Would You Let an LLM Close a Highway? Evaluating GenAI-Powered Road Closures at Waze

For Waze, accurate road closure data is critical. Minor errors can cause failed routes and erode user trust. One source of this data is public traffic updates, which we translate into road closures on the map using an LLM-based pipeline. Getting the right output was only the beginning. Trusting it in production was the real challenge. This talk is about what that took: the evaluation we built, where our assumptions broke, and why it turned out to matter more than the prompt.