"We need domain experts." Everyone agreed. Nobody knew what we should actually ask them... While building AI financial and business advisors at Intuit, we learned that the hardest problem wasn't prompting the LLM, it was figuring out how to capture our experts' knowledge in a way that actually improves the product, benefits customers, and doesn't exhaust our advisors (or ourselves). In five fast minutes I'll share the experiments that surprised us: why our "golden answer" wasn't so golden, why comparing AI responses beat writing answers from scratch, how disagreement became useful data, and why a simple yes/no question sometimes taught us more than an expert paragraph. Most importantly, how we realized we weren't looking for one perfect answer, we were discovering what "good advice" actually means. Whether your experts are accountants, doctors, lawyers or your company's "person who knows everything," you'll leave with practical ideas for turning human expertise into better AI.
Staff AI Scientist at Intuit