Scale LLM tasks to thousands of items without a giant prompt! Generative AI can feel like magic: you hand the model everything it might need and let it do its thing. It works, until the number of items grows, and every call gets slower and more expensive. There is a subtler catch: too many similar items make the model's responses inconsistent, even before the context length becomes a concern - and we have the numbers to prove it. At a few thousand items, "hand it everything" becomes less attractive as an option. This talk describes a comparative experiment ran by our team at Ai2 on 3 LLM tasks requiring scale-up to thousands of items. We will discuss techniques to reduce the number of items required to be in context while preserving quality: diverse sampling, incremental labeling, and clustered context. We will also look at how to evaluate such cases when no golden labels are available. Each technique is accompanied with experiment results for a real use-case.
Research Engineer at Ai2