No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow

Summary

Defines Corpus Task Complexity (CTC) and shows that conclusions from ordinary low-CTC long-context evaluations can reverse on high-CTC tasks. Adds 10 high-CTC tasks to a 22-task evaluation suite. Full attention remains stronger on high-CTC tasks but is costly to scale.

Keywords

evaluation · long-context · benchmarks · attention · corpus-task-complexity

Topics

evaluation, long-context, benchmarks, attention, corpus-task-complexity

Research notes

• Discovery: Posted in #random-papers on 2026-09-28; tweet by paper co-author Prasann Singhal. Canonical paper located via arXiv search: 2609.29245.
• Method: Defines Corpus Task Complexity as a measure of how hard it is to extract an answer from a corpus; constructs a 22-task suite plus 10 high-CTC tasks and compares architecture conclusions under low vs. high CTC.
• Key findings: High-CTC results can reverse conclusions drawn from ordinary low-CTC long-context evals; full attention is stronger on high-CTC tasks but expensive to scale.
• Limitations: Code/data identifier shown in the paper is 'PrasannS/corpustaskcomplexity'; the exact GitHub URL was not verified, so code_url is left null. Arxiv page fetch hit a 429 rate limit; metadata comes from search results.
• CTC-Bench: a benchmark paper relevant to long-context evaluation debates. Code identifier unverified — needs a follow-up check before linking in the sheet.