CoLoTa: A Dataset for Entity-based Commonsense Reasoning over Long-Tail Knowledge
- Type
- repo
- Venue
- SIGIR 2025 / D3Mlab (University of Toronto)
- Year
- 2025
- Source
- github
- Access
- free
- Language
- en
- Added
- 2026-08-14T20:50:00Z
- Verified
- 2026-08-14T20:50:00Z
Summary
Rewrites StrategyQA questions and CREAK claims by swapping head entities for long-tail Wikidata counterparts (popularity = number of triples), then annotates QIDs, relevant triples, a natural-language inference rule, and grounded reasoning steps. 3300 queries: 1650 QA + 1650 claim verification (CoLoTa_qa.json / CoLoTa_cv.json). o1 accuracy drops ~0.18-0.22 vs original popular-entity queries while answer rate stays high, implying more hallucinations. LLM-based KGQA (KGR, KB-Binder) near-collapse on CoLoTa. GitHub API license null; no LICENSE in README.
Keywords
colota · long-tail · commonsense · kgqa · wikidata · sigir · d3mlab
Topics
commonsense reasoning, long-tail entities, KGQA, hallucination
Research notes
- Primary: GitHub README + arxiv abs 2504.14462 (SIGIR 2025). License not stated on GitHub API or README (blank). Dataset not in datasets_local. Discord posted the GitHub repo.