Misguided Attention: Challenging the Reasoning Ability of LLMs (Reddit discussion)

Summary

This Reddit post on r/LocalLLaMA discusses the 'Misguided Attention' project (GitHub: cpldcpu/MisguidedAttention), a collection of prompts that are slight variations of commonly known thought experiments, riddles, and paradoxes. The prompts challenge LLM reasoning by introducing small modifications to well-known problems; many LLMs mistakenly recognize the unmodified problem from training data and respond with the memorized solution rather than reasoning through the modified version step-by-step. The project includes an evaluation benchmark tracking LLM performance over time, with reasoning models (e.g. o1, QwQ) showing improved but still imperfect performance.

Keywords

reasoning llm-evaluation riddles logic-puzzles benchmark misguided-attention trick-questions

Topics

LLM Evaluation / Reasoning

Research notes

• The Reddit URL itself is a discussion post (often blocked by Reddit's bot protection). The actual resource is the cpldcpu/MisguidedAttention GitHub repo. Also mirrored on Codeberg (philosopher033427615/MisguidedAttention). Discussed on Hacker News (item 42473891).