LLM-PuzzleTest (PuzzleVQA and AlgoPuzzleVQA)

Summary

DECLARE Lab repo for two generated VQA suites. PuzzleVQA (arXiv 2403.13315): abstract visual patterns over colors/numbers/sizes/shapes, single- and dual-concept templates with gold perception/induction/deduction explanations; GPT-4V ~46% on single-concept. AlgoPuzzleVQA (arXiv 2403.03864): 18 puzzle types x 100 instances = 1800, covering boolean logic, combinatorics, graphs, optimization, search, sets; answers from human-authored algorithms so difficulty/size can scale. Later tracking paper arXiv 2502.01081 compares GPT-n/o-n on open-ended vs MCQ. Repo MIT; AlgoPuzzleVQA HF Apache-2.0; PuzzleVQA HF card has no license tag.

Keywords

puzzlevqa · algopuzzlevqa · multimodal · vqa · declare-lab · abstract-reasoning

Topics

multimodal reasoning, visual puzzles, VQA

Research notes

• Primary: GitHub README + HF cards declare-lab/PuzzleVQA and declare-lab/AlgoPuzzleVQA. Papers 2403.13315, 2403.03864, 2502.01081 not previously in papers_local. Discord posted the GitHub repo.