Click-100k

Summary

Click-100k is a 101k-example GUI grounding dataset that pairs computer screen frames with low-level GUI commands and click coordinates, used to train Gelato-30B-A3B, a state-of-the-art grounding model for computer-use tasks. It was built by filtering and unifying multiple public datasets (ShowUI, AutoGUI, PC-Agent-E, WaveUI, OS-Atlas, UGround, PixMo Points, SeeClick, UI-VISION, Jedi) plus 85 professional application tutorial videos annotated with Claude 4 Sonnet, with aggressive noise filtering using OmniParser and Qwen2.5-7B-VL. Gelato achieves 63.88% accuracy on ScreenSpot-Pro and 69.15% on OS-World-G.

Keywords

gui-grounding computer-use vision ui-interaction click-coordinates agent gelato

Topics

Vision / GUI Agents

Research notes

• Associated with arxiv papers 2411.17465, 2502.01977, 2504.07981 and others. Code at github.com/mlfoundations/gelato. Maximum 50k instances per source to prevent dataset skewing.