OLMo-2-0425-1B

Summary

OLMo-2-0425-1B is the smallest model in the OLMo 2 (Open Language Model) family by Allen AI (Ai2), pre-trained on 4 trillion tokens from the OLMo-mix-1124 dataset and using Dolmino-mix-1124 for mid-training. It has 16 layers, 2048 hidden size, 16 attention heads, and a 4096 context length. The OLMo 2 family is designed to enable the science of language models by releasing all code, checkpoints, logs, and training details openly. This 1B base model also has SFT, DPO, and Instruct (RLVR) variants.

Keywords

olmo llm open-source pretraining allenai language-model 1b

Topics

NLP / Language Models

Research notes

• Part of the OLMo 2 family which includes 1B, 7B, 13B, and 32B variants. The 1B model uses the 0425 checkpoint date while larger models use 1124 or 0325 dates.