Tiny-LLM

Summary

Tiny-LLM is an extremely small functional language model with just 10-13 million parameters, built on the Llama architecture and trained on 32B tokens of the FineWeb dataset with a context length of 1024 tokens. Despite its tiny size, it is functional for text generation and serves as an educational/experimental resource for understanding LLM fundamentals, with ~37K monthly downloads and numerous community fine-tunes, adapters, and quantizations.

Keywords

tiny-lm small-model llama fineweb educational experimental text-generation lightweight

Topics

NLP / Language Modeling

Research notes

• Model checkpoint, not a dataset. One of the smallest functional LLMs available. Has 6 finetunes, 1 adapter, and 4 quantizations on HuggingFace.