Sources and further reading

Numbered in order of first appearance in this article.

  1. Common Crawl, an open repository of web crawl data maintained by a non-profit since 2008. https://commoncrawl.org
  2. Llama Team, AI at Meta (2024). The Llama 3 herd of models. https://arxiv.org/abs/2407.21783
  3. Penedo, G. et al. (2024). The FineWeb datasets: decanting the web for the finest text data at scale. https://arxiv.org/abs/2406.17557
  4. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 53: obligations for providers of general-purpose AI models. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  5. Radford, A. et al. (2019). Language models are unsupervised multitask learners (GPT-2). OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
  6. Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533-536. https://doi.org/10.1038/323533a0
  7. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. https://arxiv.org/abs/2203.02155
  8. DeepSeek-AI (2025). DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. https://arxiv.org/abs/2501.12948
  9. NVIDIA H100 SXM specifications: 16,896 CUDA cores, 528 tensor cores, 989 dense BF16 teraflops, 3.35 TB/s HBM3, 700 W (compiled by G. Lockwood). https://glennklockwood.com/garden/processors/H100
  10. Kaplan, J. et al. (2020). Scaling laws for neural language models. https://arxiv.org/abs/2001.08361
  11. Together AI. LLM inference: what determines speed (memory bandwidth ceiling and worked examples). https://www.together.ai/blog/llm-inference
  12. AMD EPYC 9005 series: up to 192 cores, twelve DDR5 memory channels per processor (Kingston server memory guide). https://www.kingston.com/en/memory/server-memory/turin
  13. Brown, T. et al. (2020). Language models are few-shot learners (GPT-3). https://arxiv.org/abs/2005.14165
  14. DeepSeek-AI (2024). DeepSeek-V3 technical report. https://arxiv.org/abs/2412.19437
  15. Cottier, B. et al., Epoch AI (2024). The rising costs of training frontier AI models. https://arxiv.org/abs/2405.21015
  16. Kingma, D. P. and Ba, J. (2014). Adam: a method for stochastic optimisation. https://arxiv.org/abs/1412.6980
  17. PyTorch. https://pytorch.org
  18. JAX. https://docs.jax.dev
  19. Hugging Face Transformers. https://huggingface.co/docs/transformers
  20. Sentence-Transformers. https://sbert.net
  21. vLLM. https://docs.vllm.ai
  22. llama.cpp. https://github.com/ggml-org/llama.cpp
  23. Ollama. https://ollama.com
  24. LlamaIndex. https://docs.llamaindex.ai
  25. LangChain. https://docs.langchain.com
  26. Meta Llama. https://www.llama.com