Sources and further reading

The papers, reports and projects the article draws on. The numbers match the markers in the text.

1. McCulloch, W. S. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115-133. https://doi.org/10.1007/BF02478259
2. Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533-536. https://doi.org/10.1038/323533a0
3. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735
4. Bahdanau, D., Cho, K. and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. https://arxiv.org/abs/1409.0473
5. Vaswani, A. et al. (2017). Attention is all you need. https://arxiv.org/abs/1706.03762
6. Firth, J. R. (1957). A synopsis of linguistic theory, 1930-1955. In Studies in Linguistic Analysis, Philological Society, Oxford, 1-32.
7. Templeton, A. et al., Anthropic (2024). Scaling monosemanticity: extracting interpretable features from Claude 3 Sonnet. https://transformer-circuits.pub/2024/scaling-monosemanticity/
8. Mikolov, T. et al. (2013). Efficient estimation of word representations in vector space. https://arxiv.org/abs/1301.3781
9. Kaplan, J. et al. (2020). Scaling laws for neural language models. https://arxiv.org/abs/2001.08361
10. Levesque, H., Davis, E. and Morgenstern, L. (2012). The Winograd schema challenge. Proceedings of KR 2012.
11. Clark, K. et al. (2019). What does BERT look at? An analysis of BERT's attention. https://arxiv.org/abs/1906.04341
12. Geva, M. et al. (2021). Transformer feed-forward layers are key-value memories. https://arxiv.org/abs/2012.14913
13. Devlin, J. et al. (2018). BERT: pre-training of deep bidirectional transformers for language understanding. https://arxiv.org/abs/1810.04805
14. Brown, T. et al. (2020). Language models are few-shot learners (GPT-3). https://arxiv.org/abs/2005.14165
15. Gage, P. (1994). A new algorithm for data compression. The C Users Journal, 12(2).
16. Sennrich, R., Haddow, B. and Birch, A. (2016). Neural machine translation of rare words with subword units. https://arxiv.org/abs/1508.07909
17. Radford, A. et al. (2019). Language models are unsupervised multitask learners (GPT-2). OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
18. DeepSeek-AI (2024). DeepSeek-V3 technical report. https://arxiv.org/abs/2412.19437
19. Common Crawl, an open repository of web crawl data maintained by a non-profit since 2008. https://commoncrawl.org
20. Llama Team, AI at Meta (2024). The Llama 3 herd of models. https://arxiv.org/abs/2407.21783
21. Penedo, G. et al. (2024). The FineWeb datasets: decanting the web for the finest text data at scale. https://arxiv.org/abs/2406.17557
22. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 53: obligations for providers of general-purpose AI models. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
23. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. https://arxiv.org/abs/2203.02155
24. DeepSeek-AI (2025). DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. https://arxiv.org/abs/2501.12948
25. Cottier, B. et al., Epoch AI (2024). The rising costs of training frontier AI models. https://arxiv.org/abs/2405.21015
26. Kingma, D. P. and Ba, J. (2014). Adam: a method for stochastic optimization. https://arxiv.org/abs/1412.6980
27. PyTorch. https://pytorch.org
28. JAX. https://docs.jax.dev
29. Hugging Face Transformers. https://huggingface.co/docs/transformers
30. Sentence-Transformers. https://sbert.net
31. vLLM. https://docs.vllm.ai
32. llama.cpp. https://github.com/ggml-org/llama.cpp
33. Ollama. https://ollama.com
34. LlamaIndex. https://docs.llamaindex.ai
35. LangChain. https://docs.langchain.com
36. Meta Llama. https://www.llama.com