Language Models

The Unreasonable Effectiveness of Recurrent Neural Networks

Review

Seminal blog post demonstrating the power of character-level RNNs. Shows Shakespeare generation, Wikipedia generation, LaTeX generation, and Linux kernel code generation. The visualizations of LSTM cells are particularly illuminating.

Infinigram: Corpus-Based Language Modeling via Suffix Arrays with LLM Probability Mixing

Infinigram: Variable-Length N-grams via Suffix Arrays

A corpus-based language model using suffix arrays for O(m log n) pattern matching. The corpus is the model.

Language Calculus: An Algebraic Framework for LLM Composition

A mathematical framework that treats language models as algebraic objects with compositional structure.

Neural Language Models: From RNNs to Transformers

The evolution of neural sequence prediction, and how it connects to classical methods

Comparing Prediction Methods: CTW vs. N-grams vs. Neural LMs

The bias-data trade-off in sequential prediction: when to use CTW, n-grams, or neural language models.

N-gram Language Models

The classical approach to sequence prediction: counting and smoothing