The simplest neural language model is almost embarrassingly direct. Look at the last $N$ tokens. Embed each one through a shared lookup table. Concatenate the embeddings into one vector. Run it through a small MLP that outputs a distribution over the next token. That is the entire model, and it is roughly what Bengio and coauthors introduced in 2003. It is the model that put neural language modeling on the map, years before recurrent networks and then transformers took over from n-gram counts.
It is worth building because it makes one prior completely explicit.
A fixed window is the Markov assumption, in hardware
The model can only see $N$ tokens back. Anything older is gone, not because the optimizer failed to learn it but because there is no wire for it to travel on. This is the order-$N$ Markov assumption, turned from a modeling approximation into a structural fact of the architecture. Choose $N$ too small and the model cannot represent long-range agreement; choose it too large and the MLP head grows linearly in $N$ and starts to overfit.
A hybrid prior worth noticing
The interesting part is how Bengio handles position. The embedding table is shared across positions: the token “the” gets the same vector whether it sits at slot 1 or slot 5. That is weight sharing across positions, the same idea as the convolution: shift the tokens and their embeddings shift with them, which makes the embedding layer equivariant. But then the concatenation step makes the head position sensitive. Token 1’s embedding always lands in the first slice of the input vector, token 2’s in the second, and so on, so the MLP can learn that the immediately preceding token matters more than the one $N$ back. A word means the same thing wherever it sits, for free, and the model still uses word order. One layer is equivariant, the next is deliberately not.
What dropping recurrence buys
It is still worth keeping in the toolbox for what it avoids. There is no recurrence, so there is no backpropagation through time and no vanishing-gradient problem (the subject of the next post). Every training example, a window and its next token, is independent of every other, so training parallelizes trivially and there is no per-sequence bookkeeping.
On character-level text (the opening of Alice in Wonderland, which I also use for the recurrent model so the comparison is fair) the fixed-context model reaches the same loss as the recurrent network in roughly a quarter of the epochs. Same endpoint, less effort, because there is no unrolled chain to push gradients back through.
What you give up is the cutoff. The model genuinely cannot use a token from a hundred positions back, where a recurrent network at least has a path for it (a thin and leaky one, as the next post shows). It is the right model when the context that matters is small and you know how small. When it is not, you reach for an architecture with unbounded reach, and pay for that reach in other ways.
The embedding and concatenation code, the parameter accounting, and the side-by-side Alice experiment against the recurrent model are in Chapter 4 of the book, Inductive Biases in Neural Networks.
Watch the Episode
This post is also a video episode: Bengio's Language Model: The Markov Assumption Made Architectural, part of the series playlist.
Discussion