The Infinite Table
Part 1 of What Your RL Algorithm Actually Assumes — tabular Q-learning makes zero assumptions about state similarity and pays for it in sample complexity.
1
of 4
← All Series
What Your RL Algorithm Actually Assumes
A four-part interactive series exploring how representation choices — tabular, linear, neural — shape what reinforcement learning agents can learn, and what they silently bet on.
Every RL algorithm is an approximation. The question is: an approximation of what, and at what cost?
This series uses interactive gridworld widgets to make the tradeoffs visible. Train agents in your browser, toggle features on and off, and watch how representation choices determine what an agent can learn — before a single gradient step is taken.
The Posts
The Infinite Table — Tabular Q-learning: the honest baseline that makes zero assumptions about state similarity and pays for it in sample complexity.
The Features You Choose Are the Assumptions You Make — Linear function approximation: how hand-crafted features compress the state space, and what you’re betting on when you pick them.
The Architecture Is the Prior — Neural function approximation: the architecture decides what kind of features can be learned, and that decision is a Bayesian prior over value functions.
What You Assume vs. What You Compute — Model-based vs. model-free, the assumptions table, AIXI as the incomputable ideal, and the unifying claim: representation is prior is assumption.
Posts in this Series
Showing 4 of 4 posts
2
of 4
The Features You Choose Are the Assumptions You Make
Part 2 of What Your RL Algorithm Actually Assumes — how hand-crafted features compress the state space, and what you're betting on when you pick them.
3
of 4
The Architecture Is the Prior
Part 3 of What Your RL Algorithm Actually Assumes — the architecture decides what kind of features can be learned, and that decision is a Bayesian prior over value functions.
4
of 4
What You Assume vs. What You Compute
Part 4 of What Your RL Algorithm Actually Assumes — model-based vs. model-free, the assumptions table, AIXI as the incomputable ideal, and the unifying claim: representation is prior is assumption.