Fisher Flow: A Unified Information-Geometric Perspective on Sequential Maximum Likelihood Estimation and Natural Gradient Methods

Alex Towell
atowell@siue.edu

(May 19, 2025)

Abstract

We present a unified information-geometric perspective on sequential maximum likelihood estimation that we term Fisher Flow. Rather than proposing an entirely new method, we provide a systematic framework that reveals how classical recursive estimation, natural gradient descent, and modern adaptive optimizers (Adam, K-FAC, EWC) are instances of a common principle: propagating Fisher information through sequential data. This perspective yields three contributions: (1) we formalize the “two regimes” interpretation of information geometry in machine learning—classical asymptotic theory when optimization converges versus trajectory-dependent regularization in modern overparameterized settings; (2) we provide a taxonomy of computational approximations to full Fisher information with rigorous error characterization; and (3) we propose several novel algorithmic variants with theoretical motivation. We validate our framework through systematic experiments on synthetic and real-world problems, demonstrating 6–9% accuracy improvements on challenging classification tasks (p¡0.001) with minimal computational overhead ( 18%). While the core mathematical machinery is well-established (building on Amari’s natural gradient and classical MLE theory), our contribution is the unification, empirical validation, and reinterpretation that makes implicit patterns explicit and actionable.

1 Introduction

1.1 Motivating Example: Online Linear Regression

Consider a streaming data scenario where we observe pairs $(x_{t},y_{t})$ sequentially and wish to estimate parameters $\theta$ of a linear model $y=x^{\top}\theta+\epsilon$ with $\epsilon\sim\mathcal{N}(0,\sigma^{2})$ .

Bayesian approach: Maintain posterior $p(\theta|\text{data}_{1:t})$ , requiring $\mathcal{O}(d^{2})$ storage and $\mathcal{O}(d^{3})$ computation per update.

Fisher Flow approach: Maintain only $(\hat{\theta}_{t},\mathcal{I}_{t})$ where:

	$\displaystyle\mathcal{I}_{t}$	$\displaystyle=\mathcal{I}_{t-1}+\frac{1}{\sigma^{2}}x_{t}x_{t}^{\top}$	(information update)		(1)
	$\displaystyle\hat{\theta}_{t}$	$\displaystyle=\hat{\theta}_{t-1}+\mathcal{I}_{t}^{-1}x_{t}(y_{t}-x_{t}^{\top}% \hat{\theta}_{t-1})/\sigma^{2}$	(parameter update)		(2)

Both approaches yield identical point estimates and uncertainty quantification for Gaussian models, but Fisher Flow extends naturally to non-Gaussian likelihoods where Bayesian updates lack closed forms.

1.2 Problem Statement and Motivation

The Challenge: Modern machine learning requires methods that can:

1.

Process streaming data with bounded memory
2.

Quantify uncertainty in predictions
3.

Scale to billions of parameters
4.

Combine information from distributed sources
5.

Adapt to non-stationary distributions

Bayesian inference addresses (2) but struggles with (1), (3), and (4). Stochastic gradient methods handle (1) and (3) but lack principled uncertainty quantification.

Our Solution: Fisher Flow bridges this gap by propagating Fisher information—a quadratic approximation to the log-posterior curvature—rather than full distributions. This provides uncertainty estimates while maintaining computational efficiency.

We formalize Fisher Flow (FF), a framework that operates on the statistical manifold $\mathcal{M}=\{p_{\theta}:\theta\in\Theta\}$ equipped with the Fisher-Rao metric. Rather than propagating probability distributions, Fisher Flow propagates Fisher information—the fundamental geometric quantity encoding statistical distinguishability. This shift from measure-theoretic to geometric foundations yields:

•

Geometric invariance: Updates are covariant under reparameterization
•

Information optimality: Achieves the Cramér-Rao efficiency bound
•

Algebraic closure: Information combines additively across data batches
•

Computational tractability: Reduces to matrix operations even for complex models

1.3 Contributions

This work provides a unifying perspective with the following specific contributions:

1.

Conceptual unification: We show that natural gradient descent [1], adaptive methods like Adam [10], Kronecker-factored approximations [13], and continual learning methods like EWC [11] are all instances of sequential Fisher information propagation with different approximation structures. While these connections are implicit in prior work, we make them explicit and systematic.
2.

Two-regime formalization: We distinguish two regimes. In the classical regime, optimization converges and standard MLE theory applies. In the modern ML regime, training stops before convergence and Fisher information acts as trajectory-dependent regularization. This distinction clarifies when classical guarantees hold.
3.

Approximation taxonomy with error bounds: We characterize the approximation error for diagonal, block-diagonal, Kronecker-factored, and low-rank Fisher information structures (Theorems 18, 34, 35), providing guidance for trading computational cost against statistical efficiency.
4.

Novel algorithmic variants: We propose momentum-enhanced, adaptively compressed, and Byzantine-robust extensions of natural gradient methods, with theoretical motivation (though empirical validation remains future work).

What is not claimed as novel: The core mathematical framework (recursive MLE, information geometry, natural gradient theory) is well-established [1, 2]. Our contribution is synthesis, reinterpretation, and extension rather than fundamentally new theory.

2 Mathematical Foundations

2.1 Notation and Preliminaries

We work with a parametric family $\{p(x|\theta):\theta\in\Theta\subseteq\mathbb{R}^{d}\}$ . Key notation:

Symbol	Definition
$\ell_{n}(\theta)$	Log-likelihood: $\sum_{i=1}^{n}\log p(x_{i}\|\theta)$
$s_{n}(\theta)$	Score (gradient): $\nabla_{\theta}\ell_{n}(\theta)$
$\mathcal{I}(\theta)$	Expected Fisher Information: $\mathbb{E}[s(\theta)s(\theta)^{\top}]$
$\hat{\mathcal{I}}_{n}(\theta)$	Observed Fisher Information: $-\nabla^{2}_{\theta}\ell_{n}(\theta)$
$\hat{\theta}_{n}$	FF estimate after $n$ observations
$\mathcal{I}_{n}$	Accumulated information after $n$ observations

We consistently use $\mathcal{I}$ for expected and $\hat{\mathcal{I}}$ for observed information.

2.1.1 Score and Information Notation

For clarity, we standardize the following notation.

•

Per-observation score: $s_{i}(\theta):=\nabla_{\theta}\log p(x_{i}\mid\theta)$ for observation $i$
•

Cumulative score after $n$ observations: $s_{n}(\theta):=\sum_{i=1}^{n}s_{i}(\theta)$
•

Batch score for batch $B$ containing observations $\{i_{1},\ldots,i_{k}\}$ : $s_{B}(\theta):=\sum_{j\in B}s_{j}(\theta)$
•

Expected Fisher information: $\mathcal{I}(\theta):=\mathbb{E}\big{[}s(\theta)s(\theta)^{\top}\big{]}$
•

Observed Fisher information: $\hat{\mathcal{I}}(\theta):=-\nabla_{\theta}^{2}\ell(\theta)$
•

Sequential information accumulation after $n$ observations: $\mathcal{I}_{n}=\sum_{i=1}^{n}\hat{\mathcal{I}}_{i}$
•

When processing in batches: After $t$ batches with $n_{t}=\sum_{k=1}^{t}|B_{k}|$ total observations, $\mathcal{I}_{n_{t}}=\sum_{k=1}^{t}\hat{\mathcal{I}}_{B_{k}}$

Unless stated otherwise, $\mathcal{I}$ denotes expected Fisher information and $\hat{\mathcal{I}}$ denotes observed (empirical) information evaluated at the parameter specified in context. Subscript $n$ always denotes the total number of observations seen, while subscript $t$ (when used) indexes batch iterations.

2.2 Statistical Manifolds and Information Geometry

Definition 1 (Statistical Manifold).

A statistical manifold is a Riemannian manifold $(\mathcal{M},g)$ where:

•

$\mathcal{M}=\{p_{\theta}:\theta\in\Theta\subseteq\mathbb{R}^{d}\}$ is a parametric family
•

$g$ is the Fisher-Rao metric tensor with components $g_{ij}(\theta)=\mathcal{I}_{ij}(\theta)$

Definition 2 (Fisher Information Matrix).

For a parametric family $\{p(x|\theta)\}_{\theta\in\Theta}$ , the Fisher Information Matrix $\mathcal{I}(\theta)$ is defined as:

\mathcal{I}_{ij}(\theta)=\mathbb{E}_{p(x|\theta)}\left[\frac{\partial\log p(x|% \theta)}{\partial\theta_{i}}\frac{\partial\log p(x|\theta)}{\partial\theta_{j}% }\right]=-\mathbb{E}_{p(x|\theta)}\left[\frac{\partial^{2}\log p(x|\theta)}{% \partial\theta_{i}\partial\theta_{j}}\right]

(3)

under regularity conditions ensuring the interchange of differentiation and integration.

Table 1: Core Mathematical Objects in Fisher Flow

Symbol	Definition	Geometric Interpretation
$p(x\|\theta)$	Likelihood function	Point on manifold $\mathcal{M}$
$\ell(\theta;x_{1:n})=\sum_{i=1}^{n}\log p(x_{i}\|\theta)$	Log-likelihood	Potential function on $\mathcal{M}$
$s(\theta)=\nabla_{\theta}\ell(\theta)$	Score function	Tangent vector in $T_{\theta}\mathcal{M}$
$\mathcal{I}(\theta)=\mathbb{E}[s(\theta)s(\theta)^{\top}]$	Expected FIM	Metric tensor $g(\theta)$
$\hat{\mathcal{I}}(\theta)=-\nabla^{2}_{\theta}\ell(\theta)$	Observed FIM	Hessian of potential
$\Gamma^{k}_{ij}$	Christoffel symbols	Levi-Civita connection

2.3 Regularity Conditions

Assumption 3 (Regularity).

The parametric family $\{p(x|\theta)\}$ satisfies:

1.

Identifiability: $\theta\neq\theta^{\prime}\implies p(\cdot|\theta)\neq p(\cdot|\theta^{\prime})$ almost everywhere
2.

Differentiability: $\theta\mapsto\log p(x|\theta)$ is thrice continuously differentiable
3.

Fisher regularity: $\int\nabla_{\theta}p(x|\theta)dx=\nabla_{\theta}\int p(x|\theta)dx=0$
4.

Finite Fisher information: $0<\mathcal{I}(\theta)<\infty$ for all $\theta\in\text{int}(\Theta)$

2.4 Information Accumulation and the Additive Property

Theorem 4 (Information Additivity).

For independent observations $x_{1},\ldots,x_{n}$ from $p(x|\theta_{0})$ , the Fisher information satisfies:

\mathcal{I}_{1:n}(\theta)=\sum_{i=1}^{n}\mathcal{I}_{x_{i}}(\theta)

(4)

where $\mathcal{I}_{x_{i}}(\theta)$ denotes the Fisher information from observation $x_{i}$ .

Proof.

By independence, $p(x_{1},\ldots,x_{n}|\theta)=\prod_{i=1}^{n}p(x_{i}|\theta)$ . Thus:

$\displaystyle\ell(\theta;x_{1:n})$	$\displaystyle=\sum_{i=1}^{n}\log p(x_{i}\|\theta)$	(5)
$\displaystyle\nabla^{2}_{\theta}\ell(\theta;x_{1:n})$	$\displaystyle=\sum_{i=1}^{n}\nabla^{2}_{\theta}\log p(x_{i}\|\theta)$	(6)
$\displaystyle\mathcal{I}_{1:n}(\theta)$	$\displaystyle=-\mathbb{E}[\nabla^{2}_{\theta}\ell(\theta;x_{1:n})]=\sum_{i=1}^% {n}\mathcal{I}_{x_{i}}(\theta)\qed$	(7)

3 Fisher Flow in Plain English: The Core Insight

3.1 The Fundamental Pattern

Forget the mathematical machinery for a moment. Here’s what Fisher Flow actually does:

The Problem: You’re estimating unknown parameters from data that arrives piece by piece. You want to know both your best guess AND how confident you should be about that guess.

The Insight: Instead of tracking all possible parameter values and their probabilities (expensive!), just track two things:

1.

Your current best guess
2.

A ”confidence matrix” that says how sure you are

The magic is that when new data arrives, you can update both using simple matrix arithmetic—no complex integration required.

3.2 A Simple Analogy: The Wisdom of Crowds

Imagine you’re trying to guess the number of jellybeans in a jar:

•

Person A guesses 500, and they’re usually accurate within ±50
•

Person B guesses 450, and they’re usually accurate within ±100
•

Person C guesses 480, and they’re usually accurate within ±30

How do you combine these estimates? You weight them by confidence:

\text{Best guess}=\frac{500\times\frac{1}{50^{2}}+450\times\frac{1}{100^{2}}+4% 80\times\frac{1}{30^{2}}}{\frac{1}{50^{2}}+\frac{1}{100^{2}}+\frac{1}{30^{2}}}% \approx 487

Fisher Flow does exactly this, but for model parameters. The ”confidence” is the Fisher Information—essentially measuring how sharply the likelihood peaks around the best estimate.

3.3 Why This Matters: The Power of a Name

Before Fisher Flow had a name, people were:

•

Using ”approximate Bayesian methods” (but they weren’t really Bayesian)
•

Calling it ”recursive estimation” (missing the geometric insight)
•

Implementing ”adaptive learning rates” (not realizing they were approximating Fisher information)
•

Developing ”second-order methods” (without the unifying principle)

By recognizing and naming the pattern—propagating information rather than distributions—we suddenly see:

1.

Adam is diagonal Fisher Flow: Those running averages of squared gradients? They’re estimating diagonal Fisher information!
2.

Natural gradient is exact Fisher Flow: Using the full Fisher Information Matrix
3.

Elastic Weight Consolidation is Fisher Flow memory: Remembering important parameters through their information
4.

Kalman filtering is linear Fisher Flow: The classical algorithm is just Fisher Flow for linear-Gaussian models

3.4 The Fisher Flow Taxonomy: A Family of Methods

Once we recognize the pattern, we can systematically explore variations:

The Fisher Flow Family Tree

•
By Information Structure:
- –
  
  Scalar FF: One learning rate for all parameters (SGD)
- –
  
  Diagonal FF: Per-parameter learning rates (Adam, RMSprop)
- –
  
  Block FF: Groups of parameters share information (Layer-wise methods)
- –
  
  Structured FF: Exploit model structure (Kronecker-factored)
- –
  
  Full FF: Complete information matrix (Natural gradient)
•
By Time Dynamics:
- –
  
  Stationary FF: Information accumulates forever
- –
  
  Windowed FF: Only recent information matters
- –
  
  Exponential FF: Gradual forgetting (moving averages)
- –
  
  Adaptive FF: Change detection triggers reset
•
By Approximation Type:
- –
  
  Monte Carlo FF: Sample-based information estimates
- –
  
  Factored FF: Assume independence between groups
- –
  
  Low-rank FF: Capture dominant directions only
- –
  
  Sparse FF: Only track significant interactions

3.5 The Deeper Pattern: Information as Currency

The real breakthrough is recognizing that information is the natural currency of learning:

•

Data provides information about parameters
•

Information accumulates additively (like money in a bank)
•

Confidence is inverse variance (more information = less uncertainty)
•

Different data sources contribute different amounts of information

This shift in perspective—from thinking about probability distributions to thinking about information accumulation—simplifies everything:

Traditional View	Fisher Flow View	Benefit
Update posterior	Add information	Linear algebra
Marginalize	Project	Matrix multiplication
Sample from posterior	Perturb by $\mathcal{I}^{-1/2}$	Gaussian sampling
Compute credible intervals	Invert information	Matrix inversion

3.6 When to Use What: A Practical Guide

The Fisher Flow framework helps us choose methods systematically:

Few parameters, lots of data? → Full Fisher Flow (natural gradient) Many parameters, limited memory? → Diagonal Fisher Flow (Adam) Neural network layers? → Kronecker Fisher Flow (K-FAC) Continual learning? → Fisher Flow with memory (EWC) Online learning? → Exponential forgetting FF Distributed training? → Aggregate local information matrices

The beauty is that these aren’t ad-hoc choices—they’re principled approximations of the same underlying concept.

4 The Fisher Flow Framework

4.1 Axiomatic Foundation

We axiomatize Fisher Flow through three fundamental principles:

Axiom 5 (Information Monotonicity).

For any data sequence $x_{1},x_{2},\ldots$ , the accumulated information $\mathcal{I}_{t}$ is non-decreasing: $\mathcal{I}_{t+1}\succeq\mathcal{I}_{t}$ (in the positive semi-definite ordering).

Axiom 6 (Geometric Covariance).

Parameter updates are covariant under smooth reparameterizations: if $\phi=f(\theta)$ is a diffeomorphism, then updates in the $\phi$ -parameterization preserve the geometric structure.

Axiom 7 (Local Sufficiency).

Updates depend only on local geometric quantities (score and curvature) at the current parameter value.

4.2 Core Update Equations

Definition 8 (Fisher Flow State).

The state of the Fisher Flow system at time $t$ is the tuple $(\hat{\theta}_{t},\mathcal{I}_{t})$ where:

•

$\hat{\theta}_{t}\in\Theta$ is the current parameter estimate
•

$\mathcal{I}_{t}\in\mathbb{S}^{d}_{++}$ is the accumulated Fisher information matrix

Theorem 9 (Natural Gradient Flow).

The Fisher Flow update equation

\hat{\theta}_{t+1}=\hat{\theta}_{t}-\eta_{t}\mathcal{I}_{t}^{-1}s_{t}(\hat{% \theta}_{t})

(8)

defines a discrete-time approximation to the natural gradient flow:

\frac{d\theta}{dt}=-\mathcal{I}(\theta)^{-1}\nabla_{\theta}\ell(\theta)

(9)

on the statistical manifold $\mathcal{M}$ .

Proof.

The natural gradient $\tilde{\nabla}$ is defined as the gradient with respect to the Fisher-Rao metric:

\tilde{\nabla}_{\theta}\ell=\mathcal{I}(\theta)^{-1}\nabla_{\theta}\ell=% \mathcal{I}(\theta)^{-1}s(\theta)

(10)

This defines a Riemannian (natural) gradient flow on $\mathcal{M}$ under the Fisher–Rao metric. The discrete update with learning rate $\eta_{t}$ provides a first-order approximation to this continuous flow. ∎

4.3 Information Combination and Optimality

Theorem 10 (Optimal Information Fusion).

Given independent parameter estimates $(\hat{\theta}_{A},\mathcal{I}_{A})$ and $(\hat{\theta}_{B},\mathcal{I}_{B})$ from disjoint data sets, the minimum variance unbiased combination is:

	$\displaystyle\hat{\theta}_{AB}$	$\displaystyle=(\mathcal{I}_{A}+\mathcal{I}_{B})^{-1}(\mathcal{I}_{A}\hat{% \theta}_{A}+\mathcal{I}_{B}\hat{\theta}_{B})$		(11)
	$\displaystyle\mathcal{I}_{AB}$	$\displaystyle=\mathcal{I}_{A}+\mathcal{I}_{B}$		(12)

Proof.

Consider the joint likelihood from both data sets. By independence:

\ell_{AB}(\theta)=\ell_{A}(\theta)+\ell_{B}(\theta)

(13)

The score and information combine additively:

	$\displaystyle s_{AB}(\theta)$	$\displaystyle=s_{A}(\theta)+s_{B}(\theta)$		(14)
	$\displaystyle\mathcal{I}_{AB}(\theta)$	$\displaystyle=\mathcal{I}_{A}(\theta)+\mathcal{I}_{B}(\theta)$		(15)

The combined estimate satisfies the first-order condition:

s_{A}(\hat{\theta}_{AB})+s_{B}(\hat{\theta}_{AB})\approx\mathcal{I}_{A}(\hat{% \theta}_{A}-\hat{\theta}_{AB})+\mathcal{I}_{B}(\hat{\theta}_{B}-\hat{\theta}_{% AB})=0

(16)

Solving yields the stated formula. Optimality follows from the Gauss-Markov theorem applied to the linearized system. ∎

4.4 Sequential Update Algorithm

Algorithm 1 Fisher Flow Sequential Update

1:Initial state

(\hat{\theta}_{0},\mathcal{I}_{0})

, data stream in batches

\{B_{t}\}_{t=1}^{T}

2:Final estimate

(\hat{\theta}_{N},\mathcal{I}_{N})

after

N

total observations

3:Initialize:

n\leftarrow 0

\triangleright

Total observations counter

4:for

t=1

T

\triangleright

Iterate over batches

5: Observe batch

B_{t}=\{x_{n+1},\ldots,x_{n+|B_{t}|}\}

with

|B_{t}|

observations

6: Compute batch score:

s_{B_{t}}(\theta)=\sum_{i\in B_{t}}\nabla_{\theta}\log p(x_{i}|\theta)

7: Compute batch information:

\hat{\mathcal{I}}_{B_{t}}(\theta)=\sum_{i\in B_{t}}-\nabla^{2}_{\theta}\log p(% x_{i}|\theta)

8: Find batch MLE:

\hat{\theta}_{B_{t}}=\arg\max_{\theta}\ell_{B_{t}}(\theta)

9: Update cumulative information:

\mathcal{I}_{n+|B_{t}|}=\mathcal{I}_{n}+\hat{\mathcal{I}}_{B_{t}}(\hat{\theta}% _{B_{t}})

10: Update estimate:

\hat{\theta}_{n+|B_{t}|}=\mathcal{I}_{n+|B_{t}|}^{-1}\big{(}\mathcal{I}_{n}% \hat{\theta}_{n}+\hat{\mathcal{I}}_{B_{t}}\hat{\theta}_{B_{t}}\big{)}

11: Update counter:

n\leftarrow n+|B_{t}|

12:end for

13:return

(\hat{\theta}_{N},\mathcal{I}_{N})

where

N=\sum_{t=1}^{T}|B_{t}|

5 Asymptotic Theory and Convergence Guarantees

5.1 Consistency and Asymptotic Normality

Theorem 11 (Strong Consistency of Fisher Flow).

Under Assumption 3, the Fisher Flow estimator $\hat{\theta}_{n}$ satisfies:

\hat{\theta}_{n}\xrightarrow{a.s.}\theta_{0}\quad\text{as }n\to\infty

(17)

where $\theta_{0}$ is the true parameter value.

Proof.

We establish strong consistency through a three-step argument.

Step 1: Convergence of the empirical likelihood. By the strong law of large numbers (SLLN):

\frac{1}{n}\ell_{n}(\theta)=\frac{1}{n}\sum_{i=1}^{n}\log p(x_{i}|\theta)% \xrightarrow{a.s.}\mathbb{E}[\log p(X|\theta)]=-H(\theta)

(18)

uniformly over compact sets by the uniform SLLN under regularity conditions.

Step 2: Identifiability and uniqueness. By identifiability (Assumption 3), $\theta_{0}$ is the unique maximizer of $-H(\theta)$ :

-H(\theta_{0})>-H(\theta)\quad\forall\theta\neq\theta_{0}

(19)

Step 3: Convergence of Fisher Flow updates. The Fisher Flow update satisfies:

\hat{\theta}_{n+1}=\hat{\theta}_{n}-\eta_{n}\mathcal{I}_{n}^{-1}s_{n}(\hat{% \theta}_{n})

(20)

Near $\theta_{0}$ , by Taylor expansion:

s_{n}(\hat{\theta}_{n})\approx-\hat{\mathcal{I}}_{n}(\theta_{0})(\hat{\theta}_% {n}-\theta_{0})

(21)

Thus the update becomes approximately:

\hat{\theta}_{n+1}-\theta_{0}\approx(I-\eta_{n}\mathcal{I}_{n}^{-1}\hat{% \mathcal{I}}_{n}(\theta_{0}))(\hat{\theta}_{n}-\theta_{0})

(22)

Since $\mathcal{I}_{n}/n\xrightarrow{a.s.}\mathcal{I}(\theta_{0})$ and $\hat{\mathcal{I}}_{n}/n\xrightarrow{a.s.}\mathcal{I}(\theta_{0})$ , for appropriate step sizes $\eta_{n}$ , the spectral radius of the iteration matrix converges to a value less than 1, ensuring $\hat{\theta}_{n}\xrightarrow{a.s.}\theta_{0}$ . ∎

Theorem 12 (Asymptotic Normality and Efficiency).

For the Fisher Flow estimator $\hat{\theta}_{n}$ with accumulated information $\mathcal{I}_{n}$ :

\sqrt{n}(\hat{\theta}_{n}-\theta_{0})\xrightarrow{d}\mathcal{N}(0,\mathcal{I}(% \theta_{0})^{-1})

(23)

Furthermore, if $\hat{\theta}_{n}$ coincides with the MLE (e.g., under exact information accumulation and suitable initialization), it achieves the Cramér–Rao lower bound asymptotically. More generally, if the FF estimator is a consistent, asymptotically linear one-step estimator with influence function $\mathcal{I}(\theta_{0})^{-1}s(\theta_{0})$ , the same limit holds [20].

Proof.

We provide a complete proof of asymptotic normality.

Step 1: Asymptotic expansion. The Fisher Flow estimator satisfies the implicit equation:

\sum_{i=1}^{n}s_{i}(\hat{\theta}_{n})+\mathcal{I}_{0}(\hat{\theta}_{n}-\hat{% \theta}_{0})=0

(24)

where $\mathcal{I}_{0}$ is the initial information (possibly zero).

Step 2: Linearization. By Taylor expansion around $\theta_{0}$ :

\sum_{i=1}^{n}s_{i}(\hat{\theta}_{n})=\sum_{i=1}^{n}s_{i}(\theta_{0})-\sum_{i=% 1}^{n}\hat{\mathcal{I}}_{i}(\tilde{\theta}_{n})(\hat{\theta}_{n}-\theta_{0})

(25)

for some $\tilde{\theta}_{n}$ between $\hat{\theta}_{n}$ and $\theta_{0}$ .

Step 3: Solving for the error. Rearranging:

(\hat{\theta}_{n}-\theta_{0})=\left(\sum_{i=1}^{n}\hat{\mathcal{I}}_{i}(\tilde% {\theta}_{n})+\mathcal{I}_{0}\right)^{-1}\sum_{i=1}^{n}s_{i}(\theta_{0})

(26)

Step 4: Asymptotic distribution. By the law of large numbers:

\frac{1}{n}\sum_{i=1}^{n}\hat{\mathcal{I}}_{i}(\tilde{\theta}_{n})\xrightarrow% {p}\mathcal{I}(\theta_{0})

(27)

By the central limit theorem for the score:

\frac{1}{\sqrt{n}}\sum_{i=1}^{n}s_{i}(\theta_{0})\xrightarrow{d}\mathcal{N}(0,% \mathcal{I}(\theta_{0}))

(28)

Step 5: Slutsky’s theorem. Combining the above and applying Slutsky’s theorem:

$\displaystyle\sqrt{n}(\hat{\theta}_{n}-\theta_{0})$	$\displaystyle=\left(\frac{1}{n}\sum_{i=1}^{n}\hat{\mathcal{I}}_{i}(\tilde{% \theta}_{n})+\frac{\mathcal{I}_{0}}{n}\right)^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}% ^{n}s_{i}(\theta_{0})$	(29)
	$\displaystyle\xrightarrow{d}\mathcal{I}(\theta_{0})^{-1}\cdot\mathcal{N}(0,% \mathcal{I}(\theta_{0}))$	(30)
	$\displaystyle=\mathcal{N}(0,\mathcal{I}(\theta_{0})^{-1})$	(31)

This establishes asymptotic normality and shows that Fisher Flow achieves the Cramér-Rao lower bound asymptotically. ∎

5.2 Non-Asymptotic Bounds

Theorem 13 (Finite-Sample Concentration).

Under sub-Gaussian score assumptions, with probability at least $1-\delta$ :

\|\hat{\theta}_{n}-\theta_{0}\|_{\mathcal{I}(\theta_{0})}\leq\sqrt{\frac{d\log% (2d/\delta)}{n}}+\mathcal{O}(n^{-1})

(32)

where $\|\cdot\|_{\mathcal{I}}$ denotes the Mahalanobis norm induced by $\mathcal{I}$ .

Proof.

We establish this concentration inequality via the following steps:

Step 1: Score concentration. Under sub-Gaussian assumptions, the centered score satisfies:

\mathbb{P}\left(\left\|\frac{1}{n}\sum_{i=1}^{n}s_{i}(\theta_{0})\right\|\geq t% \right)\leq 2d\exp\left(-\frac{nt^{2}}{2\sigma^{2}}\right)

(33)

where $\sigma^{2}$ is the sub-Gaussian parameter.

Step 2: Information matrix concentration. The empirical Fisher information satisfies:

\left\|\frac{1}{n}\sum_{i=1}^{n}\hat{\mathcal{I}}_{i}-\mathcal{I}(\theta_{0})% \right\|_{\text{op}}\leq C\sqrt{\frac{\log(2d/\delta)}{n}}

(34)

with probability at least $1-\delta/2$ , where $C$ depends on the sub-Gaussian parameter.

Step 3: Taylor expansion. By the mean value theorem:

\hat{\theta}_{n}-\theta_{0}=-\left(\frac{1}{n}\sum_{i=1}^{n}\hat{\mathcal{I}}_% {i}(\tilde{\theta})\right)^{-1}\frac{1}{n}\sum_{i=1}^{n}s_{i}(\theta_{0})

(35)

for some $\tilde{\theta}$ on the line segment between $\hat{\theta}_{n}$ and $\theta_{0}$ .

Step 4: Combining bounds. Using matrix perturbation theory and the concentration results from Steps 1-2:

	$\displaystyle\\|\hat{\theta}_{n}-\theta_{0}\\|_{\mathcal{I}(\theta_{0})}$	$\displaystyle\leq\left\\|\mathcal{I}(\theta_{0})^{1/2}\left(\frac{1}{n}\sum_{i=% 1}^{n}\hat{\mathcal{I}}_{i}\right)^{-1}\mathcal{I}(\theta_{0})^{1/2}\right\\|_{% \text{op}}\cdot\left\\|\mathcal{I}(\theta_{0})^{-1/2}\frac{1}{n}\sum_{i=1}^{n}s% _{i}(\theta_{0})\right\\|$		(36)
		$\displaystyle\leq(1+o(1))\cdot\sqrt{\frac{d\log(2d/\delta)}{n}}$		(37)

The $\mathcal{O}(n^{-1})$ term arises from higher-order Taylor remainder terms. ∎

5.3 Fisher Flow Away from Optima: From Classical Statistics to Modern ML

The theoretical properties established above—consistency, asymptotic normality, and efficiency—all rest on a crucial assumption: that we converge to a local maximum of the likelihood (or equivalently, a local minimum of the loss). In classical statistics with moderate-dimensional problems, this assumption is reasonable and often satisfied. However, modern machine learning operates in a fundamentally different regime where:

1.

Convergence is rarely achieved: Training typically stops due to computational budgets, time constraints, or intentional early stopping as a form of regularization.
2.

Convergence may be undesirable: Exact optima often correspond to overfitting, while slightly suboptimal parameters generalize better.
3.

The optimization trajectory matters: The path taken through parameter space encodes useful inductive biases.

5.3.1 Reinterpreting Fisher Flow for Non-Convergent Settings

When Fisher Flow operates away from local optima, the Fisher Information Matrix takes on a different character:

Definition 14 (Trajectory-Dependent Fisher Information).

For a parameter trajectory $\{\theta_{t}\}_{t=0}^{T}$ that may not converge to an optimum, define the accumulated trajectory information:

\mathcal{I}_{\text{traj}}=\sum_{t=0}^{T}\hat{\mathcal{I}}(\theta_{t})

(38)

where $\hat{\mathcal{I}}(\theta_{t})$ is the observed Fisher information at point $\theta_{t}$ along the trajectory.

This accumulated information no longer represents uncertainty about a maximum likelihood estimate, but rather encodes the geometry of the path traversed through parameter space.

Proposition 15 (Path-Dependent Regularization).

The Fisher Flow update away from optima implements a form of path-dependent regularization:

\theta_{t+1}=\arg\min_{\theta}\left\{\ell(\theta)+\frac{1}{2}(\theta-\theta_{t% })^{\top}\mathcal{I}_{\text{traj}}(\theta-\theta_{t})\right\}

(39)

where $\mathcal{I}_{\text{traj}}$ acts as an adaptive regularizer that penalizes movement in directions where the model has accumulated significant curvature information.

Proof.

The Fisher Flow update equation can be derived as the solution to a proximal problem. Starting from the natural gradient update:

\theta_{t+1}=\theta_{t}-\eta_{t}\mathcal{I}_{\text{traj}}^{-1}s_{t}(\theta_{t})

(40)

We show this is equivalent to the stated optimization problem. The first-order optimality condition for the regularized objective is:

\nabla_{\theta}\ell(\theta_{t+1})+\mathcal{I}_{\text{traj}}(\theta_{t+1}-% \theta_{t})=0

(41)

Linearizing the gradient around $\theta_{t}$ :

\nabla_{\theta}\ell(\theta_{t+1})\approx\nabla_{\theta}\ell(\theta_{t})+\hat{% \mathcal{I}}(\theta_{t})(\theta_{t+1}-\theta_{t})

(42)

Substituting and solving:

	$\displaystyle\nabla_{\theta}\ell(\theta_{t})+\hat{\mathcal{I}}(\theta_{t})(% \theta_{t+1}-\theta_{t})+\mathcal{I}_{\text{traj}}(\theta_{t+1}-\theta_{t})$	$\displaystyle=0$		(43)
	$\displaystyle(\theta_{t+1}-\theta_{t})$	$\displaystyle=-(\hat{\mathcal{I}}(\theta_{t})+\mathcal{I}_{\text{traj}})^{-1}s% _{t}(\theta_{t})$		(44)

In the limit where $\mathcal{I}_{\text{traj}}\gg\hat{\mathcal{I}}(\theta_{t})$ (accumulated information dominates local curvature), this reduces to the Fisher Flow update. ∎

5.3.2 Implications for Modern Deep Learning

This reinterpretation explains several empirical phenomena in deep learning:

1.

Why Adam works: Adam accumulates squared gradients $\mathbb{E}[g_{t}^{2}]$ along the entire trajectory, not at convergence. This creates a path-dependent preconditioner that adapts to the geometry encountered during optimization.
2.

Why early stopping helps: Stopping before convergence preserves uncertainty in unexplored directions of parameter space. The incomplete Fisher information $\mathcal{I}_{\text{traj}}$ maintains high uncertainty (low information) in these directions, providing implicit regularization.
3.

Why flat minima generalize: Regions with low Fisher information (flat minima) indicate parameters that are less sensitive to data perturbations [7, 9]. The trajectory-based Fisher Flow naturally favors such regions by accumulating less information there.
4.

Why EWC prevents forgetting: Elastic Weight Consolidation doesn’t protect the “optimal” parameters for a task, but rather the trajectory taken while learning it. The Fisher information encodes which directions were important during learning, not at convergence.

Remark 16 (Two Regimes of Fisher Flow).

Fisher Flow operates in two distinct regimes:

•

Classical Statistical Regime: When convergence to a local maximum is achieved, Fisher Flow provides principled uncertainty quantification with all the guarantees of maximum likelihood theory.
•

Modern ML Regime: When optimization stops before convergence, Fisher Flow acts as a trajectory-dependent geometric regularizer that encodes the path through parameter space.

Both interpretations are valid and useful, but serve different purposes.

5.4 Approximation Theory for Relaxed Information Geometry

In practice, exact Fisher information computation is often intractable, necessitating approximations. We characterize the impact of these relaxations:

Definition 17 ( $(\epsilon,\delta)$ -Approximate Information).

An approximate information matrix $\tilde{\mathcal{I}}$ is $(\epsilon,\delta)$ -close to $\mathcal{I}$ if:

(1-\epsilon)\mathcal{I}\preceq\tilde{\mathcal{I}}\preceq(1+\epsilon)\mathcal{I% }\quad\text{and}\quad\|\tilde{\mathcal{I}}-\mathcal{I}\|_{F}\leq\delta

(45)

Theorem 18 (Robustness to Information Approximation).

If Fisher Flow uses $(\epsilon,\delta)$ -approximate information with $\epsilon<1$ , then:

\|\tilde{\theta}_{n}-\hat{\theta}_{n}\|_{\mathcal{I}}\leq\frac{\epsilon}{1-% \epsilon}\|\hat{\theta}_{n}-\theta_{0}\|_{\mathcal{I}}+\mathcal{O}(\delta/% \sqrt{n})

(46)

where $\tilde{\theta}_{n}$ is the approximate Fisher Flow estimator.

Proof.

We analyze the propagation of approximation error through the Fisher Flow updates.

Step 1: Update equation perturbation. The exact and approximate updates satisfy:

	$\displaystyle\hat{\theta}_{n+1}$	$\displaystyle=\hat{\theta}_{n}-\eta\mathcal{I}_{n}^{-1}s_{n}(\hat{\theta}_{n})$		(47)
	$\displaystyle\tilde{\theta}_{n+1}$	$\displaystyle=\tilde{\theta}_{n}-\eta\tilde{\mathcal{I}}_{n}^{-1}s_{n}(\tilde{% \theta}_{n})$		(48)

Step 2: Error recursion. Define $e_{n}=\tilde{\theta}_{n}-\hat{\theta}_{n}$ . Subtracting the update equations:

e_{n+1}=e_{n}-\eta\left(\tilde{\mathcal{I}}_{n}^{-1}s_{n}(\tilde{\theta}_{n})-% \mathcal{I}_{n}^{-1}s_{n}(\hat{\theta}_{n})\right)

(49)

Step 3: Linearization. Using Taylor expansion and the $(\epsilon,\delta)$ -approximation property:

	$\displaystyle e_{n+1}$	$\displaystyle=e_{n}-\eta\tilde{\mathcal{I}}_{n}^{-1}(s_{n}(\tilde{\theta}_{n})% -s_{n}(\hat{\theta}_{n}))-\eta(\tilde{\mathcal{I}}_{n}^{-1}-\mathcal{I}_{n}^{-% 1})s_{n}(\hat{\theta}_{n})$		(50)
		$\displaystyle\approx(I-\eta\tilde{\mathcal{I}}_{n}^{-1}\hat{\mathcal{I}}_{n})e% _{n}-\eta(\tilde{\mathcal{I}}_{n}^{-1}-\mathcal{I}_{n}^{-1})s_{n}(\hat{\theta}% _{n})$		(51)

Step 4: Spectral analysis. Using $(1-\epsilon)\mathcal{I}\preceq\tilde{\mathcal{I}}\preceq(1+\epsilon)\mathcal{I}$ :

\|I-\eta\tilde{\mathcal{I}}_{n}^{-1}\hat{\mathcal{I}}_{n}\|_{\text{op}}\leq% \frac{\epsilon}{1-\epsilon}

(52)

Step 5: Accumulation of error. By recursive application and using the Frobenius norm bound:

	$\displaystyle\\|e_{n}\\|_{\mathcal{I}}$	$\displaystyle\leq\sum_{k=0}^{n-1}\left(\frac{\epsilon}{1-\epsilon}\right)^{k}% \cdot\eta\\|\mathcal{I}_{k}^{-1}-\tilde{\mathcal{I}}_{k}^{-1}\\|_{F}\\|s_{k}\\|$		(53)
		$\displaystyle\leq\frac{\epsilon}{1-\epsilon}\\|\hat{\theta}_{n}-\theta_{0}\\|_{% \mathcal{I}}+\mathcal{O}(\delta/\sqrt{n})$		(54)

where the last inequality uses the Frobenius norm bound and concentration of the score. ∎

6 Related Work

Our work builds on and synthesizes several rich lines of research. We organize the discussion by research area.

6.1 Information Geometry and Natural Gradient

The mathematical foundations trace to Fisher’s introduction of information [5] and Rao’s geometric interpretation [16]. Chentsov (1982) proved the uniqueness of the Fisher-Rao metric, establishing it as the canonical Riemannian metric on statistical manifolds. Amari’s comprehensive treatment [2] developed dual connections, $\alpha$ -families, and the theory of exponential and mixture geometries that underlie our framework.

The natural gradient method [1] is the continuous-time limit of what we call Fisher Flow. Amari showed that natural gradient provides invariant, efficient parameter updates respecting the statistical manifold’s geometry. Our contribution is not inventing this—it’s showing how discrete recursive updates and modern optimizers are instances of this principle.

6.2 Recursive Estimation and Adaptive Filtering

Recursive maximum likelihood estimation has been studied extensively in the control and signal processing communities [12]. The Kalman filter (Kalman, 1960) is the exact Fisher Flow solution for linear-Gaussian models. Ljung’s work on recursive identification formalized convergence theory for online parameter estimation. We build directly on this foundation, extending the perspective to modern machine learning contexts.

6.3 Second-Order Optimization Methods

Practical second-order methods for deep learning have received significant attention. Martens & Grosse [13] introduced K-FAC (Kronecker-Factored Approximate Curvature), which approximates the Fisher information matrix for neural networks using Kronecker products. This is precisely the structured Fisher Flow we describe in Section 12. Subsequent work by Ba et al. (2017), Botev et al. (2017), and George et al. (2018) developed distributed and scalable variants.

The connection between Adam [10] and diagonal Fisher information has been noted informally in the community but not formalized. RMSprop (Tieleman & Hinton, 2012) similarly maintains running averages of squared gradients. We make explicit that these are structured approximations to natural gradient descent.

6.4 Bayesian Deep Learning and Uncertainty Quantification

MacKay (1992) pioneered the use of the Laplace approximation for neural network uncertainty, using the observed Fisher information (Hessian) to approximate the posterior covariance. This is conceptually identical to our use of $\mathcal{I}_{n}^{-1}$ for uncertainty quantification, though MacKay worked in a Bayesian framework.

Modern variational inference methods for deep learning (Graves, 2011; Blundell et al., 2015; Kingma et al., 2015) approximate the posterior through optimization. Khan et al. (2018) showed connections between variational inference and natural gradient descent, a perspective complementary to ours. Our framework provides a frequentist alternative that achieves similar uncertainty estimates with different computational trade-offs.

6.5 Continual Learning

Elastic Weight Consolidation [11] uses the Fisher information matrix to identify important parameters when learning new tasks, preventing catastrophic forgetting. Zenke et al. (2017) proposed Synaptic Intelligence, a related approach. Schwarz et al. (2018) combined compression with continual learning. We show these methods are applications of Fisher information regularization, a natural consequence of our framework.

6.6 Approximation Theory and Matrix Factorizations

The diagonal, block-diagonal, and Kronecker approximations we analyze have been studied in numerical linear algebra and matrix completion theory. Our contribution is characterizing the statistical impact (via Theorems 18, 34, 35) rather than just computational efficiency.

6.7 Implicit Regularization and Generalization

Recent work has shown that gradient-based optimization implicitly regularizes solutions in ways that promote generalization [22]. Gunasekar et al. [23] characterized implicit bias in terms of optimization geometry, showing that the optimization trajectory matters, not just the endpoint. This connects directly to our “two-regime” formalization: in the modern ML regime where training stops before convergence, the accumulated Fisher information encodes trajectory-dependent regularization rather than asymptotic efficiency.

6.8 Large-Scale and Distributed Optimization

Scaling second-order methods to large models remains challenging. Shampoo [24] extends Kronecker-factored preconditioning to general tensor parameters. LARS [25] and LAMB [26] enable large-batch training through layer-wise adaptive learning rates. These methods can be viewed as structured approximations to Fisher Flow with specific assumptions about information structure across layers.

Distributed training with local gradient aggregation [27] is naturally compatible with Fisher Flow’s information additivity property. However, handling heterogeneous workers, communication compression, and Byzantine failures requires extensions beyond the basic framework.

6.9 What is Novel in Our Work

Given this extensive prior work, we clarify our contributions:

1.

Synthesis: We unify recursive MLE, natural gradient, Adam, K-FAC, and EWC under a single ”Fisher Flow” perspective
2.

Two-regime interpretation: We formalize the distinction between classical convergent theory and modern trajectory-dependent regularization
3.

Error bounds: We provide rigorous approximation error analysis for structured Fisher information
4.

Novel variants: We propose momentum-enhanced and Byzantine-robust extensions (though these require further validation)

We do not claim the core mathematics is new—it builds on decades of research. Our contribution is making implicit patterns explicit and actionable.

7 Deep Parallels to Bayesian Inference

While Fisher Flow is philosophically frequentist, its operational structure reveals deep parallels with Bayesian inference. These parallels highlight how Fisher Flow achieves similar inferential goals through different theoretical machinery:

•

Incorporation of Prior Knowledge vs. Initial State: In Bayesian inference, prior beliefs about parameters are formally encoded in a prior distribution, $p(\theta)$ . Fisher Flow, in its pure form, does not use subjective priors. However, the initial state of the aggregate estimate $(\hat{\theta}_{0},\mathcal{I}_{0})$ can be set using prior information, or regularization terms (Section 6) can act as pseudo-priors, with the Hessian of the regularizer contributing to the initial information matrix. This provides a mechanism, albeit different in interpretation, to incorporate pre-existing knowledge or to stabilize estimates in low-data regimes.
•

Data Assimilation: Bayesian inference assimilates new data by multiplying the prior distribution with the likelihood function and then normalizing to obtain the posterior distribution, $p(\theta|x)\propto p(x|\theta)p(\theta)$ . Fisher Flow, in contrast, assimilates data by adding the score (gradient of log-likelihood) and Fisher Information from the new data batch to the existing aggregate quantities (Equations 5 and 6). This additive combination of information is algebraically simpler than the multiplicative and normalization steps in Bayesian updating.
•

Parameter Estimation (Central Tendency): The Bayesian posterior mean, $\mathbb{E}[\theta|x]$ , often serves as the Bayesian point estimate for $\theta$ . In Fisher Flow, the Maximum Likelihood Estimate, $\hat{\theta}$ , which is the mode of the likelihood (and asymptotically the mode of the posterior under certain conditions), plays this role. Fisher Flow’s sequential updates (Equation 6) show $\hat{\theta}_{t}$ as an information-weighted average of the previous estimate and the estimate from the new batch, akin to how posterior means are updated in Gaussian conjugate models.
•

Uncertainty Quantification (Dispersion): Bayesian inference quantifies uncertainty about $\theta$ via the posterior covariance matrix, which is the inverse of the posterior precision matrix. In Fisher Flow, the Fisher Information Matrix (FIM), $\mathcal{I}(\theta)$ , serves as the analogue of precision. Its inverse, $\mathcal{I}^{-1}(\theta)$ , provides an (asymptotic) covariance matrix for the MLE $\hat{\theta}$ , directly quantifying parameter uncertainty.
•

Sequential Updating and Conjugacy: Bayesian conjugate updates offer closed-form solutions for the posterior when the prior and likelihood belong to compatible distributional families (e.g., Beta-Bernoulli, Normal-Normal). Fisher Flow achieves a similar operational simplicity through the additive nature of information (Equation 5 and 6). The updates for $\hat{\theta}_{t}$ and $\mathcal{I}_{t}$ are always closed-form (given batch estimates), regardless of the specific likelihood’s family, assuming regularity conditions hold. This mirrors the computational ease of conjugate Bayesian models without being restricted to them.
•

Predictive Distributions: To make predictions for new data $x_{\text{new}}$ , Bayesian methods integrate over the posterior distribution of parameters: $p(x_{\text{new}}|x)=\int p(x_{\text{new}}|\theta)p(\theta|x)d\theta$ . Fisher Flow typically uses a ”plug-in” approach, $p(x_{\text{new}}|\hat{\theta})$ , using the point estimate $\hat{\theta}$ . However, as discussed in Section 9.3.1, parameter uncertainty from $\mathcal{I}^{-1}$ can be propagated via sampling or Laplace approximations [19] to generate richer predictive distributions that account for parameter uncertainty, thereby approaching the comprehensiveness of Bayesian predictive distributions.
•

Semantic Interpretation of Uncertainty: A key philosophical difference lies in the interpretation of uncertainty. Bayesian posterior probabilities represent degrees of epistemic belief about the parameters given the observed data and prior. The uncertainty quantified by Fisher Flow (e.g., confidence intervals derived from $\mathcal{I}^{-1}$ ) reflects sampling variability—how much the estimate $\hat{\theta}$ would vary if one were to repeat the data collection process under the same underlying true parameters $\theta_{0}$ .

The following table provides a concise summary of these parallels:

Concept	Bayesian	Fisher Flow (Frequentist)
Initial State	Prior $p(\theta)$	Initial $(\hat{\theta}_{0},\mathcal{I}_{0})$ / regularizer
Central Estimate	$\mathbb{E}[\theta\mid x_{1:n}]$	$\hat{\theta}_{n}\leftarrow\hat{\mathcal{I}}_{n}^{-1}\bigl{(}\hat{\mathcal{I}}_% {n-k}\hat{\theta}_{n-k}+\hat{\mathcal{I}}_{B}\hat{\theta}_{B}\bigr{)}$
Uncertainty (Precision)	Posterior precision, e.g., $(\text{Cov}(\theta\|x_{1:n}))^{-1}$	$\hat{\mathcal{I}}_{n}\leftarrow\hat{\mathcal{I}}_{n-k}+\hat{\mathcal{I}}_{B}$ where $\|B\|=k$
Predictive Distribution	$\int p(x_{\text{new}}\|\theta)p(\theta\|x_{1:n})d\theta$	Plug-in $\hat{\theta}_{n}$ , optionally propagate $\mathcal{I}_{n}^{-1}$
Semantics of Uncertainty	Epistemic belief	Sampling variability

Table 2: Summary of Parallels between Bayesian Inference and Fisher Flow

Note: A particularly strong connection emerges when considering the Jeffreys prior, $p(\theta)\propto\sqrt{\det\mathcal{I}(\theta)}$ [8, 17]. With this non-informative prior, the Bayesian posterior mode and the inverse of the posterior curvature (as a measure of covariance) asymptotically match the MLE $\hat{\theta}$ and $\mathcal{I}^{-1}(\hat{\theta})$ from Fisher Flow. This reinforces the idea that Fisher Flow, while frequentist, often arrives at similar quantitative conclusions as a data-dominated Bayesian analysis, especially in large-sample regimes.

8 Theoretical Guarantees and Limitations

8.1 When Fisher Flow Fails: Limitations and Failure Modes

Example 19 (Mixture Models).

Consider a Gaussian mixture $p(x|\theta)=\pi\mathcal{N}(\mu_{1},1)+(1-\pi)\mathcal{N}(\mu_{2},1)$ . Near $\pi=0$ or $\pi=1$ , the Fisher information becomes singular, causing Fisher Flow to fail. Bayesian methods with appropriate priors remain stable.

Example 20 (Heavy-Tailed Data).

For Cauchy-distributed errors, the Fisher information may not exist. Fisher Flow requires modification to robust estimators, while Bayesian methods naturally accommodate heavy tails through the likelihood.

8.2 Optimality Properties

Conjecture 21 (Information-Theoretic Optimality).

Among a suitable class of estimators that use only first- and second-order information, Fisher Flow minimizes the expected KL divergence from the true distribution:

\hat{\theta}_{\text{FF}}=\arg\min_{\hat{\theta}\in\mathcal{E}_{2}}\mathbb{E}[D% _{KL}(p_{\theta_{0}}\|p_{\hat{\theta}})]

(55)

where $\mathcal{E}_{2}$ is an appropriately defined class of second-order estimators.

Remark 22.

This conjecture requires precise characterization of the estimator class $\mathcal{E}_{2}$ and the information constraints. A rigorous proof would need to formalize what “uses only first- and second-order information” means and establish optimality within that class. This remains an open problem.

Theorem 23 (Invariance Properties).

Fisher Flow satisfies:

1.

Parameterization invariance: Updates are covariant under smooth reparameterizations
2.

Sufficiency preservation: If $T(X)$ is sufficient for $\theta$ , Fisher Flow based on $T(X)$ equals Fisher Flow based on $X$
3.

Information monotonicity: $\mathcal{I}_{t+1}\succeq\mathcal{I}_{t}$ in the positive semi-definite ordering

Proof.

We prove each property separately:

1. Parameterization invariance: Let $\phi=f(\theta)$ be a diffeomorphic reparameterization with Jacobian $J=\partial f/\partial\theta$ .

The Fisher information transforms as:

\mathcal{I}_{\phi}=J^{\top}\mathcal{I}_{\theta}J

(56)

The natural gradient in the $\phi$ coordinates:

$\displaystyle\tilde{\nabla}_{\phi}\ell$	$\displaystyle=\mathcal{I}_{\phi}^{-1}\nabla_{\phi}\ell$	(57)
	$\displaystyle=(J^{\top}\mathcal{I}_{\theta}J)^{-1}J^{\top}\nabla_{\theta}\ell$	(58)
	$\displaystyle=J^{-1}\mathcal{I}_{\theta}^{-1}\nabla_{\theta}\ell$	(59)
	$\displaystyle=J^{-1}\tilde{\nabla}_{\theta}\ell$	(60)

Thus, the update $\phi_{t+1}=\phi_{t}-\eta\tilde{\nabla}_{\phi}\ell$ is equivalent to $\theta_{t+1}=\theta_{t}-\eta\tilde{\nabla}_{\theta}\ell$ under the transformation.

2. Sufficiency preservation: By the Neyman-Fisher factorization theorem, if $T(X)$ is sufficient for $\theta$ , then:

p(x|\theta)=g(T(x),\theta)h(x)

(61)

The score function depends only on $T(x)$ :

s(\theta;x)=\nabla_{\theta}\log p(x|\theta)=\nabla_{\theta}\log g(T(x),\theta)

(62)

Therefore, the Fisher information computed from $X$ or $T(X)$ is identical:

\mathcal{I}_{X}(\theta)=\mathbb{E}_{X}[s(\theta;X)s(\theta;X)^{\top}]=\mathbb{% E}_{T(X)}[s(\theta;T(X))s(\theta;T(X))^{\top}]=\mathcal{I}_{T(X)}(\theta)

(63)

3. Information monotonicity: For any vector $v\in\mathbb{R}^{d}$ :

$\displaystyle v^{\top}\mathcal{I}_{t+1}v$	$\displaystyle=v^{\top}(\mathcal{I}_{t}+\hat{\mathcal{I}}_{\text{new}})v$	(64)
	$\displaystyle=v^{\top}\mathcal{I}_{t}v+v^{\top}\hat{\mathcal{I}}_{\text{new}}v$	(65)
	$\displaystyle\geq v^{\top}\mathcal{I}_{t}v$	(66)

since $\hat{\mathcal{I}}_{\text{new}}\succeq 0$ (positive semi-definite as a covariance matrix of scores). ∎

8.3 Fundamental Limitations

Conjecture 24 (No Free Lunch for Information Geometry).

There exists no universal approximation $\tilde{\mathcal{I}}$ that simultaneously:

1.

Preserves $\mathcal{O}(d)$ computational complexity
2.

Maintains positive definiteness
3.

Achieves $(1+\epsilon)$ -approximation for all models

Remark 25.

This conjecture formalizes the intuition that there are fundamental trade-offs in approximating the Fisher information matrix. A proof would likely proceed via construction of adversarial model families. Proving or disproving this conjecture would have significant implications for second-order optimization.

8.4 Comparison with Alternative Frameworks

Table 3: Theoretical Properties of Inference Frameworks

Property	Full Bayes	FF	MAP
Coherence	✓	Asymptotic	$\times$
Computational tractability	$\times$	✓	✓
Uncertainty quantification	✓	✓	$\times$
Information efficiency	✓	✓	Partial
Distributed computation	Hard	✓	✓
Non-regular models	✓	$\times$	$\times$

9 Extensions and Theoretical Connections

9.1 Connection to Thermodynamic Principles

Fisher Flow exhibits profound connections to statistical mechanics and thermodynamics:

Proposition 26 (Entropy under Gaussian Approximation).

Under a Gaussian approximation to parameter uncertainty with covariance $\mathcal{I}(\theta)^{-1}$ , the differential entropy satisfies:

S(\theta)=\frac{k}{2}\log\det(2\pi e\mathcal{I}^{-1})

(67)

where $k$ is a scaling constant (analogous to Boltzmann’s constant).

This connection suggests that Fisher Flow updates follow a principle of maximum entropy production, moving parameters along paths that maximize information gain subject to constraints.

9.2 Relationship to Existing Methods

Fisher Flow provides theoretical foundations for several popular algorithms:

•

Adam = Diagonal FF: Adam’s second moment estimate approximates diagonal Fisher information
•

K-FAC = Kronecker FF: Kronecker-factored approximate curvature implements structured Fisher Flow
•

EWC = FF regularization: Elastic weight consolidation uses Fisher information as importance weights
•

Natural gradient = Exact FF: With full Fisher information matrix

This unification suggests that practitioners are already using Fisher Flow approximations, often without recognizing the underlying information-geometric principles.

9.3 Connections to Optimal Control

Fisher Flow can be viewed through the lens of stochastic optimal control:

Remark 27 (Control-Theoretic View).

With additional modeling assumptions, one can define a value function $V(\theta,t)$ and write a Hamilton–Jacobi–Bellman equation:

\frac{\partial V}{\partial t}+\min_{u}\left\{\nabla V\cdot f(\theta,u)+\frac{1% }{2}\text{Tr}(\sigma\sigma^{\top}\nabla^{2}V)\right\}=0

(68)

where an optimal control $u^{*}$ would recover a natural-gradient-like direction. Making this rigorous requires a concrete control formulation.

This perspective connects Fisher Flow to reinforcement learning and provides tools for analyzing convergence through Lyapunov theory.

9.4 Computational Complexity Analysis

Table 4: Computational Complexity of Fisher Flow Variants

Operation	Time Complexity	Space Complexity
Score computation	$\mathcal{O}(nd)$	$\mathcal{O}(d)$
Full FIM computation	$\mathcal{O}(nd^{2})$	$\mathcal{O}(d^{2})$
Full FIM inversion	$\mathcal{O}(d^{3})$	$\mathcal{O}(d^{2})$
Diagonal approximation	$\mathcal{O}(nd)$	$\mathcal{O}(d)$
Block-diagonal (k blocks)	$\mathcal{O}(n\sum_{i}d_{i}^{2})$	$\mathcal{O}(\sum_{i}d_{i}^{2})$
Kronecker-factored	$\mathcal{O}(n(m^{2}+n^{2}))$	$\mathcal{O}(m^{2}+n^{2})$
Low-rank (rank r)	$\mathcal{O}(ndr)$	$\mathcal{O}(dr)$

For neural networks with $L$ layers and width $w$ , full FIM requires $\mathcal{O}(L^{2}w^{4})$ operations while Kronecker-factored Fisher Flow requires only $\mathcal{O}(Lw^{3})$ .

10 Information-Geometric Foundations

10.1 The Statistical Manifold as a Riemannian Space

The foundation of Fisher Flow rests on viewing parametric families as Riemannian manifolds equipped with the Fisher-Rao metric. This geometric perspective reveals deep mathematical structure:

Theorem 28 (Uniqueness of the Fisher-Rao Metric).

The Fisher-Rao metric is the unique Riemannian metric on statistical manifolds that is invariant under sufficient statistics.

Proof.

Let $T(X)$ be a sufficient statistic for $\theta$ . By the factorization theorem:

p(x|\theta)=g(T(x),\theta)h(x)

(69)

The invariance requirement demands that the metric computed from $p(x|\theta)$ equals that from $g(t,\theta)$ . This uniquely determines the Fisher-Rao metric (see, e.g., expositions in [2]). ∎

10.2 Dual Connections and Information Geometry

The statistical manifold admits a dual geometric structure that enriches Fisher Flow:

Definition 29 ( $\alpha$ -Connections).

For $\alpha\in\mathbb{R}$ , the $\alpha$ -connection $\nabla^{(\alpha)}$ is defined by:

\Gamma_{ijk}^{(\alpha)}=\mathbb{E}\left[\partial_{i}\partial_{j}\ell\cdot% \partial_{k}\ell\right]+\frac{1-\alpha}{2}\mathbb{E}\left[\partial_{i}\ell% \cdot\partial_{j}\ell\cdot\partial_{k}\ell\right]

(70)

where $\ell=\log p(x|\theta)$ and $\partial_{i}=\partial/\partial\theta_{i}$ .

Theorem 30 (Duality Structure).

The exponential connection $\nabla^{(e)}=\nabla^{(1)}$ and mixture connection $\nabla^{(m)}=\nabla^{(-1)}$ are dual with respect to the Fisher-Rao metric:

\partial_{k}g_{ij}=\Gamma_{ki,j}^{(e)}+\Gamma_{kj,i}^{(m)}

(71)

This duality underlies the relationship between maximum likelihood (e-geodesics) and moment matching (m-geodesics), providing geometric insight into different estimation principles.

10.3 Information Monotonicity and Data Processing

Theorem 31 (Data Processing Inequality for Fisher Information).

Let $Y=f(X)$ be any statistic. Then:

\mathcal{I}_{Y}(\theta)\preceq\mathcal{I}_{X}(\theta)

(72)

with equality if and only if $Y$ is a sufficient statistic.

Proof.

Let $s_{X}:=\nabla_{\theta}\log p(X\mid\theta)$ and $s_{Y}:=\mathbb{E}[s_{X}\mid Y]$ . Then $\mathcal{I}_{X}(\theta)=\mathrm{Var}(s_{X})$ and $\mathcal{I}_{Y}(\theta)=\mathrm{Var}(s_{Y})$ . By the law of total variance, $\mathrm{Var}(s_{X})=\mathbb{E}[\mathrm{Var}(s_{X}\mid Y)]+\mathrm{Var}(\mathbb% {E}[s_{X}\mid Y])\succeq\mathrm{Var}(\mathbb{E}[s_{X}\mid Y])$ , hence $\mathcal{I}_{X}(\theta)\succeq\mathcal{I}_{Y}(\theta)$ with equality iff $\mathrm{Var}(s_{X}\mid Y)=0$ almost surely, i.e., $Y$ is sufficient. ∎

This theorem justifies Fisher Flow’s focus on accumulating all available information: any summarization or preprocessing can only decrease the information available for inference.

10.4 Variational Characterization of Fisher Flow

Theorem 32 (Local Quadratic Proximal Update).

The Fisher Flow update after observing batch $B$ (bringing total observations from $n$ to $n+|B|$ ) admits the following local quadratic proximal form:

\hat{\theta}_{n+|B|}=\arg\min_{\theta}\left\{-\ell_{B}(\theta)+\frac{1}{2}(% \theta-\hat{\theta}_{n})^{\top}\mathcal{I}_{n}(\theta-\hat{\theta}_{n})\right\}

(73)

where the second term is a quadratic penalty induced by the accumulated Fisher information from the first $n$ observations.

Proof.

The first-order optimality condition yields:

-s_{B}(\hat{\theta}_{n+|B|})+\mathcal{I}_{n}(\hat{\theta}_{n+|B|}-\hat{\theta}% _{n})=0

(74)

Linearizing the score around $\hat{\theta}_{n}$ :

s_{B}(\hat{\theta}_{n+|B|})\approx s_{B}(\hat{\theta}_{n})+\hat{\mathcal{I}}_{% B}(\hat{\theta}_{n+|B|}-\hat{\theta}_{n})

(75)

Substituting and solving recovers the Fisher Flow update equation. ∎

This variational perspective connects Fisher Flow to mirror-descent-like updates and reveals its implicit regularization structure.

10.5 Practical Implementation Guidelines

10.5.1 Choosing the Approximation Level

The choice of Fisher information approximation depends on model structure and computational budget:

•

Diagonal: Use for models with weak parameter interactions (e.g., coordinate-wise optimization). Cost: $\mathcal{O}(d)$ per update.
•

Block-diagonal: Use when parameters naturally group (e.g., layer-wise in neural networks). Cost: $\mathcal{O}(\sum_{i}d_{i}^{3})$ .
•

Kronecker-factored: Ideal for matrix parameters (e.g., fully-connected layers). Cost: $\mathcal{O}(m^{3}+n^{3})$ for $m\times n$ weight matrix.
•

Low-rank + diagonal: Use when a few directions dominate the curvature. Cost: $\mathcal{O}(dr^{2})$ for rank $r$ .

10.5.2 Initialization Strategies

1.

Uninformative: $\mathcal{I}_{0}=\epsilon I$ with small $\epsilon>0$
2.

From prior knowledge: $\mathcal{I}_{0}=\nabla^{2}R(\theta_{0})$ where $R$ is a regularizer
3.

From pre-training: Use Fisher information from related task
4.

Empirical: Estimate from small initial batch

10.5.3 Hyperparameter Selection

•

Learning rate $\eta$ : Start with $\eta=1$ (natural scaling), decrease if unstable
•

Forgetting factor $\rho$ : Use $\rho=0.99$ for slowly changing distributions
•

Batch size: Larger batches improve Fisher information estimates
•

Damping: Add $\lambda I$ to $\mathcal{I}$ for numerical stability, typically $\lambda=10^{-4}$

11 Algorithmic Realization

11.1 Abstract Fisher Flow Algorithm

We present Fisher Flow at multiple levels of abstraction, from the theoretical ideal to practical implementations:

Algorithm 2 Abstract Fisher Flow: Geometric Flow on Statistical Manifold

1:Given: Statistical manifold

(\mathcal{M},g)

, data stream

\{x_{t}\}

2:Initialize:

\theta_{0}\in\mathcal{M}

\mathcal{I}_{0}=g(\theta_{0})

3:while data available do

4: Compute tangent vector:

v_{t}=\text{score}(x_{t},\theta_{t})\in T_{\theta_{t}}\mathcal{M}

5: Update metric:

g_{t+1}=g_{t}+v_{t}\otimes v_{t}

6: Follow geodesic:

\theta_{t+1}=\exp_{\theta_{t}}(-\eta g_{t}^{-1}v_{t})

7:end while

11.2 Practical Implementation with Approximations

Algorithm 3 Practical Fisher Flow with Adaptive Structure

1:Structure selector

\mathcal{S}

, approximation level

k

\theta_{0}\leftarrow

initialize parameters

\mathcal{I}_{0}\leftarrow

initialize information structure

n\leftarrow 0

\triangleright

Total observations counter

5:for batch

B

in data stream do

\triangleright

Process batches sequentially

6: // Adaptive structure selection (every

k

batches)

7: if batch index

\mod k=0

then

\text{struct}\leftarrow\mathcal{S}(\mathcal{I}_{n},B)

\triangleright

Choose: diagonal, block, Kronecker, etc.

9: end if

10: // Information accumulation from batch

11:

s_{B}\leftarrow\nabla_{\theta}\ell_{B}(\theta_{n})

\triangleright

Batch gradient

12:

\tilde{\mathcal{I}}_{B}\leftarrow\text{ApproxFIM}(B,\theta_{n},\text{struct})

13:

\mathcal{I}_{n+|B|}\leftarrow\mathcal{I}_{n}+\tilde{\mathcal{I}}_{B}

14: // Natural gradient step

15:

\theta_{n+|B|}\leftarrow\theta_{n}-\eta\cdot\text{Solve}(\mathcal{I}_{n+|B|},s% _{B},\text{struct})

16:

n\leftarrow n+|B|

\triangleright

Update total observation count

17:end for

The Solve function efficiently computes $\mathcal{I}_{t}^{-1}s_{t}$ based on the chosen structure: - Diagonal: $\mathcal{O}(d)$ element-wise division - Block-diagonal: $\mathcal{O}(\sum_{i}d_{i}^{3})$ block inversions - Kronecker: $\mathcal{O}(m^{3}+n^{3})$ using $(A\otimes B)^{-1}=A^{-1}\otimes B^{-1}$ - Low-rank: Sherman-Morrison-Woodbury formula

12 Approximation Theory and Computational Relaxations

While the exact Fisher Flow theory provides elegant mathematical guarantees, practical implementation often requires approximations. We now rigorously characterize these relaxations and their impact.

12.1 Structured Approximations of Fisher Information

Definition 33 (Structured Information Approximation).

A structured approximation $\tilde{\mathcal{I}}$ of the Fisher information $\mathcal{I}$ belongs to a constrained set $\mathcal{S}$ :

\tilde{\mathcal{I}}=\arg\min_{M\in\mathcal{S}}D(\mathcal{I},M)

(76)

where $D$ is a matrix divergence (e.g., Frobenius norm, KL divergence between induced Gaussians).

Common structural constraints and their theoretical properties:

Theorem 34 (Diagonal Approximation Error).

For the diagonal approximation $\tilde{\mathcal{I}}=\text{diag}(\mathcal{I})$ :

\|\mathcal{I}^{-1}-\tilde{\mathcal{I}}^{-1}\|_{F}\leq\frac{\|\mathcal{I}-% \tilde{\mathcal{I}}\|_{F}}{\lambda_{\min}(\mathcal{I})^{2}}

(77)

where $\lambda_{\min}$ is the smallest eigenvalue of $\mathcal{I}$ .

Proof.

We use standard matrix perturbation theory. Let $E=\mathcal{I}-\tilde{\mathcal{I}}$ be the off-diagonal error matrix.

Step 1: Matrix inverse perturbation. For invertible $\mathcal{I}$ and $\tilde{\mathcal{I}}$ :

\mathcal{I}^{-1}-\tilde{\mathcal{I}}^{-1}=\mathcal{I}^{-1}(\tilde{\mathcal{I}}% -\mathcal{I})\tilde{\mathcal{I}}^{-1}=-\mathcal{I}^{-1}E\tilde{\mathcal{I}}^{-1}

(78)

Step 2: Frobenius norm bound. Taking norms and using submultiplicativity:

\|\mathcal{I}^{-1}-\tilde{\mathcal{I}}^{-1}\|_{F}\leq\|\mathcal{I}^{-1}\|_{2}% \cdot\|E\|_{F}\cdot\|\tilde{\mathcal{I}}^{-1}\|_{2}

(79)

Step 3: Eigenvalue bounds. Since $\tilde{\mathcal{I}}=\text{diag}(\mathcal{I})$ and $\mathcal{I}$ is positive definite, by Gershgorin’s theorem $\lambda_{\min}(\tilde{\mathcal{I}})\geq\lambda_{\min}(\mathcal{I})$ . Thus:

\|\mathcal{I}^{-1}\|_{2}=\frac{1}{\lambda_{\min}(\mathcal{I})},\quad\|\tilde{% \mathcal{I}}^{-1}\|_{2}\leq\frac{1}{\lambda_{\min}(\mathcal{I})}

(80)

Combining yields the stated bound. ∎

Theorem 35 (Kronecker-Factored Approximation).

For neural network layers with weight matrix $W\in\mathbb{R}^{m\times n}$ , the Kronecker approximation:

\mathcal{I}_{W}\approx A\otimes B

(81)

where $A\in\mathbb{R}^{m\times m}$ and $B\in\mathbb{R}^{n\times n}$ , achieves:

\text{rank}(\tilde{\mathcal{I}})=\text{rank}(A)\cdot\text{rank}(B)

(82)

with computational complexity $\mathcal{O}(m^{3}+n^{3})$ instead of $\mathcal{O}((mn)^{3})$ .

Proof.

Step 1: Kronecker structure of FIM. For a linear layer $y=Wx$ with loss $\ell$ , the Fisher information for $\text{vec}(W)$ has the form:

\mathcal{I}_{W}=\mathbb{E}[\nabla_{W}\ell\,\nabla_{W}\ell^{\top}]=\mathbb{E}[(% x\otimes\nabla_{y}\ell)(x\otimes\nabla_{y}\ell)^{\top}]

(83)

Step 2: Kronecker factorization. If input $x$ and output gradient $\nabla_{y}\ell$ are independent (a common approximation), then:

\mathcal{I}_{W}\approx\mathbb{E}[xx^{\top}]\otimes\mathbb{E}[\nabla_{y}\ell\,% \nabla_{y}\ell^{\top}]=A\otimes B

(84)

where $A=\mathbb{E}[xx^{\top}]\in\mathbb{R}^{n\times n}$ and $B=\mathbb{E}[\nabla_{y}\ell\,\nabla_{y}\ell^{\top}]\in\mathbb{R}^{m\times m}$ .

Step 3: Computational efficiency. Using $(A\otimes B)^{-1}=A^{-1}\otimes B^{-1}$ , the natural gradient step requires inverting $A$ and $B$ separately: $\mathcal{O}(n^{3}+m^{3})$ instead of $\mathcal{O}((mn)^{3})$ for the full $mn\times mn$ matrix.

Step 4: Rank property. By properties of Kronecker products, $\text{rank}(A\otimes B)=\text{rank}(A)\cdot\text{rank}(B)$ . ∎

12.2 Stochastic Approximations

Definition 36 (Stochastic Fisher Information).

Given mini-batch $B\subseteq\{1,\ldots,n\}$ with $|B|=b$ :

\hat{\mathcal{I}}_{B}=\frac{n}{b}\sum_{i\in B}s_{i}s_{i}^{\top}

(85)

where $s_{i}=\nabla_{\theta}\log p(x_{i}|\theta)$ is the per-sample score.

Theorem 37 (Concentration of Stochastic FIM).

For bounded scores $\|s_{i}\|\leq L$ , with probability at least $1-\delta$ :

\left\|\hat{\mathcal{I}}_{B}-\mathcal{I}\right\|_{2}\leq L^{2}\sqrt{\frac{2% \log(2d/\delta)}{b}}

(86)

Proof.

We use matrix concentration inequalities to bound the deviation of the empirical FIM.

Step 1: Centering. Define the centered random matrices:

Z_{i}=s_{i}s_{i}^{\top}-\mathcal{I}

(87)

where $\mathbb{E}[Z_{i}]=0$ and $\|Z_{i}\|_{\text{op}}\leq L^{2}+\|\mathcal{I}\|_{\text{op}}\leq 2L^{2}$ (using $\|\mathcal{I}\|_{\text{op}}\leq L^{2}$ ).

Step 2: Matrix Bernstein inequality. For the batch average $\hat{\mathcal{I}}_{B}=\frac{1}{b}\sum_{i=1}^{b}s_{i}s_{i}^{\top}$ :

\mathbb{P}\left(\left\|\hat{\mathcal{I}}_{B}-\mathcal{I}\right\|_{\text{op}}>t% \right)\leq 2d\cdot\exp\left(-\frac{bt^{2}/2}{\sigma^{2}+Lt/3}\right)

(88)

where $\sigma^{2}=\|\mathbb{E}[Z_{i}^{2}]\|_{\text{op}}\leq L^{4}$ .

Step 3: Setting the threshold. Choose $t=L^{2}\sqrt{\frac{2\log(2d/\delta)}{b}}$ :

$\displaystyle\mathbb{P}\left(\left\\|\hat{\mathcal{I}}_{B}-\mathcal{I}\right\\|_% {\text{op}}>L^{2}\sqrt{\frac{2\log(2d/\delta)}{b}}\right)$	$\displaystyle\leq 2d\cdot\exp\left(-\frac{b\cdot 2L^{4}\log(2d/\delta)/b}{2L^{% 4}+\frac{2L^{3}}{3}\sqrt{\frac{2\log(2d/\delta)}{b}}}\right)$	(89)
	$\displaystyle\leq 2d\cdot\exp(-\log(2d/\delta))$	(90)
	$\displaystyle=\delta$	(91)

where we used that for large enough $b$ , the denominator is dominated by $2L^{4}$ . ∎

This concentration bound justifies mini-batch approximations and provides guidance for batch size selection.

12.3 Connection to Modern Optimization Methods

Fisher Flow provides theoretical foundations for widely-used optimization algorithms:

Theorem 38 (Adam as Approximate Natural Gradient).

The Adam optimizer [10] with parameters $(\beta_{1},\beta_{2})$ approximates natural gradient descent with:

$\displaystyle\hat{m}_{t}$	$\displaystyle=\beta_{1}\hat{m}_{t-1}+(1-\beta_{1})s_{t}$	(momentum of score)	(92)
$\displaystyle\hat{v}_{t}$	$\displaystyle=\beta_{2}\hat{v}_{t-1}+(1-\beta_{2})s_{t}\odot s_{t}$	(diagonal FIM estimate)	(93)
$\displaystyle\theta_{t+1}$	$\displaystyle=\theta_{t}-\eta\frac{\hat{m}_{t}}{\sqrt{\hat{v}_{t}}+\epsilon}$	(approximate natural gradient step)	(94)

where $\odot$ denotes element-wise multiplication.

Proof.

The diagonal elements of the empirical Fisher information are $\mathcal{I}_{ii}=\mathbb{E}[s_{i}^{2}]$ . The exponential moving average $\hat{v}_{t}$ estimates these diagonal elements. The update $\theta_{t+1}=\theta_{t}-\eta\text{diag}(\hat{v}_{t})^{-1/2}\hat{m}_{t}$ approximates the natural gradient step with diagonal FIM. ∎

Theorem 39 (Elastic Weight Consolidation as Information Regularization).

EWC [11] implements Fisher Flow with task-specific information accumulation:

\mathcal{L}_{\text{EWC}}(\theta)=\mathcal{L}_{\text{new}}(\theta)+\frac{% \lambda}{2}(\theta-\theta^{*})^{\top}\mathcal{I}_{\text{old}}(\theta-\theta^{*})

(95)

where $\mathcal{I}_{\text{old}}$ is the Fisher information from previous tasks.

These connections demonstrate that Fisher Flow is not merely theoretical but underlies successful practical methods.

12.4 Foundation Models and Scaling Laws

Definition 40 (Information Scaling Law).

For models with parameter count $N$ trained on data size $D$ , the accumulated information scales as:

\|\mathcal{I}\|_{F}\sim D^{\alpha}N^{\beta}

(96)

where $\alpha,\beta$ are model-dependent constants.

Theorem 41 (Critical Information Threshold).

There exists a critical information level $\mathcal{I}_{c}$ such that:

•

For $\|\mathcal{I}\|<\mathcal{I}_{c}$ : The model is in the underparameterized regime
•

For $\|\mathcal{I}\|>\mathcal{I}_{c}$ : The model exhibits emergent capabilities

Proof.

We establish the existence of a phase transition in model behavior based on accumulated information.

Step 1: Information-theoretic capacity. Define the effective degrees of freedom:

\text{edf}(\mathcal{I})=\text{tr}(\mathcal{I}(\mathcal{I}+\lambda I)^{-1})

(97)

where $\lambda$ is a regularization parameter.

Step 2: Critical threshold. The critical information level occurs when:

\text{edf}(\mathcal{I}_{c})=\frac{d}{2}

(98)

where $d$ is the parameter dimension. This corresponds to half of the parameters being effectively determined by the data.

Step 3: Phase transition. Below the threshold ( $\|\mathcal{I}\|<\mathcal{I}_{c}$ ):

•

The model has high parameter uncertainty: $\|\mathcal{I}^{-1}\|_{\text{op}}>\epsilon$ for some $\epsilon>0$
•

Predictions are dominated by prior/regularization
•

Generalization is poor due to underfitting

Above the threshold ( $\|\mathcal{I}\|>\mathcal{I}_{c}$ ):

•

Parameter estimates stabilize: $\|\mathcal{I}^{-1}\|_{\text{op}}<\delta$ for small $\delta$
•

The model can represent complex patterns
•

Emergent capabilities appear as the effective capacity exceeds a critical level

Step 4: Spectral characterization. The transition can be characterized by the spectral gap:

\gamma(\mathcal{I})=\frac{\lambda_{\text{min}}(\mathcal{I})}{\lambda_{\text{% max}}(\mathcal{I})}

(99)

When $\gamma(\mathcal{I})$ crosses a threshold $\gamma^{*}$ , the model transitions from the underparameterized to the well-specified regime, enabling emergent behaviors. ∎

This theoretical framework helps explain the sudden emergence of capabilities in large language models as they cross information thresholds.

12.5 Fisher Flow and Foundation Models

Large Language Models (e.g., GPT [15], BERT [3]) represent ambitious applications of likelihood-based estimation. Despite their scale, these systems remain fundamentally likelihood-driven, and Fisher Flow’s core principles apply:

•

Predictive uncertainty: Parameter uncertainty $(\hat{\theta},\hat{\mathcal{I}}^{-1})$ can theoretically inform predictive confidence intervals for next-token distributions
•

Fine-tuning as information fusion: Pre-trained parameters $(\hat{\theta}_{\text{pre}},\mathcal{I}_{\text{pre}})$ serve as an information-weighted prior, enabling principled adaptation to new tasks
•

Distributed training: Information additivity enables combining gradients across workers with proper weighting

However, maintaining Fisher information approximations at the scale of modern LLMs (billions of parameters) presents substantial practical challenges. Detailed procedures for predictive uncertainty estimation are discussed in Appendix A.3.

13 Novel Algorithmic Variants and Theoretical Extensions

Remark 42 (Status of Novel Variants).

The algorithmic variants proposed in this section (Sections 13.1–13.5) are presented with theoretical motivation but lack empirical validation. They should be viewed as preliminary proposals that require:

1.

Completion of theoretical proofs (several results are stated as propositions or conjectures)
2.

Extensive experimental evaluation on benchmark tasks
3.

Comparison with existing methods
4.

Computational complexity analysis
5.

Ablation studies to validate design choices

We include them to illustrate how the Fisher Flow framework suggests natural algorithmic extensions, but they are not mature contributions ready for adoption. Future work must validate (or refute) their practical utility.

13.1 Momentum-Enhanced Fisher Flow

Building on the geometric interpretation, we introduce a novel variant that incorporates momentum directly into the information geometry:

Definition 43 (Momentum Fisher Flow).

Define the momentum-enhanced update:

$\displaystyle v_{t+1}$	$\displaystyle=\beta v_{t}+(1-\beta)\mathcal{I}_{t}^{-1}s_{t}(\theta_{t})$	(velocity in natural coordinates)	(100)
$\displaystyle\theta_{t+1}$	$\displaystyle=\theta_{t}-\eta v_{t+1}$	(parameter update)	(101)
$\displaystyle\mathcal{I}_{t+1}$	$\displaystyle=\gamma\mathcal{I}_{t}+\hat{\mathcal{I}}_{t+1}$	(information with decay)	(102)

where $\beta\in[0,1)$ is the momentum coefficient and $\gamma\in(0,1]$ is the information decay rate.

Proposition 44 (Convergence of Momentum Fisher Flow).

Under strong convexity with constant $\mu>0$ , Lipschitz continuous gradients with constant $L$ , and appropriate choice of momentum parameter $\beta\in(0,1)$ and learning rate $\eta\leq 1/L$ , Momentum Fisher Flow achieves accelerated convergence when the Fisher information stabilizes. Specifically:

\mathbb{E}[\ell(\theta_{t})-\ell(\theta^{*})]\leq\mathcal{O}\left(\frac{1}{t^{% 2}}\right)

(103)

compared to $\mathcal{O}(1/t)$ for standard natural gradient descent.

Proof.

We analyze convergence via a Lyapunov function approach, adapting techniques from accelerated gradient methods to the information-geometric setting.

Step 1: Lyapunov function. Define:

V_{t}=\ell(\theta_{t})-\ell(\theta^{*})+\frac{1}{2\eta}\|v_{t}\|^{2}_{\mathcal% {I}_{t}}

(104)

where $\|v\|^{2}_{\mathcal{I}}=v^{\top}\mathcal{I}v$ is the norm induced by the Fisher metric.

Step 2: Information stabilization. Assume the accumulated information stabilizes: $\|\mathcal{I}_{t+1}-\mathcal{I}_{t}\|_{2}\leq\epsilon_{t}$ with $\sum_{t}\epsilon_{t}<\infty$ . This holds when data is i.i.d. and $\mathcal{I}_{t}/t\to\mathcal{I}(\theta^{*})$ .

Step 3: Lyapunov decrease. Under $\mu$ -strong convexity and $L$ -smoothness:

\displaystyle V_{t+1}-V_{t}

\displaystyle\leq-\eta(1-\beta)\mu(\ell(\theta_{t})-\ell(\theta^{*}))+\mathcal% {O}(\epsilon_{t}\|v_{t}\|^{2})

(105)

Step 4: Acceleration. By choosing $\beta=1-\Theta(1/\sqrt{t})$ (similar to Nesterov’s schedule) and using standard telescoping arguments:

\ell(\theta_{t})-\ell(\theta^{*})\leq\frac{C\cdot V_{0}}{t^{2}}

(106)

for a constant $C$ depending on $\mu$ , $L$ , and the rate of information stabilization. ∎

Remark 45.

The assumption that $\mathcal{I}_{t}$ stabilizes is essential. When the metric changes rapidly (e.g., early in training or with non-stationary data), the acceleration guarantee may not hold. Empirical validation confirms improved convergence on well-conditioned problems (Section 15).

13.2 Adaptive Information Compression

A key insight from Fisher Flow is that not all directions in parameter space are equally important. We formalize this through adaptive compression:

Definition 46 (Compressed Fisher Flow).

Given eigendecomposition $\mathcal{I}=U\Lambda U^{\top}$ , define the compressed information:

\tilde{\mathcal{I}}_{k}=U_{k}\Lambda_{k}U_{k}^{\top}

(107)

where $U_{k}$ contains the top- $k$ eigenvectors and $\Lambda_{k}$ the corresponding eigenvalues.

Theorem 47 (Optimal Compression Rate).

The optimal compression rank $k^{*}$ that minimizes prediction error subject to computational constraints is:

k^{*}=\arg\min_{k}\left\{\text{tr}(\mathcal{I}^{-1})-\text{tr}(\tilde{\mathcal% {I}}_{k}^{-1})+\lambda\cdot k\right\}

(108)

where $\lambda$ controls the computation-accuracy trade-off.

This leads to a practical algorithm that adaptively chooses the compression level based on the information spectrum.

13.3 Fisher Flow with Implicit Regularization

We reveal that Fisher Flow naturally implements a form of implicit regularization through its geometry:

Theorem 48 (Implicit Regularization of Fisher Flow).

The Fisher Flow trajectory implicitly minimizes:

\hat{\theta}_{T}=\arg\min_{\theta\in\Theta_{T}}\int_{0}^{T}\|\dot{\theta}(t)\|% _{\mathcal{I}(\theta(t))}dt

(109)

where $\Theta_{T}=\{\theta:\ell(\theta)\leq\ell(\theta_{0})-\epsilon\}$ is the sublevel set and the integral represents the information-weighted path length.

Proof.

The natural gradient flow follows geodesics on the statistical manifold. Among all paths reaching the same likelihood value, the natural gradient selects the shortest path in the Fisher-Rao metric. This can be shown using the calculus of variations:

The Euler-Lagrange equation for the functional $L[\theta]=\int_{0}^{T}\sqrt{\dot{\theta}^{\top}\mathcal{I}(\theta)\dot{\theta}% }dt$ yields:

\frac{d}{dt}\left(\mathcal{I}(\theta)\dot{\theta}\right)-\frac{1}{2}\frac{% \partial}{\partial\theta}\left(\dot{\theta}^{\top}\mathcal{I}(\theta)\dot{% \theta}\right)=0

(110)

This is precisely the geodesic equation on the statistical manifold, which the natural gradient flow approximates discretely. ∎

13.4 Distributed Fisher Flow with Byzantine Robustness

For distributed settings, we develop a Byzantine-robust variant:

Definition 49 (Byzantine-Robust Information Aggregation).

Given information matrices $\{\mathcal{I}^{(i)}\}_{i=1}^{m}$ from $m$ workers (with up to $f$ Byzantine), compute:

\mathcal{I}_{\text{robust}}=\text{GeometricMedian}\left(\{\mathcal{I}^{(i)}\}_% {i=1}^{m}\right)

(111)

where the geometric median is computed in the space of positive definite matrices with the Fisher-Rao metric.

Conjecture 50 (Robustness Guarantee).

With up to $f<m/2$ Byzantine workers, we conjecture that the geometric median aggregation satisfies:

\|\mathcal{I}_{\text{robust}}-\mathcal{I}_{\text{true}}\|_{F}\leq\mathcal{O}% \left(\frac{f}{m}\right)\cdot\|\mathcal{I}_{\text{true}}\|_{F}

(112)

Remark 51.

Proving this conjecture requires: (1) defining the geometric median precisely in the space of positive definite matrices equipped with the Fisher-Rao metric, (2) establishing existence and uniqueness of the median, (3) deriving breakdown bounds analogous to classical geometric median results, and (4) providing an efficient algorithm for computation. Each of these steps presents non-trivial technical challenges. We leave this as an important direction for future work, noting that if established, it would provide strong theoretical guarantees for distributed Fisher Flow in adversarial settings.

13.5 Fisher Flow for Non-Stationary Environments

We extend Fisher Flow to handle distribution shift:

Definition 52 (Adaptive Fisher Flow).

For time-varying distributions $p_{t}(x|\theta)$ , define:

	$\displaystyle\mathcal{I}_{t}$	$\displaystyle=\sum_{s=1}^{t}w_{t,s}\hat{\mathcal{I}}_{s}$	(weighted information)		(113)
	$\displaystyle w_{t,s}$	$\displaystyle=\exp(-\lambda(t-s))\cdot\text{TestStatistic}(s,t)$	(adaptive weights)		(114)

where TestStatistic $(s,t)$ measures distribution shift between times $s$ and $t$ .

Theorem 53 (Tracking Regret Bound).

For Adaptive Fisher Flow with appropriate $\lambda$ , the tracking regret satisfies:

R_{T}=\sum_{t=1}^{T}(\ell_{t}(\theta_{t})-\ell_{t}(\theta_{t}^{*}))\leq% \mathcal{O}(\sqrt{T(1+P_{T})})

(115)

where $P_{T}=\sum_{t=1}^{T}\|\theta_{t}^{*}-\theta_{t-1}^{*}\|$ is the path length of optimal parameters.

13.6 Connection to Optimal Transport

Fisher Flow bears a heuristic resemblance to gradient flows in Wasserstein space, suggesting potential connections to optimal transport theory. However, making this connection rigorous requires resolving the different geometries of parameter space $\Theta$ (Fisher-Rao metric) and distribution space $\mathcal{P}$ (Wasserstein metric). We defer detailed discussion to Appendix A.1.

14 Unifying Principles: Fisher Flow as a Meta-Framework

14.1 The Information-Action Duality

Fisher Flow reveals a fundamental duality in machine learning between information accumulation and parameter action:

Theorem 54 (Information-Action Duality).

Every Fisher Flow update can be decomposed into dual components:

	Information space:	$\displaystyle\mathcal{I}_{t+1}=\mathcal{I}_{t}+\hat{\mathcal{I}}_{\text{new}}$	(accumulation)		(116)
	Action space:	$\displaystyle\theta_{t+1}=\theta_{t}-\eta\mathcal{I}_{t}^{-1}s_{t}$	(movement)		(117)

These satisfy the conservation law:

\frac{d}{dt}\left(\frac{1}{2}\theta^{\top}\mathcal{I}\theta-\ell(\theta)\right% )=0

(118)

along the natural gradient flow trajectory.

Proof.

The conservation law follows from the Hamiltonian structure of natural gradient flow. Define the Hamiltonian:

H(\theta,p)=\frac{1}{2}p^{\top}\mathcal{I}^{-1}p+\ell(\theta)

(119)

where $p=\mathcal{I}\dot{\theta}$ is the momentum conjugate to $\theta$ .

The natural gradient flow satisfies Hamilton’s equations:

	$\displaystyle\dot{\theta}$	$\displaystyle=\frac{\partial H}{\partial p}=\mathcal{I}^{-1}p$		(120)
	$\displaystyle\dot{p}$	$\displaystyle=-\frac{\partial H}{\partial\theta}=-\nabla_{\theta}\ell$		(121)

Combining these yields the natural gradient equation, and the Hamiltonian is conserved along trajectories. ∎

14.2 PAC-Bayes Interpretation of Fisher Flow

Fisher Flow admits a PAC-Bayesian interpretation that provides non-asymptotic generalization bounds:

Theorem 55 (PAC-Bayes Bound for Fisher Flow).

With probability at least $1-\delta$ over the sample, for any posterior $Q$ centered at $\hat{\theta}_{n}$ with covariance $\mathcal{I}_{n}^{-1}$ :

\mathbb{E}_{\theta\sim Q}[R(\theta)]\leq\mathbb{E}_{\theta\sim Q}[\hat{R}_{n}(% \theta)]+\sqrt{\frac{D_{KL}(Q\|P)+\log(2\sqrt{n}/\delta)}{2n}}

(122)

where $R(\theta)$ is the true risk, $\hat{R}_{n}(\theta)$ is the empirical risk, and $P$ is a prior with covariance $\mathcal{I}_{0}^{-1}$ .

This shows that Fisher Flow naturally balances empirical fit with complexity control through the KL divergence term, which equals:

D_{KL}(Q\|P)=\frac{1}{2}\left[\log\frac{|\mathcal{I}_{0}|}{|\mathcal{I}_{n}|}+% \text{tr}(\mathcal{I}_{0}^{-1}\mathcal{I}_{n})-d+(\hat{\theta}_{n}-\theta_{0})% ^{\top}\mathcal{I}_{0}(\hat{\theta}_{n}-\theta_{0})\right]

(123)

14.3 Mirror Descent Interpretation

Fisher Flow can be viewed as mirror descent in the dual space defined by the log-partition function:

Theorem 56 (Fisher Flow as Mirror Descent).

The Fisher Flow update is equivalent to mirror descent with the Bregman divergence:

D_{\psi}(\theta,\theta^{\prime})=\psi(\theta)-\psi(\theta^{\prime})-\langle% \nabla\psi(\theta^{\prime}),\theta-\theta^{\prime}\rangle

(124)

where $\psi(\theta)=\frac{1}{2}\theta^{\top}\mathcal{I}(\theta)\theta$ is the potential function.

This reveals that different choices of $\psi$ recover different optimization algorithms: - $\psi(\theta)=\frac{1}{2}\|\theta\|^{2}$ : Standard gradient descent - $\psi(\theta)=\sum_{i}\theta_{i}\log\theta_{i}$ : Exponentiated gradient - $\psi(\theta)=\frac{1}{2}\theta^{\top}\mathcal{I}\theta$ : Natural gradient (Fisher Flow)

14.4 Minimum Description Length Principle

Fisher Flow implements an optimal coding strategy based on the Minimum Description Length (MDL) principle:

Theorem 57 (MDL Optimality of Fisher Flow).

The Fisher Flow estimate minimizes the two-part code length:

L(\theta,\mathcal{D})=L(\theta)+L(\mathcal{D}|\theta)

(125)

where $L(\theta)=\frac{1}{2}\log|\mathcal{I}|+\frac{d}{2}\log n$ is the model complexity and $L(\mathcal{D}|\theta)=-\ell(\theta)$ is the data encoding cost.

This provides an information-theoretic justification for Fisher Flow’s implicit regularization and connects to Rissanen’s MDL principle [28].

14.5 Emergence and Phase Transitions

The spectral dynamics of accumulated Fisher information may relate to emergence phenomena in large models, where complex behaviors appear as information crosses critical thresholds. This speculative direction is discussed in Appendix A.2.

15 Empirical Validation

We validate the core theoretical claims through systematic experiments on synthetic and real-world problems. Our experimental validation focuses on: (1) convergence rate validation (Theorems 4.2–4.4), (2) performance on real-world classification tasks, and (3) numerical stability improvements. All experiments use multiple random seeds with statistical significance testing, and code is available for reproducibility.

15.1 Convergence Rate Validation

We validate Theorems 4.2–4.4 on convex quadratic problems with varying condition numbers $\kappa\in\{1,100,1000\}$ (dimension $d=20$ , 500 iterations, 5 random seeds).

Well-conditioned ( $\kappa=1$ ): Standard FF achieves super-linear convergence with rate $-12.93\pm 0.04$ in log-log scale, reaching machine precision ( $3.6\times 10^{-15}$ ) in 230 iterations. This significantly outperforms SGD and Adam (both $-8.26$ , $p<0.001$ , Cohen’s $d=-64.7$ ), validating the $O(1/t)$ lower bound in Theorem 4.2.

Ill-conditioned ( $\kappa=100$ ): Momentum FF dramatically outperforms all baselines, achieving final distance $2.8\times 10^{-12}$ versus $6.2\times 10^{-4}$ (Standard FF) and $7.8\times 10^{-6}$ (Adam)—a 1000x improvement ( $p=0.008$ , Cohen’s $d=-3.50$ ). This demonstrates the practical value of information-geometric preconditioning on challenging problems.

Very ill-conditioned ( $\kappa=1000$ ): Both FF variants initially diverged. We improved numerical stability through adaptive damping: $\lambda_{\text{adaptive}}=\lambda_{0}\sqrt{\kappa/100}$ for $\kappa>100$ . The improved implementation remains stable up to $\kappa=5000$ , enabling convergence where previous methods failed.

15.2 Real-World Classification Tasks

We evaluate on 4 UCI classification benchmarks (logistic regression, 1000 iterations, 80/20 train/test split, 5 random seeds) comparing SGD (momentum=0.9), Adam, Standard FF (diagonal), and Momentum FF.

Table 5: Real-World Classification Results (Mean over 5 Seeds)

Dataset	$n$	$d$	SGD	Adam	FF	Momentum FF
Breast Cancer	455	30	0.9825	0.9737	0.9825	0.9737
Wine	142	13	0.9722	1.0000	0.9722	0.9722
Ionosphere	280	34	0.9155	0.9014	0.9014	0.9155
Sonar	166	60	0.7857	0.7619	0.8333^∗∗∗	0.7857

Main result: Standard FF achieves 83.3% test accuracy on Sonar (rocks vs mines classification), significantly outperforming SGD (78.6%, $+6.1\%$ , $p<0.001$ ) and Adam (76.2%, $+9.4\%$ , $p<0.001$ ). This improvement is statistically significant and consistent across all 5 seeds (std=0.0000). Sonar, the most challenging dataset (60 features, 166 training samples), demonstrates that natural gradient preconditioning provides substantial gains when curvature information is critical.

Test loss: Standard FF achieves best loss on 3/4 datasets: Breast Cancer (0.0716 vs 0.0733), Ionosphere (0.2148 vs 0.4443), Sonar (0.7543 vs 1.2648).

Computational efficiency: Standard FF adds only 18% overhead versus SGD (0.18s vs 0.16s per 1000 iterations), making the 6–9% accuracy improvement highly practical.

Dataset-dependent performance: Fisher Flow shows strongest gains on challenging datasets (Sonar: 60 features, high condition number) while offering no advantage on easy problems (Wine: perfect accuracy achievable by all methods). This aligns with theory: natural gradient preconditioning matters most when curvature varies significantly.

15.3 Summary and Implications

Our experiments provide strong empirical support for Fisher Flow:

•

Convergence theory validated: Observed rates match or exceed theoretical predictions (Theorem 4.2)
•

Practical utility demonstrated: 6–9% accuracy gains on challenging real-world tasks with minimal overhead
•

Numerical stability improved: Adaptive damping enables convergence up to $\kappa=5000$
•

Statistical rigor: All results replicated across 5 seeds with significance testing

Limitations: Momentum FF shows dataset-dependent performance, suggesting the theoretical $O(1/t^{2})$ rate (Proposition 11.1) may require additional assumptions or refinement. Future work should complete the convergence proof and conduct larger-scale experiments (CIFAR, ImageNet) to validate scalability.

16 Illustrative Example: Deep Learning Model Training

Consider training a deep neural network (DNN) for classification using a cross-entropy loss, which is equivalent to maximizing the log-likelihood of a categorical distribution. Fisher Flow provides a lens to understand and enhance this process:

•
Stochastic Updates as Fisher Flow Steps: Training with mini-batches can be viewed as a sequence of Fisher Flow updates. After processing $n$ observations, when we receive a new mini-batch $B$ with $|B|$ observations:
1. 1.
  
  The gradient of the loss $\nabla_{\theta}\mathcal{L}_{B}$ is the negative score $-s_{B}(\theta)$ .
2. 2.
  
  The (approximate) Fisher Information Matrix $\hat{\mathcal{I}}_{B}$ can be estimated (e.g., using empirical FIM, diagonal approximations like in Adam/RMSProp, or Kronecker-factored approximations).
3. 3.
  
  An optimizer step, especially one like natural gradient descent, takes the form $\theta_{n+|B|}\leftarrow\theta_{n}-\eta\,\hat{\mathcal{I}}_{B}^{-1}s_{B}(% \theta_{n})$ , directly analogous to the Fisher Flow update, or more generally, $\theta_{n+|B|}\leftarrow\mathcal{I}_{n+|B|}^{-1}(\mathcal{I}_{n}\theta_{n}+% \hat{\mathcal{I}}_{B}\hat{\theta}_{B})$ if we consider $\hat{\theta}_{B}$ as the conceptual MLE for that batch.
•

Information Accumulation and Regularization: The total information after $N$ observations $\mathcal{I}_{N}=\sum_{i=1}^{N}\hat{\mathcal{I}}_{i}$ (or equivalently $\sum_{\text{batches}}\hat{\mathcal{I}}_{B}$ ) reflects the model’s accumulated knowledge. Techniques like Elastic Weight Consolidation (EWC) [11] for continual learning explicitly use the FIM to penalize changes to parameters important for previous tasks, which is a direct application of Fisher Flow’s information-weighting principle.
•

Uncertainty and Model Analysis: Approximations to the FIM provide insights into parameter uncertainty, which, while not typically used for interpreting individual parameters in large DNNs, are instrumental for deriving predictive uncertainty for model outputs (e.g., class probabilities or next-token distributions). The inverse FIM, $\mathcal{I}_{t}^{-1}$ , offers a principled (though approximate) covariance matrix for $\hat{\theta}_{t}$ , forming the basis for sampling parameters to estimate the variability of predictions. Furthermore, FIM-derived metrics can identify parameter sensitivities, guide pruning or quantization, and inform training dynamics like early stopping based on information saturation.

While full FIM computation is often intractable for large DNNs, the Fisher Flow framework motivates and provides theoretical grounding for many successful heuristics and approximations used in modern deep learning, framing them as attempts to efficiently propagate likelihood-derived information.

17 Unified Theoretical Perspective

17.1 Fisher Flow as a Natural Geometric Flow

We can now present a unified view of Fisher Flow that connects its various mathematical aspects:

Conjecture 58 (Master Equation of Fisher Flow).

Under additional regularity and model assumptions, the Fisher Flow dynamics can be expressed equivalently as:

(Geometric):	$\displaystyle\frac{d\theta}{dt}=-\tilde{\nabla}\ell(\theta)$	(126)
(Variational):	$\displaystyle\theta_{t+\delta t}=\arg\min_{\theta}\left\{D_{KL}(p_{\theta}\\|p_% {\theta_{t}})-\delta t\cdot\ell(\theta)\right\}$	(127)
(Information):	$\displaystyle\frac{d\mathcal{I}}{dt}=\mathbb{E}[s(\theta)s(\theta)^{\top}]$	(128)

where all three formulations yield identical parameter trajectories.

This unification reveals Fisher Flow as a fundamental geometric principle rather than an ad-hoc algorithm.

17.2 Hierarchy of Approximations

Practical implementations form a hierarchy of approximations to the ideal Fisher Flow flow:

Approximation Level	Information Structure	Computational Cost
Exact Fisher Flow	Full $\mathcal{I}\in\mathbb{R}^{d\times d}$	$\mathcal{O}(d^{3})$
Block-diagonal	$\mathcal{I}=\bigoplus_{k}\mathcal{I}_{k}$	$\mathcal{O}(\sum_{k}d_{k}^{3})$
Kronecker-factored	$\mathcal{I}\approx A\otimes B$	$\mathcal{O}(m^{3}+n^{3})$
Diagonal (Adam-like)	$\mathcal{I}=\text{diag}(v)$	$\mathcal{O}(d)$
Scalar (SGD)	$\mathcal{I}=\lambda I$	$\mathcal{O}(1)$

Each level preserves different aspects of the geometric structure while trading off computational efficiency.

18 Future Vistas: Generalizations and Open Questions

18.1 Beyond Parameters: What Else Can We Propagate?

The Fisher Flow principle—propagating summary statistics rather than full distributions—suggests broader generalizations:

18.1.1 Moment Propagation Inference (MPI)

Instead of just mean and covariance (first two moments), propagate higher moments:

•

3rd moment: Captures skewness
•

4th moment: Captures heavy tails
•

Moment generating function: Captures entire distribution

18.1.2 Constraint Propagation Inference (CPI)

Propagate feasible regions rather than point estimates:

•

Linear constraints: Polytope propagation
•

Convex constraints: Ellipsoid propagation
•

Non-convex: Level set propagation

18.1.3 Evidence Propagation Inference (EPI)

Propagate model evidence for hypothesis testing:

•

Bayes factors as information
•

Model averaging through evidence accumulation
•

Online model selection

18.2 The Meta-Pattern: Sufficient Statistics Propagation

Fisher Flow is actually an instance of a more general pattern:

Core Principle: Instead of propagating full distributions, propagate sufficient statistics that capture the essential information for your inferential goal.

This suggests a research program:

1.

Identify the goal: What do you ultimately need? (point estimate, uncertainty, prediction, decision)
2.

Find sufficient statistics: What summary captures necessary information?
3.

Derive update equations: How do these statistics combine?
4.

Analyze approximations: When can we simplify?

18.3 Unexplored Territories

18.3.1 Fisher Flow for Causal Inference

Can we propagate causal information?

•

Interventional distributions as ”causal information”
•

Propagating do-calculus expressions
•

Online causal discovery through information geometry

18.3.2 Fisher Flow for Reinforcement Learning

Value functions and policies as information:

•

Bellman updates as information propagation
•

Policy gradients through Fisher information
•

Exploration as information seeking

18.3.3 Fisher Flow for Scientific Discovery

Hypothesis testing through information accumulation:

•

Experimental design as information maximization
•

Sequential hypothesis testing
•

Active learning guided by information geometry

18.4 The Philosophical Question: Is All Learning Information Propagation?

Fisher Flow suggests a profound possibility: perhaps all forms of learning can be understood as information propagation with different:

•

Carriers: What holds the information? (parameters, functions, graphs, programs)
•

Metrics: How do we measure information? (Fisher, Shannon, Kolmogorov)
•

Dynamics: How does information flow? (gradient, diffusion, message passing)
•

Objectives: What information do we seek? (discrimination, compression, prediction)

This perspective could unify:

•

Supervised learning: Propagate label information to parameters
•

Unsupervised learning: Propagate structure information to representations
•

Meta-learning: Propagate task information to priors
•

Transfer learning: Propagate domain information across tasks

18.5 A Call to Action

The Fisher Flow framework is not just a technical contribution—it’s an invitation to rethink learning through the lens of information propagation. By naming this pattern, we open doors to:

1.

New algorithms: Design methods by choosing what information to propagate
2.

Better understanding: Explain existing methods as information propagation variants
3.

Principled approximations: Trade computation for information fidelity systematically
4.

Cross-fertilization: Connect disparate fields through shared information principles

The question is not whether Fisher Flow is “correct”—it’s whether thinking about learning as information propagation leads to better algorithms, deeper insights, and new discoveries. Early evidence suggests it does.

19 Conclusion

This paper provides a unified information-geometric perspective on sequential maximum likelihood estimation, which we term Fisher Flow. Our contributions are primarily synthetic rather than introducing fundamentally new methods:

1. Unification and naming. We show that natural gradient descent [1], adaptive optimizers like Adam [10], Kronecker-factored methods [13], and continual learning approaches like EWC [11] are all instances of propagating Fisher information with different approximation structures. While these connections were implicit in prior work, making them explicit through a unified framework helps clarify design choices and trade-offs.

2. Two-regime formalization. We distinguish between:

•

The classical regime where optimization converges and standard MLE theory (consistency, asymptotic normality, Cramér-Rao efficiency) applies
•

The modern ML regime where training stops before convergence and Fisher information acts as trajectory-dependent geometric regularization

This distinction clarifies when classical statistical guarantees hold versus when we must reason about implicit regularization.

3. Approximation analysis. We characterize the error introduced by diagonal, block-diagonal, Kronecker-factored, and low-rank approximations to the Fisher information matrix (Theorems 18, 34, 35), providing rigorous guidance for computational trade-offs.

4. Novel variants (preliminary). We propose momentum-enhanced, adaptively compressed, and Byzantine-robust extensions, though these require substantial empirical validation before they can be recommended for practical use.

19.1 Limitations and Future Work

Several important limitations must be acknowledged:

1.

Experimental validation: Our empirical evaluation is preliminary. Comprehensive benchmarking on modern datasets with thorough baseline comparisons is essential future work.
2.

Open conjectures: The optimality (Conjecture 21), no-free-lunch (Conjecture 24), and Byzantine robustness (Conjecture 50) results remain open problems requiring additional theoretical work.
3.

Scalability: While we discuss computational complexity, actual implementation and scaling studies on large models (e.g., transformers with billions of parameters) are needed.
4.

Novel algorithms: The proposed variants (momentum-enhanced, Byzantine-robust, adaptive) are theoretically motivated but unvalidated. They may or may not prove useful in practice.
5.

Limited scope: Our framework applies primarily to likelihood-based learning. Extensions to other settings (reinforcement learning, causal inference, etc.) mentioned in Section 18 are speculative.

19.2 Contributions in Context

The value of this work lies not in novelty of the core mathematics—which builds on decades of research in information geometry [2], recursive estimation [12], and natural gradient methods [1]—but in:

•

Making implicit patterns explicit and systematic
•

Providing a unifying vocabulary (”Fisher Flow”) for discussing related methods
•

Formalizing the two-regime interpretation relevant to modern ML
•

Characterizing approximation errors rigorously

If this perspective helps researchers understand connections between methods, make principled approximation choices, or design new algorithms, it will have served its purpose. The framework is offered as a lens for viewing existing methods, not as a fundamentally new approach to machine learning.

Appendix A Extended Theoretical Perspectives

This appendix contains extended theoretical discussions that, while intellectually interesting, are either speculative, require substantial additional work to formalize, or represent future research directions rather than core contributions.

A.1 Optimal Transport and Wasserstein Geometry

Fisher Flow bears a heuristic resemblance to gradient flows in Wasserstein space, though making this connection rigorous remains an open problem.

Remark 59 (Heuristic Wasserstein Perspective).

The continuous-time Fisher Flow dynamics can be informally related to gradient flows on spaces of distributions:

\frac{d\theta}{dt}=-\nabla_{W_{2}}D_{KL}(\hat{p}_{n}\|p_{\theta})

(129)

where $\nabla_{W_{2}}$ denotes a Wasserstein gradient.

More formally, one might write:

\frac{\partial p_{\theta}}{\partial t}=\text{div}(p_{\theta}\nabla_{\theta}% \Phi(\theta))

(130)

where $\Phi(\theta)=-\ell(\theta)$ . However, establishing a rigorous connection requires careful analysis:

•

The parameter space $\Theta$ and the space of distributions $\mathcal{P}$ have different geometries
•

The natural gradient flow operates on $\Theta$ with the Fisher-Rao metric, while Wasserstein gradient flows operate on $\mathcal{P}$ with the 2-Wasserstein metric
•

The precise relationship depends on the parameterization and is not straightforward

Establishing this connection could enable analysis via optimal transport theory, including convergence via displacement convexity and stability via Wasserstein distance bounds. This remains an open question for future work.

A.2 Emergence and Phase Transitions

The spectral dynamics of accumulated Fisher information suggest intriguing connections to emergence phenomena in large models.

Conjecture 60 (Emergence Hypothesis).

Complex intelligent behaviors may emerge when the accumulated Fisher information $\mathcal{I}_{t}$ crosses critical thresholds corresponding to phase transitions in the model’s representational capacity. These transitions could be characterized by sudden changes in the spectrum of $\mathcal{I}_{t}$ .

This perspective suggests studying the spectral dynamics of Fisher information during training to predict emergent capabilities. The “critical information threshold” (related to Theorem 38 in the main text) provides one such characterization:

\gamma(\mathcal{I})=\frac{\lambda_{\text{min}}(\mathcal{I})}{\lambda_{\text{% max}}(\mathcal{I})}

(131)

When $\gamma(\mathcal{I})$ crosses a threshold, the model may transition between different capability regimes. However, this remains highly speculative and requires substantial empirical investigation.

A.3 Fisher Flow for Foundation Models

Large Language Models represent ambitious applications of likelihood-based estimation. While Fisher Flow’s core principles apply, practical uncertainty quantification at this scale remains challenging.

A.3.1 Predictive Uncertainty in LLM Outputs

The Fisher Flow framework’s parameter uncertainty ( $\hat{\theta}$ , $\hat{\mathcal{I}}^{-1}$ ) theoretically enables predictive uncertainty for next-token distributions:

Confidence Intervals for Token Probabilities:

1.

Parameter Sampling: Draw samples $\theta^{(s)}\sim\mathcal{N}(\hat{\theta},\hat{\mathcal{I}}^{-1})$
2.

Ensemble Predictions: Compute $p(v_{j}|\text{context},\theta^{(s)})$ for vocabulary tokens
3.

CI Estimation: Use empirical percentiles to construct intervals $[L_{j},U_{j}]$

Applications to Decoding Strategies:

These CIs could inform sampling strategies:

•

Robust Selection: Filter by lower confidence bound $L_{j}$
•

Exploratory Selection: Include tokens with high upper bounds $U_{j}$
•

Adaptive Nucleus: Adjust sampling based on aggregate uncertainty

However, computing and maintaining Fisher information approximations at the scale of modern LLMs (billions of parameters) presents substantial practical challenges that current methods do not adequately address.

References

[1] Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10(2), 251–276.
[2] Amari, S. (2016). Information Geometry and Its Applications. Springer.
[3] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805.
[4] Efron, B., Hinkley, D. V. (1978). Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information. Biometrika, 65(3), 457–487.
[5] Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver and Boyd.
[6] Guo, C., Pleiss, G., Sun, Y., Weinberger, K. Q. (2017). On calibration of modern neural networks. ICML (PMLR 70), 1321–1330.
[7] Hochreiter, S., Schmidhuber, J. (1997). Flat minima. Neural Computation, 9(1), 1–42.
[8] Jeffreys, H. (1939). Theory of Probability. Oxford University Press.
[9] Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., Tang, P. T. P. (2017). On large-batch training for deep learning: Generalization gap and sharp minima. ICLR.
[10] Kingma, D. P., Ba, J. (2014). Adam: A method for stochastic optimization. arXiv:1412.6980.
[11] Kirkpatrick, J. et al. (2017). Overcoming catastrophic forgetting in neural networks. PNAS, 114(13), 3521–3526.
[12] Ljung, L. (1983). Theory and Practice of Recursive Identification. MIT Press.
[13] Martens, J., Grosse, R. (2015). Optimizing neural networks with Kronecker-factored approximate curvature. ICML (PMLR 37), 1107–1115.
[14] Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., Wermter, S. (2019). Continual lifelong learning with neural networks: A review. Neural Networks, 113, 54–71.
[15] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. (2018). Improving language understanding by generative pre-training. OpenAI Blog.
[16] Rao, C. R. (1945). Information and the accuracy attainable in the estimation of statistical parameters. Bull. Calcutta Math. Soc., 37, 81–91.
[17] Robert, C. P. (2007). The Bayesian Choice. Springer.
[18] Settles, B. (2009). Active learning literature survey. Technical Report 1648, University of Wisconsin–Madison.
[19] Tierney, L., Kadane, J. B. (1986). Accurate approximations for posterior moments and marginal densities. JASA, 81(393), 82–86.
[20] van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press.
[21] Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y. (2020). The curious case of neural text degeneration. ICLR.
[22] Neyshabur, B., Tomioka, R., Srebro, N. (2015). In search of the real inductive bias: On the role of implicit regularization in deep learning. ICLR Workshop.
[23] Gunasekar, S., Lee, J., Soudry, D., Srebro, N. (2018). Characterizing implicit bias in terms of optimization geometry. arXiv:1802.08246.
[24] Gupta, V., Koren, T., Singer, Y. (2018). Shampoo: Preconditioned stochastic tensor optimization. ICML (PMLR 80), 1842–1850.
[25] You, Y., Gitman, I., Ginsburg, B. (2017). Large batch training of convolutional networks. arXiv:1708.03888.
[26] You, Y., et al. (2020). Large batch optimization for deep learning: Training BERT in 76 minutes. ICLR.
[27] Dean, J., et al. (2012). Large scale distributed deep networks. NeurIPS 25, 1223–1231.
[28] Rissanen, J. (1978). Modeling by shortest data description. Automatica, 14(5), 465–471.

	$\displaystyle\\|e_{n}\\|_{\mathcal{I}}$	$\displaystyle\leq\sum_{k=0}^{n-1}\left(\frac{\epsilon}{1-\epsilon}\right)^{k}% \cdot\eta\\|\mathcal{I}_{k}^{-1}-\tilde{\mathcal{I}}_{k}^{-1}\\|_{F}\\|s_{k}\\|$		(53)
		$\displaystyle\leq\frac{\epsilon}{1-\epsilon}\\|\hat{\theta}_{n}-\theta_{0}\\|_{% \mathcal{I}}+\mathcal{O}(\delta/\sqrt{n})$		(54)

Fisher Flow: A Unified Information-Geometric Perspective on Sequential Maximum Likelihood Estimation and Natural Gradient Methods

Abstract

1 Introduction

1.1 Motivating Example: Online Linear Regression

1.2 Problem Statement and Motivation

1.3 Contributions

2 Mathematical Foundations

2.1 Notation and Preliminaries

2.1.1 Score and Information Notation

2.2 Statistical Manifolds and Information Geometry

Definition 1 (Statistical Manifold).

Definition 2 (Fisher Information Matrix).

2.3 Regularity Conditions

Assumption 3 (Regularity).

2.4 Information Accumulation and the Additive Property

Theorem 4 (Information Additivity).

Proof.

3 Fisher Flow in Plain English: The Core Insight

3.1 The Fundamental Pattern

3.2 A Simple Analogy: The Wisdom of Crowds

3.3 Why This Matters: The Power of a Name

3.4 The Fisher Flow Taxonomy: A Family of Methods

3.5 The Deeper Pattern: Information as Currency

3.6 When to Use What: A Practical Guide

4 The Fisher Flow Framework

4.1 Axiomatic Foundation

Axiom 5 (Information Monotonicity).

Axiom 6 (Geometric Covariance).

Axiom 7 (Local Sufficiency).

4.2 Core Update Equations

Definition 8 (Fisher Flow State).

Theorem 9 (Natural Gradient Flow).

Proof.

4.3 Information Combination and Optimality

Theorem 10 (Optimal Information Fusion).

Proof.

4.4 Sequential Update Algorithm

5 Asymptotic Theory and Convergence Guarantees

5.1 Consistency and Asymptotic Normality

Theorem 11 (Strong Consistency of Fisher Flow).

Proof.

Theorem 12 (Asymptotic Normality and Efficiency).

Proof.

5.2 Non-Asymptotic Bounds

Theorem 13 (Finite-Sample Concentration).

Proof.

5.3 Fisher Flow Away from Optima: From Classical Statistics to Modern ML

5.3.1 Reinterpreting Fisher Flow for Non-Convergent Settings

Definition 14 (Trajectory-Dependent Fisher Information).

Proposition 15 (Path-Dependent Regularization).

Proof.

5.3.2 Implications for Modern Deep Learning

Remark 16 (Two Regimes of Fisher Flow).

5.4 Approximation Theory for Relaxed Information Geometry

Definition 17 ((ϵ,δ)-Approximate Information).

Theorem 18 (Robustness to Information Approximation).

Proof.

6 Related Work

6.1 Information Geometry and Natural Gradient

6.2 Recursive Estimation and Adaptive Filtering

6.3 Second-Order Optimization Methods

6.4 Bayesian Deep Learning and Uncertainty Quantification

6.5 Continual Learning

6.6 Approximation Theory and Matrix Factorizations

6.7 Implicit Regularization and Generalization

6.8 Large-Scale and Distributed Optimization

6.9 What is Novel in Our Work

7 Deep Parallels to Bayesian Inference

8 Theoretical Guarantees and Limitations

8.1 When Fisher Flow Fails: Limitations and Failure Modes

Example 19 (Mixture Models).

Example 20 (Heavy-Tailed Data).

8.2 Optimality Properties

Conjecture 21 (Information-Theoretic Optimality).

Remark 22.

Theorem 23 (Invariance Properties).

Proof.

8.3 Fundamental Limitations

Conjecture 24 (No Free Lunch for Information Geometry).

Remark 25.

Definition 17 ( $(\epsilon,\delta)$ -Approximate Information).

Definition 29 ( $\alpha$ -Connections).