Sakana researchers present training algorithm enabling 1,000-layer networks without backpropagation

Predictive coding treats every hidden activation as an optimization variable and penalizes the squared gap between each layer's activation and the prediction arriving from the layer below. Inference is gradient descent on that energy; learning is a Hebbian-style weight step. The problem: supervision enters at the output and must propagate through a chain of local compromises. In deep, narrow networks the credit signal decays long before it reaches the input. Innocenti and colleagues characterized the gap between predictive coding and backpropagation as a function of width and depth, showing it becomes severe precisely when width falls below depth.

PC-ALM starts from the constrained formulation of training: minimize the supervised loss subject to a per-layer constraint. Predictive coding is a quadratic-penalty relaxation of that problem. PC-ALM uses the augmented Lagrangian instead, adding a Lagrange multiplier for each layer constraint while retaining the predictive-coding penalty. Setting λ = 0 recovers predictive coding exactly. Inference alternates two local steps: a primal gradient step on the activations and a dual step that accumulates the layer's prediction error. Completing the square shows that every primal step is a standard predictive-coding step with a prediction target shifted by the multiplier. After T steps the weight update acts on the composite signal. The researchers describe this as a PI controller for each layer: the prediction error is the proportional term and the multiplier is the integral term. α = 0 gives predictive coding; α = ρ with an exact solution of the inner problem gives the classical method of multipliers.

LeCun noted in 1988 that the multipliers of a constrained network equal the backpropagation adjoints at the KKT point. The team proves that in linear predictive-coding networks, under a spectral-radius stability condition, PC-ALM converges to that same KKT point: the activations return to their forward-pass values while every multiplier locks onto the exact backpropagation adjoint. The per-mode stability bound reduces to the predictive-coding condition when α = 0. Unlike predictive coding's monotonic gradient flow, the PC-ALM iteration matrix contains complex eigenvalues that produce damped oscillations; α sets the frequency but not the decay rate.

The team swept RMLP of widths and depths from 8 to 128 on Fashion-MNIST and MNIST under the Innocenti parameterization, with a single epoch. At an inference budget of T = 2L, PC-ALM matches backpropagation at every width, depth, and activation function (identity, tanh, ReLU), while predictive coding collapses in the deep, narrow cells. The reference cell in the repository (width 32, depth 32, ReLU, Fashion-MNIST) reports 78.66% accuracy for backpropagation, 68.13% for predictive coding, and 77.75% for PC-ALM, with gradient cosine similarity to backpropagation rising from 0.604 to 0.909. The experiment extended to 1,000 layers on MNIST (width 32, ReLU, 5 epochs) and remained within roughly 2 points of backpropagation. Improvement was also recorded on ResNet-18 on CIFAR-10 and Tiny ImageNet.

The implementation is available under an MIT license in JAX and runs on CPU, reproducing the paper's width-depth grid. It is important to emphasize: this is a training method, not a model, and it has been tested only on small image benchmarks. No performance metrics exist for language, multimodal, or production-scale settings. The results are impressive within the defined experimental regime, but the path to practical deployment remains long.