The Geometry of Invisible Forces: A Quartic Detection Theory
quartic detection law, hidden variables, Fisher information, KL divergence, statistical manifold, cross-spectral geometry, information geometry
The Problem: Forces You Cannot See
Hidden degrees of freedom are everywhere. The ocean drives climate through modes no single weather station resolves. Latent neural populations shape the signals recorded by any one electrode. Slow institutional forces move markets through channels invisible to any single asset’s price history. The central question is deceptively simple: given noisy observations, can you tell whether a hidden force is present?
Classical statistics offers a reassuring answer — collect enough data and any nonzero signal emerges. Fisher information scales as \(N\) (the sample size), so the detection boundary shrinks as \(N^{-1/2}\). For a coupling strength \(\lambda\), you need \(N \sim \lambda^{-2}\) observations. Reasonable. Manageable.
This answer is wrong. Not slightly wrong — wrong by orders of magnitude.
The Quartic Law: Why Detection Is Exponentially Harder
Consider the simplest possible hidden-variable problem. An observed time series \(X_t\) obeys
\[ X_{t+1} = a\, X_t + \lambda\, F_t + \varepsilon_t \]
where \(F_t\) is a hidden persistent driver (\(F_{t+1} = b\, F_t + \eta_t\)) and \(\varepsilon_t\) is noise. The coupling \(\lambda\) controls how strongly the hidden world leaks into the visible one.
The power spectrum of \(X_t\) is indeed perturbed at \(O(\lambda^2)\). So far, so classical. But here is the catch: when you refit a reduced model (one without the hidden variable), the best-fit parameters shift to absorb most of that perturbation. The refitted model “explains away” the hidden signal by reparametrizing itself.
What survives is not the \(O(\lambda^2)\) perturbation, but only its normal component — the part that cannot be absorbed by any reparametrization of the reduced model. And this residual is \(O(\lambda^4)\).
The result is the quartic detection law:
\[ D_{\mathrm{KL}}^{\min}(\lambda) = C\,\lambda^4 + O(\lambda^6) \]
where \(C\) is a system-specific constant and \(D_{\mathrm{KL}}^{\min}\) is the minimum Kullback-Leibler divergence between truth and the best-fit reduced model. The detection boundary becomes
\[ \boxed{\lambda_c(N) \;\propto\; \left(\frac{\log N}{N}\right)^{1/4}} \]
For a 10% coupling (\(\lambda/\sigma = 0.1\)), the quartic penalty demands \(\sim 10^3\) times more data than the naive quadratic expectation. This is not a small correction — it is a qualitative change in what is experimentally feasible.
The Geometry: Tangent Absorption on Statistical Manifolds
The quartic law is not an accident of the specific model. It is a geometric theorem on statistical manifolds.
Let \(\mathcal{M}_0\) be the manifold of reduced (null) models, parametrized by \(\boldsymbol{\theta}\). The true distribution under hidden forcing is
\[ p_{\mathrm{true}}(x \mid \lambda) = p_0(x \mid \boldsymbol{\theta}_0)\,\bigl(1 + \lambda^2\, h(x) + O(\lambda^4)\bigr) \]
The perturbation \(h\) has a unique decomposition into tangent and normal components relative to \(\mathcal{M}_0\):
\[ h = \underbrace{\Pi_T\, h}_{\text{absorbed by refitting}} + \underbrace{R}_{\text{detectable residual}} \]
where \(\Pi_T\) is the \(L^2(p_0)\)-projection onto the tangent space \(T_{\boldsymbol{\theta}_0}\mathcal{M}_0\). The tangent part is indistinguishable from a parameter shift; only the normal residual \(R = (I - \Pi_T)\,h\) is genuinely detectable. Since \(h\) enters at \(O(\lambda^2)\), the squared residual — hence the KL divergence — is \(O(\lambda^4)\).
This is Efron’s statistical curvature repurposed for detection: the “curvature” of the embedding of truth relative to the null manifold controls how much signal survives refitting.
The Pairing Principle: One Probe Is Blind
The geometry yields a stark operational consequence. Consider \(n\) sensors sharing a common hidden source:
\[ Y_{ij} = \mu + \lambda\, u_i + \varepsilon_{ij}, \qquad u_i \sim N(0,1) \]
The quartic coefficient is
\[ C = \frac{n(n-1)}{4\sigma^4} \]
The combinatorial factor \(n(n-1) = 2\binom{n}{2}\) counts sensor pairs sharing the hidden effect. When \(n = 1\), \(C = 0\) identically — no amount of data can detect the hidden variable from a single probe. This is not a power problem; it is a structural impossibility.
The pairing principle states:
Detecting a hidden degree of freedom requires at least two probes sharing the hidden effect. Detection power grows as the number of probe pairs \(\binom{n}{2}\), with diminishing returns: \(\lambda_c \propto n^{-1/2}\), not \(n^{-1}\).
For Gaussian observations, this is exact. A single electrode in a neural recording, a single climate station, a single asset price — each is provably blind to a latent common driver, regardless of record length.
The Dark Regime: When Timescales Coalesce
Even with the quartic law in hand, there is a deeper obstruction. In the spectral setting, the quartic coefficient takes the closed form
\[ C(a, b) = \frac{\sigma_\eta^4\, b^2\, (a - b)^2}{2\,\sigma_\varepsilon^4\,(1 - b^2)^3\,(1 - ab)^2} \]
This coefficient vanishes when \(a = b\) — when the intrinsic relaxation timescale of the observed system exactly matches that of the hidden driver. At this timescale coalescence, the hidden driver’s spectral signature is perfectly tangent to the null manifold. The hidden forcing becomes spectrally dark: present, dynamically active, yet locally invisible to any single-channel spectral test.
The data cost diverges as the dark boundary is approached:
\[ \lambda_c \;\propto\; |a - b|^{-1/2} \to \infty \quad \text{as } a \to b \]
This is not merely a detection difficulty — it is a geometric singularity of the inference problem itself.
Breaking the Impossibility: The Cross-Spectral Escape
The single-channel impossibility is not the end of the story. It is the beginning of a deeper one.
Lucente et al. proved that no time-irreversibility measure can detect departure from equilibrium in a scalar Gaussian time series from a linear system. This seems like a fundamental wall. But it has a geometric loophole: it applies only to diagonal (single-channel) observations.
When a second channel shares the same hidden driver, the cross spectrum — the off-diagonal block of the spectral matrix — provides a detection channel that is orthogonal to the entire diagonal null manifold. The cross-spectral contribution obeys its own quartic law:
\[ D_{\mathrm{cross}}(\lambda) = C_{\mathrm{cross}}\,\lambda^4 + O(\lambda^6) \]
with a remarkable property: the coefficient \(C_{\mathrm{cross}}\) is exactly independent of the observed-channel dynamics. All dependence on the transfer functions \(H_1, H_2\) of the observed channels cancels identically. The cross-spectral detectability is determined solely by the hidden mode’s spectral density:
\[ C_{\mathrm{cross}} = \frac{u_1^2\, u_2^2}{\sigma_{\varepsilon_1}^2\, \sigma_{\varepsilon_2}^2} \cdot I_F, \qquad I_F = \frac{1}{4\pi}\int_{-\pi}^{\pi} S_F(\omega)^2\, d\omega \]
Crucially, \(C_{\mathrm{cross}} > 0\) at exact timescale coalescence, where all single-channel measures vanish. The cross spectrum breaks the dark regime.
The Thermodynamic Bridge: Detectability Certifies Irreversibility
The connection deepens at the level of thermodynamics. For a one-way coupled Ornstein-Uhlenbeck system, the full-system entropy production rate (EPR) is exactly
\[ \dot{\Sigma}_{\mathrm{total}} = \alpha_2\,\lambda^2 \]
and the EPR-detectability bridge reads
\[ \dot{\Sigma}_{\mathrm{total}}^2 = \frac{\alpha_2^2}{C_{\mathrm{cross}}}\, D_{\mathrm{cross}} + O(\lambda^6) \]
This means: if the cross-spectral divergence is positive, the full system is certifiably out of equilibrium. Cross-spectral structure witnesses entropy production even when every single-channel estimator of time-irreversibility returns zero.
A single probe can sit inside a system with arbitrarily large true entropy production and measure exactly zero irreversibility. Two probes, sharing the same hidden bath, break this thermodynamic blindness.
The Unified Picture
The three results assemble into a coherent geometric theory of hidden-variable detectability:
| Layer | Result | Key Object |
|---|---|---|
| Universal law | Quartic onset \(D \sim \lambda^4\) | Normal residual \(R = (I - \Pi_T)\,h\) |
| Structural impossibility | Single probe is blind (\(C = 0\) at \(n=1\)) | Pairing principle: need \(\geq 2\) probes |
| Spectral darkness | \(C \to 0\) at timescale coalescence | Tangent alignment of hidden spectrum |
| Cross-spectral escape | \(C_{\mathrm{cross}} > 0\) at coalescence | Off-diagonal orthogonality |
| Thermodynamic certification | \(D_{\mathrm{cross}} > 0 \Rightarrow \dot{\Sigma} > 0\) | EPR-detectability bridge |
The hierarchy is:
- A single observation channel is exactly dark — hidden forcing is indistinguishable from equilibrium, for all coupling strengths, at all sample sizes.
- Two channels sharing the hidden driver restore quartic detectability via the cross spectrum, even at the coalescence singularity where auto-spectral methods fail.
- The cross-spectral witness certifies that the full system is out of equilibrium, linking observable spectral structure to the thermodynamic arrow of time.
Implications
For neuroscience: Detecting latent neural inputs requires multi-channel recording. The pairing principle prescribes exactly how many electrodes are needed given estimates of coupling and noise. A single fMRI voxel or EEG electrode is provably blind to a shared latent source.
For climate science: Detecting unresolved ocean forcing in climate records requires spatially distributed stations. A single station, no matter how long the record, cannot distinguish forced variability from intrinsic noise. The dark regime at timescale coalescence explains why slow ocean modes are so difficult to identify.
For stochastic thermodynamics: The single-channel impossibility is not a limitation of current methods — it is a geometric fact about the statistical manifold. The cross-spectral escape provides a constructive route to certifying irreversibility from partial observations.
For experimental design: The quartic law transforms the question “how much data do I need?” into “how many probes do I need, and where?” The answer: at least two, sharing the hidden effect, with detection power growing as the number of probe pairs.
Papers in This Series
Yuda Bi, Vince D. Calhoun. Why Single Probes Cannot Detect Hidden Forcing: A Quartic Detection Law. Under review at Physical Review Letters.
Yuda Bi, Vince D. Calhoun. Cross Spectra Break the Single-Channel Impossibility. Under review at Physical Review Letters.
Yuda Bi, Chenyu Zhang, Vince D. Calhoun. Timescale Coalescence Makes Hidden Persistent Forcing Spectrally Dark. Under review at Physical Review E. arXiv:2603.20917
Yuda Bi, Vince D. Calhoun. Conditioning on a Volatility Proxy Compresses the Apparent Timescale of Collective Market Correlation. Under review at Physical Review E. arXiv:2603.14072