Nonlinear ICA非线性 ICA

13.1 Nonlinear mixing非线性混合

The linear model $\bx = \bA\bs$ is an idealization. Sensors saturate, images are formed through occlusion, shading and perspective, and the factors that generate natural data are rarely added together with fixed weights. The natural generalization keeps independent sources but replaces the matrix by a function:

线性模型 $\bx = \bA\bs$ 是一种理想化。传感器会饱和；图像经由遮挡、明暗与透视而形成；生成自然数据的各种因素，也很少以固定的权重简单相加。最自然的推广是保留相互独立的源，而把矩阵换成一个函数：

where $\mathbf{f}\colon \R^n \to \R^n$ is an unknown invertible function, the mixing function. We assume as many observations as sources and, where needed, that $\mathbf f$ and $\mathbf f^{-1}$ are smooth. Nonlinear ICA seeks a function $\mathbf g$ such that the components of $\by = \mathbf g(\bx)$ recover the sources. With $\mathbf f(\bs) = \bA\bs$ this is ordinary ICA, which Chapter 4第 4 章 showed to be solvable up to scaling and permutation whenever at most one source is Gaussian.

其中 $\mathbf{f}\colon \R^n \to \R^n$ 是未知的可逆函数，称为混合函数。我们假设观测维数与源的个数相同，并在需要时假设 $\mathbf f$ 与 $\mathbf f^{-1}$ 都是光滑的。非线性 ICA 要找一个函数 $\mathbf g$，使 $\by = \mathbf g(\bx)$ 的各分量恢复出源。当 $\mathbf f(\bs) = \bA\bs$ 时，这就是普通的 ICA；Chapter 4第 4 章已经证明，只要至多一个源是高斯的，它就能在尺度与置换的意义下求解。

Some ambiguity is unavoidable, and it is larger than in the linear case. Let $\pi$ be a permutation of $\{1, \dots, n\}$ and $h_1, \dots, h_n$ invertible scalar functions, and set $\tilde s_i = h_i(s_{\pi(i)})$. Each $\tilde s_i$ is a function of a different source, so the $\tilde s_i$ are again independent. Write $\mathbf h$ for the invertible map $\bs \mapsto \tilde\bs$; then $\bx = \mathbf f(\bs) = \tilde{\mathbf f}(\tilde\bs)$ with $\tilde{\mathbf f} = \mathbf f \circ \mathbf h^{-1}$. The data cannot tell $(\mathbf f, \bs)$ from $(\tilde{\mathbf f}, \tilde\bs)$, so the best one can hope for is the following.

某种不确定性是无法避免的，而且比线性情形更大。设 $\pi$ 是 $\{1, \dots, n\}$ 的一个置换，$h_1, \dots, h_n$ 是可逆的标量函数，令 $\tilde s_i = h_i(s_{\pi(i)})$。每个 $\tilde s_i$ 只是某一个源的函数，且所涉及的源各不相同，因此 $\tilde s_i$ 仍然相互独立。记 $\mathbf h$ 为可逆映射 $\bs \mapsto \tilde\bs$，则 $\bx = \mathbf f(\bs) = \tilde{\mathbf f}(\tilde\bs)$，其中 $\tilde{\mathbf f} = \mathbf f \circ \mathbf h^{-1}$。数据无法区分 $(\mathbf f, \bs)$ 与 $(\tilde{\mathbf f}, \tilde\bs)$，所以最理想的结果也只能是下面这种意义下的分离。

Two features of this definition matter later. First, the marginal distributions of the sources carry no information. With $F_i$ the cumulative distribution function (CDF) of $s_i$ and $\Phi$ the standard Gaussian CDF, the map $h_i = \Phi^{-1}\circ F_i$ turns any source with a continuous, strictly increasing CDF into a standard Gaussian variable (Exercise 13.1习题 13.1). Non-Gaussianity of the marginals, the basis of linear ICA, is therefore useless here. Second, since one may as well assume Gaussian sources, the rotational ambiguity of Gaussian vectors from Section 1.4第 1.4 节 is never far away. The next section turns this observation into a theorem.

这个定义有两点后面会用到。第一，源的边缘分布不携带任何信息。记 $F_i$ 为 $s_i$ 的累积分布函数（CDF），$\Phi$ 为标准高斯分布的 CDF，则映射 $h_i = \Phi^{-1}\circ F_i$ 能把任何 CDF 连续且严格递增的源变成标准高斯变量（Exercise 13.1习题 13.1）。线性 ICA 赖以成立的边缘非高斯性，在这里毫无用处。第二，既然不妨假设源都是高斯的，Section 1.4第 1.4 节中高斯向量的旋转不确定性便近在眼前。下一节把这一观察变成定理。

13.2 Independence is not enough仅有独立性是不够的

The basic tool is the probability integral transform: if a random variable $u$ has a continuous CDF $F$, then $F(u)$ is uniform on $(0, 1)$. Applying it one coordinate at a time, each time conditionally on the coordinates already processed, turns any random vector into one with independent uniform components. Write $F_{k\mid 1:k-1}(\cdot \mid x_1, \dots, x_{k-1})$ for the conditional CDF of $x_k$ given $x_1, \dots, x_{k-1}$, and define

基本工具是概率积分变换：若随机变量 $u$ 的 CDF $F$ 是连续的，则 $F(u)$ 在 $(0, 1)$ 上均匀分布。逐个坐标地使用这一变换，每次都以已经处理过的坐标为条件，就能把任何随机向量变成分量独立且均匀分布的向量。记 $F_{k\mid 1:k-1}(\cdot \mid x_1, \dots, x_{k-1})$ 为给定 $x_1, \dots, x_{k-1}$ 时 $x_k$ 的条件 CDF，定义

where $F_1$ is the CDF of $x_1$. This triangular map is often called the Darmois construction; Hyvärinen and Pajunen (1999) used it to show that nonlinear ICA always has solutions, and far too many of them. It is a nonlinear counterpart of Gram–Schmidt orthogonalization: the $k$-th output keeps only what $x_k$ adds to the coordinates before it, and for a Gaussian vector the construction reduces to Cholesky whitening followed by $\Phi$ (Exercise 13.2习题 13.2).

其中 $F_1$ 是 $x_1$ 的 CDF。这个三角形映射通常称为 Darmois 构造；Hyvärinen 与 Pajunen（1999）用它证明了非线性 ICA 总是有解，而且解多得过了头。它相当于 Gram–Schmidt 正交化的非线性版本：第 $k$ 个输出只保留 $x_k$ 相对于前面各坐标新增的部分；对高斯向量而言，这一构造就退化为 Cholesky 白化再接上 $\Phi$（Exercise 13.2习题 13.2）。

Part (a) applies to any random vector, mixture or not. Independent outputs therefore cannot certify that a nonlinear unmixing has found the sources, even when the mixing happens to be linear.

(a) 部分适用于任何随机向量，不论它是否来自混合。因此，即使混合恰好是线性的，输出相互独立也不能证明一个非线性解混找到了源。

The following script checks both claims on $10^5$ samples.

下面的脚本在 $10^5$ 个样本上检验这两点。

import numpy as np
from scipy import stats

rng = np.random.default_rng(1)
T = 100_000
s1 = rng.uniform(0, 1, T)                 # two independent sources
s2 = rng.uniform(0, 1, T)
x1, x2 = s1, s1 + s2                      # a linear mixture

# Darmois construction, taking x2 first and x1 second
y1 = np.where(x2 <= 1, x2**2 / 2, 1 - (2 - x2)**2 / 2)   # CDF of x2 (triangular)
lo, hi = np.maximum(0, x2 - 1), np.minimum(1, x2)         # given x2, x1 ~ U(lo, hi)
y2 = (x1 - lo) / (hi - lo)                                # conditional CDF of x1

# Test that (y1, y2) is uniform on the unit square: chi-square on a 10 x 10 grid
H, _, _ = np.histogram2d(y1, y2, bins=10, range=[[0, 1], [0, 1]])
chi2 = ((H - T / 100) ** 2 / (T / 100)).sum()
print(f"chi2 = {chi2:.1f} on 99 df, p = {stats.chi2.sf(chi2, 99):.2f}")
# ... yet each output depends on both sources
print(np.round(np.corrcoef([y1, y2, s1, s2])[:2, 2:], 2))

It prints $\chi^2 = 91.3$ on 99 degrees of freedom ($p = 0.70$; over repeated seeds the $p$-values spread evenly over $(0,1)$, as they should under independence) and the correlations of $(y_1, y_2)$ with $(s_1, s_2)$, $\bigl(\begin{smallmatrix} 0.70 & 0.70 \\ 0.67 & -0.67 \end{smallmatrix}\bigr)$: both outputs depend strongly on both sources. In Figure 13.1图 13.1, lines of constant $s_1$ and of constant $s_2$, straight in the $\bx$ plane, become curves in the $\by$ square.

脚本输出 $\chi^2 = 91.3$（自由度 99，$p = 0.70$；换用不同的随机种子重复，$p$ 值均匀地散布在 $(0,1)$ 上，与独立性相符），以及 $(y_1, y_2)$ 与 $(s_1, s_2)$ 的相关系数 $\bigl(\begin{smallmatrix} 0.70 & 0.70 \\ 0.67 & -0.67 \end{smallmatrix}\bigr)$：两个输出都与两个源强相关。在Figure 13.1图 13.1 中，$\bx$ 平面里的 $s_1$ 等值线与 $s_2$ 等值线都是直线，到了 $\by$ 正方形中却成了曲线。

Part (b) of the theorem rests on a symmetry: a map that sends a distribution with independent components onto itself while mixing the coordinates. Rotations of $\Ncal(\bzero, \bI)$ are the simplest such maps, but there are many more. Let $\bR(\alpha)$ denote the rotation of the plane by the angle $\alpha$ and define

定理的 (b) 部分依赖于一种对称性：存在把分量独立的分布映到自身、同时又把坐标混在一起的映射。$\Ncal(\bzero, \bI)$ 的旋转是这类映射中最简单的，但远不止于此。记 $\bR(\alpha)$ 为平面上转角为 $\alpha$ 的旋转，定义

which turns each circle centered at the origin by an angle proportional to its squared radius. If $\bz \sim \Ncal(\bzero, \bI_2)$, then $\mathbf m(\bz) \sim \Ncal(\bzero, \bI_2)$ (Exercise 13.3习题 13.3): its components are independent standard Gaussians, although each depends on both $z_1$ and $z_2$ in a complicated way (Figure 13.2图 13.2). The map is smooth with a smooth inverse, so requiring $\mathbf g$ to be smooth does not remove the ambiguity. Composed with the componentwise maps $\Phi^{-1}\circ F_i$ and their inverses, as in the proof of part (b), such measure-preserving maps produce a continuum of spurious solutions, which are smooth when $\mathbf f$ and $\mathbf f^{-1}$ are smooth and the source densities are smooth and positive on their intervals. Without further information, independence of the outputs says almost nothing about whether they are the sources.

它把每个以原点为圆心的圆旋转一个与半径平方成正比的角度。若 $\bz \sim \Ncal(\bzero, \bI_2)$，则 $\mathbf m(\bz) \sim \Ncal(\bzero, \bI_2)$（Exercise 13.3习题 13.3）：它的两个分量是相互独立的标准高斯变量，但每个分量都以复杂的方式同时依赖 $z_1$ 与 $z_2$（Figure 13.2图 13.2）。这个映射光滑且逆映射也光滑，所以要求 $\mathbf g$ 光滑并不能消除不确定性。像 (b) 部分的证明那样，把这类保测映射与逐分量映射 $\Phi^{-1}\circ F_i$ 及其逆复合起来，就得到一整族连续变化的伪解；当 $\mathbf f$ 与 $\mathbf f^{-1}$ 光滑、源密度光滑且在各自区间上为正时，这些伪解也是光滑的。没有其他信息时，输出相互独立几乎不能说明它们是不是源。

13.3 Restricting the mixing function限制混合函数

Theorem 13.2定理 13.2 leaves two ways forward: restrict the class of mixing functions, or give the sources more structure than independence. The oldest restriction is the post-nonlinear (PNL) model of Taleb and Jutten (1999), in which a linear mixture passes through an invertible nonlinearity in each sensor,

Theorem 13.2定理 13.2留下了两条出路：限制混合函数的类别，或者赋予源比独立性更多的结构。最早的限制是 Taleb 与 Jutten（1999）提出的后非线性（post-nonlinear, PNL）模型：线性混合在每个传感器中再经过一个可逆的非线性变换，

as happens when sensors respond nonlinearly, for instance by saturating. The separating structure mirrors the model: first a scalar nonlinearity $g_i$ in each channel, then a matrix, $\by = \bW\,\mathbf g(\bx)$, where $\mathbf g$ applies $g_i$ to the $i$-th coordinate. If every composition $g_i \circ f_i$ is affine, what remains is a linear ICA problem. Taleb and Jutten (1999) showed that, under conditions on $\bA$, the sources and the nonlinearities, independence of the outputs forces exactly this: each $g_i\circ f_i$ is affine and $\bW\bA$ is a scaled permutation, so the sources are recovered up to the usual scaling, permutation and additive constants. The conditions have been weakened in later work, and we do not state them precisely here.

传感器响应非线性（例如饱和）时就是这种情形。分离结构与模型相对应：先在每个通道上作用一个标量非线性函数 $g_i$，再乘以一个矩阵，即 $\by = \bW\,\mathbf g(\bx)$，其中 $\mathbf g$ 把 $g_i$ 作用于第 $i$ 个坐标。如果每个复合函数 $g_i \circ f_i$ 都是仿射的，剩下的就是一个线性 ICA 问题。Taleb 与 Jutten（1999）证明，在关于 $\bA$、源和非线性函数的一定条件下，输出相互独立恰好迫使这一点成立：每个 $g_i\circ f_i$ 都是仿射的，且 $\bW\bA$ 是带尺度的置换矩阵，于是源可以在通常的尺度、置换以及加性常数的意义下恢复。后来的工作放宽了这些条件，这里不再精确陈述。

The PNL model (13.3) escapes Theorem 13.2定理 13.2 because its separating structure, $n$ scalar functions and one matrix, is too rigid to reproduce maps that act jointly on several coordinates, such as the Darmois map or the spiral of Figure 13.2图 13.2. Restricting $\mathbf f$ to conformal maps has a similar effect: Hyvärinen and Pajunen (1999) gave a uniqueness result in two dimensions, and Buchholz, Besserve and Schölkopf (2022) extended such results to higher dimensions, in each case up to a small group of simple transformations. The price is generality, since the processes behind images or neural recordings are neither post-nonlinear nor conformal. The rest of the chapter keeps $\mathbf f$ arbitrary and adds structure to the sources instead.

PNL 模型 (13.3) 之所以能躲开Theorem 13.2定理 13.2，是因为它的分离结构（$n$ 个标量函数加一个矩阵）过于刚性，无法复现同时作用于多个坐标的映射，例如 Darmois 映射或Figure 13.2图 13.2 中的螺旋映射。把 $\mathbf f$ 限制为共形映射也有类似效果：Hyvärinen 与 Pajunen（1999）给出了二维情形的唯一性结果，Buchholz、Besserve 与 Schölkopf（2022）把这类结果推广到更高维，两者都是在一个由简单变换构成的小群的意义下唯一。代价是一般性：图像或神经记录背后的生成过程既不是后非线性的，也不是共形的。本章其余部分对 $\mathbf f$ 不加限制，转而给源增加结构。

13.4 Temporal structure: time-contrastive and permutation-contrastive learning时间结构：时间对比学习与置换对比学习

For linear ICA, temporal structure was an alternative to non-Gaussianity (Chapter 9第 9 章). For nonlinear ICA it becomes a source of identifiability in its own right. The methods of Hyvärinen and Morioka turn estimation into classification, using a simple fact about logistic regression. Suppose a classifier is trained on equal numbers of samples from two densities $p_1$ and $p_0$ and models the posterior probability of the first class as $1/(1 + e^{-r(\bv)})$. The true posterior is $p_1(\bv)/(p_1(\bv) + p_0(\bv))$, so the optimal regression function is the log-density ratio

对线性 ICA 而言，时间结构是非高斯性之外的另一种选择（Chapter 9第 9 章）。对非线性 ICA 而言，它本身就成了可辨识性的来源。Hyvärinen 与 Morioka 的方法把估计问题变成分类问题，所依据的是逻辑回归的一个简单事实。设用来自两个密度 $p_1$ 与 $p_0$ 的等量样本训练一个分类器，它把第一类的后验概率建模为 $1/(1 + e^{-r(\bv)})$。真实后验是 $p_1(\bv)/(p_1(\bv) + p_0(\bv))$，所以最优的回归函数就是对数密度比

With flexible models and enough data, logistic regression therefore learns a log-density ratio; this is the principle of noise-contrastive estimation (Gutmann & Hyvärinen, 2012). In nonlinear ICA the two densities share the same mixing function, and their ratio turns out to be blind to it.

因此，只要模型足够灵活、数据足够多，逻辑回归学到的就是对数密度比；这正是噪声对比估计的原理（Gutmann & Hyvärinen, 2012）。在非线性 ICA 中，两个密度共享同一个混合函数，而它们的比值恰好对这个混合函数“视而不见”。

Time-contrastive learning时间对比学习

Suppose the sources are nonstationary. The time series is divided into segments $\tau = 1, \dots, K$; within each segment the sources are independent, and their log-densities change from segment to segment through an exponential family,

假设源是非平稳的。把时间序列分成 $\tau = 1, \dots, K$ 共 $K$ 段；在每一段内各源相互独立，而它们的对数密度通过一个指数族随段而变：

where $q_{i,0}$ and $q_i$ are fixed functions, $\lambda_i(\tau)$ is a modulation parameter and $Z_i(\tau)$ a normalizing constant. With $q_{i,0} = 0$ and $q_i(s) = -s^2/2$, for instance, $\lambda_i(\tau)$ is the inverse variance of source $i$ in segment $\tau$: the sources are Gaussian with power that fluctuates over time. Time-contrastive learning (TCL; Hyvärinen & Morioka, 2016) trains a multinomial logistic regression to predict the segment label from the observation,

其中 $q_{i,0}$ 与 $q_i$ 是固定的函数，$\lambda_i(\tau)$ 是调制参数，$Z_i(\tau)$ 是归一化常数。例如取 $q_{i,0} = 0$，$q_i(s) = -s^2/2$，则 $\lambda_i(\tau)$ 就是源 $i$ 在第 $\tau$ 段内方差的倒数：源是高斯的，但功率随时间起伏。时间对比学习（time-contrastive learning, TCL；Hyvärinen & Morioka, 2016）训练一个多项逻辑回归，从观测预测它所在的段：

where $\mathbf h\colon \R^n \to \R^n$ is a feature extractor, in practice a neural network, trained jointly with the weights $\bw_\tau$ and biases $b_\tau$.

其中 $\mathbf h\colon \R^n \to \R^n$ 是特征提取器，实践中是一个神经网络，与权重 $\bw_\tau$ 和偏置 $b_\tau$ 一起训练。

This is a slight generalization of Theorem 1 of Hyvärinen and Morioka (2016), with the same proof. The cancellation of the Jacobian is the heart of the matter: in the likelihood of a single segment, $\log|\det\mathbf J_{\mathbf f^{-1}}|$ is an unknown function that can absorb almost anything, but a ratio of two densities generated by the same mixing function does not contain it. The rank condition on $\mathbf L$ is a sufficient variability requirement: at least $n + 1$ segments, with source distributions that change in sufficiently different ways (Exercise 13.4习题 13.4). It remains to undo $\bB$. The $q_i(s_i)$ are independent within each segment, but generally not when the segments are pooled, because the pooled density is a mixture of product densities. The natural last step is therefore a linear method that uses the nonstationarity: within segment $\tau$ the covariance matrix of $\mathbf h(\bx)$ equals $\bB^{-1}$ times a diagonal matrix times $\bB^{-\top}$, so joint diagonalization of the segment-wise covariance matrices recovers the $q_i(s_i)$ (Chapter 9第 9 章), provided no two of them have proportional variance profiles across segments. An ordinary ICA of the pooled features is exact only when the pooled $q_i(s_i)$ are independent and non-Gaussian. Note that $q_i(s_i)$ need not be an invertible function of $s_i$: with $q_i(s) = -s^2/2$ the sign of $s_i$ is lost.

这是 Hyvärinen 与 Morioka（2016）定理 1 的一个小推广，证明完全相同。Jacobian 被消去是问题的关键：在单独一段的似然中，$\log|\det\mathbf J_{\mathbf f^{-1}}|$ 是一个几乎什么都能吸收的未知函数，而由同一混合函数生成的两个密度之比根本不含这一项。关于 $\mathbf L$ 的秩条件是一种充分变化性要求：至少要有 $n + 1$ 段，而且各源分布的变化方式要足够不同（Exercise 13.4习题 13.4）。剩下的是去掉 $\bB$。各 $q_i(s_i)$ 在每一段内相互独立，但把各段合并后一般并不独立，因为合并后的密度是若干乘积密度的混合。因此最后一步宜采用利用非平稳性的线性方法：在第 $\tau$ 段内，$\mathbf h(\bx)$ 的协方差矩阵等于 $\bB^{-1}$ 乘一个对角矩阵再乘 $\bB^{-\top}$，所以只要没有两个 $q_i(s_i)$ 的方差随段变化的轮廓成比例，对各段协方差矩阵做联合对角化就能恢复各 $q_i(s_i)$（Chapter 9第 9 章）。只有当合并后的 $q_i(s_i)$ 相互独立且非高斯时，对合并数据做普通 ICA 才是精确的。注意 $q_i(s_i)$ 未必是 $s_i$ 的可逆函数：当 $q_i(s) = -s^2/2$ 时，$s_i$ 的符号就丢失了。

Permutation-contrastive learning置换对比学习

Permutation-contrastive learning (PCL; Hyvärinen & Morioka, 2017) assumes instead that the sources are stationary with temporal dependence. It discriminates pairs of consecutive observations $(\bx(t), \bx(t-1))$ from pairs $(\bx(t), \bx(t^*))$ with a randomly chosen time $t^*$, which have the same marginals but no temporal dependence. Its regression function is a sum of terms $B_i$, each depending only on one learned feature $h_i$ at the two time points. Hyvärinen and Morioka proved that the learned $h_i$ recover the sources up to permutation and componentwise invertible transformations if $\mathbf f$ is a bijection that is twice differentiable with a twice differentiable inverse, the sources are mutually independent, stationary and ergodic processes, and each source is

置换对比学习（permutation-contrastive learning, PCL；Hyvärinen & Morioka, 2017）则假设源是平稳的、但在时间上相互依赖。它区分相邻时刻的观测对 $(\bx(t), \bx(t-1))$ 与随机选取时刻 $t^*$ 构成的观测对 $(\bx(t), \bx(t^*))$；后者的边缘分布相同，但没有时间依赖。它的回归函数是若干项 $B_i$ 之和，每一项只依赖于一个学到的特征 $h_i$ 在两个时刻的取值。Hyvärinen 与 Morioka 证明：若 $\mathbf f$ 是二阶可微且逆映射也二阶可微的双射，各源是相互独立的平稳遍历过程，并且每个源都满足下面两个条件，则学到的 $h_i$ 能在置换与逐分量可逆变换的意义下恢复源：

• Uniformly dependent: writing $p(a, b)$ for the joint density of $s_i(t)$ and $s_i(t-1)$, the cross-derivative $\partial^2 \log p(a, b)/\partial a\,\partial b$ exists, is continuous and is nonzero everywhere.
• Not quasi-Gaussian: this cross-derivative is not of the form $c\,\alpha(a)\,\alpha(b)$ for a constant $c$ and a function $\alpha$.

• 一致依赖：记 $p(a, b)$ 为 $s_i(t)$ 与 $s_i(t-1)$ 的联合密度，交叉导数 $\partial^2 \log p(a, b)/\partial a\,\partial b$ 存在、连续且处处非零。
• 非拟高斯：这个交叉导数不能写成 $c\,\alpha(a)\,\alpha(b)$ 的形式，其中 $c$ 为常数，$\alpha$ 为某个函数。

A stationary Gaussian autoregressive process has a constant cross-derivative and is therefore quasi-Gaussian. For quasi-Gaussian sources PCL recovers only an invertible linear mixture of transformed sources, which a further linear step must unmix (Hyvärinen & Morioka, 2017).

平稳的高斯自回归过程的交叉导数是常数，因而是拟高斯的。对拟高斯的源，PCL 只能恢复变换后各源的一个可逆线性混合，还需要再做一步线性解混（Hyvärinen & Morioka, 2017）。

13.5 Auxiliary variables and generalized contrastive learning辅助变量与广义对比学习

TCL and PCL share a structure. In both, an observed variable, the segment label $\tau$ in TCL and the past observation $\bx(t-1)$ in PCL, changes the distribution of the sources without changing the mixing function. Hyvärinen, Sasaki and Turner (2019) made this the definition of the model: the sources are conditionally independent given an observed auxiliary variable $\bu$,

TCL 与 PCL 有共同的结构。两者都有一个观测到的变量（TCL 中是段标号 $\tau$，PCL 中是前一时刻的观测 $\bx(t-1)$），它改变源的分布，却不改变混合函数。Hyvärinen、Sasaki 与 Turner（2019）把这一点直接作为模型的定义：给定一个观测到的辅助变量 $\bu$，各源条件独立，

where $\mathbf f$ does not depend on $\bu$. The auxiliary variable can be a time index or segment label, a past observation, a class label, a stimulus, or a simultaneously recorded second modality. The sources need not be marginally independent. Estimation is by generalized contrastive learning (GCL): a logistic regression discriminates true pairs $(\bx, \bu)$ from pairs $(\bx, \bu^*)$ in which $\bu^*$ is drawn independently of $\bx$, for instance by shuffling the observed values of $\bu$, using a regression function of the form

其中 $\mathbf f$ 与 $\bu$ 无关。辅助变量可以是时间标号或段标号、过去的观测、类别标签、刺激，或者同时记录的另一种模态。各源不必边缘独立。估计方法是广义对比学习（generalized contrastive learning, GCL）：用逻辑回归区分真实的配对 $(\bx, \bu)$ 与配对 $(\bx, \bu^*)$，其中 $\bu^*$ 的抽取与 $\bx$ 独立（例如把观测到的 $\bu$ 值随机打乱），回归函数取如下形式：

By the principle of Section 13.4第 13.4 节, the optimal regression function is

由Section 13.4第 13.4 节中的原理，最优的回归函数是

To state the identifiability result, write $q_i'$ and $q_i''$ for the first and second derivatives of $q_i(s_i, \bu)$ with respect to $s_i$, and collect their values at $(s_i, \bu)$ in a vector of length $2n$:

为了陈述可辨识性结果，用 $q_i'$、$q_i''$ 表示 $q_i(s_i, \bu)$ 关于 $s_i$ 的一阶与二阶导数，并把它们在 $(s_i, \bu)$ 处的值排成一个长度为 $2n$ 的向量：