Appendix A附录 A

Mathematical Background数学基础

Facts used throughout: orthogonal and permutation matrices, matrix square roots, the polar decomposition, Kronecker products and tensors, matrix calculus, optimization on the sphere and the orthogonal group, and the asymptotic statistics of Part IV (delta method, sandwich covariance, Fisher information).

全书用到的数学工具：正交矩阵与置换矩阵、矩阵平方根、极分解、Kronecker 积与张量、矩阵微积分、球面与正交群上的优化，以及第四部分用到的渐近统计（delta 方法、三明治协方差、Fisher 信息）。

A.1 Conventions约定

This appendix collects the matrix and statistical facts used throughout the book. Short and instructive proofs are given in full; for the rest we point to standard references. All matrices are real. For matrices of the same size, $\langle\bA, \bB\rangle = \tr(\bA^\top\bB) = \sum_{ij}a_{ij}b_{ij}$ is the Frobenius inner product and $\|\bA\|_F = \langle\bA, \bA\rangle^{1/2}$ the Frobenius norm; $\|\bx\|$ is the Euclidean norm of a vector. The identity $\tr(\bA\bB) = \tr(\bB\bA)$ is used constantly. For symmetric matrices, $\bA \succeq \bB$ (respectively $\bA \succ \bB$ ) means that $\bA - \bB$ is positive semidefinite (positive definite).

本附录汇集全书用到的矩阵与统计知识。简短而有启发的证明完整给出，其余给出标准参考文献。所有矩阵都是实矩阵。对同样大小的矩阵， $\langle\bA, \bB\rangle = \tr(\bA^\top\bB) = \sum_{ij}a_{ij}b_{ij}$ 是 Frobenius 内积， $\|\bA\|_F = \langle\bA, \bA\rangle^{1/2}$ 是 Frobenius 范数； $\|\bx\|$ 是向量的欧氏范数。恒等式 $\tr(\bA\bB) = \tr(\bB\bA)$ 会反复用到。对对称矩阵， $\bA \succeq \bB$ （或 $\bA \succ \bB$ ）表示 $\bA - \bB$ 半正定（或正定）。

Every square matrix splits as $\bM = \operatorname{sym}(\bM) + \operatorname{skew}(\bM)$ , with symmetric part $\operatorname{sym}(\bM) = (\bM + \bM^\top)/2$ and skew-symmetric part $\operatorname{skew}(\bM) = (\bM - \bM^\top)/2$ . The two parts are orthogonal: if $\bS$ is symmetric and $\bK$ is skew-symmetric, then $\tr(\bS\bK) = \tr\bigl((\bS\bK)^\top\bigr) = -\tr(\bK\bS) = -\tr(\bS\bK)$ , so $\langle\bS, \bK\rangle = 0$ . This simple fact underlies the gradients on the orthogonal group in Section A.7第 A.7 节.

每个方阵都可以分解为 $\bM = \operatorname{sym}(\bM) + \operatorname{skew}(\bM)$ ，其中对称部分为 $\operatorname{sym}(\bM) = (\bM + \bM^\top)/2$ ，反对称部分为 $\operatorname{skew}(\bM) = (\bM - \bM^\top)/2$ 。两部分相互正交：若 $\bS$ 对称、 $\bK$ 反对称，则 $\tr(\bS\bK) = \tr\bigl((\bS\bK)^\top\bigr) = -\tr(\bK\bS) = -\tr(\bS\bK)$ ，所以 $\langle\bS, \bK\rangle = 0$ 。Section A.7第 A.7 节中正交群上的梯度就建立在这个简单事实之上。

A.2 Orthogonal and permutation matrices正交矩阵与置换矩阵

A square matrix $\bQ$ is orthogonal if $\bQ^\top\bQ = \bI$ , equivalently $\bQ\bQ^\top = \bI$ or $\bQ^{-1} = \bQ^\top$ . Orthogonal matrices preserve inner products, $(\bQ\bu)^\top(\bQ\bv) = \bu^\top\bv$ , and hence lengths, angles and Frobenius norms: $\|\bQ\bM\|_F = \|\bM\bQ\|_F = \|\bM\|_F$ . They form a group, the orthogonal group $O(n)$ . Since $(\det\bQ)^2 = \det(\bQ^\top\bQ) = 1$ , every orthogonal matrix has determinant $\pm 1$ . Those with determinant $+1$ form the group $SO(n)$ of rotations; the others include reflections such as $\diag(1, \dots, 1, -1)$ .

若方阵 $\bQ$ 满足 $\bQ^\top\bQ = \bI$ （等价地， $\bQ\bQ^\top = \bI$ ，或 $\bQ^{-1} = \bQ^\top$ ），则称它是正交矩阵。正交矩阵保持内积 $(\bQ\bu)^\top(\bQ\bv) = \bu^\top\bv$ ，因而保持长度、夹角与 Frobenius 范数： $\|\bQ\bM\|_F = \|\bM\bQ\|_F = \|\bM\|_F$ 。它们构成一个群，即正交群 $O(n)$ 。由于 $(\det\bQ)^2 = \det(\bQ^\top\bQ) = 1$ ，正交矩阵的行列式为 $\pm 1$ 。行列式为 $+1$ 的正交矩阵构成旋转群 $SO(n)$ ；其余的包括 $\diag(1, \dots, 1, -1)$ 这样的反射。

Proposition A.1 (dimension of the orthogonal group)命题 A.1（正交群的维数）

$O(n)$ is a smooth manifold of dimension $n(n-1)/2$ . Its tangent space at $\bQ$ is $\{\boldsymbol{\Omega}\bQ : \boldsymbol{\Omega}^\top = -\boldsymbol{\Omega}\}$ , and for every skew-symmetric $\boldsymbol{\Omega}$ the curve $t \mapsto \exp(t\boldsymbol{\Omega})\bQ$ lies in $O(n)$ and passes through $\bQ$ with velocity $\boldsymbol{\Omega}\bQ$ .

$O(n)$ 是维数为 $n(n-1)/2$ 的光滑流形。它在 $\bQ$ 处的切空间为 $\{\boldsymbol{\Omega}\bQ : \boldsymbol{\Omega}^\top = -\boldsymbol{\Omega}\}$ ；对每个反对称矩阵 $\boldsymbol{\Omega}$ ，曲线 $t \mapsto \exp(t\boldsymbol{\Omega})\bQ$ 落在 $O(n)$ 中，并以速度 $\boldsymbol{\Omega}\bQ$ 经过 $\bQ$ 。

Proof.证明

$O(n)$ is the level set $\{\bQ : \bQ\bQ^\top = \bI\}$ of the map $\bQ \mapsto \bQ\bQ^\top$ , whose derivative $\dot\bQ \mapsto \dot\bQ\bQ^\top + \bQ\dot\bQ^\top$ maps onto the symmetric matrices (given a symmetric $\bS$ , take $\dot\bQ = \tfrac12\bS\bQ$ ). By the regular value theorem, $O(n)$ is a smooth manifold whose tangent space at $\bQ$ is the kernel of this derivative. Writing $\dot\bQ = \boldsymbol{\Omega}\bQ$ , the kernel condition becomes $\boldsymbol{\Omega} + \boldsymbol{\Omega}^\top = \bzero$ . A skew-symmetric matrix is determined by its $n(n-1)/2$ entries above the diagonal, which gives the dimension. Finally, $\exp(t\boldsymbol{\Omega})^\top\exp(t\boldsymbol{\Omega}) = \exp(-t\boldsymbol{\Omega})\exp(t\boldsymbol{\Omega}) = \bI$ , and the derivative at $t = 0$ is $\boldsymbol{\Omega}\bQ$ .

$O(n)$ 是映射 $\bQ \mapsto \bQ\bQ^\top$ 的水平集 $\{\bQ : \bQ\bQ^\top = \bI\}$ ，该映射的导数 $\dot\bQ \mapsto \dot\bQ\bQ^\top + \bQ\dot\bQ^\top$ 映满全体对称矩阵（对给定的对称矩阵 $\bS$ ，取 $\dot\bQ = \tfrac12\bS\bQ$ 即可）。由正则值定理， $O(n)$ 是光滑流形，它在 $\bQ$ 处的切空间就是这个导数的核。记 $\dot\bQ = \boldsymbol{\Omega}\bQ$ ，核的条件就变成 $\boldsymbol{\Omega} + \boldsymbol{\Omega}^\top = \bzero$ 。反对称矩阵由对角线以上的 $n(n-1)/2$ 个元素确定，由此得到维数。最后， $\exp(t\boldsymbol{\Omega})^\top\exp(t\boldsymbol{\Omega}) = \exp(-t\boldsymbol{\Omega})\exp(t\boldsymbol{\Omega}) = \bI$ ，且它在 $t = 0$ 处的导数为 $\boldsymbol{\Omega}\bQ$ 。

For $n = 2$ , $\exp\bigl(\theta\bigl(\begin{smallmatrix} 0 & -1 \\ 1 & 0 \end{smallmatrix}\bigr)\bigr) = \bigl(\begin{smallmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{smallmatrix}\bigr)$ , the rotation by $\theta$ : one parameter, as $n(n-1)/2 = 1$ predicts. In general every rotation is the exponential of a skew-symmetric matrix. An orthogonal change of basis $\bB$ brings any orthogonal $\bQ$ to a block-diagonal form with $2 \times 2$ blocks $\exp(\theta_k\bJ)$ , $\bJ = \bigl(\begin{smallmatrix} 0 & -1 \\ 1 & 0 \end{smallmatrix}\bigr)$ , and diagonal entries $\pm 1$ (Horn & Johnson, 2013). When $\det\bQ = 1$ the entries $-1$ come in pairs, each pair forming a block $\exp(\pi\bJ) = -\bI_2$ , so $\bQ = \exp(\boldsymbol{\Omega})$ with the skew-symmetric matrix $\boldsymbol{\Omega} = \bB\,\diag(\theta_1\bJ, \dots, \theta_k\bJ, 0, \dots, 0)\,\bB^\top$ . Conversely, $\det\exp(\boldsymbol{\Omega}) = e^{\tr\boldsymbol{\Omega}} = 1$ .

当 $n = 2$ 时， $\exp\bigl(\theta\bigl(\begin{smallmatrix} 0 & -1 \\ 1 & 0 \end{smallmatrix}\bigr)\bigr) = \bigl(\begin{smallmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{smallmatrix}\bigr)$ ，即旋转 $\theta$ 角：只有一个参数，与 $n(n-1)/2 = 1$ 一致。一般地，每个旋转都是某个反对称矩阵的指数。通过正交基变换 $\bB$ ，任何正交矩阵 $\bQ$ 都可以化为分块对角形式，其中的 $2 \times 2$ 块为 $\exp(\theta_k\bJ)$ ， $\bJ = \bigl(\begin{smallmatrix} 0 & -1 \\ 1 & 0 \end{smallmatrix}\bigr)$ ，其余对角元为 $\pm 1$ （Horn & Johnson, 2013）。当 $\det\bQ = 1$ 时，对角元 $-1$ 成对出现，每一对构成一个块 $\exp(\pi\bJ) = -\bI_2$ ，于是 $\bQ = \exp(\boldsymbol{\Omega})$ ，其中 $\boldsymbol{\Omega} = \bB\,\diag(\theta_1\bJ, \dots, \theta_k\bJ, 0, \dots, 0)\,\bB^\top$ 是反对称矩阵。反之， $\det\exp(\boldsymbol{\Omega}) = e^{\tr\boldsymbol{\Omega}} = 1$ 。

Givens rotations. The Givens rotation $\bG(i, j, \theta)$ equals the identity except for the entries $g_{ii} = g_{jj} = \cos\theta$ , $g_{ij} = -\sin\theta$ and $g_{ji} = \sin\theta$ ; it rotates the $(i, j)$ coordinate plane by $\theta$ and leaves the other coordinates unchanged. There are $n(n-1)/2$ such planes, matching the dimension of $O(n)$ . Every orthogonal matrix is a product of at most $n(n-1)/2$ Givens rotations and a diagonal matrix of signs: applying Givens rotations to zero the entries below the diagonal of $\bQ$ column by column, as in the Givens QR factorization (Golub & Van Loan, 2013), leaves an upper triangular orthogonal matrix, which must be diagonal with entries $\pm 1$ . Jacobi-type algorithms such as JADE (Chapter 9第 9 章) search over $O(n)$ by sweeping through the planes and optimizing one angle at a time.

Givens 旋转。Givens 旋转 $\bG(i, j, \theta)$ 除 $g_{ii} = g_{jj} = \cos\theta$ 、 $g_{ij} = -\sin\theta$ 、 $g_{ji} = \sin\theta$ 之外与单位阵相同；它把 $(i, j)$ 坐标平面旋转 $\theta$ 角，而保持其余坐标不变。这样的平面共有 $n(n-1)/2$ 个，恰好等于 $O(n)$ 的维数。每个正交矩阵都是至多 $n(n-1)/2$ 个 Givens 旋转与一个符号对角矩阵的乘积：像用 Givens 旋转做 QR 分解那样（Golub & Van Loan, 2013），逐列消去 $\bQ$ 对角线以下的元素，剩下一个上三角的正交矩阵，它必然是对角元为 $\pm 1$ 的对角矩阵。JADE（Chapter 9第 9 章）等 Jacobi 型算法在 $O(n)$ 中搜索的方式，就是依次扫过各个平面，每次只优化一个角度。

Permutation matrices. For a permutation $\pi$ of $\{1, \dots, n\}$ , the permutation matrix $\bPi$ has entries $\pi_{ij} = 1$ if $i = \pi(j)$ and $0$ otherwise, so $(\bPi\bs)_{\pi(j)} = s_j$ : it moves component $j$ to position $\pi(j)$ . Permutation matrices are orthogonal, and so are the signed permutation matrices $\bPi\bD$ with $\bD = \diag(\pm 1)$ . These are the only orthogonal scaled permutations.

置换矩阵。对 $\{1, \dots, n\}$ 的置换 $\pi$ ，置换矩阵 $\bPi$ 的元素为：当 $i = \pi(j)$ 时 $\pi_{ij} = 1$ ，否则为 $0$ 。于是 $(\bPi\bs)_{\pi(j)} = s_j$ ：它把第 $j$ 个分量移到第 $\pi(j)$ 个位置。置换矩阵是正交的，带符号的置换矩阵 $\bPi\bD$ （ $\bD = \diag(\pm 1)$ ）也是正交的，而且它们是仅有的正交的带尺度置换矩阵。

Lemma A.2 (orthogonal scaled permutations)引理 A.2（正交的带尺度置换矩阵）

Let $\bPi$ be a permutation matrix and $\bD$ a diagonal matrix. Then $\bPi\bD$ is orthogonal if and only if $\bD = \diag(\pm 1)$ . The signed permutation matrices form a subgroup of $O(n)$ with $2^n n!$ elements.

设 $\bPi$ 为置换矩阵， $\bD$ 为对角矩阵。则 $\bPi\bD$ 正交当且仅当 $\bD = \diag(\pm 1)$ 。带符号的置换矩阵构成 $O(n)$ 的一个子群，共有 $2^n n!$ 个元素。

Proof.证明

$(\bPi\bD)^\top(\bPi\bD) = \bD\bPi^\top\bPi\bD = \bD^2$ , which equals $\bI$ exactly when every $d_i^2 = 1$ . For two signed permutations, $\bD\bPi' = \bPi'(\bPi'^\top\bD\bPi')$ and $\bPi'^\top\bD\bPi'$ is again a diagonal sign matrix, so the product $(\bPi\bD)(\bPi'\bD')$ is a signed permutation; so is the inverse $(\bPi\bD)^\top = \bPi^\top(\bPi\bD\bPi^\top)$ . Distinct pairs $(\bPi, \bD)$ give distinct matrices, because the position of the nonzero entry in each column determines $\bPi$ and its sign determines $\bD$ ; hence there are $n!\cdot 2^n$ of them.

$(\bPi\bD)^\top(\bPi\bD) = \bD\bPi^\top\bPi\bD = \bD^2$ ，它等于 $\bI$ 当且仅当每个 $d_i^2 = 1$ 。对两个带符号置换矩阵， $\bD\bPi' = \bPi'(\bPi'^\top\bD\bPi')$ ，而 $\bPi'^\top\bD\bPi'$ 仍是对角符号矩阵，所以乘积 $(\bPi\bD)(\bPi'\bD')$ 仍是带符号置换矩阵；逆矩阵 $(\bPi\bD)^\top = \bPi^\top(\bPi\bD\bPi^\top)$ 也是。不同的 $(\bPi, \bD)$ 给出不同的矩阵，因为每列非零元的位置确定 $\bPi$ ，其符号确定 $\bD$ ；因此共有 $n!\cdot 2^n$ 个。

In ICA, separation means that the global matrix is a scaled permutation $\bPi\bD$ (Proposition 1.3命题 1.3). After whitening the global matrix is orthogonal (Proposition 3.4命题 3.4), and the lemma shows that the scaling is then reduced to signs: the solutions form the finite group of signed permutations.

在 ICA 中，分离意味着全局矩阵是带尺度的置换矩阵 $\bPi\bD$ （Proposition 1.3命题 1.3）。白化之后全局矩阵是正交的（Proposition 3.4命题 3.4），由引理可知，此时尺度只剩下符号：所有解构成带符号置换矩阵这个有限群。

A.3 Symmetric matrices and matrix functions对称矩阵与矩阵函数

Spectral theorem. Every symmetric $\bC \in \R^{n \times n}$ can be written $\bC = \mathbf{E}\bLambda\mathbf{E}^\top = \sum_i \lambda_i\be_i\be_i^\top$ , with $\mathbf{E}$ orthogonal with columns $\be_i$ and $\bLambda = \diag(\lambda_1, \dots, \lambda_n)$ real (Horn & Johnson, 2013). The columns are orthonormal eigenvectors, $\bC\be_i = \lambda_i\be_i$ . If the eigenvalues are distinct, each eigenvector is unique up to sign; for a repeated eigenvalue, any orthonormal basis of its eigenspace can be used.

谱定理。每个对称矩阵 $\bC \in \R^{n \times n}$ 都可以写成 $\bC = \mathbf{E}\bLambda\mathbf{E}^\top = \sum_i \lambda_i\be_i\be_i^\top$ ，其中 $\mathbf{E}$ 是以 $\be_i$ 为列的正交矩阵， $\bLambda = \diag(\lambda_1, \dots, \lambda_n)$ 为实对角矩阵（Horn & Johnson, 2013）。各列是标准正交的特征向量， $\bC\be_i = \lambda_i\be_i$ 。若特征值互不相同，则每个特征向量在相差一个符号的意义下唯一；对重特征值，可以取其特征子空间的任意一组标准正交基。

Definiteness. $\bC$ is positive semidefinite, $\bC \succeq \bzero$ , if $\bv^\top\bC\bv \ge 0$ for all $\bv$ , equivalently if all $\lambda_i \ge 0$ ; it is positive definite, $\bC \succ \bzero$ , if $\bv^\top\bC\bv > 0$ for all $\bv \ne \bzero$ , equivalently if all $\lambda_i > 0$ . Covariance matrices are positive semidefinite, since $\bv^\top\bC_{\bx}\bv = \Var(\bv^\top\bx)$ . A positive definite matrix has a unique Cholesky factorization $\bC = \mathbf{L}\mathbf{L}^\top$ with $\mathbf{L}$ lower triangular with a positive diagonal (Golub & Van Loan, 2013).

正定性。若对所有 $\bv$ 都有 $\bv^\top\bC\bv \ge 0$ ，等价地，所有 $\lambda_i \ge 0$ ，则称 $\bC$ 半正定，记作 $\bC \succeq \bzero$ ；若对所有 $\bv \ne \bzero$ 都有 $\bv^\top\bC\bv > 0$ ，等价地，所有 $\lambda_i > 0$ ，则称 $\bC$ 正定，记作 $\bC \succ \bzero$ 。协方差矩阵是半正定的，因为 $\bv^\top\bC_{\bx}\bv = \Var(\bv^\top\bx)$ 。正定矩阵有唯一的 Cholesky 分解 $\bC = \mathbf{L}\mathbf{L}^\top$ ，其中 $\mathbf{L}$ 是对角元为正的下三角矩阵（Golub & Van Loan, 2013）。

Rayleigh quotient. For a unit vector $\bw$ , $\bw^\top\bC\bw = \sum_i \lambda_i(\be_i^\top\bw)^2$ is an average of the eigenvalues with weights $(\be_i^\top\bw)^2$ that sum to one. Hence $\max_{\|\bw\| = 1}\bw^\top\bC\bw = \lambda_{\max}$ and $\min_{\|\bw\| = 1}\bw^\top\bC\bw = \lambda_{\min}$ , attained at the corresponding eigenvectors. This variational characterization is the basis of PCA, and Section A.7第 A.7 节 revisits it as an optimization problem on the sphere.

Rayleigh 商。对单位向量 $\bw$ ， $\bw^\top\bC\bw = \sum_i \lambda_i(\be_i^\top\bw)^2$ 是各特征值以 $(\be_i^\top\bw)^2$ 为权重的加权平均，权重之和为一。因此 $\max_{\|\bw\| = 1}\bw^\top\bC\bw = \lambda_{\max}$ ， $\min_{\|\bw\| = 1}\bw^\top\bC\bw = \lambda_{\min}$ ，分别在相应的特征向量处取到。这一变分刻画是 PCA 的基础，Section A.7第 A.7 节还会把它作为球面上的优化问题重新讨论。

Matrix functions. For a function $f$ defined on the eigenvalues, $f(\bC) = \mathbf{E}f(\bLambda)\mathbf{E}^\top = \sum_i f(\lambda_i)\be_i\be_i^\top$ . The result does not depend on the choice of eigenvectors, because $f(\bC) = p(\bC)$ for any polynomial $p$ with $p(\lambda_i) = f(\lambda_i)$ for all $i$ . Examples are powers, $\exp(\bC)$ , $\log\bC$ for $\bC \succ \bzero$ , and square roots. Matrix functions commute with orthogonal similarity: $f(\bQ\bC\bQ^\top) = \bQ f(\bC)\bQ^\top$ .

矩阵函数。对定义在各特征值上的函数 $f$ ，令 $f(\bC) = \mathbf{E}f(\bLambda)\mathbf{E}^\top = \sum_i f(\lambda_i)\be_i\be_i^\top$ 。这个定义与特征向量的选取无关，因为对任何满足 $p(\lambda_i) = f(\lambda_i)$ （对所有 $i$ ）的多项式 $p$ ，都有 $f(\bC) = p(\bC)$ 。例子有幂、 $\exp(\bC)$ 、 $\bC \succ \bzero$ 时的 $\log\bC$ ，以及平方根。矩阵函数与正交相似变换可交换： $f(\bQ\bC\bQ^\top) = \bQ f(\bC)\bQ^\top$ 。

Proposition A.3 (matrix square roots)命题 A.3（矩阵平方根）

Every positive semidefinite $\bC$ has a unique positive semidefinite square root, that is, a matrix $\bB \succeq \bzero$ with $\bB^2 = \bC$ , namely $\bC^{1/2} = \mathbf{E}\bLambda^{1/2}\mathbf{E}^\top$ . If $\bC \succ \bzero$ , then $\bC^{-1/2} = (\bC^{1/2})^{-1} = \mathbf{E}\bLambda^{-1/2}\mathbf{E}^\top$ and $\bC^{-1/2}\bC\,\bC^{-1/2} = \bI$ .

每个半正定矩阵 $\bC$ 都有唯一的半正定平方根，即满足 $\bB^2 = \bC$ 的矩阵 $\bB \succeq \bzero$ ，它就是 $\bC^{1/2} = \mathbf{E}\bLambda^{1/2}\mathbf{E}^\top$ 。若 $\bC \succ \bzero$ ，则 $\bC^{-1/2} = (\bC^{1/2})^{-1} = \mathbf{E}\bLambda^{-1/2}\mathbf{E}^\top$ ，且 $\bC^{-1/2}\bC\,\bC^{-1/2} = \bI$ 。

Proof.证明

$\mathbf{E}\bLambda^{1/2}\mathbf{E}^\top$ is positive semidefinite and its square is $\mathbf{E}\bLambda\mathbf{E}^\top = \bC$ . For uniqueness, let $\bB \succeq \bzero$ with $\bB^2 = \bC$ . Then $\bB$ commutes with $\bC = \bB^2$ , so it maps each eigenspace of $\bC$ into itself: if $\bC\bv = \lambda\bv$ , then $\bC(\bB\bv) = \bB\bC\bv = \lambda\bB\bv$ . Restricted to the eigenspace of $\lambda$ , $\bB$ is symmetric and positive semidefinite with square $\lambda\bI$ , so its eigenvalues are nonnegative square roots of $\lambda$ , and it equals $\sqrt\lambda\,\bI$ there. Hence $\bB = \sum_i \sqrt{\lambda_i}\,\be_i\be_i^\top$ . The statements about $\bC^{-1/2}$ follow from the eigendecomposition.

$\mathbf{E}\bLambda^{1/2}\mathbf{E}^\top$ 是半正定的，且其平方为 $\mathbf{E}\bLambda\mathbf{E}^\top = \bC$ 。再证唯一性：设 $\bB \succeq \bzero$ 且 $\bB^2 = \bC$ 。则 $\bB$ 与 $\bC = \bB^2$ 可交换，所以 $\bB$ 把 $\bC$ 的每个特征子空间映到自身：若 $\bC\bv = \lambda\bv$ ，则 $\bC(\bB\bv) = \bB\bC\bv = \lambda\bB\bv$ 。把 $\bB$ 限制在特征值 $\lambda$ 的特征子空间上，它是对称半正定的，且平方为 $\lambda\bI$ ，所以它的特征值都是 $\lambda$ 的非负平方根，从而它在该子空间上等于 $\sqrt\lambda\,\bI$ 。因此 $\bB = \sum_i \sqrt{\lambda_i}\,\be_i\be_i^\top$ 。关于 $\bC^{-1/2}$ 的结论由特征分解直接得到。

Non-symmetric square roots, that is, matrices $\bB$ with $\bB\bB^\top = \bC$ such as the Cholesky factor, are not unique. If $\bC \succ \bzero$ and $\bB\bB^\top = \bB'\bB'^\top = \bC$ , then $\bQ = \bB^{-1}\bB'$ satisfies $\bQ\bQ^\top = \bB^{-1}\bC\bB^{-\top} = \bI$ , so $\bB' = \bB\bQ$ with $\bQ$ orthogonal, and every such $\bB\bQ$ works. Inverting gives the whitening matrices of Proposition 3.2命题 3.2.

非对称的平方根，即满足 $\bB\bB^\top = \bC$ 的矩阵 $\bB$ （例如 Cholesky 因子），并不唯一。若 $\bC \succ \bzero$ 且 $\bB\bB^\top = \bB'\bB'^\top = \bC$ ，则 $\bQ = \bB^{-1}\bB'$ 满足 $\bQ\bQ^\top = \bB^{-1}\bC\bB^{-\top} = \bI$ ，所以 $\bB' = \bB\bQ$ ， $\bQ$ 正交；反之每个这样的 $\bB\bQ$ 都满足要求。取逆即得Proposition 3.2命题 3.2中的白化矩阵。

A.4 The SVD and the polar decomposition奇异值分解与极分解

Every $\bW \in \R^{n \times m}$ has a singular value decomposition (SVD) $\bW = \mathbf{L}\bSigma\bR^\top$ , with $\mathbf{L} \in O(n)$ , $\bR \in O(m)$ , and $\bSigma \in \R^{n \times m}$ zero except for the diagonal entries $\sigma_1 \ge \sigma_2 \ge \dots \ge 0$ , the singular values (Golub & Van Loan, 2013). We call the factors $\mathbf{L}$ and $\bR$ , for left and right, to keep $\bU$ and $\bV$ for the rotation and the whitening matrix of the main text. The $\sigma_i^2$ are the eigenvalues of $\bW\bW^\top$ and of $\bW^\top\bW$ , whose eigenvectors are the columns of $\mathbf{L}$ and of $\bR$ . A square $\bW$ is invertible exactly when all its singular values are positive.

每个 $\bW \in \R^{n \times m}$ 都有奇异值分解（SVD） $\bW = \mathbf{L}\bSigma\bR^\top$ ，其中 $\mathbf{L} \in O(n)$ ， $\bR \in O(m)$ ， $\bSigma \in \R^{n \times m}$ 除对角元 $\sigma_1 \ge \sigma_2 \ge \dots \ge 0$ （即奇异值）外全为零（Golub & Van Loan, 2013）。我们把两个因子记作 $\mathbf{L}$ 与 $\bR$ （取 left 与 right 之意），以便把 $\bU$ 与 $\bV$ 留给正文中的旋转与白化矩阵。 $\sigma_i^2$ 是 $\bW\bW^\top$ 与 $\bW^\top\bW$ 的特征值，相应的特征向量分别是 $\mathbf{L}$ 与 $\bR$ 的各列。方阵 $\bW$ 可逆，当且仅当它的奇异值全为正。

Polar decomposition. An invertible square $\bW$ factors as

极分解。可逆方阵 $\bW$ 可以分解为

\bW = (\bW\bW^\top)^{1/2}\,\bQ_{\bW}, \qquad \bQ_{\bW} = (\bW\bW^\top)^{-1/2}\,\bW = \mathbf{L}\bR^\top,\tag{A.1}

where $(\bW\bW^\top)^{1/2} = \mathbf{L}\bSigma\mathbf{L}^\top$ is positive definite and $\bQ_{\bW}$ is orthogonal; indeed $(\bW\bW^\top)^{-1/2}\bW = \mathbf{L}\bSigma^{-1}\mathbf{L}^\top\mathbf{L}\bSigma\bR^\top = \mathbf{L}\bR^\top$ . This is the matrix analogue of writing a nonzero complex number as a modulus times a phase. The same orthogonal factor appears on the other side, $\bW = \bQ_{\bW}(\bW^\top\bW)^{1/2}$ . A singular $\bW$ also has polar decompositions, but its orthogonal factor is not unique.

其中 $(\bW\bW^\top)^{1/2} = \mathbf{L}\bSigma\mathbf{L}^\top$ 正定， $\bQ_{\bW}$ 正交；事实上 $(\bW\bW^\top)^{-1/2}\bW = \mathbf{L}\bSigma^{-1}\mathbf{L}^\top\mathbf{L}\bSigma\bR^\top = \mathbf{L}\bR^\top$ 。这相当于把非零复数写成模乘以相位的矩阵版本。同一个正交因子也出现在另一侧： $\bW = \bQ_{\bW}(\bW^\top\bW)^{1/2}$ 。奇异的 $\bW$ 也有极分解，但其正交因子不唯一。

Proposition A.4 (the closest orthogonal matrix)命题 A.4（最近的正交矩阵）

Let $\bW \in \R^{n \times n}$ be invertible. Then $(\bW\bW^\top)^{-1/2}\bW = \mathbf{L}\bR^\top$ is the unique orthogonal matrix closest to $\bW$ in Frobenius norm: $\|\bW - \bQ\|_F > \|\bW - (\bW\bW^\top)^{-1/2}\bW\|_F$ for every other orthogonal $\bQ$ .

设 $\bW \in \R^{n \times n}$ 可逆。则 $(\bW\bW^\top)^{-1/2}\bW = \mathbf{L}\bR^\top$ 是在 Frobenius 范数下离 $\bW$ 最近的唯一正交矩阵：对其他任何正交矩阵 $\bQ$ ， $\|\bW - \bQ\|_F > \|\bW - (\bW\bW^\top)^{-1/2}\bW\|_F$ 。

Proof.证明

For orthogonal $\bQ$ , $\|\bW - \bQ\|_F^2 = \|\bW\|_F^2 + n - 2\tr(\bQ^\top\bW)$ , so we must maximize $\tr(\bQ^\top\bW) = \tr(\bQ^\top\mathbf{L}\bSigma\bR^\top) = \tr(\bM\bSigma)$ with $\bM = \bR^\top\bQ^\top\mathbf{L}$ , which is orthogonal. Then $\tr(\bM\bSigma) = \sum_i m_{ii}\sigma_i \le \sum_i \sigma_i$ , because each column of $\bM$ is a unit vector and so $|m_{ii}| \le 1$ . Since every $\sigma_i > 0$ , equality holds if and only if $m_{ii} = 1$ for all $i$ ; a unit column with $i$ -th entry 1 is the $i$ -th column of the identity, so this means $\bM = \bI$ , that is, $\bQ = \mathbf{L}\bR^\top$ . By (A.1), $\mathbf{L}\bR^\top = (\bW\bW^\top)^{-1/2}\bW$ .

对正交矩阵 $\bQ$ ， $\|\bW - \bQ\|_F^2 = \|\bW\|_F^2 + n - 2\tr(\bQ^\top\bW)$ ，所以问题化为最大化 $\tr(\bQ^\top\bW) = \tr(\bQ^\top\mathbf{L}\bSigma\bR^\top) = \tr(\bM\bSigma)$ ，其中 $\bM = \bR^\top\bQ^\top\mathbf{L}$ 是正交矩阵。于是 $\tr(\bM\bSigma) = \sum_i m_{ii}\sigma_i \le \sum_i \sigma_i$ ，因为 $\bM$ 的每一列都是单位向量，所以 $|m_{ii}| \le 1$ 。由于所有 $\sigma_i > 0$ ，等号成立当且仅当对所有 $i$ 都有 $m_{ii} = 1$ ；第 $i$ 个分量为 1 的单位列向量就是单位阵的第 $i$ 列，所以这意味着 $\bM = \bI$ ，即 $\bQ = \mathbf{L}\bR^\top$ 。由 (A.1)， $\mathbf{L}\bR^\top = (\bW\bW^\top)^{-1/2}\bW$ 。

Uses in ICA. Symmetric FastICA orthogonalizes its current estimate by $\bW \leftarrow (\bW\bW^\top)^{-1/2}\bW$ ((8.5)), which by the proposition is the orthogonal matrix closest to $\bW$ . It treats all rows alike: since matrix functions commute with orthogonal similarity, replacing $\bW$ by $\bPi\bW$ replaces the result by $\bPi(\bW\bW^\top)^{-1/2}\bW$ , whereas Gram–Schmidt orthogonalization depends on the order of the rows. ZCA whitening produces the whitened mixing matrix $(\bA\bA^\top)^{-1/2}\bA$ , the polar factor of $\bA$ (Section 3.4第 3.4 节). Numerically, $\bQ_{\bW}$ is computed from the SVD, or by iterations such as Newton's iteration $\bX \leftarrow (\bX + \bX^{-\top})/2$ started at $\bX = \bW$ (Higham, 1986). Applied to $\bB\bA^\top$ in place of $\bW$ , the same argument solves the orthogonal Procrustes problem $\min_{\bQ \in O(n)}\|\bQ\bA - \bB\|_F$ .

在 ICA 中的应用。对称 FastICA 用 $\bW \leftarrow (\bW\bW^\top)^{-1/2}\bW$ 对当前估计作正交化（(8.5)），由命题可知，这正是离 $\bW$ 最近的正交矩阵。它对各行一视同仁：由于矩阵函数与正交相似变换可交换，把 $\bW$ 换成 $\bPi\bW$ ，结果就变成 $\bPi(\bW\bW^\top)^{-1/2}\bW$ ；而 Gram–Schmidt 正交化的结果依赖于各行的次序。ZCA 白化给出的白化混合矩阵 $(\bA\bA^\top)^{-1/2}\bA$ 正是 $\bA$ 的极因子（Section 3.4第 3.4 节）。数值上， $\bQ_{\bW}$ 可由 SVD 求得，也可以用迭代法计算，例如从 $\bX = \bW$ 出发的 Newton 迭代 $\bX \leftarrow (\bX + \bX^{-\top})/2$ （Higham, 1986）。把 $\bW$ 换成 $\bB\bA^\top$ ，同样的论证还可以解决正交 Procrustes 问题 $\min_{\bQ \in O(n)}\|\bQ\bA - \bB\|_F$ 。

A.5 Kronecker products, vec and tensorsKronecker 积、vec 与张量

Kronecker products. For $\bA \in \R^{p \times q}$ and $\bB \in \R^{r \times s}$ , the Kronecker product $\bA \otimes \bB \in \R^{pr \times qs}$ is the block matrix whose $(i, j)$ block is $a_{ij}\bB$ . It satisfies $(\bA \otimes \bB)(\bC \otimes \bD) = \bA\bC \otimes \bB\bD$ whenever the products exist, $(\bA \otimes \bB)^\top = \bA^\top \otimes \bB^\top$ , and $(\bA \otimes \bB)^{-1} = \bA^{-1} \otimes \bB^{-1}$ . For square $\bA$ ( $p \times p$ ) and $\bB$ ( $r \times r$ ), the eigenvalues of $\bA \otimes \bB$ are the products of an eigenvalue of $\bA$ and an eigenvalue of $\bB$ , so $\tr(\bA \otimes \bB) = \tr\bA\,\tr\bB$ and $\det(\bA \otimes \bB) = (\det\bA)^r(\det\bB)^p$ . By the first two rules, the Kronecker product of orthogonal matrices is orthogonal.

Kronecker 积。对 $\bA \in \R^{p \times q}$ 与 $\bB \in \R^{r \times s}$ ，Kronecker 积 $\bA \otimes \bB \in \R^{pr \times qs}$ 是第 $(i, j)$ 块为 $a_{ij}\bB$ 的分块矩阵。它满足：只要乘积有意义，就有 $(\bA \otimes \bB)(\bC \otimes \bD) = \bA\bC \otimes \bB\bD$ ； $(\bA \otimes \bB)^\top = \bA^\top \otimes \bB^\top$ ； $(\bA \otimes \bB)^{-1} = \bA^{-1} \otimes \bB^{-1}$ 。对方阵 $\bA$ （ $p \times p$ ）与 $\bB$ （ $r \times r$ ）， $\bA \otimes \bB$ 的特征值是 $\bA$ 的一个特征值与 $\bB$ 的一个特征值之积，所以 $\tr(\bA \otimes \bB) = \tr\bA\,\tr\bB$ ， $\det(\bA \otimes \bB) = (\det\bA)^r(\det\bB)^p$ 。由前两条规则，正交矩阵的 Kronecker 积仍是正交矩阵。

The vec operator. $\vecop(\bX)$ stacks the columns of $\bX$ into one long vector. The key identity is

vec 算子。 $\vecop(\bX)$ 把 $\bX$ 的各列依次叠成一个长向量。关键的恒等式是

\vecop(\bA\bX\bB) = (\bB^\top \otimes \bA)\,\vecop(\bX).\tag{A.2}

To see it, note that column $k$ of $\bA\bX\bB$ is $\sum_j b_{jk}\bA\bx_j$ , where $\bx_j$ is column $j$ of $\bX$ ; this is block $k$ of $(\bB^\top \otimes \bA)\vecop(\bX)$ , whose $(k, j)$ block is $b_{jk}\bA$ . Special cases are $\vecop(\ba\bb^\top) = \bb \otimes \ba$ and $\tr(\bA^\top\bB) = \vecop(\bA)^\top\vecop(\bB)$ . In NumPy, which stores arrays by rows, $\vecop(\bX)$ is X.reshape(-1, order="F"); flattening by rows instead gives the identity with $\bA \otimes \bB^\top$ in place of $\bB^\top \otimes \bA$ .

验证如下： $\bA\bX\bB$ 的第 $k$ 列为 $\sum_j b_{jk}\bA\bx_j$ ，其中 $\bx_j$ 是 $\bX$ 的第 $j$ 列；这正是 $(\bB^\top \otimes \bA)\vecop(\bX)$ 的第 $k$ 块，因为 $\bB^\top \otimes \bA$ 的第 $(k, j)$ 块为 $b_{jk}\bA$ 。特例有 $\vecop(\ba\bb^\top) = \bb \otimes \ba$ 与 $\tr(\bA^\top\bB) = \vecop(\bA)^\top\vecop(\bB)$ 。NumPy 按行存储数组， $\vecop(\bX)$ 对应 X.reshape(-1, order="F")；若按行展平，恒等式中的 $\bB^\top \otimes \bA$ 要换成 $\bA \otimes \bB^\top$ 。

Tensors. An order- $k$ tensor is a $k$ -way array $\mathcal{T} = (t_{i_1 \dots i_k})$ . The outer product of vectors $\ba, \bb, \mathbf{c}, \mathbf{d}$ is the order-4 tensor $\ba \circ \bb \circ \mathbf{c} \circ \mathbf{d}$ with entries $a_ib_jc_kd_l$ ; tensors of this form have rank one. A tensor is symmetric if its entries are invariant under permutations of the indices. A CP (canonical polyadic) decomposition writes $\mathcal{T} = \sum_{r=1}^R \lambda_r\,\ba_r \circ \bb_r \circ \mathbf{c}_r \circ \mathbf{d}_r$ , and the smallest such $R$ is the CP rank. Tensor rank behaves differently from matrix rank. It can exceed every dimension of the array, computing it is NP-hard in general (Hillar & Lim, 2013), and the CP decomposition is often essentially unique, whereas a low-rank matrix factorization is unique only up to an invertible linear transformation (Kolda & Bader, 2009). Uniqueness conditions, such as Kruskal's, are discussed in Section 12.4第 12.4 节.

张量。 $k$ 阶张量是一个 $k$ 维数组 $\mathcal{T} = (t_{i_1 \dots i_k})$ 。向量 $\ba, \bb, \mathbf{c}, \mathbf{d}$ 的外积是四阶张量 $\ba \circ \bb \circ \mathbf{c} \circ \mathbf{d}$ ，其元素为 $a_ib_jc_kd_l$ ；这种形式的张量秩为一。若张量的元素在下标的任意置换下不变，则称它是对称的。CP（canonical polyadic）分解把张量写成 $\mathcal{T} = \sum_{r=1}^R \lambda_r\,\ba_r \circ \bb_r \circ \mathbf{c}_r \circ \mathbf{d}_r$ ，这样的 $R$ 的最小值称为 CP 秩。张量秩与矩阵秩的性质很不一样：它可以超过数组的每一维，一般情形下计算它是 NP 难的（Hillar & Lim, 2013），而且 CP 分解常常在本质上唯一，而低秩矩阵分解只在相差一个可逆线性变换的意义下唯一（Kolda & Bader, 2009）。Kruskal 条件等唯一性条件见Section 12.4第 12.4 节。

Multilinear transformation. For a matrix $\bM$ with $n$ columns and an order-4 tensor $\mathcal{T}$ with all dimensions equal to $n$ , define

多重线性变换。对有 $n$ 列的矩阵 $\bM$ 与各维都等于 $n$ 的四阶张量 $\mathcal{T}$ ，定义

(\mathcal{T} \times_1 \bM \times_2 \bM \times_3 \bM \times_4 \bM)_{ijkl} = \sum_{p,q,r,t} m_{ip}\,m_{jq}\,m_{kr}\,m_{lt}\,t_{pqrt}.

It maps the rank-one tensor $\ba \circ \ba \circ \ba \circ \ba$ to $(\bM\ba) \circ (\bM\ba) \circ (\bM\ba) \circ (\bM\ba)$ , and for orthogonal $\bM$ it preserves the Frobenius norm $\|\mathcal{T}\|_F^2 = \sum_{ijkl}t_{ijkl}^2$ (Exercise A.3习题 A.3). The fourth-order cumulant tensor $\mathcal{K}_{\bx}$ of a random vector, with entries $\cum(x_i, x_j, x_k, x_l)$ , is symmetric, and by multilinearity (Proposition 2.8命题 2.8) $\mathcal{K}_{\bM\bx} = \mathcal{K}_{\bx} \times_1 \bM \times_2 \bM \times_3 \bM \times_4 \bM$ . For independent sources, $\mathcal{K}_{\bs}$ is diagonal, its only nonzero entries being $(\mathcal{K}_{\bs})_{pppp} = \kappa_p$ . Hence, for $\bx = \bA\bs$ ,

它把秩一张量 $\ba \circ \ba \circ \ba \circ \ba$ 映为 $(\bM\ba) \circ (\bM\ba) \circ (\bM\ba) \circ (\bM\ba)$ ；当 $\bM$ 正交时，它保持 Frobenius 范数 $\|\mathcal{T}\|_F^2 = \sum_{ijkl}t_{ijkl}^2$ 不变（Exercise A.3习题 A.3）。随机向量的四阶累积量张量 $\mathcal{K}_{\bx}$ 以 $\cum(x_i, x_j, x_k, x_l)$ 为元素，它是对称张量；由多重线性（Proposition 2.8命题 2.8）， $\mathcal{K}_{\bM\bx} = \mathcal{K}_{\bx} \times_1 \bM \times_2 \bM \times_3 \bM \times_4 \bM$ 。对独立的源， $\mathcal{K}_{\bs}$ 是对角张量，唯一的非零元为 $(\mathcal{K}_{\bs})_{pppp} = \kappa_p$ 。因此对 $\bx = \bA\bs$ ，

\mathcal{K}_{\bx} = \sum_{p=1}^n \kappa_p\,\ba_p \circ \ba_p \circ \ba_p \circ \ba_p,\tag{A.3}

a symmetric CP decomposition of rank at most $n$ whose factors are the columns of $\bA$ ; this is (2.3) in tensor form. Arranging the entries as an $m^2 \times m^2$ matrix, with rows indexed by $(i, j)$ and columns by $(k, l)$ , gives $\sum_p \kappa_p\,\vecop(\ba_p\ba_p^\top)\vecop(\ba_p\ba_p^\top)^\top$ , a symmetric matrix of rank at most $n$ . For whitened data the factors are orthogonal, which JADE exploits (Chapter 9第 9 章); with more sources than sensors the rank exceeds the dimension, which only tensors allow (Chapter 12第 12 章).

这是一个秩至多为 $n$ 的对称 CP 分解，其因子就是 $\bA$ 的各列；它正是 (2.3) 的张量形式。把各元素排成 $m^2 \times m^2$ 的矩阵，行以 $(i, j)$ 为指标、列以 $(k, l)$ 为指标，得到 $\sum_p \kappa_p\,\vecop(\ba_p\ba_p^\top)\vecop(\ba_p\ba_p^\top)^\top$ ，这是秩至多为 $n$ 的对称矩阵。对白化后的数据，这些因子相互正交，JADE 正是利用了这一点（Chapter 9第 9 章）；当源多于传感器时，秩超过了维数，而这只有张量才允许（Chapter 12第 12 章）。

A.6 Matrix calculus矩阵微积分

For a scalar function $f$ of a matrix $\bW \in \R^{n \times m}$ , the gradient $\partial f/\partial\bW$ is the $n \times m$ matrix of partial derivatives $\partial f/\partial w_{ij}$ . The most reliable way to compute it is through the differential: if $df = \tr(\bG^\top d\bW)$ for every direction $d\bW$ , then $\partial f/\partial\bW = \bG$ (Magnus & Neudecker, 2019). The basic rules are

对矩阵 $\bW \in \R^{n \times m}$ 的标量函数 $f$ ，梯度 $\partial f/\partial\bW$ 是由偏导数 $\partial f/\partial w_{ij}$ 构成的 $n \times m$ 矩阵。最可靠的计算方法是通过微分：若对每个方向 $d\bW$ 都有 $df = \tr(\bG^\top d\bW)$ ，则 $\partial f/\partial\bW = \bG$ （Magnus & Neudecker, 2019）。基本规则如下：

\begin{aligned} d\,\tr(\bA\bW) &= \tr(\bA\,d\bW), & \frac{\partial\,\tr(\bA\bW)}{\partial\bW} &= \bA^\top, \\[6pt] d(\ba^\top\bW\bb) &= \tr(\bb\ba^\top d\bW), & \frac{\partial\,(\ba^\top\bW\bb)}{\partial\bW} &= \ba\bb^\top, \\[6pt] d\,\tr(\bW^\top\bM\bW) &= \tr\bigl(\bW^\top(\bM + \bM^\top)\,d\bW\bigr), & \frac{\partial\,\tr(\bW^\top\bM\bW)}{\partial\bW} &= (\bM + \bM^\top)\bW, \\[6pt] d(\bW^{-1}) &= -\bW^{-1}(d\bW)\,\bW^{-1}. \end{aligned}

The last rule follows by differentiating $\bW\bW^{-1} = \bI$ . The determinant, which brings the Jacobian of $\by = \bW\bx$ into every likelihood and entropy of this book, has a particularly simple gradient.

最后一条由对 $\bW\bW^{-1} = \bI$ 求微分得到。行列式把变换 $\by = \bW\bx$ 的 Jacobi 因子带进了本书中的每个似然与熵，而它的梯度格外简单。

Proposition A.5 (gradient of the log-determinant)命题 A.5（对数行列式的梯度）

For invertible $\bW$ , $d\log|\det\bW| = \tr(\bW^{-1}d\bW)$ , hence $\partial\log|\det\bW|/\partial\bW = \bW^{-\top}$ .

对可逆的 $\bW$ ， $d\log|\det\bW| = \tr(\bW^{-1}d\bW)$ ，因此 $\partial\log|\det\bW|/\partial\bW = \bW^{-\top}$ 。

Proof.证明

$\det(\bW + d\bW) = \det\bW\,\det(\bI + \bW^{-1}d\bW)$ , and $\det(\bI + \boldsymbol{\Delta}) = 1 + \tr\boldsymbol{\Delta} + O(\|\boldsymbol{\Delta}\|^2)$ : in the expansion of the determinant, the product of the diagonal entries contributes $1 + \tr\boldsymbol{\Delta} + O(\|\boldsymbol{\Delta}\|^2)$ , and every other term contains at least two off-diagonal entries of $\boldsymbol{\Delta}$ . Hence $\log|\det(\bW + d\bW)| = \log|\det\bW| + \tr(\bW^{-1}d\bW) + O(\|d\bW\|^2)$ . Writing $\tr(\bW^{-1}d\bW) = \tr\bigl((\bW^{-\top})^\top d\bW\bigr)$ identifies the gradient.

$\det(\bW + d\bW) = \det\bW\,\det(\bI + \bW^{-1}d\bW)$ ，而 $\det(\bI + \boldsymbol{\Delta}) = 1 + \tr\boldsymbol{\Delta} + O(\|\boldsymbol{\Delta}\|^2)$ ：在行列式的展开式中，对角元之积贡献 $1 + \tr\boldsymbol{\Delta} + O(\|\boldsymbol{\Delta}\|^2)$ ，其余每一项都至少含有 $\boldsymbol{\Delta}$ 的两个非对角元。因此 $\log|\det(\bW + d\bW)| = \log|\det\bW| + \tr(\bW^{-1}d\bW) + O(\|d\bW\|^2)$ 。把 $\tr(\bW^{-1}d\bW)$ 写成 $\tr\bigl((\bW^{-\top})^\top d\bW\bigr)$ ，即得梯度。

Example A.6 (gradient of the ICA log-likelihood)例 A.6（ICA 对数似然的梯度）

Let $f(\bW) = \log|\det\bW| + \sum_i \E[\log p_i(\bw_i^\top\bx)]$ , where $\bw_i^\top$ is row $i$ of $\bW$ , $\by = \bW\bx$ , and $\psi_i = -p_i'/p_i$ is the score function of source $i$ . Then $df = \tr(\bW^{-1}d\bW) - \E\bigl[\sum_i \psi_i(y_i)\,d\bw_i^\top\bx\bigr] = \tr(\bW^{-1}d\bW) - \tr\bigl(\E[\bx\bpsi(\by)^\top]\,d\bW\bigr)$ , so

设 $f(\bW) = \log|\det\bW| + \sum_i \E[\log p_i(\bw_i^\top\bx)]$ ，其中 $\bw_i^\top$ 是 $\bW$ 的第 $i$ 行， $\by = \bW\bx$ ， $\psi_i = -p_i'/p_i$ 是源 $i$ 的得分函数。则 $df = \tr(\bW^{-1}d\bW) - \E\bigl[\sum_i \psi_i(y_i)\,d\bw_i^\top\bx\bigr] = \tr(\bW^{-1}d\bW) - \tr\bigl(\E[\bx\bpsi(\by)^\top]\,d\bW\bigr)$ ，所以

\frac{\partial f}{\partial\bW} = \bW^{-\top} - \E\bigl[\bpsi(\by)\,\bx^\top\bigr], \qquad \frac{\partial f}{\partial\bW}\,\bW^\top\bW = \bigl(\bI - \E[\bpsi(\by)\,\by^\top]\bigr)\bW.

The second expression, the gradient multiplied on the right by the positive definite matrix $\bW^\top\bW$ , is the relative or natural gradient of Chapter 7第 7 章; it is still an ascent direction, because $\langle\bG, \bG\bW^\top\bW\rangle = \|\bG\bW^\top\|_F^2 \ge 0$ for $\bG = \partial f/\partial\bW$ .

第二个表达式，即右乘正定矩阵 $\bW^\top\bW$ 之后的梯度，就是Chapter 7第 7 章中的相对梯度（自然梯度）；它仍是上升方向，因为对 $\bG = \partial f/\partial\bW$ 有 $\langle\bG, \bG\bW^\top\bW\rangle = \|\bG\bW^\top\|_F^2 \ge 0$ 。

Formulas like these are easy to get wrong by a transpose, and easy to check. The script below verifies (A.2), Proposition A.5命题 A.5 by central differences, and Proposition A.4命题 A.4 against 20,000 random orthogonal matrices drawn uniformly from $O(4)$ .

这类公式很容易在转置上出错，也很容易检验。下面的脚本验证 (A.2)，用中心差分验证Proposition A.5命题 A.5，并用从 $O(4)$ 中均匀抽取的 20000 个随机正交矩阵检验Proposition A.4命题 A.4。

import numpy as np

rng = np.random.default_rng(0)

# vec(AXB) = (B^T kron A) vec(X), with vec stacking columns
A, X, B = (rng.standard_normal(s) for s in [(3, 4), (4, 5), (5, 2)])
vec = lambda M: M.reshape(-1, order="F")
print(np.allclose(vec(A @ X @ B), np.kron(B.T, A) @ vec(X)))

# gradient of log|det W| by central differences
W, h = rng.standard_normal((4, 4)), 1e-6
logdet = lambda M: np.linalg.slogdet(M)[1]
G = np.zeros((4, 4))
for i in range(4):
    for j in range(4):
        D = np.zeros((4, 4)); D[i, j] = h
        G[i, j] = (logdet(W + D) - logdet(W - D)) / (2 * h)
print(f"{np.abs(G - np.linalg.inv(W).T).max():.1e}")

# closest orthogonal matrix: L R^T = (W W^T)^(-1/2) W
L, sv, Rt = np.linalg.svd(W)
d, E = np.linalg.eigh(W @ W.T)
print(np.allclose(L @ Rt, E @ np.diag(d ** -0.5) @ E.T @ W))
def random_orthogonal(n):                 # uniformly distributed on O(n)
    Q, R = np.linalg.qr(rng.standard_normal((n, n)))
    return Q * np.sign(np.diag(R))
dist = min(np.linalg.norm(W - random_orthogonal(4)) for _ in range(20000))
print(f"{np.linalg.norm(W - L @ Rt):.3f} {dist:.3f}")

It prints True, a finite-difference error of 5.7e-08, True, and the distances 2.886 from $\bW$ to its polar factor and 3.047 to the best of the random orthogonal matrices.

程序依次输出 True、有限差分误差 5.7e-08、True，以及 $\bW$ 到其极因子的距离 2.886 和到随机正交矩阵中最近者的距离 3.047。

A.7 Optimization on the sphere and the orthogonal group球面与正交群上的优化

Many ICA contrasts are optimized under constraints. One-unit methods search over unit vectors $\bw$ (Chapter 5第 5 章, Chapter 8第 8 章), and methods for whitened data search over orthogonal matrices. Both constraint sets are smooth manifolds, and the same three ingredients recur: first-order (Lagrange) conditions, the Riemannian gradient, and retractions, which map a step taken in the tangent space back onto the manifold (Edelman, Arias & Smith, 1998; Absil, Mahony & Sepulchre, 2008).

许多 ICA 对比函数都是在约束下优化的。单单元方法在单位向量 $\bw$ 中搜索（Chapter 5第 5 章，Chapter 8第 8 章），针对白化数据的方法则在正交矩阵中搜索。这两个约束集都是光滑流形，反复出现的是同样三样东西：一阶（拉格朗日）条件、黎曼梯度，以及把在切空间中迈出的一步拉回流形的收缩映射（retraction）（Edelman, Arias & Smith, 1998; Absil, Mahony & Sepulchre, 2008）。

Proposition A.7 (optimality conditions on the sphere)命题 A.7（球面上的最优性条件）

Let $f$ be twice continuously differentiable on $\R^n$ , let $\bw$ be a unit vector, and write $\lambda = \bw^\top\nabla f(\bw)$ . (a) If $\bw$ is a local maximum or minimum of $f$ on the unit sphere, then $\nabla f(\bw) = \lambda\bw$ ; equivalently, the Riemannian gradient $(\bI - \bw\bw^\top)\nabla f(\bw)$ vanishes. (b) At such a critical point, if $\bv^\top(\nabla^2 f(\bw) - \lambda\bI)\bv < 0$ for every nonzero $\bv$ with $\bw^\top\bv = 0$ , then $\bw$ is a strict local maximum on the sphere; if $\bv^\top(\nabla^2 f(\bw) - \lambda\bI)\bv > 0$ for some such $\bv$ , it is not a local maximum.

设 $f$ 在 $\R^n$ 上二阶连续可微， $\bw$ 是单位向量，记 $\lambda = \bw^\top\nabla f(\bw)$ 。（a）若 $\bw$ 是 $f$ 在单位球面上的局部极大点或极小点，则 $\nabla f(\bw) = \lambda\bw$ ；等价地，黎曼梯度 $(\bI - \bw\bw^\top)\nabla f(\bw)$ 为零。（b）在这样的临界点处，若对每个满足 $\bw^\top\bv = 0$ 的非零 $\bv$ 都有 $\bv^\top(\nabla^2 f(\bw) - \lambda\bI)\bv < 0$ ，则 $\bw$ 是球面上的严格局部极大点；若对某个这样的 $\bv$ 有 $\bv^\top(\nabla^2 f(\bw) - \lambda\bI)\bv > 0$ ，则它不是局部极大点。

Proof.证明

For a unit vector $\bv \perp \bw$ , the curve $\bw(t) = \cos t\,\bw + \sin t\,\bv$ stays on the sphere, with $\bw(0) = \bw$ , $\bw'(0) = \bv$ and $\bw''(0) = -\bw$ . Hence the first derivative of $f(\bw(t))$ at $t = 0$ is $\nabla f(\bw)^\top\bv$ , and the second is $\bv^\top\nabla^2 f(\bw)\bv - \nabla f(\bw)^\top\bw$ . At a local extremum the first derivative vanishes for every tangent $\bv$ , so $\nabla f(\bw)$ is parallel to $\bw$ , which is (a). At a critical point the second derivative equals $\bv^\top(\nabla^2 f(\bw) - \lambda\bI)\bv$ . If it is positive for some $\bv$ , $f$ increases along that curve, so $\bw$ is not a local maximum. Every point of the sphere near $\bw$ has the form $\bw(t)$ for a small $t$ and some unit $\bv \perp \bw$ , and by Taylor's theorem $f(\bw(t)) = f(\bw) + \tfrac{t^2}{2}\bv^\top(\nabla^2 f(\bw) - \lambda\bI)\bv + o(t^2)$ uniformly in $\bv$ , because the unit tangent vectors form a compact set; a negative definite form therefore gives $f(\bw(t)) < f(\bw)$ for small $t \ne 0$ .

对与 $\bw$ 正交的单位向量 $\bv$ ，曲线 $\bw(t) = \cos t\,\bw + \sin t\,\bv$ 始终在球面上，且 $\bw(0) = \bw$ ， $\bw'(0) = \bv$ ， $\bw''(0) = -\bw$ 。因此 $f(\bw(t))$ 在 $t = 0$ 处的一阶导数为 $\nabla f(\bw)^\top\bv$ ，二阶导数为 $\bv^\top\nabla^2 f(\bw)\bv - \nabla f(\bw)^\top\bw$ 。在局部极值点，一阶导数对每个切向量 $\bv$ 都为零，所以 $\nabla f(\bw)$ 与 $\bw$ 平行，这就是（a）。在临界点处，二阶导数等于 $\bv^\top(\nabla^2 f(\bw) - \lambda\bI)\bv$ 。若它对某个 $\bv$ 为正，则 $f$ 沿这条曲线增大，所以 $\bw$ 不是局部极大点。球面上靠近 $\bw$ 的每一点都可以写成 $\bw(t)$ 的形式（ $t$ 很小， $\bv \perp \bw$ 为单位向量）；由于单位切向量构成紧集，由 Taylor 定理， $f(\bw(t)) = f(\bw) + \tfrac{t^2}{2}\bv^\top(\nabla^2 f(\bw) - \lambda\bI)\bv + o(t^2)$ 关于 $\bv$ 一致成立，因此当这个二次型负定时，对很小的 $t \ne 0$ 有 $f(\bw(t)) < f(\bw)$ 。

For example, $f(\bw) = \bw^\top\bC\bw$ with $\bC$ symmetric has $\nabla f = 2\bC\bw$ and $\nabla^2 f = 2\bC$ , so its critical points on the sphere are the unit eigenvectors, with $\lambda = 2\lambda_k$ at $\be_k$ . Suppose $\lambda_1 > \lambda_2 \ge \dots \ge \lambda_n$ . At $\be_1$ , the form $2\bv^\top(\bC - \lambda_1\bI)\bv$ is negative for every nonzero $\bv \perp \be_1$ , so $\be_1$ is a strict local maximum. At $\be_k$ with $k \ge 2$ , the tangent direction $\bv = \be_1$ gives $2(\lambda_1 - \lambda_k) > 0$ , so $\be_k$ is not a local maximum. The only local maxima are $\pm\be_1$ : PCA has no spurious local optima. Chapter 5第 5 章 carries out the same analysis for kurtosis (Theorem 5.3定理 5.3).

举例来说，对对称矩阵 $\bC$ ， $f(\bw) = \bw^\top\bC\bw$ 有 $\nabla f = 2\bC\bw$ 与 $\nabla^2 f = 2\bC$ ，所以它在球面上的临界点就是单位特征向量，在 $\be_k$ 处 $\lambda = 2\lambda_k$ 。设 $\lambda_1 > \lambda_2 \ge \dots \ge \lambda_n$ 。在 $\be_1$ 处，二次型 $2\bv^\top(\bC - \lambda_1\bI)\bv$ 对每个与 $\be_1$ 正交的非零 $\bv$ 都为负，所以 $\be_1$ 是严格局部极大点。在 $\be_k$ （ $k \ge 2$ ）处，取切方向 $\bv = \be_1$ 得 $2(\lambda_1 - \lambda_k) > 0$ ，所以 $\be_k$ 不是局部极大点。唯一的局部极大点是 $\pm\be_1$ ：PCA 没有虚假的局部最优。Chapter 5第 5 章对峭度作了同样的分析（Theorem 5.3定理 5.3）。

The vector $(\bI - \bw\bw^\top)\nabla f(\bw)$ , the projection of the gradient onto the tangent space $\{\bv : \bw^\top\bv = 0\}$ , is the Riemannian gradient of $f$ on the sphere. Gradient ascent takes a step $\mu\bv$ along it and maps back by normalization, $\bw \leftarrow (\bw + \mu\bv)/\|\bw + \mu\bv\|$ , the simplest retraction; the normalization step of FastICA has the same form.

向量 $(\bI - \bw\bw^\top)\nabla f(\bw)$ 是梯度在切空间 $\{\bv : \bw^\top\bv = 0\}$ 上的投影，称为 $f$ 在球面上的黎曼梯度。梯度上升沿它走一步 $\mu\bv$ ，再通过归一化拉回球面： $\bw \leftarrow (\bw + \mu\bv)/\|\bw + \mu\bv\|$ ，这是最简单的收缩映射；FastICA 的归一化步骤也是这种形式。

The orthogonal group. Let $f$ be a smooth function of $\bU \in \R^{n \times n}$ with Euclidean gradient $\bG = \partial f/\partial\bU$ . By Proposition A.1命题 A.1, the directions tangent to $O(n)$ at $\bU$ are $\boldsymbol{\Omega}\bU$ with $\boldsymbol{\Omega}$ skew-symmetric, and the derivative of $f$ along $t \mapsto \exp(t\boldsymbol{\Omega})\bU$ at $t = 0$ is $\langle\bG, \boldsymbol{\Omega}\bU\rangle = \langle\bG\bU^\top, \boldsymbol{\Omega}\rangle = \langle\operatorname{skew}(\bG\bU^\top), \boldsymbol{\Omega}\rangle$ , because the symmetric part of $\bG\bU^\top$ is orthogonal to $\boldsymbol{\Omega}$ (Section A.1第 A.1 节). Therefore

正交群。设 $f$ 是 $\bU \in \R^{n \times n}$ 的光滑函数，欧氏梯度为 $\bG = \partial f/\partial\bU$ 。由Proposition A.1命题 A.1， $O(n)$ 在 $\bU$ 处的切方向为 $\boldsymbol{\Omega}\bU$ （ $\boldsymbol{\Omega}$ 反对称）， $f$ 沿 $t \mapsto \exp(t\boldsymbol{\Omega})\bU$ 在 $t = 0$ 处的导数为 $\langle\bG, \boldsymbol{\Omega}\bU\rangle = \langle\bG\bU^\top, \boldsymbol{\Omega}\rangle = \langle\operatorname{skew}(\bG\bU^\top), \boldsymbol{\Omega}\rangle$ ，因为 $\bG\bU^\top$ 的对称部分与 $\boldsymbol{\Omega}$ 正交（Section A.1第 A.1 节）。因此

\operatorname{grad} f(\bU) = \operatorname{skew}(\bG\bU^\top)\,\bU\tag{A.4}

is the Riemannian gradient, and $\bU$ is a critical point of $f$ on $O(n)$ if and only if $\bG\bU^\top$ is symmetric. This is the Lagrange condition for the constraint $\bU\bU^\top = \bI$ , whose multiplier is a symmetric matrix $\bLambda$ with $\bG = \bLambda\bU$ . Retractions include the matrix exponential $\bU \leftarrow \exp(\mu\boldsymbol{\Omega})\bU$ , the Cayley transform $\bU \leftarrow (\bI - \tfrac{\mu}{2}\boldsymbol{\Omega})^{-1}(\bI + \tfrac{\mu}{2}\boldsymbol{\Omega})\bU$ (Exercise A.1习题 A.1), and the polar factor of $\bU + \mu\boldsymbol{\Omega}\bU$ , that is, symmetric orthogonalization (Section A.4第 A.4 节). In two dimensions each of them simply changes the rotation angle.

就是黎曼梯度； $\bU$ 是 $f$ 在 $O(n)$ 上的临界点，当且仅当 $\bG\bU^\top$ 对称。这正是约束 $\bU\bU^\top = \bI$ 下的拉格朗日条件，其乘子是满足 $\bG = \bLambda\bU$ 的对称矩阵 $\bLambda$ 。常用的收缩映射有矩阵指数 $\bU \leftarrow \exp(\mu\boldsymbol{\Omega})\bU$ 、Cayley 变换 $\bU \leftarrow (\bI - \tfrac{\mu}{2}\boldsymbol{\Omega})^{-1}(\bI + \tfrac{\mu}{2}\boldsymbol{\Omega})\bU$ （Exercise A.1习题 A.1），以及 $\bU + \mu\boldsymbol{\Omega}\bU$ 的极因子，即对称正交化（Section A.4第 A.4 节）。在二维情形，它们都只是改变旋转角。

A.8 Asymptotic statistics渐近统计

Part IV and several other chapters make statements about large samples. We write $X_T = O_P(r_T)$ if $X_T/r_T$ is bounded in probability, that is, for every $\varepsilon > 0$ there is an $M$ with $P(|X_T/r_T| > M) < \varepsilon$ for all $T$ , and $X_T = o_P(r_T)$ if $X_T/r_T \to 0$ in probability. By the central limit theorem, the average of $T$ i.i.d. random vectors with mean $\bmu$ and covariance $\bSigma$ satisfies $\sqrt T(\bar\bx - \bmu) \to \Ncal(\bzero, \bSigma)$ in distribution, so $\bar\bx - \bmu = O_P(T^{-1/2})$ . Slutsky's lemma allows us to add $o_P(1)$ terms to a sequence that converges in distribution, or to multiply it by a sequence that converges in probability to a constant, without changing the limit beyond the obvious (van der Vaart, 1998).

第四部分以及其他若干章都涉及大样本下的结论。若 $X_T/r_T$ 依概率有界，即对任意 $\varepsilon > 0$ ，存在 $M$ 使对所有 $T$ 都有 $P(|X_T/r_T| > M) < \varepsilon$ ，就记 $X_T = O_P(r_T)$ ；若 $X_T/r_T$ 依概率收敛到 0，就记 $X_T = o_P(r_T)$ 。由中心极限定理， $T$ 个均值为 $\bmu$ 、协方差为 $\bSigma$ 的独立同分布随机向量的平均满足 $\sqrt T(\bar\bx - \bmu) \to \Ncal(\bzero, \bSigma)$ （依分布），所以 $\bar\bx - \bmu = O_P(T^{-1/2})$ 。Slutsky 引理允许我们给依分布收敛的序列加上 $o_P(1)$ 项，或乘以依概率收敛到常数的序列，而极限只发生显然的变化（van der Vaart, 1998）。

Proposition A.8 (delta method)命题 A.8（delta 方法）

Let $\sqrt T(\hat\btheta_T - \btheta) \to \Ncal(\bzero, \bSigma)$ in distribution, and let $g$ be a real function differentiable at $\btheta$ . Then $\sqrt T\bigl(g(\hat\btheta_T) - g(\btheta)\bigr) \to \Ncal\bigl(0,\ \nabla g(\btheta)^\top\bSigma\,\nabla g(\btheta)\bigr)$ . For a vector-valued $g$ , replace $\nabla g(\btheta)^\top$ by the Jacobian matrix of $g$ at $\btheta$ .

设 $\sqrt T(\hat\btheta_T - \btheta)$ 依分布收敛到 $\Ncal(\bzero, \bSigma)$ ， $g$ 是在 $\btheta$ 处可微的实函数。则 $\sqrt T\bigl(g(\hat\btheta_T) - g(\btheta)\bigr) \to \Ncal\bigl(0,\ \nabla g(\btheta)^\top\bSigma\,\nabla g(\btheta)\bigr)$ 。若 $g$ 取向量值，把 $\nabla g(\btheta)^\top$ 换成 $g$ 在 $\btheta$ 处的 Jacobi 矩阵即可。

Proof.证明

Differentiability gives $g(\hat\btheta_T) - g(\btheta) = \nabla g(\btheta)^\top(\hat\btheta_T - \btheta) + o_P(\|\hat\btheta_T - \btheta\|)$ , and $\|\hat\btheta_T - \btheta\| = O_P(T^{-1/2})$ . Multiply by $\sqrt T$ : the remainder becomes $o_P(1)$ , and the linear term converges to $\Ncal(0, \nabla g^\top\bSigma\nabla g)$ . Slutsky's lemma completes the proof.

由可微性， $g(\hat\btheta_T) - g(\btheta) = \nabla g(\btheta)^\top(\hat\btheta_T - \btheta) + o_P(\|\hat\btheta_T - \btheta\|)$ ，且 $\|\hat\btheta_T - \btheta\| = O_P(T^{-1/2})$ 。乘以 $\sqrt T$ 后，余项变为 $o_P(1)$ ，线性项收敛到 $\Ncal(0, \nabla g^\top\bSigma\nabla g)$ 。由 Slutsky 引理即得结论。

Example A.9 (variance of the sample kurtosis)例 A.9（样本峭度的方差）

Let $y$ have zero mean and moments $\mu_j = \E[y^j]$ up to order 8, and let $\hat k = \hat m_4 - 3\hat m_2^2$ with $\hat m_j = \frac1T\sum_t y(t)^j$ . By the central limit theorem, $\sqrt T(\hat m_2 - \mu_2, \hat m_4 - \mu_4)$ is asymptotically normal with variances $\mu_4 - \mu_2^2$ and $\mu_8 - \mu_4^2$ and covariance $\mu_6 - \mu_2\mu_4$ , and $g(a, b) = b - 3a^2$ has gradient $(-6\mu_2, 1)$ . The delta method gives $\sqrt T(\hat k - \kurt(y)) \to \Ncal(0, v)$ with

设 $y$ 均值为零，直到 8 阶的矩 $\mu_j = \E[y^j]$ 都有限，令 $\hat k = \hat m_4 - 3\hat m_2^2$ ，其中 $\hat m_j = \frac1T\sum_t y(t)^j$ 。由中心极限定理， $\sqrt T(\hat m_2 - \mu_2, \hat m_4 - \mu_4)$ 渐近正态，两个分量的方差分别为 $\mu_4 - \mu_2^2$ 与 $\mu_8 - \mu_4^2$ ，协方差为 $\mu_6 - \mu_2\mu_4$ ；而 $g(a, b) = b - 3a^2$ 的梯度为 $(-6\mu_2, 1)$ 。由 delta 方法， $\sqrt T(\hat k - \kurt(y)) \to \Ncal(0, v)$ ，其中

v = \mu_8 - \mu_4^2 - 12\,\mu_2(\mu_6 - \mu_2\mu_4) + 36\,\mu_2^2(\mu_4 - \mu_2^2).\tag{A.5}

For a standard Gaussian, $(\mu_2, \mu_4, \mu_6, \mu_8) = (1, 3, 15, 105)$ and $v = 24$ , the $\sqrt{24/T}$ used in Exercise 1.5习题 1.5. For the unit-variance Laplace law, $(\mu_2, \mu_4, \mu_6, \mu_8) = (1, 6, 90, 2520)$ and $v = 1656$ : kurtosis estimates of super-Gaussian sources are far noisier (Section 5.6第 5.6 节). For symmetric laws, estimating the mean as well does not change $v$ : to first order, centering adds $-4\mu_3\bar y$ to $\hat k$ , which changes $v$ by $64\mu_2\mu_3^2 - 8\mu_3\mu_5$ , and $\mu_3 = 0$ for symmetric laws. A simulation with $T = 2000$ and $20\,000$ repetitions, subtracting the sample mean, gives $T\Var(\hat k) \approx 23.5$ , $9.85$ and $1655$ for Gaussian, uniform and Laplace data, with Monte Carlo standard errors of about $0.25$ , $0.10$ and $40$ , against the predicted $24$ , $9.87$ (Exercise A.4习题 A.4) and $1656$ .

对标准高斯分布， $(\mu_2, \mu_4, \mu_6, \mu_8) = (1, 3, 15, 105)$ ， $v = 24$ ，这正是Exercise 1.5习题 1.5中用到的 $\sqrt{24/T}$ 。对单位方差的拉普拉斯分布， $(\mu_2, \mu_4, \mu_6, \mu_8) = (1, 6, 90, 2520)$ ， $v = 1656$ ：超高斯源的峭度估计要嘈杂得多（Section 5.6第 5.6 节）。对对称分布，即使同时估计均值， $v$ 也不变：在一阶近似下，中心化给 $\hat k$ 加上 $-4\mu_3\bar y$ ，使 $v$ 改变 $64\mu_2\mu_3^2 - 8\mu_3\mu_5$ ，而对称分布的 $\mu_3 = 0$ 。用 $T = 2000$ 、重复 $20\,000$ 次并减去样本均值的模拟，对高斯、均匀与拉普拉斯数据得到 $T\Var(\hat k) \approx 23.5$ 、 $9.85$ 与 $1655$ （蒙特卡罗标准误分别约为 $0.25$ 、 $0.10$ 与 $40$ ），预言值分别为 $24$ 、 $9.87$ （Exercise A.4习题 A.4）与 $1656$ 。

M-estimators. Most ICA estimators are defined as solutions of estimating equations, such as $\E[\varphi_i(y_i)\,y_j] = 0$ for $i \ne j$ in Chapter 6第 6 章, with the expectation replaced by a sample average. Their asymptotic covariance has a standard form.

M 估计量。大多数 ICA 估计量都定义为估计方程的解，例如Chapter 6第 6 章中的 $\E[\varphi_i(y_i)\,y_j] = 0$ （ $i \ne j$ ），其中期望换成样本平均。它们的渐近协方差有标准的形式。

Proposition A.10 (asymptotic normality of M-estimators)命题 A.10（M 估计量的渐近正态性）

Let $\bx(1), \dots, \bx(T)$ be i.i.d., and let $\hat\btheta_T$ solve $\frac1T\sum_{t=1}^T \mathbf{h}(\bx(t), \btheta) = \bzero$ , where $\E[\mathbf{h}(\bx, \btheta_0)] = \bzero$ . Assume that $\hat\btheta_T \to \btheta_0$ in probability, that $\mathbf{h}$ is twice continuously differentiable in $\btheta$ with first and second derivatives dominated near $\btheta_0$ by an integrable function of $\bx$ , that $\bH = \E[\partial\mathbf{h}(\bx, \btheta_0)/\partial\btheta^\top]$ is invertible, and that $\bB = \E[\mathbf{h}(\bx, \btheta_0)\,\mathbf{h}(\bx, \btheta_0)^\top]$ is finite. Then $\sqrt T(\hat\btheta_T - \btheta_0) \to \Ncal(\bzero,\ \bH^{-1}\bB\bH^{-\top})$ .

设 $\bx(1), \dots, \bx(T)$ 独立同分布， $\hat\btheta_T$ 是方程 $\frac1T\sum_{t=1}^T \mathbf{h}(\bx(t), \btheta) = \bzero$ 的解，且 $\E[\mathbf{h}(\bx, \btheta_0)] = \bzero$ 。假设 $\hat\btheta_T$ 依概率收敛到 $\btheta_0$ ； $\mathbf{h}$ 关于 $\btheta$ 二阶连续可微，且其一阶与二阶导数在 $\btheta_0$ 附近被某个关于 $\bx$ 可积的函数控制； $\bH = \E[\partial\mathbf{h}(\bx, \btheta_0)/\partial\btheta^\top]$ 可逆； $\bB = \E[\mathbf{h}(\bx, \btheta_0)\,\mathbf{h}(\bx, \btheta_0)^\top]$ 有限。则 $\sqrt T(\hat\btheta_T - \btheta_0) \to \Ncal(\bzero,\ \bH^{-1}\bB\bH^{-\top})$ 。

Proof.证明

(Sketch.) A Taylor expansion of the estimating equations around $\btheta_0$ gives $\bzero = \frac1T\sum_t \mathbf{h}(\bx(t), \btheta_0) + \hat\bH_T(\hat\btheta_T - \btheta_0)$ , where $\hat\bH_T$ is an average of derivatives evaluated at points between $\hat\btheta_T$ and $\btheta_0$ ; by the law of large numbers and the domination condition, $\hat\bH_T \to \bH$ in probability. Hence $\sqrt T(\hat\btheta_T - \btheta_0) = -\hat\bH_T^{-1}\,T^{-1/2}\sum_t \mathbf{h}(\bx(t), \btheta_0)$ with probability tending to one, and the central limit theorem and Slutsky's lemma give the limit. Van der Vaart (1998, Chapter 5) gives the details and weaker conditions.

（证明概要。）把估计方程在 $\btheta_0$ 附近作 Taylor 展开，得 $\bzero = \frac1T\sum_t \mathbf{h}(\bx(t), \btheta_0) + \hat\bH_T(\hat\btheta_T - \btheta_0)$ ，其中 $\hat\bH_T$ 是在 $\hat\btheta_T$ 与 $\btheta_0$ 之间的点处取值的导数的平均；由大数定律与控制条件， $\hat\bH_T$ 依概率收敛到 $\bH$ 。因此，在概率趋于一的事件上， $\sqrt T(\hat\btheta_T - \btheta_0) = -\hat\bH_T^{-1}\,T^{-1/2}\sum_t \mathbf{h}(\bx(t), \btheta_0)$ ，再由中心极限定理与 Slutsky 引理即得极限。细节以及更弱的条件见 van der Vaart（1998）第 5 章。

The covariance $\bH^{-1}\bB\bH^{-\top}$ is called the sandwich. Chapter 10第 10 章 evaluates it for ICA estimators (Theorem 10.8定理 10.8), and Exercise A.5习题 A.5 works out the simplest case, the estimation of a location parameter.

协方差 $\bH^{-1}\bB\bH^{-\top}$ 称为三明治（sandwich）协方差。Chapter 10第 10 章对 ICA 估计量计算了它（Theorem 10.8定理 10.8），Exercise A.5习题 A.5则处理最简单的情形，即位置参数的估计。

Fisher information and the Cramér–Rao bound. For a family of densities $p(\bx; \btheta)$ , the Fisher information is $\mathcal{I}(\btheta) = \E\bigl[\nabla_{\btheta}\log p(\bx; \btheta)\,\nabla_{\btheta}\log p(\bx; \btheta)^\top\bigr]$ , which equals $-\E[\nabla^2_{\btheta}\log p(\bx; \btheta)]$ under regularity conditions. The Cramér–Rao bound states that, under regularity conditions, every unbiased estimator based on $T$ i.i.d. observations satisfies $\Cov(\hat\btheta) \succeq (T\,\mathcal{I}(\btheta))^{-1}$ ; the maximum-likelihood estimator is asymptotically normal with covariance $\mathcal{I}(\btheta)^{-1}/T$ , so it attains the bound asymptotically (Lehmann & Casella, 1998; van der Vaart, 1998). In the sandwich formula, maximum likelihood takes $\mathbf{h}$ to be the score, so that $\bH = -\mathcal{I}$ , $\bB = \mathcal{I}$ and $\bH^{-1}\bB\bH^{-\top} = \mathcal{I}^{-1}$ . For ICA, Chapter 10第 10 章 derives the bound for the gain matrix (Theorem 10.10定理 10.10).

Fisher 信息与 Cramér–Rao 界。对密度族 $p(\bx; \btheta)$ ，Fisher 信息定义为 $\mathcal{I}(\btheta) = \E\bigl[\nabla_{\btheta}\log p(\bx; \btheta)\,\nabla_{\btheta}\log p(\bx; \btheta)^\top\bigr]$ ，在正则条件下它等于 $-\E[\nabla^2_{\btheta}\log p(\bx; \btheta)]$ 。Cramér–Rao 界是说：在正则条件下，基于 $T$ 个独立同分布观测的任何无偏估计量都满足 $\Cov(\hat\btheta) \succeq (T\,\mathcal{I}(\btheta))^{-1}$ ；最大似然估计量渐近正态，协方差为 $\mathcal{I}(\btheta)^{-1}/T$ ，所以它渐近地达到这个界（Lehmann & Casella, 1998; van der Vaart, 1998）。在三明治公式中，最大似然取 $\mathbf{h}$ 为得分，于是 $\bH = -\mathcal{I}$ ， $\bB = \mathcal{I}$ ， $\bH^{-1}\bB\bH^{-\top} = \mathcal{I}^{-1}$ 。对 ICA，Chapter 10第 10 章推导了增益矩阵的这个界（Theorem 10.10定理 10.10）。

For a location family $p(x - \theta)$ , the score with respect to $\theta$ is $-p'(x - \theta)/p(x - \theta) = \psi(x - \theta)$ , where $\psi = -p'/p$ is the score function in the book's notation, and the Fisher information is $\mathcal{J} = \E[\psi(s)^2]$ . For the standard Gaussian, $\psi(u) = u$ and $\mathcal{J} = 1$ ; for the unit-variance Laplace law, $\psi(u) = \sqrt2\,\sign(u)$ and $\mathcal{J} = 2$ . Among laws of a given variance, the Gaussian carries the least information.

对位置族 $p(x - \theta)$ ，关于 $\theta$ 的得分为 $-p'(x - \theta)/p(x - \theta) = \psi(x - \theta)$ ，其中 $\psi = -p'/p$ 即本书记号中的得分函数，Fisher 信息为 $\mathcal{J} = \E[\psi(s)^2]$ 。对标准高斯分布， $\psi(u) = u$ ， $\mathcal{J} = 1$ ；对单位方差的拉普拉斯分布， $\psi(u) = \sqrt2\,\sign(u)$ ， $\mathcal{J} = 2$ 。在方差给定的分布中，高斯分布携带的信息最少。

Proposition A.11 (Fisher information at unit variance)命题 A.11（单位方差下的 Fisher 信息）

Let $s$ have zero mean, unit variance and a continuously differentiable density $p > 0$ with score $\psi = -p'/p$ , $\mathcal{J} = \E[\psi(s)^2] < \infty$ , and $u\,p(u) \to 0$ as $|u| \to \infty$ . Then $\mathcal{J} \ge 1$ , with equality if and only if $s \sim \Ncal(0, 1)$ .

设 $s$ 均值为零、方差为一，具有连续可微的密度 $p > 0$ ，其得分为 $\psi = -p'/p$ ， $\mathcal{J} = \E[\psi(s)^2] < \infty$ ，并且当 $|u| \to \infty$ 时 $u\,p(u) \to 0$ 。则 $\mathcal{J} \ge 1$ ，等号成立当且仅当 $s \sim \Ncal(0, 1)$ 。

Proof.证明

Integrating by parts, $\E[s\,\psi(s)] = -\int u\,p'(u)\,du = \bigl[-u\,p(u)\bigr]_{-\infty}^{\infty} + \int p(u)\,du = 1$ ; the integral converges absolutely because $\E|s\,\psi(s)| \le (\E[s^2]\,\mathcal{J})^{1/2}$ . By the Cauchy–Schwarz inequality, $1 = (\E[s\,\psi(s)])^2 \le \E[s^2]\,\E[\psi(s)^2] = \mathcal{J}$ . Equality holds if and only if $\psi(s) = c\,s$ almost surely for a constant $c$ ; since $\psi$ is continuous and $p > 0$ , this means $\psi(u) = cu$ for all $u$ , so $(\log p)' = -cu$ and $p$ is a Gaussian density with variance $1/c$ . Unit variance forces $c = 1$ .

分部积分得 $\E[s\,\psi(s)] = -\int u\,p'(u)\,du = \bigl[-u\,p(u)\bigr]_{-\infty}^{\infty} + \int p(u)\,du = 1$ ；由于 $\E|s\,\psi(s)| \le (\E[s^2]\,\mathcal{J})^{1/2}$ ，这个积分绝对收敛。由 Cauchy–Schwarz 不等式， $1 = (\E[s\,\psi(s)])^2 \le \E[s^2]\,\E[\psi(s)^2] = \mathcal{J}$ 。等号成立当且仅当存在常数 $c$ 使 $\psi(s) = c\,s$ 几乎必然成立；由于 $\psi$ 连续且 $p > 0$ ，这意味着对所有 $u$ 有 $\psi(u) = cu$ ，于是 $(\log p)' = -cu$ ， $p$ 是方差为 $1/c$ 的高斯密度。单位方差迫使 $c = 1$ 。

The same integration by parts applies to absolutely continuous densities such as the Laplace density, so the inequality holds for them too. Consequently, for two unit-variance sources $\mathcal{J}_i\mathcal{J}_j \ge 1$ , with equality only when both are Gaussian, which is why the conditions $\mathcal{J}_i\mathcal{J}_j > 1$ in Chapter 7第 7 章 and Chapter 10第 10 章 fail only for pairs of Gaussian sources.

同样的分部积分也适用于拉普拉斯密度这样的绝对连续密度，所以不等式对它们同样成立。因此，对两个单位方差的源有 $\mathcal{J}_i\mathcal{J}_j \ge 1$ ，仅当二者都是高斯的时取等号；这就是为什么Chapter 7第 7 章与Chapter 10第 10 章中的条件 $\mathcal{J}_i\mathcal{J}_j > 1$ 只对一对高斯源才不成立。

Key points

本章要点

$O(n)$ has dimension $n(n-1)/2$ , and its tangent directions at $\bQ$ are $\boldsymbol{\Omega}\bQ$ with $\boldsymbol{\Omega}$ skew-symmetric. Givens rotations generate $SO(n)$ , and together with a diagonal sign matrix they give every element of $O(n)$ . A scaled permutation is orthogonal only when its scales are $\pm 1$ .
A positive semidefinite matrix has a unique positive semidefinite square root, $\bC^{1/2} = \mathbf{E}\bLambda^{1/2}\mathbf{E}^\top$ ; non-symmetric square roots differ by orthogonal factors.
The polar factor $(\bW\bW^\top)^{-1/2}\bW = \mathbf{L}\bR^\top$ is the orthogonal matrix closest to $\bW$ ; it is what symmetric orthogonalization computes.
$\vecop(\bA\bX\bB) = (\bB^\top \otimes \bA)\vecop(\bX)$ . Cumulant tensors transform multilinearly, and for $\bx = \bA\bs$ the fourth-order cumulant tensor is a symmetric CP decomposition along the columns of $\bA$ .
Compute gradients through differentials; in particular $\partial\log|\det\bW|/\partial\bW = \bW^{-\top}$ .
On the sphere, critical points satisfy $\nabla f(\bw) = \lambda\bw$ and the Riemannian gradient is $(\bI - \bw\bw^\top)\nabla f$ ; on $O(n)$ it is $\operatorname{skew}(\bG\bU^\top)\bU$ .
The delta method, the sandwich covariance of M-estimators and the Cramér–Rao bound are the tools of Part IV; a unit-variance law has Fisher information $\mathcal{J} \ge 1$ , with equality only for the Gaussian.

$O(n)$ 的维数为 $n(n-1)/2$ ，在 $\bQ$ 处的切方向为 $\boldsymbol{\Omega}\bQ$ （ $\boldsymbol{\Omega}$ 反对称）。Givens 旋转生成 $SO(n)$ ，再乘上一个符号对角矩阵即可得到 $O(n)$ 的任意元素。带尺度的置换矩阵仅当尺度为 $\pm 1$ 时才是正交的。
半正定矩阵有唯一的半正定平方根 $\bC^{1/2} = \mathbf{E}\bLambda^{1/2}\mathbf{E}^\top$ ；非对称的平方根之间相差正交因子。
极因子 $(\bW\bW^\top)^{-1/2}\bW = \mathbf{L}\bR^\top$ 是离 $\bW$ 最近的正交矩阵，对称正交化计算的正是它。
$\vecop(\bA\bX\bB) = (\bB^\top \otimes \bA)\vecop(\bX)$ 。累积量张量按多重线性方式变换；对 $\bx = \bA\bs$ ，四阶累积量张量是沿 $\bA$ 各列的对称 CP 分解。
通过微分来计算梯度；特别地， $\partial\log|\det\bW|/\partial\bW = \bW^{-\top}$ 。
在球面上，临界点满足 $\nabla f(\bw) = \lambda\bw$ ，黎曼梯度为 $(\bI - \bw\bw^\top)\nabla f$ ；在 $O(n)$ 上则为 $\operatorname{skew}(\bG\bU^\top)\bU$ 。
delta 方法、M 估计量的三明治协方差与 Cramér–Rao 界是第四部分的工具；单位方差分布的 Fisher 信息 $\mathcal{J} \ge 1$ ，仅对高斯分布取等号。

A.9 Exercises习题

Exercise A.1习题 A.1 basic基础

Let $\boldsymbol{\Omega}$ be skew-symmetric. Show that $\bI - \boldsymbol{\Omega}$ is invertible and that the Cayley transform $\bQ = (\bI - \boldsymbol{\Omega})^{-1}(\bI + \boldsymbol{\Omega})$ is orthogonal. For $n = 2$ and $\boldsymbol{\Omega} = \tfrac{\theta}{2}\bigl(\begin{smallmatrix} 0 & -1 \\ 1 & 0 \end{smallmatrix}\bigr)$ , which rotation is $\bQ$ ?

设 $\boldsymbol{\Omega}$ 反对称。证明 $\bI - \boldsymbol{\Omega}$ 可逆，并且 Cayley 变换 $\bQ = (\bI - \boldsymbol{\Omega})^{-1}(\bI + \boldsymbol{\Omega})$ 是正交矩阵。当 $n = 2$ 、 $\boldsymbol{\Omega} = \tfrac{\theta}{2}\bigl(\begin{smallmatrix} 0 & -1 \\ 1 & 0 \end{smallmatrix}\bigr)$ 时， $\bQ$ 是哪个旋转？

Show solution查看解答

For every $\bv$ , $\bv^\top\boldsymbol{\Omega}\bv$ is a scalar equal to its transpose $\bv^\top\boldsymbol{\Omega}^\top\bv = -\bv^\top\boldsymbol{\Omega}\bv$ , so it is zero. Hence $\bv^\top(\bI \mp \boldsymbol{\Omega})\bv = \|\bv\|^2 > 0$ for $\bv \ne \bzero$ , and both $\bI - \boldsymbol{\Omega}$ and $\bI + \boldsymbol{\Omega}$ are invertible. Since $(\bI - \boldsymbol{\Omega})^\top = \bI + \boldsymbol{\Omega}$ and $(\bI + \boldsymbol{\Omega})^\top = \bI - \boldsymbol{\Omega}$ , we get $\bQ^\top = (\bI - \boldsymbol{\Omega})(\bI + \boldsymbol{\Omega})^{-1}$ . The matrices $(\bI + \boldsymbol{\Omega})^{-1}$ and $(\bI - \boldsymbol{\Omega})^{-1}$ commute, being functions of the same matrix, so $\bQ^\top\bQ = (\bI - \boldsymbol{\Omega})(\bI + \boldsymbol{\Omega})^{-1}(\bI - \boldsymbol{\Omega})^{-1}(\bI + \boldsymbol{\Omega}) = (\bI - \boldsymbol{\Omega})(\bI - \boldsymbol{\Omega})^{-1}(\bI + \boldsymbol{\Omega})^{-1}(\bI + \boldsymbol{\Omega}) = \bI$ . For $n = 2$ , write $a = \theta/2$ ; then $\bQ = \frac{1}{1 + a^2}\bigl(\begin{smallmatrix} 1 - a^2 & -2a \\ 2a & 1 - a^2 \end{smallmatrix}\bigr)$ , the rotation by the angle $2\arctan(\theta/2)$ , since $\cos\phi = (1 - a^2)/(1 + a^2)$ and $\sin\phi = 2a/(1 + a^2)$ for $\phi = 2\arctan a$ . For small $\theta$ the angle is $\theta + O(\theta^3)$ , so the Cayley transform agrees with the exponential to first order, as a retraction should.

对任意 $\bv$ ，标量 $\bv^\top\boldsymbol{\Omega}\bv$ 等于它的转置 $\bv^\top\boldsymbol{\Omega}^\top\bv = -\bv^\top\boldsymbol{\Omega}\bv$ ，所以为零。于是对 $\bv \ne \bzero$ 有 $\bv^\top(\bI \mp \boldsymbol{\Omega})\bv = \|\bv\|^2 > 0$ ， $\bI - \boldsymbol{\Omega}$ 与 $\bI + \boldsymbol{\Omega}$ 都可逆。由于 $(\bI - \boldsymbol{\Omega})^\top = \bI + \boldsymbol{\Omega}$ ， $(\bI + \boldsymbol{\Omega})^\top = \bI - \boldsymbol{\Omega}$ ，得 $\bQ^\top = (\bI - \boldsymbol{\Omega})(\bI + \boldsymbol{\Omega})^{-1}$ 。 $(\bI + \boldsymbol{\Omega})^{-1}$ 与 $(\bI - \boldsymbol{\Omega})^{-1}$ 都是同一个矩阵的函数，因而可交换，所以 $\bQ^\top\bQ = (\bI - \boldsymbol{\Omega})(\bI + \boldsymbol{\Omega})^{-1}(\bI - \boldsymbol{\Omega})^{-1}(\bI + \boldsymbol{\Omega}) = (\bI - \boldsymbol{\Omega})(\bI - \boldsymbol{\Omega})^{-1}(\bI + \boldsymbol{\Omega})^{-1}(\bI + \boldsymbol{\Omega}) = \bI$ 。当 $n = 2$ 时，记 $a = \theta/2$ ，则 $\bQ = \frac{1}{1 + a^2}\bigl(\begin{smallmatrix} 1 - a^2 & -2a \\ 2a & 1 - a^2 \end{smallmatrix}\bigr)$ ；由于对 $\phi = 2\arctan a$ 有 $\cos\phi = (1 - a^2)/(1 + a^2)$ 、 $\sin\phi = 2a/(1 + a^2)$ ，它是旋转 $2\arctan(\theta/2)$ 角的矩阵。当 $\theta$ 很小时这个角为 $\theta + O(\theta^3)$ ，所以 Cayley 变换与矩阵指数在一阶上一致，这正是收缩映射应有的性质。

Exercise A.2习题 A.2 basic基础

Let $\bz$ be white, let $G$ be a smooth function with derivative $g$ , and let $f(\bw) = \E[G(\bw^\top\bz)]$ ; assume that the expectations exist and that differentiation under the expectation is allowed. Show that the critical points of $f$ on the unit sphere satisfy $\E[\bz\,g(\bw^\top\bz)] = \beta\bw$ with $\beta = \E[(\bw^\top\bz)\,g(\bw^\top\bz)]$ . What happens for $G(u) = u^2$ ?

设 $\bz$ 是白的， $G$ 是光滑函数、导数为 $g$ ，令 $f(\bw) = \E[G(\bw^\top\bz)]$ ；假设各期望存在，且可以在期望号下求导。证明 $f$ 在单位球面上的临界点满足 $\E[\bz\,g(\bw^\top\bz)] = \beta\bw$ ，其中 $\beta = \E[(\bw^\top\bz)\,g(\bw^\top\bz)]$ 。当 $G(u) = u^2$ 时会怎样？

Show solution查看解答

Differentiating under the expectation, $\nabla f(\bw) = \E[\bz\,g(\bw^\top\bz)]$ . By Proposition A.7命题 A.7(a), a critical point satisfies $\nabla f(\bw) = \lambda\bw$ with $\lambda = \bw^\top\nabla f(\bw) = \E[(\bw^\top\bz)\,g(\bw^\top\bz)] = \beta$ . This is the equation from which FastICA starts (Chapter 8第 8 章). For $G(u) = u^2$ , $g(u) = 2u$ and $\nabla f(\bw) = 2\E[\bz\bz^\top]\bw = 2\bw$ because $\bz$ is white, so every unit vector is a critical point: $f \equiv 1$ on the sphere. Second-order statistics carry no information about directions in whitened data, as Section 3.5第 3.5 节 explains.

在期望号下求导得 $\nabla f(\bw) = \E[\bz\,g(\bw^\top\bz)]$ 。由Proposition A.7命题 A.7（a），临界点满足 $\nabla f(\bw) = \lambda\bw$ ，其中 $\lambda = \bw^\top\nabla f(\bw) = \E[(\bw^\top\bz)\,g(\bw^\top\bz)] = \beta$ 。这正是 FastICA 的出发点（Chapter 8第 8 章）。当 $G(u) = u^2$ 时， $g(u) = 2u$ ，由于 $\bz$ 是白的， $\nabla f(\bw) = 2\E[\bz\bz^\top]\bw = 2\bw$ ，所以每个单位向量都是临界点： $f$ 在球面上恒等于 1。白化数据的二阶统计量不携带任何关于方向的信息，正如Section 3.5第 3.5 节所解释的那样。

Exercise A.3习题 A.3 intermediate进阶

Show that for orthogonal $\bQ$ the multilinear transform $\mathcal{T} \mapsto \mathcal{T} \times_1 \bQ \times_2 \bQ \times_3 \bQ \times_4 \bQ$ preserves the Frobenius norm. Deduce that if $\bz$ is white and $\by = \bU\bz$ with $\bU$ orthogonal, then $\sum_{ijkl}\cum(y_i, y_j, y_k, y_l)^2$ does not depend on $\bU$ , so that maximizing $\sum_i\kurt(y_i)^2$ over $\bU$ is the same as minimizing the sum of squares of all cross-cumulants (Comon, 1994).

证明：对正交矩阵 $\bQ$ ，多重线性变换 $\mathcal{T} \mapsto \mathcal{T} \times_1 \bQ \times_2 \bQ \times_3 \bQ \times_4 \bQ$ 保持 Frobenius 范数不变。由此推出：若 $\bz$ 是白的， $\by = \bU\bz$ 且 $\bU$ 正交，则 $\sum_{ijkl}\cum(y_i, y_j, y_k, y_l)^2$ 与 $\bU$ 无关；因而在 $\bU$ 上最大化 $\sum_i\kurt(y_i)^2$ ，等价于最小化所有交叉累积量的平方和（Comon, 1994）。

Show solution查看解答

Arrange the entries of $\mathcal{T}$ in a vector $\bt$ , with the first index running fastest. The transformed entries $\sum_{pqrt} q_{ip}q_{jq}q_{kr}q_{lt}\,t_{pqrt}$ then form the vector $(\bQ \otimes \bQ \otimes \bQ \otimes \bQ)\bt$ . By the rules of Section A.5第 A.5 节, a Kronecker product of orthogonal matrices is orthogonal, so the Euclidean norm of $\bt$ , which is the Frobenius norm of $\mathcal{T}$ , is preserved. For $\by = \bU\bz$ , multilinearity gives $\mathcal{K}_{\by} = \mathcal{K}_{\bz} \times_1 \bU \times_2 \bU \times_3 \bU \times_4 \bU$ , so $\|\mathcal{K}_{\by}\|_F = \|\mathcal{K}_{\bz}\|_F$ for every $\bU$ . Split $\|\mathcal{K}_{\by}\|_F^2$ into the diagonal entries, $\cum(y_i, y_i, y_i, y_i)^2 = \kurt(y_i)^2$ , and the squared cross-cumulants. Their total is fixed, so maximizing the first part is the same as minimizing the second. Independent outputs have zero cross-cumulants (Proposition 2.8命题 2.8), so at a separating rotation the whole norm sits on the diagonal and $\sum_i\kurt(y_i)^2$ attains its largest possible value $\|\mathcal{K}_{\bz}\|_F^2$ .

把 $\mathcal{T}$ 的元素按第一个下标变化最快的次序排成向量 $\bt$ 。变换后的元素 $\sum_{pqrt} q_{ip}q_{jq}q_{kr}q_{lt}\,t_{pqrt}$ 构成向量 $(\bQ \otimes \bQ \otimes \bQ \otimes \bQ)\bt$ 。由Section A.5第 A.5 节中的规则，正交矩阵的 Kronecker 积仍是正交矩阵，所以 $\bt$ 的欧氏范数，也就是 $\mathcal{T}$ 的 Frobenius 范数，保持不变。对 $\by = \bU\bz$ ，由多重线性得 $\mathcal{K}_{\by} = \mathcal{K}_{\bz} \times_1 \bU \times_2 \bU \times_3 \bU \times_4 \bU$ ，所以对每个 $\bU$ 都有 $\|\mathcal{K}_{\by}\|_F = \|\mathcal{K}_{\bz}\|_F$ 。把 $\|\mathcal{K}_{\by}\|_F^2$ 分成对角元部分 $\cum(y_i, y_i, y_i, y_i)^2 = \kurt(y_i)^2$ 与交叉累积量的平方两部分。二者之和固定，所以最大化前者等价于最小化后者。独立的输出没有交叉累积量（Proposition 2.8命题 2.8），所以在实现分离的旋转处，全部范数都集中在对角元上， $\sum_i\kurt(y_i)^2$ 达到可能的最大值 $\|\mathcal{K}_{\bz}\|_F^2$ 。

Exercise A.4习题 A.4 intermediate进阶

Using (A.5), compute the asymptotic variance $v$ of the sample kurtosis for the unit-variance uniform law, whose moments are $\mu_2 = 1$ , $\mu_4 = 9/5$ , $\mu_6 = 27/7$ and $\mu_8 = 9$ . Compare it with the Gaussian and Laplace values, and find the sample size at which the standard error of $\hat k$ equals $0.1$ for each of the three laws.

利用 (A.5)，计算单位方差均匀分布的样本峭度的渐近方差 $v$ ，其各阶矩为 $\mu_2 = 1$ ， $\mu_4 = 9/5$ ， $\mu_6 = 27/7$ ， $\mu_8 = 9$ 。把它与高斯分布和拉普拉斯分布的值比较，并对这三个分布分别求出使 $\hat k$ 的标准误等于 $0.1$ 的样本量。

Show solution查看解答

$v = 9 - \tfrac{81}{25} - 12\bigl(\tfrac{27}{7} - \tfrac95\bigr) + 36\bigl(\tfrac95 - 1\bigr) = \tfrac{144}{25} - \tfrac{864}{35} + \tfrac{144}{5} = \tfrac{1008 - 4320 + 5040}{175} = \tfrac{1728}{175} \approx 9.87$ . So $v$ is about $9.87$ for the uniform law, $24$ for the Gaussian and $1656$ for the Laplace law: bounded, sub-Gaussian laws give precise kurtosis estimates, and heavy tails give very imprecise ones. The standard error $\sqrt{v/T}$ equals $0.1$ when $T = 100\,v$ , that is, at about 990 samples for the uniform law, 2400 for the Gaussian and 165 600 for the Laplace law.

$v = 9 - \tfrac{81}{25} - 12\bigl(\tfrac{27}{7} - \tfrac95\bigr) + 36\bigl(\tfrac95 - 1\bigr) = \tfrac{144}{25} - \tfrac{864}{35} + \tfrac{144}{5} = \tfrac{1008 - 4320 + 5040}{175} = \tfrac{1728}{175} \approx 9.87$ 。所以均匀分布的 $v$ 约为 $9.87$ ，高斯分布为 $24$ ，拉普拉斯分布为 $1656$ ：有界的亚高斯分布给出精确的峭度估计，重尾分布给出的估计则很不精确。标准误 $\sqrt{v/T}$ 等于 $0.1$ 当且仅当 $T = 100\,v$ ，即均匀分布约需 990 个样本，高斯分布需 2400 个，拉普拉斯分布需 165 600 个。

Exercise A.5习题 A.5 advanced硬核

Let $x = \theta + s$ , where $s$ has a symmetric, continuously differentiable density $p > 0$ with score $\psi = -p'/p$ and Fisher information $\mathcal{J} = \E[\psi(s)^2] < \infty$ . Let $\varphi$ be odd and twice continuously differentiable with $\varphi'$ and $\varphi''$ bounded, with $\E[\varphi(s)^2] < \infty$ , $\E[\varphi'(s)] \ne 0$ and $\varphi(u)p(u) \to 0$ as $|u| \to \infty$ , and let $\hat\theta$ be a consistent solution of $\sum_t \varphi(x(t) - \theta) = 0$ . (a) Use Proposition A.10命题 A.10 to show that the asymptotic variance of $\sqrt T(\hat\theta - \theta)$ is $\E[\varphi(s)^2]/(\E[\varphi'(s)])^2$ . (b) Show that $\E[\varphi'(s)] = \E[\varphi(s)\psi(s)]$ , and deduce that this variance is at least $1/\mathcal{J}$ , with equality if and only if $\varphi$ is proportional to $\psi$ . (c) Evaluate the bound and the variance for the unit-variance Laplace law and $\varphi = \tanh$ .

设 $x = \theta + s$ ，其中 $s$ 具有对称、连续可微的密度 $p > 0$ ，得分为 $\psi = -p'/p$ ，Fisher 信息 $\mathcal{J} = \E[\psi(s)^2] < \infty$ 。设 $\varphi$ 是二阶连续可微的奇函数， $\varphi'$ 与 $\varphi''$ 有界，并满足 $\E[\varphi(s)^2] < \infty$ 、 $\E[\varphi'(s)] \ne 0$ ，且当 $|u| \to \infty$ 时 $\varphi(u)p(u) \to 0$ ；令 $\hat\theta$ 是 $\sum_t \varphi(x(t) - \theta) = 0$ 的相合解。（a）利用Proposition A.10命题 A.10证明 $\sqrt T(\hat\theta - \theta)$ 的渐近方差为 $\E[\varphi(s)^2]/(\E[\varphi'(s)])^2$ 。（b）证明 $\E[\varphi'(s)] = \E[\varphi(s)\psi(s)]$ ，并由此推出这个方差不小于 $1/\mathcal{J}$ ，等号成立当且仅当 $\varphi$ 与 $\psi$ 成比例。（c）对单位方差的拉普拉斯分布与 $\varphi = \tanh$ ，计算这个下界与方差。

Show solution查看解答

(a) Here $\mathbf{h}(x, \theta) = \varphi(x - \theta)$ , and $\E[\varphi(s)] = 0$ because $\varphi$ is odd and $p$ symmetric, so the true $\theta$ solves the population equation. Since $\partial\mathbf{h}/\partial\theta = -\varphi'(x - \theta)$ , we have $\bH = -\E[\varphi'(s)] \ne 0$ and $\bB = \E[\varphi(s)^2] < \infty$ . The first and second derivatives of $\mathbf{h}$ , $-\varphi'(x - \theta)$ and $\varphi''(x - \theta)$ , are bounded by constants, so all the conditions of Proposition A.10命题 A.10 hold, and the sandwich is $\bB/\bH^2 = \E[\varphi(s)^2]/(\E[\varphi'(s)])^2$ . (b) Integrating by parts and using $\varphi p \to 0$ at $\pm\infty$ , $\E[\varphi'(s)] = \int\varphi'(u)p(u)\,du = -\int\varphi(u)p'(u)\,du = \int\varphi(u)\psi(u)p(u)\,du = \E[\varphi(s)\psi(s)]$ . By the Cauchy–Schwarz inequality, $(\E[\varphi\psi])^2 \le \E[\varphi^2]\,\E[\psi^2] = \E[\varphi^2]\,\mathcal{J}$ , so $\E[\varphi^2]/(\E[\varphi'])^2 \ge 1/\mathcal{J}$ , with equality if and only if $\varphi = c\psi$ . The maximum-likelihood choice $\varphi = \psi$ attains the Cramér–Rao bound $1/(T\mathcal{J})$ . (c) The Laplace density is not differentiable at 0, but it is absolutely continuous and the integration by parts in (b) still holds. The function $\tanh$ is odd and bounded, and its derivatives $1 - \tanh^2$ and $-2\tanh\,(1 - \tanh^2)$ are bounded, so it satisfies the hypotheses. With $b = 1/\sqrt2$ , $\mathcal{J} = 1/b^2 = 2$ , so the bound is $1/2$ . Numerical integration gives $\E[\tanh^2(s)] \approx 0.3235$ and $\E[\tanh'(s)] = \E[1 - \tanh^2(s)] \approx 0.6765$ , so the asymptotic variance of the tanh estimator is about $0.3235/0.6765^2 \approx 0.707$ , some 41% above the bound. Chapter 10第 10 章 carries out the same comparison for ICA (Proposition 10.9命题 10.9).

（a）这里 $\mathbf{h}(x, \theta) = \varphi(x - \theta)$ ；由于 $\varphi$ 是奇函数且 $p$ 对称， $\E[\varphi(s)] = 0$ ，所以真值 $\theta$ 满足总体方程。由 $\partial\mathbf{h}/\partial\theta = -\varphi'(x - \theta)$ 得 $\bH = -\E[\varphi'(s)] \ne 0$ ， $\bB = \E[\varphi(s)^2] < \infty$ 。 $\mathbf{h}$ 的一阶与二阶导数 $-\varphi'(x - \theta)$ 与 $\varphi''(x - \theta)$ 都被常数控制，所以Proposition A.10命题 A.10的条件全部成立，三明治协方差为 $\bB/\bH^2 = \E[\varphi(s)^2]/(\E[\varphi'(s)])^2$ 。（b）分部积分并利用 $\varphi p$ 在 $\pm\infty$ 处趋于零，得 $\E[\varphi'(s)] = \int\varphi'(u)p(u)\,du = -\int\varphi(u)p'(u)\,du = \int\varphi(u)\psi(u)p(u)\,du = \E[\varphi(s)\psi(s)]$ 。由 Cauchy–Schwarz 不等式， $(\E[\varphi\psi])^2 \le \E[\varphi^2]\,\E[\psi^2] = \E[\varphi^2]\,\mathcal{J}$ ，所以 $\E[\varphi^2]/(\E[\varphi'])^2 \ge 1/\mathcal{J}$ ，等号成立当且仅当 $\varphi = c\psi$ 。最大似然的选择 $\varphi = \psi$ 达到 Cramér–Rao 界 $1/(T\mathcal{J})$ 。（c）拉普拉斯密度在 0 处不可微，但它绝对连续，（b）中的分部积分仍然成立。 $\tanh$ 是有界的奇函数，其导数 $1 - \tanh^2$ 与 $-2\tanh\,(1 - \tanh^2)$ 也有界，所以它满足题设条件。取 $b = 1/\sqrt2$ ， $\mathcal{J} = 1/b^2 = 2$ ，下界为 $1/2$ 。数值积分给出 $\E[\tanh^2(s)] \approx 0.3235$ ， $\E[\tanh'(s)] = \E[1 - \tanh^2(s)] \approx 0.6765$ ，所以 tanh 估计量的渐近方差约为 $0.3235/0.6765^2 \approx 0.707$ ，比下界高约 41%。Chapter 10第 10 章对 ICA 作了同样的比较（Proposition 10.9命题 10.9）。

A.10 References参考文献

Absil, P.-A., Mahony, R., & Sepulchre, R. (2008). Optimization Algorithms on Matrix Manifolds. Princeton University Press.
Comon, P. (1994). Independent component analysis, a new concept? Signal Processing, 36(3), 287–314.
Edelman, A., Arias, T. A., & Smith, S. T. (1998). The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications, 20(2), 303–353.
Golub, G. H., & Van Loan, C. F. (2013). Matrix Computations (4th ed.). Johns Hopkins University Press.
Higham, N. J. (1986). Computing the polar decomposition—with applications. SIAM Journal on Scientific and Statistical Computing, 7(4), 1160–1174.
Hillar, C. J., & Lim, L.-H. (2013). Most tensor problems are NP-hard. Journal of the ACM, 60(6), Article 45.
Horn, R. A., & Johnson, C. R. (2013). Matrix Analysis (2nd ed.). Cambridge University Press.
Kolda, T. G., & Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review, 51(3), 455–500.
Lehmann, E. L., & Casella, G. (1998). Theory of Point Estimation (2nd ed.). Springer.
Magnus, J. R., & Neudecker, H. (2019). Matrix Differential Calculus with Applications in Statistics and Econometrics (3rd ed.). Wiley.
van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press.