1 矩阵求导的本质结构

对于向量和标量的函数/自变量, 我们常常对他们的导数的定义感到困惑. 这里简单看几个例子回顾一下定义.

一般地,

1. ${\displaystyle f:{\mathbb{R}}^{n\times 1}\to \mathbb{R}}$, 则 $$D_{x}f(\overrightarrow{x})=\frac{\partial f(x)}{\partial x^{\mathrm{T}}}=(\frac{\partial f}{\partial x_1},\cdots ,\frac{\partial f}{\partial x_n}),{\mathrm{\nabla }}_{x}f(x)=\frac{\partial f(x)}{\partial x}.$$
2. ${\displaystyle f:{\mathbb{R}}^{m\times n}\to \mathbb{R}}$, 定义 ${\displaystyle \mathrm{vec}(X)=(x_{11},\cdots ,x_{n1},x_{12},\cdots ,x_{m2},\cdots ,x_{1n},\cdots ,x_{mn}{)}^{\mathrm{T}}}$ ^[1], 定义 $$\begin{array}{rl}{D}_{\mathrm{vec}(X)}f(X)& =\frac{\partial f(X)}{\partial {\mathrm{vec}}^{\mathrm{T}}(X)}=(\frac{\partial f}{\partial x_{11}},\cdots ,\frac{\partial f}{\partial x_{m1}},\cdots ,\frac{\partial f}{\partial x_{1n}},\cdots ,\frac{\partial f}{\partial x_{mn}}),\\ D_{X}f(X)& =\frac{\partial f(X)}{\partial X_{m\times n}^{\mathrm{T}}}={\left(\begin{array}{ccc}\frac{\partial f}{\partial X_{11}}& \cdots & \frac{\partial f}{\partial X_{m1}}\\ ⋮& \ddots & ⋮\\ \frac{\partial f}{\partial X_{1n}}& \cdots & \frac{\partial f}{\partial X_{mn}}\end{array}\right)}_{n\times m},\end{array}$$ 以及类似的 ${\displaystyle {\mathrm{\nabla }}_{\mathrm{vec}(X)}f(X),{\mathrm{\nabla }}_{X}f(X)}$.
3. ${\displaystyle F:{\mathbb{R}}^{m\times n}\to {\mathbb{R}}^{p\times q}}$, 则类似地可定义 ${\displaystyle \mathrm{vec}(F(X))}$, 则 $$\begin{array}{rl}{D}_{X}F(X)& ={\left(\frac{\partial {\mathrm{vec}}_{pq\times 1}(F(X))}{\partial {\mathrm{vec}}_{mn\times 1}^{\mathrm{T}}(X)}\right)}_{pq\times mn},\\ {\mathrm{\nabla }}_{X}F(X)& ={\left(\frac{\partial {\mathrm{vec}}_{pq\times 1}^{\mathrm{T}}(F(X))}{\partial {\mathrm{vec}}_{mn\times 1}(X)}\right)}_{mn\times pq}.\end{array}$$

2 基于本质结构的数学推导

也即, 直接采用定义计算.

1. ${\displaystyle \overrightarrow{x}=(x_1,\cdots ,x_n{)}^{\mathrm{T}}}$, 用上述定义计算得:
   1. 设 ${\displaystyle c\in \mathbb{R}}$. $$\begin{array}{rl}\frac{\partial c}{\partial x}& =0_{n\times 1},\\ \frac{\partial [c_{1}f(x)+c_{2}g(x)]}{\partial x}& =c_1\frac{\partial f}{\partial x}+c_2\frac{\partial g}{\partial x},\\ \frac{\partial (f\cdot g)}{\partial x}& =\frac{\partial f}{\partial x}g(x)+\frac{\partial g}{\partial x}f(x),\\ \frac{\partial (f/g)}{\partial x}& =\frac{1}{g^2(x)}(\frac{\partial f}{\partial x}g\left(x\right)-f\frac{\partial g}{\partial x}).\end{array}$$
   2. 设 ${\displaystyle a=(a_1,\cdots ,a_n{)}^{\mathrm{T}}}$, ${\displaystyle A=(a_{ij}{)}_{n\times n}}$. $$\begin{array}{rl}\frac{\partial x^{\mathrm{T}}a}{\partial x}& =\frac{\partial (a^{\mathrm{T}}x)}{\partial x}=a,\\ \frac{\partial (x^{\mathrm{T}}x)}{\partial x}& =2x,\\ \frac{\partial (x^{\mathrm{T}}Ax)}{\partial x}& =(A+A^{\mathrm{T}})x,\\ \frac{\partial (a^{\mathrm{T}}x{x}^{\mathrm{T}}b)}{\partial x}& =(a{b}^{\mathrm{T}}+b{a}^{\mathrm{T}})x.\end{array}$$

2. ${\displaystyle X=(x_{ij}{)}_{m\times n}}$,
   1. 与 1 (1) 完全相同
   2. $$\text{begin{align*}}$$

\frac{\partial (a ^{\mathrm{T}}X ^{\mathrm{T}}b)}{\partial X}&= ba ^{\mathrm{T}},\
\frac{\partial (a ^{\mathrm{T}}XX ^{\mathrm{T}}b)}{\partial X}&= (ab ^{\mathrm{T}}+ba ^{\mathrm{T}})X,\
\frac{\partial (a ^{\mathrm{T}}X ^{\mathrm{T}}Xb)}{\partial X}&= X(ba ^{\mathrm{T}}+ab ^{\mathrm{T}}).
\end{align*}$$

3 基于迹的快速求导法

首先回顾迹的性质: $$\mathrm{tr}(A^{\mathrm{T}})=\mathrm{tr}(A),\mathrm{tr}(AB)=\mathrm{tr}(BA).$$
此外, 定义全微分. 若 ${\displaystyle f:{\mathbb{R}}^{m\times n}\to \mathbb{R}}$, ${\displaystyle \frac{\partial f}{\partial X}\in {\mathbb{R}}^{m\times n}}$, 则 $$\mathrm{d}f=\sum _{i=1}^m\sum _{j=1}^n\frac{\partial f}{\partial X_{ij}}\mathrm{d}{X}_{ij}.$$ 又 $$\mathrm{tr}(A^{\mathrm{T}}B)=\sum _{i=1}^m\sum _{j=1}^n{A}_{ij}{B}_{ij},$$ 则联立两式得 $$\begin{array}{}\text{(3.1)}& \mathrm{d}f=\mathrm{tr}({\left(\frac{\partial f}{\partial X}\right)}^{\mathrm{T}}\cdot \mathrm{d}X).\end{array}$$

接下来我们来看一些具体的例子. 首先注意, 对 ${\displaystyle f:{\mathbb{R}}^{m\times n}\to \mathbb{R}}$, $$\begin{array}{}\text{(3.2)}& \text{tr}(f(X))=f(X)\Rightarrow \mathrm{d}\mathrm{tr}f(X)=\mathrm{d}f(X)=\mathrm{tr}\mathrm{d}f(X).\end{array}$$