矩阵微分

雅可比矩阵和矩阵微分

实值函数分类

雅可比矩阵

在向量分析中，雅可比矩阵是函数以一阶偏导数以一定方式排列成的矩阵，其行列式称为雅可比行列式

• $1\times m$行向量偏导算子定义为$$D_x\stackrel{def}{=}\frac{\partial }{\partial x^T}=[\frac{\partial }{\partial x_1}...\frac{\partial }{\partial x_m}]$$所以实值标量函数f（x）在x的偏导向量由$1\times m$行向量给出

$$D_{x}f(x)=\frac{\partial f(x)}{\partial x^T}$$$$=[\frac{\partial f(x)}{\partial x_1},...,\frac{\partial f(x)}{\partial x_m}]$$

当实值标量函数f（X）的变元为实值矩阵$X\in R^{m\times n}$，可能存在两种定义

• Jacobian偏导$$D_{x}f(x)\stackrel{def}{=}\frac{\partial f(X)}{\partial X^T}=\left[\begin{array}{ccc}\frac{\partial f(X)}{\partial x_{11}}& ...& \frac{\partial f(X)}{\partial x_{m1}}\\ ...& ...& ...\\ \frac{\partial f(X)}{\partial x_{1n}}& ...& \frac{\partial f(X)}{\partial x_{mn}}\end{array}\right]$$
• 行向量偏导$$D_{vecX}f(X)=\frac{\partial f(X)}{{\partial }_{ve{c}^T(X)}}=[\frac{\partial f(X)}{\partial x_{11}}...\frac{\partial f(X)}{\partial x_{m1}}...\frac{\partial f(X)}{\partial x_{1n}}...\frac{\partial f(X)}{\partial x_{mn}}]$$
• 两者关系$$D_{vec}f(X)=rvec(D_{X}f(X))=(vec(D_X^{T}f(X)){)}^T$$即实值标量函数f(X)的行向量偏导$Dve{c}_{X}f(X)$等于Jacobian矩阵的转置$D_X^{T}f(X)$的列向量化

矩阵函数雅克比矩阵

函数为矩阵，变元为矩阵

$$D_{X}F(X)=\frac{\partial f(X)}{\partial X^T}=\left[\begin{array}{ccc}\frac{\partial f(X)}{\partial X^T}& ...& \frac{\partial f(X)}{\partial X^T}\\ ...& ...& ...\\ \frac{\partial f(X)}{\partial X^T}& ...& \frac{\partial f(X)}{\partial X^T}\end{array}\right]$$$$=\left[\begin{array}{ccccccc}\frac{\partial f_{11}}{\partial x_{11}}& ...& \frac{\partial f_{11}}{\partial x_{m1}}& ...& \frac{\partial f_{11}}{\partial x_{1n}}& ...& \frac{\partial f_{11}}{\partial x_{m1}}\\ ...& ...& ...& ...& ...& ...\\ \frac{\partial f_{p1}}{\partial x_{11}}& ...& \frac{\partial f_{p1}}{\partial x_{m1}}& ...& \frac{\partial f_{p1}}{\partial x_{1n}}& ...& \frac{\partial f_{p1}}{\partial x_{m1}}\\ ...& ...& ...& ...& ...& ...\\ \frac{\partial f_{1q}}{\partial x_{11}}& ...& \frac{\partial f_{1q}}{\partial x_{m1}}& ...& \frac{\partial f_{1q}}{\partial x_{1n}}& ...& \frac{\partial f_{1q}}{\partial x_{m1}}\\ ...& ...& ...& ...& ...& ...\\ \frac{\partial f_{pq}}{\partial x_{11}}& ...& \frac{\partial f_{pq}}{\partial x_{m1}}& ...& \frac{\partial f_{pq}}{\partial x_{1n}}& ...& \frac{\partial f_{pq}}{\partial x_{m1}}\end{array}\right]$$

梯度矩阵

采用列向量作为偏导算子称为偏导算子

$${\mathrm{\nabla }}_x\stackrel{def}{=}[\frac{\partial }{\partial x_1},...,\frac{\partial }{\partial x_m}{]}^T=\frac{\partial }{\partial x}$$

• 标量函数的梯度向量

  $${\mathrm{\nabla }}_{x}f(x)=[\frac{\partial f(x)}{\partial x_1},...,\frac{\partial f(x)}{\partial x_m}{]}^T$$

• 矩阵梯度向量

  $${\mathrm{\nabla }}_{vecX}f(X)=\frac{\partial f(X)}{{\partial }_{vecX}}=[\frac{\partial f(X)}{\partial x_{11}}...\frac{\partial f(X)}{\partial x_{m1}}...\frac{\partial f(X)}{\partial x_{1n}}...\frac{\partial f(X)}{\partial x_{mn}}{]}^T$$
• 梯度矩阵$${\mathrm{\nabla }}_{x}f(x)=\frac{\partial f(X)}{\partial X^T}=\left[\begin{array}{ccc}\frac{\partial f(X)}{\partial x_{11}}& ...& \frac{\partial f(X)}{\partial x_{1n}}\\ ...& ...& ...\\ \frac{\partial f(X)}{\partial x_{m1}}& ...& \frac{\partial f(X)}{\partial x_{mn}}\end{array}\right]$$

实标量函数和矩阵函数的梯度矩阵是雅克比矩阵的转置

偏导和梯度计算

• 若$F(X)=c$为常数，其中X为$m\times n$矩阵，则梯度$\frac{\partial c}{\partial X}=O_{m\times n}$
• 线性法则
• 乘积法则
• 商法则
• 链式法则

独立性基本假设

假定实值函数的向量变元和矩阵变元无任何特殊结构则

$$\frac{\partial x_i}{\partial x_j}=\{\begin{array}{ll}1& i=j\\ 0& other\end{array}$$$$\frac{\partial x_{kl}}{\partial x_{ij}}=\{\begin{array}{ll}1& k=i,l=j\\ 0& other\end{array}$$

example

求实值函数$f(x)=x^{T}Ax$的Jacobian矩阵，$x^TAx=\sum{k=1}^n\sum{l=1}^na_{kl}x_kx_l$，\mathrm{我}\mathrm{们}\mathrm{可}\mathrm{以}\mathrm{求}\mathrm{出}\mathrm{行}\mathrm{偏}\mathrm{导}\mathrm{向}\mathrm{量}$\frac{\partial x^TAx}{\partial x^T}$的第i个分量为

$$\frac{\partial x^{T}Ax}{\partial x^T}=\frac{\partial }{\partial x_i}\sum _{k=1}^n\sum _{l=1}^n{a}_{kl}{x}_k{x}_l=\sum _{k=1}^n{x}_k{a}_{ki}+\sum _{l=1}^n{x}_l{a}_{il}$$

我可以得到行偏导向量和梯度向量

$$Df(x)=x^{T}A+x^T{A}^T=x^T(A+A^T)$$$${\mathrm{\nabla }}_{X}f(x)=(A^T+A)x$$

一阶实矩阵微分

• 标量函数tr(U)的微分$$d[tr(U)]=d(\sum _{i=1}^n{u}_{ii})=\sum _{i=1}^{n}d{u}_{ii}=tr(dU)$$
• 矩阵乘积UV的微分矩阵$$d(UV)=(dU)V+U(dV)$$
• 矩阵的迹的矩阵微分等于矩阵微分的迹$$d(tr(X))=tr(dX)$$

标量函数f（x）的jacobian矩阵辨识

• 以向量为变元的标量函数f(x)的微分$$df(x)=\frac{\partial f(x)}{\partial x_1}d{x}_1+...+\frac{\partial f(x)}{\partial x_m}d{x}_m$$$$=\frac{\partial f(x)}{\partial x^T}dx=(dx{)}^T\frac{\partial f(x)}{\partial x}$$如果令$$A=\frac{\partial f(x)}{\partial x^T}$$我们可以把一阶微分写成迹的形式$$df(x)=tr(Adx)$$$$D_{x}f(x)=\frac{\partial f(x)}{\partial x^T}=A$$

标量函数f（X）的jacobian矩阵辨识

$$df(X)=(vec(A^T){)}^{T}d(vecX)$$

A为标量函数f（X）的Jacobin矩阵

$$df(x)=tr(Adx)\to D_{x}f(x)=\frac{\partial f(x)}{\partial x^T}=A$$$$df(X)=tr(AdX)\to D_{X}f(X)=\frac{\partial f(X)}{\partial X^T}=A$$$$D_{X}f(X)=\frac{\partial f(X)}{\partial X^T}=A\to {\mathrm{\nabla }}_{X}f(X)=A^T$$

Hessian 矩阵

$$H[f(x)]=\frac{{\partial }^{2}f(x)}{\partial x\partial x^T}$$$$=\left[\begin{array}{ccc}\frac{{\partial }^{2}f}{\partial x_1\partial x_1}& ...& \frac{\partial f^2}{\partial x_1\partial x_m}\\ ...& ...& ...\\ \frac{{\partial }^{2}f}{\partial x_m\partial x_1}& ...& \frac{{\partial }^{2}f}{\partial x_m\partial x_m}\end{array}\right]\in R^{m\times m}$$$$H[f(x)]={\mathrm{\nabla }}_x^{2}f(x)={\mathrm{\nabla }}_x(D_{x}f(x))$$$$[Hf(x){]}_{i,j}=[\frac{{\partial }^{2}f(x)}{\partial x\partial x^T}]=\frac{\partial }{\partial x_i}[\frac{\partial f(x)}{\partial x_j}]$$

共轭梯度和复Hessian矩阵

形式偏导

$$\frac{\partial }{\partial z}=\frac{1}{2}>(\frac{\partial }{\partial x}-j\frac{\partial }{\partial y})$$

实部和虚部相互独立

单个复变量梯度

$${\mathrm{\nabla }}_{z}f(z,z^{\ast })=\frac{\partial f(z,z^{\ast })}{\partial }{|}_{z^{\ast }=C}$$

单个复变量微分

$$df(z,z^{\ast })=\frac{\partial f(z,z^{\ast }}{\partial z}dz+\frac{\partial f(z,z^{\ast })}{\partial z^{\ast }}d{z}^{\ast }$$

复变元向量违法

$$df(z,z^{\ast })=\frac{\partial f(z,z^{\ast })}{\partial z^T}dz+\frac{\partial (z,z^{\ast })}{\partial z^H}d{z}^{\ast }$$

标量函数的梯度向量和共轭梯度向量

$${\mathrm{\nabla }}_{z}f(z,z^{\ast })=(D_{z}f(z,z^{\ast }){)}^T$$

常见矩阵偏导

$$\frac{\partial a^{T}x}{\partial x}=a$$$$\frac{\partial x^{T}a}{\partial x}=a$$$$\frac{\partial x^{T}Ax}{\partial x}=(A+A^T)x$$$$\frac{\partial x^{T}Ax}{\partial x^T}=x^T(A+A^T)$$$$\frac{\partial x^{T}A}{\partial x}=A$$$$\frac{\partial (a-Ax{)}^{T}B(a-Ax)}{\partial x}=-2A^{T}B(a-Ax)$$