通过微扰分析求解等式约束优化问题

对于无约束的光滑函数来说，求极值的通用方法自然是求导（梯度）并让导数等于零了。那带等式约束的极值问题呢？很多读者应该也不陌生，标准方法是拉格朗日乘子法。然而，拉格朗日乘子总让人觉得是一种纯粹的代数技巧，没有求无约束极值时那种微扰分析的直观感觉。

这篇文章我们就来尝试一下：如果将无约束极值的微扰思想贯彻到约束优化中，会得到什么结果？

驻点条件[#](https://spaces.ac.cn/archives/11928#%E9%A9%BB%E7%82%B9%E6%9D%A1%E4%BB%B6)

首先来重温一下，为什么梯度等于零就是无约束极值的必要条件（驻点条件）。考虑目标函数$f \left(\right. \mathbf{\mathit{x}} \left.\right)$，其中$\mathbf{\mathit{x}} \in \mathbb{R}^{d}$，我们想要求$f \left(\right. \mathbf{\mathit{x}} \left.\right)$的最小值点，我们在$\mathbf{\mathit{x}}$附近做一阶近似

$$ (\text{1}) f \left(\right. \mathbf{\mathit{x}} + \delta \mathbf{\mathit{x}} \left.\right) = f \left(\right. \mathbf{\mathit{x}} \left.\right) + \delta \mathbf{\mathit{x}}^{\top} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) $$

如果$f \left(\right. \mathbf{\mathit{x}} \left.\right)$已经是最小值，那么对于任意微扰$\delta \mathbf{\mathit{x}} \neq 0$，必然恒成立$f \left(\right. \mathbf{\mathit{x}} + \delta \mathbf{\mathit{x}} \left.\right) \geq f \left(\right. \mathbf{\mathit{x}} \left.\right)$，这意味着$\delta \mathbf{\mathit{x}}^{\top} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) \geq 0$，但$\delta \mathbf{\mathit{x}}^{\top} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right)$是关于$\delta \mathbf{\mathit{x}}$的线性函数，它要想始终非负，那么只有

$$ (\text{2}) \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) = 0 $$

这一选择。同样的讨论对最大值也成立，所以这是极大值、极小值共同的驻点条件。而当驻点条件成立时，近似展开的下一主导项是

$$ (\text{3}) f \left(\right. \mathbf{\mathit{x}} + \delta \mathbf{\mathit{x}} \left.\right) = f \left(\right. \mathbf{\mathit{x}} \left.\right) + \frac{1}{2} \delta \mathbf{\mathit{x}}^{\top} \nabla_{\mathbf{\mathit{x}}}^{2} f \left(\right. \mathbf{\mathit{x}} \left.\right) \delta \mathbf{\mathit{x}} $$

根据同样的原理，要想$f \left(\right. \mathbf{\mathit{x}} + \delta \mathbf{\mathit{x}} \left.\right) \geq f \left(\right. \mathbf{\mathit{x}} \left.\right)$恒成立，那么$\delta \mathbf{\mathit{x}}^{\top} \nabla_{\mathbf{\mathit{x}}}^{2} f \left(\right. \mathbf{\mathit{x}} \left.\right) \delta \mathbf{\mathit{x}} \geq 0$恒成立，即Hessian矩阵$\nabla_{\mathbf{\mathit{x}}}^{2} f \left(\right. \mathbf{\mathit{x}} \left.\right)$半正定，这是最小值点的必要条件；反之，最大值点的必要条件则是$\nabla_{\mathbf{\mathit{x}}}^{2} f \left(\right. \mathbf{\mathit{x}} \left.\right)$半负定。 当然这个讨论并不是严格的数学证明，只能算是一个启发式推导。

等式约束[#](https://spaces.ac.cn/archives/11928#%E7%AD%89%E5%BC%8F%E7%BA%A6%E6%9D%9F)

现在再来看，当引入等式约束后，微扰过程将会发生什么变化。考虑

$$ (\text{4}) \underset{\mathbf{\mathit{x}}}{min} f \left(\right. \mathbf{\mathit{x}} \left.\right) \text{s}.\text{t}. \mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} \left.\right) = 0 $$

其中$\mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} \left.\right) \in \mathbb{R}^{k}$，代表有$k$个等式约束。任取一个满足$\mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} \left.\right) = 0$的$\mathbf{\mathit{x}}$，同样考虑微扰的一阶近似得

$$ (\text{5}) f \left(\right. \mathbf{\mathit{x}} + \delta \mathbf{\mathit{x}} \left.\right) = f \left(\right. \mathbf{\mathit{x}} \left.\right) + \delta \mathbf{\mathit{x}}^{\top} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) $$

类似地，$f \left(\right. \mathbf{\mathit{x}} \left.\right)$是最小值的必要条件依然是$\delta \mathbf{\mathit{x}}^{\top} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) \geq 0$恒成立，但区别在于，此时不需要对任意的微扰$\delta \mathbf{\mathit{x}}$都恒成立，而只需要对所有满足$\mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} + \delta \mathbf{\mathit{x}} \left.\right) = 0$的$\delta \mathbf{\mathit{x}}$恒成立。对$\mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} + \delta \mathbf{\mathit{x}} \left.\right) = 0$做一阶近似，得

$$ (\text{6}) \mathbf{\mathit{J}} \delta \mathbf{\mathit{x}} = 0 , \mathbf{\mathit{J}} = \nabla_{\mathbf{\mathit{x}}} \mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} \left.\right) \in \mathbb{R}^{k \times d} $$

注意到$\mathbf{\mathit{J}} \delta \mathbf{\mathit{x}} = 0$是关于$\delta \mathbf{\mathit{x}}$的线性方程组，可以用“伪逆”写出它的通解：

$$ (\text{7}) \delta \mathbf{\mathit{x}} = \left(\right. \mathbf{\mathit{I}} - \mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} \left.\right) \delta \mathbf{\mathit{z}} $$

其中$\mathbf{\mathit{J}}^{\dagger}$是$\mathbf{\mathit{J}}$的伪逆，$\delta \mathbf{\mathit{z}}$表示任意无约束的微扰向量，当$\mathbf{\mathit{J}} \mathbf{\mathit{J}}^{\top}$可逆时$\mathbf{\mathit{J}}^{\dagger} = \mathbf{\mathit{J}}^{\top} \left(\right. \mathbf{\mathit{J}} \mathbf{\mathit{J}}^{\top} \left.\right)^{- 1}$，由此可见$\mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}}$是对称矩阵。将上式代入$\delta \mathbf{\mathit{x}}^{\top} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) \geq 0$并由$\delta \mathbf{\mathit{z}}$的任意性得驻点条件

$$ (\text{8}) \left(\right. \mathbf{\mathit{I}} - \mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} \left.\right) \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) = 0 $$

其中$\left(\right. \mathbf{\mathit{I}} - \mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} \left.\right) \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right)$称为$f \left(\right. \mathbf{\mathit{x}} \left.\right)$在流形$\mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} \left.\right) = 0$上的投影梯度，它是欧氏梯度$\nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right)$在切空间上的正交投影，在黎曼几何的语言中，它又叫黎曼梯度。

拉格朗日[#](https://spaces.ac.cn/archives/11928#%E6%8B%89%E6%A0%BC%E6%9C%97%E6%97%A5)

现在我们将它与拉格朗日乘子法做个对比。拉格朗日乘子法通过引入待定系数$\mathbf{\mathit{c}} \in \mathbb{R}^{k}$，将目标函数改为

$$ (\text{9}) F \left(\right. \mathbf{\mathit{x}} , \mathbf{\mathit{c}} \left.\right) = f \left(\right. \mathbf{\mathit{x}} \left.\right) - \mathbf{\mathit{c}}^{\top} \mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} \left.\right) $$

然后改为求无约束目标$F \left(\right. \mathbf{\mathit{x}} , \mathbf{\mathit{c}} \left.\right)$的驻点，那么直接令两边梯度等于零得

$$ (\text{10}) \nabla_{\mathbf{\mathit{x}}} F \left(\right. \mathbf{\mathit{x}} , \mathbf{\mathit{c}} \left.\right) = \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) - \mathbf{\mathit{J}}^{\top} \mathbf{\mathit{c}} = 0 , \nabla_{\mathbf{\mathit{c}}} F \left(\right. \mathbf{\mathit{x}} , \mathbf{\mathit{c}} \left.\right) = - \mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} \left.\right) = 0 $$

注意伪逆成立$\mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} = \mathbf{\mathit{J}}^{\top} \left(\right. \mathbf{\mathit{J}}^{\top} \left.\right)^{\dagger}$，所以式$(\text{8})$又可以写成$\nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) - \mathbf{\mathit{J}}^{\top} \left(\right. \mathbf{\mathit{J}}^{\top} \left.\right)^{\dagger} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) = 0$，如果记$\left(\right. \mathbf{\mathit{J}}^{\top} \left.\right)^{\dagger} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right)$为$\mathbf{\mathit{c}}$，那么就跟上式一模一样了，即从式$(\text{8})$可以推出跟拉格朗日乘子法一样的条件。 那么反过来，从拉格朗日乘子法能否推出式$(\text{8})$呢？自然也可以。假设$k < d$且$\mathbf{\mathit{J}}$的秩为$k$，那么将$\nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) - \mathbf{\mathit{J}}^{\top} \mathbf{\mathit{c}} = 0$看作关于$\mathbf{\mathit{c}}$的方程，它是一个超定方程，如果有解只能是$\mathbf{\mathit{c}} = \left(\right. \mathbf{\mathit{J}}^{\top} \left.\right)^{\dagger} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right)$，代回方程后便得到式$(\text{8})$。

这表明两者给出的条件在数学上是完全等价的。当然，拉格朗日乘子法的写法看起来会更干净一些，但笔者认为微扰思想上与无约束情形更为一致，所以更偏爱这种理解。

此外，拉格朗日乘子法的简单只是形式上的，要想真正理解它，并不比微扰视角容易。读者完全可以将从微扰思想到投影梯度的过程，看作是拉格朗日乘子法的一种推导。

两个例子[#](https://spaces.ac.cn/archives/11928#%E4%B8%A4%E4%B8%AA%E4%BE%8B%E5%AD%90)

我们通过以下两个例子来加深理解。

球面约束[#](https://spaces.ac.cn/archives/11928#%E7%90%83%E9%9D%A2%E7%BA%A6%E6%9D%9F)

第一个例子，考虑简单的球面极值

$$ (\text{11}) \underset{\mathbf{\mathit{x}}}{min} f \left(\right. \mathbf{\mathit{x}} \left.\right) \text{s}.\text{t}. \parallel \mathbf{\mathit{x}} \parallel_{2} = 1 $$

容易求得$\nabla_{\mathbf{\mathit{x}}} \parallel \mathbf{\mathit{x}} \parallel_{2} = \mathbf{\mathit{x}} / \parallel \mathbf{\mathit{x}} \parallel_{2} = \mathbf{\mathit{x}}$，所以$\mathbf{\mathit{J}} = \mathbf{\mathit{x}}^{\top}$，由于$\mathbf{\mathit{x}}$是单位向量，所以$\mathbf{\mathit{J}}^{\dagger} = \mathbf{\mathit{x}}$，因此驻点条件为

$$ (\text{12}) 0 = \left(\right. \mathbf{\mathit{I}} - \mathbf{\mathit{x}} \mathbf{\mathit{x}}^{\top} \left.\right) \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) = \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) - \mathbf{\mathit{x}} \left(\right. \mathbf{\mathit{x}}^{\top} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) \left.\right) $$

最后的式子，正是梯度在球面切空间上的投影。

正交约束[#](https://spaces.ac.cn/archives/11928#%E6%AD%A3%E4%BA%A4%E7%BA%A6%E6%9D%9F)

另一个例子是Stiefel流形上的优化问题

$$ (\text{13}) \underset{\mathbf{\mathit{X}}}{min} f \left(\right. \mathbf{\mathit{X}} \left.\right) \text{s}.\text{t}. \mathbf{\mathit{X}}^{\top} \mathbf{\mathit{X}} = \mathbf{\mathit{I}}_{m} , \mathbf{\mathit{X}} \in \mathbb{R}^{n \times m} \left(\right. n \geq m \left.\right) $$

如果只是想求驻点条件，用拉格朗日乘子法做会快不少，但这里还是完整地把投影梯度求出来。这里的困难在于$\mathbf{\mathit{X}}$本身就是一个矩阵，那么$\mathbf{\mathit{J}}$将会是一个高阶矩阵，思考的心智负担较大。 对此，我们采用线性算子视角，对$\mathbf{\mathit{X}}^{\top} \mathbf{\mathit{X}} - \mathbf{\mathit{I}}_{m}$做一阶近似得$\delta \mathbf{\mathit{X}}^{\top} \mathbf{\mathit{X}} + \mathbf{\mathit{X}}^{\top} \delta \mathbf{\mathit{X}}$，按照前面的推导，它应该等于$\mathbf{\mathit{J}} \delta \mathbf{\mathit{x}}$，我们直接定义线性算子

$$ (\text{14}) \mathbf{\mathit{J}} \left[\right. \mathbf{\mathit{V}} \left]\right. = \mathbf{\mathit{V}}^{\top} \mathbf{\mathit{X}} + \mathbf{\mathit{X}}^{\top} \mathbf{\mathit{V}} $$

下面我们需要推导$\mathbf{\mathit{J}}^{\top} \left[\right. \mathbf{\mathit{V}} \left]\right.$，它按照$\langle \mathbf{\mathit{U}} , \mathbf{\mathit{J}}^{\top} \left[\right. \mathbf{\mathit{V}} \left]\right. \rangle_{F} = \langle \mathbf{\mathit{V}} , \mathbf{\mathit{J}} \left[\right. \mathbf{\mathit{U}} \left]\right. \rangle_{F}$来理解（类比$\mathbf{\mathit{u}}^{\top} \mathbf{\mathit{J}}^{\top} \mathbf{\mathit{v}} = \mathbf{\mathit{v}}^{\top} \mathbf{\mathit{J}} \mathbf{\mathit{u}}$），$\mathbf{\mathit{J}} \left[\right. \mathbf{\mathit{U}} \left]\right.$和$\langle \cdot , \cdot \rangle_{F}$我们都是已知的，所以有

$$ (\text{15}) \langle \mathbf{\mathit{U}} , \mathbf{\mathit{J}}^{\top} \left[\right. \mathbf{\mathit{V}} \left]\right. \rangle_{F} = \text{tr} ⁡ \left(\right. \mathbf{\mathit{V}}^{\top} \left(\right. \mathbf{\mathit{U}}^{\top} \mathbf{\mathit{X}} + \mathbf{\mathit{X}}^{\top} \mathbf{\mathit{U}} \left.\right) \left.\right) = \text{tr} ⁡ \left(\right. \mathbf{\mathit{U}}^{\top} \mathbf{\mathit{X}} \left(\right. \mathbf{\mathit{V}} + \mathbf{\mathit{V}}^{\top} \left.\right) \left.\right) = \langle \mathbf{\mathit{U}} , \mathbf{\mathit{X}} \left(\right. \mathbf{\mathit{V}} + \mathbf{\mathit{V}}^{\top} \left.\right) \rangle_{F} $$

由此可读出$\mathbf{\mathit{J}}^{\top} \left[\right. \mathbf{\mathit{V}} \left]\right. = \mathbf{\mathit{X}} \left(\right. \mathbf{\mathit{V}} + \mathbf{\mathit{V}}^{\top} \left.\right)$，根据$\mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} = \mathbf{\mathit{J}}^{\top} \left(\right. \mathbf{\mathit{J}} \mathbf{\mathit{J}}^{\top} \left.\right)^{- 1} \mathbf{\mathit{J}}$，我们还要求$\mathbf{\mathit{J}} \mathbf{\mathit{J}}^{\top}$和$\left(\right. \mathbf{\mathit{J}} \mathbf{\mathit{J}}^{\top} \left.\right)^{- 1}$：

$$ (\text{16}) \mathbf{\mathit{J}} \mathbf{\mathit{J}}^{\top} \left[\right. \mathbf{\mathit{V}} \left]\right. = \mathbf{\mathit{J}} \left[\right. \mathbf{\mathit{X}} \left(\right. \mathbf{\mathit{V}} + \mathbf{\mathit{V}}^{\top} \left.\right) \left]\right. = \left(\right. \mathbf{\mathit{V}} + \mathbf{\mathit{V}}^{\top} \left.\right) \mathbf{\mathit{X}}^{\top} \mathbf{\mathit{X}} + \mathbf{\mathit{X}}^{\top} \mathbf{\mathit{X}} \left(\right. \mathbf{\mathit{V}} + \mathbf{\mathit{V}}^{\top} \left.\right) = 2 \left(\right. \mathbf{\mathit{V}} + \mathbf{\mathit{V}}^{\top} \left.\right) $$

如果$\mathbf{\mathit{V}}$是对称矩阵，那么$\mathbf{\mathit{J}} \mathbf{\mathit{J}}^{\top} \left[\right. \mathbf{\mathit{V}} \left]\right. = 4 \mathbf{\mathit{V}}$，所以$\left(\right. \mathbf{\mathit{J}} \mathbf{\mathit{J}}^{\top} \left.\right)^{- 1} \left[\right. \mathbf{\mathit{V}} \left]\right. = \mathbf{\mathit{V}} / 4$，而$\mathbf{\mathit{J}} \left[\right. \mathbf{\mathit{V}} \left]\right.$刚好是对称的，所以我们只需要这个特殊情况。综合起来有

$$ (\text{17}) \mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} \left[\right. \mathbf{\mathit{V}} \left]\right. = \mathbf{\mathit{J}}^{\top} \left(\right. \mathbf{\mathit{J}} \mathbf{\mathit{J}}^{\top} \left.\right)^{- 1} \mathbf{\mathit{J}} \left[\right. \mathbf{\mathit{V}} \left]\right. = \mathbf{\mathit{J}}^{\top} \left[\right. \left(\right. \mathbf{\mathit{V}}^{\top} \mathbf{\mathit{X}} + \mathbf{\mathit{X}}^{\top} \mathbf{\mathit{V}} \left.\right) / 4 \left]\right. = \mathbf{\mathit{X}} \left(\right. \mathbf{\mathit{V}}^{\top} \mathbf{\mathit{X}} + \mathbf{\mathit{X}}^{\top} \mathbf{\mathit{V}} \left.\right) / 2 $$

记$\mathbf{\mathit{G}} = \nabla_{\mathbf{\mathit{X}}} f \left(\right. \mathbf{\mathit{X}} \left.\right)$，那么它在Stiefel流形上的投影梯度为

$$ (\text{18}) \mathbf{\mathit{G}} - \mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} \left[\right. \mathbf{\mathit{G}} \left]\right. = \mathbf{\mathit{G}} - \mathbf{\mathit{X}} \left[\right. \mathbf{\mathit{X}}^{\top} \mathbf{\mathit{G}} \left]\right._{\text{sym}} $$

其中$\left[\right. \mathbf{\mathit{Z}} \left]\right._{\text{sym}} = \left(\right. \mathbf{\mathit{Z}} + \mathbf{\mathit{Z}}^{\top} \left.\right) / 2$为对称化算子。令它等于零，然后两边左乘$\mathbf{\mathit{X}}^{\top}$得$\mathbf{\mathit{X}}^{\top} \mathbf{\mathit{G}} = \left[\right. \mathbf{\mathit{X}}^{\top} \mathbf{\mathit{G}} \left]\right._{\text{sym}}$，这也就是说，驻点条件为$\mathbf{\mathit{X}}^{\top} \mathbf{\mathit{G}}$本身就是对称矩阵！

二阶条件[#](https://spaces.ac.cn/archives/11928#%E4%BA%8C%E9%98%B6%E6%9D%A1%E4%BB%B6)

那等式约束优化之下，有没有类似$\nabla_{\mathbf{\mathit{x}}}^{2} f \left(\right. \mathbf{\mathit{x}} \left.\right)$正/负定的最小/大值判断方法呢？自然也是有的，而且有一个很巧妙的推导。跟拉格朗日乘子法一样，我们定义

$$ (\text{19}) F \left(\right. \mathbf{\mathit{x}} , \mathbf{\mathit{c}} \left.\right) = f \left(\right. \mathbf{\mathit{x}} \left.\right) - \mathbf{\mathit{c}}^{\top} \mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} \left.\right) $$

对于任意驻点$\mathbf{\mathit{x}}$，自动有$\mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} \left.\right) = 0$，所以对于任意$\mathbf{\mathit{c}} \in \mathbb{R}^{k}$都有$F \left(\right. \mathbf{\mathit{x}} , \mathbf{\mathit{c}} \left.\right) \equiv f \left(\right. \mathbf{\mathit{x}} \left.\right)$，于是展开到二阶有

$$ (\text{20}) f \left(\right. \mathbf{\mathit{x}} + \delta \mathbf{\mathit{x}} \left.\right) = F \left(\right. \mathbf{\mathit{x}} + \delta \mathbf{\mathit{x}} , \mathbf{\mathit{c}} \left.\right) = \underset{\equiv f \left(\right. \mathbf{\mathit{x}} \left.\right)}{\underbrace{F \left(\right. \mathbf{\mathit{x}} , \mathbf{\mathit{c}} \left.\right)}} + \delta \mathbf{\mathit{x}}^{\top} \left(\right. \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) - \mathbf{\mathit{J}}^{\top} \mathbf{\mathit{c}} \left.\right) + \frac{1}{2} \delta \mathbf{\mathit{x}}^{\top} \mathbf{\mathit{H}} \left(\right. \mathbf{\mathit{c}} \left.\right) \delta \mathbf{\mathit{x}} $$

这里$\mathbf{\mathit{H}} \left[\right. \mathbf{\mathit{c}} \left]\right. \triangleq \nabla_{\mathbf{\mathit{x}}}^{2} \left(\right. f \left(\right. \mathbf{\mathit{x}} \left.\right) - \mathbf{\mathit{c}}^{\top} \mathbf{\mathit{g}} \left(\right. \mathbf{\mathit{x}} \left.\right) \left.\right)$。由于$\mathbf{\mathit{c}}$的任意性，我们可以直接取$\mathbf{\mathit{c}} = \left(\right. \mathbf{\mathit{J}}^{\top} \left.\right)^{\dagger} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right)$，使得一阶项为零，于是

$$ (\text{21}) f \left(\right. \mathbf{\mathit{x}} + \delta \mathbf{\mathit{x}} \left.\right) = f \left(\right. \mathbf{\mathit{x}} \left.\right) + \frac{1}{2} \delta \mathbf{\mathit{x}}^{\top} \mathbf{\mathit{H}} \left[\right. \left(\right. \mathbf{\mathit{J}}^{\top} \left.\right)^{\dagger} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) \left]\right. \delta \mathbf{\mathit{x}} $$

现在只剩二阶项，所以$\delta \mathbf{\mathit{x}}$的近似精度只需一阶，即$\delta \mathbf{\mathit{x}} = \left(\right. \mathbf{\mathit{I}} - \mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} \left.\right) \delta \mathbf{\mathit{z}}$足矣，代入得

$$ (\text{22}) f \left(\right. \mathbf{\mathit{x}} + \delta \mathbf{\mathit{x}} \left.\right) = f \left(\right. \mathbf{\mathit{x}} \left.\right) + \frac{1}{2} \delta \mathbf{\mathit{z}}^{\top} \left(\right. \mathbf{\mathit{I}} - \mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} \left.\right) \mathbf{\mathit{H}} \left[\right. \left(\right. \mathbf{\mathit{J}}^{\top} \left.\right)^{\dagger} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) \left]\right. \left(\right. \mathbf{\mathit{I}} - \mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} \left.\right) \delta \mathbf{\mathit{z}} $$

由$\delta \mathbf{\mathit{z}}$的任意性知，$f \left(\right. \mathbf{\mathit{x}} \left.\right)$为最小值的必要条件是$\left(\right. \mathbf{\mathit{I}} - \mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} \left.\right) \mathbf{\mathit{H}} \left[\right. \left(\right. \mathbf{\mathit{J}}^{\top} \left.\right)^{\dagger} \nabla_{\mathbf{\mathit{x}}} f \left(\right. \mathbf{\mathit{x}} \left.\right) \left]\right. \left(\right. \mathbf{\mathit{I}} - \mathbf{\mathit{J}}^{\dagger} \mathbf{\mathit{J}} \left.\right)$半正定。同理，最大值则为该矩阵半负定。 当然，不管是无约束还是有约束优化，通过Hessian矩阵的正负定来判断最大/最小，通常只具有理论价值，实践中通常都是筛出所有符合驻点条件的位置，然后逐一具体情况具体分析。

文章小结[#](https://spaces.ac.cn/archives/11928#%E6%96%87%E7%AB%A0%E5%B0%8F%E7%BB%93)

本文将无约束极值的微扰分析思想用于等式约束优化中，推导了切空间上的投影梯度，而极值的驻点条件便是让相应的投影梯度为零，最后我们还证明了它与拉格朗日乘子法在结论上的等价性。

_转载到请包括本文地址：[https://spaces.ac.cn/archives/11928](https://spaces.ac.cn/archives/11928 "通过微扰分析求解等式约束优化问题")_

_更详细的转载事宜请参考：_[《科学空间FAQ》](https://spaces.ac.cn/archives/6508#%E6%96%87%E7%AB%A0%E5%A6%82%E4%BD%95%E8%BD%AC%E8%BD%BD/%E5%BC%95%E7%94%A8 "《科学空间FAQ》")