作业19

提交截至时间：暂定 2022/06/03 周五 20:00（晚上）

理论部分

习题 1. (互信息) 假设 \(X _ { 1 } \to X _ { 2 } \to X _ { 3 } \to \cdot \cdot \cdot \to X _ { n }\) 是一个马尔科夫链，即

\[ p \left(x _ {1}, x _ {2}, \dots , x _ {n}\right) = p \left(x _ {1}\right) p \left(x _ {2} \mid x _ {1}\right) \dots p \left(x _ {n} \mid x _ {n - 1}\right) \]

试化简 \(I \left( X _ { 1 } ; X _ { 2 } , \ldots , X _ { n } \right)\)

解.

\[ \begin{array}{l} I \left(X _ {1}; X _ {2}, \dots , X _ {n}\right) = H \left(X _ {1}\right) - H \left(X _ {1} \mid X _ {2}, \dots , X _ {n}\right) \\ = H \left(X _ {1}\right) - \left[ H \left(X _ {1}, X _ {2}, \dots , X _ {n}\right) - H \left(X _ {2}, \dots , X _ {n}\right) \right] \\ = H \left(X _ {1}\right) - \left[ \sum_ {i = 1} ^ {n} H \left(X _ {i} \mid X _ {i - 1}, \dots , X _ {1}\right) - \sum_ {i = 2} ^ {n} H \left(X _ {i} \mid X _ {i - 1}, \dots , X _ {2}\right) \right] \\ = H \left(X _ {1}\right) - \left[ \left(H \left(X _ {1}\right) + \sum_ {i = 2} ^ {n} H \left(X _ {i} \mid X _ {i - 1}\right)\right) - \left(H \left(X _ {2}\right) + \sum_ {i = 3} ^ {n} H \left(X _ {i} \mid X _ {i - 1}\right)\right) \right] \\ = H \left(X _ {2}\right) - H \left(X _ {2} \mid X _ {1}\right) \\ = I \left(X _ {2}; X _ {1}\right) \\ = I \left(X _ {1}; X _ {2}\right) \\ \end{array} \]

习题 2. (通过 \(K L\) 散度理解 MLE) 假设 \(\mathbf { x } _ { 1 } , \ldots , \mathbf { x } _ { n }\) 来自密度为 \(p ( \mathbf { x } )\) 的分布 \(P\) ，试说明如果采用具有密度函数 \(q _ { \theta } ( \mathbf { x } )\) 的分布族 \(Q _ { \theta }\) 来计算 \(M L E\) ，那么 MLE 将试图找到在 \(K L\) 散度意义上最接近真实分布 \(P\) 的分布 \(Q _ { \theta }\) 。

即证明

\[ \arg \max _ {\theta} \prod_ {i = 1} ^ {n} q _ {\theta} \left(\mathbf {x} _ {i}\right) \Longleftrightarrow \arg \min _ {\theta} D _ {\mathrm {KL}} (P \| Q _ {\theta}) \]

解.

\[ \begin{array}{l} \arg \max _ {\theta} \prod_ {i = 1} ^ {n} q _ {\theta} \left(\mathbf {x} _ {i}\right) \Longleftrightarrow \arg \min _ {\theta} - \frac {1}{n} \sum_ {i = 1} ^ {n} \log q _ {\theta} \left(\mathbf {x} _ {i}\right) \\ \xrightarrow {P} \arg \min _ {\theta} - E _ {P} \log q _ {\theta} (\mathbf {x}) \Longleftrightarrow \arg \min _ {\theta} - \int p (\mathbf {x}) \log q _ {\theta} (\mathbf {x}) d \mathbf {x} \\ \Longleftrightarrow \arg \min _ {\theta} H (P, Q _ {\theta}) \Longleftrightarrow \arg \min _ {\theta} (H (P, Q _ {\theta}) - H (P)) \\ \Longleftrightarrow \arg \min _ {\theta} \left\{- \int p (\mathbf {x}) \log q _ {\theta} (\mathbf {x}) d \mathbf {x} + \int p (\mathbf {x}) \log p (\mathbf {x}) d \mathbf {x} \right\} \\ \Longleftrightarrow \arg \min _ {\theta} \left\{- \int p (\mathbf {x}) \log \frac {q _ {\theta} (\mathbf {x})}{p (\mathbf {x})} \right\} \Longleftrightarrow \arg \min _ {\theta} D _ {\mathrm {KL}} (P \| Q _ {\theta}) \\ \end{array} \]

其实，从优化模型参数角度来说，最小化负对数似然，交叉熵(多分类问题中)，KL散度这3种方式是一样的。