aImplicit and Explicit Gradients Let f be a function of z,x such that z=cos(x)${\displaystyle z=cos(x)}$ and f(z,x) = z*x2${\displaystyle f(z,x)=z*x^2}$. How do we calculate ∂f(z,x)

∂x=?${\displaystyle \frac{\partial f(z,x)}{\partial x}=?}$. One simple way is to subsitute z=cos(x) in f${\displaystyle z=cos(x)inf}$ and then applying chain rule. Another way is to use explicit and implicit gradient. This is used when a function parameters are itself a function of another parameter. (1)∂f(z,x)

∂x = ∂+f(z,x)

∂x+∂f(z,x)

∂z∂z

∂x where ∂+f(z,x)

∂x is called implicit gradientand ∂f(z,x)

∂z∂z

∂x is called explicit gradient$$\begin{array}{c}\frac{\partial f(z,x)}{\partial x}=\frac{{\partial }^{+}f(z,x)}{\partial x}+\frac{\partial f(z,x)}{\partial z}\frac{\partial z}{\partial x}\\ \\ where\frac{{\partial }^{+}f(z,x)}{\partial x}iscalled\mathbf{i}\mathbf{m}\mathbf{p}\mathbf{l}\mathbf{i}\mathbf{c}\mathbf{i}\mathbf{t}\mathbf{g}\mathbf{r}\mathbf{a}\mathbf{d}\mathbf{i}\mathbf{e}\mathbf{n}\mathbf{t}\\ and\frac{\partial f(z,x)}{\partial z}\frac{\partial z}{\partial x}iscalled\mathbf{e}\mathbf{x}\mathbf{p}\mathbf{l}\mathbf{i}\mathbf{c}\mathbf{i}\mathbf{t}\mathbf{g}\mathbf{r}\mathbf{a}\mathbf{d}\mathbf{i}\mathbf{e}\mathbf{n}\mathbf{t}\end{array}$$ To calculate implicit gradient, ∂+f(z,x)

∂x${\displaystyle \frac{{\partial }^{+}f(z,x)}{\partial x}}$ assume all other variables except x as constants.∂f(z,x)

∂x = ∂+f(z,x)

∂x+∂f(z,x)

∂z∂z

∂x∂+f(z,x)

∂x = cos(x)*2x∂f(z,x)

∂z = x2∂z

∂x = -sin(x) ∂f(z,x)

∂x = cos(x)*2x -x2*sin(x)$$\begin{array}{c}\frac{\partial f(z,x)}{\partial x}=\frac{{\partial }^{+}f(z,x)}{\partial x}+\frac{\partial f(z,x)}{\partial z}\frac{\partial z}{\partial x}\\ \frac{{\partial }^{+}f(z,x)}{\partial x}=cos(x)*2x\\ \frac{\partial f(z,x)}{\partial z}=x^2\\ \frac{\partial z}{\partial x}=-sin(x)\\ \\ \frac{\partial f(z,x)}{\partial x}=cos(x)*2x-x^2*sin(x)\end{array}$$ Applying Chain Rule to ∇𝛼Lval(w', α)${\displaystyle {\nabla }_{𝛼}{L}_{val}(w^{\text{'}},\alpha )}$, where w' = w − ξ∇w Ltrain(w, α)${\displaystyle w^{\text{'}}=w-\xi {\nabla }_w{L}_{train}(w,\alpha )}$? Observe w' is a function of 𝛼${\displaystyle 𝛼}$ itself. To apply chain rule we will use implicit and explicit gradients. Roughly speaking we can write this as ∂Lval(w',𝛼)

∂𝛼=∂L+val(w',𝛼)

∂𝛼+∂Lval(w',𝛼)

∂w'*∂w'

∂𝛼∂L+val(w',𝛼)

∂𝛼=∇𝛼L+val(w', α)∂Lval(w',𝛼)

∂w'=-ξ∇2𝛼w'Ltrain(w', α)∂w'

∂𝛼=∇w'Lval(w', α)∂Lval(w',𝛼)

∂𝛼=∇𝛼L+val(w', α)-ξ∇2𝛼w'Ltrain(w', α)∇w'Lval(w', α) $$\begin{array}{c}\frac{\partial L_{val}(w^{\text{'}},𝛼)}{\partial 𝛼}=\frac{\partial L_{val}^{+}(w^{\text{'}},𝛼)}{\partial 𝛼}+\frac{\partial L_{val}(w^{\text{'}},𝛼)}{\partial w^{\text{'}}}*\frac{\partial w^{\text{'}}}{\partial 𝛼}\\ \frac{\partial L_{val}^{+}(w^{\text{'}},𝛼)}{\partial 𝛼}={\nabla }_{𝛼}{L}_{val}^{+}(w^{\text{'}},\alpha )\\ \frac{\partial L_{val}(w^{\text{'}},𝛼)}{\partial w^{\text{'}}}=-\xi {\nabla }_{𝛼w\text{'}}^2{L}_{train}(w^{\text{'}},\alpha )\\ \frac{\partial w^{\text{'}}}{\partial 𝛼}={\nabla }_{w\text{'}}{L}_{val}(w^{\text{'}},\alpha )\\ \frac{\partial L_{val}(w^{\text{'}},𝛼)}{\partial 𝛼}={\nabla }_{𝛼}{L}_{val}^{+}(w^{\text{'}},\alpha )-\xi {\nabla }_{𝛼w\text{'}}^2{L}_{train}(w^{\text{'}},\alpha ){\nabla }_{w\text{'}}{L}_{val}(w^{\text{'}},\alpha )\\ \end{array}$$ Finite Difference Method (2)∂f(x,z)

∂x=lim𝜖→0f(x+𝜖,z)-f(x-𝜖,z)

2𝜖=lim𝜖→0f(x+k𝜖,z)-f(x-k𝜖,z)

2k𝜖 (3)∂f(x,z)

∂x*k=lim𝜖→0f(x+k𝜖,z)-f(x-k𝜖,z)

2𝜖 $$\begin{array}{c}\frac{\partial f(x,z)}{\partial x}=\underset{𝜖\to 0}{\overset{}{lim}}\frac{f(x+𝜖,z)-f(x-𝜖,z)}{2𝜖}\\ =\underset{𝜖\to 0}{\overset{}{lim}}\frac{f(x+k𝜖,z)-f(x-k𝜖,z)}{2k𝜖}\\ \\ \frac{\partial f(x,z)}{\partial x}*k=\underset{𝜖\to 0}{\overset{}{lim}}\frac{f(x+k𝜖,z)-f(x-k𝜖,z)}{2𝜖}\\ \end{array}$$ Using equation 2∇2𝛼w'Ltrainl(w', α)=∇𝛼(∇w'(Ltrain(w',𝛼))=∇w'(∇𝛼(Ltrain(w',𝛼))=lim𝜖→0∇𝛼(Ltrain(w'+𝜖,𝛼)-∇𝛼(Ltrain(w'-𝜖,𝛼)

2𝜖$$\begin{array}{c}{\nabla }_{𝛼w\text{'}}^2{L}_{trainl}(w^{\text{'}},\alpha )={\nabla }_{𝛼}({\nabla }_{w\text{'}}(L_{train}(w\text{'},𝛼))\\ ={\nabla }_{w\text{'}}({\nabla }_{𝛼}(L_{train}(w\text{'},𝛼))\\ =\underset{𝜖\to 0}{\overset{}{lim}}\frac{{\nabla }_{𝛼}(L_{train}(w\text{'}+𝜖,𝛼)-{\nabla }_{𝛼}(L_{train}(w\text{'}-𝜖,𝛼)}{2𝜖}\end{array}$$Using equation 3∇2𝛼w'Lval(w', α)=lim𝜖→0∇𝛼(Lva(w'+∇w'Lval(w', α)𝜖,𝛼)-∇𝛼(Lval(w'-∇w'Lval(w', α)𝜖,𝛼)

2𝜖=lim𝜖→0∇𝛼(Lval(w+,𝛼)-∇𝛼(Lval(w-,𝛼)

2𝜖where w+= w'+∇w'Lval(w', α)𝜖and w-=w'-∇w'Lval(w', α)𝜖$$\begin{array}{c}{\nabla }_{𝛼w\text{'}}^2{L}_{val}(w^{\text{'}},\alpha )=\underset{𝜖\to 0}{\overset{}{lim}}\frac{{\nabla }_{𝛼}(L_{va}(w\text{'}+{\nabla }_{w\text{'}}{L}_{val}(w^{\text{'}},\alpha )𝜖,𝛼)-{\nabla }_{𝛼}(L_{val}(w\text{'}-{\nabla }_{w\text{'}}{L}_{val}(w^{\text{'}},\alpha )𝜖,𝛼)}{2𝜖}\\ =\underset{𝜖\to 0}{\overset{}{lim}}\frac{{\nabla }_{𝛼}(L_{val}(w^{+},𝛼)-{\nabla }_{𝛼}(L_{val}(w^{-},𝛼)}{2𝜖}\\ where{w}^{+}=w\text{'}+{\nabla }_{w\text{'}}{L}_{val}(w^{\text{'}},\alpha )𝜖\\ and{w}^{-}=w\text{'}-{\nabla }_{w\text{'}}{L}_{val}(w^{\text{'}},\alpha )𝜖\end{array}$$