4.5. 計算式の簡単化

(Formula used in machine_learning header file package; 4. Derivative of the cross entropy error \(E\) with respect to model parameters \(W_{j,i}^{(m)}\); 4.5. Simpler formula)

前節までで以下の式が得られた。
The formula below were obtained in the previous sections.

4.3節の(3)式
Eq. (3) of section 4.3:
\[\begin{eqnarray} \PartialDiff{E}{W_{j,i}^{(m)}} &=& -\frac{1}{N}\sum_{n=0}^{N-1} \left[\left(1-\delta_{iJ^{(m)}}\right)x_{i}^{(m,n)} +\delta_{iJ^{(m)}}\right] \nonumber \\ & & \sum_{i’=0}^{J^{(M+1)}-1} \frac{t_{i’}^{(n)}}{x_{i’}^{(M+1,n)}} \left[D_{i’,j}^{(m,n)}-D_{i’,J^{(M+1)}-1}^{(M,n)} \delta_{mM}\right] \nonumber \\ & & \left(m=0,\cdots,M; i=0,\cdots,J^{(m)}; j=0,\cdots,J^{(m+1)}-1; \right. \nonumber \\ & & \left. m\neq M \mbox{ or } j\neq J^{(m+1)}-1\right) \label{eq.dEdW.useD} \end{eqnarray}\]
4.4節の(4)式
Eq. (4) of section 4.4:
\[\begin{eqnarray} D_{i’,j}^{(M,n)} &=& \PartialDiff{f_{i’}^{(M)} \left(y_0^{(M,n)},\cdots,y_{J^{(M+1)}-1}^{(M,n)}\right)} {y_{j}^{(M,n)}} \nonumber \\ & & \left(n=0,\cdots,N-1; i’=0,\cdots,J^{(M+1)}-1; j=0,\cdots,J^{(M+1)}-1\right) \label{eq.D.M1} \end{eqnarray}\]
4.4節の(7)式
Eq. (7) of section 4.4:
\[\begin{eqnarray} D_{i’,j}^{(m,n)} &=& \sum_{j’=0}^{J^{(m+2)}-1}D_{i’,j’}^{(m+1,n)} \sum_{i{’}{’}=0}^{J^{(m+1)-1}} W_{j’,i{’}{’}}^{(m+1)} \PartialDiff{f_{i{’}{’}}^{(m)} \left(y_0^{(m,n)},\cdots,y_{J^{(m+1)}-1}^{(m,n)}\right)} {y_{j}^{(m,n)}} \nonumber \\ & & \left(m=0,\cdots,M-1; n=0,\cdots,N-1; \right. \nonumber \\ & & \left. i’=0,\cdots,J^{(M+1)}-1; j=0,\cdots,J^{(m+1)}-1\right) \label{eq.D.recursive} \end{eqnarray}\]

式をより簡単にするために \[\begin{eqnarray} E_j^{(m,n)} &\equiv& \sum_{i’=0}^{J^{(M+1)}-1} \frac{t_{i’}^{(n)}}{x_{i’}^{(M+1,n)}} D_{i’,j}^{(m,n)} \nonumber \\ & & \left(m=0,\cdots,M; n=0,\cdots,N-1;j=0,\cdots,J^{(m+1)}-1\right) \label{eq.Ej} \end{eqnarray}\] とおく。(\ref{eq.Ej})を(\ref{eq.dEdW.useD})に代入すると \[\begin{eqnarray} \PartialDiff{E}{W_{j,i}^{(m)}} &=& -\frac{1}{N}\sum_{n=0}^{N-1} \left[\left(1-\delta_{iJ^{(m)}}\right)x_{i}^{(m,n)} +\delta_{iJ^{(m)}}\right] \nonumber \\ & & \left[\sum_{i’=0}^{J^{(M+1)}-1} \frac{t_{i’}^{(n)}}{x_{i’}^{(M+1,n)}} D_{i’,j}^{(m,n)} -\sum_{i’=0}^{J^{(M+1)}-1} \frac{t_{i’}^{(n)}}{x_{i’}^{(M+1,n)}} D_{i’,J^{(M+1)}-1}^{(M,n)}\delta_{mM}\right] \nonumber \\ &=& -\frac{1}{N}\sum_{n=0}^{N-1} \left[\left(1-\delta_{iJ^{(m)}}\right)x_{i}^{(m,n)} +\delta_{iJ^{(m)}}\right] \left[E_j^{(m,n)}-E_{J^{(M+1)}-1}^{(M,n)}\delta_{mM}\right] \nonumber \\ & & \left(m=0,\cdots,M; i=0,\cdots,J^{(m)}; j=0,\cdots,J^{(m+1)}-1; \right. \nonumber \\ & & \left. m\neq M \mbox{ or } j\neq J^{(m+1)}-1\right) \label{eq.dEdW.useEj} \end{eqnarray}\] となる。\(E_j^{(m,n)}\)は以下のように計算できる。まず\(m=M\)のとき、(\ref{eq.D.M1})式より \[\begin{eqnarray} E_j^{(M,n)} &=& \sum_{i’=0}^{J^{(M+1)}-1} \frac{t_{i’}^{(n)}}{x_{i’}^{(M+1,n)}}D_{i’,j}^{(M,n)} \nonumber \\ &=& \sum_{i’=0}^{J^{(M+1)}-1} \frac{t_{i’}^{(n)}}{x_{i’}^{(M+1,n)}} \PartialDiff{f_{i’}^{(M)} \left(y_0^{(M,n)},\cdots,y_{J^{(M+1)}-1}^{(M,n)}\right)} {y_{j}^{(M,n)}} \nonumber \\ & & \left(n=0,\cdots,N-1;j=0,\cdots,J^{(M+1)}-1\right) \label{eq.Ej.M1} \end{eqnarray}\] が成り立つ。また\(m<M\)のとき、(\ref{eq.D.recursive})式より \[\begin{eqnarray} E_j^{(m,n)} &=& \sum_{i’=0}^{J^{(M+1)}-1} \frac{t_{i’}^{(n)}}{x_{i’}^{(M+1,n)}}D_{i’,j}^{(m,n)} \nonumber \\ &=& \sum_{i’=0}^{J^{(M+1)}-1} \frac{t_{i’}^{(n)}}{x_{i’}^{(M+1,n)}} \sum_{j’=0}^{J^{(m+2)}-1}D_{i’,j’}^{(m+1,n)} \nonumber \\ & & \sum_{i{’}{’}=0}^{J^{(m+1)-1}} W_{j’,i{’}{’}}^{(m+1)} \PartialDiff{f_{i{’}{’}}^{(m)} \left(y_0^{(m,n)},\cdots,y_{J^{(m+1)}-1}^{(m,n)}\right)} {y_{j}^{(m,n)}} \nonumber \\ &=& \sum_{j’=0}^{J^{(m+2)}-1} \sum_{i’=0}^{J^{(M+1)}-1} \frac{t_{i’}^{(n)}}{x_{i’}^{(M+1,n)}} D_{i’,j’}^{(m+1,n)} \nonumber \\ & & \sum_{i{’}{’}=0}^{J^{(m+1)-1}} W_{j’,i{’}{’}}^{(m+1)} \PartialDiff{f_{i{’}{’}}^{(m)} \left(y_0^{(m,n)},\cdots,y_{J^{(m+1)}-1}^{(m,n)}\right)} {y_{j}^{(m,n)}} \nonumber \\ &=& \sum_{j’=0}^{J^{(m+2)}-1}E_{j’}^{(m+1,n)} \sum_{i{’}{’}=0}^{J^{(m+1)-1}} W_{j’,i{’}{’}}^{(m+1)} \PartialDiff{f_{i{’}{’}}^{(m)} \left(y_0^{(m,n)},\cdots,y_{J^{(m+1)}-1}^{(m,n)}\right)} {y_{j}^{(m,n)}} \nonumber \\ & & \left(m=0,\cdots,M-1; n=0,\cdots,N-1;j=0,\cdots,J^{(m+1)}-1\right) \label{eq.Ej.recursive} \end{eqnarray}\] が成り立つ。まず(\ref{eq.Ej.M1})を用いて\(E_j^{(M,n)}\)を求め、次に\(m\)を1つずつ小さくしながら漸化式(\ref{eq.Ej.recursive})を繰り返し用いれば全ての\(m\)に対する\(E_j^{(m,n)}\)が求まる。それらの結果を(\ref{eq.dEdW.useEj})に代入すれば \(\partial E/\partial W_{j,i}^{(m)}\)が求まる。 (\ref{eq.dEdW.useD})(\ref{eq.D.M1})(\ref{eq.D.recursive})を用いる場合と比べ、 (\ref{eq.dEdW.useEj})(\ref{eq.Ej.M1})(\ref{eq.Ej.recursive})を用いる方が \(i’\)に関する和の計算が(\ref{eq.Ej.M1})のみで済み、計算量を大幅に削減できる。
To simplify the formula, let us introduce a new quantity \(E_j^{(m,n)}\) defined by Eq. (\ref{eq.Ej}). Inserting this equation into (\ref{eq.dEdW.useD}) results in (\ref{eq.dEdW.useEj}). The values of \(E_j^{(m,n)}\) can be computed as follows. When \(m=M\), Eq. (\ref{eq.D.M1}) gives (\ref{eq.Ej.M1}). When \(m<M\), Eq. (\ref{eq.Ej.recursive}) is derived based on Eq. (\ref{eq.D.recursive}). First calculating \(E_j^{(M,n)}\) using Eq. (\ref{eq.Ej.M1}) and then repeatedly using the recursive equation (\ref{eq.Ej.recursive}) yields \(E_j^{(m,n)}\) for all \(m\). Inserting these results into (\ref{eq.dEdW.useEj}), we can calculate \(\partial E/\partial W_{j,i}^{(m)}\). Compared to Eqs. (\ref{eq.dEdW.useD}), (\ref{eq.D.M1}), and (\ref{eq.D.recursive}), using Eqs. (\ref{eq.dEdW.useEj}), (\ref{eq.Ej.M1}), and (\ref{eq.Ej.recursive}) significantly reduces the computational cost since the summation with regard to \(i’\) appears only in (\ref{eq.Ej.M1}).