W!o+ 的《小伶鼬工坊演義》︰神經網絡【backpropagation】一

談及『反向傳播算法』之前， Michael Nielsen 先生首起『記號法』之用︰

……

Let’s begin with a notation which lets us refer to weights in the network in an unambiguous way. We’ll use $w^l_{jk}$ $w_{j k}^{l}$ to denote the weight for the connection from the $k^{\rm th}$ $k^{t h}$ neuron in the $(l-1)^{\rm th}$ $(l - 1)^{t h}$ layer to the $j^{\rm th}$ $j^{t h}$ neuron in the $l^{\rm th}$ $l^{t h}$ layer. So, for example, the diagram below shows the weight on a connection from the fourth neuron in the second layer to the second neuron in the third layer of a network:

This notation is cumbersome at first, and it does take some work to master. But with a little effort you’ll find the notation becomes easy and natural. One quirk of the notation is the ordering of the

j

and $k$

k

indices. You might think that it makes more sense to use $j$

j

to refer to the input neuron, and

k

to the output neuron, not vice versa, as is actually done. I’ll explain the reason for this quirk below.

We use a similar notation for the network’s biases and activations. Explicitly, we use $b^l_j$ $b_{j}^{l}$ for the bias of the $j^{\rm th}$ $j^{t h}$ neuron in the $l^{t h}$ layer. And we use $a^l_j$ $a_{j}^{l}$ for the activation of the $j^{\rm th}$ $j^{t h}$ neuron in the $l^{\rm th}$ $l^{t h}$ layer. The following diagram shows examples of these notations in use:

With these notations, the activation $a^l_j$

a_{j}^{l}

of the $j^{\rm th}$

j^{t h}

neuron in the $l^{\rm th}$

l^{t h}

layer is related to the activations in the

(l - 1)^{t h}

layer by the equation (compare Equation (4) and surrounding discussion in the last chapter)

$a^{l}_j = \sigma\left( \sum_k w^{l}_{jk} a^{l-1}_k + b^l_j \right), \ \ \ \ (23)$

where the sum is over all neurons $k$ $k$ in the $(l-1)^{\rm th}$ $(l - 1)^{t h}$ layer. To rewrite this expression in a matrix form we define a weight matrix $w^l$ $w^{l}$ for each layer, $l$ $l$ . The entries of the weight matrix $w^l$ $w^{l}$ are just the weights connecting to the $l^{\rm th}$ $l^{t h}$ layer of neurons, that is, the entry in the $j^{\rm th}$ $j^{t h}$ row and $k^{\rm th}$ $k^{t h}$ column is $w^l_{jk}$ $w_{j k}^{l}$ . Similarly, for each layer $l$ we define a bias vector, $b^l$ $b^{l}$ . You can probably guess how this works – the components of the bias vector are just the values $b^l_j$ $b_{j}^{l}$ , one component for each neuron in the $l^{\rm th}$ $l^{t h}$ layer. And finally, we define an activation vector $a^l$ $a^{l}$ whose components are the activations $a^l_j$ $a_{j}^{l}$ .