人世間不祇需『問』之精神，還要有『追』的毅力︰

……黑傑克終於發現了咔嗎的『密室』，在多次的探訪後，找到一道『暗門』，不知通往何鄉何處？心中掂量著該進或退？最終發了一則『簡訊』，自此消逝於人世間……

1H ㄍㄞˋ 5W ㄉㄨㄟ一˙

1H 解？5W 追？

。黑傑克的言語駁雜，這則簡訊又太短，實在不容易『定論』，就妄說一通吧！『1H 解』當是說他已經了解了『Know How』；順著說『5W 追』自該是『誰』Who？『地』Where？『時』When？『事』What？『◎』Why？由於『誰地時事』大體已解，那可以『追』的也只剩下『◎』──  Why 為什麼 ── 了。此處用◎是有原因的，英文的 Why 又說成 For What，這就是『為什麼』的翻譯由來，因為見到黑傑克用了『追問』之追，頗感『學問之道』的不容易，故此特用◎，彰顯他的 Why not ── 追追追 ──精神！！

─── 摘自《黑傑克的咔嗎！！明暗之交》

 

如是者！或能趕得上 Michael Nielsen 先生之步伐耶？？

Why we only applied dropout to the fully-connected layers: If you look carefully at the code above, you’ll notice that we applied dropout only to the fully-connected section of the network, not to the convolutional layers. In principle we could apply a similar procedure to the convolutional layers. But, in fact, there’s no need: the convolutional layers have considerable inbuilt resistance to overfitting. The reason is that the shared weights mean that convolutional filters are forced to learn from across the entire image. This makes them less likely to pick up on local idiosyncracies in the training data. And so there is less need to apply other regularizers, such as dropout.

Going further: It’s possible to improve performance on MNIST still further. Rodrigo Benenson has compiled an informative summary page, showing progress over the years, with links to papers. Many of these papers use deep convolutional networks along lines similar to the networks we’ve been using. If you dig through the papers you’ll find many interesting techniques, and you may enjoy implementing some of them. If you do so it’s wise to start implementation with a simple network that can be trained quickly, which will help you more rapidly understand what is going on.

For the most part, I won’t try to survey this recent work. But I can’t resist making one exception. It’s a 2010 paper by Cireșan, Meier, Gambardella, and Schmidhuber*

*Deep, Big, Simple Neural Nets Excel on Handwritten Digit Recognition, by Dan Claudiu Cireșan, Ueli Meier, Luca Maria Gambardella, and Jürgen Schmidhuber (2010)..

What I like about this paper is how simple it is. The network is a many-layer neural network, using only fully-connected layers (no convolutions). Their most successful network had hidden layers containing , , , , and neurons, respectively. They used ideas similar to Simard et al to expand their training data. But apart from that, they used few other tricks, including no convolutional layers: it was a plain, vanilla network, of the kind that, with enough patience, could have been trained in the 1980s (if the MNIST data set had existed), given enough computing power(!) They achieved a classification accuracy of percent, more or less the same as ours. The key was to use a very large, very deep network, and to use a GPU to speed up training. This let them train for many epochs. They also took advantage of their long training times to gradually decrease the learning rate from to . It’s a fun exercise to try to match these results using an architecture like theirs.

Why are we able to train? We saw in the last chapter that there are fundamental obstructions to training in deep, many-layer neural networks. In particular, we saw that the gradient tends to be quite unstable: as we move from the output layer to earlier layers the gradient tends to either vanish (the vanishing gradient problem) or explode (the exploding gradient problem). Since the gradient is the signal we use to train, this causes problems.

How have we avoided those results?

Of course, the answer is that we haven’t avoided these results. Instead, we’ve done a few things that help us proceed anyway. In particular: (1) Using convolutional layers greatly reduces the number of parameters in those layers, making the learning problem much easier; (2) Using more powerful regularization techniques (notably dropout and convolutional layers) to reduce overfitting, which is otherwise more of a problem in more complex networks; (3) Using rectified linear units instead of sigmoid neurons, to speed up training – empirically, often by a factor of 3–5; (4) Using GPUs and being willing to train for a long period of time. In particular, in our final experiments we trained for epochs using a data set times larger than the raw MNIST training data. Earlier in the book we mostly trained for epochs using just the raw training data. Combining factors (3) and (4) it’s as though we’ve trained a factor perhaps times longer than before.

Your response may be “Is that it? Is that all we had to do to train deep networks? What’s all the fuss about?”

Of course, we’ve used other ideas, too: making use of sufficiently large data sets (to help avoid overfitting); using the right cost function (to avoid a learning slowdown); using good weight initializations (also to avoid a learning slowdown, due to neuron saturation); algorithmically expanding the training data. We discussed these and other ideas in earlier chapters, and have for the most part been able to reuse these ideas with little comment in this chapter.

With that said, this really is a rather simple set of ideas. Simple, but powerful, when used in concert. Getting started with deep learning has turned out to be pretty easy!

How deep are these networks, anyway? Counting the convolutional-pooling layers as single layers, our final architecture has hidden layers. Does such a network really deserve to be called a deep network? Of course, hidden layers is many more than in the shallow networks we studied earlier. Most of those networks only had a single hidden layer, or occasionally hidden layers. On the other hand, as of 2015 state-of-the-art deep networks sometimes have dozens of hidden layers. I’ve occasionally heard people adopt a deeper-than-thou attitude, holding that if you’re not keeping-up-with-the-Joneses in terms of number of hidden layers, then you’re not really doing deep learning. I’m not sympathetic to this attitude, in part because it makes the definition of deep learning into something which depends upon the result-of-the-moment. The real breakthrough in deep learning was to realize that it’s practical to go beyond the shallow – and -hidden layer networks that dominated work until the mid-2000s. That really was a significant breakthrough, opening up the exploration of much more expressive models. But beyond that, the number of layers is not of primary fundamental interest. Rather, the use of deeper networks is a tool to use to help achieve other goals – like better classification accuracies.

A word on procedure: In this section, we’ve smoothly moved from single hidden-layer shallow networks to many-layer convolutional networks. It’s all seemed so easy! We make a change and, for the most part, we get an improvement. If you start experimenting, I can guarantee things won’t always be so smooth. The reason is that I’ve presented a cleaned-up narrative, omitting many experiments – including many failed experiments. This cleaned-up narrative will hopefully help you get clear on the basic ideas. But it also runs the risk of conveying an incomplete impression. Getting a good, working network can involve a lot of trial and error, and occasional frustration. In practice, you should expect to engage in quite a bit of experimentation. To speed that process up you may find it helpful to revisit Chapter 3’s discussion of how to choose a neural network’s hyper-parameters, and perhaps also to look at some of the further reading suggested in that section.

The code for our convolutional networks

Alright, let’s take a look at the code for our program, network3.py. Structurally, it’s similar to network2.py, the program we developed in Chapter 3, although the details differ, due to the use of Theano. We’ll start by looking at the FullyConnectedLayer class, which is similar to the layers studied earlier in the book. Here’s the code (discussion below):

………