W!o+ 的《小伶鼬工坊演義》︰神經網絡【MNIST】三

若是每次測試那七十四行的程式,都得重新訓練神經網絡之

是由稱為神經元神經細胞所組成的神經系統控制中心,是所有脊椎動物和大部分無脊椎動物都具有的一個器官,只有少數的無脊椎動物沒有腦,例如海綿水母、成年的海鞘海星,它們以分散或者局部的神經網絡代替。

許多動物的腦位於部,通常是靠近主要的感覺器官,例如視覺聽覺前庭系統味覺嗅覺。腦是脊椎動物身體中最複雜的器官 。在普通人類的大腦皮質(腦中最大的部分)中,包含150-330億個神經元,[1]每一個神經元都通過突觸和其他數千個神經元相連接 。這些神經元之間通過稱作軸突原生質纖維進行較長距離互相聯結,可以將一種稱作動作電位的衝動信號,在腦的不同區域之間或者向身體的特定接收細胞傳遞。脊椎動物的腦由顱骨保護。腦與脊髓構成中樞神經系統。中樞神經系統的細胞依靠複雜的聯繫來處理傳遞信息。腦是感情思考、生命得以維持的中樞。它控制和協調行為、身體內穩態(身體功能,例如心跳血壓體溫等)以及精神活動(例如認知情感記憶學習)。

生理上來說,腦的功能就是控制身體的其他器官。腦對其他器官的作用方式,一是調製肌肉的運動模式,二是通過分泌一些稱為荷爾蒙的化學物質。集中的控制方式,可以對環境的變化做出迅速而一致的反應。 一些基本的反應,例如反射,可以通過脊髓或者周邊神經節來控制,然而基於多種感官輸入,有心智、有目的的動作,只有通過腦中樞的整合能力才能控制。

關於單個腦細胞的運作機制,現今已經有了比較詳細的了解;然而數以兆億的神經元如何以集群的方式合作,還是一個未解決的問題 。[2]現代神經科學中,新近的模型將腦看作一種生物計算機,雖然運行的機制和電子計算機很不一樣,但是它們從周圍世界中獲得信息、存儲信息、以多種方式處理信息的功能是類似的,它有點像計算機中的中央處理器(CPU)。

本文會對各種動物的腦進行比較,特別是脊椎動物的腦,而人腦將被作為各種腦的其中一種進行討論。人腦的特別之處會在人腦條目中探討,因為其中很多話題在人腦的前提下討論,內容會豐富得多 。其中最重要的,是腦疾病與腦損傷造成的後果,它會被放在人腦條目中探討,因為人腦的大多數常見疾病並不見於其他物種,即使有,它們的表現形式也可能不同。

220px-Frontal_lobe_animation

人腦立體旋轉剖面圖

───

 

,顯然有點荒謬!所以此處設想如何『保存大腦』?甚或該怎麼『移植』的呢??研究一下『network.py』︰

"""
network.py
~~~~~~~~~~

A module to implement the stochastic gradient descent learning
algorithm for a feedforward neural network.  Gradients are calculated
using backpropagation.  Note that I have focused on making the code
simple, easily readable, and easily modifiable.  It is not optimized,
and omits many desirable features.
"""

#### Libraries
# Standard library
import random

# Third-party libraries
import numpy as np

class Network(object):

    def __init__(self, sizes):
        """The list ``sizes`` contains the number of neurons in the
        respective layers of the network.  For example, if the list
        was [2, 3, 1] then it would be a three-layer network, with the
        first layer containing 2 neurons, the second layer 3 neurons,
        and the third layer 1 neuron.  The biases and weights for the
        network are initialized randomly, using a Gaussian
        distribution with mean 0, and variance 1.  Note that the first
        layer is assumed to be an input layer, and by convention we
        won't set any biases for those neurons, since biases are only
        ever used in computing the outputs from later layers."""
        self.num_layers = len(sizes)
        self.sizes = sizes
        self.biases = [np.random.randn(y, 1) for y in sizes[1:]]
        self.weights = [np.random.randn(y, x)
                        for x, y in zip(sizes[:-1], sizes[1:])]

 

就會發現這個『腦』就是保存在

self.sizes

self.biases

self.weights

裡。如是假使知道『numpy』的 array 檔案存取之法︰

numpy.savez

numpy.savez(file, *args, **kwds)
Save several arrays into a single file in uncompressed .npz format.

If arguments are passed in with no keywords, the corresponding variable names, in the .npz file, are ‘arr_0’, ‘arr_1’, etc. If keyword arguments are given, the corresponding variable names, in the .npz file will match the keyword names.

Parameters:

file : str or file

Either the file name (string) or an open file (file-like object) where the data will be saved. If file is a string, the .npz extension will be appended to the file name if it is not already there.

args : Arguments, optional

Arrays to save to the file. Since it is not possible for Python to know the names of the arrays outside savez, the arrays will be saved with names “arr_0”, “arr_1”, and so on. These arguments can be any expression.

kwds : Keyword arguments, optional

Arrays to save to the file. Arrays will be saved in the file with the keyword names.

Returns:

None

See also

save
Save a single array to a binary file in NumPy format.
savetxt
Save an array to a file as plain text.
savez_compressed
Save several arrays into a compressed .npz archive

───

 

以及

numpy.load

numpy.load(file, mmap_mode=None, allow_pickle=True, fix_imports=True, encoding=’ASCII’)
Load arrays or pickled objects from .npy, .npz or pickled files.

Parameters:

file : file-like object or string

The file to read. File-like objects must support the seek() and read() methods. Pickled files require that the file-like object support the readline() method as well.

mmap_mode : {None, ‘r+’, ‘r’, ‘w+’, ‘c’}, optional

If not None, then memory-map the file, using the given mode (see numpy.memmap for a detailed description of the modes). A memory-mapped array is kept on disk. However, it can be accessed and sliced like any ndarray. Memory mapping is especially useful for accessing small fragments of large files without reading the entire file into memory.

allow_pickle : bool, optional

Allow loading pickled object arrays stored in npy files. Reasons for disallowing pickles include security, as loading pickled data can execute arbitrary code. If pickles are disallowed, loading object arrays will fail. Default: True

fix_imports : bool, optional

Only useful when loading Python 2 generated pickled files on Python 3, which includes npy/npz files containing object arrays. If fix_imports is True, pickle will try to map the old Python 2 names to the new names used in Python 3.

encoding : str, optional

What encoding to use when reading Python 2 strings. Only useful when loading Python 2 generated pickled files on Python 3, which includes npy/npz files containing object arrays. Values other than ‘latin1’, ‘ASCII’, and ‘bytes’ are not allowed, as they can corrupt numerical data. Default: ‘ASCII’

Returns:

result : array, tuple, dict, etc.

Data stored in the file. For .npz files, the returned instance of NpzFile class must be closed to avoid leaking file descriptors.

Raises:

IOError

If the input file does not exist or cannot be read.

ValueError

The file contains an object array, but allow_pickle=False given.

See also

save, savez, savez_compressed, loadtxt

memmap
Create a memory-map to an array stored in a file on disk.

───

 

再輔之以 list → array 轉換之法︰

Array creation

Introduction

There are 5 general mechanisms for creating arrays:

  1. Conversion from other Python structures (e.g., lists, tuples)
  2. Intrinsic numpy array array creation objects (e.g., arange, ones, zeros, etc.)
  3. Reading arrays from disk, either from standard or custom formats
  4. Creating arrays from raw bytes through the use of strings or buffers
  5. Use of special library functions (e.g., random)

This section will not cover means of replicating, joining, or otherwise expanding or mutating existing arrays. Nor will it cover creating object arrays or structured arrays. Both of those are covered in their own sections.

Converting Python array_like Objects to Numpy Arrays

In general, numerical data arranged in an array-like structure in Python can be converted to arrays through the use of the array() function. The most obvious examples are lists and tuples. See the documentation for array() for details for its use. Some objects may support the array-protocol and allow conversion to arrays this way. A simple way to find out if the object can be converted to a numpy array using array() is simply to try it interactively and see if it works! (The Python Way).

Examples:

>>> x = np.array([2,3,1,0])
>>> x = np.array([2, 3, 1, 0])
>>> x = np.array([[1,2.0],[0,0],(1+1j,3.)]) # note mix of tuple and lists,
    and types
>>> x = np.array([[ 1.+0.j, 2.+0.j], [ 0.+0.j, 0.+0.j], [ 1.+1.j, 3.+0.j]])

 

果可保存『大腦』乎??!!

>>> s = network.np.array(net.sizes)
>>> w1 = net.weights[0]
>>> w2 = net.weights[1]
>>> b1 = net.biases[0]
>>> b2 = net.biases[1]
>>> network.np.savez("swb",s=s,w1=w1,w2=w2,b1=b1,b2=b2)
>>> npzfile = network.np.load("swb.npz")
>>> npzfile.files
['s', 'b2', 'w2', 'w1', 'b1']
>>> list(npzfile["s"])
[784, 30, 10] 
>>> 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

W!o+ 的《小伶鼬工坊演義》︰神經網絡【MNIST】二

固然我們可以用『純數學』或『抽象演算法』的方式來了解『手寫阿拉伯數字』辨識之『神經網絡』程式。不過要是能輔之以

資料視覺化

資料視覺化是關於資料之視覺表現形式的研究;其中,這種資料的視覺表現形式被定義為一種以某種概要形式抽提出來的資訊,包括相應資訊單位的各種屬性和變數[1]

……

歷史

資料視覺化領域的起源可以追溯到二十世紀50年代電腦圖形學的早期。當時,人們利用電腦建立出了首批圖形圖表。1987年,由布魯斯·麥考梅克湯姆斯·蒂凡提瑪克辛·布朗所編寫的美國國家科學基金會報告《Visualization in Scientific Computing》(意為「科學計算之中的視覺化」)[4],對於這一領域產生了大幅度的促進和刺激。這份報告之中強調了新的基於電腦視覺化技術方法的必要性。隨著電腦運算能力的迅速提升,人們建立了規模越來越大,複雜程度越來越高的數值模型,從而造就了形形色色體積龐大的數值型資料集。同時,人們不但利用醫學掃描器和顯微鏡之類的資料採集裝置產生大型的資料集,而且還利用可以儲存文字、數值和多媒體資訊的大型資料庫來收集資料。因而,就需要高階的電腦圖形學技術與方法來處理和視覺化這些規模龐大的資料集[3]

短語「Visualization in Scientific Computing」(意為「科學計算之中的視覺化」)後來變成了「Scientific Visualization」(即「科學視覺化」),而前者最初指的是作為科學計算之組成部分的視覺化:也就是科學與工程實踐當中對於電腦建模模擬的運用。更近一些的時候,視覺化也日益尤為關注資料,包括那些來自商業財務行政管理數字媒體等方面的大型異質性資料集合。二十世紀90年代初期,人們發起了一個新的,稱為「資訊視覺化」的研究領域,旨在為許多應用領域之中對於抽象的異質性資料集的分析工作提供支援。因此,目前人們正在逐漸接受這個同時涵蓋科學視覺化資訊視覺化領域的新生術語「資料視覺化」[3]

自那時起,資料視覺化就是一個處於不斷演變之中的概念,其邊界在不斷地擴大;因而,最好是對其加以寬泛的定義。資料視覺化指的是技術上較為高階的技術方法,而這些技術方法允許利用圖形、圖像處理電腦視覺以及使用者介面,通過表達、建模以及對立體、表面、屬性以及動畫的顯示,對資料加以視覺化解釋。與立體建模之類的特殊技術方法相比,資料視覺化所涵蓋的技術方法要廣泛得多[5]

1280px-Minard

法國工程師查爾斯·約瑟夫·密納德於1861年繪製的關於拿破崙入侵俄羅斯的資訊圖

───

 

難道不會更生動的嗎?或許還可以讓想像力飛昇耶??這就是所謂『一圖勝千言』之古義吧!!

只因短短文本無法解說『matplotlib』強大繪圖能力,也只能遁之以相關『範例教程』的了︰

matplotlib

Pyplot tutorial

matplotlib.pyplot is a collection of command style functions that make matplotlib work like MATLAB. Each pyplot function makes some change to a figure: e.g., creates a figure, creates a plotting area in a figure, plots some lines in a plotting area, decorates the plot with labels, etc. In matplotlib.pyplot various states are preserved across function calls, so that it keeps track of things like the current figure and plotting area, and the plotting functions are directed to the current axes (please note that “axes” here and in most places in the documentation refers to the axes part of a figure and not the strict mathematical term for more than one axis).

───

 

Image tutorial

Startup commands

First, let’s start IPython. It is a most excellent enhancement to the standard Python prompt, and it ties in especially well with Matplotlib. Start IPython either at a shell, or the IPython Notebook now.

With IPython started, we now need to connect to a GUI event loop. This tells IPython where (and how) to display plots. To connect to a GUI loop, execute the %matplotlib magic at your IPython prompt. There’s more detail on exactly what this does at IPython’s documentation on GUI event loops.

If you’re using IPython Notebook, the same commands are available, but people commonly use a specific argument to the %matplotlib magic:

───

 

就此展開『探索程式』、『程式探索』之旅乎!!??

假使按造 Michael Nielsen 所說的步驟執行︰

pi@raspberrypi ~ git clone https://github.com/mnielsen/neural-networks-and-deep-learning.git Cloning into 'neural-networks-and-deep-learning'... remote: Counting objects: 1141, done. remote: Total 1141 (delta 0), reused 0 (delta 0), pack-reused 1141 Receiving objects: 100% (1141/1141), 20.33 MiB | 4.30 MiB/s, done. Resolving deltas: 100% (574/574), done. Checking connectivity... done. pi@raspberrypi ~ 
pi@raspberrypi ~ cd neural-networks-and-deep-learning/ pi@raspberrypi ~/neural-networks-and-deep-learning cd src
pi@raspberrypi ~/neural-networks-and-deep-learning/src python Python 2.7.9 (default, Mar  8 2015, 00:52:26)  [GCC 4.9.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import mnist_loader >>> training_data, validation_data, test_data = \ ... mnist_loader.load_data_wrapper() >>> import network >>> net = network.Network([784, 30, 10]) >>> net.SGD(training_data, 30, 10, 3.0, test_data=test_data) Epoch 0: 9013 / 10000 Epoch 1: 9181 / 10000 Epoch 2: 9277 / 10000 Epoch 3: 9333 / 10000 Epoch 4: 9374 / 10000 Epoch 5: 9374 / 10000 Epoch 6: 9375 / 10000 Epoch 7: 9369 / 10000 Epoch 8: 9415 / 10000 Epoch 9: 9435 / 10000 Epoch 10: 9428 / 10000 Epoch 11: 9423 / 10000 Epoch 12: 9452 / 10000 Epoch 13: 9442 / 10000 Epoch 14: 9434 / 10000 Epoch 15: 9453 / 10000 Epoch 16: 9480 / 10000 Epoch 17: 9468 / 10000 Epoch 18: 9461 / 10000 Epoch 19: 9483 / 10000 Epoch 20: 9462 / 10000 Epoch 21: 9477 / 10000 Epoch 22: 9474 / 10000 Epoch 23: 9491 / 10000 Epoch 24: 9474 / 10000 Epoch 25: 9487 / 10000 Epoch 26: 9453 / 10000 Epoch 27: 9462 / 10000 Epoch 28: 9472 / 10000 Epoch 29: 9482 / 10000 >>>  </pre>    <span style="color: #666699;">大約經過五十分鐘後,在樹莓派 3 上可以訓練完成!或許是作者的運氣不好,最好的成績只有9491 / 10000$ 而已??

如果依據《易經》之『啟蒙』之說︰

蒙 ䷃ :亨。 匪我求童蒙,童蒙求我。 初噬告,再三瀆,瀆則不告。利貞。

彖曰:蒙,山下有險,險而止,蒙。 蒙亨,以亨行時中也。匪我求童蒙,童蒙求我,志應也。 初噬告,以剛中也。再三瀆, 瀆則不告,瀆蒙也。 蒙以養正,聖功也。

象曰:山下出泉,蒙﹔君子以果行育德。

,果應『再』乎!真不怕『三瀆』耶!!或可聽聽來知德這麼講︰

蒙卦 坎下艮上

蒙,昧也。其卦以坎遇艮,山下有險,艮山在外,坎水在內。水乃必行之物,遇山而止,內既險陷不安,外又行之不去,莫知所往,昏蒙之象也。《序卦》「屯者,物之始生也。物生必蒙,故受之以蒙」,所以次屯。

蒙,亨,匪我求童蒙,童蒙求我。初筮告,再三瀆,瀆則不吉,利貞。(告,古毒反)

蒙亨者,言蒙者亨也,不終于蒙也。匪我求童蒙二句,正理也。再,指四,陽一陰二,二再則四矣。三,指三。瀆者煩瀆也。初筮者,初筮下卦,得剛中也。比卦坎之剛中在上卦,故曰再筮。告者,二告乎五也。不告者,二不告乎三四也。凡陽則明,陰則暗,所以九二發六五之蒙。利貞者,教之以正也。

 

再筮得
Epoch 0: 9059 / 10000
Epoch 1: 9274 / 10000
Epoch 2: 9330 / 10000
Epoch 3: 9327 / 10000
Epoch 4: 9374 / 10000
Epoch 5: 9426 / 10000
Epoch 6: 9443 / 10000
Epoch 7: 9396 / 10000
Epoch 8: 9417 / 10000
Epoch 9: 9426 / 10000
Epoch 10: 9452 / 10000
Epoch 11: 9457 / 10000
Epoch 12: 9439 / 10000
Epoch 13: 9428 / 10000
Epoch 14: 9413 / 10000
Epoch 15: 9460 / 10000
Epoch 16: 9459 / 10000
Epoch 17: 9467 / 10000
Epoch 18: 9443 / 10000
Epoch 19: 9472 / 10000
Epoch 20: 9467 / 10000
Epoch 21: 9455 / 10000
Epoch 22: 9472 / 10000
Epoch 23: 9454 / 10000
Epoch 24: 9451 / 10000
Epoch 25: 9494 / 10000
Epoch 26: 9460 / 10000
Epoch 27: 9474 / 10000
Epoch 28: 9470 / 10000
Epoch 29: 9437 / 10000
>>> 

 

又何必在意那『結果』的呢?如以『隨機』之義而言,難到不該是『陰陽不測謂之神』的嗎??『大數平均值』方才是法則的吧 !!

That is, the trained network gives us a classification rate of about 95 percent - 95.42 percent at its peak ("Epoch 28")! That's quite encouraging as a first attempt. I should warn you, however, that if you run the code then your results are not necessarily going to be quite the same as mine, since we'll be initializing our network using (different) random weights and biases. To generate results in this chapter I've taken best-of-three runs.

─── Michael Nielsen

 

【練習題】

>>> import matplotlib.pyplot as plt
>>> img = training_data[0][0].reshape(28,28)
>>> plt.imshow(img)
<matplotlib.image.AxesImage object at 0x73f414f0>
>>> plt.show()

Figure 1_079

 

>>> plt.imshow(img, cmap='Greys', interpolation='nearest')
<matplotlib.image.AxesImage object at 0x740a5270>
>>> plt.show()

Figure 1_080

 

>>> len(training_data[0][0])
784
>>> len(training_data[0][1])
10
>>> net.feedforward(training_data[0][0])
array([[  8.31077720e-05],
       [  5.83811178e-08],
       [  3.52833999e-09],
       [  4.06550775e-05],
       [  1.80642282e-08],
       [  9.98592617e-01],
       [  1.64964140e-09],
       [  4.81507294e-06],
       [  6.47293120e-04],
       [  5.81232092e-07]])
>>> network.np.argmax(net.feedforward(training_data[0][0]))
5
>>> 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

W!o+ 的《小伶鼬工坊演義》︰神經網絡【MNIST】一

由於 Michael Nielsen 先生已經將那個長七十四行之派生程式講解的十分清楚的了,此處也就不再贅述。直接跳接執行段文本︰

How well does the program recognize handwritten digits? Well, let’s start by loading in the MNIST data. I’ll do this using a little helper program, mnist_loader.py, to be described below. We execute the following commands in a Python shell,

>>> import mnist_loader
>>> training_data, validation_data, test_data = \
... mnist_loader.load_data_wrapper()

Of course, this could also be done in a separate Python program, but if you’re following along it’s probably easiest to do in a Python shell.

After loading the MNIST data, we’ll set up a Network with 30 hidden neurons. We do this after importing the Python program listed above, which is named network,

>>> import network
>>> net = network.Network([784, 30, 10])

Finally, we’ll use stochastic gradient descent to learn from the MNIST training_data over 30 epochs, with a mini-batch size of 10, and a learning rate of \eta = 3.0,

>>> net.SGD(training_data, 30, 10, 3.0, test_data=test_data)

Note that if you’re running the code as you read along, it will take some time to execute – for a typical machine (as of 2015) it will likely take a few minutes to run. I suggest you set things running, continue to read, and periodically check the output from the code. If you’re in a rush you can speed things up by decreasing the number of epochs, by decreasing the number of hidden neurons, or by using only part of the training data. Note that production code would be much, much faster: these Python scripts are intended to help you understand how neural nets work, not to be high-performance code! And, of course, once we’ve trained a network it can be run very quickly indeed, on almost any computing platform. For example, once we’ve learned a good set of weights and biases for a network, it can easily be ported to run in Javascript in a web browser, or as a native app on a mobile device. In any case, here is a partial transcript of the output of one training run of the neural network. The transcript shows the number of test images correctly recognized by the neural network after each epoch of training. As you can see, after just a single epoch this has reached 9,129 out of 10,000, and the number continues to grow,

Epoch 0: 9129 / 10000
Epoch 1: 9295 / 10000
Epoch 2: 9348 / 10000
...
Epoch 27: 9528 / 10000
Epoch 28: 9542 / 10000
Epoch 29: 9534 / 10000

That is, the trained network gives us a classification rate of about 95 percent – 95.42 percent at its peak (“Epoch 28”)! That’s quite encouraging as a first attempt. I should warn you, however, that if you run the code then your results are not necessarily going to be quite the same as mine, since we’ll be initializing our network using (different) random weights and biases. To generate results in this chapter I’ve taken best-of-three runs.

…… ※學習會退化耶 ?

 

然而此刻我們並不談這些執行結果。為求能更深入理解那個程式之『學習過程』,我們需要安裝一些 Python 『工具』程式庫。就像 Michael Nielsen 先生不只打包了 MNIST 之『資料庫』︰

neural-networks-and-deep-learning/data/

mnist.pkl.gz

。還寫了一個方便的『導入』程式︰

neural-networks-and-deep-learning/src/mnist_loader.py

。但當改天需要使用 MNIST 原始資料庫時,
THE MNIST DATABASE
of handwritten digits
Yann LeCun, Courant Institute, NYU
Corinna Cortes, Google Labs, New York
Christopher J.C. Burges, Microsoft Research, Redmond

………

 

自己還是得了解它的檔案格式

FILE FORMATS FOR THE MNIST DATABASE

The data is stored in a very simple file format designed for storing vectors and multidimensional matrices. General info on this format is given at the end of this page, but you don’t need to read that to use the data files.

All the integers in the files are stored in the MSB first (high endian) format used by most non-Intel processors. Users of Intel processors and other low-endian machines must flip the bytes of the header.

There are 4 files:

train-images-idx3-ubyte: training set images
train-labels-idx1-ubyte: training set labels
t10k-images-idx3-ubyte:  test set images
t10k-labels-idx1-ubyte:  test set labels

The training set contains 60000 examples, and the test set 10000 examples.

The first 5000 examples of the test set are taken from the original NIST training set. The last 5000 are taken from the original NIST test set. The first 5000 are cleaner and easier than the last 5000.

TRAINING SET LABEL FILE (train-labels-idx1-ubyte):

[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  60000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
........
xxxx     unsigned byte   ??               label

The labels values are 0 to 9.

TRAINING SET IMAGE FILE (train-images-idx3-ubyte):

[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000803(2051) magic number
0004     32 bit integer  60000            number of images
0008     32 bit integer  28               number of rows
0012     32 bit integer  28               number of columns
0016     unsigned byte   ??               pixel
0017     unsigned byte   ??               pixel
........
xxxx     unsigned byte   ??               pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).

TEST SET LABEL FILE (t10k-labels-idx1-ubyte):

[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  10000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
........
xxxx     unsigned byte   ??               label

The labels values are 0 to 9.

TEST SET IMAGE FILE (t10k-images-idx3-ubyte):

[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000803(2051) magic number
0004     32 bit integer  10000            number of images
0008     32 bit integer  28               number of rows
0012     32 bit integer  28               number of columns
0016     unsigned byte   ??               pixel
0017     unsigned byte   ??               pixel
........
xxxx     unsigned byte   ??               pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).

───

 

。故而於此介紹兩套派生程式庫︰

matplotlib

Introduction

matplotlib is a python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. matplotlib can be used in python scripts, the python and ipython shell (ala MATLAB®* or Mathematica®), web application servers, and six graphical user interface toolkits.

screenshots

matplotlib tries to make easy things easy and hard things possible. You can generate plots, histograms, power spectra, bar charts, errorcharts, scatterplots, etc, with just a few lines of code. For a sampling, see the screenshots, thumbnail gallery, and examples directory

For simple plotting the pyplot interface provides a MATLAB-like interface, particularly when combined with IPython. For the power user, you have full control of line styles, font properties, axes properties, etc, via an object oriented interface or via a set of functions familiar to MATLAB users.

───

 

以及

Python Imaging Library (PIL)

The Python Imaging Library (PIL) adds image processing capabilities to your Python interpreter. This library supports many file formats, and provides powerful image processing and graphics capabilities.

Status

The current free version is PIL 1.1.7. This release supports Python 1.5.2 and newer, including 2.5 and 2.6. A version for 3.X will be released later.

……

 

或應改為

python-pillow/Pillow

About

Goals

The fork author’s goal is to foster and support active development of PIL through:

License

Like PIL, Pillow is licensed under the MIT-like open source PIL Software License:

Software License

The Python Imaging Library (PIL) is

    Copyright © 1997-2011 by Secret Labs AB
    Copyright © 1995-2011 by Fredrik Lundh

By obtaining, using, and/or copying this software and/or its associated documentation, you agree that you have read, understood, and will comply with the following terms and conditions:

Permission to use, copy, modify, and distribute this software and its associated documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appears in all copies, and that both that copyright notice and this permission notice appear in supporting documentation, and that the name of Secret Labs AB or the author not be used in advertising or publicity pertaining to distribution of the software without specific, written prior permission.

SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Why a fork?

PIL is not setuptools compatible. Please see this Image-SIG post for a more detailed explanation. Also, PIL’s current bi-yearly (or greater) release schedule is too infrequent to accommodate the large number and frequency of issues reported.

What about PIL?

Note

Prior to Pillow 2.0.0, very few image code changes were made. Pillow 2.0.0 added Python 3 support and includes many bug fixes from many contributors.

As more time passes since the last PIL release, the likelihood of a new PIL release decreases. However, we’ve yet to hear an official “PIL is dead” announcement. So if you still want to support PIL, please report issues here first, then open corresponding Pillow tickets here.

Please provide a link to the first ticket so we can track the issue(s) upstream.

───

 

以方便 MNIST 手寫阿拉伯數字之『圖像顯示』和『圖形處理』,或將更可以『驗證』它之『辨識』能力乎??!!

 

 

 

 

 

 

 

 

 

 

 

 

 

 

W!o+ 的《小伶鼬工坊演義》︰神經網絡【Sigmoid】八

此一章節 Michael Nielsen 先生開始說明手寫阿拉伯數字辨識的 Python 程式內容︰

Implementing our network to classify digits

Alright, let’s write a program that learns how to recognize handwritten digits, using stochastic gradient descent and the MNIST training data. We’ll do this with a short Python (2.7) program, just 74 lines of code! The first thing we need is to get the MNIST data. If you’re a git user then you can obtain the data by cloning the code repository for this book,

git clone https://github.com/mnielsen/neural-networks-and-deep-learning.git

If you don’t use git then you can download the data and code here.

Incidentally, when I described the MNIST data earlier, I said it was split into 60,000 training images, and 10,000 test images. That’s the official MNIST description. Actually, we’re going to split the data a little differently. We’ll leave the test images as is, but split the 60,000-image MNIST training set into two parts: a set of 50,000 images, which we’ll use to train our neural network, and a separate 10,000 image validation set. We won’t use the validation data in this chapter, but later in the book we’ll find it useful in figuring out how to set certain hyper-parameters of the neural network – things like the learning rate, and so on, which aren’t directly selected by our learning algorithm. Although the validation data isn’t part of the original MNIST specification, many people use MNIST in this fashion, and the use of validation data is common in neural networks. When I refer to the “MNIST training data” from now on, I’ll be referring to our 50,000 image data set, not the original 60,000 image data set* *As noted earlier, the MNIST data set is based on two data sets collected by NIST, the United States’ National Institute of Standards and Technology. To construct MNIST the NIST data sets were stripped down and put into a more convenient format by Yann LeCun, Corinna Cortes, and Christopher J. C. Burges. See this link for more details. The data set in my repository is in a form that makes it easy to load and manipulate the MNIST data in Python. I obtained this particular form of the data from the LISA machine learning laboratory at the University of Montreal (link)..

Apart from the MNIST data we also need a Python library called Numpy, for doing fast linear algebra. If you don’t already have Numpy installed, you can get it here.

Let me explain the core features of the neural networks code, before giving a full listing, below. The centerpiece is a Network class, which we use to represent a neural network. Here’s the code we use to initialize a Network object:

class Network(object):

    def __init__(self, sizes):
        self.num_layers = len(sizes)
        self.sizes = sizes
        self.biases = [np.random.randn(y, 1) for y in sizes[1:]]
        self.weights = [np.random.randn(y, x) 
                        for x, y in zip(sizes[:-1], sizes[1:])]

In this code, the list sizes contains the number of neurons in the respective layers. So, for example, if we want to create a Network object with 2 neurons in the first layer, 3 neurons in the second layer, and 1 neuron in the final layer, we’d do this with the code:

net = Network([2, 3, 1])

The biases and weights in the Network object are all initialized randomly, using the Numpy np.random.randn function to generate Gaussian distributions with mean 0 and standard deviation 1. This random initialization gives our stochastic gradient descent algorithm a place to start from. In later chapters we’ll find better ways of initializing the weights and biases, but this will do for now. Note that the Network initialization code assumes that the first layer of neurons is an input layer, and omits to set any biases for those neurons, since biases are only ever used in computing the outputs from later layers.

───

 

打算深入理解者,對

NumPy

NumPy is the fundamental package for scientific computing with Python. It contains among other things:

  • a powerful N-dimensional array object
  • sophisticated (broadcasting) functions
  • tools for integrating C/C++ and Fortran code
  • useful linear algebra, Fourier transform, and random number capabilities

Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases.

Numpy is licensed under the BSD license, enabling reuse with few restrictions.

───

 

科學計算派生程式庫之了解大概不能少。

有志於此者,何不閱讀 Travis E. Oliphant 先生寫的

Guide to NumPy
Travis E. Oliphant, PhD
Dec 7, 2006
This book is under restricted distribution using a Market-D
etermined, Temporary, Distribution-Restriction (MDTDR) system (see http://www.trelgol.com) until October 31, 2010 at the latest. If you receive this book, you a re asked not to copy it in any form (electronic or paper) until the temporary distribution-restriction lapses. If you have multiple users at an institution, you should eith
er share a single copy using some form of digital library check-out, or buy multiple copies. The more copies purchased, the sooner the documentation can be released from this inconvenient distribution restriction. After October 31, 2010 this book may be freely
copied in any format and used as source material for other books as long as acknowledgement of the original author is given. Your support of this temporary distribution restriction plays an essential role in allowing the author and others like him to produce more quality books and software.

───

 

指南耶!!總是用得着乎??

 

 

 

 

 

 

 

 

 

 

 

 

 

W!o+ 的《小伶鼬工坊演義》︰神經網絡【Sigmoid】七

Michael Nielsen 先生所講的這段結論︰

Let me conclude this section by discussing a point that sometimes bugs people new to gradient descent. In neural networks the cost C is, of course, a function of many variables – all the weights and biases – and so in some sense defines a surface in a very high-dimensional space. Some people get hung up thinking: “Hey, I have to be able to visualize all these extra dimensions”. And they may start to worry: “I can’t think in four dimensions, let alone five (or five million)”. Is there some special ability they’re missing, some ability that “real” supermathematicians have? Of course, the answer is no. Even most professional mathematicians can’t visualize four dimensions especially well, if at all. The trick they use, instead, is to develop other ways of representing what’s going on. That’s exactly what we did above: we used an algebraic (rather than visual) representation of \Delta C to figure out how to move so as to decrease C. People who are good at thinking in high dimensions have a mental library containing many different techniques along these lines; our algebraic trick is just one example. Those techniques may not have the simplicity we’re accustomed to when visualizing three dimensions, but once you build up a library of such techniques, you can get pretty good at thinking in high dimensions. I won’t go into more detail here, but if you’re interested then you may enjoy reading this discussion of some of the techniques professional mathematicians use to think in high dimensions. While some of the techniques discussed are quite complex, much of the best content is intuitive and accessible, and could be mastered by anyone.

───

 

讓作者想起『二維生物』的故事。它們生活在一個很大的『球面』上,那麼它們可能『發現』那不是『平面』的嗎?或是它們也能用『三維』空間概念來表現的呢??若問『想像』一個『超曲面』

S(x_1, x_2, x_3, \cdots) = c

困難,還是理解伽羅瓦之『能想』困難︰

Permutations_RGB.svg

220px-15-Puzzle

220px-Symmetric_group_3;_Cayley_table;_matrices.svg

220px-Permutations_with_repetition.svg

就讓我們略窺一下『伽羅瓦』的思考法吧。假使 x_1,x_2,\cdots, x_n 是『多項式P(x) = \sum \limits_{k=0}^{k=n} c_k x^k = 0 的『』,此處係數 c_k 都是『有理數』。如果我們建構一個『對稱函數

f(x_1,x_2,x_3,\cdots,x_n) = (x-x_1)(x-x_2)(x-x_3)\cdots(x-x_n)

,將它展開後 c_n f(x_1,x_2,x_3,\cdots,x_n) 應該就是 P(x) 的吧。如果將這些『x_1,x_2,\cdots, x_n,作任意的『排列』permutation \begin{pmatrix} x_1 & x_2 & x_3 & \cdots & x_n \\ x_2 & x_n & x_4 & \cdots & x_1\end{pmatrix},此處是說上一排的『』的『位置』用下一排的『』來『置換』,由於 f 函數的特殊『形式』,我們會得到 f(x_1,x_2,x_3, \cdots,x_n)=f(x_2,x_n,x_4,\cdots,x_1)。事實上對於任意的『置換』,都會有 f(x_1,x_2,...,x_n)=f(x_2,x_1,\cdots,x_n)=f(x_3,x_1,\cdots,x_n,x_{n−1})。所以函數 f 稱之為『對稱函數』,這個『置換』的『不變性』就是『伽羅瓦』 主要研究的對象。舉例來說,考慮一個二次方程式 x^2 + A x + B = 0 有兩個根 \lambda_1, \lambda_2F(\lambda_1, \lambda_2) = (x - \lambda_1)(x - \lambda_2)
= x^2 -(\lambda_1 + \lambda_2) x +  \lambda_1 \cdot \lambda_2
= x^2 + A x + B
,比對後得到
\lambda_1 + \lambda_2 = -A,和
\lambda_1 \cdot \lambda_2 = B
。這兩個『代數式』對於 \lambda_1, \lambda_2 來講,也是『對稱的』,如果將它們看成兩個變數的『聯立方程組』,化簡後所得到的也定然就是『對等的』二次方程式 {\lambda_1}^2 + A \lambda_1 + B = 0{\lambda_2}^2 + A \lambda_2 + B = 0

這產生了很重要的結果,假使 \lambda_1 = a + b \sqrt{Q} 是方程式的一個根,假設另一個根是 \lambda_2 = c + d \sqrt{Q},由於
\lambda_1 + \lambda_2 = -A
\Longrightarrow  (a + b \sqrt{Q}) + (c + d \sqrt{Q}) = -A
\Longrightarrow  (a + c +A) + (b + d) \sqrt{Q} = 0
\therefore a + c + A = 0, \ b + d =0
,再由
\lambda_1 \cdot \lambda_2 = B
\Longrightarrow  (a + b \sqrt{Q}) \cdot (c + d \sqrt{Q}) = B
\Longrightarrow (a c + b d Q - B) + (a d + b c) \sqrt{Q}) =0
\therefore a c + b d Q = B, \ a d + b c =0
。因此得到 d = -b, \ c = a。而且 a = - \frac{A}{2}, \ b \sqrt{Q} = \frac{\sqrt{A^2 - 4B}}{2}。也就是說這兩個根是熟悉的 \frac{- A + \sqrt{A^2 - 4B}}{2}\frac{- A - \sqrt{A^2 - 4B}}{2}。於是一個『對稱函數f(x_1,x_2,x_3,\cdots,x_n) = (x-x_1)(x-x_2)(x-x_3)\cdots(x-x_n) 如果某一個根 x_ka + b \sqrt{Q} 的形式﹐那麼必然有另一個根 x_ja - b \sqrt{Q} 的形式,這就是由於那個『多項式』的係數是『有理數』的原故,它的『二次方根』的解,總是『成對』出現的啊!於是『二次方根』解的個數也必然是『偶數』的了!!

如 果我們探討一個三次方程式 x^3 + A x^2 + B x + C = 0 有三個根 \lambda_1, \lambda_2, \lambda_3 的情況,此時 F(\lambda_1, \lambda_2, \lambda_3) = (x - \lambda_1)(x - \lambda_2)(x - \lambda_3),假使說 \lambda_1, \lambda_2 是一對『二次方根』的解,那麼 \lambda_3 就必然是『有理數』。而 且從 x \approx +\infty \Longrightarrow F(\lambda_1, \lambda_2, \lambda_3) \approx +\inftyx \approx -\infty \Longrightarrow F(\lambda_1, \lambda_2, \lambda_3) \approx -\infty 來看,三次方程式至少有一個實數解。如果我們用 x = z - \frac{A}{3}  來消去 A x^2 這個『平方項

{(z - \frac{A}{3})}^3 + A {(z - \frac{A}{3})}^2 + B {(z - \frac{A}{3})} + C

= \left( z^3 - z^2 A + \frac{z A^2}{3} - \frac{A^3}{27} \right) + A \left( z^2 - \frac{2 z A}{3} + \frac{A^2}{9} \right) + B \left( z - \frac{A}{3} \right) + C

= z^3 + \left(- \frac{A^2}{3} + B \right) z + \left( \frac{2 A^3}{27} - \frac{B A}{3} + C \right)

\equiv_{df} z^3 + p z + q

。那麼為什麼要消去『平方項』的呢?如果考察

F(\lambda_1, \lambda_2, \lambda_3) = (x - \lambda_1)(x - \lambda_2)(x - \lambda_3)

= x^3 - (\lambda_1 + \lambda_2 + \lambda_3) x^2 + \left[\lambda_1 \cdot \lambda_2  + \lambda_3 \cdot (\lambda_1 + \lambda_2) \right]  x - \lambda_1 \lambda_2 \lambda_3

,當 『平方項』為『』時,- (\lambda_1 + \lambda_2 + \lambda_3) = 0,這建議著 \lambda_3 = (- \lambda_1) + (- \lambda_2) =  u + v,也就是說有一個根可以表示成『特殊兩數』之和。假使我們將 z = u + v 代入方程式,得到

(u + v)^3 + p(u + v) + q = 0

= (u^3 + v^3 + q) + (u + v)(3uv + p) = 0,假使『選擇3uv + p = 0,又可以得到 u^3 + v^3 + q = 0,是這一組 u,v 所滿足的『聯立方程式』,可以將它改寫成

uv = - \frac{p}{3} \Longrightarrow u^3 \cdot v^3 = - \frac{p^3}{27},與

u^3 + v^3 = -q ,這卻正是說 u^3, v^3 是一個『二次方程式』的根。求解後可以得到

u^{3}=-{q\over 2} + \sqrt{{q^{2}\over 4}+{p^{3}\over 27}}, \  v^{3}=-{q\over 2} - \sqrt{{q^{2}\over 4}+{p^{3}\over 27}}

, 所以 z=u+v=\sqrt[3]{-{q\over 2}+ \sqrt{{q^{2}\over 4}+{p^{3}\over 27}}} +\sqrt[3]{-{q\over 2}- \sqrt{{q^{2}\over 4}+{p^{3}\over 27}}}。然而三次方程式不是應該有三個解的嗎?假使 \frac{q^2}{4}+\frac{p^3}{27} >0,那個 『根號\sqrt{{q^{2}\over 4}+{p^{3}\over 27}}  是正實數,因此 u^3, v^3 也是實數,而 z 是兩個『立方根』 之和,所以也是實數。那麼要如何求得另外兩個解的呢?假使設想 y^3 = \alpha = y^3 \cdot w^3w^3 = 1,然而 w^3 - 1 = (w - 1)(w^2 + w +1) = 0,所以可以解得 w = 1, \ w = \omega = \frac{-1 + i \sqrt{3}}{2}, \  w = {\omega}^2 = \frac{-1 - i \sqrt{3}}{2},於是 y 的三個解是 \sqrt[3]{\alpha}, \sqrt[3]{\alpha} \ \omega, \sqrt[3]{\alpha} \  {\omega}^2,最終我們可以得到 (u, v)(u \ \omega, v \  { \omega}^2),以及 (u \ {\omega}^2, v \ \omega) 三組答案,現今人們將這個方法叫做『卡爾達諾法』。『吉羅拉莫‧卡爾達諾』 Girolamo Cardano 是意大利文藝復興時期百科全書式的學者,主要成就在數學、物理、醫學方面。在一五四五年出版的《大術》一書中,他首先發表了三次方程式的一般解法。然而就數學史而言,真正發現此三次代數方程式解法的或許是『尼科洛‧塔塔利亞』 Niccolò Tartaglia,兩人也因此而結怨。這本書中還記載了四次代數方程的一般解法,其實是由他的學生『費拉里』所發現的?這簡直是特別為那個『實驗哲學』  x-phi 所舉的例子的吧!!

── 摘自《【Sonic π】電路學之補充《四》無窮小算術‧中下下‧中

 

總是見仁見智的吧!!

到底是誰?為什麼會開始將 28 \times 28 = 784 看成『行向量』的耶?所謂『手寫阿拉伯數字』難道不是『二維圖象』的乎??