什麼是事物的『特徵』呢？為什麼它的『提取方法』很重要？維基百科詞條這麼說︰

Feature extraction

In machine learning, pattern recognition and in image processing, feature extraction starts from an initial set of measured data and builds derived values (features) intended to be informative and non-redundant, facilitating the subsequent learning and generalization steps, and in some cases leading to better human interpretations. Feature extraction is related to dimensionality reduction.

When the input data to an algorithm is too large to be processed and it is suspected to be redundant (e.g. the same measurement in both feet and meters, or the repetitiveness of images presented as pixels), then it can be transformed into a reduced set of features (also named a features vector). This process is called feature selection. The selected features are expected to contain the relevant information from the input data, so that the desired task can be performed by using this reduced representation instead of the complete initial data.

───

 

假使考慮如何『定義』事物耶？也許『特徵』就是『界定性徵』，可以用來『區分』相異的東西！所以人們自然懂得『汪星人』不同於『喵星人』的也！！

於是乎好奇那『聲音』本有『調子』，可以用

Cepstrum

A cepstrum (/ˈkɛpstrəmˈˌˈsɛpstrəmˈ/) is the result of taking the Inverse Fourier transform (IFT) of the logarithm of the estimated spectrum of a signal. It may be pronounced in the two ways given, the second having the advantage of avoiding confusion with ‘kepstrum’ which also exists (see below). There is a complex cepstrum, a real cepstrum, a power cepstrum, and a phase cepstrum. The power cepstrum in particular finds applications in the analysis of human speech.

The name “cepstrum” was derived by reversing the first four letters of “spectrum”. Operations on cepstra are labelled quefrency analysis (aka quefrency alanysis^[1]), liftering, or cepstral analysis.

Steps in forming cepstrum from time history

───

 

來探討。那麼『圖象』可有『調子』乎？！能否依樣畫葫蘆來研究的呢！？不管『笨鳥先飛』、『菜鳥忘飛』、『老鳥已飛』……… 科技史裡滿載『傻問題』之『大成就』矣！！？？何不就效法一下嘛？？！！

【還是用五】

 

>>> img = training_data[0][0].reshape(28,28)
>>> f_img = network.np.fft.rfft2(img)
>>> logp_img = 2*network.np.log(network.np.abs(f_img))
>>> plt.imshow(logp_img)
<matplotlib.image.AxesImage object at 0x51af290>
>>> plt.show()

 

 

>>> ilogpf_img = network.np.fft.irfft2(logp_img)
>>> cf_img = network.np.abs(ilogpf_img)**2
>>> plt.imshow(cf_img)
<matplotlib.image.AxesImage object at 0x51d1050>
>>> plt.show()

 

 

【依舊選零】

 

>>> img1 = training_data[1][0].reshape(28,28)
>>> f_img1 = network.np.fft.rfft2(img1)
>>> logp_img1 = 2*network.np.log(network.np.abs(f_img1))
>>> plt.imshow(logp_img1)
<matplotlib.image.AxesImage object at 0x5091e50>
>>> plt.show()

 

 

>>> ilogpf_img1 = network.np.fft.irfft2(logp_img1)
>>> cf_img1 = network.np.abs(ilogpf_img1)**2
>>> plt.imshow(cf_img1)
<matplotlib.image.AxesImage object at 0x51b9b50>
>>> plt.show()
>>> 

 

 

【參考資料】

Discrete Fourier Transform (numpy.fft)

───

 

竟然會看起來很像？？似乎又有點不一樣！！到底該說是『行』還是『不行』的呀？？？

 

 

 

 

 

 

 

 

 

 

 

 

 

若說身在『影像處理』之領域不知道

快速傅立葉變換

快速傅立葉變換（英語：Fast Fourier Transform, FFT），是計算序列的離散傅立葉變換（DFT）或其逆變換的一種演算法。傅立葉分析將訊號從原始域（通常是時間或空間）轉換到頻域的表示或者逆過來轉換。FFT會通過把DFT矩陣分解為稀疏（大多為零）因子之積來快速計算此類變換。^[1] 因此，它能夠將計算DFT的複雜度從只用DFT定義計算需要的 ，降低到 ，其中 為資料大小。

快速傅立葉變換廣泛的應用於工程、科學和數學領域。這裡的基本思想在1965年得到才普及，但早在1805年就已推匯出來。^[2] 1994年吉爾伯特·斯特朗把FFT描述為「我們一生中最重要的數值演算法」^[3]，它還被IEEE科學與工程計算期刊列入20世紀十大演算法。^[4]

───

 

大概不可思議！！若問『手寫阿拉伯數字辨識』能不能用手寫數字 Spatial Domain 『空間域』來處理，誠是『大哉問』耶？？

就讓我們略窺一下那個『空間域』的圖像︰

>>> import mnist_loader
>>> training_data, validation_data, test_data = \
... mnist_loader.load_data_wrapper()
>>> import network
>>> net = network.Network([784, 30, 10])
>>> npzfile = network.np.load("swb.npz")
>>> net.weights[0] = npzfile["w1"]
>>> net.weights[1] = npzfile["w2"]
>>> net.biases[0] = npzfile["b1"]
>>> net.biases[1] = npzfile["b2"]
>>> import matplotlib.pyplot as plt
>>> img = training_data[0][0].reshape(28,28)
>>> plt.imshow(img,cmap='Greys', interpolation='nearest')
<matplotlib.image.AxesImage object at 0x56e33d0>
>>> plt.show()
>>>

 

【5 之原圖】

 

 

>>> f_img = network.np.fft.fft2(img)
>>> sf_img = network.np.fft.fftshift(f_img)
>>> dbf_img = 20*network.np.log(network.np.abs(sf_img))
>>> plt.imshow(dbf_img, cmap='Greys', interpolation='nearest')
<matplotlib.image.AxesImage object at 0x570a150>
>>> plt.show()
>>>

 

【5 之 FFT db 頻譜】

 

 

>>> phase_img = network.np.angle(f_img)
>>> plt.imshow(phase_img, cmap='Greys', interpolation='nearest')
<matplotlib.image.AxesImage object at 0x51bd690>
>>> plt.show()
>>> 

 

【5 之 FFT phase 頻譜】

 

 

>>> iphase_img = network.np.fft.ifft2(phase_img)
>>> iphase_img_p = network.np.abs(iphase_img)
>>> plt.imshow(iphase_img_p, cmap='Greys', interpolation='nearest')
<matplotlib.image.AxesImage object at 0x51c0d90>
>>> plt.show()
>>>

 

【單從相位頻譜重建】

 

由於涉及『複數』 Complex number

複數，為實數的延伸，它使任一多項式方程式都有根。複數當中有個「虛數單位」，它是的一個平方根，即。任一複數都可表達為，其中及皆為實數，分別稱為複數之「實部」和「虛部」。

複數的發現源於三次方程的根的表達式。數學上，「複」字表明所討論的數體為複數，如複矩陣、複變函數等。

───

 

那個『學習法則』該怎麼建立的呢？有興趣者或可以到此一遊︰

Welcome

 

The Computational Intelligence Laboratory (CIL) is doing research in the areas of Complex-Valued Neural Networks and Intelligent Image Processing. The CIL is an integrated part of the College of Science, Technology, Engineering and Mathematics of Texas A&M University-Texarkana.

Our research on Complex-Valued Neural Networks is concentrated on the development of the Multi-Valued Neuron (MVN) and MVN-based neural networks paradigms.

Our research on Intelligent Image Processing is concentrated on applications of MVN-based neural networks in image processing and image recognition.

The Director of the Laboratory is Dr. Igor Aizenberg.

An NSF Grant Recipient in 2009-2012

───

 

Complex-Valued Neurons

Complex-Valued Neural Networks

The primarily CIL research area is Complex-Valued Neural Networks (CVNNs), mainly Multi-Valued Neurons and neural networks based on them.

Complex-Valued Neural Networks become increasingly popular. The use of complex-valued inputs/outputs, weights and activation functions make it possible to increase the functionality of a single neuron and of a neural network, to improve their performance and to reduce the training time.

The history of complex numbers shows that although it took a long time for them to be accepted (almost 300 years from the first reference to “imaginary numbers” by Girolamo Cardano in 1545 to Leonard Euler’s and Carl Friedrich Gauss’ works published in 1748 and 1831, respectively), they have become an integral part of engineering and mathematics. It is difficult to imagine today how signal processing, aerodynamics, hydrodynamics, energy science, quantum mechanics, circuit analysis, and many other areas of engineering and science could develop without complex numbers. It is a fundamental mathematical fact that complex numbers are a necessary and absolutely natural part of numerical world. Their necessity clearly follows from the Fundamental Theorem of Algebra, which states that every non-constant single-variable polynomial of degree n with complex coefficients has exactly n complex roots, if each root is counted up to its multiplicity.

Answering a question frequently asked by some “conservative” people, what one can get using complex-valued neural networks (“twice more” parameters, more computations, etc.), we may say that one may get the same as using the Fourier transform, but not just the Walsh transform in signal processing. There are many engineering problems in the modern world where complex-valued signals and functions of complex variables are involved and where they are unavoidable. Thus, to employ neural networks for their analysis, approximation, etc., the use of complex-valued neural networks is natural. However, even in the analysis of real-valued signals (for example, images or audio signals) one of the most frequently used approaches is frequency domain analysis, which immediately leads us to the complex domain. In fact, analyzing signal properties in the frequency domain, we see that each signal is characterized by magnitude and phase that carry different information about the signal. This fundamental fact was deeply discovered by A.V. Oppenheim and J.S. Lim in their paper “The importance of phase in signals”, IEEE Proceedings, v. 69, No 5, 1981,pp.: 529- 541. They have shown that the phase in the Fourier spectrum of a signal is much more informative than the magnitude: particularly in the Fourier spectrum of images, just phase contains the information about all shapes, edges, orientation of all objects.

This property can be illustrated by the following example. Let us consider two popular test images �Lena� and �Bridge�.

Lena	Bridge

 

Let us take their Fourier transforms and then let us swap magnitude and phase of their Fourier spectra combining the phase of �Lena� with the magnitude of �Bridge� and wise-versa. After taking the inverse Fourier transform we clearly realize that those images were restored whose phases were combined with the counterpart magnitude:

Restored from Lena Phase + Bridge Magnitude	Restored from Bridge phase + Lena Magnitude

 

Thus, in fact, phase contains information of what is represented by the corresponding signal. To use this information properly, the most appropriate solution is movement to the complex domain. Hence, one of the most important characteristics of Complex-Valued Neural Networks is the proper treatment of amplitude and phase information, e.g., the treatment of wave-related phenomena such as electromagnetism, light waves, quantum waves and oscillatory phenomenon.

───

 

也可讀讀

https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2011-42.pdf

 

多點了解乎！！？？

 

 

 

 

 

 

 

 

 

 

 

 

 

 

今天又是『五四』的了。不知那位『德』先生曾否來過？這位『賽』先生可曾長住？？卻見世界烽煙不斷！『人道精神』正慾火鍛鍊中！！想起

《論語》‧學而

子貢曰：貧而無諂，富而無驕，何如？

子曰：可也。未若貧而樂，富而好禮者也。

子貢曰：《詩》云：『如切如磋，如琢如磨。』其斯之謂與？

子曰：賜也，始可與言詩已矣！告諸往而知來者。

，感嘆『貪、嗔、痴』果是『娑婆世界』之現象耶？？！！

於此篇章之末，與其講 Michael Nielsen 先生做了個『總結』︰

Toward deep learning

While our neural network gives impressive performance, that performance is somewhat mysterious. The weights and biases in the network were discovered automatically. And that means we don’t immediately have an explanation of how the network does what it does. Can we find some way to understand the principles by which our network is classifying handwritten digits? And, given such principles, can we do better?

To put these questions more starkly, suppose that a few decades hence neural networks lead to artificial intelligence (AI). Will we understand how such intelligent networks work? Perhaps the networks will be opaque to us, with weights and biases we don’t understand, because they’ve been learned automatically. In the early days of AI research people hoped that the effort to build an AI would also help us understand the principles behind intelligence and, maybe, the functioning of the human brain. But perhaps the outcome will be that we end up understanding neither the brain nor how artificial intelligence works!

To address these questions, let’s think back to the interpretation of artificial neurons that I gave at the start of the chapter, as a means of weighing evidence. Suppose we want to determine whether an image shows a human face or not:

Credits: 1. Ester Inbar. 2. Unknown. 3. NASA, ESA, G. Illingworth, D. Magee, and P. Oesch (University of California, Santa Cruz), R. Bouwens (Leiden University), and the HUDF09 Team. Click on the images for more details.

……

The end result is a network which breaks down a very complicated question – does this image show a face or not – into very simple questions answerable at the level of single pixels. It does this through a series of many layers, with early layers answering very simple and specific questions about the input image, and later layers building up a hierarchy of ever more complex and abstract concepts. Networks with this kind of many-layer structure – two or more hidden layers – are called deep neural networks.

Of course, I haven’t said how to do this recursive decomposition into sub-networks. It certainly isn’t practical to hand-design the weights and biases in the network. Instead, we’d like to use learning algorithms so that the network can automatically learn the weights and biases – and thus, the hierarchy of concepts – from training data. Researchers in the 1980s and 1990s tried using stochastic gradient descent and backpropagation to train deep networks. Unfortunately, except for a few special architectures, they didn’t have much luck. The networks would learn, but very slowly, and in practice often too slowly to be useful.

Since 2006, a set of techniques has been developed that enable learning in deep neural nets. These deep learning techniques are based on stochastic gradient descent and backpropagation, but also introduce new ideas. These techniques have enabled much deeper (and larger) networks to be trained – people now routinely train networks with 5 to 10 hidden layers. And, it turns out that these perform far better on many problems than shallow neural networks, i.e., networks with just a single hidden layer. The reason, of course, is the ability of deep nets to build up a complex hierarchy of concepts. It’s a bit like the way conventional programming languages use modular design and ideas about abstraction to enable the creation of complex computer programs. Comparing a deep network to a shallow network is a bit like comparing a programming language with the ability to make function calls to a stripped down language with no ability to make such calls. Abstraction takes a different form in neural networks than it does in conventional programming, but it’s just as important.

───

 

不如說祇是個『勸學篇』︰

訊︰☿ 把酒飛斝是同道，欲法荀子《勸學篇》趁年少︰

君子曰：學不可以已。青，取之於藍而青於藍；冰，水為之而寒於水。〈以喻學則才過其本性也。〉木直中繩，輮以為輪，其曲中規，雖有槁暴，不復挺者，輮使之然也。〈輮，屈。槁，枯。曓，乾。挻，宜也。《晏子春秋》作「不復贏也」。〉故木受繩則直，金就礪則利，君子博學而日參省乎己，則知明而行無過矣。〈參，三也。曾子曰︰「日三省吾身。」知，讀爲智。行，下孟反。〉故不登高山，不知天之高也；不臨深谿，不知地之厚也；不聞先王之遺言，不知學問之大也。〈大，謂有益於人。〉干、越、夷、貉之子，生而同聲，長而異俗，教使之然也。〈干、越，猶言吳、越。《呂氏春秋》「荊有次非得寶劍於干、越」，高誘曰︰「吳邑也。」貉，東北夷。同聲，謂啼聲同。貉，莫革反。〉《詩》曰：「嗟爾君子，無恆安息。靖共爾位，好是正直。神之聽之，介爾景福。」〈《詩》，《小雅‧小明》之篇。靖，謀。介，助。景，大也。無恆安息，戒之不使懷安也。言能謀恭其位，好正宜之道，則神聽而助之福，引此詩以喻勤學也。〉神莫大於化道，福莫長於無禍。〈爲學則自化道，故神莫大焉。修身則自無禍，故福莫長焉。〉吾嘗終日而思矣，不如須臾之所學也，吾嘗跂而望矣，不如登高之博見也 。〈跂，舉足也。〉登高而招，臂非加長也，而見者遠；順風而呼，聲非加疾也，而聞者彰。假輿馬者，非利足也，而致千里；假舟楫者，非能水也，而絕江河。〈能，善。絶，過。〉君子生非異也，善假於物也。〈皆以喻修身在假於學。生非異，言與衆人同也。〉南方有鳥焉，名曰蒙鳩，以羽為巢而編之以髮，繫之葦苕，風至苕折，卵破子死。巢非不完也 ，所繫者然也。〈蒙 鳩，鷦鷯也。苕，葦之秀也，今巧婦鳥之巢至精密，多繫於葦竹之上是也。「蒙」當爲「蔑」。《方言》雲︰「鷦鷯，關而西謂之桑飛，或謂之蔑雀。」或曰︰一名 蒙鳩，亦以其愚也。言人不知學問，其所置身亦猶繫葦之危也。《說苑》︰「客謂孟嘗君曰︰『鷦鷯巢於葦苕，箸之髮毛，可謂完堅矣，大風至則苕折卵破子死者何 也？其所託者然也。』〉西方有木焉，名曰射干，莖長四寸，生於高山之上，而臨百仞之淵，木莖非能長也，所立者然也 。〈《本草》藥名有射干，一名烏扇。陶弘景雲︰「花白莖長，如射人之執竿。」又引阮公詩云「射干臨層城」，是生於高處也。據《本草》在《草部》中，又生南陽川穀，此雲「西方有木」，未詳。或曰︰「長四寸」卽是草，雲木，誤也。蓋生南陽，亦生西方也。射音夜。〉蓬生麻中，不扶而直。蘭槐之根是爲芷。其漸之滫，君子不近，庶人不服，其質非不美也，所漸者然也。〈蘭槐，香草，其根是爲芷也。《本草》︰「白芷一名白茝。」陶弘景雲︰「卽《離騷》所謂蘭茝也。」葢苗名蘭茝，根名芷也。弱槐當是蘭茝別名，故云「蘭槐之根是爲芷」也。漸，漬也，染也。滫，溺也。言雖香草，浸漬於溺中，則可惡也。漸，子廉反。滫，思酒反。〉故君子居必擇鄉，遊必就士，所以防邪僻而近中正也。物類之起，必有所始。榮辱之來，必象其德。肉腐出蟲，魚枯生蠹。怠慢忘身，禍災乃作。強自取柱，柔自取束。〈凡物強則以爲柱而任勞，柔則見束而約急，皆其自取也。〉邪穢在身，怨之所構。〈構，結也。言亦所自取。〉施薪若一，火就燥也；〈布薪於地，均若一，火就燥而焚之矣。〉平地若一，水就溼也。草木疇生，禽獸羣焉，物各從其類也。〈疇與儔同，類也。〉是故質的張而弓矢至焉，林木茂而斧斤至焉，〈所謂召禍也。質，射矦。的，正鵠也。〉樹成蔭而衆鳥息焉，醯酸而蜹聚焉。〈喻有德則慕之者衆。〉故言有召禍也，行有招辱也，君子慎其所立乎！〈禍福如此，不可不慎所立。所立，卽謂學也。〉

積土成山，風雨興焉；積水成淵，蛟龍生焉；積善成德，而神明自得，聖心備焉。〈神明自得，謂自通於神明。〉故不積蹞步，無以致千里；〈半步曰蹞。蹞與跬同。〉不積小流，無以成江海。騏驥一躍，不能十步；駑馬十駕，〈言駑馬十度引車，則亦及騏驥之一躍。據下雲「駑馬十駕，則亦及之」，此亦當同，疑脫一句。〉功在不舍。鍥而舍之，朽木不折；鍥而不舍 ，金石可鏤。〈言立功在於不舍。舍與捨同。鍥，刻也，苦結反。《春秋傳》曰「陽虎借邑人之車，鍥其軸」也。〉螾無爪牙之利，筋骨之強，上食埃土，下飲黃泉，用心一也。〈螾與蚓同，蚯蚓也。〉蟹八跪而二螯，非虵蟺之穴無可寄託者，用心躁也。〈跪，足也。《韓子》以刖足爲刖跪。螫，蟹首上如鉞者。許叔重《說文》雲「蟹六足二螫」也。〉是故無冥冥之志者無昭昭之明 ，無惛惛之事者無赫赫之功。〈冥冥、惛惛，皆專默精誠之謂也。〉行衢道者不至，事兩君者不容。〈《爾雅》雲︰「四達謂之衢。」孫炎雲︰「衢，交道四出也。」或曰︰衢道，兩道也。不至，不能有所至。下篇有「楊朱哭衢塗」。今秦俗猶以兩爲衢，古之遺言歟？〉目不能兩視而明，耳不能兩聽而聰。螣蛇無足而飛，〈《爾雅》云：「螣，螣蛇。」郭璞雲「龍類，能興雲霧而遊其中」也。〉梧鼠五技而窮。〈「梧鼠」當爲「鼫鼠」，蓋本誤爲「鼯」字，傳寫又誤爲「梧」耳。技，才能也。言技能雖多而不能如螣蛇專一，故窮。五技，謂能飛不能上屋，能緣不能窮木，能游不能渡谷，能穴不能掩身，能走不能先人。〉《詩》曰 ：「屍鳩在桑，其子七兮。淑人君子，其儀一兮。其儀一兮，心如結兮。」故君子結於一也。〈《詩》，《曹風‧屍鳩》之篇。毛雲︰「屍鳩，鴶鞠也。屍鳩之養七子，旦從上而下，暮從下而上，平均如一。善人君子，其執義亦當如屍鳩之一。執義一則用心堅固。」故曰「心如結」也。

─── 摘自《M♪o 之學習筆記本《編者跋》》

 

而那個『應用之道』尚待『切磋琢磨』乎！！？？

所 謂『相由心生』是說精神外顯的『形貌』從『用心方向』而來，這個『習焉不察』之內在『心相』，常可以用來分辨『行業』。一行有一行的規矩，百業有百業的訣 竅，入了行，從了業，自然帶有某種『氣息』的吧！如何才能夠不著『相』？若可『無所住』而生其『心』，那麼既無『我心』何來『我相』的呢！！

那麼這個《子之武城》一事，是否有個『前言』對上『後語』，可分出『對錯好壞』的呢？也許有個『禮樂』之『理』和『禮樂』之『用』的差別，想那『子游』為武城宰，採用『禮樂』教化之道，孔夫子卻『莞爾』笑，豈有不『以子之言，擊子之語』的哩！然而夫子所謂『戲之』果真是說『割雞焉用牛刀？』是錯了嗎？恐是不樂見『禮樂』被當作了『名器』的吧！就像到了宋代的『存天理，去人欲』，導致『死生事小，失節事大』，終演成『禮教殺人』之憾事 ！！於是

祇『能』這樣『用』，不『會』那樣『使』，終究難了『用大』之道 ── 無用而不通達 ── ，如如不動，應事而動，因事制宜。

正說著『以正治國』和『以奇用兵』，『為學之法』與『用學之法 』的不同，也須避免那『紙上談兵』之過。此事《孫子兵法》

地形‧第十

孫子曰：地形有通者、有掛者、有支者、有隘者、有險者、有遠者 。我可以往，彼可以來，曰通。通形者， 先居高陽，利糧道，以戰則利。可以往，難以返，曰掛。掛形者，敵無備，出而勝之，敵若有備，出而不勝，難以返，不利。我出而不利，彼出而不利，曰支 。支形 者，敵雖利我，我無出也，引而去之，令敵半出而擊之，利 。隘形者，我先居之，必盈之以待敵。若敵先居之，盈而勿從，不盈而從之。險形者，我先居之，必居高 陽以待敵；若敵先居之，引而去之，勿從也。遠形者，勢均，難以挑戰，戰而不利。凡此六者 ，地之道也，將之至任，不可不察也。

故兵有走者、有弛者、有陷者、有崩者、有亂者、有北者。凡此六者，非天之災，將之過也。夫勢均，以一擊十，曰走；卒強吏弱，曰馳；吏強卒弱，曰陷；大吏怒而不服，遇敵懟 而自戰，將不知其能，曰崩；將弱不嚴，教道不明，吏卒無常，陳兵縱橫，曰亂；將不能料敵，以少合衆，以弱擊強，兵無選鋒，曰北。凡此六者，敗之道也，將之 至任，不可不察也。

夫地形者，兵之助也。料敵制勝，計險厄遠近，上將之道也。知此而用戰者必勝，不知此而用戰者必敗。故戰道必勝，主曰無戰，必戰可也；戰道不勝，主曰必戰，無戰可也。故進不求名，退不避罪 ，唯民是保，而利合於主，國之寶也。

視卒如嬰兒，故可以與之赴深溪；視卒如愛子，故可與之俱死 。厚而不能使，愛而不能令，亂而不能治，譬若驕子，不可用也。

知吾卒之可以擊，而不知敵之不可擊，勝之半也；知敵之可擊，而不知吾卒之不可以擊，勝之半也；知敵之可擊，知吾卒之可以擊，而不知地形之不可以戰，勝之半也。故知兵者，動而不迷，舉而不窮。故曰：知彼知己，勝乃不殆；知天知地，勝乃可全。

講的好。

─── 摘自《字詞網絡︰ WordNet 《六》 相 □ 而用 ○ ！！》

 

 

 

 

 

 

 

 

 

 

 

 

 

 

對於那個七十四行 Python 小程式， Michael Nielsen 先生寫到︰

I said above that our program gets pretty good results. What does that mean? Good compared to what? It’s informative to have some simple (non-neural-network) baseline tests to compare against, to understand what it means to perform well. The simplest baseline of all, of course, is to randomly guess the digit. That’ll be right about ten percent of the time. We’re doing much better than that!

What about a less trivial baseline? Let’s try an extremely simple idea: we’ll look at how dark an image is. For instance, an image of a 2 will typically be quite a bit darker than an image of a 1, just because more pixels are blackened out, as the following examples illustrate:

This suggests using the training data to compute average darknesses for each digit, 0,1,2,…,9. When presented with a new image, we compute how dark the image is, and then guess that it’s whichever digit has the closest average darkness. This is a simple procedure, and is easy to code up, so I won’t explicitly write out the code – if you’re interested it’s in the GitHub repository. But it’s a big improvement over random guessing, getting 2,225 of the 10,000 test images correct, i.e., 22.25 percent accuracy.

It’s not difficult to find other ideas which achieve accuracies in the 20 to 50 percent range. If you work a bit harder you can get up over 50 percent. But to get much higher accuracies it helps to use established machine learning algorithms. Let’s try using one of the best known algorithms, the support vector machine or SVM. If you’re not familiar with SVMs, not to worry, we’re not going to need to understand the details of how SVMs work. Instead, we’ll use a Python library called scikit-learn, which provides a simple Python interface to a fast C-based library for SVMs known as LIBSVM.

If we run scikit-learn’s SVM classifier using the default settings, then it gets 9,435 of 10,000 test images correct. (The code is available here.) That’s a big improvement over our naive approach of classifying an image based on how dark it is. Indeed, it means that the SVM is performing roughly as well as our neural networks, just a little worse. In later chapters we’ll introduce new techniques that enable us to improve our neural networks so that they perform much better than the SVM.

That’s not the end of the story, however. The 9,435 of 10,000 result is for scikit-learn’s default settings for SVMs. SVMs have a number of tunable parameters, and it’s possible to search for parameters which improve this out-of-the-box performance. I won’t explicitly do this search, but instead refer you to this blog post by Andreas Mueller if you’d like to know more. Mueller shows that with some work optimizing the SVM’s parameters it’s possible to get the performance up above 98.5 percent accuracy. In other words, a well-tuned SVM only makes an error on about one digit in 70. That’s pretty good! Can neural networks do better?

In fact, they can. At present, well-designed neural networks outperform every other technique for solving MNIST, including SVMs. The current (2013) record is classifying 9,979 of 10,000 images correctly. This was done by Li Wan, Matthew Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. We’ll see most of the techniques they used later in the book. At that level the performance is close to human-equivalent, and is arguably better, since quite a few of the MNIST images are difficult even for humans to recognize with confidence, for example:

I trust you’ll agree that those are tough to classify! With images like these in the MNIST data set it’s remarkable that neural networks can accurately classify all but 21 of the 10,000 test images. Usually, when programming we believe that solving a complicated problem like recognizing the MNIST digits requires a sophisticated algorithm. But even the neural networks in the Wan et al paper just mentioned involve quite simple algorithms, variations on the algorithm we’ve seen in this chapter. All the complexity is learned, automatically, from the training data. In some sense, the moral of both our results and those in more sophisticated papers, is that for some problems:

sophisticated algorithm ≤ simple learning algorithm + good training data.───

 

假使說用『猜的』，恐怕講『百分之十』都只能是『想當然爾』吧 ，故而沒什麼可以多說的了！

至於說用『平均暗度』，即使僅從人寫字習慣上講︰

 

或大或小、或粗或細，可知其不可為！不過讀讀

"""
mnist_average_darkness
~~~~~~~~~~~~~~~~~~~~~~

A naive classifier for recognizing handwritten digits from the MNIST
data set.  The program classifies digits based on how dark they are
--- the idea is that digits like "1" tend to be less dark than digits
like "8", simply because the latter has a more complex shape.  When
shown an image the classifier returns whichever digit in the training
data had the closest average darkness.

The program works in two steps: first it trains the classifier, and
then it applies the classifier to the MNIST test data to see how many
digits are correctly classified.

Needless to say, this isn't a very good way of recognizing handwritten
digits!  Still, it's useful to show what sort of performance we get
from naive ideas."""

#### Libraries
# Standard library
from collections import defaultdict

# My libraries
import mnist_loader

def main():
    training_data, validation_data, test_data = mnist_loader.load_data()
    # training phase: compute the average darknesses for each digit,
    # based on the training data
    avgs = avg_darknesses(training_data)
    # testing phase: see how many of the test images are classified
    # correctly
    num_correct = sum(int(guess_digit(image, avgs) == digit)
                      for image, digit in zip(test_data[0], test_data[1]))
    print "Baseline classifier using average darkness of image."
    print "%s of %s values correct." % (num_correct, len(test_data[1]))

def avg_darknesses(training_data):
    """ Return a defaultdict whose keys are the digits 0 through 9.
    For each digit we compute a value which is the average darkness of
    training images containing that digit.  The darkness for any
    particular image is just the sum of the darknesses for each pixel."""
    digit_counts = defaultdict(int)
    darknesses = defaultdict(float)
    for image, digit in zip(training_data[0], training_data[1]):
        digit_counts[digit] += 1
        darknesses[digit] += sum(image)
    avgs = defaultdict(float)
    for digit, n in digit_counts.iteritems():
        avgs[digit] = darknesses[digit] / n
    return avgs

def guess_digit(image, avgs):
    """Return the digit whose average darkness in the training data is
    closest to the darkness of ``image``.  Note that ``avgs`` is
    assumed to be a defaultdict whose keys are 0...9, and whose values
    are the corresponding average darknesses across the training data."""
    darkness = sum(image)
    distances = {k: abs(v-darkness) for k, v in avgs.iteritems()}
    return min(distances, key=distances.get)

if __name__ == "__main__":
    main()

 

倒是滿有意思的。在樹莓派 3 上，實測結果如下︰

pi@raspberrypi:~/neural-networks-and-deep-learning/src 9,435 / 10,000 python mnist_svm.py 
Baseline classifier using an SVM.
9435 of 10000 values correct.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

隨著 Michael Nielsen 先生介紹『mnist_loader.py』程式，第一章也將步入尾聲。正好借此機緣，說說如何用 Python 『struct』程式庫

7.3. struct — Interpret strings as packed binary data

This module performs conversions between Python values and C structs represented as Python strings. This can be used in handling binary data stored in files or from network connections, among other sources. It uses Format Strings as compact descriptions of the layout of the C structs and the intended conversion to/from Python values.

……

7.3.2.1. Byte Order, Size, and Alignment

By default, C types are represented in the machine’s native format and byte order, and properly aligned by skipping pad bytes if necessary (according to the rules used by the C compiler).

Alternatively, the first character of the format string can be used to indicate the byte order, size and alignment of the packed data, according to the following table:

Character	Byte order	Size	Alignment
@	native	native	native
=	native	standard	none
<	little-endian	standard	none
>	big-endian	standard	none
!	network (= big-endian)	standard	none

If the first character is not one of these, '@' is assumed.

……

7.3.2.2. Format Characters

Format characters have the following meaning; the conversion between C and Python values should be obvious given their types. The ‘Standard size’ column refers to the size of the packed value in bytes when using standard size; that is, when the format string starts with one of '<', '>', '!' or '='. When using native size, the size of the packed value is platform-dependent.

Format	C Type	Python type	Standard size	Notes
x	pad byte	no value
c	char	string of length 1	1
b	signed char	integer	1	(3)
B	unsigned char	integer	1	(3)
?	_Bool	bool	1	(1)
h	short	integer	2	(3)
H	unsigned short	integer	2	(3)
i	int	integer	4	(3)
I	unsigned int	integer	4	(3)
l	long	integer	4	(3)
L	unsigned long	integer	4	(3)
q	long long	integer	8	(2), (3)
Q	unsigned long long	integer	8	(2), (3)
f	float	float	4	(4)
d	double	float	8	(4)
s	char[]	string
p	char[]	string
P	void *	integer		(5), (3)

───

 

讀取原始『MNIST』之手寫阿拉伯數字資料庫︰

FILE FORMATS FOR THE MNIST DATABASE

TRAINING SET LABEL FILE (train-labels-idx1-ubyte):

[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  60000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
........
xxxx     unsigned byte   ??               label

The labels values are 0 to 9.

TRAINING SET IMAGE FILE (train-images-idx3-ubyte):

[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000803(2051) magic number
0004     32 bit integer  60000            number of images
0008     32 bit integer  28               number of rows
0012     32 bit integer  28               number of columns
0016     unsigned byte   ??               pixel
0017     unsigned byte   ??               pixel
........
xxxx     unsigned byte   ??               pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).

TEST SET LABEL FILE (t10k-labels-idx1-ubyte):

[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  10000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
........
xxxx     unsigned byte   ??               label

The labels values are 0 to 9.

TEST SET IMAGE FILE (t10k-images-idx3-ubyte):

[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000803(2051) magic number
0004     32 bit integer  10000            number of images
0008     32 bit integer  28               number of rows
0012     32 bit integer  28               number of columns
0016     unsigned byte   ??               pixel
0017     unsigned byte   ??               pixel
........
xxxx     unsigned byte   ??               pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).

───

 

希望短短的幾行互動程式足以盡其意也︰

>>> import struct
>>> import numpy as np

>>> with open("train-images.idx3-ubyte","rb") as imagefile:
...     magic, ni, nr, nc = struct.unpack(">IIII", imagefile.read(16))
...     images = np.fromfile(imagefile, dtype=np.uint8).reshape(60000,784)
... 
>>> len(images)
60000
>>> type(images[0])
<type 'numpy.ndarray'>
>>> len((images[0]))
784

>>> with open("train-labels.idx1-ubyte", "rb") as labelfile:
...     magic, ni =  struct.unpack(">II", labelfile.read(8))
...     labels = np.fromfile(labelfile, dtype=np.uint8)
... 
>>> len(labels)
60000
>>> labels[0]
5

>>> import matplotlib.pyplot as plt
>>> img = images[0].reshape(28,28)
>>> plt.imshow(img,cmap='Greys', interpolation='nearest')
<matplotlib.image.AxesImage object at 0x30f7b10>
>>> plt.show()
>>>