隨著 Michael Nielsen 先生介紹『mnist_loader.py』程式,第一章也將步入尾聲。正好借此機緣,說說如何用 Python 『struct』程式庫
7.3. struct
— Interpret strings as packed binary data
This module performs conversions between Python values and C structs represented as Python strings. This can be used in handling binary data stored in files or from network connections, among other sources. It uses Format Strings as compact descriptions of the layout of the C structs and the intended conversion to/from Python values.
Note
By default, the result of packing a given C struct includes pad bytes in order to maintain proper alignment for the C types involved; similarly, alignment is taken into account when unpacking. This behavior is chosen so that the bytes of a packed struct correspond exactly to the layout in memory of the corresponding C struct. To handle platform-independent data formats or omit implicit pad bytes, use standard
size and alignment instead of native
size and alignment: see Byte Order, Size, and Alignment for details.
……
7.3.2.1. Byte Order, Size, and Alignment
By default, C types are represented in the machine’s native format and byte order, and properly aligned by skipping pad bytes if necessary (according to the rules used by the C compiler).
Alternatively, the first character of the format string can be used to indicate the byte order, size and alignment of the packed data, according to the following table:
Character | Byte order | Size | Alignment |
---|---|---|---|
@ |
native | native | native |
= |
native | standard | none |
< |
little-endian | standard | none |
> |
big-endian | standard | none |
! |
network (= big-endian) | standard | none |
If the first character is not one of these, '@'
is assumed.
……
7.3.2.2. Format Characters
Format characters have the following meaning; the conversion between C and Python values should be obvious given their types. The ‘Standard size’ column refers to the size of the packed value in bytes when using standard size; that is, when the format string starts with one of '<'
, '>'
, '!'
or '='
. When using native size, the size of the packed value is platform-dependent.
Format | C Type | Python type | Standard size | Notes |
---|---|---|---|---|
x |
pad byte | no value | ||
c |
char |
string of length 1 | 1 | |
b |
signed char |
integer | 1 | (3) |
B |
unsigned char |
integer | 1 | (3) |
? |
_Bool |
bool | 1 | (1) |
h |
short |
integer | 2 | (3) |
H |
unsigned short |
integer | 2 | (3) |
i |
int |
integer | 4 | (3) |
I |
unsigned int |
integer | 4 | (3) |
l |
long |
integer | 4 | (3) |
L |
unsigned long |
integer | 4 | (3) |
q |
long long |
integer | 8 | (2), (3) |
Q |
unsigned long long |
integer | 8 | (2), (3) |
f |
float |
float | 4 | (4) |
d |
double |
float | 8 | (4) |
s |
char[] |
string | ||
p |
char[] |
string | ||
P |
void * |
integer | (5), (3) |
───
讀取原始『MNIST』之手寫阿拉伯數字資料庫︰
FILE FORMATS FOR THE MNIST DATABASE
TRAINING SET LABEL FILE (train-labels-idx1-ubyte):
[offset] [type] [value] [description]
0000 32 bit integer 0x00000801(2049) magic number (MSB first)
0004 32 bit integer 60000 number of items
0008 unsigned byte ?? label
0009 unsigned byte ?? label
........
xxxx unsigned byte ?? label
The labels values are 0 to 9.
TRAINING SET IMAGE FILE (train-images-idx3-ubyte):
[offset] [type] [value] [description]
0000 32 bit integer 0x00000803(2051) magic number
0004 32 bit integer 60000 number of images
0008 32 bit integer 28 number of rows
0012 32 bit integer 28 number of columns
0016 unsigned byte ?? pixel
0017 unsigned byte ?? pixel
........
xxxx unsigned byte ?? pixel
Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).
TEST SET LABEL FILE (t10k-labels-idx1-ubyte):
[offset] [type] [value] [description]
0000 32 bit integer 0x00000801(2049) magic number (MSB first)
0004 32 bit integer 10000 number of items
0008 unsigned byte ?? label
0009 unsigned byte ?? label
........
xxxx unsigned byte ?? label
The labels values are 0 to 9.
TEST SET IMAGE FILE (t10k-images-idx3-ubyte):
[offset] [type] [value] [description]
0000 32 bit integer 0x00000803(2051) magic number
0004 32 bit integer 10000 number of images
0008 32 bit integer 28 number of rows
0012 32 bit integer 28 number of columns
0016 unsigned byte ?? pixel
0017 unsigned byte ?? pixel
........
xxxx unsigned byte ?? pixel
Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).
───
希望短短的幾行互動程式足以盡其意也︰
>>> import struct >>> import numpy as np >>> with open("train-images.idx3-ubyte","rb") as imagefile: ... magic, ni, nr, nc = struct.unpack(">IIII", imagefile.read(16)) ... images = np.fromfile(imagefile, dtype=np.uint8).reshape(60000,784) ... >>> len(images) 60000 >>> type(images[0]) <type 'numpy.ndarray'> >>> len((images[0])) 784 >>> with open("train-labels.idx1-ubyte", "rb") as labelfile: ... magic, ni = struct.unpack(">II", labelfile.read(8)) ... labels = np.fromfile(labelfile, dtype=np.uint8) ... >>> len(labels) 60000 >>> labels[0] 5 >>> import matplotlib.pyplot as plt >>> img = images[0].reshape(28,28) >>> plt.imshow(img,cmap='Greys', interpolation='nearest') <matplotlib.image.AxesImage object at 0x30f7b10> >>> plt.show() >>>