python读取mnist文件

From: https://www.cnblogs.com/x1957/archive/2012/06/02/2531503.html

从  http://yann.lecun.com/exdb/mnist/ 可以下载原始的文件。

train-images-idx3-ubyte.gz:  training set images (9912422 bytes)
train-labels-idx1-ubyte.gz: training set labels (28881 bytes)
t10k-images-idx3-ubyte.gz:  test set images (1648877 bytes)
t10k-labels-idx1-ubyte.gz:  test set labels (4542 bytes)

There are 4 files:

train-images-idx3-ubyte: training set images
train-labels-idx1-ubyte: training set labels
t10k-images-idx3-ubyte:  test set images
t10k-labels-idx1-ubyte:  test set labels

The training set contains 60000 examples, and the test set 10000 examples.

The first 5000 examples of the test set are taken from the originalNIST training set. The last 5000 are taken from the original NIST testset. The first 5000 are cleaner and easier than the last 5000.

TRAINING SET LABEL FILE (train-labels-idx1-ubyte):

[offset] [type]         [value]          [description]
0000     32 bit integer  0x00000801(2049)magic number (MSB first)
0004     32 bit integer  60000           number of items
0008     unsigned byte   ??              label
0009     unsigned byte   ??              label
........
xxxx     unsigned byte   ??              label

The labels values are 0 to 9.

TRAINING SET IMAGE FILE (train-images-idx3-ubyte):

[offset] [type]         [value]          [description]
0000     32 bit integer  0x00000803(2051)magic number
0004     32 bit integer  60000           number of images
0008     32 bit integer  28              number of rows
0012     32 bit integer  28              number of columns
0016     unsigned byte   ??              pixel
0017     unsigned byte   ??              pixel
........
xxxx     unsigned byte   ??              pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background(white), 255 means foreground (black).

TEST SET LABEL FILE (t10k-labels-idx1-ubyte):

[offset] [type]         [value]          [description]
0000     32 bit integer  0x00000801(2049)magic number (MSB first)
0004     32 bit integer  10000           number of items
0008     unsigned byte   ??              label
0009     unsigned byte   ??              label
........
xxxx     unsigned byte   ??              label

The labels values are 0 to 9.

TEST SET IMAGE FILE (t10k-images-idx3-ubyte):

[offset] [type]         [value]          [description]
0000     32 bit integer  0x00000803(2051)magic number
0004     32 bit integer  10000           number of images
0008     32 bit integer  28              number of rows
0012     32 bit integer  28              number of columns
0016     unsigned byte   ??              pixel
0017     unsigned byte   ??              pixel
........
xxxx     unsigned byte   ??              pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background(white), 255 means foreground (black).
 

THE IDX FILE FORMAT

the IDX file format is a simple format for vectors and multidimensionalmatrices of various numerical types.

The basic format is

magic number
size in dimension 0
size in dimension 1
size in dimension 2
.....
size in dimension N
data

The magic number is an integer (MSB first). The first 2 bytes are always0.

The third byte codes the type of the data:
0x08: unsigned byte
0x09: signed byte
0x0B: short (2 bytes)
0x0C: int (4 bytes)
0x0D: float (4 bytes)
0x0E: double (8 bytes)

The 4-th byte codes the number of dimensions of the vector/matrix: 1for vectors, 2 for matrices....

The sizes in each dimension are 4-byte integers (MSB first, high endian,like in most non-Intel processors).

The data is stored like in a C array, i.e. the index in the last dimensionchanges the fastest.

python读取mnist文件其实就是python怎么读取binnary file。mnist的结构如下，选取train-images

TRAINING SET IMAGE FILE (train-images-idx3-ubyte):

[offset] [type]          [value]          [description] 
0000     32 bit integer  0x00000803(2051) magic number 
0004     32 bit integer  60000            number of images 
0008     32 bit integer  28               number of rows 
0012     32 bit integer  28               number of columns 
0016     unsigned byte   ??               pixel 
0017     unsigned byte   ??               pixel 
........ 
xxxx     unsigned byte   ??               pixel

也就是之前我们要读取4个 32 bit integer

试过很多方法，觉得最方便的，至少对我来说还是使用

struct.unpack_from()

 先使用二进制方式把文件都读进来

 然后使用struc.unpack_from

‘>IIII‘是说使用大端法读取4个unsinged int32

然后读取一个图片测试是否读取成功

 ‘>784B‘的意思就是用大端法读取784个unsigned byte

import numpy as np
import struct
import matplotlib.pyplot as plt

filename = ‘train-images.idx3-ubyte‘
binfile = open(filename, ‘rb‘)
buf = binfile.read()

index = 0
magic, numImages, numRows, numColumns = struct.unpack_from(‘>IIII‘, buf, index)
index += struct.calcsize(‘>IIII‘)

im = struct.unpack_from(‘>784B‘, buf, index)
index += struct.calcsize(‘>784B‘)

im = np.array(im)
im = im.reshape(28, 28)

fig = plt.figure()
plotwindow = fig.add_subplot(111)
plt.imshow(im, cmap=‘gray‘)
plt.show()

### 其实就是python怎么读取binnary file
### https://www.cnblogs.com/x1957/archive/2012/06/02/2531503.html
###