制作mnist格式数据集
时间: 2023-09-08 13:09:25 浏览: 50
MNIST(Modified National Institute of Standards and Technology)是一个手写数字识别的经典数据集,包含60,000个训练样本和10,000个测试样本,每个样本都是一个28x28像素的灰度图像。以下是创建MNIST格式数据集的步骤:
1. 下载MNIST数据集
可以从官网下载MNIST数据集,也可以使用TensorFlow等深度学习框架内置的MNIST数据集。下载后的数据集包含四个文件:
- train-images-idx3-ubyte.gz:训练集图像
- train-labels-idx1-ubyte.gz:训练集标签
- t10k-images-idx3-ubyte.gz:测试集图像
- t10k-labels-idx1-ubyte.gz:测试集标签
2. 解压数据集
使用gzip库解压缩数据集文件:
```python
import gzip
def extract_data(filename, num_data, data_size, offset):
with gzip.open(filename) as f:
f.read(offset)
buf = f.read(data_size * num_data)
data = np.frombuffer(buf, dtype=np.uint8).astype(np.float32)
return data.reshape(num_data, data_size)
train_images = extract_data('train-images-idx3-ubyte.gz', 60000, 784, 16)
train_labels = extract_data('train-labels-idx1-ubyte.gz', 60000, 1, 8)
test_images = extract_data('t10k-images-idx3-ubyte.gz', 10000, 784, 16)
test_labels = extract_data('t10k-labels-idx1-ubyte.gz', 10000, 1, 8)
```
3. 将数据集转为TFRecord格式
TFRecord格式是一种二进制格式,可以更高效地存储和读取数据集。可以使用TensorFlow内置的tf.data.Dataset API将数据集转为TFRecord格式:
```python
import tensorflow as tf
def write_tfrecord(images, labels, filename):
with tf.io.TFRecordWriter(filename) as writer:
for i in range(images.shape[0]):
image_raw = images[i].tostring()
example = tf.train.Example(features=tf.train.Features(feature={
'image': tf.train.Feature(bytes_list=tf.train.BytesList(value=[image_raw])),
'label': tf.train.Feature(int64_list=tf.train.Int64List(value=[labels[i]]))
}))
writer.write(example.SerializeToString())
write_tfrecord(train_images, train_labels, 'train.tfrecord')
write_tfrecord(test_images, test_labels, 'test.tfrecord')
```
4. 读取TFRecord格式数据集
可以使用TensorFlow内置的tf.data.Dataset API读取TFRecord格式数据集:
```python
def read_tfrecord(filename):
feature_description = {
'image': tf.io.FixedLenFeature([], tf.string),
'label': tf.io.FixedLenFeature([], tf.int64)
}
def _parse_example(example_string):
feature_dict = tf.io.parse_single_example(example_string, feature_description)
image = tf.io.decode_raw(feature_dict['image'], tf.uint8)
image = tf.cast(image, tf.float32) / 255.0
image = tf.reshape(image, [28, 28, 1])
label = tf.cast(feature_dict['label'], tf.int32)
return image, label
dataset = tf.data.TFRecordDataset(filename)
dataset = dataset.map(_parse_example)
return dataset
train_dataset = read_tfrecord('train.tfrecord')
test_dataset = read_tfrecord('test.tfrecord')
```