My Dataset looks like this:
dataset1 = tf.data.Dataset.from_tensor_slices((
tf.random.uniform([4, 100], maxval=100, dtype=tf.int32),
tf.random.uniform([4])))
for record in dataset1.take(2):
print(record)
print(type(record))
(<tf.Tensor: shape=(100,), dtype=int32, numpy=
array([28, 96, 6, 22, 36, 33, 34, 29, 20, 77, 40, 82, 45, 81, 62, 59, 30,
86, 44, 17, 43, 32, 19, 32, 96, 24, 14, 65, 43, 59, 0, 96, 20, 17,
54, 31, 88, 72, 88, 55, 57, 63, 92, 50, 95, 76, 99, 63, 95, 82, 22,
36, 87, 56, 44, 29, 12, 45, 82, 27, 56, 32, 44, 66, 77, 99, 97, 58,
52, 81, 42, 54, 78, 3, 29, 86, 59, 98, 67, 39, 25, 27, 16, 46, 68,
81, 72, 30, 53, 95, 33, 71, 93, 82, 95, 55, 13, 53, 30, 21],
dtype=int32)>, <tf.Tensor: shape=(), dtype=float32, numpy=0.42071342>)
(<tf.Tensor: shape=(100,), dtype=int32, numpy=
array([71, 52, 9, 25, 94, 45, 64, 56, 99, 92, 62, 96, 13, 97, 39, 10, 27,
41, 81, 37, 38, 20, 77, 11, 26, 28, 55, 99, 50, 7, 89, 2, 66, 64,
11, 97, 4, 30, 34, 20, 81, 86, 68, 84, 75, 4, 22, 35, 87, 44, 57,
94, 27, 19, 60, 37, 38, 83, 39, 75, 65, 80, 97, 72, 20, 69, 35, 20,
37, 5, 60, 11, 84, 46, 25, 30, 13, 74, 5, 82, 34, 1, 79, 91, 41,
83, 94, 80, 79, 6, 3, 26, 84, 20, 53, 78, 93, 36, 54, 44],
dtype=int32)>, <tf.Tensor: shape=(), dtype=float32, numpy=0.73927164>)
<class 'tuple'>
So each record is a tuple of two Tensors, one is an input and another is an output to a model. I am trying to convert this Dataset into a .tfrecord
file which requires me to make an Example
out of each record. Here is my attempt:
def _bytes_feature(value):
"""Returns a bytes_list from a string / byte."""
if isinstance(value, type(tf.constant(0))):
value = value.numpy() # BytesList won't unpack a string from an EagerTensor.
return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))
def _float_feature(value):
"""Returns a float_list from a float / double."""
return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))
def serialize_example(feature1, feature2):
feature = {
'feature1': _bytes_feature(tf.io.serialize_tensor(feature1)),
'feature2': _float_feature(feature2),
}
example_proto = tf.train.Example(features=tf.train.Features(feature=feature))
return example_proto.SerializeToString()
When I do dataset1.map(serialize_example)
, I expect my code to work before doing
writer = tf.data.experimental.TFRecordWriter(some_path)
writer.write(dataset1)
However, I get the following error when I try dataset1.map(serialize_example)
:
...
value = value.numpy() # BytesList won't unpack a string from an EagerTensor.
AttributeError: 'Tensor' object has no attribute 'numpy'
How should I convert this dataset into a .tfrecord
file?
from Serialize a tf.train.Dataset with Tensor features into a tfrecord file?
No comments:
Post a Comment