使用 keras text_dataset_from_directory 时出现尺寸问题,我得到(无,)并且无法将其加载到模型中
我正在使用 keras.utils.text_dataset_from_directory (参见代码)。当我到达 model.fit 时,我收到有关输入尺寸的警告和错误(我认为与输出有关)。 代码:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers import Rescaling
MAX_VAL = 0.5
num_classes = 2
train_ds = keras.utils.text_dataset_from_directory(
directory='.../training_data/',
labels='inferred',
label_mode='categorical',
class_names=None,
batch_size=32,
max_length=None,
shuffle=True,
seed=None,
validation_split=None,
subset=None,
follow_links=False)
validation_ds = keras.utils.text_dataset_from_directory(
directory='.../validation_data/',
labels='inferred',
label_mode='categorical',
class_names=None,
batch_size=32,
max_length=None,
shuffle=True,
seed=None,
validation_split=None,
subset=None,
follow_links=False)
inputs = keras.Input(shape=(None,))
x = layers.Reshape((-1, 1))(inputs)
x = Rescaling(scale=1.0 / MAX_VAL)(x)
x = layers.Dense(32, activation="softmax")(x)
outputs = layers.Dense(num_classes, activation="softmax")(x)
model = keras.Model(inputs=inputs, outputs=outputs)
model.summary()
keras.utils.plot_model(model, "my_first_model.png")
model.compile(optimizer='Adam', loss='categorical_crossentropy')
history = model.fit(train_ds, epochs=10, validation_data=validation_ds)
输出:
Epoch 1/10
WARNING:tensorflow:Model was constructed with shape (None, None) for input KerasTensor(type_spec=TensorSpec(shape=(None, None), dtype=tf.float32, name='input_1'), name='input_1', description="created by layer 'input_1'"), but it was called on an input with incompatible shape (None,).
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-6-3656f32a72a9> in <module>
1 model.compile(optimizer='Adam', loss='categorical_crossentropy')
----> 2 history = model.fit(train_ds, epochs=10, validation_data=validation_ds)
~\AppData\Roaming\Python\Python38\site-packages\keras\utils\traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\func_graph.py in autograph_handler(*args, **kwargs)
1145 except Exception as e: # pylint:disable=broad-except
1146 if hasattr(e, "ag_error_metadata"):
-> 1147 raise e.ag_error_metadata.to_exception(e)
1148 else:
1149 raise
ValueError: in user code:
File "...\Python\Python38\site-packages\keras\engine\training.py", line 1021, in train_function *
return step_function(self, iterator)
File "...\Python\Python38\site-packages\keras\engine\training.py", line 1010, in step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "...\Python\Python38\site-packages\keras\engine\training.py", line 1000, in run_step **
outputs = model.train_step(data)
File "...\Python\Python38\site-packages\keras\engine\training.py", line 860, in train_step
loss = self.compute_loss(x, y, y_pred, sample_weight)
File "...\Python\Python38\site-packages\keras\engine\training.py", line 918, in compute_loss
return self.compiled_loss(
File "...\Python\Python38\site-packages\keras\engine\compile_utils.py", line 201, in __call__
loss_value = loss_obj(y_t, y_p, sample_weight=sw)
File "...\Python\Python38\site-packages\keras\losses.py", line 141, in __call__
losses = call_fn(y_true, y_pred)
File "...\Python\Python38\site-packages\keras\losses.py", line 245, in call **
return ag_fn(y_true, y_pred, **self._fn_kwargs)
File "...\Python\Python38\site-packages\keras\losses.py", line 1789, in categorical_crossentropy
return backend.categorical_crossentropy(
File "...\Python\Python38\site-packages\keras\backend.py", line 5083, in categorical_crossentropy
target.shape.assert_is_compatible_with(output.shape)
ValueError: Shapes (None, 2) and (None, 1, 2) are incompatible
模型的输入是保存在 .txt
文件中的一维向量(长度约为 27000): 0.101471743,0.099917953,0.103334975,0.099364908,0.099035715,...,0.097369999,0.099680934
从目录读取数据集,格式为:
/training_data/
...class_a/
......a_text_1.txt
......a_text_2.txt
...class_b/
......b_text_1.txt
......b_text_2.txt
/validation_data/
...class_a/
......a_text_1.txt
......a_text_2.txt
...class_b/
......b_text_1.txt
......b_text_2.txt
如何获得正确的尺寸?
编辑: 我将数据保存为 .jpg
文件,并使用 image_dataset_from_directory
加载它。这解决了这个问题。但我仍然想了解为什么我无法正确从 .txt
文件获取数据。 (我丢失了很多从 .jpg
中从浮点数据传输到 8 位 int 的数据。并且使用 image_dataset_from_directory
要求所有 img 大小具有相同的长度,我希望我的数据尺寸不同)。
I am using keras.utils.text_dataset_from_directory
(see code). when I reach model.fit
I get a warning about input dimentions and a error (that I think has something to do with the output).
code:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers import Rescaling
MAX_VAL = 0.5
num_classes = 2
train_ds = keras.utils.text_dataset_from_directory(
directory='.../training_data/',
labels='inferred',
label_mode='categorical',
class_names=None,
batch_size=32,
max_length=None,
shuffle=True,
seed=None,
validation_split=None,
subset=None,
follow_links=False)
validation_ds = keras.utils.text_dataset_from_directory(
directory='.../validation_data/',
labels='inferred',
label_mode='categorical',
class_names=None,
batch_size=32,
max_length=None,
shuffle=True,
seed=None,
validation_split=None,
subset=None,
follow_links=False)
inputs = keras.Input(shape=(None,))
x = layers.Reshape((-1, 1))(inputs)
x = Rescaling(scale=1.0 / MAX_VAL)(x)
x = layers.Dense(32, activation="softmax")(x)
outputs = layers.Dense(num_classes, activation="softmax")(x)
model = keras.Model(inputs=inputs, outputs=outputs)
model.summary()
keras.utils.plot_model(model, "my_first_model.png")
model.compile(optimizer='Adam', loss='categorical_crossentropy')
history = model.fit(train_ds, epochs=10, validation_data=validation_ds)
output:
Epoch 1/10
WARNING:tensorflow:Model was constructed with shape (None, None) for input KerasTensor(type_spec=TensorSpec(shape=(None, None), dtype=tf.float32, name='input_1'), name='input_1', description="created by layer 'input_1'"), but it was called on an input with incompatible shape (None,).
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-6-3656f32a72a9> in <module>
1 model.compile(optimizer='Adam', loss='categorical_crossentropy')
----> 2 history = model.fit(train_ds, epochs=10, validation_data=validation_ds)
~\AppData\Roaming\Python\Python38\site-packages\keras\utils\traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\func_graph.py in autograph_handler(*args, **kwargs)
1145 except Exception as e: # pylint:disable=broad-except
1146 if hasattr(e, "ag_error_metadata"):
-> 1147 raise e.ag_error_metadata.to_exception(e)
1148 else:
1149 raise
ValueError: in user code:
File "...\Python\Python38\site-packages\keras\engine\training.py", line 1021, in train_function *
return step_function(self, iterator)
File "...\Python\Python38\site-packages\keras\engine\training.py", line 1010, in step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "...\Python\Python38\site-packages\keras\engine\training.py", line 1000, in run_step **
outputs = model.train_step(data)
File "...\Python\Python38\site-packages\keras\engine\training.py", line 860, in train_step
loss = self.compute_loss(x, y, y_pred, sample_weight)
File "...\Python\Python38\site-packages\keras\engine\training.py", line 918, in compute_loss
return self.compiled_loss(
File "...\Python\Python38\site-packages\keras\engine\compile_utils.py", line 201, in __call__
loss_value = loss_obj(y_t, y_p, sample_weight=sw)
File "...\Python\Python38\site-packages\keras\losses.py", line 141, in __call__
losses = call_fn(y_true, y_pred)
File "...\Python\Python38\site-packages\keras\losses.py", line 245, in call **
return ag_fn(y_true, y_pred, **self._fn_kwargs)
File "...\Python\Python38\site-packages\keras\losses.py", line 1789, in categorical_crossentropy
return backend.categorical_crossentropy(
File "...\Python\Python38\site-packages\keras\backend.py", line 5083, in categorical_crossentropy
target.shape.assert_is_compatible_with(output.shape)
ValueError: Shapes (None, 2) and (None, 1, 2) are incompatible
The inputs to the model are 1d vectors (of length ~27000) saved in .txt
files:0.101471743,0.099917953,0.103334975,0.099364908,0.099035715,...,0.097369999,0.099680934
readin dataset frome dir formated as:
/training_data/
...class_a/
......a_text_1.txt
......a_text_2.txt
...class_b/
......b_text_1.txt
......b_text_2.txt
/validation_data/
...class_a/
......a_text_1.txt
......a_text_2.txt
...class_b/
......b_text_1.txt
......b_text_2.txt
How can I get the dimensions right?
EDIT:
I saved the data as a .jpg
file and loaded it using image_dataset_from_directory
. this fixed the issue. but I would still like to understand why I cant get the data from .txt
files properly. (I lose a lot of data transferring from float data to 8bit int in .jpg
. and using image_dataset_from_directory
requires all img size to be the same length, I wand my data to be different sizes).
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论