使用 keras text_dataset_from_directory 时出现尺寸问题,我得到(无,)并且无法将其加载到模型中

发布于 01-19 06:31 字数 5113 浏览 3 评论 0原文

我正在使用 keras.utils.text_dataset_from_directory (参见代码)。当我到达 model.fit 时,我收到有关输入尺寸的警告和错误(我认为与输出有关)。 代码:

from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers import Rescaling
MAX_VAL = 0.5
num_classes = 2

train_ds = keras.utils.text_dataset_from_directory(
    directory='.../training_data/',
    labels='inferred',
    label_mode='categorical',
    class_names=None,
    batch_size=32,
    max_length=None,
    shuffle=True,
    seed=None,
    validation_split=None,
    subset=None,
    follow_links=False)
validation_ds = keras.utils.text_dataset_from_directory(
    directory='.../validation_data/',
    labels='inferred',
    label_mode='categorical',
    class_names=None,
    batch_size=32,
    max_length=None,
    shuffle=True,
    seed=None,
    validation_split=None,
    subset=None,
    follow_links=False)

inputs = keras.Input(shape=(None,))
x = layers.Reshape((-1, 1))(inputs)
x = Rescaling(scale=1.0 / MAX_VAL)(x)
x = layers.Dense(32, activation="softmax")(x)
outputs = layers.Dense(num_classes, activation="softmax")(x)
model = keras.Model(inputs=inputs, outputs=outputs)
model.summary()
keras.utils.plot_model(model, "my_first_model.png")

model.compile(optimizer='Adam', loss='categorical_crossentropy')
history = model.fit(train_ds, epochs=10, validation_data=validation_ds)

输出:

Epoch 1/10
WARNING:tensorflow:Model was constructed with shape (None, None) for input KerasTensor(type_spec=TensorSpec(shape=(None, None), dtype=tf.float32, name='input_1'), name='input_1', description="created by layer 'input_1'"), but it was called on an input with incompatible shape (None,).
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-3656f32a72a9> in <module>
      1 model.compile(optimizer='Adam', loss='categorical_crossentropy')
----> 2 history = model.fit(train_ds, epochs=10, validation_data=validation_ds)

~\AppData\Roaming\Python\Python38\site-packages\keras\utils\traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\func_graph.py in autograph_handler(*args, **kwargs)
   1145           except Exception as e:  # pylint:disable=broad-except
   1146             if hasattr(e, "ag_error_metadata"):
-> 1147               raise e.ag_error_metadata.to_exception(e)
   1148             else:
   1149               raise

ValueError: in user code:

    File "...\Python\Python38\site-packages\keras\engine\training.py", line 1021, in train_function  *
        return step_function(self, iterator)
    File "...\Python\Python38\site-packages\keras\engine\training.py", line 1010, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "...\Python\Python38\site-packages\keras\engine\training.py", line 1000, in run_step  **
        outputs = model.train_step(data)
    File "...\Python\Python38\site-packages\keras\engine\training.py", line 860, in train_step
        loss = self.compute_loss(x, y, y_pred, sample_weight)
    File "...\Python\Python38\site-packages\keras\engine\training.py", line 918, in compute_loss
        return self.compiled_loss(
    File "...\Python\Python38\site-packages\keras\engine\compile_utils.py", line 201, in __call__
        loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    File "...\Python\Python38\site-packages\keras\losses.py", line 141, in __call__
        losses = call_fn(y_true, y_pred)
    File "...\Python\Python38\site-packages\keras\losses.py", line 245, in call  **
        return ag_fn(y_true, y_pred, **self._fn_kwargs)
    File "...\Python\Python38\site-packages\keras\losses.py", line 1789, in categorical_crossentropy
        return backend.categorical_crossentropy(
    File "...\Python\Python38\site-packages\keras\backend.py", line 5083, in categorical_crossentropy
        target.shape.assert_is_compatible_with(output.shape)

    ValueError: Shapes (None, 2) and (None, 1, 2) are incompatible

模型的输入是保存在 .txt 文件中的一维向量(长度约为 27000): 0.101471743,0.099917953,0.103334975,0.099364908,0.099035715,...,0.097369999,0.099680934

从目录读取数据集,格式为:

/training_data/
...class_a/
......a_text_1.txt
......a_text_2.txt
...class_b/
......b_text_1.txt
......b_text_2.txt
/validation_data/
...class_a/
......a_text_1.txt
......a_text_2.txt
...class_b/
......b_text_1.txt
......b_text_2.txt

如何获得正确的尺寸?

编辑: 我将数据保存为 .jpg 文件,并使用 image_dataset_from_directory 加载它。这解决了这个问题。但我仍然想了解为什么我无法正确从 .txt 文件获取数据。 (我丢失了很多从 .jpg 中从浮点数据传输到 8 位 int 的数据。并且使用 image_dataset_from_directory 要求所有 img 大小具有相同的长度,我希望我的数据尺寸不同)。

I am using keras.utils.text_dataset_from_directory (see code). when I reach model.fit I get a warning about input dimentions and a error (that I think has something to do with the output).
code:

from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers import Rescaling
MAX_VAL = 0.5
num_classes = 2

train_ds = keras.utils.text_dataset_from_directory(
    directory='.../training_data/',
    labels='inferred',
    label_mode='categorical',
    class_names=None,
    batch_size=32,
    max_length=None,
    shuffle=True,
    seed=None,
    validation_split=None,
    subset=None,
    follow_links=False)
validation_ds = keras.utils.text_dataset_from_directory(
    directory='.../validation_data/',
    labels='inferred',
    label_mode='categorical',
    class_names=None,
    batch_size=32,
    max_length=None,
    shuffle=True,
    seed=None,
    validation_split=None,
    subset=None,
    follow_links=False)

inputs = keras.Input(shape=(None,))
x = layers.Reshape((-1, 1))(inputs)
x = Rescaling(scale=1.0 / MAX_VAL)(x)
x = layers.Dense(32, activation="softmax")(x)
outputs = layers.Dense(num_classes, activation="softmax")(x)
model = keras.Model(inputs=inputs, outputs=outputs)
model.summary()
keras.utils.plot_model(model, "my_first_model.png")

model.compile(optimizer='Adam', loss='categorical_crossentropy')
history = model.fit(train_ds, epochs=10, validation_data=validation_ds)

output:

Epoch 1/10
WARNING:tensorflow:Model was constructed with shape (None, None) for input KerasTensor(type_spec=TensorSpec(shape=(None, None), dtype=tf.float32, name='input_1'), name='input_1', description="created by layer 'input_1'"), but it was called on an input with incompatible shape (None,).
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-3656f32a72a9> in <module>
      1 model.compile(optimizer='Adam', loss='categorical_crossentropy')
----> 2 history = model.fit(train_ds, epochs=10, validation_data=validation_ds)

~\AppData\Roaming\Python\Python38\site-packages\keras\utils\traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\func_graph.py in autograph_handler(*args, **kwargs)
   1145           except Exception as e:  # pylint:disable=broad-except
   1146             if hasattr(e, "ag_error_metadata"):
-> 1147               raise e.ag_error_metadata.to_exception(e)
   1148             else:
   1149               raise

ValueError: in user code:

    File "...\Python\Python38\site-packages\keras\engine\training.py", line 1021, in train_function  *
        return step_function(self, iterator)
    File "...\Python\Python38\site-packages\keras\engine\training.py", line 1010, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "...\Python\Python38\site-packages\keras\engine\training.py", line 1000, in run_step  **
        outputs = model.train_step(data)
    File "...\Python\Python38\site-packages\keras\engine\training.py", line 860, in train_step
        loss = self.compute_loss(x, y, y_pred, sample_weight)
    File "...\Python\Python38\site-packages\keras\engine\training.py", line 918, in compute_loss
        return self.compiled_loss(
    File "...\Python\Python38\site-packages\keras\engine\compile_utils.py", line 201, in __call__
        loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    File "...\Python\Python38\site-packages\keras\losses.py", line 141, in __call__
        losses = call_fn(y_true, y_pred)
    File "...\Python\Python38\site-packages\keras\losses.py", line 245, in call  **
        return ag_fn(y_true, y_pred, **self._fn_kwargs)
    File "...\Python\Python38\site-packages\keras\losses.py", line 1789, in categorical_crossentropy
        return backend.categorical_crossentropy(
    File "...\Python\Python38\site-packages\keras\backend.py", line 5083, in categorical_crossentropy
        target.shape.assert_is_compatible_with(output.shape)

    ValueError: Shapes (None, 2) and (None, 1, 2) are incompatible

The inputs to the model are 1d vectors (of length ~27000) saved in .txt files:
0.101471743,0.099917953,0.103334975,0.099364908,0.099035715,...,0.097369999,0.099680934

readin dataset frome dir formated as:

/training_data/
...class_a/
......a_text_1.txt
......a_text_2.txt
...class_b/
......b_text_1.txt
......b_text_2.txt
/validation_data/
...class_a/
......a_text_1.txt
......a_text_2.txt
...class_b/
......b_text_1.txt
......b_text_2.txt

How can I get the dimensions right?

EDIT:
I saved the data as a .jpg file and loaded it using image_dataset_from_directory. this fixed the issue. but I would still like to understand why I cant get the data from .txt files properly. (I lose a lot of data transferring from float data to 8bit int in .jpg. and using image_dataset_from_directory requires all img size to be the same length, I wand my data to be different sizes).

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文