如何在 Keras 中制作端到端 3D-2D CNN？

Question

我有两个 CAE 模型，一个是 3D，另一个是 2D。此 2D CAE 将第一个生成的新表示作为输入。我的目标是弄清楚如何将它们结合起来，以便我可以拥有端到端完整的 3D-2D CAE 模型以及如何训练它？

以下是每个模型的代码：

#3D CAE (I have just implemented the first encoding part since my aim is to generate the new representation z)

in_3D = Input((100,100, 288, 1))
model_3D = Conv3D(8, (5, 5, 5), activation='relu', padding='same')(in_3D)
model_3D = MaxPooling3D((2, 2, 2), strides=(1, 1, 4), padding='same')(model_3D)
model_3D = Reshape((10000,72*8))(model_3D)
model_3D = Dense(350, activation="relu")(model_3D)
model_3D = Dense(250, activation="relu")(model_3D)
model_3D = Dense(198, activation="relu")(model_3D)
model_3D = Reshape((100,100, 198))(model_3D)
z = Permute((3,2, 1))(model_3D)

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 100, 100, 288, 1  0         
                             )]                                  
                                                                 
 conv3d_1 (Conv3D)           (None, 100, 100, 288, 8)  1008      
                                                                 
 max_pooling3d_1 (MaxPooling  (None, 100, 100, 72, 8)  0         
 3D)                                                             
                                                                 
 reshape (Reshape)         (None, 10000, 576)        0         
                                                                 
 dense (Dense)            (None, 10000, 350)        201950    
                                                                 
 dense_1 (Dense)            (None, 10000, 250)        87750     
                                                                 
 dense_2 (Dense)            (None, 10000, 198)        49698     
                                                                 
 reshape_1 (Reshape)         (None, 100, 100, 198)     0         
                                                                 
 permute (Permute)           (None, 198, 100, 100)     0         
                                                              

第二个 2D CAE 模型接收第一个模型生成的新 z (198,100,100) 作为输入。这里 198 作为 None 传递

#2D CAE  

in_2D = Input((100,100, 1))
model_2D= Conv2D(16, (3, 3), activation='relu', padding='same', name='Conv1')(in_2D)
model_2D = MaxPooling2D((2, 2), padding='same')(model_2D)
model_2D = Flatten()(model_2D)
model_2D = Dense(48, activation='relu')(model_2D)
model_2D = Dense(36, activation='relu')(model_2D)
model_2D = Dense(12)(model_2D)
model_2D= Dense(100*100, activation='linear')(model_2D)

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 100, 100, 1)]     0         
                                                                 
 Conv1 (Conv2D)              (None, 100, 100, 16)      160       
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 50, 50, 16)       0         
 2D)                                                             
                                                                 
 flatten (Flatten)          (None, 40000)             0         
                                                                 
 dense (Dense)              (None, 48)                1920048   
                                                                 
 dense_1 (Dense)             (None, 36)                1764      
                                                                 
 dense_2 (Dense)             (None, 12)                444       
                                                                 
 dense_3 (Dense)            (None, 10000)             130000    

任何帮助将不胜感激。

Answer 1

要组合这两个模型，您可以首先像这样单独声明它们：

import keras
import tensorflow as tf
from tensorflow.keras.layers import *

in_3D = Input((100,100, 288, 1))
model_3D = Conv3D(8, (5, 5, 5), activation='relu', padding='same')(in_3D)
model_3D = MaxPooling3D((2, 2, 2), strides=(1, 1, 4), padding='same')(model_3D)
model_3D = Reshape((10000,72*8))(model_3D)
model_3D = Dense(350, activation="relu")(model_3D)
model_3D = Dense(250, activation="relu")(model_3D)
model_3D = Dense(198, activation="relu")(model_3D)
model_3D = Reshape((100,100, 198))(model_3D)
z = Permute((3,2, 1))(model_3D)
cae_model_3D = keras.Model(in_3D, z)

in_2D = Input((100,100, 1))
model_2D= Conv2D(16, (3, 3), activation='relu', padding='same', name='Conv1')(in_2D)
model_2D = MaxPooling2D((2, 2), padding='same')(model_2D)
model_2D = Flatten()(model_2D)
model_2D = Dense(48, activation='relu')(model_2D)
model_2D = Dense(36, activation='relu')(model_2D)
model_2D = Dense(12)(model_2D)
model_2D= Dense(100*100, activation='linear')(model_2D)
cae_model_2D = keras.Model(in_2D, model_2D)

然后声明一个组合模型，将第一个模型的输出作为第二个模型的输入：

combined_model_input = Input((100, 100, 288, 1))
cae_model_3D_output = cae_model_3D(combined_model_input)
cae_model_3D_output_reshaped = tf.reshape(cae_model_3D_output, (-1, 100, 100, 1))
combined_model_output = cae_model_2D(cae_model_3D_output_reshaped)
combined_model = keras.Model(combined_model_input, combined_model_output)

请注意，我们必须重塑第一个模型的输出以使其保持一致您希望将 198 作为批次维度传递（None）。

训练模型的最简单方法是调用它的 compile 和 fit 方法。您传递给这些方法的确切参数将取决于您要解决的问题和您自己的偏好。以下是官方文档的链接以获取帮助。