训练损失减少，但验证损失稳定

Question

我正在尝试训练我从本文中获得的神经网络 https://scholarworks.rit.edu/cgi/cgi/viewcontent.cgi?referer =＆amp;httpsredir=1& article = 10455＆amp; atsemp；请参阅此图像： neural Network Architechture

我正在使用Pytorch-Lightning使用Multi-GPU培训。

我正在喂养该网络3通道光流（UVC：U是水平的时间位移，V是垂直时间位移，C表示置信图）。

Ouputs表示框架姿势的框架，它们的形式为6个浮动值（TranslationX，Tanslationy，Translationz，yaw，Yaw，Pitch，Roll）。翻译从-0.25到3米，旋转从-6到6度不等。

输出数据集取自Kitti-odometry数据集，有11个视频序列，我将前8个用于训练，其余3个序列的一部分在训练过程中进行评估。

我训练了200个时期的模型（在8 GPU上花费了33个小时）。

在此培训期间，培训损失减少，但在整个培训过程中验证损失保持恒定。

transform = transforms.Compose(
    [cv_resize((370,1242)),
     flow_transform_and_uint8_and_tensor(),
     transforms.Normalize((0.3973, 0.2952, 0.4500), (0.4181, 0.4362, 0.3526))])

batch_size = 8
val_data_percentage = 0.06
epochs = 200
learning_rate = 0.0001

train_dataset = FlowsAndPoses("./uvc_flows_png/train/", "./relative_poses/train/", transform)
test_dataset = FlowsAndPoses("./uvc_flows_png/test/", "./relative_poses/test/", transform)

dataset_length = test_dataset.__len__()
test_dataset, val_dataset = random_split(test_dataset,[int(dataset_length*(1-val_data_percentage)),dataset_length - int(dataset_length*(1-val_data_percentage))])
print("Train: ",train_dataset.__len__(), "   Validation: ", val_dataset.__len__())

criterion = nn.L1Loss()
class Net(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 64, 7, 2)
        self.conv2 = nn.Conv2d(64, 128, 5, 2)
        self.conv3 = nn.Conv2d(128, 256, 5, 2)
        self.conv4 = nn.Conv2d(256, 256, 3, 1)
        self.conv5 = nn.Conv2d(256, 512, 3, 2)
        self.conv6 = nn.Conv2d(512, 512, 3, 1)
        self.conv7 = nn.Conv2d(512, 512, 3, 2)
        self.conv8 = nn.Conv2d(512, 512, 3, 1)
        self.conv9 = nn.Conv2d(512, 1024, 3, 2)
        
        self.fc1 = nn.Linear(32768, 1024)
        self.drop = nn.Dropout(0.5)
        self.fc2 = nn.Linear(1024, 6)

        self.net_relu = nn.LeakyReLU(0.1)

    def forward(self, x):
        x = self.net_relu(self.conv1(x))
        x = self.net_relu(self.conv2(x))
        x = self.net_relu(self.conv3(x))
        x = self.net_relu(self.conv4(x))
        x = self.net_relu(self.conv5(x))
        x = self.net_relu(self.conv6(x))
        x = self.net_relu(self.conv7(x))
        x = self.net_relu(self.conv8(x))
        x = self.net_relu(self.conv9(x))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = self.net_relu(self.fc1(x))
        x = self.drop(x)
        x = self.fc2(x)
        return x

    def training_step(self, batch, batch_idx):
        running_loss = 0
        print("Training: ")
        inputs, labels = batch
        outputs = self.forward(inputs.float())
        loss = criterion(outputs, labels.float())
        self.log("my_loss", loss, on_epoch=True)
        return loss

    def training_epoch_end(self, training_step_outputs):
        training_loss_file = open("losses/training_loss"+str(self.current_epoch)+"_"+str(self.global_step), "w")
        training_loss_file.write(str(training_step_outputs))
        training_loss_file.close()
        try:
            torch.save(self.state_dict(), "checkpoints/trained_model_epoch"+str(self.current_epoch)+".pth")
        except:
            print("error saving")

    def validation_step(self, batch, batch_idx):
        inputs, labels = batch
        outputs = self.forward(inputs.float())
        loss = criterion(outputs, labels.float())
        self.log("val_loss", loss)
        return loss

    def validation_epoch_end(self, validation_step_outputs):
        valid_loss_file = open("losses/validation_loss"+str(self.current_epoch)+"_"+str(self.global_step), "w")
        valid_loss_file.write(str(validation_step_outputs))
        valid_loss_file.close()

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=learning_rate)
        return optimizer


autoencoder = Net()
trainer = pl.Trainer(gpus=[0,1,2,3,4,5,6,7], accelerator="gpu", strategy="ddp", enable_checkpointing=True, max_epochs=epochs, check_val_every_n_epoch=1)
trainer.fit(autoencoder, DataLoader(train_dataset, batch_size=batch_size, shuffle=True), DataLoader(val_dataset, batch_size=batch_size, shuffle=True))

零级和优化器。

我得到的结果在以下图像中：

训练损失

如果有人对如何解决此问题有建议，我会真正地对待它。