将Relu函数的输入归一化遇到运行时错误

Question

当我尝试将最大激活值从上一层传递到下一层中的relu的输入时，我会遇到下面的运行时错误。但是，当我传递固定值时，它可以很好地工作而没有任何错误。

File "/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py", line 175, in backward allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
RuntimeError: Trying to backward through the graph a second time (or directly access saved
tensors after they have already been freed). Saved intermediate values of the graph are freed
when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to
backward through the graph a second time or if you need to access saved tensors after calling
backward.

如您在下面的此代码中看到的那样，我将参数prev_layer_max从上一层传递，并遇到错误：

class th_norm_ReLU(nn.Module):
    def __init__(self, modify):
        super(th_norm_ReLU, self).__init__()
        self.therelu = F.relu
       

    def forward(self, input, prev_layer_max):
      output = input * (prev_layer_max / input.max())
      norm_output = self.therelu (output)
      
      return norm_output

但是，如果我使用固定值而不是传递prev_layer_max参数，则为下面的此代码我使其等于1它正常起作用而没有任何错误：

    def forward(self, input, prev_layer_max = 1):
        output = input * (1 / input.max())
        norm_output = self.therelu (output)

训练环如下：

for epoch in range(params.epochs):
    running_loss = 0

    start_time = time.time()
    for i, (images, labels) in enumerate(train_loader):
        model.train()
        model.zero_grad()  
        optimizer.zero_grad()  
        labels.to(device)
        images = images.float().to(device)
        outputs = model(images, epoch)
        loss = criterion(outputs.cpu(), labels)  
        running_loss += loss.item()  
        loss.backward()  
        optimizer.step()

这是我在列表中记录每个层的最大值（thresh_list ）：

def forward(self, input, epoch):
    x = self.conv1(input)
    x = self.relu(x,1)
    self.thresh_list[0] = max(self.thresh_list[0], x.max()) #to get the max activation 
    x = self.conv_dropout(x)
    x = self.conv2(x)
    x = self.relu(x, self.thresh_list[0])
    self.thresh_list[1] = max(self.thresh_list[1], x.max())
    x = self.pool1(x)
    x = self.conv_dropout(x)
    x = self.conv3(x)
    x = self.relu(x, self.thresh_list[1] )
    self.thresh_list[2] = max(self.thresh_list[2], x.max())

我称之为的Relue函数是：

self.relu = th_norm_ReLU(True)
        

the_norm_relu模型如上所述。