为什么要为NAN记录的keras-rl2 dqn代理的平均值和MAE

Question

从，但是只记录了奖励和数量的步骤，并且错误指标都是NAN，

memory = SequentialMemory(limit=1000000, window_length=WINDOW_LENGTH)

processor = AtariProcessor()

policy = LinearAnnealedPolicy(EpsGreedyQPolicy(), attr='eps', value_max=1., value_min=.1, value_test=.05,nb_steps=1000000)

dqn = DQNAgent(model=model1, nb_actions=nb_actions, policy=policy, memory=memory,
           processor=processor, nb_steps_warmup=50000, gamma=.99, 
target_model_update=10000,train_interval=4, delta_clip=1.)

adamOptimizer = adam_v2.Adam(learning_rate=0.00025)

dqn.compile(adamOptimizer ,metrics=['mae'])

env_name = 'PongNoFrameskip-v4'

weights_filename = 'dqn_{}_weights.h5f'.format(env_name)

checkpoint_weights_filename = 'dqn_' + env_name + '_weights_{step}.h5f'

log_filename = 'dqn_{}_log.json'.format(env_name)

callbacks = [ModelIntervalCheckpoint(checkpoint_weights_filename, interval=250000)]

callbacks += [FileLogger(log_filename, interval=100)]

trainLog = dqn.fit(env, callbacks=callbacks, nb_steps=1750000, log_interval=10000)    

我只让IT训练几千步为了显示，在 dqn _ {}。全是NAN，下面是JSON日志文件内容的屏幕截图

dqn代理训练日志文件

当打印回调历史记录键时，不包括损失和MAE

print(trainLog.history.keys())

输出：dict_keys（['emotive_reward'，'nb_episode_steps'，'nb_steps']）

Answer 1

他们没有实施它（由于库是现在的档案库，因此可能不会实现它）。但是，我通过在第219行中的 keras-rl2/rl/core.py 修改源代码来解决此问题，或者在 ######之间添加代码。 ## 。

if done:
    # We are in a terminal state but the agent hasn't yet seen it. We therefore
    # perform one more forward-backward call and simply ignore the action before
    # resetting the environment. We need to pass in `terminal=False` here since
    # the *next* state, that is the state of the newly reset environment, is
    # always non-terminal by convention.
    self.forward(observation)
    self.backward(0., terminal=False)

    # This episode is finished, report and reset.

    episode_logs = {
        'episode_reward': episode_reward,
        'nb_episode_steps': episode_step,
        'nb_steps': self.step,
        #################################
        **{name:metrics[i] for i, name in enumerate(self.metrics_names)},
        'info': accumulated_info,
        #################################
    }
    callbacks.on_episode_end(episode, episode_logs)

    episode += 1
    observation = None
    episode_step = None
    episode_reward = None

我还添加了信息，以防万一。不用担心，这不会改变代理培训过程或行为，我们只是在检索其他信息。