将一个训练好的模型分别保存为.ckpt和.pb格式，然后利用这两种保存的模型分别进行预测。为什么得到的结果不同？

Question

将一个训练好的模型分别保存为.ckpt和.pb格式，然后利用这两种保存的模型分别进行预测，得到的结果不同？而且对于同一张图片，每次预测的结果也都稍有区别，请教下这是为什么？

# coding: utf-8

import tensorflow as tf
from tensorflow.python.framework import graph_util
import os
import numpy as np
import cv2

os.environ["CUDA_VISIBLE_DEVICES"] = "0"


def restore_pb(img_data, pb_file_path=None):
    '''restore model from .pb to infer
    '''
    if not pb_file_path:
        pb_file_path = "test/inceptionV3_0.pb"
    with tf.Graph().as_default():
        graph_def = tf.GraphDef()
        with tf.gfile.FastGFile(pb_file_path, "rb") as f:
            graph_def = tf.GraphDef()
            graph_def.ParseFromString(f.read())
            return_elems = ['tower_0/InceptionV3/predictions/Softmax:0']
            predictions = tf.import_graph_def(graph_def, return_elements=return_elems)

        with tf.Session() as sess:
            # print(tf.contrib.graph_editor.get_tensors(tf.get_default_graph()))
            # for node in sess.graph.node:
            #     print(node.name)
            sess.run(tf.global_variables_initializer())
            image = sess.graph.get_tensor_by_name('import/x:0')
            #label = sess.graph.get_tensor_by_name('y:0')

            predictions = sess.run(predictions, feed_dict={image: img_data})
            print('pb predictions:', predictions[0])

def restore_ckpt(img_data):
    '''restore model from .ckpt to infer
    '''
    cpkt_meta = 'test/inception_mom1to2_rand5.ckpt-0.meta'
    ckpt = 'test/inception_mom1to2_rand5.ckpt-0'
    with tf.Graph().as_default():
        graph = tf.Graph()
        config = tf.ConfigProto(allow_soft_placement=True)
        with tf.Session(graph=graph, config=config) as sess:
            saver = tf.train.import_meta_graph(cpkt_meta)
            saver.restore(sess, ckpt)
            # print(tf.contrib.graph_editor.get_tensors(tf.get_default_graph()))
            image = graph.get_tensor_by_name('x:0')
            preds = graph.get_tensor_by_name('tower_0/InceptionV3/predictions/Softmax:0')
            predictions = sess.run(preds, feed_dict={image: img_data})
            print('ckpt predictions:', predictions)

def ckpt2pb():
    '''convert .ckpt to .pb file
    '''
    # some path
    output_nodes = ['tower_0/InceptionV3/predictions/Softmax']
    cpkt_meta = 'test/inception_mom1to2_rand5.ckpt-0.meta'
    ckpt = 'test/inception_mom1to2_rand5.ckpt-0'
    pb_file_path = 'test/inceptionV3_ckpt2pb.pb'

    with tf.Graph().as_default():
        graph = tf.Graph()
        config = tf.ConfigProto(allow_soft_placement=True)

        with tf.Session(graph=graph, config=config) as sess:
            # restored model from ckpt
            saver = tf.train.import_meta_graph(cpkt_meta)
            saver.restore(sess, ckpt)

            # save freeze graph into .pb file
            graph_def = tf.get_default_graph().as_graph_def()
            constant_graph = graph_util.convert_variables_to_constants(sess, graph_def, output_nodes)
            with tf.gfile.FastGFile(pb_file_path, mode='wb') as f:
                f.write(constant_graph.SerializeToString())

分别对一张图片进行预测：

img_data = cv2.imread('tumor_009_19327_175114_0.jpeg')
img_data = cv2.cvtColor(img_data, cv2.COLOR_BGR2RGB)
img_data = (np.array(img_data).astype(np.float32)) / 256.0
img_data = np.reshape(img_data, [-1, 256, 256, 3])

restore_pb(img_data)
restore_ckpt(img_data)
ckpt2pb()
restore_pb(img_data, pb_file_path='test/inceptionV3_ckpt2pb.pb')

结果如下：

2019-05-10 16:56:42.793083: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-05-10 16:56:44.553334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:04:00.0
totalMemory: 22.38GiB freeMemory: 22.21GiB
2019-05-10 16:56:44.553413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2019-05-10 16:56:44.804127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 16:56:44.804196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2019-05-10 16:56:44.804204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2019-05-10 16:56:44.804687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21557 MB memory) -> physical GPU (device: 0, name: Tesla P40, pcibus id: 0000:04:00.0, compute capability: 6.1)
pb predictions: [[0.31244308 0.6875569 ]]
2019-05-10 16:56:47.791268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2019-05-10 16:56:47.791313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 16:56:47.791321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2019-05-10 16:56:47.791326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2019-05-10 16:56:47.791480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21557 MB memory) -> physical GPU (device: 0, name: Tesla P40, pcibus id: 0000:04:00.0, compute capability: 6.1)
ckpt predictions: [[0.27373236 0.7262676 ]]
2019-05-10 16:56:52.778329: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2019-05-10 16:56:52.778398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 16:56:52.778406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2019-05-10 16:56:52.778414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2019-05-10 16:56:52.778624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21557 MB memory) -> physical GPU (device: 0, name: Tesla P40, pcibus id: 0000:04:00.0, compute capability: 6.1)
Converted 190 variables to const ops.
2019-05-10 16:56:58.946692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2019-05-10 16:56:58.946815: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 16:56:58.946827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2019-05-10 16:56:58.946834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2019-05-10 16:56:58.947005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21557 MB memory) -> physical GPU (device: 0, name: Tesla P40, pcibus id: 0000:04:00.0, compute capability: 6.1)
pb predictions: [[0.28257495 0.71742505]]

只看预测结果，也就是：

pb predictions: [[0.31244308 0.6875569 ]] # this model gerated from last running

ckpt predictions: [[0.27373236 0.7262676 ]]

pb predictions: [[0.28257495 0.71742505]]

可以看到，上面得到的3个结果都是略有区别的，而且每次运行都略有不同。请问这是为什么？

Answer 1

1.激活函数选择导致的；输入256维一个矩阵，经过映射，过滤，激活函数soft抽取进化是在一个范围内随机抽取负例正例来做选择的

2.因为网络中应用了dropout

3.输入模型的图片的预处理的方式与训练莫模型的图片预处理方式不一致，例如图片归一化的方式不一致
以上都是可能导致差别的原因。