您应该如何测试 2 个分类准确度分数的显着性：配对排列测试

Question

我有一个经过训练的分类器在 2 个相关的多类分类任务上进行了测试。由于分类任务的每次试验都是相关的，因此 2 组预测构成配对数据。我想运行配对排列测试来查明两个预测集之间的分类准确性差异是否显着。

因此，我的数据由 2 个预测类别列表组成，其中每个预测都与同一索引的其他测试集中的预测相关。

示例：

actual_classes = [1, 3, 6, 1, 22, 1, 11, 12, 9, 2]
predictions1 = [1, 3, 6, 1, 22, 1, 11, 12, 9 10] # 90% acc.
predictions2 = [1, 3, 7, 10, 22, 1, 7, 12, 2, 10] # 50% acc.

H0：分类准确率没有显着差异。

如何运行配对排列测试来测试分类准确性差异的显着性？

Answer 1

我一直在考虑这个问题，我将发布一个提议的解决方案，看看是否有人批准或解释我为什么错了。

actual_classes = [1, 3, 6, 1, 22, 1, 11, 12, 9, 2]
predictions1 = [1, 3, 6, 1, 22, 1, 11, 12, 9 10] # 90% acc.
predictions2 = [1, 3, 7, 10, 22, 1, 7, 12, 2, 10] # 50% acc.
paired_predictions = [[1,1], [3,3], [6,7], [1,10], [22,22], [1,1], [11,7], [12,12], [9,2], [10,10]]

actual_test_statistic = predictions1 - predictions2 # 90%-50%=40 # 0.9-0.5=0.4
all_simulations = [] # empty list
for number_of_iterations:
    shuffle(paired_predictions) # only shuffle between pairs, not within
    simulated_predictions1 = paired_predictions[first prediction of each pair]
    simulated_predictions2 = paired_predictions[second prediction of each pair]
    simulated_accuracy1 = proportion of times simulated_predictions1 equals actual_classes
    simulated_accuracy2 = proportion of times simulated_predictions2 equals actual_classes
    all_simulations.append(simulated_accuracy1 - simulated_accuracy2) # Put the simulated difference in the list

p = count(absolute(all_simulations) > absolute(actual_test_statistic ))/number_of_iterations

如果您有任何想法，请在评论中告诉我。或者更好的是，在您自己的答案中提供您自己的更正版本。谢谢你！