one epoch = one forward pass and one backward pass of all the training examples
batch size = the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you'll need.
number of iterations = number of passes, each pass using [batch size] number of examples. To be clear, one pass = one forward pass + one backward pass (we do not count the forward pass and backward pass as two different passes).
For example: if you have 1000 training examples, and your batch size is 500, then it will take 2 iterations to complete 1 epoch.
The term "batch" is ambiguous: some people use it to designate the entire training set, and some people use it to refer to the number of training examples in one forward/backward pass (as I did in this answer). To avoid that ambiguity and make clear that batch corresponds to the number of training examples in one forward/backward pass, one can use the term mini-batch.
An epoch describes the number of times the algorithm sees the entire data set. So, each time the algorithm has seen all samples in the dataset, an epoch has been completed.
Iteration
An iteration describes the number of times a batch of data passed through the algorithm. In the case of neural networks, that means the forward pass and backward pass. So, every time you pass a batch of data through the NN, you completed an iteration.
Example
An example might make it clearer.
Say you have a dataset of 10 examples (or samples). You have a batch size of 2, and you've specified you want the algorithm to run for 3 epochs.
Therefore, in each epoch, you have 5 batches (10/2 = 5). Each batch gets passed through the algorithm, therefore you have 5 iterations per epoch. Since you've specified 3 epochs, you have a total of 15 iterations (5*3 = 15) for training.
Many neural network training algorithms involve making multiple presentations of the entire data set to the neural network. Often, a single presentation of the entire data set is referred to as an "epoch". In contrast, some algorithms present data to the neural network a single case at a time.
"Iteration" is a much more general term, but since you asked about it together with "epoch", I assume that your source is referring to the presentation of a single case to a neural network.
Before I start with the actual answer, I would like to build some background.
A batch is the complete dataset. Its size is the total number of training examples in the available dataset.
mini-batch size is the number of examples the learning algorithm processes in a single pass (forward and backward).
A Mini-batch is a small part of the dataset of given mini-batch size.
Iterations is the number of batches of data the algorithm has seen (or simply the number of passes the algorithm has done on the dataset).
Epochs is the number of times a learning algorithm sees the complete dataset. Now, this may not be equal to the number of iterations, as the dataset can also be processed in mini-batches, in essence, a single pass may process only a part of the dataset. In such cases, the number of iterations is not equal to the number of epochs.
In the case of Batch gradient descent, the whole batch is processed on each training pass. Therefore, the gradient descent optimizer results in smoother convergence than Mini-batch gradient descent, but it takes more time. The batch gradient descent is guaranteed to find an optimum if it exists.
Stochastic gradient descent is a special case of mini-batch gradient descent in which the mini-batch size is 1.
I guess in the context of neural network terminology:
Epoch: When your network ends up going over the entire training set (i.e., once for each training instance), it completes one epoch.
In order to define iteration (a.k.a steps), you first need to know about batch size:
Batch Size: You probably wouldn't like to process the entire training instances all at one forward pass as it is inefficient and needs a huge deal of memory. So what is commonly done is splitting up training instances into subsets (i.e., batches), performing one pass over the selected subset (i.e., batch), and then optimizing the network through backpropagation. The number of training instances within a subset (i.e., batch) is called batch_size.
Iteration: (a.k.a training steps) You know that your network has to go over all training instances in one pass in order to complete one epoch. But wait! when you are splitting up your training instances into batches, that means you can only process one batch (a subset of training instances) in one forward pass, so what about the other batches? This is where the term Iteration comes into play:
Definition: The number of forwarding passes (The number of batches that you have created) that your network has to do in order to complete one epoch (i.e., going over all training instances) is called Iteration.
For example, when you have 10,000 training instances and you want to do batching with the size of 10; you have to do 10,000/10 = 1,000 iterations to complete 1 epoch.
You have training data which you shuffle and pick mini-batches from it. When you adjust your weights and biases using one mini-batch, you have completed one iteration.
Once you run out of your mini-batches, you have completed an epoch. Then you shuffle your training data again, pick your mini-batches again, and iterate through all of them again. That would be your second epoch.
Typically, you'll split your test set into small batches for the network to learn from, and make the training go step by step through your number of layers, applying gradient-descent all the way down. All these small steps can be called iterations.
An epoch corresponds to the entire training set going through the entire network once. It can be useful to limit this, e.g. to fight to overfit.
To my understanding, when you need to train a NN, you need a large dataset that involves many data items. when NN is being trained, data items go into NN one by one, that is called an iteration; When the whole dataset goes through, it is called an epoch.
I believe iteration is equivalent to a single batch forward+backprop in batch SGD. Epoch is going through the entire dataset once (as someone else mentioned).
An epoch contains a few iterations. That's actually what this epoch is. Let's define epoch as the number of iterations over the data set in order to train the neural network.
Epoch is 1 complete cycle where the Neural network has seen all the data.
One might have said 100,000 images to train the model, however, memory space might not be sufficient to process all the images at once, hence we split training the model on smaller chunks of data called batches. e.g. batch size is 100.
We need to cover all the images using multiple batches. So we will need 1000 iterations to cover all the 100,000 images. (100 batch size * 1000 iterations)
Once Neural Network looks at the entire data it is called 1 Epoch (Point 1). One might need multiple epochs to train the model. (let us say 10 epochs).
def SGD(self, training_data, epochs, mini_batch_size, eta,
test_data=None):
"""Train the neural network using mini-batch stochastic
gradient descent. The "training_data" is a list of tuples
"(x, y)" representing the training inputs and the desired
outputs. The other non-optional parameters are
self-explanatory. If "test_data" is provided then the
network will be evaluated against the test data after each
epoch, and partial progress printed out. This is useful for
tracking progress, but slows things down substantially."""
if test_data: n_test = len(test_data)
n = len(training_data)
for j in xrange(epochs):
random.shuffle(training_data)
mini_batches = [
training_data[k:k+mini_batch_size]
for k in xrange(0, n, mini_batch_size)]
for mini_batch in mini_batches:
self.update_mini_batch(mini_batch, eta)
if test_data:
print "Epoch {0}: {1} / {2}".format(
j, self.evaluate(test_data), n_test)
else:
print "Epoch {0} complete".format(j)
Note that the page has a code for the gradient descent algorithm which uses epoch
def SGD(self, training_data, epochs, mini_batch_size, eta,
test_data=None):
"""Train the neural network using mini-batch stochastic
gradient descent. The "training_data" is a list of tuples
"(x, y)" representing the training inputs and the desired
outputs. The other non-optional parameters are
self-explanatory. If "test_data" is provided then the
network will be evaluated against the test data after each
epoch, and partial progress printed out. This is useful for
tracking progress, but slows things down substantially."""
if test_data: n_test = len(test_data)
n = len(training_data)
for j in xrange(epochs):
random.shuffle(training_data)
mini_batches = [
training_data[k:k+mini_batch_size]
for k in xrange(0, n, mini_batch_size)]
for mini_batch in mini_batches:
self.update_mini_batch(mini_batch, eta)
if test_data:
print "Epoch {0}: {1} / {2}".format(
j, self.evaluate(test_data), n_test)
else:
print "Epoch {0} complete".format(j)
Look at the code. For each epoch, we randomly generate a subset of the inputs for the gradient descent algorithm. Why epoch is effective is also explained on the page. Please take a look.
"A full training pass over the entire dataset such that each example has been seen once. Thus, an epoch represents N/batch_size training iterations, where N is the total number of examples."
If you are training model for 10 epochs with batch size 6, given total 12 samples that means:
the model will be able to see the whole dataset in 2 iterations ( 12 / 6 = 2) i.e. single epoch.
overall, the model will have 2 X 10 = 20 iterations (iterations-per-epoch X no-of-epochs)
re-evaluation of loss and model parameters will be performed after each iteration!
A full training pass over the entire dataset such that each example has been seen once. Thus, an epoch represents N/batch size training iterations, where N is the total number of examples.
A single update of a model's weights during training. An iteration consists of computing the gradients of the parameters with respect to the loss on a single batch of data.
发布评论
评论（14）
在神经网络术语中：
例如：如果您有 1000 个训练样本，并且您的批量大小为 500，那么需要 2 次迭代才能完成 1 个 epoch。
仅供参考：权衡批量大小与训练神经网络的迭代次数
术语“批量”含糊不清：一些人们用它来指定整个训练集，有些人用它来指一次前向/后向传递中的训练示例的数量（就像我在这个答案中所做的那样）。为了避免这种歧义并明确批次对应于一次前向/后向传递中的训练示例数量，可以使用术语小批量。
In the neural network terminology:
For example: if you have 1000 training examples, and your batch size is 500, then it will take 2 iterations to complete 1 epoch.
FYI: Tradeoff batch size vs. number of iterations to train a neural network
The term "batch" is ambiguous: some people use it to designate the entire training set, and some people use it to refer to the number of training examples in one forward/backward pass (as I did in this answer). To avoid that ambiguity and make clear that batch corresponds to the number of training examples in one forward/backward pass, one can use the term mini-batch.
Epoch 和 iteration 描述不同的事物。
历元
历元描述了算法查看整个数据集的次数。因此，每当算法看到数据集中的所有样本时，一个纪元就完成了。
迭代
迭代描述了一批数据通过算法的次数。对于神经网络来说，这意味着前向传递和后向传递。因此，每次通过神经网络传递一批数据时，您就完成了一次迭代。
示例
举个例子可能会更清楚。
假设您有一个包含 10 个示例（或样本）的数据集。您的批量大小为 2，并且您已指定希望算法运行 3 轮。
因此，在每个 epoch 中，您有 5 个批次 (10/2 = 5)。每个批次都会通过算法，因此每个时期有 5 次迭代。
由于您指定了 3 个 epoch，因此总共需要 15 次迭代 (5*3 = 15) 进行训练。
Epoch and iteration describe different things.
Epoch
An epoch describes the number of times the algorithm sees the entire data set. So, each time the algorithm has seen all samples in the dataset, an epoch has been completed.
Iteration
An iteration describes the number of times a batch of data passed through the algorithm. In the case of neural networks, that means the forward pass and backward pass. So, every time you pass a batch of data through the NN, you completed an iteration.
Example
An example might make it clearer.
Say you have a dataset of 10 examples (or samples). You have a batch size of 2, and you've specified you want the algorithm to run for 3 epochs.
Therefore, in each epoch, you have 5 batches (10/2 = 5). Each batch gets passed through the algorithm, therefore you have 5 iterations per epoch.
Since you've specified 3 epochs, you have a total of 15 iterations (5*3 = 15) for training.
许多神经网络训练算法涉及将整个数据集多次呈现给神经网络。通常，整个数据集的单个表示被称为“纪元”。相比之下，一些算法一次向神经网络呈现单个案例的数据。
“迭代”是一个更通用的术语，但由于您与“纪元”一起询问它，我认为您的来源是指向神经网络呈现单个案例。
Many neural network training algorithms involve making multiple presentations of the entire data set to the neural network. Often, a single presentation of the entire data set is referred to as an "epoch". In contrast, some algorithms present data to the neural network a single case at a time.
"Iteration" is a much more general term, but since you asked about it together with "epoch", I assume that your source is referring to the presentation of a single case to a neural network.
要了解它们之间的区别，您必须了解梯度下降算法及其变体。
在开始给出实际答案之前，我想先介绍一些背景知识。
批次是完整的数据集。它的大小是可用数据集中训练示例的总数。
小批量大小是学习算法在单次传递（向前和向后）中处理的示例数量。
小批量是给定小批量大小数据集的一小部分。
迭代是算法已经看到的数据批次的数量（或者简称为算法在数据集上完成的传递次数）。
Epochs 是学习算法看到完整数据集的次数。现在，这可能不等于迭代次数，因为数据集也可以小批量处理，本质上，单次传递可能只处理数据集的一部分。 在这种情况下，迭代次数不等于纪元数量。
在批量梯度下降的情况下，每次训练都会处理整个批次。因此，梯度下降优化器比 Mini-batch 梯度下降收敛更平滑，但需要更多时间。批量梯度下降保证找到最优值（如果存在）。
随机梯度下降是小批量梯度下降的一种特殊情况，其中小批量大小为1。
To understand the difference between these you must understand the Gradient Descent Algorithm and its Variants.
Before I start with the actual answer, I would like to build some background.
A batch is the complete dataset. Its size is the total number of training examples in the available dataset.
mini-batch size is the number of examples the learning algorithm processes in a single pass (forward and backward).
A Mini-batch is a small part of the dataset of given mini-batch size.
Iterations is the number of batches of data the algorithm has seen (or simply the number of passes the algorithm has done on the dataset).
Epochs is the number of times a learning algorithm sees the complete dataset. Now, this may not be equal to the number of iterations, as the dataset can also be processed in mini-batches, in essence, a single pass may process only a part of the dataset. In such cases, the number of iterations is not equal to the number of epochs.
In the case of Batch gradient descent, the whole batch is processed on each training pass. Therefore, the gradient descent optimizer results in smoother convergence than Mini-batch gradient descent, but it takes more time. The batch gradient descent is guaranteed to find an optimum if it exists.
Stochastic gradient descent is a special case of mini-batch gradient descent in which the mini-batch size is 1.
我想在神经网络术语的背景下：
为了定义迭代（又名步骤），您首先需要了解批量大小：
批量大小：< /strong> 您可能不希望一次性处理整个训练实例，因为它效率低下并且需要大量内存。因此，通常所做的是将训练实例分割为子集（即批次），对选定的子集（即批次）执行一次传递，然后通过反向传播优化网络。子集（即批次）内的训练实例数量称为batch_size。
迭代：（也称为训练步骤）您知道，您的网络必须一次性遍历所有训练实例才能完成一个周期。但是等等！当您将训练实例分成批次时，这意味着您只能在一次前向传递中处理一个批次（训练实例的子集），那么其他批次呢？这就是术语迭代发挥作用的地方：
定义：网络必须执行的转发传递次数（您创建的批次数量）为了完成一个时期（即遍历所有训练实例）被称为迭代。
例如，当您有 10,000 个训练实例并且您想要以 10 的大小进行批处理时；您必须执行 10,000/10 = 1,000 次迭代才能完成 1 个纪元。
希望这可以回答您的问题！
I guess in the context of neural network terminology:
In order to define iteration (a.k.a steps), you first need to know about batch size:
Batch Size: You probably wouldn't like to process the entire training instances all at one forward pass as it is inefficient and needs a huge deal of memory. So what is commonly done is splitting up training instances into subsets (i.e., batches), performing one pass over the selected subset (i.e., batch), and then optimizing the network through backpropagation. The number of training instances within a subset (i.e., batch) is called batch_size.
Iteration: (a.k.a training steps) You know that your network has to go over all training instances in one pass in order to complete one epoch. But wait! when you are splitting up your training instances into batches, that means you can only process one batch (a subset of training instances) in one forward pass, so what about the other batches? This is where the term Iteration comes into play:
Definition: The number of forwarding passes (The number of batches that you have created) that your network has to do in order to complete one epoch (i.e., going over all training instances) is called Iteration.
For example, when you have 10,000 training instances and you want to do batching with the size of 10; you have to do 10,000/10 = 1,000 iterations to complete 1 epoch.
Hope this could answer your question!
您拥有训练数据，可以对其进行洗牌并从中选择小批量。当您使用一个小批量调整权重和偏差时，您就完成了一次迭代。
一旦你用完你的小批量，你就完成了一个纪元。然后你再次打乱你的训练数据，再次选择你的小批量，并再次迭代所有它们。那将是你的第二个时代。
You have training data which you shuffle and pick mini-batches from it. When you adjust your weights and biases using one mini-batch, you have completed one iteration.
Once you run out of your mini-batches, you have completed an epoch. Then you shuffle your training data again, pick your mini-batches again, and iterate through all of them again. That would be your second epoch.
通常，您会将测试集分成小批量，以便网络进行学习，并通过层数逐步进行训练，一直向下应用梯度下降。所有这些小步骤都可以称为迭代。
一个epoch对应于整个训练集经过整个网络一次。限制这一点可能很有用，例如防止过度拟合。
Typically, you'll split your test set into small batches for the network to learn from, and make the training go step by step through your number of layers, applying gradient-descent all the way down. All these small steps can be called iterations.
An epoch corresponds to the entire training set going through the entire network once. It can be useful to limit this, e.g. to fight to overfit.
据我了解，当你需要训练神经网络时，你需要一个涉及许多数据项的大型数据集。神经网络训练时，数据项一项一项地进入神经网络，称为迭代；当整个数据集经过时，称为一个纪元。
To my understanding, when you need to train a NN, you need a large dataset that involves many data items. when NN is being trained, data items go into NN one by one, that is called an iteration; When the whole dataset goes through, it is called an epoch.
我相信迭代相当于批量SGD中的单批量前向+反向传播。 Epoch 会遍历整个数据集一次（正如其他人提到的）。
I believe iteration is equivalent to a single batch forward+backprop in batch SGD. Epoch is going through the entire dataset once (as someone else mentioned).
一个纪元包含一些迭代。这实际上就是这个时代。让我们将epoch定义为数据集上的迭代次数，以训练神经网络。
An epoch contains a few iterations. That's actually what this epoch is. Let's define epoch as the number of iterations over the data set in order to train the neural network.
Epoch 是神经网络看到所有数据的 1 个完整周期。
有人可能会说需要 100,000 张图像来训练模型，但是，内存空间可能不足以一次处理所有图像，因此我们在称为批次的较小数据块上分割训练模型。例如批量大小为 100。
我们需要使用多个批量来覆盖所有图像。因此我们需要 1000 次迭代才能覆盖所有 100,000 张图像。 （100 批量大小 * 1000 次迭代）
一旦神经网络查看整个数据，它就被称为 1 Epoch（点 1）。人们可能需要多个时期来训练模型。 （假设 10 个 epoch）。
Epoch is 1 complete cycle where the Neural network has seen all the data.
One might have said 100,000 images to train the model, however, memory space might not be sufficient to process all the images at once, hence we split training the model on smaller chunks of data called batches. e.g. batch size is 100.
We need to cover all the images using multiple batches. So we will need 1000 iterations to cover all the 100,000 images. (100 batch size * 1000 iterations)
Once Neural Network looks at the entire data it is called 1 Epoch (Point 1). One might need multiple epochs to train the model. (let us say 10 epochs).
epoch 是用于训练的样本子集的迭代，例如神经网络中的梯度下降算法。一个很好的参考是： http://neuralnetworksanddeeplearning.com/chap1.html
请注意，该页面有一个使用 epoch 的梯度下降算法的代码
查看代码。对于每个时期，我们随机生成梯度下降算法的输入子集。页面上还解释了为什么 epoch 有效。请看一下。
An epoch is an iteration of a subset of the samples for training, for example, the gradient descent algorithm in a neural network. A good reference is: http://neuralnetworksanddeeplearning.com/chap1.html
Note that the page has a code for the gradient descent algorithm which uses epoch
Look at the code. For each epoch, we randomly generate a subset of the inputs for the gradient descent algorithm. Why epoch is effective is also explained on the page. Please take a look.
根据 Google 机器学习词汇表，纪元定义为
“对整个数据集进行完整的训练，使得每个示例都被看到一次，因此，一个时期代表
N/batch_size次训练迭代，其中 N 是示例的总数。”如果您正在使用批量大小 6 训练10 epoch模型，给定总共12个样本，这意味着：
模型将能够通过 2 次迭代（12 / 6 = 2）即单个纪元查看整个数据集。
总体而言，模型将进行 2 X 10 = 20 次迭代（每个时期的迭代次数 X 时期数）
每次迭代后将重新评估损失和模型参数！
According to Google's Machine Learning Glossary, an epoch is defined as
"A full training pass over the entire dataset such that each example has been seen once. Thus, an epoch represents
N/batch_sizetraining iterations, where N is the total number of examples."If you are training model for 10 epochs with batch size 6, given total 12 samples that means:
the model will be able to see the whole dataset in 2 iterations ( 12 / 6 = 2) i.e. single epoch.
overall, the model will have 2 X 10 = 20 iterations (iterations-per-epoch X no-of-epochs)
re-evaluation of loss and model parameters will be performed after each iteration!
作为奖励：
来源：https://developers.google.com/machine-learning/glossary/
as bonus:
source: https://developers.google.com/machine-learning/glossary/