从带有样本的多项分布中抽取一个巨大的样本（1e09）

发布于 2024-12-11 23:08:51 字数 392 浏览 0 评论 0原文

我想从多项分布中采样。我将通过使用样本并指定一些概率来做到这一点。例如：我有 3 个类别，我想采样 10 次。

> my_prob = c(0.2, 0.3, 0.5)
> x = sample(c(0:2), 100, replace = T, prob = my_prob)
> head(x)
[1] 2 0 2 1 1 2

我的设置现在仅在以下方面有所不同：我想采样很多（例如 1e09）数字。实际上我只对每个类别的频率感兴趣。因此，在上面提到的示例中，这意味着：

> table(x)
x
 0  1  2 
27 29 44

有人知道如何尽可能高效地计算它吗？

谢谢，斯特菲

原文

I would like to sample from a multinomial distribution. I would do this by using sample and specifying some probabilites.
E.g: I have 3 categories, and I want to sample 10 times.

> my_prob = c(0.2, 0.3, 0.5)
> x = sample(c(0:2), 100, replace = T, prob = my_prob)
> head(x)
[1] 2 0 2 1 1 2

My setting is now only different in the following aspect: I want to sample a lot (e.g. 1e09) numbers. And actually I am only interested in the frequency of each category.
So in the above mentioned example this would mean:

> table(x)
x
 0  1  2 
27 29 44

Does anybody have an idea how to compute this as efficient as possible?

thanks,
steffi

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

稍尽春風 2024-12-18 23:08:51

您需要rmultinom。

my_prob <- c(0.2,0.3,0.5)
number_of_experiments <- 10
number_of_samples <- 100
experiments <- rmultinom(n=number_of_experiments, size=number_of_samples, prob=my_prob)
experiments

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
     [1,]   14   18   15   19   14   17   23   18   24    15
     [2,]   33   34   36   30   40   30   27   38   24    30
     [3,]   53   48   49   51   46   53   50   44   52    55

You need rmultinom.

my_prob <- c(0.2,0.3,0.5)
number_of_experiments <- 10
number_of_samples <- 100
experiments <- rmultinom(n=number_of_experiments, size=number_of_samples, prob=my_prob)
experiments

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
     [1,]   14   18   15   19   14   17   23   18   24    15
     [2,]   33   34   36   30   40   30   27   38   24    30
     [3,]   53   48   49   51   46   53   50   44   52    55

回复收藏 0 原文

谜兔 2024-12-18 23:08:51

如果问题是您无法将长度为 1e9 的向量放入 RAM，那么您可以针对较少数量的样本重复计算该表并将总数相加。

n_total <- 1e9
n_chunk <- 1e6
n_iter <- n_total / n_chunk
my_prob = c(0.2, 0.3, 0.5)
totals <- numeric(3)
for(i in seq_len(n_iter))
{
  totals <- totals + table(sample(0:2, n_chunk, replace = TRUE, prob = my_prob))
}
totals
stopifnot(sum(totals) == n_total)

就像Max说的，您可能更喜欢 rmultinom 而不是示例。获取其 experiments 变量的 rowSums。

If the problem is that you can't fit a vector of length 1e9 into RAM, then you can repeatedly calculate the table for a smaller number of samples and add up the totals.

n_total <- 1e9
n_chunk <- 1e6
n_iter <- n_total / n_chunk
my_prob = c(0.2, 0.3, 0.5)
totals <- numeric(3)
for(i in seq_len(n_iter))
{
  totals <- totals + table(sample(0:2, n_chunk, replace = TRUE, prob = my_prob))
}
totals
stopifnot(sum(totals) == n_total)

Like Max said, you might prefer rmultinom over sample. Take the rowSums of his experiments variable.

回复收藏 0 原文

~没有更多了~