在2个附加的EBS量之间传输数据的最快方法

发布于 2025-02-13 20:11:48 字数 1828 浏览 1 评论 0原文

我试图以最快的方式在EC2实例上复制2个EBS卷,同时可以安全地使用数据。从较大尺寸的音量到较小尺寸的音量时,我无法进行块级副本。我本质上是运行一个将“收缩”这些EBS卷的脚本(将原始卷的数据复制到较小的卷)。

我已经分类了许多文章,并尝试了CP -A和RCLONE -A之类的事情,因为我需要保留卷上的权限和元数据(因为Rclone不保留权限,因此排除了Rclone(排除Rclone)。我之所以降落在MSRSYNC上,是因为我认为它是最快的选择。

我还试图将较大体积的分区缩小到较小体积的大小,并使用DD和PV进行块级副本,但事实证明这很危险。我会在大多数时候看到一个损坏的驱动器,并在大多数时候看到一些怪异的故障,但是非常快。 200多M/s。

目前与MSRSYNC一起使用,仅需40个演出即可花费一个多小时。我将列出正在使用的变量。

  • AWS EC2 M5.8XLARGE运行AMI AMZN2-AMI-KERNEL-5.10-HVM-2.0.20220606.1-X86_64-GP2(认为较大的卷具有较高的吞吐量,并且M5的M5是EBS的EBS,但是它是EBS的优化。不如我想的那样多)。
  • EBS GP3在两个驱动器上都有10,000 IOP和750吞吐量。更少,需要很多小时。
  • MSRSYNC复制命令(documentation- https://github.com/jbd/msrsync ) :sudo ./msrsync -p16 -p -rsync“ -avaxewslhh -info = progress2“/mnt/datacopy $ j//mnt/newdataCopy $ j。我选择在可能的情况下使用16个线程,因为M5.8xlarge有许多核心。
  • 测试文件来自 https://fastest.fish/test-files 。我正在用ddfallocate生成我的测试文件,但是由于某些原因,它们不会显示为卷上的用法(确保权限是正常的)。我将尝试多个文件和一个文件,MSRSYNC可以帮助多个文件,但不能对单个文件有很大帮助。

MSRSYNC当前显示

[1/6 entries] [16.0 K/78.1 G transferred] [235 entries/s] [3.7 M/s bw] [monq 8] [jq 1]


[2/6 entries] [9.1 G/78.1 G transferred] [0 entries/s] [5.5 M/s bw] [monq 1] [jq 1]
[3/6 entries] [19.1 G/78.1 G transferred] [0 entries/s] [10.5 M/s bw] [monq 0] [jq 1]
[4/6 entries] [29.1 G/78.1 G transferred] [0 entries/s] [15.8 M/s bw] [monq 1] [jq 1]
[5/6 entries] [39.1 G/78.1 G transferred] [0 entries/s] [21.2 M/s bw] [monq 1] [jq 1]

有5个10GB测试文件,总计500GB EBS卷。

看完Iotop和HTOP后,我不确定自己在哪里瓶颈。即使正在使用的吞吐量和IOPS,EBS量是否也很慢?任何信息或提示都将不胜感激。

编辑:有关权限的调整后措辞

I'm trying to copy data between 2 ebs volumes on an EC2 instance the fastest way possible while being safe with the data. I cannot do a block level copy as I'm going from a bigger sized volume to a smaller sized volume. I'm essentially running a script that will "shrink" these ebs volumes (copying data from the original volume to a smaller volume).

I have sorted through many articles and tried things like cp -a and rclone -a as I need to retain the permissions and metadata on the volumes (ruling out rclone, as rclone doesn't retain permissions). I have landed on msrsync because of this as I had believed it to be the fastest option.

I have also tried to shrink the partition of the larger volume to the size of the smaller volume and do a block level copy with dd and pv, but that turned out to be dangerous. I would get a corrupted drive and see some weird glitches most of the time, however it is very fast. 200+ m/s.

Currently with msrsync, copying as little as 40 gigs is taking over an hour. I will list the variables I'm working with.

  • aws ec2 m5.8xlarge running ami amzn2-ami-kernel-5.10-hvm-2.0.20220606.1-x86_64-gp2 (thinking that larger volumes have higher throughput, and m5's are ebs optimized however that is turning out to not help as much as I thought).
  • ebs gp3 with 10,000 iops and 750 throughput on both drives. any less and it takes many hours.
  • msrsync copy command (documentation - https://github.com/jbd/msrsync) with the following flags: sudo ./msrsync -p16 -P --rsync "-avAXEWSlHh --info=progress2" /mnt/datacopy$j/ /mnt/newdatacopy$j. I opted to use 16 threads when possible as the m5.8xlarge has many cores.
  • test files are from https://fastest.fish/test-files. I WAS generating my test files with dd and fallocate, but for some reason they would not show up as usage on the volumes (after ensuring permissions were normal). I will try multiple and a single file, and msrsync helps the multiple files but can't help the single file much.

msrsync is showing

[1/6 entries] [16.0 K/78.1 G transferred] [235 entries/s] [3.7 M/s bw] [monq 8] [jq 1]


[2/6 entries] [9.1 G/78.1 G transferred] [0 entries/s] [5.5 M/s bw] [monq 1] [jq 1]
[3/6 entries] [19.1 G/78.1 G transferred] [0 entries/s] [10.5 M/s bw] [monq 0] [jq 1]
[4/6 entries] [29.1 G/78.1 G transferred] [0 entries/s] [15.8 M/s bw] [monq 1] [jq 1]
[5/6 entries] [39.1 G/78.1 G transferred] [0 entries/s] [21.2 M/s bw] [monq 1] [jq 1]

currently with 5 10GB test files, and a 500GB total ebs volume.

after looking at iotop and htop, I'm not sure where I'm bottlenecking. Are ebs volumes just slow even with the throughput and iops I'm using? Any information or tips would be extremely appreciated.

EDIT: adjusted phrasing regarding permissions

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文