从具有快速网络连接的虚拟机克隆 git 存储库时,我的瓶颈是什么?

发布于 2024-12-17 07:17:48 字数 686 浏览 1 评论 0 原文

我遇到过这样的情况,一个相对较大的 git 存储库位于本地网络上一台旧的、缓慢的主机上的虚拟机中,需要相当长的时间才能进行初始克隆。

ravn@bamboo:~/git$ git clone gitosis@gitbox:git00
Initialized empty Git repository in /home/ravn/git/git00/.git/
remote: Counting objects: 89973, done.
remote: Compressing objects: 100% (26745/26745), done.
remote: Total 89973 (delta 50970), reused 85013 (delta 47798)
Receiving objects: 100% (89973/89973), 349.86 MiB | 2.25 MiB/s, done.
Resolving deltas: 100% (50970/50970), done.
Checking out files: 100% (11722/11722), done.
ravn@bamboo:~/git$

gitosis 中没有特定于 git 的配置更改。

有什么方法可以将接收比特的速度加快到网络的能力吗?


我需要新的存储库与上游存储库正确连接。据我了解,这需要 git 进行克隆,因此在 git 之外进行原始位复制将不起作用。

I have a situation with a relatively large git repository located in a virtual machine on an elderly, slow host on my local network where it takes quite a while to do the initial clone.

ravn@bamboo:~/git$ git clone gitosis@gitbox:git00
Initialized empty Git repository in /home/ravn/git/git00/.git/
remote: Counting objects: 89973, done.
remote: Compressing objects: 100% (26745/26745), done.
remote: Total 89973 (delta 50970), reused 85013 (delta 47798)
Receiving objects: 100% (89973/89973), 349.86 MiB | 2.25 MiB/s, done.
Resolving deltas: 100% (50970/50970), done.
Checking out files: 100% (11722/11722), done.
ravn@bamboo:~/git$

There is no git specific configuration changes in gitosis.

Is there any way of speeding up the receiving bit up to what the network is capable of?


I need the new repositories to be properly connected with the upstream repository. To my understanding, this requires git to do the cloning, and thus raw bit copying outside of git will not work.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(6

七月上 2024-12-24 07:17:48

使用深度创建浅克隆。

git clone --depth 1 <repository>

Use the depth to create a shallow clone.

git clone --depth 1 <repository>
扭转时空 2024-12-24 07:17:48

附注。公平警告:

git 通常被认为速度非常快。您应该尝试从 darcs、bazaar、hg 克隆完整的存储库(上帝禁止:TFS 或 subversion...)。另外,如果您经常从头开始克隆完整的存储库,那么您无论如何都会做错事。您始终可以git远程更新并获得增量更改。

有关保持完整存储库同步的各种其他方法,请参见,例如

(包含其他相关 SO 帖子的链接)

哑副本

如前所述,您可以仅使用“哑”文件传输来复制存储库。

这当然不会浪费时间压缩、重新打包、增量化和/或过滤。

另外,您将获得

  • 挂钩
  • 配置(远程、推送分支、设置(空格、合并、别名、用户详细信息等)
  • 存储(请参阅我可以吗从远程存储库获取存储到本地分支? 也)
  • rerere 缓存
  • 引用
  • 日志备份(例如,来自过滤器分支)和各种其他事物(来自 rebase、bisect 等的中间状态) .)

这可能是也可能不是您所需要的,但很高兴知道


Bundle

Git clone 默认情况下会优化带宽,因为默认情况下 git clone 不会优化带宽。 镜像所有分支(请参阅--mirror),仅按原样转储包文件是没有意义的(因为这可能会发送超出所需数量的文件)。

当分发给真正大量的客户端时,考虑使用捆绑

如果您想要快速克隆而不需要服务器端成本,< em>git way 是 bundle创建。您现在可以分发捆绑包,甚至无需服务器参与。如果您的意思是 bundle... --all 包含的不仅仅是简单的 git clone,请考虑例如 bundle ... master 来减少体积。

git bundle create snapshot.bundle --all # (or mention specific ref names instead of --all)

并分发快照包。这是两全其美的方法,当然您不会从上面的项目符号列表中获得这些项目。在接收端,只需

git clone snapshot.bundle myclonedir/

压缩配置

即可通过减少/删除压缩来降低服务器负载。
看看这些配置设置(我假设 pack.compression 可以帮助您降低服务器负载)

核心.压缩

整数-1..9,表示默认压缩级别。 -1 是 zlib 默认值。 0 表示不压缩,1..9 是各种速度/大小的权衡,9 是最慢的。如果设置,这将为其他压缩变量提供默认值,例如
core.loosecompression 和 pack.compression。

core.loosecompression

整数 -1..9,表示不在包文件中的对象的压缩级别。 -1 是 zlib 默认值。 0 表示不压缩,1..9 是各种速度/大小的权衡,9 是最慢的。如果没有设置,则默认为
核心.压缩。如果未设置,则默认为 1(最佳速度)。

pack.compression

整数-1..9,表示包文件中对象的压缩级别。 -1 是 zlib 默认值。 0 表示不压缩,1..9 是各种速度/大小的权衡,9 是最慢的。如果未设置,则默认为 core.compression。如果那样的话
未设置,默认为 -1,zlib 默认值,这是“速度和压缩之间的默认折衷方案(当前相当于级别 6)。”

请注意,更改压缩级别不会自动重新压缩所有现有对象。您可以通过将 -F 选项传递给 git-repack(1) 来强制重新压缩。

如果网络带宽充足,这实际上会带来更快的克隆速度。 当您决定对其进行基准测试时,不要忘记 git-repack -F

PS. Fair warning:

git is generally considered blazingly fast. You should try cloning a full repo from darcs, bazaar, hg (god forbid: TFS or subversion...). Also, if you routinely clone full repos from scratch, you'd be doing something wrong anyway. You can always just git remote update and get incremental changes.

For various other ways to keep full repos in synch see, e.g.

(The contain links to other relevant SO posts)

Dumb copy

As mentioned you could just copy a repository with 'dumb' file transfer.

This will certainly not waste time compressing, repacking, deltifying and/or filtering.

Plus, you will get

  • hooks
  • config (remotes, push branches, settings (whitespace, merge, aliases, user details etc.)
  • stashes (see Can I fetch a stash from a remote repo into a local branch? also)
  • rerere cache
  • reflogs
  • backups (from filter-branch, e.g.) and various other things (intermediate state from rebase, bisect etc.)

This may or may not be what you require, but it is nice to be aware of the fact


Bundle

Git clone by default optimizes for bandwidth. Since git clone, by default, does not mirror all branches (see --mirror) it would not make sense to just dump the pack-files as-is (because that will send possibly way more than required).

When distributing to a truly big number of clients, consider using bundles.

If you want a fast clone without the server-side cost, the git way is bundle create. You can now distribute the bundle, without the server even being involved. If you mean that bundle... --all includes more than simple git clone, consider e.g. bundle ... master to reduce the volume.

git bundle create snapshot.bundle --all # (or mention specific ref names instead of --all)

and distribute the snapshot bundle instead. That's the best of both worlds, while of course you won't get the items from the bullet list above. On the receiving end, just

git clone snapshot.bundle myclonedir/

Compression configs

You can look at lowering server load by reducing/removing compression.
Have a look at these config settings (I assume pack.compression may help you lower the server load)

core.compression

An integer -1..9, indicating a default compression level. -1 is the zlib default. 0 means no compression, and 1..9 are various speed/size tradeoffs, 9 being slowest. If set, this provides a default to other compression variables, such as
core.loosecompression and pack.compression.

core.loosecompression

An integer -1..9, indicating the compression level for objects that are not in a pack file. -1 is the zlib default. 0 means no compression, and 1..9 are various speed/size tradeoffs, 9 being slowest. If not set, defaults to
core.compression. If that is not set, defaults to 1 (best speed).

pack.compression

An integer -1..9, indicating the compression level for objects in a pack file. -1 is the zlib default. 0 means no compression, and 1..9 are various speed/size tradeoffs, 9 being slowest. If not set, defaults to core.compression. If that
is not set, defaults to -1, the zlib default, which is "a default compromise between speed and compression (currently equivalent to level 6)."

Note that changing the compression level will not automatically recompress all existing objects. You can force recompression by passing the -F option to git-repack(1).

Given ample network bandwidth, this will in fact result in faster clones. Don't forget about git-repack -F when you decide to benchmark that!

静水深流 2024-12-24 07:17:48

2014 年建议的 git clone --depth=1 ... 建议将在第二季度变得更快2019 年使用 Git 2.22。
这是因为,在初始“git clone --depth=...”部分克隆期间,它是
为大部分连接花费周期是毫无意义的
检查是否枚举并跳过承诺对象(根据定义,这是从另一端获取的所有对象)。
这已经被优化掉了。

克隆:对部分克隆进行更快的对象检查

对于部分克隆,进行完整的连接检查是浪费的;我们跳过
promisor 对象(对于部分克隆,是所有已知对象),以及
枚举它们全部以将它们排除在连接检查之外可以
在大型仓库上花费大量时间。

最多,我们想确保我们得到任何引用的对象
想要参考。
对于部分克隆,只需检查这些对象是否已转移。

结果:

 测试 dfa33a2^ dfa33a2
 -------------------------------------------------- -----------------------
 5600.2:无斑点克隆 18.41(22.72+1.09) 6.83(11.65+0.50) -62.9%
 5600.3: 检验结果 1.82(3.24+0.26) 1.84(3.24+0.26) +1.1%

速度提高 62%!


在 Git 2.26(2020 年第一季度)中,现在在提取部分克隆时会禁用不需要的连接检查。

请参阅提交2df1aa2提交 5003377(2020 年 1 月 12 日),作者:乔纳森·谭(jhowtan
(由 Junio C Hamano -- gitster -- 合并于 提交 8fb3945,2020 年 2 月 14 日)

已连接:验证部分的承诺性克隆

签字人:Jonathan Tan
审阅者:Jonathan Nieder

提交dfa33a298d(“克隆:做得更快部分克隆的对象检查”,2019-04-21,Git v2.22.0-rc0 -- merge) 优化了使用 < 进行克隆时的连接检查code>--filter 仅检查直接指向的对象是否存在参考文献。
但这还不够:它们还需要是承诺对象。
通过检查这些对象是否是承诺对象(即它们出现在承诺包中)来使此检查更加稳健。

和:

fetch:如果<,则放弃完整连接检查代码>--过滤器

签字人:Jonathan Tan
审阅者:Jonathan Nieder

如果指定了过滤器,我们不需要对刚刚获取的包文件的内容进行完整的连接检查;我们只需要检查引用的对象是否是promisor对象。

这显着加快了对具有许多承诺者对象的存储库的提取速度,因为在连接检查期间,所有承诺者对象都被枚举(以将它们标记为无趣),并且这需要大量时间。


而且,仍然在 Git 2.26(2020 年第一季度)中,对象可达性位图机制和部分克隆机制无法很好地协同工作,因为部分克隆使用的一些对象过滤标准本质上依赖于对象遍历,但是位图机制是一种绕过该对象遍历的优化。

然而,在某些情况下他们可以一起工作,并且他们被教导了这些。

请参阅 提交 20a5fd8(2020 年 2 月 18 日),作者:Junio C Hamano (gitster)
请参阅 提交 3ab3185提交 84243da, 提交 4f3bd56, 提交 cc4aa28, 提交 2aaeb9a, 提交 6663ae0, 提交 4eb707e, 提交 ea047a8, 提交 608d9c9, 提交 55cb10f, 提交 792f811, 提交 d90fe06(2020 年 2 月 14 日),以及 提交 e03f928, 提交 acac50d, 提交 551cf8b(2020 年 2 月 13 日),作者:杰夫·金(peff
(由 Junio C Hamano -- gitster -- 合并于 提交 0df82d9,2020 年 3 月 2 日)

pack-bitmap:实现 BLOB_LIMIT 过滤

签字人:杰夫·金

正如之前的提交实现BLOB_NONE一样,我们可以支持BLOB_LIMIT< /code> 通过查看结果中任何 blob 的大小并根据需要取消设置它们的位来进行过滤。
这比 BLOB_NONE 稍微贵一点,但仍然会产生明显的加速(这些结果位于 git.git):

测试 HEAD~2 HEAD
-------------------------------------------------- ----------------------------------
5310.9:带 blob 的 rev-list 计数:无 1.80(1.77+0.02) 0.22(0.20+0.02) -87.8%
5310.10:带有 blob 的 rev-list 计数:limit=1k 1.99(1.96+0.03) 0.29(0.25+0.03) -85.4%

该实现与 BLOB_NONE 类似,不同之处在于我们在遍历 blob 类型位图时必须逐个对象(因为我们无法屏蔽掉匹配项,但必须单独查找每个斑点的大小)。
使用 ctz64() 的技巧取自 show_objects_for_type(),它同样需要找到各个位(但希望快速跳过没有斑点的大块)。


Git 2.27(2020 年第二季度)将简化部分克隆存储库中的提交祖先连通性检查,其中假定“承诺”对象可以从承诺者远程存储库按需延迟获取。

请参阅 提交 2b98478(2020 年 3 月 20 日),作者:Jonathan Tan (jhowtan)
(由 Junio C Hamano -- gitster -- 合并于 提交 0c60105,2020 年 4 月 22 日)

已连接:始终使用部分克隆优化< /h2>

签字人:Jonathan Tan
审阅者:Josh Steadmon

使用50033772d5(“已连接:验证承诺者-部分克隆的性质”, 2020-01-30,Git v2.26.0-rc0 -- 合并列于 <一个href="https://github.com/git/git/commit/7ae7e234c7e75dace3ba8aa8d87166ac7d75d7dd" rel="nofollow noreferrer">batch #5),check_connected() 中的快速路径(检查 promisor 包) 现在传递慢速路径的子集 (rev-list) > - 如果所有要检查的对象都在承诺包中找到,则快速路径和慢速路径都将通过;

  • 否则,快路肯定过不了。

这意味着每当我们需要执行慢速路径时,我们始终可以尝试快速路径。

快速路径当前由旗帜守卫;因此,删除该标志。
另外,使快速路径回退到慢速路径 - 如果快速路径失败,失败的 OID 和所有剩余的 OID 将传递到 rev-list。

用户可见的主要好处是从部分克隆获取的性能 - 具体来说,是在获取之前完成的连接检查的加速。
特别是,对我的计算机上的部分克隆进行无操作提取的时间从 7 秒加快到 0.01 秒。这是对 2df1aa239c ("fetch:放弃完全连接检查 if --filter", 2020-01-30,Git v2.26.0-rc0 -- 合并列于 <一个href="https://github.com/git/git/commit/7ae7e234c7e75dace3ba8aa8d87166ac7d75d7dd" rel="nofollow noreferrer">batch #5),它是上述 50033772d5。在那次提交中,提取速度之后的连接检查速度加快了。

在这些情况下,添加快速路径可能会导致性能降低:

  • 如果部分克隆或获取部分克隆失败,Git 将徒劳地运行 rev-list (预计获取的所有内容都会进入 promisor 包,因此如果没有不会发生,rev-list 很可能也会失败)。

  • 在部分克隆服务 receive-pack 的情况下(在我看来,不太可能),receive-pack 完成的任何连接检查。

我认为这些情况非常罕见,并且这种情况下的性能下降足够小(额外的对象数据库访问),避免使用标志的好处超过了这些。


在 Git 2.27(2020 年第 2 季度)中,使用对象过滤器“--filter=tree:0”的对象遍历现在可以利用可用的包位图。

请参阅 提交 9639474提交 5bf7f1e(2020 年 5 月 4 日),作者:杰夫·金(peff
请参阅提交b0a8d48提交 856e12c(2020 年 5 月 4 日),作者:泰勒·布劳 (ttaylorr)
(由 Junio C Hamano -- gitster -- 合并于 提交 69ae8ff,2020 年 5 月 13 日)

pack-bitmap.c:支持 '树:0'过滤

签字人:Taylor Blau

在上一个补丁中,我们可以轻松定义排除某种类型的所有对象的其他过滤器。当“n”等于 0 时,使用它可以为“--filter=tree:”过滤器实现位图级过滤.

一般情况下位图没有帮助,因为对于 'n > 的值0',对象过滤机制需要完整的树遍历才能确定给定树的深度。
缓存这一点也是不明显的,因为同一树对象可以根据上下文具有不同的深度(例如,树在两次提交之间的目录层次结构中向上移动)。

但是,“n = 0”的情况可以得到帮助,而这个补丁就是这样做的。
在该树和带有内核的 master 上运行 p5310.11 ,我们可以看到这种情况得到了很大的帮助:

测试掌握这棵树
-------------------------------------------------- ------------------------------------------
5310.11:树的转速列表计数:0 10.68(10.39+0.27) 0.06(0.04+0.01) -99.4%

并且:

请参阅 提交9639474提交 5bf7f1e(2020 年 5 月 4 日),作者:杰夫·金 (peff)
请参阅提交b0a8d48提交 856e12c(2020 年 5 月 4 日),作者:泰勒·布劳 (ttaylorr)
(由 Junio C Hamano -- gitster -- 合并于 提交 69ae8ff,2020 年 5 月 13 日)

pack-bitmap:将对象过滤器传递给填充遍历

签署人:杰夫·金
签字人:Taylor Blau

有时,位图遍历仍然需要手动遍历一些提交,因为这些提交未包含在位图包文件中(例如,由于自上次完全重新打包以来的推送或提交)。

如果给我们一个对象过滤器,我们不会将其传递给此遍历。
这对于正确性来说不是必需的,因为位图代码有自己的过滤器来对位图结果进行后处理(它必须这样做,以过滤掉位图包文件中提到的对象)。

对于 blob 过滤器,也没有任何性能原因来传递这些过滤器。填充遍历可能会从结果中省略它们,但这不会节省我们任何时间,因为我们仍然需要遍历每个树条目来查看它是否是一个 blob。

但现在我们支持树过滤器,因此有机会节省成本。 tree:depth=0 过滤器意味着我们可以完全避免访问树,因为我们知道我们不会访问它们(或它们指向的任何子树或 blob)。
p5310 中的新测试证明了这一点(“部分位图”状态是指 HEAD~100 及其祖先都位于位图包中,但 HEAD ~100..HEAD 不是)。

以下是结果(针对 linux.git 运行):

测试 HEAD^ HEAD
-------------------------------------------------- -----------------------------------------------------------
[...]
5310.16:带树过滤器的转速列表(部分位图)0.19(0.17+0.02) 0.03(0.02+0.01) -84.2%

节省的绝对数量并不是巨大,但请记住,我们只省略了 100 个第一个父链接(在此处的 linux.git 版本中,这是 894 次实际提交)。

在更病态的情况下,我们可能有更大比例的非位图提交。我没有费心在 perf 脚本中创建这样的情况,因为设置成本很高,这足以以百分比形式显示节省的费用。


在 Git 2.32(2021 年第 2 季度)中,允许某些对象丢失和可延迟检索的“承诺包”的处理已经得到优化(有点)。

请参阅提交c1fa951提交 45a187c, 提交 fcc07e9(2021 年 4 月 13 日),作者:杰夫·金(peff
(由 Junio C Hamano -- gitster -- 合并于 提交 13158b9,2021 年 4 月 30 日)

修订:避免使用 --exclude 进行解析-承诺对象

签字人:杰夫·金

当给出 --exclude-promisor-objects 时,在遍历任何对象之前,我们会迭代任何 promisor 包中的所有对象,将它们标记为 UNINTERESTING 和 SEEN。
我们使用 parse_object() 将迭代包中获得的 oid 转换为对象,但这有两个问题:

  • 速度很慢;我们正在 zlib 膨胀(并从增量重建)packfile 中每个对象的每个字节
  • 它使树缓冲区附加到其结构,这意味着我们的堆使用量将增长以同时存储每个未压缩的树。
    这可能是千兆字节。

我们显然可以通过在解析树缓冲区后释放它们来解决第二个问题。
但我们可以观察到该函数根本不查看对象内容!我们调用 parse_object() 的唯一原因是我们需要一个“struct object”来设置标志。
这里有两个选项:

  • 我们可以通过 oid_object_info() 查找对象类型,然后调用适当的 lookup_foo() 函数
  • 我们可以调用lookup_unknown_object(),它给我们一个OBJ_NONE结构(稍后将通过object_as_type()自动转换)调用 lookup_commit() 等)。

第一个更接近当前代码,但我们确实付出了代价来查找每个对象的类型。
后者在 CPU 上应该更高效,尽管它浪费了一点内存(“未知”对象结构是所有对象类型的联合,因此某些结构比它们需要的要大)。
它还存在触发直接调用 lookup_object() 但尚未准备好处理 OBJ_NONE 的代码中潜在错误的风险(此类代码本来就存在错误,但我们使用 lookup_unknown_object() 的频率太低,以至于它可能会隐藏)。

我在这里选择了第二个选项。
我认为风险并不高(无论如何我们都希望找到并修复任何此类错误),并且总体上应该更高效。

p5600 中的新测试展示了改进(位于 git.git):< /p>

测试 HEAD^ HEAD
-------------------------------------------------- ----------------------------
5600.5:提交计数 0.37(0.37+0.00) 0.38(0.38+0.00) +2.7%
5600.6:非承诺者提交计数 11.74(11.37+0.37) 0.04(0.03+0.00) -99.7%

此脚本中的改进特别大,因为新克隆的部分存储库中的每个对象都是 promisor 对象。
所以在标记完它们之后,就没有什么可以遍历的了。


较早的优化丢弃了仍在使用的树对象缓冲区,该缓冲区已用GIT 2.38(Q3 2022)校正。

参见 noreflow noreferrer“ github.com/peff“ rel =” nofollow noreferrer“> jeff king( peff
(由 Junio C Hamano -- gitster -- 合并于 commit 01a30a5 ,2022年8月25日)

报告:安德鲁·奥尔森
签名:Jeff King

以来解析,2021-04-13),我们将始终释放搜索承诺链接后,附加到“结构树”上的缓冲区。
但是,有一个重要的情况,我们不想这样做:如果其他人已经在使用这棵树!

这可以在“ rev-list-missing =允许启示录”中发生,在缺少一个或多个树或斑点的部分克隆中进行遍历。
即使是“ - 缺少=允许启动”的微不足道使用也会触发此问题,因为随附的测试证明了(这只是一个香草 - blob:none clone )。

The git clone --depth=1 ... suggested in 2014 will become faster in Q2 2019 with Git 2.22.
That is because, during an initial "git clone --depth=..." partial clone, it is
pointless to spend cycles for a large portion of the connectivity
check that enumerates and skips promisor objects (which by definition is all objects fetched from the other side).
This has been optimized out.

clone: do faster object check for partial clones

For partial clones, doing a full connectivity check is wasteful; we skip
promisor objects (which, for a partial clone, is all known objects), and
enumerating them all to exclude them from the connectivity check can
take a significant amount of time on large repos.

At most, we want to make sure that we get the objects referred to by any
wanted refs.
For partial clones, just check that these objects were transferred.

Result:

 Test                          dfa33a2^         dfa33a2
 -------------------------------------------------------------------------
 5600.2: clone without blobs   18.41(22.72+1.09)   6.83(11.65+0.50) -62.9%
 5600.3: checkout of result    1.82(3.24+0.26)     1.84(3.24+0.26) +1.1%

62% faster!


With Git 2.26 (Q1 2020), an unneeded connectivity check is now disabled in a partial clone when fetching into it.

See commit 2df1aa2, commit 5003377 (12 Jan 2020) by Jonathan Tan (jhowtan).
(Merged by Junio C Hamano -- gitster -- in commit 8fb3945, 14 Feb 2020)

connected: verify promisor-ness of partial clone

Signed-off-by: Jonathan Tan
Reviewed-by: Jonathan Nieder

Commit dfa33a298d ("clone: do faster object check for partial clones", 2019-04-21, Git v2.22.0-rc0 -- merge) optimized the connectivity check done when cloning with --filter to check only the existence of objects directly pointed to by refs.
But this is not sufficient: they also need to be promisor objects.
Make this check more robust by instead checking that these objects are promisor objects, that is, they appear in a promisor pack.

And:

fetch: forgo full connectivity check if --filter

Signed-off-by: Jonathan Tan
Reviewed-by: Jonathan Nieder

If a filter is specified, we do not need a full connectivity check on the contents of the packfile we just fetched; we only need to check that the objects referenced are promisor objects.

This significantly speeds up fetches into repositories that have many promisor objects, because during the connectivity check, all promisor objects are enumerated (to mark them UNINTERESTING), and that takes a significant amount of time.


And, still with Git 2.26 (Q1 2020), The object reachability bitmap machinery and the partial cloning machinery were not prepared to work well together, because some object-filtering criteria that partial clones use inherently rely on object traversal, but the bitmap machinery is an optimization to bypass that object traversal.

There however are some cases where they can work together, and they were taught about them.

See commit 20a5fd8 (18 Feb 2020) by Junio C Hamano (gitster).
See commit 3ab3185, commit 84243da, commit 4f3bd56, commit cc4aa28, commit 2aaeb9a, commit 6663ae0, commit 4eb707e, commit ea047a8, commit 608d9c9, commit 55cb10f, commit 792f811, commit d90fe06 (14 Feb 2020), and commit e03f928, commit acac50d, commit 551cf8b (13 Feb 2020) by Jeff King (peff).
(Merged by Junio C Hamano -- gitster -- in commit 0df82d9, 02 Mar 2020)

pack-bitmap: implement BLOB_LIMIT filtering

Signed-off-by: Jeff King

Just as the previous commit implemented BLOB_NONE, we can support BLOB_LIMIT filters by looking at the sizes of any blobs in the result and unsetting their bits as appropriate.
This is slightly more expensive than BLOB_NONE, but still produces a noticeable speedup (these results are on git.git):

Test                                         HEAD~2            HEAD
------------------------------------------------------------------------------------
5310.9:  rev-list count with blob:none       1.80(1.77+0.02)   0.22(0.20+0.02) -87.8%
5310.10: rev-list count with blob:limit=1k   1.99(1.96+0.03)   0.29(0.25+0.03) -85.4%

The implementation is similar to the BLOB_NONE one, with the exception that we have to go object-by-object while walking the blob-type bitmap (since we can't mask out the matches, but must look up the size individually for each blob).
The trick with using ctz64() is taken from show_objects_for_type(), which likewise needs to find individual bits (but wants to quickly skip over big chunks without blobs).


Git 2.27 (Q2 2020) will simplify the commit ancestry connectedness check in a partial clone repository in which "promised" objects are assumed to be obtainable lazily on-demand from promisor remote repositories.

See commit 2b98478 (20 Mar 2020) by Jonathan Tan (jhowtan).
(Merged by Junio C Hamano -- gitster -- in commit 0c60105, 22 Apr 2020)

connected: always use partial clone optimization

Signed-off-by: Jonathan Tan
Reviewed-by: Josh Steadmon

With 50033772d5 ("connected: verify promisor-ness of partial clone", 2020-01-30, Git v2.26.0-rc0 -- merge listed in batch #5), the fast path (checking promisor packs) in check_connected() now passes a subset of the slow path (rev-list) > - if all objects to be checked are found in promisor packs, both the fast path and the slow path will pass;

  • otherwise, the fast path will definitely not pass.

This means that we can always attempt the fast path whenever we need to do the slow path.

The fast path is currently guarded by a flag; therefore, remove that flag.
Also, make the fast path fallback to the slow path - if the fast path fails, the failing OID and all remaining OIDs will be passed to rev-list.

The main user-visible benefit is the performance of fetch from a partial clone - specifically, the speedup of the connectivity check done before the fetch.
In particular, a no-op fetch into a partial clone on my computer was sped up from 7 seconds to 0.01 seconds. This is a complement to the work in 2df1aa239c ("fetch: forgo full connectivity check if --filter", 2020-01-30, Git v2.26.0-rc0 -- merge listed in batch #5), which is the child of the aforementioned 50033772d5. In that commit, the connectivity check after the fetch was sped up.

The addition of the fast path might cause performance reductions in these cases:

  • If a partial clone or a fetch into a partial clone fails, Git will fruitlessly run rev-list (it is expected that everything fetched would go into promisor packs, so if that didn't happen, it is most likely that rev-list will fail too).

  • Any connectivity checks done by receive-pack, in the (in my opinion, unlikely) event that a partial clone serves receive-pack.

I think that these cases are rare enough, and the performance reduction in this case minor enough (additional object DB access), that the benefit of avoiding a flag outweighs these.


With Git 2.27 (Q2 2020), the object walk with object filter "--filter=tree:0" can now take advantage of the pack bitmap when available.

See commit 9639474, commit 5bf7f1e (04 May 2020) by Jeff King (peff).
See commit b0a8d48, commit 856e12c (04 May 2020) by Taylor Blau (ttaylorr).
(Merged by Junio C Hamano -- gitster -- in commit 69ae8ff, 13 May 2020)

pack-bitmap.c: support 'tree:0' filtering

Signed-off-by: Taylor Blau

In the previous patch, we made it easy to define other filters that exclude all objects of a certain type. Use that in order to implement bitmap-level filtering for the '--filter=tree:<n>' filter when 'n' is equal to 0.

The general case is not helped by bitmaps, since for values of 'n > 0', the object filtering machinery requires a full-blown tree traversal in order to determine the depth of a given tree.
Caching this is non-obvious, too, since the same tree object can have a different depth depending on the context (e.g., a tree was moved up in the directory hierarchy between two commits).

But, the 'n = 0' case can be helped, and this patch does so.
Running p5310.11 in this tree and on master with the kernel, we can see that this case is helped substantially:

Test                                  master              this tree
--------------------------------------------------------------------------------
5310.11: rev-list count with tree:0   10.68(10.39+0.27)   0.06(0.04+0.01) -99.4%

And:

See commit 9639474, commit 5bf7f1e (04 May 2020) by Jeff King (peff).
See commit b0a8d48, commit 856e12c (04 May 2020) by Taylor Blau (ttaylorr).
(Merged by Junio C Hamano -- gitster -- in commit 69ae8ff, 13 May 2020)

pack-bitmap: pass object filter to fill-in traversal

Signed-off-by: Jeff King
Signed-off-by: Taylor Blau

Sometimes a bitmap traversal still has to walk some commits manually, because those commits aren't included in the bitmap packfile (e.g., due to a push or commit since the last full repack).

If we're given an object filter, we don't pass it down to this traversal.
It's not necessary for correctness because the bitmap code has its own filters to post-process the bitmap result (which it must, to filter out the objects that are mentioned in the bitmapped packfile).

And with blob filters, there was no performance reason to pass along those filters, either. The fill-in traversal could omit them from the result, but it wouldn't save us any time to do so, since we'd still have to walk each tree entry to see if it's a blob or not.

But now that we support tree filters, there's opportunity for savings. A tree:depth=0 filter means we can avoid accessing trees entirely, since we know we won't them (or any of the subtrees or blobs they point to).
The new test in p5310 shows this off (the "partial bitmap" state is one where HEAD~100 and its ancestors are all in a bitmapped pack, but HEAD~100..HEAD are not).

Here are the results (run against linux.git):

Test                                                  HEAD^               HEAD
-------------------------------------------------------------------------------------------------
[...]
5310.16: rev-list with tree filter (partial bitmap)   0.19(0.17+0.02)     0.03(0.02+0.01) -84.2%

The absolute number of savings isn't huge, but keep in mind that we only omitted 100 first-parent links (in the version of linux.git here, that's 894 actual commits).

In a more pathological case, we might have a much larger proportion of non-bitmapped commits. I didn't bother creating such a case in the perf script because the setup is expensive, and this is plenty to show the savings as a percentage.


With Git 2.32 (Q2 2021), handling of "promisor packs" that allows certain objects to be missing and lazily retrievable has been optimized (a bit).

See commit c1fa951, commit 45a187c, commit fcc07e9 (13 Apr 2021) by Jeff King (peff).
(Merged by Junio C Hamano -- gitster -- in commit 13158b9, 30 Apr 2021)

revision: avoid parsing with --exclude-promisor-objects

Signed-off-by: Jeff King

When --exclude-promisor-objects is given, before traversing any objects we iterate over all of the objects in any promisor packs, marking them as UNINTERESTING and SEEN.
We turn the oid we get from iterating the pack into an object with parse_object(), but this has two problems:

  • it's slow; we are zlib inflating (and reconstructing from deltas) every byte of every object in the packfile
  • it leaves the tree buffers attached to their structs, which means our heap usage will grow to store every uncompressed tree simultaneously.
    This can be gigabytes.

We can obviously fix the second by freeing the tree buffers after we've parsed them.
But we can observe that the function doesn't look at the object contents at all! The only reason we call parse_object() is that we need a "struct object" on which to set the flags.
There are two options here:

  • we can look up just the object type via oid_object_info(), and then call the appropriate lookup_foo() function
  • we can call lookup_unknown_object(), which gives us an OBJ_NONE struct (which will get auto-converted later by object_as_type() via calls to lookup_commit(), etc).

The first one is closer to the current code, but we do pay the price to look up the type for each object.
The latter should be more efficient in CPU, though it wastes a little bit of memory (the "unknown" object structs are a union of all object types, so some of the structs are bigger than they need to be).
It also runs the risk of triggering a latent bug in code that calls lookup_object() directly but isn't ready to handle OBJ_NONE (such code would already be buggy, but we use lookup_unknown_object() infrequently enough that it might be hiding).

I went with the second option here.
I don't think the risk is high (and we'd want to find and fix any such bugs anyway), and it should be more efficient overall.

The new tests in p5600 show off the improvement (this is on git.git):

Test                                 HEAD^               HEAD
-------------------------------------------------------------------------------
5600.5: count commits                0.37(0.37+0.00)     0.38(0.38+0.00) +2.7%
5600.6: count non-promisor commits   11.74(11.37+0.37)   0.04(0.03+0.00) -99.7%

The improvement is particularly big in this script because every object in the newly-cloned partial repo is a promisor object.
So after marking them all, there's nothing left to traverse.


An earlier optimization discarded a tree-object buffer that is still in use, which has been corrected with Git 2.38 (Q3 2022).

See commit 1490d7d (14 Aug 2022) by Jeff King (peff).
(Merged by Junio C Hamano -- gitster -- in commit 01a30a5, 25 Aug 2022)

is_promisor_object(): fix use-after-free of tree buffer

Reported-by: Andrew Olsen
Signed-off-by: Jeff King

Since commit fcc07e9 (is_promisor_object(): free tree buffer after parsing, 2021-04-13, Git v2.32.0-rc0 -- merge listed in batch #13) (is_promisor_object(): free tree buffer after parsing, 2021-04-13), we'll always free the buffers attached to a "struct tree" after searching them for promisor links.
But there's an important case where we don't want to do so: if somebody else is already using the tree!

This can happen during a "rev-list --missing=allow-promisor" traversal in a partial clone that is missing one or more trees or blobs.
Even a trivial use of "--missing=allow-promisor" triggers this problem, as the included test demonstrates (it's just a vanilla --blob:none clone).

友谊不毕业 2024-12-24 07:17:48

我正在对 git clone 进行基准测试。

如果项目包含子模块,使用 --jobs 选项可以更快
前任:

git clone --recursive --shallow-submodules --depth 1 --branch "your tag or branch" --jobs 5 --  "your remote repo"

I'm bench marking git clone.

It can be faster with --jobs options if the project include submodules
ex:

git clone --recursive --shallow-submodules --depth 1 --branch "your tag or branch" --jobs 5 --  "your remote repo"
遇到 2024-12-24 07:17:48

在意识到数据传输速度的上限是在git“外部”建立的ssh连接之后,我做了一些实验,发现使用pcsp(Putty scp)的上限是3,0 MB/s因为河豚加密方案选择得当。使用raw ftp进行控制实验表明传输速度为3.1 MB/s,因此表明这是网络的上限。

它在 vmware hypervisor 内运行,并且由于执行网络 I/O 的进程几乎使用了 100% cpu,因此表明瓶颈是 Ubuntu 网卡驱动程序。然后我发现,即使安装了 vmware 工具,由于某种原因,内核仍然使用 vlance 驱动程序(模拟具有 IRQ 等的 10 MBps 网卡)而不是 vmxnet 驱动程序(直接与虚拟机管理程序对话)。现在等待服务窗口进行更改。

换句话说,问题不在于git,而在于底层的“硬件”。

After realizing that the upper limit to the transfer speed of data, is the ssh connection which is established "outside" of git I did some experiments, and found that the upper limit of using pcsp (Putty scp) was 3,0 MB/s as the blowfish encryption scheme was properly chosen. A control experiment with raw ftp showed that the transfer speed was 3.1 MB/s, so it indicated that this was the upper bound of the network.

This runs inside a vmware hypervisor, and as the process doing network I/O utilized almost 100% cpu it indicated that the bottleneck was the Ubuntu network card driver. I then found that even though vmware tools were installed, for some reason the kernel still used the vlance driver (emulating a 10 MBps network card with IRQ's and all) instead of the vmxnet driver (which speaks directly to the hypervisor). This now awaits a service window to be changed.

In other words, the problem was not with git but the underlying "hardware".

一人独醉 2024-12-24 07:17:48

从日志来看,您似乎已经完成了克隆,如果您的问题是您需要在不同的计算机上多次执行此过程,您只需将存储库目录从一台计算机复制到另一台计算机即可。这种方式将保留每个副本和您克隆的存储库之间的关系(远程)。

From the log it seems you already finished the clone, if your problem is that you need to do this process multiple times on different machines, you can just copy the repository directory from one machine to the other. This way will preserve the relationship (remotes) between each copy and the repository you cloned from.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文