适用于 Linux 的可复制自定义发行版构建系统
问题
我有一个大型基础设施，由多种运行 Linux 的服务器组成。例如，数据库服务器、负载平衡器、特定于应用程序的服务器。每种服务器都有许多实例，并且所有实例都需要可重现。
每种服务器基本上都是自定义发行版。自定义包括对上游包（其他上游版本、构建选项、补丁等）的更改，以及可能的一些额外的自定义包。
例如，我需要一台运行最新 OpenLDAP slapd 的服务器，并使用特定选项和一些补丁进行编译。这就是事情变得复杂的地方。
更新到最新的 slapd 还需要更新它所依赖的库，这意味着还要重建依赖于这些库的所有包。也就是说，我基本上需要重建发行版的重要部分。我正在寻找一种有助于自动化此过程的解决方案。
解决方案要求
有点模糊。我想准备构建自定义发行版所需的一切，为其指定一个名称（例如 ldap-server），并在需要重现构建时随时将该名称指定给自动构建系统。
我觉得这是 Gengoo 或者 LFS 社区应该有的东西。我也见过 ALT Linux Hasher、Fedora Mock、Debian pbuilder/sbuild 等项目，但从未使用过任何项目其中。
有什么想法吗？
提前致谢！
The Problem
I have a big infrastructure consisting of several kinds of servers running Linux. For instance, database servers, load balancers, application-specific servers. There are many instances of every kind of server, and all of them need to be reproducible.
Every kind of server is basically a custom distribution. Customisations include changes to the upstream packages (other upstream version, build options, patches, whatever) and, possibly, some extra custom packages.
For example, I need a server running the latest OpenLDAP slapd compiled with specific options and some patches. And this is where things get complicated.
Updating to the latest slapd will also require updating libraries it depends on, which means rebuilding all packages that depend on these libraries, too. That is I basically need to rebuild significant part of the distribution. I'm looking for a solution that helps automate this process.
Solution requirements
Kind of vague. I want to prepare everything necessary for building my custom distro, give it a name (e.g ldap-server) and give that name to the automated build system any time I need to reproduce the build.
I think this is something Gengoo or LFS community should have. Also I've seen projects like ALT Linux Hasher, Fedora Mock, Debian pbuilder/sbuild but never used any of them.
Any ideas?
Thanks in advance!
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（5）
您还可以使用 Nix 包管理器和 Hydra 构建系统来完成您的任务。
Nix 是一个纯函数式包管理器。
Hydra 是一个基于 Nix 的持续构建系统。 （AFAIU，它会在必要时重建依赖包。）
Nix 不仅可以跟踪包依赖项及其修改，还可以跟踪您的主机配置 - 以允许回滚到一致的先前状态。 （这就是基于 Nix 的 Linux 发行版 NixOS 背后的想法。）
还有：
You might also make use of the Nix package manager and Hydra build system for your task.
Nix is a purely functional package manager.
Hydra is a Nix-based continuous build system. (AFAIU, it does the rebuilding of dependent packages when necessary.)
Nix can not only track package dependencies and their modification, but also your host configuration--to allow rollbacks to consistent previous states. (That's the idea behind NixOS, a Nix-based Linux distribution.)
There is also:
我不会问为什么你选择为你的生产服务器维护一个自定义发行版......但是......我有过这种黑客马拉松的一些经验......以及巨大的头痛< /strong> 与之相配。
为了自动构建发行版，我使用了构建顺序和依赖项的 XML 定义，并使用脚本化的 GNU Make 来构建并行独立分支并构建二进制包。 XML+shell-script+python+Make/Autotools 位的输出是一组特殊“核心”工具和附加工具的完整构建。
第二步是将这些二进制文件/原始构建目录安装到系统中。我使用 installwatch （我认为）来使用 inotify 来监视东西的安装位置。然后，我输出该文件的 XML 以及任何二进制文件的依赖项。
此后，我有了一个构建清单 (XML)，并为每个包提供了一个包含已安装包详细信息的 XML 文件。然后，我制作了一个工具，将 XML 和就地二进制文件转换为各种格式（RPM 等）
现在（发挥你的想象力）我有一个安装脚本来自动构建，构建的包及其上的大量元数据依赖关系，以及将元数据转换为可部署包的方法
接下来，我为各种服务器制作了构建脚本，从 glib 开始:) ...并运行这些构建。系统知道哪些packages/./configure是通用的并共享这些包。这给我留下了
o 名为 /common 的存储库
o 每种构建类型和架构的存储库
一些脚本/rsync-over-ssh 和补丁管理脚本，您就可以离开。
显然这是我为公共环境构建多个发行版的方法的非常粗略的概述。有些包是影响源代码树的元包（但在构建时被视为普通包。一个例子是首先运行并向内核应用补丁的元包）。
然后是工具链自动化的问题。
这一切都始于 LFS……但正如你所看到的，事情变得有点冒险。
底线是，它非常有趣，但我只是为了 BSD 和 Fedora 放弃了一切。
像 Suse Build Service 这样的东西可能会让人感兴趣。将稳定源组合查找和编译外包出去将使事情变得更简单！您甚至不需要构建任何与 Suse 相关的东西。
I wont ask why you chose to maintain a custom distro for your production servers ... but ... I have had some experience of this kind of hackathon ... and the massive headaches that go with it.
To automate your build of the distro, I used an XML definition of the build order and dependencies and scripted GNU Make to build in parallel independent branches and construct the binary packages. The resulting output from the XML+shell-script+bit of python+Make/Autotools was a complete build of a special set of 'core' tools and then extras.
The second step was installing these binaries/raw build directories into a system. I used installwatch (i think) to use inotify to keep an eye on where things were installed to. I then output XML of this along with the dependencies of any binaries.
After this I had a build manifest (XML) and for each package an XML file with the details of the installed packages. I then made a tool to convert the XML and the in-place binaries into various formats (RPM etc)
Now (use your imagination) I have an install script to automate build, tons of meta-data on built packages and their dependencies, and a method of turning that meta-data into deployable packages
Next, I made build-scripts for various servers, from glib up :) ... and ran those builds. The system knew which packages/./configure's were common and shared those packages. This left me with
o A repo called /common
o A repo for each build type and architecture
A few scripts/rsync-over-ssh and patch management scripts and you are away.
obviously this is a very rough overview of my approach to building multiple distros for a common environment. Some packages were meta-packages that affected the source-tree (but were treated like normal packages as build time. One example was a meta-package that ran first and applied patches to the kernel).
Then there is the matter of tool-chain automation.
It all started off with LFS ... but as you can see, things got a little adventurous.
Bottom line is, it was very fun but I just ditched it all for a BSD and Fedora.
Things like the Suse Build Service might be of interest. Farming out the stable-source-combination-finding and compilation will make things simpler! You don't even need to build anything to do with Suse.
ALTLinux girar-builder 是系统（使用 girar-builder）。 org/Hasher”rel="nofollow">hasher 内部）来重建包并维护一致的包存储库。 Hasher 是一个隔离构建过程的工具，以便可以准确地“跟踪”所有需求，从而对构建过程的可重复性有一定的保证。
除此之外，girar-builder 还执行 依赖性检查 在向存储库添加（更新、删除）新构建的包时，这样新的包如果破坏了其他包的依赖关系，则不会被接受，除非其他依赖包也被添加到同一构建任务（= repo 更改事务）并在新包之后重建。这是一种经常可以观察到的讨论情况（带有 dep 的示例由于共享库中符号消失而损坏，包删除的示例）位于 ALTLinux 开发人员邮件列表（< a href="https://lists.altlinux.org/mailman/listinfo/devel-en" rel="nofollow">列表的英文版本）：“检测到新的未满足的依赖项”。为了继续进行，依赖包应该由其维护者添加到该任务中。
girar-builder 还对新软件包进行安装测试，仅举 git.alt (girar-bulder) 完成的另一项检查。
为了确保构建包可以在包存储库的当前状态下重现，会不时（非常定期）检查存储库中的每个包（称为 Sisyphus) 目前是可重建的 -- 重建测试状态报告，每个包的最后一次重建测试的日志。
ALTLinux girar-builder is the system (that uses hasher internally) to rebuild packages and maintain a consistent repository of packages. Hasher is a tool to isolate the build processes, so that all the requirements can be accurately "tracked" so that there are some guarantees as to the reproducibility of the build process.
Among other things, girar-builder does dependency checks when adding (updating, deleting) a newly built package to the repository, so that the new package won't be accepted if it breaks the dependencies of other packages, unless the other dependent packages are also added to the same build task (= repo changing transaction) and rebuilt after the new package. This is a situation that can often be observed being discussed (an example with a dep broken due to the disappearance of a symbol in a shared library, an example of a package deletion) in the ALTLinux developers mailing list (the English counterpart of the list): that "NEW unmet dependencies detected". In order to proceed, the dependent packages should be added by their maintainers to that task.
girar-builder also does an installation test for the new packages, just to name another check done by git.alt (girar-bulder).
In order to be sure that building the packages can be reproduced in the current state of the repository of packages, it is being checked from time to time (quite regularly) that every package in the repository (called Sisyphus) is rebuildable at the current moment -- a rebuild test status report, the logs of the last rebuild test, per package.
以防万一，自最初的问题以来，已经出现了对类似问题的另一个答案： mkimage-profiles它基于 ALT Linux 发行版相关的工具链，但通过映像配置管理工具对其进行了扩展，该工具试图使出现的分叉变得简约和简洁。到目前为止，它主要以俄语正式记录（这是我的决定，有几个原因），但代码本身有很好的英语注释。
要了解该方法，请参阅 conf.d/server.mk：
对 OpenVZ 模板缓存、VM 图像、ARM/PPC 拱门、git（如提交使用有意义的描述生成的配置文件的各个阶段）和配置树图等。
PXE 启动支持应该非常容易在框架内实现（并获得上游），但实际上尚未完成 - 我知道这些位，但必须解决它们。
对网络安装映像的初步支持从大约 17MB 开始（示例）。
我也对你认为 ALT 不切实际的特殊原因感兴趣 - 当然有一些已知的原因，但你的原因对我来说可能是新的:-) PS：特别是当或多或少准备好去 LFS 时。
PS2：您可以使用 live-builder.iso 在具有 4+ GB RAM 和启用 DHCP 的互联网路由以太网连接的系统上，只需以 altlinux 身份登录，cd /usr/share/ mkimage-profiles 和 make server-mini.iso
Just in case, a yet another answer to the similar set of problems has become available since the original question: mkimage-profiles which is based on ALT Linux distribution related toolchain but extends it with an image configuration management tool that is trying to make the occuring forks minimalistic and concise. It's mostly formally documented in Russian by now (it was my decision for several reasons) but the code itself is pretty well-commented in English.
To get a feeling of the approach see e.g. conf.d/server.mk:
There's some support for OpenVZ template caches, VM images, ARM/PPC arches, git (as in committing the stages of the profile being generated with meaningful descriptions) and configuration tree graphing, among the rest.
PXE boot support should be pretty straightforward to implement (and get upstream) within the framework but not actually done yet -- I know the bits but have to get around to them.
There's a preliminary support for netinstall images starting from ~17MB in size (example).
I'd also be interested in your particular reasons to find ALT impractical -- there are some known ones of course but yours might be new to me :-) PS: especially when being more or less ready to go as far as LFS.
PS2: you can try out the thing in live mode with live-builder.iso on a system with 4+ GB RAM and DHCP-enabled Internet-routed ethernet connection, just login as altlinux, cd /usr/share/mkimage-profiles and make server-mini.iso
我对此了解不多，但我认为 Suse Studio 值得一看。
I don't know much about it, but I think Suse Studio is worth a look.