降级构建？如何在 Jenkins 中删除升级版本并在删除时运行指定脚本
在我正在工作的项目中，我们有一个持续部署设置。目标是始终将最新的工作版本安装到生产中，除非有人手动覆盖此功能。
为了使这项工作正常进行，我们
- 运行静态代码分析
- 运行单元测试运行
- 集成测试
- 运行自动 UI 测试（在可行的范围内）
如果上述任何步骤失败，则构建过程将停止，并将构建标记为失败。如果创建了安装包，则分步骤安装到
CI -->分期-->生产
在每个步骤中，我们都会对环境运行集成和 UI 测试，以确保我们没有引入一些在后续环境中失败的新内容。如果没有一个测试失败，并且 N 分钟过去了，没有人按下紧急按钮，则构建将升级到下一个环境。如果测试失败，我们要删除该包，并将其完全丢弃。然而，安装包是传递到其他服务器的，因此我们需要运行一堆远程（shell）脚本来完成这一步。
问题是，有大量的失败案例我们无法在正常的自动化周期中进行可靠的测试，例如页面布局，或者某些集成仅在生产中失败等等。
所以实际问题：一旦升级后，我该如何降级/删除构建？是否可以在执行删除构建时运行远程脚本或使用任何升级插件来实现此功能？是否有一些我可能没有想到的跳出框框思考的解决方案？
In the project I'm working for we're having a continuous deployment setup. The goal is to always install the latest working build to production, unless someone manually overrides this functionality.
In order to make this working we
- Run static code analysis
- Run unit tests
- Run integration tests
- Run automatic UI tests, to the extent this is feasible
If any of the above steps fail, the build process is halted, and the build marked as failed. If the installation package is created it is then in steps installed to
CI --> staging --> production
At each step we run a integration and UI tests for the environment, to make sure we didn't introduce some new things which fail on on the subsequent environments. If none of the tests fail, and N minutes pass without anyone pressing the panic button, the build gets promoted to the next env. If the tests fail, we want to delete the package, and discard it completely. The installation packages are, however, delivered to other servers, so we need to run a bunch of remote (shell) scripts to make this step happen.
The problem is, that there are a big set of failure cases which we cannot reliably test in the normal automated cycle, e.g. page layout, or some integrations fail only production and so on.
So the actual question: How shall I demote/delete builds, once they've been promoted? Is it possible to either run a remote script when doing delete build or use any of the promotion plugins to achieve this functionality? Is there some think-outside-the-box solution for this that I might not have thought about?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
您可以编写一个 Jenkins 作业，该作业接受内部版本号作为参数，删除它，然后执行其余的内务处理，而不是手动删除内部版本。您可以配置 Jenkins 访问权限，以便人们不会意外手动删除构建。
Instead of deleting builds manually, you may write a Jenkins job that accepts the build number as a parameter, deletes it, and then does the rest of the housekeeping. You can configure Jenkins access privileges so that people do not delete builds manually by accident.
这可能是一种非常特殊的情况，但我们决定不创建一个单独的作业来删除构建，原因很简单，即将与特定构建号相关的所有日志记录保存在一个位置。设置如下：
此处的升级意味着使安装包 (RPM) 可用于给定服务器，其中自动更新处理包的实际升级。
我们有一个主要构建，每次有新的提交可用时都会构建。我们对安静时间等进行了一些微调，但基本上每一个新推送的提交集都会产生一个新的构建。该构建包含所有相关且可用的测试，但它远未完成，而且可能永远不会完成。
每小时都有一个单独的升级步骤处理从暂存到生产的升级。此构建启动了一项单独的促销活动，将最新接受的构建从 CI 转移到暂存阶段。在构建升级 CI--> staging 之前有 30 分钟的延迟，以防止最后一秒提交的意外升级。通过一些 bash find 脚本实现了延迟。升级的顺序是这样的，以确保在升级之前（至少）一小时内可以使用构建版本。
实际答案：
升级步骤是作为单独的构建完成的。为了进行真正的升级，而不是使用单独的日志进行单独的构建，构建会在主构建中使用curl并调用远程 HTTP API。这会在主构建日志中留下一个相关的促销明星。使用不同的颜色，促销信息一眼就能看出。
为了降级构建，我决定创建一个单独的“降级构建”升级步骤。然后，这将发出一个紫色星号，作为构建有缺陷的标志，从而被删除。降级是通过访问 UI 中的正确版本并按“删除版本”按钮来完成的。此步骤没有添加自动化，但通过创建单独的测试步骤，自动降级也相当容易。然而，我们还没有走到这一步。
这种方法的好处包括
我们仍然需要改进的紧急事项包括在构建的后期阶段自动进行测试管道，也是降级后降级构建的合适方法。例如，在生产中，有缺陷的构建和降级必须始终导致安装最后一个良好的构建，而事实证明这是相当难以实现的。生产数据中心很少被允许从 CI 系统所在的开发 DC 访问到这个级别。此外，停止和启动构建管道必须自动化，否则有可能回到手动状态。
当然，本着持续改进的精神，总会有需要改进的地方。整个设置有点像 bash/perl 脚本编写的混乱，但由于它是脚本化且可重复的，因此总是可以选择一次改进一小部分。最重要的是自动化，因为它允许增量步骤，而任何手动步骤或多或少都会阻止这些步骤。
This might be a very particular case, but we decided against creating a separate job for removing the builds, for the very simple reason of keeping all the logging related to a specific build number in one single place. The setup was the following:
Promotion here means make the installation package (RPM) available to the given server, where auto-update handles the actual upgrade of the package.
We have one main build, that builds every time a new commit is available. We had some fine-tuning related to quiet times etc. but basically every new pushed set of commits resulted in a new build. The build contains all the relevant and available testing, which is far from being complete, and probably never will be.
Every hour a separate promotion step handles promotion from staging to production. This build kicks off a separate promotion which takes the latest accepted build from CI to staging. There is a 30min delay before builds were promoted CI-->staging, to prevent accidental promotions for last second commits. Delays were achieved with some bash find scripting. The order of promotions is this, to make sure a build is available in staging for (at least) one hour before going to promotion.
The actual answer:
The promotion steps were done as separate builds. In order to do a real promotion, rather than a separate build with a separate log, the build kicks off an actual promotion in the main build, using curl and calling the remote HTTP API. This leaves a relevan promotions star in the main build log. Using different colors, the promotions are visible with one look.
To demote builds I decided to create a separate "demote build" promotion step. This would then issue a purple star as a sign of the build being defective, and thus removed. The demotion is done by accessing the correct build in the UI, and pressing the "Remove build" button. No automation has been added to this step, but by creating a separate test step, it would be fairly easy to automate the demotion as well. We, however, have not gotten quite this far yet.
The benefits of this approach include
Acute things we still have to improve include automating the testing on the later stages of the build pipeline, and also a suitable way of downgrading builds after demotion. E.g. in production a defective build and a demotion must always lead to installing the last good build, which has turned out to be fairly hard to achieve. Production data centers are rarely allowed to be accessible to this level from the development DC where the CI system sits. Also stopping and starting the build pipeline must be automated, as else there is the chance of slipping back to the manual state.
Naturally, in the spirit of continuous improvement, there are always things to improve. The whole setup is something of a bash/perl scripting mess, but since it's scripted and repeatable, there is always the option of improving one small piece at a time. The most important thing is the automation, as it allows for incremental steps, which any manual steps more or less prevent.
对于任何寻找通过自定义步骤删除构建的简单方法的人：
_
要立即删除构建，只需转到促销，将选项翻转为“是”，然后单击“批准”。
For anyone looking for an easy way to delete a build with custom steps:
_
To delete a build now, just go to promotions, flip the choice to YES and click approve.