作为构建的一部分重新配置并重新启动 Hudson/Jenkins 从属设备
我有一个 Jenkins (Hudson) 服务器设置,可以在各种从机上运行测试。我想做的是重新配置从站(使用远程 API),重新启动从站以使更改生效,然后继续其余的测试。到目前为止,我遇到了两个障碍:
- 一旦 Jenkins 作业开始在从属设备上运行,从属设备就不能宕机或断开与服务器的网络连接,否则 Jenkins 会立即无法通过测试。通常,我会说这是完全可取的行为。但在这种情况下,我希望 Jenkins 接受中断,直到从设备重新上线并且 Jenkins 可以重新连接到它 - 或者从设备重新连接到 Jenkins。
- 在附加到从属设备的作业中,我需要在 Jenkins 主设备上运行一些构建任务,而不是在从设备上运行。
这可能吗?到目前为止,我还没有找到使用 Jenkins 或其任何插件来执行此操作的方法。
编辑 - 进一步说明 我真的非常喜欢 Jenkins 从架构。与现有的插件相结合,它可以非常轻松地将作业分配给从站、运行并返回结果。并且选择任何匹配的从属设备的能力允许自动分配作业/测试。
在我们的情况下,我们使用虚拟化 (VMware) 从机。编写一个脚本非常容易,该脚本将使 Jenkins 在需要在从属设备上运行时使用 VMware PowerCLI 启动虚拟机,然后将作业发送给它并拉回结果。一切都好。
例外 每个测试的部分设置是以某种方式稍微重新配置虚拟机。禁用 UAC、以不同用户身份登录、安装不同驱动程序等 - 每项更改都要求在更改生效之前重新启动测试 VM/从属设备。尽管我可以编写从站按需脚本(启动方法=通过在主站上执行命令来启动从站)来处理此重新配置和重新启动,但它必须在作业运行之前完成。这就是问题发生的地方 - 我无法那么早配置从站,因为配置更改的类型取决于正在运行的作业,而该作业仅在从站启动后发生。
可能的解决方案
1) 在单个VM上使用多个从属实例。这是行不通的——一些配置是相互排斥的,但詹金斯不知道这一点。因此,它会尝试为一项作业启动一个从属配置,为另一项作业启动另一个从属配置 - 并且两个从属配置将位于同一虚拟机上。作业上的锁定并不能阻止这种情况,因为从属启动不是作业的一部分。
2)(最佳)一个构建步骤,允许作业知道它的从属连接可能会被中断。构建步骤可能必须包含一些选项,以便 Jenkins 知道如何重新连接从属设备(从属设备是否会自动重新连接、Jenkins 是否必须运行脚本、简单的 SSH 就足够了)。构建步骤将处理从属设备的断开连接,忽略通常导致作业失败的断开连接,然后执行重新连接。一旦从属设备恢复并运行,就可以进行下一个构建步骤。如果从站在一定时间内无法重新连接,则可能会超时导致作业失败。
** 当前解决方案 ** - 达不到最佳
目前,我无法使用Jenkins的slave功能。相反,我使用一系列构建步骤(在主服务器上运行),使用 Windows 和 PowerShell 脚本来启动虚拟机、进行配置并重新启动它。该虚拟机上运行着一个 SSH 服务器,我用它来将测试文件上传到测试虚拟机,然后远程执行它们。然后将结果下载回 Jenkins 以供作业处理。该解决方案很实用,但比典型的 Jenkins 从方法要多做很多工作。此外,这些脚本是针对单个虚拟机的;我不能轻易使用一群奴隶。
I have a Jenkins (Hudson) server setup that runs tests on a variety of slave machines. What I want to do is reconfigure the slave (using remote APIs), reboot the slave so that he changes take effect, then continue with the rest of the test. There are two hurdles that I've encountered so far:
- Once a Jenkins job begins to run on the slave, the slave cannot go down or break the network connection to the server otherwise Jenkins immediately fails the test. Normally, I would say this is completely desirable behavior. But in this case, I would like for Jenkins to accept the disruption until the slave comes back online and Jenkins can reconnect to it - or the slave reconnects to Jenkins.
- In a job that has been attached to the slave, I need to run some build tasks on the Jenkins master - not on the slave.
Is this possible? So far, I haven't found a way to do this using Jenkins or any of its plugins.
EDIT - Further Explanation
I really, really like the Jenkins slave architecture. Combined with the plugins already available, it makes it very easy to get jobs to a slave, run, and the results pulled back. And the ability to pick any matching slave allows for automatic job/test distribution.
In our situation, we use virtualized (VMware) slave machines. It was easy enough to write a script that would cause Jenkins to use VMware PowerCLI to start the VM up when it needed to run on a slave, then ship the job to it and pull the results back. All good.
EXCEPT Part of the setup of each test is to slightly reconfigure the virtual machine in some fashion. Disable UAC, logon as a different user, have a different driver installed, etc - each of these changes requires that the test VM/slave be rebooted before the changes take affect. Although I can write slave on-demand scripts (Launch Method=Launch slave via execution of command on the master) that handle this reconfig and restart, it has to be done BEFORE the job is run. That's where the problem occurs - I cannot configure the slave that early because the type of configuration changes are dependent on the job being run, which occurs only after the slave is started.
Possible Solutions
1) Use multiple slave instances on a single VM. This wouldn't work - several of the configurations are mutually exclusive, but Jenkins doesn't know that. So it would try to start one slave configuration for one job, another slave for a different job - and both slaves would be on the same VM. Locks on the jobs don't prevent this since slave starting isn't part of the job.
2) (Optimal) A build step that allows a job to know that it's slave connection MIGHT be disrupted. The build step may have to include some options so that Jenkins knows how to reconnect the slave (will the slave reconnect automatically, will Jenkins have to run a script, will simple SSH suffice). The build step would handle the disconnect of the slave, ignore the usually job-failing disconnect, then perform the reconnect. Once the slave is back up and running, the next build step can occur. Perhaps a timeout to fail the job if the slave isn't reconnectable in a certain amount of time.
** Current Solution ** - less than optimal
Right now, I can't use the slave function of Jenkins. Instead, I use a series of build steps - run on the master - that use Windows and PowerShell scripts to power on the VM, make the configurations, and restart it. The VM has a SSH server running on it and I use that to upload test files to the test VM, then remote execute them. Then download the results back to Jenkins for handling by the job. This solution is functional - but a lot more work than the typical Jenkins slave approach. Also, the scripts are targeted towards a single VM; I can't easily use a pool of slaves.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(2)
不确定这是否适合您,但您可以尝试让 Jenkins 代理节点以编程方式告诉主节点它已离线。
我遇到过这样的情况,我需要创建一个 Jenkins 作业来执行这些步骤(所有这些步骤都在主节点上运行):
,启动仅限于在代理节点虚拟机上运行的单独作业我使用curl POST请求执行代理断开连接步骤,但可能有一种更简洁的方法来执行此操作:
然后,当我启动代理节点时,代理启动并自动连接,并且主节点注意到代理重新联机(并且将然后向其发送作业)。
我还可以使用此命令打开和关闭节点的可用性(使用“toggleOffline”而不是“doDisconnect”):(
再次运行相同的命令将使节点状态恢复正常。)
上述内容可能不适用于您,因为听起来您想从代理节点上运行的一项詹金斯作业中完成所有操作。我不确定如果代理节点在运行作业过程中断开连接或将自身标记为离线,会发生什么。 :)
不过,您可能会在这个远程访问 API 文档 稍微看看这种方法还有什么可能。
Not sure if this will work for you, but you might try making the Jenkins agent node programmatically tell the master node that it's offline.
I had a situation where I needed to make a Jenkins job that performs these steps (all while running on the master node):
I perform the agent disconnect step with a curl POST request, but there might be a cleaner way to do it:
Then when I boot the agent node, the agent launches and automatically connects, and the master notices the agent is back online (and will then send it jobs).
I was also able to toggle a node's availability on and off with this command (using 'toggleOffline' instead of 'doDisconnect'):
(Running the same command again puts the node status back to normal.)
The above may not apply to you since it sounds like you want to do everything from one jenkins job running on the agent node. And I'm not sure what happens if an agent node disconnects or marks itself offline in the middle of running a job. :)
Still, you might poke around in this Remote Access API doc a bit to see what else is possible with this kind of approach.
很容易。您创建一个在 Master 上运行的主作业,从主作业中您将客户端作业称为构建步骤(这是一种新的构建步骤,我喜欢它)。您需要检查主作业是否应等待客户端作业完成。然后,您可以运行脚本来重新配置客户端并在客户端上运行第二个测试。
更好的策略是在从机上运行两个节点。您需要在 Jenkins 中配置两个节点。我在 UNIX 从机上成功地使用了该策略。原因是我需要设置不同的环境变量,并且我不想将其推入工作中。我使用 ssh 客户端,所以我不知道是否可以使用不同的客户端类型。您可能能够同时运行这两个测试,或者链接作业或使用上面提到的主策略。
Very easy. You create a Master job that runs on the Master, from the master job you call the client job as a build step (it's a new kind of build step and I love it). You need to check that the master job should wait for the client job to finish. Then you can run your script to reconfigure your client and run the second test on the client.
An even better strategy is to have two nodes running on your slave machines. You need to configure two nodes in Jenkins. I used that strategy successfully with a unix slave. The reason was that I needed different environment variables to be set up and I didn't wanted to push that into the jobs. I used ssh clients, so I don't know if it is possible with different client types. Than you might be able to run both tests at the same time or you chain the jobs or use the master strategy mentioned above.