Question

Caution

Buildbot no longer supports Python 2.7 on the Buildbot master.

Amazon Web Services Elastic Compute Cloud (“AWS EC2”)

class buildbot.worker.ec2.EC2LatentWorker

To start off, to use the AWS EC2 latent worker, you need to get an AWS developer account and sign up for EC2. Although Amazon often changes this process, these instructions should help you get started:

1. Go to http://aws.amazon.com/ and click to “Sign Up Now” for an AWS account.

2. Once you are logged into your account, you need to sign up for EC2. Instructions for how to do this have changed over time because Amazon changes their website, so the best advice is to hunt for it. After signing up for EC2, it may say it wants you to upload an x.509 cert. You will need this to create images (see below) but it is not technically necessary for the buildbot master configuration.

3. You must enter a valid credit card before you will be able to use EC2. Do that under ‘Payment Method’.

4. Make sure you’re signed up for EC2 by going to Your Account ‣ Account Activity and verifying EC2 is listed.

Now you need to create an AMI and configure the master. You may need to run through this cycle a few times to get it working, but these instructions should get you started.

Creating an AMI is out of the scope of this document. The EC2 Getting Started Guide is a good resource for this task. Here are a few additional hints.

• When an instance of the image starts, it needs to automatically start a buildbot worker that connects to your master (to create a buildbot worker, Creating a worker; to make a daemon, Launching the daemons).

• You may want to make an instance of the buildbot worker, configure it as a standard worker in the master (i.e., not as a latent worker), and test and debug it that way before you turn it into an AMI and convert to a latent worker in the master.

• In order to avoid extra costs in case of master failure, you should configure the worker of the AMI with maxretries option (see Worker Options) Also see example systemd unit file example

Now let’s assume you have an AMI that should work with the

If you want to attach existing volumes to an ec2 latent worker, use the volumes attribute. This mechanism can be valuable if you want to maintain state on a conceptual worker across multiple start/terminate sequences. volumes expects a list of (volume_id, mount_point) tuples to attempt attaching when your instance has been created.

If you want to attach new ephemeral volumes, use the the block_device_map attribute. This follows the AWS API syntax, essentially acting as a passthrough. The only distinction is that the volumes default to deleting on termination to avoid leaking volume resources when workers are terminated. See boto documentation for further details.

from buildbot.plugins import worker
c['workers'] = [
    worker.EC2LatentWorker('bot1', 'sekrit', 'm1.large',
                           ami='ami-12345',
                           keypair_name='latent_buildbot_worker',
                           security_name='latent_buildbot_worker',
                           block_device_map= [
                             {
                                "DeviceName": "/dev/xvdb",
                                "Ebs" : {
                                   "VolumeType": "io1",
                                   "Iops": 1000,
                                   "VolumeSize": 100
                                }
                             }
                           ]
                           )
]

If you are managing workers within a VPC, your worker configuration must be modified from above. You must specify the id of the subnet where you want your worker placed. You must also specify security groups created within your VPC as opposed to classic EC2 security groups. This can be done by passing the ids of the vpc security groups. Note, when using a VPC, you can not specify classic EC2 security groups (as specified by security_name).

from buildbot.plugins import worker
c['workers'] = [
    worker.EC2LatentWorker('bot1', 'sekrit', 'm1.large',
                           ami='ami-12345',
                           keypair_name='latent_buildbot_worker',
                           subnet_id='subnet-12345',
                           security_group_ids=['sg-12345','sg-67890']
                           )
]

If you would prefer to use spot instances for running your builds, you can accomplish that by passing in a True value to the spot_instance parameter to the EC2LatentWorker constructor. Additionally, you may want to specify max_spot_price and price_multiplier in order to limit your builds’ budget consumption.

from buildbot.plugins import worker
c['workers'] = [
    worker.EC2LatentWorker('bot1', 'sekrit', 'm1.large',
                           'ami-12345', region='us-west-2',
                           identifier='publickey',
                           secret_identifier='privatekey',
                           elastic_ip='208.77.188.166',
                           keypair_name='latent_buildbot_worker',
                           security_name='latent_buildbot_worker',
                           placement='b', spot_instance=True,
                           max_spot_price=0.09,
                           price_multiplier=1.15,
                           product_description='Linux/UNIX')
]

This example would attempt to create a m1.large spot instance in the us-west-2b region costing no more than $0.09/hour. The spot prices for ‘Linux/UNIX’ spot instances in that region over the last 24 hours will be averaged and multiplied by the price_multiplier parameter, then a spot request will be sent to Amazon with the above details. If the multiple exceeds the max_spot_price, the bid price will be the max_spot_price.

Either max_spot_price or price_multiplier, but not both, may be None. If price_multiplier is None, then no historical price information is retrieved; the bid price is simply the specified max_spot_price. If the max_spot_price is None, then the multiple of the historical average spot prices is used as the bid price with no limit.