Automated AWS provisioning with Fabric

One of these days I decided to give a try to an opensource project which happens to run only on Linux. This sounds like nothing special unless your development machine happens to be Windows 7 and you don’t want to bother with downloading and maintaining virtual images. True, Vagrant would be a good option but getting it (and VirtualBox) to work with my enterprise-grade proxy access is a real PITA. Add to that the less-than-maintained vagrant-proxyconf plugin and the zillions of tools every one with own config files for proxies (gradle, git, maven, you-name-it)… Not at least, a local virtual machine is local and we’re heading to a cloud-based society right? So, AWS to go!

There’s only this thing with your AWS EC2 instance: it costs real money though unless you terminate it, and if you terminate your instance your whole setup work is… gone! Ouch! Luckily one can nowadays automate the whole provisioning – it’s just an embarrassing variety of choices. Puppet? Well, it needs incoming connections on port 8140, so totally unreachable behind my above mentioned enterprise firewall. Ansible? Well, the master must be on a Linux machine, mine is Windows and installing Ansible in Cygwin although possible and fully documented looks like 2398746987324 steps…

So then: Fabric it is! As I already had Active Python installed, getting Fabric was just a matter of running

pypm install fabric

Oops, I mean a matter of setting the http(s)_proxy environment variables first, then there it was. I’m aware I could have used now python’s virtualenv to have a real clean state of library versions but blah… For the Fabric-style helloworld I’m going to use I can live safely without virtualenv. So, I created a as below and ran it with “fab hello”:

 hello:name="from the other side"
    def hello(name="world"):
        print("Hello %s!" % name)

I mean, I added Fabric’s scripts directory to PATH first, then it worked. Now off to do real stuff with AWS – for which I’ll need the Python boto library. After a pypm install boto and a bit of more googling, the following basic connectivity sample should work like a charm:

    from fabric.api import env
    import boto.ec2
    env.aws_region = 'eu-west-1'
    def init():
        if 'ec2' not in env:
            conn = boto.ec2.connect_to_region(env.aws_region)
            if conn is not None:
                env.ec2 = conn
                print "Connected to EC2 region %s" % env.aws_region
                msg = "Unable to connect to EC2 region %s"
                raise IOError(msg % env.aws_region)
        return env.ec2

Except it doesn’t, because boto’s parsing of http_proxy environment variable is too simplistic and fails on usernames like DOMAIN%5Cuser (that’s urlencoding for DOMAIN\user). The hint was the error message in the console that no proxy port was specified, uh. So I rolled up my sleeves and extended the parsing regex with \% in re.compile line 672 of my Python27\site-packages\boto\ Then rerun the command only to get:

boto.exception.NoAuthHandlerFound: No handler was ready to authenticate

Oh right! One needs the AWS credentials someplace… I’ll have them saved in the file C:/Users/Yourname/.aws/credentials like this:

    aws_access_key_id = <yourkeyid>
    aws_secret_access_key = <youraccesskey>

Just make sure you created already that IAM user (d’uh) and put it in a security group allowing it remote SSH access (and HTTP just in case you’ll need it). Although my AWS login uses MFA, the first reading on how to handle MFA with API calls got me confused, so I left it without MFA this time. Lo, and the script connected successfully. Now let’s see how it works to create and terminate an EC2 instance.

While fiddling around, at some point I noticed there’s a new boto version called boto3 so let’s get it, why staying with outdated versions:

pypm install boto3

Oops, how come this installs a very old version of boto3 0.0.30 while the latest documentation I can see is like 1.2.4? I’d say PyPM repositories aren’t that up-to-date, but no worries, one can always fall back to pip and upgrade the installed package version:

pip install boto3 --upgrade

Then of course, in order to use boto3 my above code needed an upgrade:

import boto3

def init():
    if 'ec2' in env:
        return env.ec2
    sess = boto3.Session()
    ec2 = sess.resource('ec2')
    if ec2 is None:
        print "Error: Unable to connect to EC2 region %s" % sess.region_name
        return None
    print "Connected to EC2 region %s" % sess.region_name
    env.ec2 = ec2

By the way, the region is now configured externally, in the .aws/config file, and will be picked up automatically:


(check for many tips)

Everything looks in place now. As a side note, I noticed that not all AWS EC2 instances can be run as such – many are usable only in a VPC. Connecting to a virtual private cloud is adding a minor hourly connection price and some more setup effort, so I ended up with a more expensive m3 instance – which ended up costing just as much as the cheap t2 plus its vpc cost. Either way, for my short testing it doesn’t really matter. I’ll decide the best option later as soon as I have all project’s install/upgrade commands baked into the Fabric script as well. Back to the drawing board…


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s