No Open Blockers

I’ve never liked mock object frameworks. Especially in Java, they were cumbersome to set up and never let me do all the things I wanted to. I always found it quicker just to roll my own mock.

Even in Python I had adopted this practice. Recently, Pulp settled on the Mock framework (http://www.voidspace.org.uk/python/mock/), so I decided to give mock frameworks another shot.

Installation

Installation is nice and simple:

$ easy_install mock

Setup

Even simpler, there’s nothing global you have to do in your unit test setup. When you’re ready to use the mock, instantiate it and go.

Usage

Creating a mock instance is as simple as importing the package and instantiating it. There’s no need to provide any sort of description of what should or should not be supported by the object. For instance:

In [1]: import mock
 
In [2]: m = mock.Mock()
 
In [3]: m.foo()
Out[3]: <Mock name='mock.foo()' id='22148496'>

That aspect alone is really cool. There’s no arcane setup I need to remember to do when writing my unit test. I want a mock, I create it, and go on with life.

Behavior

Of course, as cool as such little setup is, it also doesn’t really give us all that much on its own.

Modifications to the behavior of an individual method on the mock are done by setting attributes on the method itself. Again, there’s no setup step to let the mock know it should support a method. You just set the behavior on the method and the mock magically remembers it.

For example, to simulate a return value to the foo method above:

In [4]: m.foo.return_value = 'fus ro dah'
 
In [5]: m.foo()
Out[5]: 'fus ro dah'

All good unit tests should exercise error conditions too, which is done through the side_effect attribute:

In [7]: m.foo.side_effect = Exception()
 
In [8]: m.foo()
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/home/jdob/<ipython-input-8-518df19c1ba0> in <module>()
----> 1 m.foo()
[snip]

Assertions

Chances are, your mock is being called deep within another method that’s being tested. You’ll want to know that the method being tested is actually calling the mock and that it’s passing the correct values. Again, this information is provided as attributes on the mock’s method object.

If I want to make sure my method is actually calling the mock, I can determine how many times a particular method has been invoked:

In [6]: m.foo.call_count
Out[6]: 2

If I’m really careful in my tests, I want to make sure the proper values are being passed to the mock. This is especially important in Python where it’s all too easy to reverse the order of parameters in the invocation. This check is a bit tricker since it returns the call arguments in a tuple. The first entry in the tuple is any ordered arguments and the second is any keyword arguments.

In [12]: m.bar('wombat', 'zombie', blah='blargh')
Out[12]: <Mock name='mock.bar()' id='23023760'>
 
In [13]: m.bar.call_args[0]
Out[13]: ('wombat', 'zombie')
 
In [14]: m.bar.call_args[1]
Out[14]: {'blah': 'blargh'}

Clean Up

If you’re doing multiple runs that use the same mock, you’ll want to reset the state of the mock between runs. The thing to remember here is that it doesn’t reset the return value or side effects of any methods you’ve configured. That’s important to keep in mind if your tests individually pass but when run as a suite start to fail; chances are you have a rogue side effect on a method that hasn’t been cleaned up.

Resetting the mock is done through the reset_mock method on the mock itself:

In [15]: m.foo.call_count
Out[15]: 3
 
In [16]: m.reset_mock()
 
In [17]: m.foo.call_count
Out[17]: 0

And Beyond

There’s obviously more to the Mock framework than I’ve outlined. I’ve covered the basics and things I use on a regular basis. Their documentation is pretty solid; in particular I have the Mock class documentation (http://www.voidspace.org.uk/python/mock/mock.html) bookmarked to quickly refer to.

Go forth and test

With the announcement of RHEL’s offering on Amazon Web Services, I wanted to write up some notes from the work I’ve done with EC2 and python. Amazon provides a capable web console, but (not surprisingly) I’d rather do most of my work through a programmable API. The rest of this blog covers the steps necessary to launch an instance, along with some other random notes from my experience.

Boto

The first step is to grab the boto library. Boto is a python interface to Amazon’s web services (not just EC2 but S3 as well). Their site provides downloads, installation instructions, and source, so I won’t go into any more detail besides saying I use it.

Gather Amazon Information

There are a few things needed from your Amazon AWS account in order create and connect to instances, all of which can be retrieved from the AWS web console.

Account Access Keys

The account access keys are effectively your username/password when connecting through boto. From the AWS console, click the Account tab at the top and navigate to Security Credentials. In the middle of the page you’ll find “Access Key ID” and “Secret Access Key”. Make note of these but be sure to keep them safe; these pretty much give full access to your environment.

Key Pairs

The key pairs are used for SSH authentication when connecting to your instances. These are generated through the AWS console itself (the Account tab from the previous section will simply link you back to the AWS console). It’s pretty self-explanatory how to generate a key pair, just be sure to download and then keep the private key safe; there is no way to retrieve a private key from Amazon other than that initial download link.

Image ID

To create an instance, you have to specify which Amazon Machine Image (AMI) to base the instance on. These can be found under the AMIs section of the web console. Determine which image you want based on what it provides and make a note of the ID. It will look something like “ami-12345678″.

Existing Red Hat customers can find more information on Red Hat’s Cloud Access page.

Connect to Amazon

There are two ways of passing your AWS Key and Secret Key to boto, either through environment variables or as arguments to the connect calls. If you choose the environment variable route, they must be named:

AWS_ACCESS_KEY_ID=foo
AWS_SECRET_ACCESS_KEY=bar

Once those are set, create a connection to EC2 in python with the following snippet:

import boto
ec2conn = boto.connect_ec2()

If you choose to skip the environment variables, the keys can be passed directly to the connect call:

import boto
ec2conn = boto.connect_ec2(aws_access_key_id='foo', aws_secret_access_key='bar')

In either case, it is important to realize that these calls default to the US east EC2 region. If you want to make this explicit or, more likely, connect to one of the other two regions, you can pass the optional region argument:

region = # one of 'us-east', 'us-west', 'eu-west'
ec2conn = boto.connect_ec2(region=region)

That’s the main connection to EC2 and the one we’ll use for creating instances. There are others with different purposes, such as connecting to S3, the AWS load balancer features, and so on. They are all named “connect_”, so looking through the help for boto will give you a good idea of what’s available.

Create a new Security Group

A security group is basically Amazon’s firewall to your instances. The default security group is pretty restrictive, so we’ll create a new one that allows us access to SSH and HTTP:

name = 'SSH and HTTP Security Group'
description = 'Test security group'
 
ec2conn.create_security_group(name, description)
group = ec2conn.get_all_security_groups(groupnames=[name])[0]
 
group.authorize(ip_protocol='tcp', from_port='22', to_port='22', cidr_ip='0.0.0.0/0')
group.authorize(ip_protocol='tcp', from_port='80', to_port='80', cidr_ip='0.0.0.0/0')

Note: The create_security_group call returns a handle to the group, but I wanted to demonstrate retrieving an existing group as well.

The above should be pretty self-explanatory. The biggest thing to note is the line where the group is retrieved. Since a list is passed to groupnames we get back a list of matching groups. I can’t tell you how many times I attempted to act on the returned result without indexing a specific group inside of it. This is a common pattern all over boto, so you’d think I’d have learned after the first 30 times.

After this is complete, the web console will show a new security group with the firewall holes we created. This will come in handy when we want to SSH into our instance to, ya know, actually do stuff.

Create the Instance

We’re now ready to actually create an instance.

ami_id = 'ami-12345678'
ami = ec2conn.get_all_images([ami_id])[0]
 
ssh_key_name = # name of the key pair created above
security_groups = # name of the security group created above; must be a list
instance_size = # 'm1.large', 'm1.xlarge', etc. see amazon docs for more info
 
reservation = ami.run(key_name=ssh_key_name, security_groups=security_groups,
                      instance_type=instance_size)
print('New instance [%s]' % reservation.instances[0].public_dns_name)

The call is pretty simple at this point, we just need to pass in the data we’ve been collecting. Remember the security_groups argument must be a list. Also, keep in mind a reservation is returned from the create call, not the instance itself. The boto documentation can provide more information on the distinction.

SSH into the Instance

The above code should have output the public DNS name of the newly created instance. Once it’s finished starting (you can watch the progress in the web console or there are ways to do it in boto, I just haven’t included them here) you can SSH into it by passing the SSH key created earlier (substitute in the relevant information):

ssh -i $SSH_KEY root@$INSTANCE_DNS

Conclusion

As you’d expect, there is a lot more to boto than just creating instances, such as creating/attaching Elastic Block Storage (EBS) volumes, creating/configuring Elastic Load Balancers (ELB), and adding Autoscaling Groups to load balancers. Many of the APIs look similar to the code used in creating an instance, so it’s just a matter of figuring out what you want to do.

I’ve been getting into Django recently. I’ll go into it more in another entry, but I ran into a small issue where my database seemed to get out of sync with my model. Running syncdb didn’t throw any errors, but when I tried to access the model from the server I’d get an error about “no such column”, even though I could see it created in the generated DDL.

It took me a bit of digging (in other words, it wasn’t in the tutorial), but there’s a manage command to reset the database for a particular app. Running that and re-syncing my database got me moving again.

python manage.py reset [appname]
python manage.py syncdb

I’m not completely sure why, but I’m a bit embarrassed to admit to Planet Fedora how little my Python experience is; the majority of my experience is in Java. I was able to read and bug fix the Python code in Spacewalk, but I hadn’t really dug deep into my own project. Now that I’m not teaching any longer and have some free time (one of my main reasons for quitting), I can finally sit down and dork around with the language. After spending some time working on some basic games and a simple IRC bot, I figured I’d step back and think about what the transition from Java to Python has felt like.

Don’t Fear The Whitespace

I constantly hear people mention the indentation in Python as the first thing when talking about moving to the language. Not only is it not as jarring of an experience as people make it out to be, it’s downright awesome. I’ve always been compulsive about my code format anyway, so the biggest difference is the lack of curly braces.

Collections Are Awesome

It’s much lighter-weight to throw things into a list or map (dictionary in Python) than it is in Java. Get out of the mentality that you have to jump through import hoops and rigid notation to create, access, or return collections. In Python, they even let you do cool things like assign multiple variables as a return from a call:

exceptionType, exceptionValue, exceptionTraceback = sys.exc_info()

Looping Feels Weird At First

I got a little thrown off by this initially. Most loops read really well:

for square in openSquares:

However, when looping through a set of numbers, you need to use the range method:

for i in range(0, 10):

Looking at both of those examples brings me to my next point…

Don’t Forget The Colon

This keeps throwing me off, but after declaring a function*, loop, or if statement, don’t forget to end the line with a colon. I’m happy to be rid of curly braces, but I get over-ambitious and forget the colon too.

Don’t Over-engineer Configuration

Depending on what you’re doing, you can likely just stuff configuration values into a script and import it (not needing to compile really is liberating in this respect). That’ll also give you the use of lists and maps by default. If you’re not reading between the lines I’ll spell it out: no need for XML-based configuration, which is one of the more evil trends in Java.


There are definitely more things I could mention; don’t take this to be the only lessons I’ve learned (any other hints/tips are appreciated). But I do want to avoid a mammoth blog post that causes readers to go into a zombie-like trance, so I’ll stop it here for now. I do want to thank Devan (dgoodwin) and Jesus (zeus) for dealing with the Java-veteran-turned-Python-noob and not finding a way to crash my chat client to avoid more questions.

* I haven’t seen a solid explanation of “Call them ‘functions’ because you’ll sound like a Java guy calling them ‘methods’”, but this feels like something where using the wrong term will make me stand out as a Java developer in a Python world. So I’ve been advised to take a militant approach of “Yes, I’m a Java guy learning Python, deal with the occasional terminology missteps.”