Good luck Aaron
All of us at Level Up are sorry to announce that Aaron is leaving us, with today being his last day actively working for the company.
He will still be involved in a reduced capacity and will remain on the board, but will now be focusing his day-to-day working hours elsewhere. We would like to thank Aaron for the hard work and dedication he has put in for the past year, helping the company exceed our 1st year targets and he leaves us in a fantastic position moving forward.
We will update this blog over the coming weeks with more on Level Up, our first year in business and our plans for the future, but for now Aaron, be well, do good work, and keep in touch.
Allan
For more from Aaron, you can follow him on twitter @aaronbassett!
Earn some easy brownie points from your SO with Python, CRON & Pair
Pair has been cropping up a lot recently. Nearly every ‘Top X from Y Combinator’s Demo Day’ has Pair somewhere on the list. One of the cutsey features is a “Thinking of you” button. A single tap and you can let your loved one know they’re on your mind. ♥ ♥ Awwwww.
But it can work to your disadvantage. What if you don’t tap that button often enough? Instant cold shoulder. Now I’m not suggesting that you don’t think about your loved one on a regular basis, but when you do you’re not going to immediately reach for your phone, open the app and tap the button. So how can we make things easier? Lets estimate you think about your SO once an hour. Anytime I have a task I perform on a regular basis I tend to automate it with CRON. So lets do that here.
First thing I had to do was create a Pair account, and persuade my fiancé to actually pair with me (after a lot of eye-rolling she finally agreed). Then it was time to break out mitmproxy. There’s quite a big API to explore, but for this task I just focused on the “Thinking of you” requests, which is called a “nudge” by their API. But before you can send a nudge you have to authenticate.
The authenticate request returns an ‘authenticationToken’ (when successful) which we can use in subsequent requests. I’m unsure as to the life-span of this token so I request a new one before each nudge, you might want to experiment a little here.
Once we have authenticated we can now send our nudge request.
I might have got a little carried away while testing…
I’ve written a quick Python script to automate everything (it uses the excellent requests HTTP library). You can download it from Github. Once you have it setup just add python brownie_points.py > /dev/null to CRON at your preferred interval and begin racking up those brownie points.
If you’re worried about your partner smelling a rat when you suddenly begin “thinking of them” on the hour every hour, get creative. Use CI? Add the script to Jenkins so it runs when you deploy or you could call the script every-time you have a coffee (via RFID).
BTW, Level Up is taking bookings now for new projects. Got an interesting project you think we could help with? Or just need some friendly Python/Django developers? Then drop me an email or tweet @aaronbassett
import unix
Ask any Python developer what happens when you import this and most will respond with “The Zen of Python”, some might even be able to quote the whole thing. So what is the Zen of Python? It is a set of rules written by Tim Peters which every Python developer should follow.
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
I can’t read that list without nodding my head in agreement with every single point, except maybe the Dutch one. But long before the Zen of Python we had the Unix Philosophy (by Mike Gancarz). Written in 1994 the Unix Philosophy was around for ten years before PEP20 came on the scene. And even though it is written for a different type of development, much of it is still very relevant to the code we write every day.
Small is beautiful.
Make each program do one thing well.
Build a prototype as soon as possible.
Choose portability over efficiency.
Store data in flat text files.
Use software leverage to your advantage.
Use shell scripts to increase leverage and portability.
Avoid captive user interfaces.
Make every program a filter.
As you can see there are some similarities between the two. But it is the first three points which I find extremely relevant to our work.
Small is beautiful (No giggling!)
This mirrors the point that “Simple is better than complex”. As a system grows and becomes more complex we introduce more room for error. This is why we always try to follow K.I.S.S, a principle coined by the guys who brought us the SR-71 Blackbird. If it is good enough for developing something as amazing as the Blackbird, it’s good enough for our sites!
Make each program do one thing well
This is one of the reasons I like Django, the modular nature of its applications. For an example of what I mean watch James Bennett’s 2008 talk on Reusable Apps.
Build a prototype as soon as possible.
Our development methodology started agile, but is now very much lean. Both focus on delivering quickly, something which is very important to us, and which shows in our development process. If you’re interested in our setup make sure you read Raymond’s series on continuous integration.
I love the fact that I can view the Zen of Python with a few simple key strokes. It has been an easter egg in Python for a while now. Simply open a Python terminal and type ‘import this’ to see what I mean. As I believe the Unix Philosophy is also very important I wanted to ensure I could remind myself of it in much the same way. So I did what any developer would do, and wrote an app for it.
You can download the app from Level Up’s Github page. Place it somewhere on your Python path and then enter import unix.
Related fact: When we first started talking about starting our own agency I wanted to call it import this but we ditched that idea as only Python developers would get the joke, and everyone else might mistake us for an import/export business! @aaronbassett
Deployments (pt4 Continuous Integration)
Sysadmins who use Jenkins (formerly Hudson) seem to have an unnatural attachment to the old chap, he’s an anthropomorphised Continuous Integration server who does your bidding with minimum fuss and drama. The theory of continuous integration is that changes to the code base should be made early and often, so that integration and any possible issues it may cause can be dealt with incrementally rather than in a compounded block. The role of the CI server is to make this theory practical by automating the process of watching for changes, running tests and making the deployment.
Basically, Jenkins allows you to centralise the management of hosts using a master/slave relationship between the CI server and its networked nodes. Jobs can be chained so Job C will not be run unless A and B have been successfully completed first.
For example, we have a job that monitors a branch on the Git repo for changes. That code is then pulled to a separate workplace and tests are run (the changes have not been deployed to the server at this point). On success we then run the job that deploys the code, makes schema changes and restarts the application. If we have a failure at any point in this process, emails are sent to the admin and to the developer who made the offending commit.
The process described above uses nearly all the tools I have outlined so far.
- Code is pulled from Gitolite server by a user with read only privilges using public/private key authentication.
- Virtualenv lets us install packages as a non-privileged user in a sandboxed environment.
- Schema changes are made with South, which also gives us the ability to roll back database structure changes.
- Supervisord enables us to give process management control to an individual unprivileged user to restart the application server.
Because jobs can be scheduled, Jenkins can also be a nice replacement for Cron. The benefit being that you can centralise the management of Cron jobs and easily apply standard jobs (backups, log rotation, etc) to newly commissioned nodes. Another great strength of Jenkins is it’s large selection of plugins. Among the ones we use are the git plugin, python scripts plugin and perhaps most vital the green balls plugin (for some reason the default colour for success in Jenkins is blue?!).
So Jenkins is a really nice all purpose tool for running scheduled or triggered tasks of any kind over multiple servers. You also get a couple of nice bonuses, such as having an online changelog and project filebrowser and giving the ability for a non technical user to trigger build. Our project manager can do a live deployment once client approval has been given by clicking a button. These factors combined with the fact that Jenkins is easy to install and runs reliably make Jenkins a tool that has made my life a lot easier.
Initial Set Up (Deployments pt 3)
In this post I’m going to look at the tools used in the initial configuration of a project specifically with regard to Fabric and Virtualenv.
There are a number of scenarios where initiating a project may be required; starting a new project, deploying a project for the first time to a server or deploying to a local VM. In each of these cases the same tasks have to be carried out.
- Get or create the project files
- Install the project apps
- Create a database
- Configure the app server
- Configure the static media server
We use Fabric to create “fab-files” for these routine tasks, because we have standard project configurations and naming conventions these they can be run with very little input, usually the project name is enough (although settings can be specified through optional arguments). Fabric allows us to carry out these task on a remote server by negotiating the ssh connection so we can run commands and transfer files from our local machines. Cutting down on the need to login to the server and carry out repetitive tasks makes for better security and lessens the chance of error.
Fabric is also useful for automating other routine tasks. For example, since we have a distributed development tier it is useful for us to be able to copy the database and media assets to and from the staging server so we can propagate non-code changes (like the creation of pages in the CMS).
Another important factor in automating the initial set-up is being able to control and standardize the environment. Virtualenv allow us to create a fully or partially self contained Python environment with it’s own executables and libraries. This has two functions, firstly it lets us contain the application and it’s requirements within it’s own directory and secondly it allows us to install Python packages as a non privileged user. Virtualenv is used in conjunction with pip, we maintain our own package repository so we can serve our own bespoke modules along with third party ones. These dependencies are installed from the requirements list as a part of a build.
In the next part of the series I’ll be looking at the ongoing deployment of updates using Jenkins CI server.
Gitolite (Deployments pt2)
Gitolite as the name suggests is a lightweight git server which we use for our master repos. As I mentioned in the first post in this series we have a tiered development environment, these branches are centralised in the master repo (which is monitored by the CI server) so they can be pushed and pulled from the different servers as required.
Previously we used bare git repos on a server as the master but this had some drawbacks in terms of security and ease of set up (i.e someone had to ask me to set up a new master repo). Controlling levels of access to different users and individual repos was not easy.
With Gitolite the server is run as a user with no external login, individual git users authenticate using their own public/private keys. Administration is done using an admin repo with config changes being triggered by pushing to the server. The benefits of this method are obvious, config changes can be made without logging on the server and because all changes are linked to commits you have an audit trail and easy way to revert changes.
To add users you upload the public key of the new users to the config repos keydir, the file name minus the suffix .pub is the username used in the config files. For example the keyfile raymond@lvlup.us.pub would add the user raymond@lvlup.us.
Creating a repo is done by adding a new entry in the config file, each repo also has it’s own permissions that can be set on a user or group level. This allows us to easily set up different roles with in the repo. For example we may wish to give a frontend designer the ability to push and pull from staging and dev branches but not the live branch or enforce policy by giving the live and staging servers read only permission (to discourage cowboy coding). Permission can be even set to the file/directory level.
So with our main repo in place and using public/private key authentication code changes can be pushed and pulled from a central location to the different deployment targets. We have also made it possible for controlled access to be given to commiters out-with Level Up.
In part three I’m going to look at initial configuration stage of a project..
Our Deployments (pt1)
At Level Up we use a suite of tools to automate as much of our development process as possible, from setting up our development environment to deploying our apps. In this series of posts Ray is going to give an overview of these tools and our processes in general.
Toolkit
- Fabric - A tool for running remote commands on servers
- Gitolite - An easily configured git server
- Jenkins CI server - Continuous Integration server.
- South - Manages Django database migrations.
- Supervisord - Controls processes running on server
- Virtualenv - Creates a self contained Python Environment
The Aims
- Speed up deployments
- Limit direct access to servers
- Make life easier for me
Overview
We have a tiered development environment with rigid conventions for naming and project configuration. This means that simply by naming the project we can extrapolate the file paths, database names, git repository names etc… This also makes the process of setting up a project using scripts and templates a lot easier.
Development is done on the dev’s invidual virtual machines that run on their own hardware. These VMS are as much as possible a replica of the deployment target.
Next we have the staging server which enables clients to see completed work before sign off and also lets us test functionality that may require call backs etc.. Code pushed to staging should in theory be production ready or at least broken in a way that we anticipate.
Lastly we have the live server where the application is finally deployed to the delight of all. Each stage of this process is one step or in some cases no step from the developer’s point of view and the developer should not have to ssh into the server, which is a win for everyone!
The first of these tools we will be looking at is the git server Gitolite.
Reading time
We’re all very busy people, wouldn’t it be great if we could find out ahead of time how long an article is going to take to read? Aaron has ported readingtime.js to Python (you can get it from Github).
from datetime import timedelta
import math
class ReadingTime:
def __init__(self, text, wpm=180):
self.words = text.split(" ")
self.wpm = wpm
def duration(self):
minutes = math.floor(len(self.words) / self.wpm)
return timedelta(minutes=minutes)
def humanize(self):
d = self.duration()
if d < timedelta(minutes=1):
return "not even a minute"
elif d == timedelta(minutes=1):
return "about a minute"
elif d < timedelta(hours=1):
return "about %s minutes" % int(math.floor(d.seconds / 60))
elif d <= timedelta(hours=1, minutes=20):
return "about an hour"
else:
return "some time"
how_long = property(humanize)
It takes a string, splits it into words (no tokenization, just a basic whitespace split) and then calculates how long it would take to read at 180 words per minute (reading speed from Wikipedia). Using it in your own scripts is really easy.
>>> from reading_time import ReadingTime
>>> test_str = "The quick brown fox jumps over the lazy dog."
>>> ReadingTime(test_str).how_long
'not even a minute'
If you are a super quick (or super slow!) reader you can alter the wpm by passing in a second argument.
>>> ReadingTime(test_str,1).how_long
'about 9 minutes'
>>> ReadingTime(test_str,9).how_long
'about a minute'
One suggested use might be to combine it with an RSS parser, such as this one in ~6 lines of Python;
>>> import feedparser
>>> import re
>>>
>>> rss = feedparser.parse("http://www.thedrum.co.uk/rss.xml")
>>>
>>> for entry in rss["entries"]:
... title = entry["title"]
... summary = re.sub(r'<[^>]*?>', '', entry["summary_detail"]["value"])
...
... print "(%s) %s..." % (ReadingTime(summary).how_long, title[:30])
...
(not even a minute) Nokia mounts smartphone fightb...
(about 2 minutes) The Times editor says editoria...
(not even a minute) Google battling US government ...
(about 2 minutes) BBC Radio 5 Live to scrap seco...
(about a minute) Scotland’s First Minister pays...
(not even a minute) Aegis Media agrees acquisition...
Python Human time
Aaron has written a script to enable Python to output the time in a more human way. For example datetime.datetime(2011, 9, 1, 2, 9) will be output as “nearly ten past two”, datetime(2011, 9, 1, 11, 56) will return “five to twelve” and so on.
Using it is really straight forward.
On OS X
python human_time.py | say
On Ubuntu
python human_time.py | espeak
In your own Python scripts
import human_time
print human_time.human_time()
By default it uses the current date and time, but you can also pass in a datetime object for it to use.
from datetime import datetime
import human_time
human_time.human_time(datetime(2011, 9, 1, 1, 2))
The script with tests is on Github.
