The Go experiment

We’re always looking to improve speed here at Teamwork, as you might have seen in the past with “Obsessed with Speed & Still Obsessed with Speed (Part 2)”. However, we still find ourselves wanting our project management software to be even faster. A rather large bottleneck in this process is our software’s main language; ColdFusion. As it is an interpreted language, it can never really match the speed of a compiled, strictly typed language. This is where our experimentation with Go comes in.

Why go with Go?

Go (also known as Golang) is a relatively new programming language developed by Google. While still in its early stages as far as languages go, it is already quite impressive. One of the key goals of Go is to provide the efficient workflow that interpreted languages have, in a strictly typed compiled language. There are also some other aspects of Go that make it interesting, such as GoFMT which will format your code automatically to a standard that is established for the language. In addition, there is a standard documentation system which can easily be generated from a single command and is based on your existing code. I could go on about Go forever, but there’s already a ton of information out there about it already.

So fueled by our obsession with speed, we started experimenting.

Choice of libraries

Before getting started, we need to choose some base libraries to develop our API on top of. After a bit of researching, we decided to go with Martini for the handling of web requests as we liked the idea of being able to drop in middleware easily and even perhaps create our own middleware (like rate limiting). In terms of handling our database queries and results, we decided to go with gorp. What we use it for is relatively simple, but it does that quite well (turn query results into structures for JSON outputting).

Importantly, both of these core libraries have good documentation, a good community built around them and are relatively mature.

Porting API calls to Go

We decided that we’d start by porting our most heavily used API calls, which are tasks. Upon beginning development, a potential issue appeared immediately. ColdFusion being a dynamically typed language makes it a lot of hassle when trying to output types in JSON as what they actually are. Results from queries are treated as strings. Ideally we would convert them to the proper types before encoding them as JSON. So what you’ll notice from a lot of our API calls is that almost everything is a string. Attempting to recreate this in Go would be rather annoying, as throughout the code we would have to go back and forth between strings, integers and booleans.

We decided to simply start returning types properly in the JSON responses and have had very few issues with this as our handling of the responses aren’t that strict. Unfortunately some issues were discovered with the Android/iPhone application. The particular fields causing these issues just had to be left as strings for now (by changing the struct field to a string, gorp will map it as a string), even though they are integers.

So, after a week or two of working on the task API calls, I felt comfortable with my code going through some beta testing.

Internal Testing

Immediately upon getting the Go server running on our internal beta version of Teamwork, it became clear that in order to truely see the improvements and find out detailed information about each API call, we need a system to track statistics.

I developed a system by which we could see some basic statistics about each API call, how long it took, the average time taken for the last x number of requests etc. As time went on, this tracking became a lot more complex and as a result we're able to discover detailed information about our API calls that we never would have seen before.

Here we can see each API call, their total number of calls, how many of those used API level caching (memcached), how many of them were 304 responses (etags), the average response time for the last 100 calls and how much time was taken overall.

In addition to this, we can actually click into an API call and see details about the last 100 requests. These details include each SQL query that was executed in that request, how long the query took, what the query was and various other tidbits of information like who the user was and what browser they were using.

MySQL madness

After a few days of internal testing, we noticed through the statistics tracker that there were rogue queries that were on occasion being very slow. With the tracking tools available, we were able to dig into the particular queries causing the issue. After investigating further, we found that oddly enough, the issue was relatively simple, although obscure.

It boiled down to using IS NOT NULL on certain columns in certain join statements. It would cause the database system to use an intersect instead of the indexes we had set on that column, which defeated the purpose of using those indexes!

Solving this issue was rather simple, we changed the query from column IS NOT NULL to ifnull(column, -1) != -1 which in essence is achieving the exact same result. Such a small change brought our query down from ~100ms to ~15ms.

This is an example of how our new statistic tracking system in Go could help us enourmously in finding and discovering slowless.

Speed improvements

So once we had these slow queries out of the way, we could really see what kind of improvements we were getting from switching the API call to Go. When dealing with small datasets, the improvement isn’t particularly amazing. However, when dealing with bigger datasets, the improvement becomes glaringly obvious.

Here is a call to tasks.json in ColdFusion with 250 tasks in the result set:

Having users waiting nearly half a second for information to come back? Here is the same call with the same dataset, except this time Go is doing all of the work:

As you can see, we’ve gone from nearly half a second down to half a tenth of a second!

It turns out that as you keep scaling up the size of the datasets, ColdFusion gets much, much slower and Go is barely affected at all. By inspecting the query using our statistics tool, we can immediately see that 55ms of that time is from queries. That leaves only 4 milliseconds for everything else that Go is doing, including rate limiting, checking session information, logging and outputting all of this as a JSON response.

Well, optimisation in terms of the language our application is written in can’t really get much better than that! The only further optimizations we can make from this point on is our database and the queries.

Going into production

Before going into production it was important to us to be able to disable/enable the new API calls with the click of a button, should things go wrong. So we developed a system where we are able to do just that. The system was then expanded by creating custom middleware for Martini which would handle any type of error that should occur and disable the new Go calls as a result. When disabled, the API calls go back to ColdFusion.

With this safety net in place, our internal testing done and the promising results we were seeing, we decided that it was time to get this out into production. We were immediately faced with the dialemma of how we could share session states from our existing ColdFusion code to our new Go API calls. We decided upon using memcached to store the user’s session so Go can obtain the user’s session information from memcached based on their cookie before each request.

This was working wonderfully, for about a week. You see, in theory we should be able to just continuously put information into memcached without worrying about deleting it, as when memcached fills up, oldest/least used keys are deleted to make room. What we didn’t expect was that memcached would respond extremely slowly when these operations were taking place, so once we had filled up our memcached server, we started getting bombarded with emails about Go not being able to retreive session information from memcached due to an I/O timeout.

Upon discovering this we were able to resolve it by simply making sure we never fill up memcached by expiring keys and deleting them when they are no longer being used.

Other than that, there were few issues with going from beta into production. We had the odd inconsistency where the response was slightly different than the original API in certain, edge case circumstances. These were quickly discovered and ironed out.

Working with Go

After working with Go now for a few months, I’m finding that it’s quite a nice language to work with. The environment I use mostly for development is Sublime Text 2 with the GoSublime package. Having a consistent code format and stricter than average rules helps guide you into writing code that is easier for everyone to read and work with, without being heavily inconvenienced. Development was fast and painless thanks to the speedy Go compiler. In addition to this, deployment was very easy as it compiles easily on any platform and automatically gathers library dependencies based on your code with a simple go get command.

The resources that Go uses are also wonderfully low compared to the overhead that we have using ColdFusion. While our Go server is obviously a lot smaller and less complex than the ColdFusion application, the maximum amount of memory we’ve seen it using is 80 MB compared to the immense ~8 GB that our ColdFusion application uses. Asides from this, it’s nice not having to rely on Java and rather a natively compiled binary to run the application.

Conclusion

Going forward, we’re going to be moving all of our project management system’s API GET calls to Go. Additionally, any future web related projects we do will also be using Go for the backend.

The speed, efficiency, workflow and deployability of Go is hard to compete with.