A Tale of Two CI Tools, Differentiating Travis and CircleCI

Introduction

As we introspect on the development process that we have here at Mediocre, we try to identify current pain points and try to alleviate them. There have been several grumblings about the current process so it was a good time to compare tools and see if we couldn’t find improvement either by adjusting what we have, or using something new.

It is worth mentioning that these aren’t the only two tools that were evaluated, they are the only two that made it past preliminary evaluation (one by virtue of being our current solution, but would have passed on its own merits). The other tools investigated were Wercker, Ship.io, Codeship, drone.io, and Shippable.

Categories

The categories listed here were things we felt were important to get out of our CI tool for one reason or another. They are not listed in any type of weighted order, just as we thought about them.

SSH Connection To Build Boxes

This is a big deal for us. It helps us to easily identify why a build/test is failing when it's passing locally.

Travis used to have the ability to download vagrant files of the build box. This was amazing in that you could download the box and spin it up locally, and if you made changes you could destroy it and recreate the box. Unfortunately, it appears they no longer offer this feature which leaves a hole in the ability to debug issues They do offer a debug VM that they can make available for 24 hours. But you have to manually request it and wait for their support cycle.

Circle by default allows for SSH connections while the build is running provided you have SSH keys installed as part of your build. Additionally, the build can be re-run with ssh enabled allowing for 30 minutes of connection time after a build is complete. However, this utilizes a build slot while active preventing other builds from progressing if there’s no other capacity.

Github Integration

Since we use GitHub as our source code repository, it’s important for our CI system to have a tight integration here. Mainly we use this for the ability to see if tests have passed on a Pull request before we merge into master. We also use the authentication and authorization from GitHub to simplify our stack, managing users and roles in 2 different places for largely the same function is not cool.

In this case, both tools perform well in each category. They are almost identical in the implementation of these features.

Gitter Integration

Gitter is how we stay connected to each other and discuss development related issues. To that end, it also serves as an activity feed for git, CI, and deployments.

Travis posts to the activity feed with build statuses, including color-coding and faces for at a glance insight into the build status. This adequately covers our requirements for staying informed.

CircleCI activity messages also include the commit message from the corresponding build trigger. It uses color-coding and faces to give at a glance insight to the status of the build including a 'meh' face for fixed builds. Of course we give this the edge purely based on one of the products we produce (meh.com)

Build Artifacts

While using SSH is the be all, end all, for debugging a build, it can be a little tedious. So having the ability to access certain files output from the build is not only helpful from a debugging standpoint, in the case of logs and screenshots from Testium, but useful for the purposes of Continuous Delivery to have the software bundles attached to a build.

Travis supports automatically uploading of build artifacts, but requires an S3 bucket to do so. While configuring this feature is simple enough, takes seconds to create an S3 bucket and to configure the artifacts themselves, there is an added cost of $0.03/GB/month. Plus the overhead of having to support another system.

CircleCI supports automatic uploading of build artifacts to a built-in repository, and/or alternate locations. Easily accessible in the UI for that build via an "Artifacts" link as well as the CircleCI API. The artifacts have an indefinite retention period, and the feature is included in the base cost.

Debugging

While this is somewhat a vague category, we use it to mean the ability to discern the cause of an error based on what the tool readily makes available via the UI. Obviously the more we can figure out without special configuration, or direct intervention, the better.

TravisCI shows recent repository activity in a pane on the left side. The build page shows who authored and committed. Build output is chunked out into individual commands, except for test output, and can be downloaded in a unified log file. TravisCI only exposes what runs in the foreground on stdout/stderr. No direct support for background processes.

CircleCI provides a view of all repos with at a glance build history of each repo up to 5 builds back in a pane on the left side. This view can filter to only show the user's own activity. Build page shows who authored and committed. Build output is chunked out into individual commands but can’t be directly downloaded. CircleCI automatically makes all nohup process logs available in build artifacts, separated into stdout and stderr.

Update Cycle

While neither tool has a published build cycle, it’s important for them to stay on top of current technology and make it available or at least ensure the platform supports it. Both tools do a pretty good job of publishing updates multiple times a month. CircleCI provides a changelog page which details all updates published.

Support

When things go horribly wrong outside of our control, or we find the documentation lacking in certain areas, we need to be able to get support or guidance from the company.

Travis support supplies an email address for all requests. While their responses are thorough and helpful, it can be a little slow with typical responses happening in 2-3 days.

CircleCI also provides support via email. But they provide support through their UI and have a phone number available. Their email response time is typically a little faster than Travis at 1-2 days.

Test Metadata

As we increase our test coverage, it’s useful to have more details made available about the tests as they run and provide more data than what’s available on the console.

Travis has an open feature request for this from Sept 2011. Still not implemented as of the posting of this report.

CircleCI can parse junit type xml files and attach the data to a build. If timing information is included it will use this data when deciding which tests to parallelize.

Test Parallelization

Longer running tests, or more complex integration tests, can cause a backup in the build queue, or cause a build to take longer than is desirable. If the tests can be split across multiple machines, it reduces the load and improves our time to deliver.

Both tools provide mechanisms for breaking up tests across multiple machines. Circle provides additional functionality in that it can use timing data from the aforementioned test metadata to more effectively split up the test load.

Software Support

The CI tools that we use obviously need to be able to support our technology stack. Between dependency resolution and running tests, this is an essential feature.

Both tools support most of our technology stack with two notable exceptions. MongoDB 3 and PhantomJS 2 are not yet supported by either tool. Both provide instructions for working around the issue, and both are working towards making them available permanently as part of their build images.

Node support is provided by both through the use of NVM (Node Version Manager). However, CircleCI also provides several versions on the build VM which reduces the time taken to setup the build.

Build VM Specs

While not the most important feature, it’s useful to know what the underlying system is for stability and performance.

Travis has their VMs based on Ubuntu 12.04, linux kernel 2.6, and 3 GB of RAM.

CircleCI has their VMs based on Ubuntu 12.04, linux kernel 3.2, and 4 GB of RAM. Additionally, if your build happens to run out of memory, it will include a memory usage report in the build artifacts for post mortem analysis.

Boot Time

The time elapsed before the build process begins is a pain that must be felt regardless of the configuration of the builds themselves. Minimizing this is helpful in improving throughput.

Travis takes ~1 minute before the build process starts. This time is not included in the duration statistics and there is no feedback on progress of this in the UI.

CircleCI takes ~3 seconds before the build process starts. The UI logs the progress of the boot, and keeps this time separate from the rest of the build process.

Deployment Support

Eventually, we want to have continuous deployment on our development branch to a staging environment. As we develop this, the tools should support several technologies in this space as well as custom overrides.

Travis provides out of the box support for deployments to S3 and Elastic Beanstalk. While Elastic Beanstalk is the direction we’re planning on going, the complexity of our needs around this preclude our using this functionality. The bigger story here is the S3 support as it is simple and straightforward.

CircleCI provides out of the box support for deployments to S3 and CodeDeploy. CodeDeploy may better fit our needs further down the road, but at the time of writing this it’s not part of our deployment stack. The S3 support appears to be somewhat limited and will require more effort than Travis to ensure it is properly used.

Conclusion

While both systems meet our needs from a CI/CD perspective, there are several items that are attractive in CircleCI and, in our opinion, set it apart from Travis. Specifically, the features around debugging, performance, and it’s overall look and feel make it a desirable tool in the space.