False positives on Travis CI with CodeClimate / SimpleCov

We noticed in one of our projects that Travis was giving us green when the tests themselves were actually failing. After some digging we figured out that Travis CI was using the result of the CodeClimate coverage tool over the test suite. After some digging through code and Issues we found the following Issues which describe our problem:

codeclimate/ruby-test-reporter#22

colszowka/simplecov#269

The bug is in how SimpleCov, the code coverage tool used by CodeClimate, handles exit status codes. Some changes were made in the 0.8 series that force a certain exit status, clobbering any other code that would be set. The only solution at this time is to use an earlier version of SimpleCov, 0.7.1. If you’re using Bundler, add the following line to the test group:

gem "simplecov", "~> 0.7.1", :require => false

We’ll update this post when a fixed version of SimpleCov is available.

UPDATE: This issue is fixed as of version 0.9.0.

Photo of Jason Roelofs

Jason is a senior developer who has worked in the front-end, back-end, and everything in between. He has a deep understanding of all things code and can craft solutions for any problem. Jason leads development of our hosted CMS, Harmony.

Comments