Vegetable cucumbers is licensed under Creative Commons Zero
Cucumber exiting with 0 on failure?
Now what?
The solution:
In your features/support/env.rb, if you have this:
at_exit do
cleanup_after_tests
end
Use this:
at_exit do
begin
e = $! # last exception
cleanup_after_tests
ensure
raise e if $! != e
end
end
What happened?
After upgrading Rails from 4.0 to 4.1, my Cucumber tests were failing, but also exiting with a status code of 0, which means successful execution. Because of that rake thinks everything is fine and moves on. After searching and finding a lot of what wasn’t the problem, I went spelunking.
The tunnels
minitest
After this change in minitest 5, exit is unconditionally called as part of the autorun. This turned out to be where the exit code was changing to 0. But why?
multi_test
multi_test overrides the minitest runner so that it can report the exit status from Cucumber. It uses $! which should be set to a SystemExit exception and contain the status code it needs to report, but $! was nil. Why?
features/support/env.rb
The problem was our at_exit was triggering something that cleared $!, making it so that multi_test couldn’t do its job. It turned out to be a call to FileUtils.mkdir_p that was resetting $!, but I added the ensure block so that we hopefully don’t end up accidentally triggering this again.
Comments
Great post - this solves a really important issue for us, ensuring Cuke fails will fail our CI build. We adapted your code based on HoundCI comments like so:
``` at_exit do begin e = $ERROR_INFO # last exception cleanup_after_tests ensure fail e if $ERROR_INFO != e end end ```
but get:
AgileVentures/LocalSupport/features/support/env.rb:163:in `fail’: exception object expected (TypeError)
we get the same with ‘raise’. I think that where there are no execeptions, e is nil, so perhaps the code could be:
``` at_exit do begin e = $ERROR_INFO # last exception cleanup_after_tests ensure fail e if not e.nil? and $ERROR_INFO != e end end ```
actually I’m finding that the following seems to work better for us:
``` at_exit do exit 1 if Cucumber.wants_to_quit end ```