Collective Idea

Collective Idea Logo

Steve Richert

Test Your API with Cucumber and json_spec

By Steve Richert on July 12, 2011 in api, cucumber, json, and json_spec

At Collective Idea, we do a lot of work with RESTful JSON APIs. They can be a joy to build but a pain to test. We’re currently working on a project that’s all API all the time, so we developed some reusable Cucumber steps for testing. Now, we’ve abstracted all that goodness out into its own gem… json_spec.

The Gem

The json_spec gem is a collection of RSpec matchers and a collection of Cucumber steps to wrap them up. In an API-centric project a Cucumber scenario may look like this:

Scenario: User list
  Given I post to "/users.json" with:
    """
    {
      "first_name": "Steve",
      "last_name": "Richert"
    }
    """
  And I keep the JSON response at "id" as "USER_ID"
  When I get "/users.json"
  Then the JSON response should have 1 user
  And the JSON response at "0" should be:
    """
    {
      "id": %{USER_ID},
      "first_name": "Steve",
      "last_name": "Richert"
    }
    """

Besides the “I post” and “I get” steps, the steps above are provided by json_spec.

The Steps

There are only a handful of json_spec steps but they’re versatile, taking any of the following shapes:

the JSON should have "path"
I keep the JSON as "name"
I keep the JSON at "path" as "name"
the JSON should be:
the JSON should be {"key":"value"}
the JSON should be ["entry"]
the JSON should be %{name}
the JSON at "path" should be:
the JSON at "path" should be {"key":"value"}
the JSON at "path" should be ["entry"]
the JSON at "path" should be %{name}
the JSON at "path" should be "string"
the JSON at "path" should be 10
the JSON at "path" should be 10.0
the JSON at "path" should be true
the JSON at "path" should be false
the JSON at "path" should be null
the JSON should be a hash
the JSON at "path" should be an array
the JSON should have 1 entry
the JSON at "path" should have 2 entries
the JSON should have 3 keys
the JSON at "path" should have 4 whatevers

On top of these variations, each “JSON” can be written as “JSON response” and can be lowercase if you prefer. Each “should” can be “should not” as well.

Usage

In order to use json_spec in your Cucumber suite, in env.rb you must:

require "json_spec/cucumber"

Lastly you’ll need to define a last_json method so that json_spec knows what to evaluate. If you’re using Capybara to access your API, this may look like:

def last_json
  page.source
end

JSON Paths

You’ll see references to “path” in the steps above. These paths are slash-separated strings and they refer to specific locations in a JSON document. For instance, given a list of users:

[
  {
    "id": 1,
    "first_name": "Steve",
    "last_name": "Richert"
  }
]

You could access the first name of the first user at the path:

0/first_name

And More…

The json_spec gem is young and features are still being added quickly. One of these features (used above) is the ability to store snippets of JSON for use in later steps.

Keep an eye on the json_spec repository to stay up on the latest and greatest developments. The README also has more information on the underlying RSpec matchers and how can they be used directly.

This approach has worked well for us but we’re very open to suggestions and contributions to improve json_spec. Check out the issue tracker to help. We hope you’ll get as much use out of json_spec as we do.

By Steve Richert on July 12, 2011 in api, cucumber, json, and json_spec

23 Comments

  1. Clemens Kofler

    Clemens Kofler July 12, 2011 http://www.railway.at

    WHY would I want to use Cucumber to test this instead of writing plain old rspec controller specs? If you think about what Cucumber aims to be (although it rarely seems to succeed at it), it should first and foremost be a tool to describe the behavior of your application in your customer’s domain language. I don’t know about your customers but if my (end) customers actually know what JSON is, they usually only know that you can use it “for APIs and stuff” – so it’s not Cucumber’s domain.

    I feel that you as experienced Ruby developers (as you probably are) should be encouraging what’s right and not just go with the hype of using Cucumber for every little thing.

    Just my 2 cents, of course.

  2. Daniel Morrison

    Daniel Morrison July 12, 2011 http://collectiveidea.com

    Clemens : I’ll give you two reasons:

    1.) The application in question (as well as gems we’ve developed) is 100% API, so the “customer” is an API user.

    2.) We find Cucumber to be much easier to figure out intent later. You don’t question why you’re testing some strange edge case, you have built the reasoning into the test.

    Personally, I (and I think I can speak for the rest of the team) love writing Cucumber. We don’t think it is the right tool for every job, but it makes a lot of what we do much easier, faster, and cleaner.

  3. John

    John July 12, 2011 http://www.thinknear.com

    I’ve done something similar with cucumber to test API’s before. Nice to see that something’s been factored out.

    To date I had done somewhat rigid blanket tests, like:
    Then the response should contain Foo
    and the I test the contents of Foo int he step_definition

  4. Jeroen van Dijk

    Jeroen van Dijk July 13, 2011 http://www.zwapp.com

    Really good to see this! I have created similar steps for my own projects, but never found the time to extract it to a gem. I also think you found a nice solution for things like id’s. How do you treat timestamps? I use timecop and steps like:

    And the current timestamp is "1296039053"
    

    And what about response codes? I have a step like:

    Then I should get a response '200' with the following json:
      """
      {}
      """
    

    Clemens: When working on the API for http://www.zwapp.com it is really helpful for our IPhone developer to exactly see what he can expect of the API. Since we are a young project we cannot afford the time for the duplication in the form of documentation and an other form of tests. To me testing API’s is a lot different than testing a webpage. Whereas Capybara etc. code might not become ugly for webpages fill_in 'username'. For API’s I find it very hard to read and have an overview. Cucumber makes it feel as if you are automating curl. I often find myself using the features as documentation for myself.

    On the other hand I would love the be convinced to do it differently by someone having a good example using something else than Cucumber.

    I’ll keep an eye on your repo and see if we can migrate to it and extend it.

  5. Steve Richert

    Steve Richert July 13, 2011 http://collectiveidea.com

    Jeroen: Thanks! As far as timestamps, json_spec allows you to ignore certain keys when comparing JSON hashes. By default, it ignores id, created_at and updated_at. README for more info. But in the few cases where we need to assert timestamp values, we do use Timecop.

    We also assert status codes like you mention but in separate steps from our JSON assertions. So although the status code step is useful, it’s unrelated to JSON so it falls outside the scope of json_spec.

  6. Michael Guterl

    Michael Guterl July 16, 2011 http://diminishing.org

    The fact that ruby hashes are embedded in the scenarios indicates a problem to me. 

    You’re also building variable assignment step definitions, "And I keep the JSON response at “id” as "USER_ID"", makes me cringe.  Why not just write something like this?

    I can’t help but feel like projects like this are the reason Cucumber gets a bad reputation.

    Daniel: Seeing that the “customer” is an API user, is it not conceivable that they would understand code scenarios?

  7. Daniel Morrison

    Daniel Morrison July 16, 2011 http://collectiveidea.com

    Michael: Our choice to use cucumber here wasn’t because of the customer, it is because we find Cucumber quicker to write and easier to understand, especially when you have to come back to a project or chunk of code after a period of time.

    It isn’t for everyone, and certainly this gem isn’t for every project, but we’ve found it helpful.

    Definitely check out the rspec matchers, as that’s what everything is built on. You can use them without cucumber at all!

  8. Michael Guterl

    Michael Guterl July 16, 2011 http://diminishing.org

    Daniel: “Our choice to use cucumber here wasn’t because of the customer”, that is interesting, seeing as that is your first reason in response to Clemens.

    Your second reason is that the reasoning is built into the tests.  I don’t see any additional context in the Cucumber scenarios versus what is in my gist.  Maybe these examples are too simple to start to see the benefits?  I’d love to see some more real world examples with this extra reasoning built into the scenarios.

    I’m also not sure how typing more words to describe the same thing is any quicker to write.

  9. Daniel Morrison

    Daniel Morrison July 16, 2011 http://collectiveidea.com

    Michael: Good catch, I wasn’t being very clear. Our “customer” in this case is indeed both an API and a client that understands APIs. That said, we always want to pick the best tool for the job. In this case, we chose Cucumber because we find it clearer and easier to maintain for all involved. Your mileage may vary. ;) 

  10. Michael Guterl

    Michael Guterl July 16, 2011 http://diminishing.org

    Daniel: I’m still trying to understand where the additional context/reasoning comes from.  I don’t see any additional context in the Cucumber scenarios versus what is in my gist.  Maybe these examples are too simple to start to see the benefits?  I’d love to see some more real world examples with this extra reasoning built into the scenarios.  Does the extra reasoning come from the feature descriptions at the top of a .feature file?

  11. Daniel Morrison

    Daniel Morrison July 16, 2011 http://collectiveidea.com

    Michael: It is a readability thing. Your example works great, but as it grows to tens or hundreds of lines (which unfortunately happens as you find complex use cases and workflows) you lose track of the “why”. 

    With Cucumber, you can more clearly see the flow because you’re reading plain English (well, not quite in our case with this gem) as opposed to raw code. 

    Many of our test suites grow huge over time, and when you come back to code even a month or two later, Cucumber is easier for us to grok; we remember what the test case was there for, maybe a quirky bug or a very specific case.

    Do you have an aversion to Cucumber in general? I find many people do, but after using it for 2+ years, I can honestly say it is a huge competitive advantage for our business and makes us work better and happier.

  12. Chris Gaffney

    Chris Gaffney July 16, 2011 http://collectiveidea.com

    Michael: One big win for Cucumber is that you’re dealing with actual JSON and posted data. In your gist you’re doing a ruby abstraction of it. It’s easy to take for granted that you give #post a hash and it gets converted to JSON.

    Ideally someone could copy the data out of the scenario and post it to the actual application with curl. It leads to much better documentation when you’re dealing with a separate team that is trying to figure out exactly what a request and response will look like.

  13. Michael Guterl

    Michael Guterl July 17, 2011 http://diminishing.org

    Daniel: When using RSpec to specify JSON API behavior, I’m able to remember the “why” because of the context surrounding it.  If the examples read exactly the same and the context describes the “why”, I’m still not sure where the benefit comes from…

    Are you wrapping these step definitions in other steps?  How are you managing the duplication between scenarios that happens when you get the type of complexity and growth you describe?  Maybe the cost of duplication outweighs the benefits of generated documentation.

    I do not have an aversion to Cucumber, I actually really like it and use it everyday.  I have not used Cucumber to drive the design of an HTTP/JSON API, I typically use a combination of RSpec integration and controller specs.
    Chris: Thanks for the insight, it does make sense that the Cucumber scenarios are operating at a higher-level.  I think I have a better understanding now, you’re using this as a communication tool for your customers.  Daniel has wavered back and forth whether it was for the customer or the developer.  I think it is very important to make the distinction between developer tests and customer tests.

  14. alovak

    alovak July 18, 2011 http://alovak.com

    Cucumber scenarios can be used as example in documentation. You can see an example at Chargify site.

    I used the same approach to develop application with XML API.
    For that I developed test_xml gem.

    It can be used with Cucumber, TestUnit or RSpec.

  15. Steve Richert

    Steve Richert July 18, 2011 http://collectiveidea.com

    alovak: You hit the nail on the head. Clear documentation is a big part of its usefulness for us.

    I hadn’t seen your test_xml gem. Well done. And strangely similar to json_spec. We prefer JSON when it’s up to us… but it’s not always up to us, so we’ll keep an eye on test_xml. Thanks!

  16. Michael Guterl

    Michael Guterl July 18, 2011 http://diminishing.org

    alovak: which parts of the chargify documentation is being generated from Cucumber?

  17. ZeMan

    ZeMan July 25, 2011

    Michale Guteri, Example:

    http://docs.chargify.com/api-products#api-usage-json-products-list

  18. Daryl

    Daryl August 25, 2011

    json_spec looks great. I’m trying to get it to work, but the step definitions don’t seem to be picked up; all the json_spec steps are “undefined step”. I made the changes to env.rb. Anything else I should be checking? I’m using 0.7.0.

  19. Daryl

    Daryl August 25, 2011

    json_spec looks great. I’m trying to get it to work, but the step definitions don’t seem to be picked up; all the json_spec steps are “undefined step”. I made the changes to env.rb. Anything else I should be checking? I’m using 0.7.0.

  20. Chuck van der Linden

    Chuck van der Linden February 15, 2012

    FYI: to define last_json if you are using something a gem like rest-client or http-party which returns a ‘response’ object then, in the step that makes the call capture the response along these lines

    @last_response = RestClient.post(url, request_body, :content_type => ‘application/json’)

    Then wherever you are maintaining world extensions (usually a file under features\support) add something like the following


    module JSONSpecInterface
    def last_json
    @last_response.body
    end
    end


    World(JSONSpecInterface)

  21. jun

    jun July 19, 2012

    I like the gem. Very helpful. 
    However, I do not like the cucumber step definition part. It does not provide sufficient debug information. When I use the statement "the JSON response at ‘xxx’ should be ‘yyyy’, the function returns me something like ’Can’t convert Hash to String’. That is all I get. Clearly, the path part works. The return value may be a hash with one value instead of a plain value. That is fine. The bad thing is that it should spit out the return value to me so that I can debug and change my expectation. 

    Meanwhile, should a utility gem include cucumber steps? I am debating on it. Maybe overkill. 

    Therefore, I switch to jsonpath. I use httparty to submit requests and use jsonpath to analyze the response. Everything is under my control. 

  22. Barrett Griffith

    Barrett Griffith October 18, 2012

    I have been looking this gem over. Good work. This greatly helps testing APIs and JSON. I am trying to understand how to use it for testing JSON files opposed to responses from requests. Can you suggest how to do this? How does the last_json that needs to be defined?

    Thank You!

  23. Hartog de Mik

    Hartog de Mik October 10, 2013 http://organisedminds.com/

    Hi folks!

    Thank you for writing json_spec! It has inspired me to write jason_spec: https://github.com/organisedminds/jason_spec, something I would not have done without your good work. 

Post a Comment

Contact Us

Find us on Google Maps
Collective Idea
44 East 8th Street, Suite 410
Holland, Michigan 49423 USA 42.790334-86.105251

Follow us on the Interwebs

We are currently available for medium and long term projects. Please get in touch if we can be of service.