Example

To illustrate the power of with_mock_API, let’s try to add tests to an R package that wraps a prominent web API. The twitteR package is quite popular (13k downloads per month, at the time of writing), but like most R packages, it doesn’t have a test suite. Indeed, it is hard to test the querying of an API that requires OAuth authentication, and thus a user account to use in testing. That’s tricky to set up to run locally, and even more challenging to run on a continuous-integration service like Travis-CI. But httptest can help.

First, we need to set up some boilerplate for testthat testing, which httptest uses. Add a “tests” directory, and in that, a “testthat.R” with

library(httptest)
test_check("twitteR")

Note that it says library(httptest) instead of the library(testthat) that you would otherwise use with testthat. Next, let’s add a “testthat” directory to hold our test files. In that directory, we’ll start with a “helper.R” file for common setup code. From some experimenting, it’s clear that the package checks to see if you have an OAuth token set. We’re not actually going to hit the real Twitter API, so we don’t need a valid token—we just need a token to exist so the R code that looks for a token finds one:

use_oauth_token("foo")

Now let’s write a test. Add “test-user.R” and put a test for getting a user record, via the getUser function. According to the source code, this function should hit the “show user” Twitter API, documented here. But, we’re going to prevent that request from actually happening by wrapping our test in the with_mock_API.

context("Get a user")

with_mock_API({
    test_that("We can get a user object", {
        user <- getUser("twitterdev")
    })
})

When we run the tests, it fails with

Get a user: Error:
    GET https://api.twitter.com/1.1/users/show.json?screen_name=twitterdev
    (api.twitter.com/1.1/users/show.json-84627b.json)

The error message reveals a few things about how with_mock_API works. First, it tells us what the request method and URL was, and if there had been a request body, it would have been part of the error message as well. Second, the final part of the error message is a file name. That’s the mock file that the test context was looking for and didn’t find. If the file had existed, it would have been loaded and the code would have continued executing as if the server had returned it.

Requests are translated to mock file paths according to several rules that incorporate the request method, URL, query parameters, and body. Query parameters and request bodies are incorporated into the file path by hashing—hence the 84627b in the getUser example. If a request method other than GET is used, it will be appended to the end of the end of the file name. Mock file paths also have an extension appended because in an HTTP API, a “directory” itself is a resource. The extension allows distinguishing directories and files in the file system. That is, a mocked GET("http://example.com/api/") may read a “example.com/api.json” file, while GET("http://example.com/api/object1/") reads “example.com/api/object1.json”, and POST("api/object1/?a=1") would map to “api/object1-b64371-POST.json”. The file extension also gives information on content type. Two extensions are currently supported: .json and .R. JSON mocks can be stored in .json files, and when they are loaded by with_mock_API, relevant request metadata (headers, status code, etc.) are inferred. If your API doesn’t return JSON, or if you want to simulate requests with other behavior (201 Location response, or 400 Bad Request, for example), you can store full response objects in .R files that with_mock_API will source to load. Any request can be stored as a .R mock, but the .json mocks offer a simplified, more readable alternative.

Back to the getUser example. The error message tells us that the request it is making—GET https://api.twitter.com/1.1/users/show.json?screen_name=twitterdev—is what we should expect based on the API documentation, so that’s good. Now let’s provide a mock response. The API documentation page has an example JSON response, which looks like

{
  "id": 2244994945,
  "id_str": "2244994945",
  "name": "TwitterDev",
  "screen_name": "TwitterDev",
  "location": "Internet",
  "profile_location": null,
  "description": "Developer and Platform Relations @Twitter. We are
      developer advocates. We can't answer all your questions, but we listen
      to all of them!",
  "url": "https://t.co/66w26cua1O",
  ...
}

Let’s copy that example response to the fixture file path that the message indicated, api.twitter.com/1.1/users/show.json-84627b.json. When we run the tests again, there’s no more error. Great! This means that with_mock_API loaded our mock when it reached the GET request, and the rest of the code continued executing. getUser returns a “user” object, so let’s now assert some things about it and test some of its methods:

test_that("We can get a user object", {
    user <- getUser("twitterdev")
    expect_is(user, "user")
    expect_identical(user$name, "TwitterDev")
    expect_output(print(user), "TwitterDev")
})

We can do the same for the lookupUsers function. It should hit the users/lookup.json endpoint and the function should return a list of user objects:

test_that("lookupUsers retrieves many", {
    result <- lookupUsers(c("twitterapi", "twitter"))
    expect_is(result, "list")
    expect_true(all(vapply(result, inherits, logical(1), what="user")))
})

Drop the example response from the API documentation in the right location, and that passes as well.

We just went from zero tests to 25 percent line coverage in a few minutes, using 16 lines of code. We’ve tested a lot of the code that prepares the requests of the user API, and we’ve tested much of the code that handles the server’s response, the “user” objects that get created in R, and their methods. Our resulting test directory, containing both our test files and our API fixtures, looks like this:

tests
├── testthat
│   ├── api.twitter.com
│   │   └── 1.1
│   │       └── users
│   │           ├── lookup.json-342984.json
│   │           └── show.json-84627b.json
│   ├── helper.R
│   └── test-user.R
└── testthat.R

The full test code looks like this:

context("Get a user")

use_oauth_token("foo") # Pulled here from helper.R for clarity

with_mock_API({
    test_that("We can get a user object", {
        user <- getUser("twitterdev")
        expect_is(user, "user")
        expect_identical(user$name, "TwitterDev")
        expect_identical(name(user), "TwitterDev")
        expect_output(print(user), "TwitterDev")
    })

    test_that("lookupUsers retrieves many", {
        result <- lookupUsers(c("twitterapi", "twitter"))
        expect_is(result, "list")
        expect_true(all(vapply(result, inherits, logical(1), what="user")))
    })
})

Note that none of the test code inside the with_mock_API block looks any different from how you’d write it if you were testing against a live server using just testthat. The goal is to make your tests just as natural to write as if you were using your package normally. The with_mock_API context handles all of the HTTP mocking seamlessly.

Assert the shape of request payloads

Sometimes it is more clear what you’re testing if you focus only on the requests. One case is when the response itself isn’t that interesting or doesn’t tell you that the request did the correct thing. For example, if you’re testing a POST request that alters the state of something on the server and returns 204 No Content status on success, nothing in the response itself (which would be stored in the mock file) tells you that the request you made was shaped correctly—the response has no content. A more transparent, readable test would just assert that the POST request was made to the right URL and had the expected request body.

Both without_internet and with_mock_API allow you to make assertions about requests—method, URL, and payload—that should be made. The various expect_VERB expectation functions facilitate this testing. In this example from the crunch package, inside the with_mock_API context, there is a catalog resource containing three entities, each of which has an “archived” attribute. The is.archived method returns the value of that attribute:

test_that("is.archived", {
    expect_identical(is.archived(catalog), c(FALSE, TRUE, FALSE))
})

When we update the attributes of the catalog’s entities, we send a PATCH request, and we only want to send values that are changing. That way, we don’t unintentionally collide with any other concurrent actions happening on the server, and we can send smaller messages, which should be faster. In this example, if we were to set “archived” to TRUE for the second and third elements of the catalog, we’d only want to send a PATCH request that referenced the third element because the second one is already TRUE.

test_that("'archive' sets the archived attribute and only PATCHes changes", {
    expect_PATCH(archive(catalog[2:3]),
        'https://app.crunch.io/api/datasets/',
        '{"https://app.crunch.io/api/datasets/3/":{"archived":true}}')
})

The resulting state of the system is the same whether the smaller PATCH request is sent or whether the overly verbose one is sent. If you’ve written logic in your R code to ensure that the smaller PATCH is sent, testing the shape of the request being made is the clearest way to demonstrate and assert that the desired behavior is happening.

Another instance of when you might care more about request body shape rather than the resulting response is when there are multiple paths in your R code that should lead to the same request being made. If you can assert that all of those variations result in the same request, then when it comes to testing the response and how your code handles it, you can do that once and not have to repeat for all of the input variations. This is particularly useful in conjunction with integration tests that run against a live server because it means you can have the same test coverage with fewer integration tests.

An example of this from the crunch package is in testing a join function, which has a similar syntax to the base R merge function. merge takes “by.x” and “by.y” arguments, which point to the variables in the “x” and “y” data.frames on which to match the rows when merging. It has a shortcut for the case where the variable have the same names in both data.frame, in which case you can just specify a “by” argument. To test that all of those combinations of specifying join keys result in the same request, the test defines a payload string to reuse

testPayload <- paste0(
    '{"https://app.crunch.io/api/datasets/1/joins/95c0b45fe0af492594863f818cb913d2/":',
    '{"left_key":"https://app.crunch.io/api/datasets/1/variables/birthyr/",',
    '"right_key":"https://app.crunch.io/api/datasets/3/variables/birthyr/"}}')

and then asserts that three different ways of calling join result in the same PATCH request being made

test_that("Can specify 'by' variables several ways", {
    expect_PATCH(join(ds1, ds2, by.x=ds1$birthyr, ds2$birthyr),
        'https://app.crunch.io/api/datasets/1/joins/',
        testPayload)
    expect_PATCH(join(ds1, ds2, by.x="birthyr", by.y="birthyr"),
        'https://app.crunch.io/api/datasets/1/joins/',
        testPayload)
    expect_PATCH(join(ds1, ds2, by="birthyr"),
        'https://app.crunch.io/api/datasets/1/joins/',
        testPayload)
})

Subsequent integration tests that assert that the dataset is correctly modified on the server by join then only test with one of those ways of specifying the “by” variables. The R code that constructs the request is fully covered by these assertions.