Pop in a cassette, because we're using a VCR for integration testing

By Mo Omer

What is an integration?

Anything that facilitates connecting [sub-]system boundaries through inter-system communication.

(can be internal OR external)

Should you test integrations?

Only if...

You expect one or more particular responses to a request over some communication protocol or file format.

Surely, that's not right!?

Consider: any protocol or file format over which an application sends data and expects a response is covered.

So basically, yes, all integrations should be tested.

Ambitious statements (lies) we might tell ourselves

• "The code only executes X branch only if we get a 200 response header, and it's so simple we don't need to test it."
• "That's an internal service, and our implementation matches the docs, so we're good to go."
• "I wrote that code, it'll never change, and if it does, I'll remember to update this."

Well, what sort of failure modes are we looking at?

• "The unit tests pass, but something is wrong in production!"
• "But we mocked the tests with data that came straight from the documentation website!"
• "I pulled the mock data from a curl request, so all we have to do is figure out what changed! I'll compare with a failing request/response pair from production!"

Are unit tests under assault?

Unit testing helps!

"Does the code do what we expect it to do?"

func (s *ClientMockTestSuite) TestPlanClient_All() {
	s.serveMux.Get(bonsai.PlanAPIBasePath, func(w http.ResponseWriter, _ *http.Request) {
		respStr := `{
		    "SOME": "JSON DATA THAT WE EXPECT"
        }`
		_, err := w.Write([]byte(respStr))
	})

	expect := []bonsai.Plan{...} // SOME UNMARSHALLED STRUCT WE EXPECT
	plans, err := s.client.Plan.All(context.Background())

	s.ElementsMatch(expect, plans, "elements expected match elements in received plans")
}
	

Unit tests are a big win for testing expected functionality!

Testing

Unit tests: test your code vs. expectations

Integration tests: test your code vs. reality

Integration tests

"Does our code actually fetch, receive, and handle data from the other system as expected?"

    // Excerpt from StackOverflow Answer
    func Test_create(t *testing.T) {
        tests := []struct {
            name string // Scenario name
            args args // Scenario arguments/config
            want bool // Expected result
        }{
            ... // test cases
        }
        for _, tt := range tests {
            t.Run(tt.name, func(t *testing.T) {
                if got := create(tt.args.id); got != tt.want {
                    t.Errorf("create() = %v, want %v", got, tt.want)
                }
            })
        }
    }
    

Integration tests...

• Require real integrations(!)
• Demonstrate that the actual integration works as expected (and produces expected data structures).

Integration tests...

• Are slow.
• Are flakey.

Solution(!?): Disable the tests!

• Build tags (`+build integration`)
• Test flags (`go test -long` or `go test -short`)

Ref: Stack Overflow (>300 votes in answers)

System entropy

What happens when the system behaves differently than it once did?

• Breaking changes
• Non-breaking changes

Maybe it breaks, maybe it doesn't. Maybe you just miss things changing around the system.

How can we fix the main issues?

• Slowness
• Flakiness

Cache ALL the things!

• HTTP Request
• HTTP Response
• Even the HTTP Response handler!

Enter: Go-VCR

Inspired by the VCR gem for Ruby

"go-vcr simplifies testing by recording your HTTP interactions and replaying them in future runs in order to provide fast, deterministic and accurate testing of your code."

What the !@#$ is a "VCR?"

What the !@#$ is a "VCR?"

"VCR" - Primary [system] components

Recorder

            type Recorder struct {
                // Cassette used by the recorder
                cassette *cassette.Cassette

                // Recorder options
                options *Options

                // Other stuff
                ...
            }
        

A Recorder handles and manages recordings

Recorder

• Recording ("cassette") storage
• Playback/Record modes
• Playback hooks/callbacks

Go-VCR transparently wraps HTTP to your tests

A Recorder provides an HTTP Transport, to transparently wrap live or recorded HTTP request/responses while recording incoming and outgoing data into a cassette.

"VCR" - Primary [system] components

Cassette

"VCR" - Primary [system] components

Cassette

• A grouping of interactions between client/server
• Cassette/File name

A cassette provides HTTP replayability

A cassette stores, and thus provides, all the details associated with HTTP interactions, allowing for transparent replayability.

Note: Cassettes can hold more than one interaction!

Cassette HTTP interactions are as 1:1 as you want them to be.

Your code won't even believe it's not live HTTP.

VCR Recording modes

These are important toggles to keep in the back of your mind!

• Record-only (no playback at all).
• Replay-only (no recording at all).
• Replay-with-new-episodes (record missing interactions, but keep the old).
• Record-once (record once, playback all interactions, error if interaction is missing).
• No replay/record (pass-through only).

[optional] Add a caching mechanism for unmarshaled HTTP response results in tests.

• Read previously marshaled data from file.
• Compare to newly marshaled data (from output of HTTP transport).
• Fail if different.

Note: I used SourceGraph's Apache 2.0 licensed golden tests @ this test helper.

Demonstration: putting it all together

1. Configure the test suite to use VCR's HTTP Transport.
2. [optional] Add a caching mechanism for unmarshaled HTTP response results.
3. Run the fast tests!

Note: the first test-run will be about as slow as your previous test runs (cache is empty; no cassettes exist).

References