How can we push test data through our platform without breaking anything?

Photo by Osman Rana on Unsplash

Those weren't the actual words, but that was the basic idea Pawel G, our lead architect, wanted to try out at Outfittery. He's a person that others look up to. Black-power-style-curly-hair, super intelligent, easy going all along, always up to trying something different. The rationale behind his idea was twofold.

On one hand, we wanted to have a more reliable environment for our regression test suite. And what could be more reliable than our live environment? On the other hand, we have been creating test data in our live environment since forever in a non reliable way. So being able to securely identify and prevent reaction to those orders would actually be good for everyone in the company.

Coming up with a strategy to cope with that was supposed to be a big deal. I mean, our backend is made of 50+ microservices, split across 4 teams. We have integrations with external systems to deal with logistics, payments, warehouse, among other topics. We have BPM, finite state machine, BI reports, machine learning algorithms, all of them feeding our decision making process with data that'd better be reliable.

Turns out it was quite simple. All it took was 3 steps.

Get all experts in one room

Well, that's intuitive, no? What else would anyone do? What we did differently was ignoring hierarchy.

If you ask around, many companies would have only team leads in the room. We don't really care about that, we just want to get the job done. And so we got in the room everyone that could contribute to the discussion, regardless of being a formal representative of a team. Pawel G, Jakub, Pawel W, Jacek and myself ended up there. Great deals of knowledge about our platform but still a small group, ideal for making this a productive session.

We booked a full day workshop with everyone and got it going at 9am.

Keep it short

This is where this story begins to get interesting.

There's nothing worse than attending a meeting where people just talk about random topics as if everyone's sitting at a beach bar, enjoying the sunset. We wanted to keep the focus, get it done and storm out of that meeting room.

So before before letting the first word out, we just distributed post-it's and gave everyone 5 minutes to quietly think about what pieces of code would need to be touched. Spring Boot apps, Grails apps, Python scripts, SQL queries, anything that needed to be touched should become one post-it on the board.

Try a hypothesis

Once everyone had explained their concerns about each of the post-it's, we didn't actually know what to do next. It was our first time doing this kind of thing, what else would you expect?

So instead of going into a rabbit hole of opinions about how to tackle it, we decided to do something different. What if we put together a hypothesis and see whether it fits our needs? If it does, then we're good to go, no need to stay longer in the room.

The hypothesis was: by having flag F in table T, when a request comes in, every part of the our platform will know this is test data and we can ignore it.

We went back to our board and started trying our hypothesis. For every post-it on the board, we'd ask ourselves: "Can we make this component work with the proposed flag?"

We didn't have answers for everything. Sometimes further investigation was needed, in which cases we'd write down plans A & B for that particular post-it.

As we progressed through the board, small adjustments were made to our initial hypothesis and after roughly 100 mins we were done. That was a record! We were expecting to spend the whole day in that meeting room; instead, we had barely gone out of our regular routine.

We still had to regroup a week later to discuss the findings of our follow up investigations, but that would most likely need to be done anyway, regardless of the method we used to plan this stunt.

We're not there yet

We haven't had time to put that plan in action, not just yet. 2018 has been a crazy year for us so far: we've migrated our backend landscape to a Kubernetes cluster (JS apps are next), our SPA's are being migrated to React, we've enabled our customers to manage when they want their boxes to be automatically delivered to their doorsteps… It has been a lot of fun!

But we're not too worried. We have a tech roadmap laid out for the foreseeable future, this is currently our #1 priority there, hopefully it'll be our Christmas gift to Pawel G.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store