Walls, doors, and the microservices hangover
Everyone is falling out of love with microservices this year. I think the real lesson is about how expensive walls are, and nobody's drawing that part.
You move into a new apartment. Someone had divided up one of the bedrooms into two with a new wall. Looks like a good wall. It has the unfortunate side effect of splitting the only window in half, giving each room a narrow slice of light. The radiator gets moved to the non-sleeping room. Whoever put the wall there was smart. They had it all figured out. They just weren’t ever going to live through a winter in there.
I’ve been thinking about this hypothetical wall quite a bit this year. With the microservice backlash.
For most of the last decade, it was the adult thing to do. To break out your system into a bunch of little microservices. You’d see the diagrams in conference talks. Hundreds of boxes. Lines everywhere. It’s the subway map of a town that doesn’t yet exist. You saw these diagrams for products no one had bought yet on small startups. It’ll be useful eventually, so why not start there?
With money pouring in it was easy to not question. This year we’ve stopped growing budgets. People are line-by-line reviewing their cloud bills. And in some cases, they realize they were spending every month having components in their system talk to each other over the network when they could’ve been in the same room.
Now the tide’s turned. So I hear “Just use a monolith.” with the fervor of someone who’s recently given up sugar.
And I tend to agree. Even. But man that old slogan was cool. Look what happened.
As I strip away most of the labels from a lot of the stories people are telling, it’s... pretty standard. People divided up some work into a bunch of little pieces. The pieces talked to each other a lot. Talking takes time and money. Someone finally measures and realizes there are wrong walls. So they move some walls. That’s a refactor. It’s a good refactor. It’s not a religious experience.
What I keep circling back to is that a wall between two parts of a system is expensive in a way that’s easy to forget when you’re drawing it. On a whiteboard a wall is one line. In production it’s a small country with its own customs office.
Take a made-up checkout flow. In the many-boxes version, the order service asks the user service who this person is, asks the pricing service what the cart costs, asks the inventory service to hold the items, asks the payment service to charge the card, and then tells the notification service to send an email. Five hops, each one a function call that grew a passport.
Each hop does the same little dance. Serialize the arguments, borrow a connection, maybe pass through a sidecar proxy, wait, deserialize the answer, check whether it’s actually an error wearing a 200, decide whether to retry. Inside a single process the same call costs nanoseconds and fails only if your code is wrong. Across the network it costs a millisecond or ten, and it can fail because a pod got rescheduled, a DNS cache went stale, or a certificate expired over the weekend.
The latency is the part people underestimate, and it’s because of how tail latency stacks. Say each of those services is pretty good, and only one request in a hundred is slow. If the order service calls five of them in sequence, the chance that at least one is slow is about one minus 0.99 to the fifth, which is a little under five percent. Call ten and you’re near ten percent. Your fast services have combined into a slow product, and every single team can show you a lovely dashboard proving it isn’t them. (Calling them in parallel helps with the waiting but not the odds.)
Then there’s the part that keeps me up, which is that the database transaction is gone. In the monolith, “create the order, decrement inventory, record the payment” can be one transaction. Either all three happen or none do, and the database does the hard thinking. Spread across services, you have to do that thinking yourself. Payment succeeded but the inventory call timed out. Did inventory get decremented or not? You don’t know. So now you need idempotency keys so the retry doesn’t double count, and a saga or some compensating action to refund the card if the hold falls through, and probably an outbox table so the “order created” event goes out if and only if the order row actually got written. Each is a fine pattern, and a little machine someone has to understand at two in the morning.
There’s quieter stuff too. A contract someone has to version, and since two services never deploy at the same instant, old and new versions of it coexist, so every field change happens in two careful steps. A second set of dashboards and alerts. Local development that turns into a compose file with fourteen containers eating a laptop’s memory. And on a small team all of that lands on the same handful of people who are also trying to ship the thing customers actually asked for.
So the wall had better buy something.
I can only think of a few things it reliably buys, honestly. If two different teams own the two sides and need to release on their own schedules without negotiating, a wall is worth a lot. There’s that old observation that systems end up shaped like the organizations that build them, and I think it’s best read as a permission slip: put the walls where the people already are. If one piece has wildly different load, or needs different hardware, you can make the case, provided you can point at the numbers. If one piece going down really shouldn’t take everything else with it, isolation earns its keep. And sometimes there are security or data rules that say some code simply shouldn’t share a process with the rest.
Beyond those, I’m not sure. Maybe there are others. “It’ll be cleaner” doesn’t feel like one, though it gets said a lot. You can have clean boundaries inside one process and terrible ones across fifty.
The terrible version has a name people use with a wince, the distributed monolith. It’s what you get when the services all read and write the same database tables. You pay every networking cost from above, and you get none of the independence, because a column rename still breaks four services at once and they all have to deploy together. The tell is a release calendar where “service A must go out before service B, which must go out before C” is written down somewhere, usually in a channel nobody can find anymore.
Personally, I’d go with a middle road if you asked. Deployable units, but with honest rooms. Individual tables per module. Other modules “call” through the door (defined interface) instead of sneaking into the room (read the data) through a window. No cross-module joins (db would let you join even across modules), even though it’s tempting. The day you want to take that module out, every cross-module join is a little nail you need to pry up.
But you need to make that honesty mandatory or it will slowly decay. One deadline is all the good intentions are going to get you. Most ecosystems have some way to make the build fail if you’re importing another package’s internals (visibility in build tool, or dependency linter aware of which modules can talk to which, or just a boring test which parses imports and complains). You want to be doing that from week one so you don’t get a few hundred violations building up with a good excuse for each one.
Rooms are cheap to rearrange. When you want to merge two modules that end up being one thing, compiler will tell you what broke. When you have a module that will end up wanting to be a building of its own at some point, you can take it out without ripping up the floor because you already had an interface for it that other code could “call” through. And you already had the data be its own. There’s still work to get an in-process call replaced with network call (you get all the baggage of the checkout story), but it’s work on one seam. Not archaeology for all the code.
Otherwise, if you don’t have honest rooms, you get the worst of both worlds, an inseparable monolith and an unmergeable bunch of services.
(I know “build monolith but with good boundaries” is as exciting as “eat veggies but also get some sleep”. Didn’t say that it’s fun answer.)
The modular monolith has its own ways to go wrong, and I’d rather say them out loud. One process means one blast radius. A memory leak in the reporting module takes down checkout too. One badly behaved background job can starve the request threads. Deploys are all or nothing, so a team that wants to ship hourly is tied to whoever is nervous about Friday. The test suite grows until nobody runs it locally. There are known answers (separate worker processes from the same codebase, bulkheads around thread pools), but the problems are real, and anyone selling the monolith as a cure for complexity is doing the same thing the microservices crowd did ten years ago with a different diagram.
The other direction matters too. If someone wants to fold a service back into the main app, it seems fair to ask what that wall was supposed to buy in the first place, and whether it still does. Quite often the honest answer is that the team who owned it was reorganized away two years ago and the wall is just standing there, holding up nothing, splitting a window in half.
Merging is less glamorous than splitting, which may be why it happens so rarely. You get your transactions back and get to delete a client library and a pipeline. You also inherit whatever the service’s schema became after years of being nobody else’s business. Renovation dust, mostly.
I wonder how many of those orphan walls there are. Probably more than anyone would guess from the diagrams. Somewhere a request is crossing three of them to fetch a user’s display name, and the radiator is still in the wrong room.