Chapter 1. The claim
A payout is waiting on settlement when the worker restarts. A compliance case has been waiting three days for an analyst. In many payments stacks, each of those lives in a different service, with its own database and its own idea of what happened. In ours, each is one workflow, and it continues from the last step that finished.
The microservice premium is the ongoing cost of distribution: separate deploys, calls across a network, partial failure, and state spread across several databases. Distribution is what that money buys. It does not, by itself, give you a process that survives a crash, retries a partner, and waits days for a person. Those come from a durable workflow. We kept the workflow and declined the boundary.
Chapter 2. The pressure
A payment on our platform is not a request. The customer submits it and the money is reserved. Someone in their organisation may need to approve it. We screen the person or company being paid, and if that raises a question, the payment waits for compliance. Then it waits for settlement. Only then is it finished. Lose track of any one of those waits and the customer has paid for a transfer that never left.
Opening an account has the same shape, and so does clearing a compliance case. The work outlives the request that started it. That creates the real pressure: when a decision arrives hours or days later, it has to join the process that is already running. It cannot start a second one.
The design that suggests itself is a service for each stage, with a queue between each pair. Beside the pipeline sits a saga: a further service whose job is to undo earlier stages when a later one fails. Each service deploys on its own and keeps its own status.
That last part is where the cost hides. Ask "where is this payment?" and the answer is spread across five databases, and none of them is the whole truth.
Count what that design asks you to run: five services to deploy, four handoffs that can each fail, and a sixth piece of software whose only job is to undo the work when one does.
Chapter 3. Two decisions, bundled as one
The pipeline answers two different questions at once.
How does a payment remember where it is?
How should our teams and deploys be split?
Most designs answer both with the same tool: services. That makes them feel like one question. They are not.
Here is what we actually needed, and whether a separate service gives it to us.
Services are good at the second question. They let teams ship alone, scale parts separately and contain failures. Those are real benefits, but we did not need them.
What we had were waits. A check that takes a second and an approval that takes a day are both just waits. A wait does not need its own service, its own database or its own on-call rotation. Split at every wait, and a small team ends up running a platform built for a large one.
Chapter 4. The shape we kept
Every row in that table has the same answer: a durable workflow.
A request starts a workflow. The workflow is the whole sequence, including the waits. Steps that touch the outside world, like reserving the money, checking with a partner, notifying the customer or writing the books, are retried when they fail.
A wait is a place where the process sleeps until a person or a partner answers. It is not a row in a table that some other service has to keep checking.
Customers and operators each have their own screens, but neither is a separate process. When someone approves, rejects, or settles a payment, that decision goes to the workflow already holding it. The engine keeps the decision until the workflow reaches the matching wait, then hands it over.
The customer starts the process. Calls go out to partners. Decisions come in from operations. Approval, compliance and settlement are waits inside the workflow, so there is no queue between them.
When we need more capacity, we run more workers. Scaling the workers does not mean splitting the process. Onboarding, payments and reconciliation live in one codebase and ship together.
Chapter 5. A payment, without the machinery
A customer submits a payment. The workflow reserves the funds. If the reservation fails, the process stops. No money has moved, so there is nothing to undo.
If the payment needs a second pair of eyes, the workflow marks it as waiting and sleeps. Approval moves it forward. Rejection releases the reservation. If nobody answers before the deadline, that counts as a rejection. It is the same payment the whole time.
Next, the workflow checks the recipient with a partner. If the call times out, that step simply runs again. The failure is not handed to another service. If the check raises a concern, the workflow sleeps for compliance, the same way it slept for approval. Rejection releases the money. Approval sends the payment on to settlement, just as a clean check would have.
Notice what is missing. There is no separate saga service. Releasing a reservation is another step in the same workflow that reserved the money.
Settlement is the one step we do not undo on our own. If it is declined, we release the reservation. If it is confirmed, we record the payment. That confirmation is a decision sent to the workflow, the same way an approval is. If the recording fails, the money may already have left. Reversing it automatically would move the money a second time, on top of a first movement we can no longer be sure of. So the workflow stops in a failed state a person can see and resolve.
Opening an account follows the same pattern with a different wait. The request starts one process. A decision that arrives later joins it. The payment is not a special case.
Chapter 6. What does the work
Temporal is the engine these processes run on. Four of its properties do the work in the payment above. The chapter uses plain words: process, step, message. Temporal's own names are in the note at the end.
The process survives restarts. After a crash or a deploy, it continues from the last step that finished. The step that was in progress runs again. The payment keeps one identity the whole time. We never rebuild "the same payment" from logs.
Steps that touch the outside world are retried. A blip from a partner fails that one step, and the step runs again. It does not become a message sitting on a queue, waiting for someone to work out what it meant.
Because a step can run more than once, a debit, a refund, and a settlement each carry an idempotency key. If that step already committed, a retry does not move the money again.
A message is how a person rejoins a waiting process. Approval, review and settlement are decisions the workflow already knows how to receive. Operations sends the decision to the process itself. It does not update a status somewhere else and hope the right service notices.
The history is the record of the process. It says which step finished and which wait is open. The status a customer sees is written onto the payment by those steps. Operators act on the process itself. The status is a display of the process, not a second process.
Temporal's names. The process is a workflow: a function the engine runs to completion and can resume from its history. A step that touches the outside world is an activity, which the engine retries. A message from outside is a signal, which the workflow can wait for. The history is the workflow's event history.
Chapter 7. The bill we didn't get
Apply the Chapter 2 pipeline to the whole business and you get a service for every long-running process: onboarding, screening, pay-ins, payouts, settlement and reconciliation. Each one is a deployable, a build pipeline, a contract with its neighbours, a dashboard, and someone who gets woken up when it stops. Then there is the saga that undoes earlier steps when later ones fail. It is more software, and it is the piece most often left off the estimate.
We do pay for the workflow engine. Here is how it nets out.
The hosting bill is the smaller saving. The bigger one is time. Building that split, and keeping the undo logic correct across it, would have taken roughly N engineer-months, and very little of that work is about payments. We are a small team, so those months would have come straight out of the product.
The engine has costs of its own, which the next chapter covers.
Chapter 8. What we gave up
Deploys are shared. We cannot change how a payment settles without shipping the same codebase that opens accounts. A bug in a shared step affects every process that uses it. Our only team boundary is who owns which code, and that is a weaker wall than a network.
The engine is on the path of every payment. If we cannot reach it, a new payment does not start, and a waiting payment cannot receive a decision. Keeping the engine reachable and the workers healthy is the operational cost we accepted. Removing the queues removed one thing that could fail. The engine is another.
Changing a flow takes care. A payment that has been waiting for three days will resume by replaying the steps it already finished. If we change the flow in the meantime, the new code has to agree with what that payment already did. Workflow code also has rules that ordinary code does not: it must make the same decisions every time it replays. The team had to learn those rules. Separate services would have needed similar care, through versioned contracts between them. We traded one kind of care for another.
Separate screens, not split steps. The customer and operator interfaces are still deployed separately from the workers. We are not claiming one deployable for the whole company. We are claiming that the steps of a business process are not service boundaries.
When we would split. If a team needs to ship on its own schedule, if one process needs to scale very differently from the rest, or if a failure in one area must never touch another, that is a real reason for a service. None of these is true for us yet. When one is, we will draw the boundary there, and not at a wait.
Chapter 9. When the premium is worth paying
We would move a process into its own service for one of two reasons. The first is a team that needs to ship it on its own schedule. The second is a step with a scaling or compliance constraint that the rest of the codebase must not share: a partner who demands its own audit boundary, say, or a ledger that must never ship in the same release as the public API. Neither has happened yet.
A call to a partner is not one of those reasons. A wait for a person is not one of them. A diagram that looks more impressive with more boxes is not one of them. Those three are what turn Chapter 2's pipeline into Chapter 7's bill.
Monolith First argues that a split should be earned by real pressure, once the domain is understood. A durable workflow let us follow that advice without giving up what teams often hope a split will give them: a process that survives crashes, retries its partners, and waits as long as it has to.
What we needed was a payment that could wait days for a person and still be the same payment when the answer came. A durable workflow is what holds that wait. A service boundary was the expensive way to host it.
Comments
No comments yet.