Architecture Field Notes

Irreversible Decisions

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Someone asks “is this an architecture decision or just an implementation detail?” The honest answer is usually a question back: how much would it cost to undo in six months?

That is the only definition that has held up in practice. Not “is it drawn on the diagram”, not “did an architect approve it”, not “does it span services”. Cost of reversal.

The default

Sort every decision into one of two buckets before you spend time on it.

Constraints First

Thu, 20 Aug 2026 00:00:00 +0000

The situation

You get a requirements document, a ticket, or a conversation that starts with “we need a service that…”. Whatever it says, it is describing a solution someone already imagined, not the constraints that make a solution correct.

Designing against stated requirements without finding the real constraints is the most common way to build something technically excellent and operationally useless.

The default

Before drawing anything, get numbers for these. Guessed numbers with an order of magnitude are fine. No numbers is not fine.

Coupling and Cohesion in Practice

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Two things are coupled when changing one forces you to change the other. That definition is uselessly abstract until you attach it to concrete pain: a deployment that must be ordered, a schema change that breaks a team you have never met, a test suite that cannot run without six containers.

Every architecture pattern in existence — layering, microservices, events, hexagonal, whatever — is an attempt to control which things are coupled and how. Learning to name the coupling type is more useful than learning the patterns.

Trade-off Sliders

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Someone proposes a design and someone else objects. The argument goes in circles because both people are optimising for different qualities and neither has said which one.

The fix is boring and works: force the trade-off to be named out loud, in advance, with a ranking. Not “we value all of these” — a ranking.

The default

Pick the top three, in order, before evaluating options. Write them down. The rest are constraints to satisfy, not goals to maximise.

The Boring Baseline

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Every new system starts with a choice you make before you notice you are making it: how much novelty to accept. Novelty is not just new technology — it is any component the team has not operated before, including a familiar technology used in an unfamiliar way.

Novelty has a budget. Most teams spend it on the wrong things.

The default

The boring baseline, roughly, for a system with real users and a team under twenty people:

Monolith First, Split on Evidence

Thu, 20 Aug 2026 00:00:00 +0000

The situation

A new system, a small team, and a strong instinct that “we should build it as microservices so it scales.” The instinct conflates two different kinds of scale: load scale and organisational scale. Microservices help with the second. They rarely help with the first, and they actively hurt it at small sizes by adding network hops to what used to be function calls.

The default

Start with a modular monolith. One deployable, one datastore, clear internal module boundaries enforced by package structure, build rules, or a linter.

Service Boundaries That Survive Reorgs

Thu, 20 Aug 2026 00:00:00 +0000

The situation

You have decided to split. Now the question is where, and most teams answer it by looking at the domain model and putting a service around each noun. User service, Order service, Product service, Notification service.

This produces boundaries that look tidy on a diagram and generate constant cross-service chatter in production, because business operations do not respect noun boundaries.

The default

Draw the boundary where three things line up:

Synchronous vs Asynchronous Integration

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Service A needs something from service B. HTTP call, or a message on a broker? This gets argued as a philosophy question (“event-driven architecture”) when it is actually a question with a mechanical answer.

The default

Ask what the caller does while waiting.

If the caller cannot proceed without the answer and a human is waiting → synchronous. A user staring at a spinner needs a response, and hiding a request-response interaction behind a queue just means you rebuilt request-response badly, with correlation IDs and a timeout you forgot.
If the caller can proceed and the work merely needs to eventually happen → asynchronous. Sending an email, updating a search index, recalculating a recommendation, emitting an audit record.

That single question resolves the large majority of cases correctly.

API Contracts and Versioning

Thu, 20 Aug 2026 00:00:00 +0000

The situation

You own an API. Someone consumes it. You need to change it. The question is not “how do I version this” — it is “how do I change this without a synchronised deploy,” and versioning is only one of the answers.

The default

Expand, migrate, contract. Never change a contract in place.

sequenceDiagram
 participant P as Producer
 participant C as Consumers
 Note over P: 1. EXPAND
 P->>P: Add new field/endpoint.<br/>Old one still works.
 P->>C: Deploy. Nothing breaks.
 Note over C: 2. MIGRATE
 C->>C: Move to the new shape,<br/>at each consumer's own pace
 C->>P: Telemetry confirms<br/>old path unused
 Note over P: 3. CONTRACT
 P->>P: Remove the old field.<br/>Only now.

The third step is the one teams skip, which is why every mature codebase has deprecated fields from 2019 still in the response. Skipping it is survivable; skipping steps 1 and 2 is not.

Idempotency and Retries

Thu, 20 Aug 2026 00:00:00 +0000

The situation

A request times out. The client does not know whether the work happened. It retries. Now you may have two orders, two charges, or two emails — and the client still does not know.

This is not an edge case. In any distributed system, “I did not get a response” is indistinguishable from “it did not happen,” and that ambiguity is permanent. The only question is whether you designed for it.

Choosing a Datastore Without Regret

Thu, 20 Aug 2026 00:00:00 +0000

The situation

New system, and someone has to pick the database. This is among the most irreversible decisions available — data outlives code, and every consumer you did not plan for will be reading it by the time you want to change.

The default

A relational database, almost certainly Postgres. Then make every alternative argue against it.

This is not conservatism for its own sake. A modern relational database gives you, in one system with one operational model: transactions, joins, secondary indexes, JSON documents with indexing, full-text search, geospatial queries, range partitioning, logical replication, materialised views, and a constraint system that catches data corruption before it happens.

Schema Migrations Without Downtime

Thu, 20 Aug 2026 00:00:00 +0000

The situation

You need to rename a column, split a table, add a constraint, or change a type. During the deploy, old code and new code both run against the same database. Whatever you do must be correct for both, simultaneously, in both directions — because you might roll back.

The default

Expand, migrate, contract — the same shape as API contracts, applied to storage. Never change a column in place.

Consistency Models You Actually Need

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Someone says “the data must be consistent” and everyone nods. Nobody has defined consistent, and the requirement has quietly been interpreted as “every read everywhere sees every write immediately,” which is expensive, sometimes impossible, and almost never what the business needs.

The default

Strong consistency inside a transactional boundary; eventual consistency across boundaries. Then find the specific invariants that must never be violated and defend those explicitly.

Caching: The Four Questions

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Something is slow. Someone suggests a cache. It works immediately, everyone is happy, and six months later you are debugging why one customer sees another customer’s data on Tuesdays.

Caching is the highest ratio of “obvious win” to “subtle disaster” in the field. The wins are real. So are the disasters.

The default

Before adding any cache, answer these four in writing:

What invalidates it? If the answer is “a TTL,” you have accepted a window of wrongness. Name the window and confirm someone is okay with it.
What is the key, exactly? Including tenant, user, locale, permission context, and API version. Most cache-poisoning incidents are a key missing a dimension.
What happens when it is empty? Cold start, eviction, or a flush. If the origin cannot survive a 0% hit rate, the cache is not an optimisation — it is a load-bearing component with no redundancy.
What happens when it is wrong? Stale price, stale permission, stale feature flag. Rank the consequence.

Question 3 is the one that causes outages, and question 2 is the one that causes incidents you have to disclose.

Event Sourcing and CDC: When It Pays

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Someone proposes “event sourcing” and the conversation immediately conflates three separate things: storing state as a log of events, publishing events for other services to consume, and streaming database changes downstream.

They have different costs and different payoffs. Separating them is most of the work.

The three things

	Event sourcing	Event-driven integration	Change data capture
What it is	Your state is the event log; current state is a fold over it	Services publish domain events others subscribe to	Read the database’s replication log, emit row changes
Source of truth	The event log	Each service’s own store	The database
Who defines the schema	The domain	The publishing team, deliberately	The table structure, accidentally
Main benefit	Perfect audit trail, temporal queries, rebuild any projection	Decoupling, extensibility	Zero application change; get data out of a system you cannot modify
Main cost	Very high. Schema evolution on the log is forever	Moderate	Couples consumers to your physical schema
When it pays	Domains where history is the product: ledgers, trading, compliance	Most systems that have grown past two teams	Analytics pipelines, migrations, legacy integration

The default

Use event-driven integration where you have publishers and consumers on different teams. Use CDC to feed analytics. Do not use event sourcing unless history is the product.

Trunk-Based Development and Branch Reality

Thu, 20 Aug 2026 00:00:00 +0000

The situation

The branching model argument is usually framed as a workflow preference. It is not — it is an architecture decision about batch size, and batch size determines how bad your worst deploy can be.

A branch that lives for three weeks is three weeks of unintegrated, unvalidated, unreviewed-in-context change. The merge conflict is the visible cost. The invisible cost is that nobody knows whether the combined system works until the moment it all lands.

Pipeline Design: Fast Feedback, Slow Gates

Thu, 20 Aug 2026 00:00:00 +0000

The situation

The pipeline takes 45 minutes. People stop watching it. They merge and walk away. Failures are discovered hours later by whoever notices, context is gone, and the fix is a scramble.

Pipeline duration is not a convenience issue — it directly sets how large a batch people are willing to work in, which sets how risky each change is.

The default

Order stages by (probability of catching a failure) ÷ (time to run). Cheap, high-yield checks first; expensive, low-yield checks last.

Progressive Delivery

Thu, 20 Aug 2026 00:00:00 +0000

The situation

A deploy either works or it takes down production for everyone at once. That binary is a choice, not a law — and it is the choice that makes releases scary, which makes them rare, which makes them bigger, which makes them scarier.

The default

Separate deploy from release. Deploying puts code on servers. Releasing exposes behaviour to users. Once these are separate events, every release becomes a dial rather than a switch.

Build Reproducibility and Artifact Promotion

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Production is broken. You want to know what changed. The answer should be a commit SHA and a diff. Too often it is “well, the build pulled latest on Tuesday, so it might be a transitive dependency.”

Reproducibility is not purity for its own sake. It is the property that makes debugging, rollback, and incident forensics possible at all.

The default

One artifact, built once, identified by content, promoted unchanged.

Rollback Is a Feature

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Something is wrong in production. The fastest path to a working system is almost always to go back to the last thing that worked. Whether you can do that in two minutes or two hours was decided weeks earlier, by design choices nobody labelled as being about rollback.

The default

Every change ships with a known, tested way to undo it — and the undo path is exercised, not theoretical.

Kubernetes: What You Sign Up For

Thu, 20 Aug 2026 00:00:00 +0000

The situation

The decision is usually framed as “should we use Kubernetes?” and answered with either enthusiasm or cynicism. Neither is useful. The honest framing is: what problems does it solve, what does it charge, and are you big enough for the trade to be positive?

What it actually gives you

A declarative API for infrastructure. You describe desired state; a control loop reconciles. This is genuinely powerful and is the real product.
A standard vocabulary across teams and clouds. Deployment, Service, Ingress, ConfigMap mean the same thing everywhere.
An extension point. Operators and CRDs let you encode your own operational knowledge as automation. This is where large organisations get the most value.
Bin-packing and self-healing for stateless workloads.

What it charges

Cost	Reality
Conceptual surface	Pods, Services, Ingress, RBAC, network policy, CNI, CSI, admission control, PDBs, HPAs, taints, affinities. Everyone on the team needs a working model of most of it
Upgrade treadmill	Roughly quarterly minor releases with ~14 months of support. API deprecations break manifests. This is permanent work
Debugging depth	A failing request can be application, sidecar, CNI, kube-proxy, DNS, ingress controller, or cloud LB. The failure surface is deep
DNS and networking gotchas	CoreDNS behaviour, `ndots:5`, conntrack limits, and MTU issues will each cost you a day at some point
Resource management	Requests and limits are a genuine skill. Set them wrong and you get either OOMKills or a cluster at 20% utilisation costing full price
Security config	Default settings are permissive. Pod security, network policy, and RBAC are all opt-in work

The upgrade treadmill is the one that surprises people. It is not a one-time adoption cost; it is a recurring tax, forever, and it needs an owner.

Infrastructure as Code That Doesn't Drift

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Everything is in Terraform. Then someone fixes an urgent problem in the console at 2am, and now the code and reality disagree. Six months later a routine apply proposes to delete a security group that is holding production together.

Drift is not a tooling failure. It is what happens when the code is not the only way changes get made.

The default

The declared state is the only state, enforced by a mechanism, not a rule.

Environments, Config, and Secrets

Thu, 20 Aug 2026 00:00:00 +0000

The situation

The app needs a database URL, a feature flag, an API key, and a log level. Some of these vary per environment, some are sensitive, some change at runtime, and teams routinely handle all four with the same mechanism — usually environment variables in a deploy script — and then wonder why rotating a key requires a redeploy of nine services.

The default

Separate config by how it changes and who may read it:

Multi-Tenancy and Cost Boundaries

Thu, 20 Aug 2026 00:00:00 +0000

The situation

You have customers. Their data has to be separated. How separate is a decision that touches the data model, the deployment topology, the pricing, the compliance story, and the on-call load — and it is close to impossible to change later.

The three models

graph TD
 subgraph "Pooled"
 P1[App instance] --> PDB[(One DB,<br/>tenant_id column)]
 end
 subgraph "Bridged"
 B1[App instance] --> BDB1[(Schema A)]
 B1 --> BDB2[(Schema B)]
 end
 subgraph "Siloed"
 S1[App A] --> SDB1[(DB A)]
 S2[App B] --> SDB2[(DB B)]
 end

	Pooled	Bridged (schema/DB per tenant)	Siloed (stack per tenant)
Cost per tenant	Lowest	Medium	Highest
Blast radius	All tenants	All tenants (shared compute)	One tenant
Noisy neighbour	Real problem, needs quotas	Partially isolated	None
Per-tenant restore	Hard	Easy	Trivial
Onboarding a tenant	Insert a row	Run a migration	Provision a stack
Migrating 5000 tenants	One migration	5000 migrations	5000 deploys
Compliance story	“We isolate logically”	Better	“Your data is physically separate”
Deployment complexity	Low	Medium	High, needs real automation

The default

Pooled, with tenant isolation enforced at the lowest layer you can manage.

The Internal Platform as a Product

Thu, 20 Aug 2026 00:00:00 +0000

The situation

An organisation grows past the point where every team can invent its own deployment, monitoring, and infrastructure approach. Someone forms a platform team. Six months later there is a platform, a mandate to use it, and a significant number of teams quietly working around it.

The failure is almost never technical. It is that the platform was built as infrastructure rather than as a product with users who have alternatives.

SLOs, Error Budgets, and Saying No

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Leadership wants “five nines.” Engineering wants time to pay down debt. Product wants features. Everyone is arguing from intuition, and the loudest voice wins.

An SLO converts that argument into arithmetic. That is its real function — reliability improvement is a side effect.

The default

One or two SLOs per user-facing service, measured from the user’s perspective, with an error budget that has an agreed consequence.

Observability: What to Actually Wire

Thu, 20 Aug 2026 00:00:00 +0000

The situation

You have logs, metrics, and traces, a five-figure monthly bill for them, and during the last incident nobody could work out why checkout was failing for one customer segment.

Volume is not observability. The test is whether you can answer a new question about production without shipping code.

The default

Wire these four things, in this order. Each is cheap and each unlocks a class of question.

Capacity, Load Shedding, and Backpressure

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Traffic doubles. The service slows. Queues grow. Timeouts fire. Clients retry, which adds load. Latency rises further. Eventually everything times out and the service is effectively down — and when traffic returns to normal, it stays down, because the retry backlog keeps it saturated.

This is metastable failure. It is the most common way a healthy system dies, and it is entirely preventable with limits.

The default

Every queue is bounded. Every request has a deadline. Every dependency has a concurrency limit. Reject early rather than degrade for everyone.

Failure Modes: Timeouts, Breakers, Bulkheads

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Every distributed system needs to handle a dependency being slow, failing, or gone. The patterns are well known. What is less well known is that the standard configurations are usually wrong, and a misconfigured circuit breaker causes outages that the unprotected system would not have had.

Timeouts

Every network call has a timeout. There are no exceptions. A missing timeout means a thread or connection held indefinitely, and enough of those means resource exhaustion and a dead service.

Incident Response and Blameless Postmortems

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Something is broken. Several people are in a channel. Someone is running commands. Someone else is also running commands. Nobody is sure who is talking to the customer, and the timeline is being reconstructed from memory afterwards.

Incident response is a skill and a structure, and teams that have not practised it improvise badly under stress.

The default

Declare early, assign roles, restore first, investigate second.

Threat Modeling in One Hour

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Threat modelling has a reputation for being a heavyweight process with specialist tooling that happens once, produces a document, and is never read again. That reputation is earned, and it is why most teams do not do it.

The lightweight version takes an hour, catches most of what matters, and can be repeated whenever the design changes.

The default

Four questions, in order, with the people who actually built the thing:

Identity, Authentication, Authorisation

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Authentication is a solved problem you should not be solving — use an identity provider. Authorisation is not solved, is specific to your domain, and is where the real bugs live: the endpoint that checks you are logged in but not that the resource is yours.

The default

Authenticate at the edge. Authorise at the resource. Never trust the client about identity.

flowchart LR
 A[Client] -->|"credentials"| B["Identity provider<br/>(OIDC / SAML)"]
 B -->|"token"| A
 A -->|"token"| C[Gateway]
 C -->|"verify signature,<br/>expiry, audience"| C
 C -->|"verified identity<br/>in a trusted header<br/>or re-signed token"| D[Service]
 D -->|"authorise:<br/>may THIS subject do<br/>THIS action on<br/>THIS resource?"| E[(Data)]
 style C fill:#fd6,stroke:#c90,color:#000
 style D fill:#2d7,stroke:#0a5,color:#000

The gateway proves who. Only the service knows whether — because authorisation depends on the resource, and only the service sees the resource.

Secrets and Key Management

Thu, 20 Aug 2026 00:00:00 +0000

The situation

A database password, an API key for a payment provider, a signing key for tokens, an SSH key for a legacy box. Each has a different lifetime, blast radius, and rotation story, and most teams handle all of them with the same mechanism and the same (never) rotation cadence.

The default

A ladder, best to worst. Move each secret as far up as your platform allows.

Supply Chain: Dependencies, SBOM, Signing

Thu, 20 Aug 2026 00:00:00 +0000

The situation

A typical application has a handful of direct dependencies and several hundred transitive ones. Any of them can execute code at install time, at build time, and at runtime. Your CI has credentials. This is a supply chain, and it is attacked because it works.

The default

Three layers, in order of value per effort:

1. Know what you have.

Lockfiles committed, exact versions, digest-pinned base images.
Generate an SBOM at build time and store it with the artifact. When the next widely-exploited library vulnerability is announced, the question “are we affected, and where” should take minutes, not days.
An inventory of which services use which dependency, queryable.

2. Reduce what you have.

Least Privilege and Auditability

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Assume a component is compromised — a leaked credential, a malicious dependency, a vulnerable endpoint. The question that determines whether this is an incident or a catastrophe is: what can it reach?

That answer was determined by permission decisions made long before, usually under time pressure, usually by granting more than necessary because narrowing it was fiddly.

The default

Start from zero and add what is needed, with expiry. The opposite — starting broad and narrowing later — never happens, because narrowing risks breaking something and nobody is rewarded for it.

Architecture Decision Records That Get Read

Thu, 20 Aug 2026 00:00:00 +0000

The situation

Someone asks “why is it built this way?” The people who decided have left, moved teams, or genuinely do not remember. The code shows what was decided and nothing about why, and the reasons — the constraints, the alternatives, the thing that was true in 2022 and is not true now — are gone.

An ADR is the cheapest possible fix: one page, written once, at the time.

Diagrams That Age Well

Thu, 20 Aug 2026 00:00:00 +0000

The situation

There is a diagram in the wiki. It has fourteen boxes, no legend, arrows that might mean data flow or dependency or deployment, and it was last edited two years ago by someone who has left. Nobody trusts it, everybody references it, and it is subtly wrong in ways that mislead new joiners for months.

The default

Use the C4 levels — not as a formal methodology, but as a rule that each diagram has exactly one level of abstraction and says which.

Runbooks and On-Call Docs

Thu, 20 Aug 2026 00:00:00 +0000

The situation

3am. An alert fires. The person on call did not write this service, has never seen this alert, and has about ninety seconds of clear thinking before stress degrades their judgement.

The document they open determines how the next hour goes. Most such documents are written by the person who built the system, for a reader who already understands it — which is precisely the wrong audience.

Design Reviews and RFCs

Thu, 20 Aug 2026 00:00:00 +0000

The situation

A design gets built, shipped, and then somebody senior says “wait, this conflicts with what the payments team is doing.” That objection existed at design time. It did not surface because there was no forum, or because the forum was a meeting where nobody wanted to be the one slowing things down.

The purpose of a design review is not approval. It is making disagreement cheap and early.

Building a Team Architecture Memory

Thu, 20 Aug 2026 00:00:00 +0000

The situation

A team has been running a system for four years. The three people who made the foundational decisions have left. What remains is code, a wiki with a lot of stale pages, and a set of behaviours nobody can explain — a retry count that is oddly specific, a service that must be deployed before another, a table nobody will touch.

The system still works. Nobody can safely change it. This is the end state of purely tacit architecture knowledge, and it is extremely common.

Architecture Field Notes

Irreversible Decisions

The situation link

The default link

Constraints First

The situation link

The default link

Coupling and Cohesion in Practice

The situation link

Trade-off Sliders

The situation link

The default link

The Boring Baseline

The situation link

The default link

Monolith First, Split on Evidence

The situation link

The default link

Service Boundaries That Survive Reorgs

The situation link

The default link

Synchronous vs Asynchronous Integration

The situation link

The default link

API Contracts and Versioning

The situation link

The default link

Idempotency and Retries

The situation link

Choosing a Datastore Without Regret

The situation link

The default link

Schema Migrations Without Downtime

The situation link

The default link

Consistency Models You Actually Need

The situation link

The default link

Caching: The Four Questions

The situation link

The default link

Event Sourcing and CDC: When It Pays

The situation link

The three things link

The default link

Trunk-Based Development and Branch Reality

The situation link

Pipeline Design: Fast Feedback, Slow Gates

The situation link

The default link

Progressive Delivery

The situation link

The default link

Build Reproducibility and Artifact Promotion

The situation link

The default link

Rollback Is a Feature

The situation link

The default link

Kubernetes: What You Sign Up For

The situation link

What it actually gives you link

What it charges link

Infrastructure as Code That Doesn't Drift

The situation link

The default link

Environments, Config, and Secrets

The situation link

The default link

Multi-Tenancy and Cost Boundaries

The situation link

The three models link

The default link

The Internal Platform as a Product

The situation link

SLOs, Error Budgets, and Saying No

The situation link

The default link

Observability: What to Actually Wire

The situation link

The situation

The default

The situation

The default

The situation

The situation

The default

The situation

The default

The situation

The default

The situation

The default

The situation

The default

The situation

The default

The situation

The situation

The default

The situation

The default

The situation

The default

The situation

The default

The situation

The three things

The default

The situation

The situation

The default

The situation

The default

The situation

The default

The situation

The default

The situation

What it actually gives you

What it charges

The situation

The default

The situation

The default

The situation

The three models

The default

The situation

The situation

The default

The situation

The default

The situation

The default

The situation

Timeouts

The situation

The default

The situation

The default

The situation

The default

The situation

The default

The situation

The default

The situation

The default

The situation

The situation

The default

The situation

The situation

The situation