Why Your Systems Behave StrangelySystems Thinking for Software Architects

Carl Chesser
@che55er | che55er.io

Today's Outline

Basics: Definitions and non-linear thinking.
Anatomy: Stocks, flows, delays, and types of feedback loops.
Leverage Points: Where are effective areas to intervene in our systems.

Thinking in Systems

Much of today's vocabulary and framing comes from Donella Meadows' Thinking in Systems.

We'll use its ideas to ask better architecture questions about behavior, feedback, and leverage.

What is a System?

A set of elements or parts that is coherently organized and interconnected in a pattern or structure that produces a characteristic set of behaviors, often classified as its "function" or "purpose."
- Donella Meadows, Thinking in Systems

Zipper Merge

Fixed One Thing Broken Another

Added database connections exhausted database memory
Auto-scaled a service triggered a cascading outage
Increased retries made an overloaded dependency worse

Linear Thinking

Reductionistic thinking, breaking something down into parts.

What Is Linear Thinking?

Linear thinking assumes a predictable chain of cause and effect: A B.

One action has one primary result.
More input produces roughly more output.
Fix the broken component and the system is fixed.

What Is Non-Linear Thinking?

Non-linear thinking recognizes that systems rarely behave as simple chains of cause and effect.

Multiple things interact, effects can feed back into their causes, and a small change can sometimes produce a huge result, or almost no result.

Behavior of the whole system cannot always be predicted simply by adding up the behavior of its individual parts.

The "Root Cause" Trap

Traditional Root Cause Analysis (RCA) looks for a single case.
In complex systems, there is rarely a single root cause.
Attributing outages to "human error" or "a bad deployment" stops learning.
Systems View: The system structure allowed a routine event to trigger a failure.

The Iceberg Model

Iceberg model showing events, patterns, and system structure

Iceberg model with deeper systems-thinking layers

System Anatomy: Stocks, Flows & Delays

Stocks: What accumulates? Requests, queue messages, connections, memory, and cached data.
Flows: What changes the stocks? Requests per second, message consumption, and processing throughput.
Delays: How long before the system knows or reacts? Telemetry, network, queue, deployment, and human response time.

The Retry Storm

Each failure causes clients to retry. Those retries add additional load to an already struggling system, which causes more failures, which triggers even more retries.

Feedback Loops

Reinforcing loop

(amplifies)

A change amplifies itself, causing more change in the same direction.

Example: Money investment more money
The more money you have, the more you can invest; successful investments give you more money to invest.

Balancing loop

(stabilizes)

A change triggers a response that pushes the system back toward a goal or target.

Example:
Room temp thermostat furnace
If the room gets too cold, the thermostat turns on the furnace, which raises the temperature back toward the set point.

What kind of feedback loop was the retry storm?

Reinforcing feedback loop(a harmful one)

Fulcrum Hunting

Seeking leverage points in our systems.

Leverage Points

Effort required to change system behavior

Numbers: Constants and parameters such as subsidies, taxes, and standards.
Buffers: The sizes of stablilizing stocks relative to their flows
Stock and Flow Structures: Physical systems and their nodes of intersection.
Delays: The lengths of time relative to the rates of system changes.
Balancing Feedback Loops: The strength of the feedbacks relative to the impacts they are trying to correct.
Reinforcing Feedback Loops: The strength of the gain of driving loops.
Information Flows: The structure of who does and does not have access to information.
Rules: Incentives, punishments, contraints.
Self-Organization: The power to add, change, or evolve system structure.
Goals: The purpose of the system.
Paradigms: The mind-set out of which the system - it's goals, structure, rules, delays, parameters - arises.
Transcending Paradigms

Explore the Leverage Points

10-12 (Tune )

Parameters, buffers, structure

7-9 (Control )

Delays and feedback loops

4-6 (Change )

Information, rules, self-organization

1-3 (Question )

Goals, paradigms, transcendence

Categorizing our leverage points as we look at examples of how we would apply them in our software systems.

Software Systems are Socio-Technical

A software system is shaped by both its technical components and the people / organizations around it.

Technical

Code and hardware
Networks and infrastructure

Social

People and teams
Organizational policies and user behavior

When we evaluate leverage points, we must consider both the technical and social dimensions of the system.

Tune #12) Parameters, buffers, structure

Change a constant, threshold, or parameter.

Intervention

HTTP timeout
Thread pool
DB connection limit

Example

DB connection pool: 50 200

Parameters are useful, but low-leverage. A larger pool may simply transfer the bottleneck to the database.

Tune #11) Buffers

Increase or redesign the capacity of a stabilizing stock.

Intervention

Message queues
Capacity headroom
Ready engineering backlog

Example

A service receives 10,000 events/sec but processes 5,000. Add a durable queue to absorb several minutes of traffic.

Tune #10) Structure of Stocks & Flows

Change the topology through which data, requests, and resources flow.

Intervention

Monolith to services
Synchronous to asynchronous processing

Example

Replace a synchronous payment API call with Payment Event Queue Payment Worker, then return an immediate response.

Control #9) Delays

Change how quickly the system knows and reacts.

Intervention

Faster monitoring and observability
Shorter deployment pipelines
Quicker user feedback

Example

Use automated tests, telemetry, monitoring, and canaries to reduce bug detection from three weeks to five minutes.

A control loop that reacts too slowly can overshoot; one that reacts too quickly can oscillate.

Control #8) Balancing Feedback

Strengthen the mechanism that pushes the system toward stability.

Intervention

Autoscaling
Circuit breakers
Automated rollback

Example

When error rate rises, automatically roll back the deployment instead of waiting for engineers to notice.

Control #7) Reinforcing Feedback

Reduce the gain of the runaway loop.

Intervention

Retry-storm controls
Technical-debt reduction
Design for adoption network effects

Example

Add exponential backoff, jitter, and bounded retries to reduce retry-storm amplification.

Bulkheads

Applies across multiple leverage points.

#10) Structure of Stocks & Flows Partitions shared resources, such as threads, connections, or worker pools, so one failing component cannot consume resources needed by others.

#8) Balancing Feedback Create a stabilizing boundary by limiting the amount of capacity that can be consumed by a failing dependency.

#7) Reinforcing Feedback Reduce the amplification of cascading failures by preventing one failure from consuming shared resources and triggering additional failures elsewhere.

Change #6) Information Flows

Give the system information it did not have.

Intervention

Production observability
Architecture decision records
Product telemetry

Example

Route customer failures directly into telemetry and engineering dashboards so developers learn about them immediately.

Circuit Breakers

Applies across multiple leverage points.

#9) Delays Automates the response to failure, reducing the time between detecting unhealthy behavior and changing the system's behavior.

#8) Balancing Feedback Detects unhealthy behavior and automatically reduces or stops requests, creating a balancing response that helps prevent further damage and allows the dependency to recover.

#7) Reinforcing Feedback Reduces the amplification of failure cascades by preventing failures from generating additional load, retries, and resource exhaustion.

#6) Information Flows Turns information about failures, latency, and dependency health into actionable information that directly changes how future requests are handled.

Change #5) Rules

Change what the system is allowed to do.

Intervention

Deployment policies
Reliability-focused team incentives
Architectural constraints

Example

Make teams accountable for both feature delivery and production reliability, not feature throughput alone.

Change #4) Self-Organization

Give the system the ability to change itself.

Intervention

Platform teams
Plugin architectures
Autonomous teams

Example

Give each team ownership to deploy and operate its service, with automated guardrails.

Question #3) Change the Goal

Intervention

Feature delivery to customer outcomes
Utilization to reliability
Growth to sustainable growth

Example

Change success from number of features released to customer problems solved.

Question #2) Change the Paradigm

Intervention

Software built once to continuously evolving system
Treat infrastructure as disposable rather than precious

Example

Moving from monthly large monthly releases to favor releases that are small, frequent, observable, and reversible.

Question #1) Transcend Paradigms

Intervention

Avoid architectural dogma
Treat methodologies as tools
Question existing assumptions

Example

Ask what problem microservices solve before assuming they are the right structure.

Summary

Beware the Event Trap

Events are symptoms; structure generates behavior over time.

Find the Loops

Ask what amplifies failure and what counteracts it.

Find Stocks, Flows & Delays

What accumulates, moves, and waits?

Traverse the Leverage Ladder

Do not stop at parameters when the problem is structural.

Question the Goal

Sometimes the highest-leverage change is changing what success means.

References

Meadows, Donella H. Thinking in Systems: A Primer (2008).
Montalion, Diana. Learning Systems Thinking (2024).
Nygard, Michael. Release It! Second Edition Design and Deploy Production-Ready Software (2018).

Thank you!

Carl Chesser
@che55er | che55er.io