What I learned applying the AWS Well-Architected Framework

I started using the AWS Well-Architected Framework the way a lot of people do: as a checklist before a review. That was the wrong entry point.

The framework is a set of questions about trade-offs. It exists so you can say, out loud, what a workload is supposed to do, what you are willing to operate, and which risks you are choosing to live with. The AWS Well-Architected Tool is useful because it records those answers. A review that does not produce a backlog and a list of accepted risks is just a meeting.

The six pillars — operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability — are not independent. The whitepaper is explicit about that. Improving one usually moves another. Most of what I learned came from walking a real EKS and GitOps platform through those questions, not from reading the design principles once.

How I use a review

I scope one workload first. For platform teams that is usually “how we provision and run Kubernetes,” not “the whole company.” I want to know the users of the platform, the environments it serves, and which failure would keep people up at night.

Then I go through the questions in the tool. I care less about a perfect score than about two lists:

  1. High and medium risks we will actually fix.
  2. Risks we are accepting, with an owner and a reason.

That matches how AWS describes the process: a constructive conversation about architectural decisions, not an audit.

Operational excellence

The pillar is about supporting development, running workloads, getting insight, and improving the process. The design principles that stuck with me were perform operations as code, make frequent, small, reversible changes, and anticipate failure.

The first gap I kept hitting was that cluster and application onboarding lived in people’s heads. Console clicks, copied YAML, and a different path per environment. Well-Architected asks whether you have a consistent way to make changes. For us that meant putting the platform in Git.

Infrastructure went into Terraform so VPC peering, shared add-ons, and cluster shape were reviewable. Application delivery went into GitOps so the desired state was not whatever last succeeded in a pipeline. When a single GitOps control plane became a dependency for every environment, the same pillar asked a harder question: can we still operate if that control plane is unhealthy? Splitting control planes and encoding shard topology in config was the answer, not a slogan about “GitOps best practices.”

The same questions pushed onboarding toward one config file and one command: network, add-ons, registration. A Kubernetes operator that opened the GitHub PR to register a new spoke cluster was not an interesting side project. It was operations as code for a step that used to be a runbook nobody wanted to follow twice.

Observability showed up here too. Operational excellence wants you to understand the health of the workload and respond to events. CPU graphs on nodes are not enough if on-call still spends the first half hour assembling context. Wiring Datadog, incident state, and GitOps health into a scheduled report did more for operations than adding another dashboard. The framework also asks you to refine procedures. A certificate rotation that required tribal knowledge failed that test. Writing a staged rollout and a recovery runbook that any on-call engineer could follow was the operational-excellence work. The rotation itself was reliability and security. The runbook was how we learned from the failure mode.

Kubernetes upgrades fit the “evolve” part of the pillar. If the only way you stay current is someone reading release notes by hand, you will drift. Generating upgrade PRs from changelogs is a small example of improving the process instead of heroically remembering it.

Security

The security pillar is identity, detection, infrastructure protection, data protection, and incident response. The design principle I used most was apply security at all layers, together with implement a strong identity foundation.

On EKS, the identity question is concrete. If pods inherit the node instance role, every workload on that node has a wide AWS blast radius. IRSA with OIDC, and a role per service account, is the Well-Architected answer in Kubernetes terms: least privilege at the workload boundary, not a prettier IAM policy on a shared role.

I stopped treating “tighten this policy” as the whole security review. The more useful questions were: can one compromised workload reach every environment? Are humans on SSO instead of long-lived keys? Is break-glass access rare and logged? AWS Organizations and SCPs belong here as guardrails, not as a compliance sticker.

Detection only counts if someone owns it. CloudTrail at the organization, Config in the accounts that matter, GuardDuty and Security Hub with an owner — that is the detection and response path the framework describes. Findings with no responder are cost, not security.

Service mesh certificates sat at the intersection of security and operations. Expired mesh certs are an availability incident with a security root cause. The framework’s incident-response guidance is to prepare, not to discover the runbook during the event.

Reliability

Reliability is the ability of the workload to perform its intended function correctly and consistently, including through change and failure. The questions that mattered were about change management and failure management, not whether we had checked the Multi-AZ box.

Multi-AZ for RDS, load balancers, and node groups is a foundation. It does not cover a bad deploy, a failed database migration, or a dependency you never tested without. The framework asks you to define recovery objectives and to test recovery. Backups you have never restored do not meet that bar. PITR on DynamoDB does not either, unless someone has practiced recovering from a bad write.

The work that mapped cleanly here was staged production cluster migration: validate in lower environments, run controlled experiments against AWS networking and service dependencies, promote workloads by tier, keep a rehearsed rollback. That is change management as the reliability pillar describes it. Horizontal scaling of GitOps control planes was the same idea: remove a single point of failure instead of hoping the one control plane stays healthy.

I also learned to be careful with “more regions, more clusters.” Reliability design principles include testing recovery and automating change. If the team cannot operate one cluster through failure, a second cluster is another failure domain, not automatically a better RTO.

When people ask for 99.99%, the reliability pillar is the right place to answer. Availability is an operating model: failover you rehearse, dependencies you understand, changes you can reverse. Duplicate infrastructure without those practices is not what the framework means by reliability.

Performance efficiency

Performance efficiency is using computing resources efficiently to meet requirements, and keeping that efficiency as demand and technology change. Selection, monitoring, and trade-offs are the three areas I kept coming back to.

On EKS, selection is requests, limits, instance families, and the signal you scale on. A CPU-only HPA on a workload bound by I/O or lock contention adds nodes and does not improve tail latency. The framework asks you to use architecture that best supports your performance requirements, then monitor those requirements. For APIs that meant p95 and p99, not average CPU. For anything in front of RDS it meant connection pooling and timeouts, because connection churn shows up as “the database is slow.”

Graviton sits in this pillar and in cost. It is a valid compute choice. It is also a pipeline and compatibility problem: multi-arch images, sidecars, and agents that actually run on arm64. I treat that as the “review” part of the pillar. You do not pick a new architecture family once. You keep checking that the software stack still matches.

Caching with CloudFront is the same kind of trade-off the pillar warns about. It can absorb load. Wrong cache headers can also hide bugs and make invalidation the outage. Performance work has to be tied to a workload, not to a list of AWS services we enabled.

Cost optimization

Cost optimization is delivering business value at the lowest price point. The design principles that showed up in practice were implement cloud financial management, attribute expenditure, and stop spending on undifferentiated work.

The first gap was tagging. A FinOps pass that finds missing AWS (and data-platform) cost tags is the expenditure-awareness question. If you cannot attribute spend to a workload, you cannot decide whether that spend is buying reliability, developer time, or idle comfort.

The waste I kept seeing on EKS was not only node size. NAT Gateways on every private route, one load balancer per service, cross-AZ traffic, image pulls and log shipping across the wrong boundaries, and agents on every node. The framework’s “right-size” and “select the appropriate resource type” guidance applies to those line items as much as to instance families.

Unit cost helped more than a raw monthly bill. Cost per request, per active tenant, or per GB ingested, reviewed on a cadence, tells you whether a change improved the architecture or just moved spend around. Right-sizing always-on databases, shutting non-prod when nobody is using it, and deleting unattached EBS volumes and old snapshots are the unglamorous practices the cost pillar actually lists.

Vendor evaluation belongs here too. Proofs of concept and a gap analysis before committing spend is cloud financial management. Buying a tool because the current platform is painful, without stating which Well-Architected gap it closes, is how cost and complexity grow together.

Terraform and GitOps affect cost because they copy defaults. A module that always creates a NAT Gateway and an NLB will do that a hundred times. Consistency is operational excellence. The cost pillar asks whether those defaults are still the right resources.

S3 Intelligent-Tiering is a good example of reading the service against the pillar instead of enabling it as a best practice. It helps when access patterns are unknown and objects live long enough. The monitoring fee per object can dominate if you have millions of tiny, short-lived objects. Lifecycle rules may be the cheaper, clearer policy.

Sustainability

Sustainability is reducing energy and increasing efficiency by maximizing the benefit from provisioned resources. In practice it overlapped with cost and operational excellence almost completely.

Fewer idle environments, shorter retention where it is safe, less unnecessary data movement, modern instance families when the stack is ready. The design principle of maximize utilization is the same work as turning non-prod off at night — with the operational caveat that developers still need a way to boot a seeded environment. If shutting staging breaks every workflow, you have not improved sustainability. You have moved the problem onto people, which the operational-excellence pillar will catch.

I do not treat this pillar as a separate program. When we stopped keeping extra always-on clusters “just in case,” we reduced spend and waste at the same time. That is the overlap the framework expects you to notice.

Trade-offs the framework expects you to write down

The definitions paper is clear: you often cannot maximize every pillar at once. The reviews that helped were the ones where we wrote the trade-off down.

Reliability wants redundancy; cost and sustainability want fewer idle copies. Security wants tighter identity and more audit; operational excellence wants reversible change without a ticket for every deploy. Performance wants more capacity or a different architecture; cost wants you to prove the unit metric moved.

The platform work I trust now is the work that came out of those questions: operations in Git, identity at the workload boundary, recovery you have rehearsed, scaling signals that match the bottleneck, spend you can attribute, and idle capacity you can defend.

What I do after a review

  1. Keep the workload small enough that the answers are specific.
  2. Record high and medium risks in the Well-Architected Tool, with owners.
  3. Pick a small number of fixes that map to design principles, not a rewrite of the platform.
  4. Write down the risks you are accepting. “We are not multi-region yet” is a valid answer if someone owns the implication for RPO and RTO.

The official docs are the source of truth for the questions. Experience is only how those questions look once you are running EKS, GitOps, and the AWS bill that comes with them.

References