I will remember fondly my time at Stripe, and especially all of the engineers I had the opportunity to work with. Working at Stripe was a great learning and growth opportunity, and I have no regrets about my last four and a half years. However, I did have a few things I wanted to share before closing out this chapter of my life. 

Edit the company

Every transition has elements of push and pull. I am leaving primarily because of the opportunity I have at AWS. However, I still want to take a moment to reflect on the secondary factors that made Stripe a less-appealing place for me to stay. What I share here I share only with the hope of inspiring others to edit the company in ways which I was ultimately unable to, with the hope that we make a great employer even better.1 

Process, not products

Stripe has a very “ship” based culture. The processes and norms feel optimized for delivering a big chunk of work where we can clearly say “we did it!” However, not every type of work fits into this paradigm. 

The over emphasis on bundling everything into a “project” that we could declare victory over means teams would struggle to incorporate work that requires small-but-constant improvement. In my case, I’m specifically thinking of reliability and operations.

As an example, reliability is not something we can ship or declare victory over (though it often felt like we tried). Reliability requires constant vigilance, the flexibility to tackle problems as they occur, and the knowledge that problems will occur constantly. 

Much of what made Stripe reliable and a generally productive place to work is the work of engineers who diligently sand away the rough edges that cause toil. Paraphrasing one of the lessons from Toyota Production Systems, the most critical work is the work that smooths the way for other work.

Execution versus impact

Stripe is also very “impact” driven. This alone is not a problem, and I definitely appreciate how we got here. When you have a large number of high-agency engineers working on various problems, under budget and time constraints, you want folks to leverage their efforts to achieve the highest impact. And it follows from that we want to close the loop by holding people accountable to the impact they deliver. 

However, as Stripe grows and folks are more removed from the decision making processes, I see the expectation of having folks speak to the impact of their work causing more and more tension with engineers. Especially folks earlier in their career. Put another way, an employee that doesn’t have input into the project they are taking on should not have to speak to the impact of the work. At a certain point (at least for junior) we should be holding engineering managers accountable for impact and engineers accountable to execution. 

Overloaded engineers

It was not uncommon during my time at Stripe for teams to be loaded with work such that there are 2 active work streams per engineer. This is a higher amount of loading than I think is healthy, and it impacted us in a couple of ways. 

First, there is very little slack in the system to pick up work on an as-needed basis. A certain amount of slack in resources is needed to polish issues as blemishes are noticed.

Secondly, this type of loading negatively impacts the sense of comradery on the team. A team where everyone is working on a different thing is not really a team; it is just a group of engineers sharing a sprint board. For teams with a broad scope of ownership, this also limits the amount of context anyone will have relative to all the systems they’re expected to operate. This makes on-call much more stressful and risks burning out the team. 

Lastly, a smaller set of goals that everyone on the team has context on makes it easier for teams to self organize and allows engineers to experience a higher degree of agency within the team. 

Interfaces

I saw a number of initiatives during my time at Stripe that focused on shoring up reliability and security for critical components. These generally took the form of evaluating each component of a system against a rubric. 

Complexity, however, is at the edges. It is at the interfaces of our systems. This turns out to be where the reliability risks are as well. 

I’m not sure what my point is here. But my advice is if you ever find yourself asking about the reliability or safety of a service, you’re almost certainly asking the wrong question. The focus should be on systems; the flows and the units of functionality provided to internal and external customers. 

Incomplete migrations

Stripe struggled to complete migrations, and we did not properly account for the weight of the work not finished. 

Sometimes the migrations were not finished because we did not budget the work to move load to the new system after creating it. Often, these migrations were very expensive to actually do because of how untested the flows the run on them are; moving is too scary and stressful.

Other times we took a dependency on another team's “migration” which never completed. 

We could reduce the context engineers need to maintain by a significant percentage by deprecating old systems before building new ones. 

On my team, we had at least four incomplete migrations. This work-not-finished was a major drag on the productivity of the team. It made changes more difficult to reason about because of the multitude of code paths that need to be considered. Reliability was impacted for similar reasons: more to think about when making changes or performing mitigations during incidents. 

I recommend folks adopt a budget on the number of systems or code paths their team will own. Space is created in the budget by turning off systems which are no longer needed. 

Shared context

My largest personal struggle at Stripe was trying to get where I felt like I was “in the room” where strategy was being formed and solidified. As a staff-engineer wearing a tech-lead hat, I need shared context with my manager and their manager to correctly prioritize and do my work. This means knowing the criticism of plans and ideas from higher up. Receiving a filtered view of the world hinders my ability to affect change within the company. 

This is actually my largest source of dissatisfaction with working at Stripe. At my previous employer, even though I was nominally operating one level below where I am at Stripe, I had more access to my leaders 2-3 levels up the organization hierarchy. There was significantly more transparency around the reality of resource constraints and business needs (and dare I say, internal politics.)2 I felt I had a seat at the table when it came to roadmap building and decision making. Lastly, at my previous employer, I felt that I had leaders willing to lend me their authority in a way that I do not have here. 

It is possible that when it comes to Stripe that I am “holding it wrong” and that getting things done here requires a different skill set than the one I currently possess. 

Parting advice to engineers

Finally, I want to share some parting advice to the engineers I worked with, especially those more early in their career. You’re all amazing people and you’re going to go places for sure. Hopefully the things I say below were thing you heard from me while we worked together, but in case you didn’t (and for everyone else), here they are:

You can do it

The teams that I worked on during my time at Stripe dealt with a lot of gnarly interoperability issues with external companies and services. We often discovered issues in open source packages or had to create our own libraries for obscure protocols. Issues were often found only after a system had been misbehaving for a while and large amounts of money were at stake.

This can be an intimidating and paralyzing environment to work in. 

Believe that you can find the answer by reading the code, by reading the RFC, by looking at raw packets. Everyone is capable of seeing the Matrix. It isn’t a quality folks are born with. It just takes practice. Gumption. 

It does also often require muting the Zoom call and going heads down for some time. The job of your leaders is to play defense so that you have the room to do what you do best. Hold them accountable to this task. During an emergency the first priority is to mitigate. Let others focus on communication.3

Sometimes you fail

A senior engineer is just a junior engineer that has made a lot of mistakes and seen others make a lot of mistakes. Making mistakes, dragging teammates into incidents, and even occasionally breaking production comes with the territory. 

It doesn’t feel good when it happens. It usually feels really shitty. But you are growing. And it sucks, but this is just what it feels like. You are pushing your own abilities. Failure is truly the best teacher. 

You don’t owe the world perfection. Just do your best to make sure you have learned something from the experience.4

One way to grow from a mistake is to involve yourself as much as possible in the incident review/RCA process. Anecdotally I have found a well written analysis of an incident with a good list of remediation items has earned me more professional praise than causing the underlying incident earned me blame. 

Ask the hard questions

Much of the value I provided at Stripe was being that guy. When engineers and leaders are sitting in a meeting reviewing a retrospective on an incident or current operational metrics, there is an ever present risk of complacency. This is a mode that even the most disciplined, operational rigorous teams can get into. And it requires someone to break from the social cohesion. 

When you’re looking at a design or reading an incident report, notice when something gives you pause or causes you unease. When something doesn’t feel right, call it out. 

Raise your hand.

Practice putting it into words. 

Onward

As mentioned earlier, my new role is at AWS Cryptography. I am joining as a Principal Engineer and will work with my team to continue to innovate in the realm of applied cryptography. I’m excited to join an organization that is eager to take on large, moonshot projects and tackle some hard problems. 

Previous to my current position at Stripe, I was a software developer at Amazon Web Services (AWS), working on the Key Management Service (KMS) from 2015 through 2019. 

The AWS Key Management Service (KMS) is, as the name implies, a service for managing cryptographic keys in the cloud. In short, the service allows you to create keys, use them to encrypt and sign data through an API, and grant or restrict access to the keys using a few different access control mechanisms. AWS KMS is used by other AWS services to protect customer data, and some customers also build applications directly on top of the KMS APIs. 

Disclaimer: though informed by my experience as an employee of Amazon, my thoughts and opinions are my own and may not be shared by my former or current employer. 

While working on this team, I had the privilege of leading the development of two high-impact products:

• The Bring Your Own Key (BYOK) feature allowed customers to import key material into AWS KMS from their own, on-premise HSM infrastructure.1 This provided an option for customers that wanted to take responsibility for the generation of keys themselves and/or did want their keys to exist only within AWS. 

• Custom Key Stores (CKS) allowed customers to create a KMS key that was backed by a key in a CloudHSM cluster they controlled.2 Similar to BYOK, CKS gave customers a way to use KMS while meeting requirements they may have to retain key material in devices that were more similar to traditional on-premise HSMs.

Both features were key in enabling strategic customers to move to AWS. It was truly an honor to be a part of project work which grew the cloud business by such a significant amount.3

However, what really excited me about this work was how we executed. And that is what I want to write about today. 

Both projects were performed under time pressure from the AWS business and customers that were eager to utilize these features. As the lead engineer, I was responsible for scoping the work involved and arguing for the time and resources we would need. In both cases, I was given both less time and fewer people than I originally thought were necessary. 

Otherwise, where would be the fun in it? 

(From this point on, I will jump between discussing the two different projects because there were a lot of similarities, and also because I can’t remember exactly which memory was from which effort.)

Planning

I found when the project planning really kicked into high gear by looking at my Amazon order history for “red string.”

I exaggerate, but not that much. 

My goal at the beginning of both projects was to get a birds-eye view of the work that would need to be completed, the effort required for each piece of work, and the dependencies between them. Having this laid out at the beginning would allow me to have an approximation of how long the project would take to complete. By understanding the dependencies between the tasks, it was also possible to understand the impact on headcount to the delivery of the project. 

My notecard and red string approach went like this: 

1. Write a note card with the end goal, such as deliver feature X. 

2. Think about what composes that end goal. In the case of an AWS product, this might be “API for X”, “console website for X”, “documentation for X”, etc. Attach each of these with a red string under the main goal. 

3. Repeat for each subgoal. For example, “control plane API”, “data plane API”, etc. If at any point, there is a dependency between two tasks, connect them with a red string and make the task that must be completed first lower. 

4. Repeat, decomposing tasks into smaller tasks (design API, review API with committee, etc.), until you’re down to tasks which can be completed by a single person in a couple of days, and they are not blocked by other dependencies. 

The end result is effectively a dependency graph. For each task, I would add an approximate time estimate. The leafs (tasks with no other tasks connected below them) should be tasks that developers can pick up and get started with. The longest path (adding up the estimates on all the tasks) from a leaf to the root is the shortest theoretical length of time the project could take, assuming human resources were not a constraint.4

As the project progressed, it would be easy to keep an eye on the critical path, add tasks that were uncovered, and identify work that was ready for someone to pick up. 

This is more or less how I coordinated the work for the BYOK feature. 

The downsides of this approach were that it relied on me being close to the board with all the cards and string in order to answer questions about the project, so for the CKS project work I acquired a Microsoft Project license5 and repeated the exercise, except with gantt charts.

The advantage of using Microsoft Project was that the information was more portable. It was also in a format that other stakeholders such as project managers could consume and understand. However, the downside is that Microsoft Project asks a lot of questions, and puts an over-emphasis (in my opinion) on calendar dates when laying this out.6

However, the start-with-goal-and-work-backwards goal is overall the approach I continue to use today to both understand the work a project will require and to provide accurate timelines.

Consensus building

Both the BYOK and CKS features involved expanding the capabilities for the AWS Key Management Service, and therefore were under a lot of scrutiny by the security-minded folks of AWS Cryptography (of which there are many). An important part of delivering both projects without unexpected delays was ensuring there was consensus within the principal engineering community insofar that the security of the system was concerned. 

An objection later in the project cycle, even if it were to be eventually withdrawn, would risk disrupting project timelines. Therefore, early on I made sure to meet with certain key Principal Engineers, one on one, to hear out their concerns for the feature, understand their recommendations, and explain my thinking about how we’d ensure the security of our system would not be compromised. 

As the project progressed, and certain aspects of the design were refined or clarified, I continued to check in to ensure there would be no surprise concerns or objections raised close to the deadline. 

In all cases where concerns were raised by principal engineers (not necessarily about security), I prioritized responding in writing with the mitigations we had planned and our thinking on the topic. Whatever I wrote would also become a permanent appendix to the design document portfolio for the project, so that we would not have to rehash the same issues over and over. 

What we needed and why we need it

For both projects, there was time pressure to deliver artifacts sooner than (at least on paper) the requirements and resources would allow for. 

This pressure came from two different sources:

• External: we had made or wanted to make commitments to strategic customers about when a feature would be available in order to grow business and win contracts. 

• Internal: both projects were seen as risky and pushing the capabilities of our technology stack; there was a desire to know sooner than later if there would be unexpected difficulties. 

The solution to satisfy both concerns was an incremental approach to delivery. I’ll speak more about reducing the risk of project execution for internal stakeholders momentarily, but I want to speak to it from an external, customer facing perspective first. 

For both projects, we knew of specific customers that were particularly interested in the new features we were building. Indeed, it was critical that the product we built would meet the requirements of these particular customers. However, large customers rarely will adopt a new AWS product into their production environment on the day they are given access. Just as we need time to build something, customers need time to validate it meets their requirements and understand how they will integrate it. This can take months (or even years for larger enterprises). 

With this understanding, we prioritized the customer-facing aspects of the project work which would best allow us to collect feedback from the customers. We built APIs with the backends stubbed out, as well as documentation, that would give strategic customers the opportunity to start playing around with our product in a non-production, even if it would not be fully functional. 

Just as importantly, this approach provided us an opportunity to change direction earlier in the product life cycle in a way that would be much more difficult if the backend functionality had already been built out.

Validating early

“Build a skateboard before a car.” 

I’ve heard this advice many times, but it can be difficult to apply. After all, a car is very little like a skateboard and the use cases are generally pretty different for each. 

Reframed, the advice I would give to other engineers is to do the riskiest work up front. If you’re not sure you know how to make wheels that spin on an axle, make sure you can do that on a skateboard before you go about building the entire car. 

In the realm of building novel key management products, this has most often meant validating that the new infrastructure can provide the throughput (operations per second) at the desired latency (time to complete each operation) early on. More generally, it means first doing the parts of the work which are least like what you’ve done before. 

Ideally, after each project milestone, behavior of the entire system should be revalidated. For example, raw benchmarks against the hardware were satisfactory, does it still work when we add network hops? How about after we add authentication and authorization checks? How about after logging? 

If there is a regression, it is useful to be able to narrow down where it happened. The worse case would be getting to the end and finding that none of the performance requirements are met.

Priorizing for iterative development

Validating often is easier when validating is easy. 

The Custom Key Store (CKS) project specifically required a lot of new infrastructure components to be built. Each performance and functional test required a lot of up front orchestration work. For this reason, we spent a large portion of our early time building the test harness which would allow this testing to be performed automatically. 

It was a risk to spend so much time on automated testing before the product was even at a minimum level of functionality, but the investment paid off greatly. Every change made by any developer on the project was automatically tested for functional regressions. This saved weeks of developer time over the course of the project on testing alone, and once the testing infrastructure was there, allowed people to proceed with a much higher level of confidence. 

Similar to automated testing, ensuring the continuous integration and deployment infrastructure is working early on also allows a project to proceed at a faster pace.  

After launch

Because of the constraints we were often working under, a product seldom shipped to customers with everything we initially had wanted. Often there were features excluded from the API, the web console, or issues on the backend that impacted operations for a short period.7

After completing both projects, I took some time to compile a roadmap document. This document outlined the known deficiencies we’d need to address, work we might need to do to scale up in the future, and opportunities for growth later on. 

Writing this down was valuable for two reasons: it closed the loop on the project with my leadership, providing a sort of bookend to the work while still communicating there was work to be done in the next planning cycle. Because it was written, it also allowed other engineers on the project to add their feedback and point out anything I missed. 

Wrap up and takeaways

I would like to conclude by drawing attention to the fact that the title of this essay is “how we did it”, not “how I did it.” I worked with a fantastic team at AWS; truly one for the books. Nothing would have been possible without my excellent colleagues, my leaders, and the superb Principal Engineering community at AWS.

What I would like folks to take away from all of this:

• Product management is critical. It is a gift to work with leadership who take the time to understand the needs of customers below the superficial level; to truly understand their motivations, concerns, and timelines. 

• Ensure you have the ability to validate early: validate with your customers and validate your own assumptions about how new technology will perform.

• Prioritize automated deployment and testing infrastructure. This will lead to efficiency gains and allow developers to iterate and experiment more quickly and with more safety.

Disclaimer: though informed by my experience as an employee of Amazon and Stripe, my thoughts and opinions are my own and may not be shared by my former or current employer. 

Glossary

• EM: Engineering1 Manager. Someone who manages teams of around 5 to 8 ICs.

• IC: Individual Contributors, generally software developers that spend their time building and maintaining things. 

• MoM: Manager of managers. An individual that manages a handful of EMs.

• SLA: Service Level Agreement. A promise (often contractual) to customers or users about how your service will perform or how reliable it will be. 

• SLC: Service Level Commitment. An internal commitment to how services should perform or how reliable they should be. Less formal than an SLA. At Amazon, the term “internal SLA” was used in lieu of SLC.

• SLO: Service Level Objective. A goal set by a team for how well it wants a service to perform, but not as formal or broadly communicated as an SLC.

Note that these terms may vary in their usage company to company, or even within the same company. My goal is to define them as I am using them in this essay.  

What is an operational review?

An operational review (ops review, for short) is any process put into place to periodically review the health of the systems operated by a team. To keep things simple, I will scope this discussion to teams that operate software services and other computer infrastructure. 

My journey through operational review cultures

Amazon Marketplace

I joined Amazon Marketplace (part of the retail side of Amazon) in 2010, fresh out of college. In my early career at Amazon, the organization I worked in had a very minimal process for oversight of operations. Teams had metrics, dashboards, and their own monitoring, but there was no concept of an SLA on these services (from the perspective of ICs at least). EMs would be asked to discuss outages or unexpected behavior during internal operational reviews, but ICs did not take part in this process aside from the ICs ending their on-call rotations preparing notes on issues that occurred during their shift.

This lax approach could be attributed to the asynchronous nature of the systems my teams managed. The systems we owned either worked on asynchronous tasks as part of order fulfillment workflows, or provided information for merchant-facing dashboards and APIs, which were relatively low throughput. High latency and blips of unavailability could be tolerated as long things eventually worked.

Operational Reviews at Amazon Web Services

In 2015, I moved to Amazon Web Services, where I would work for 5 years as an software developer on the Key Management Service. AWS was a completely different beast. AWS consists of customer-facing functionality and APIs. Large enterprise customers build their businesses on these APIs and services. As a result, some customers have very high expectations. Even for services which do not have an externally published SLA, teams are expected to set an internal SLA (i.e. SLC). 

In general, most teams would review their core metrics (those most representative of customer experience) weekly during their on-call handoff (including any metrics they have an SLA on.) My team in particular would review our most core metrics, and then also randomly select another dashboard to audit both for clarity and for issues illustrated by the metrics themselves. This meeting was minimally attended by the previous and next on-calls and EMs, but was also well attended by other ICs.

At the director level there would typically be another organizational ops review. This meeting would involve a review of issues and major incidents affecting the org and would involve a deep dive into one of the products under that director each week. ICs were generally not expected at this meeting unless speaking to a specific issue.

The cornerstone of AWS operational review culture was the company-wide Wednesday ops review meeting. The basic agenda of this meeting was to celebrate operational wins, review upcoming changes folks needed to be aware of (code freezes, migrations, etc.), dive deep into critical incidents, and lastly a dive deep into a random service’s dashboard.

This meeting was open to all in the company, and was regularly attended by senior leadership and the Principal Engineer community. Every team was expected to send a representative (whether this was an EM or IC was discretionary, as long as they were able to speak for the team and could present the team’s metrics if chosen).

Having had opportunities to present both incident reports and my team’s metrics at this meeting, I can attest that this was both an incredibly intense and rewarding experience. Intense, because you’re speaking to (and questioned by) some of the most senior engineers and leaders at the company. Rewarding, because the culture was blameless, and because of how clear it was in those moments how ruthlessly the business cared about getting the fundamentals right.

Stripe

In 2020, I moved to Stripe to work on systems related to the secure storage and processing of credit card data.

When I first joined Stripe, there was no operational review process as such that was visible to ICs. There were no SLAs at the service level. There was a healthy culture of dashboards, and at least within my organization, detectors that alerted us when things were amiss. We had an on-call handoff meeting, but it generally did not involve a retrospective of service health over the last week, aside from pages or incidents.

As Stripe grew, there was an increased focus on operational reviews. We started defining SLCs for our services and tracked if we were meeting the SLCs. A number of other dashboards were created to help teams understand their posture with respect to service health and initiatives at Stripe where action was expected to be taken.

In 2021, within my organization a weekly meeting was created to review these dashboards and commitments with EMs and some ICs. EM attendance was expected, and ICs were invited but not required to attend. The same meeting was also used to deep dive into incident reports for the organization, and was generally used to perform the “review” expected of an incident, unless a higher level of review was deemed required due to the severity of the impact. The meeting was facilitated by volunteers (usually EMs).

Taking the reins

While I was pleased to see Stripe mature to the point that we were regularly tracking our service health, there were several areas where I felt the process fell short:

• The process was overly focused on the previous week and was not set up to reveal long term trends or potentially systemic issues.

• The meeting didn’t have a clear agenda, and the audience was not well defined. Often the meeting did not have the right attendees to discuss more severe issues. 

• Folks in attendance from various teams were often not prepared to discuss operational incidents or abnormalities, and there was no mechanism to follow up after the meeting.

• The meeting did not provide a forum for on-call engineers to share their perspective or concerns about system health. 

In early 2022, my organization went through a restructuring and the result for me was a new leadership chain. As a part of this restructuring, we also lost the management sponsor for our operational review process. This presented an excellent opportunity to start fresh.

With full support from my EM and MoM chain, I took on the responsibility of rewriting our operational review process.

I started with interviewing folks that had expressed an interest in sharing an opinion on our ops review process, both positive and negative. In these meetings, my goal was primarily to understand what folks hoped to get out of an ops review (that is, what would make it useful for them), and what folks liked and didn’t like about the previous format.

With this information in hand, as well as my own experience, I drafted a document which described the shortcomings of the previous format, our shared goals for the new format, and a clear statement of purpose. I ensured we all broadly agreed on these points. Next, using these previous points as an input, I drafted an outline for what the meeting would look like. After reviewing the planned format again with the critical stakeholders, and anyone who had a strong opinion, it was time to dive in and try it out. We’ve been iterating and making small improvements ever since then.

I will spend the remainder discussing what we developed.

Shared Meaning

The first challenge was developing a shared understanding of why we had this process:

Why are we having this meeting? Why should folks be here rather than doing other productive things? Why do we care about ops?

At AWS, where I walked into a functioning operational review process, this step was not necessary. I believe a meeting that folks find useful can be a justification in of itself, even if you cannot put it into words. That meaning might even be partially of symbolic value. I think this was sometimes the case at AWS. The size and attendance of the Wednesday ops review meeting and who attended broadcasted a very clear message: “operations are important here. Our customers care. We care. We will make sure you understand the health of your systems. If something goes wrong, we will be working with you to figure out a solution.” 

That was a message that came from the top. With that signal, local ops reviews at different levels tended to fall into place.

At Stripe, that shared culture did not exist to the same degree. 

And that meant defining a purpose for ourselves. The challenge here was to define a statement of purpose that resonated with both ICs (whose participation we wanted to encourage) but also was pragmatic in that it reflected the needs of EMs and MoMs to have a clear picture in their heads of how we’re doing so they can both communicate to their leaders and prioritize work.

The “purpose” was thus synthesized from input from both the EM and IC interviews. 

Here are the goals we came up with for the meeting:

• Verify our systems are behaving as expected and understand the ways in which we are not.

• Review the health of the people-driven processes that support the operations of our services (e.g. pages, on-call workloads) and ensure workloads are sustainable.

• Verify that for recent incidents we’ve learned the correct lessons, correctly assessed impact, and the appropriate artifacts have been generated.

• Understand our long-term trends and ensure our services will deliver the expected functionality over the next 12 months.

When a more succinct summary is called for, I say the purpose of the meeting is “situational awareness.” Having a weekly meeting where we come up for air and confirm to ourselves that everything looks like it is running well allows us to be more heads-down and productive the rest of the week.

Facilitation

Even with an agreed-upon purpose in place, we decided a facilitator is useful for consistency. 

The facilitator of this meeting is responsible for making sure:

• The meeting is useful.

• The time is well spent. 

Ensuring the meeting is useful and efficiently run helps drive continued participation.

This, of course, involves the usual duties of time management, making sure notes are recorded, agendas updated, and so on. 

To ensure the content is useful, and the time is well spent, the facilitator is also responsible for following up with owners of content in the meeting to make sure they are prepared is essential. Folks should not be scrambling to figure out what they need to say, nor should they be surprised about what might come up during the meeting. This also means letting presenters know if we’re going to ask about an incident, for example.

During the meeting, the facilitator runs the presentation, shares visuals for the most part, cues others when it is their turn, and manages time. 

We currently rotate facilitator duties within a group of interested volunteers. The goal is to balance redundancy (so that there is always a facilitator available) while making sure the facilitator has the experience and interest in fulfilling the responsibilities. 

Facilitator as Auditor 

There is an additional role that the facilitator must play that is important and unique enough to warrant its own section. 

The facilitator is responsible for ensuring we’re holding ourselves to a high bar for clarity and understanding. 

Other contributors are responsible for reporting on their systems, incidents, etc., but the facilitator is responsible for ensuring that the information presented makes sense, is understood by everyone, and to ask the questions that other folks might not be asking. 

Unfortunately, this is the most difficult aspect of the role to teach. In general, my advice to prospective facilitators is to trust your gut: if something doesn’t make sense, or you have a question, ask it. Don’t assume that everyone besides you understands it; be the backstop to misunderstanding. 

If a question is important to our understanding of our systems or an incident, but no one is able to answer it during the meeting, ensure it is assigned as an action item at the end to follow up. Make sure it is followed up on. 

Structure and Format

Attendance

We request that all EMs and the organization leadership (including any technical leads) attend, if at all possible. Aside from that, each team is expected to have folks present which can speak to the operations for the team for the last week. 

The attendee or attendees for each team are responsible for preparing slide content assigned to their team. 

Anyone else interested is welcome to join. 

Contents and Structure

The basic structure of the ops review format we’ve developed has four sections. 

Each section has a clear owner, and the owner presents. 

Across the four sections, there is the common goal presenting information which shows deviations from the baseline. This is in line with making sure the time is well spent and the content is useful. If there is nothing to report, the meeting progresses quickly and we spend time on other sections (or end early).

Team/Service level metrics review

Each team is responsible for discussing anomalies from how their services usually operate. The goal is to give each team space to speak to how the last week went for them, but to use the time efficiently by only speaking to deviants from the norm. We ask each team to independently review their operational metrics (defining operational metrics could be its own document), any incidents that they were involved in (this does not necessarily mean caused), and lastly any FYIs they have for other teams in the organization. 

In our case, we also selected some overall organization metrics and assigned responsibility to the Tech Lead for our organization to gather and present these each week. 

Projects and Migrations

Any team doing work, or aware of work being done by teams external to the org, that impacts teams in our organization can use this space to share updates. 

In our case, we use this most frequently to inform teams about datacenter work. We also notify folks of database migrations, or other changes to how services operate. 

If any systems owned by the organization have been flagged (through an incident or otherwise) as an area of risk or concern, we ensure we reserve time to review trends for that specific system as well. 

This information is presented by the teams involved or most adjacent to the work. 

On-Call/Runner Metrics

We use this section to review the health and sustainability of our on-call. 

Key metrics are the number of pages and tickets. We also ask offgoing on-calls to submit subjective scores for their run experience. This provides EMs and our MoM visibility into the sustainability of the human side of our operations, and to ensure on-calls are not at risk of being overwhelmed and burned out. 

As a practical matter, this section is prepared by the facilitator from the available data. 

Incidents

For each incident where our organization was implicated, the owner of the incident report prepares a summary of the incident. We include a summary of the event, the proximate causes of the event, and action items. 

We do not do a deep dive into the incident, as we found that it is better to have a dedicated meeting for those purposes. That way, the incident review can have the appropriate invitee list, which may be different from what is appropriate for the ops review. 

Format and Implementation

When we originally started running this meeting, participants were very strongly in favor of using a Google Doc template for each meeting, with the thought that this would be easiest to fill in. 

After a couple of weeks, we decided as a group to move to a slide deck format. Using a presentation format made it easier to divide up the content and assign ownership per slide. The format also encouraged brevity. Lastly, it was easier for the presenter to share during meetings. 

We maintain a slide deck that is cloned for each week’s meeting. The template is updated as we make improvements to the format. Speaker notes are used in the template to provide instructions for where to pull data from and suggestions for how to use each slide. 

The slide deck used for each meeting, as well as a video conference recording, are retained and linked to from a confluence page. 

Before the Meeting

The meeting takes place weekly, and covers events from the previous week. 

On Thursday before the meeting, an automated Slack reminder goes out to the facilitator to remind them to clone the presentation template, fill in dates, and update links. This ensures it is available for folks to start filling out.

Before the meeting, an automated Slack reminder goes out to on-calls to remind them to fill out the deck. The facilitator is also responsible for identifying if there is an incident that needs to be discussed this week and reminding the responsible team. 

The facilitator is responsible for gathering data for some common aspects, such as the on call metrics.

Lastly, before the meeting, the facilitator is responsible for nudging any teams which have still not provided content for their sections. 

Running the Meeting

At this point, running the meeting is fairly straightforward. 

The slide deck acts as the agenda for the meeting and the meeting should just go through slide by slide, allowing owners of each slide to present content when it is their turn.

The facilitator should ensure the meeting runs on time and politely cut off discussions that may lead to not getting through the content for the meeting. 

Lastly, the facilitator should ask questions themselves (preferably after others have had a chance to ask) to ensure all information presented is clear. 

Afterwards

The facilitator is responsible for sending an email with a link to the recording and slide deck. The email should also clearly summarize any action items that were assigned during the meeting, or actions that need to be taken by folks.

The same should also be recorded on any wiki pages for future reference. 

Measuring Success

Given the amount of work that had gone into planning this format, there was a strong desire by our organization leaders to understand the impact this new structure was having and the sentiments of participants.

We initially attempted to do this by sending out small surveys to participants after each meeting. Even though we kept the surveys short (2-3 multiple choice questions), I observed that people get quickly burned out filling one out each week, and the data quickly becomes sparse. 

I found it was more useful to ask folks during our regular one on one meetings for feedback. This also provides the opportunity for folks to share more nuanced sentiments that might be lost in a multiple choice survey. 

To this day, we continue to iterate on the format, how we collect and present data, and what items go on the agenda. The greatest signal I have that I’ve built something valued and enduring is that I am not the only contributor. Other folks in the organization bring their contributions to the format, and I overhear them evangelizing it to others. 

I have been especially pleased to see the meeting continue to run itself while I am out on parental leave, and I am excited to jump back in later this year and see how things have grown and the improvements folks have made. 

This is part two of my reflections on incidents so far in 2023. You can find part one here.

To recap: consuming a large amount of information about how systems fail is one of the greatest privileges of working at Stripe (and formerly in Amazon Web Services). This year, I set a goal for myself to sit in as many of these incident discussions as possible to try to extract some common threads to share back to the organization. I’m taking the opportunity now to share some of the observations I’ve made so far that are more generalizable and appropriate to publish externally. 

I devoted part one to testing. For part two, I will go into some other areas. 

Change management

Change management often gets a bad rap. When I speak of change management, I am specifically talking about the policies, procedures, and tooling used to manage infrastructure changes and software deployments. This usually centers around a form that should be filled out to describe a planned change and track its execution, but it also encompasses the policies and practices around when such a form is used, who must approve changes, and so on. 

Change management can have a bad reputation because of how it is perceived to be (or actually becomes) a bureaucratic process that gets in the way of productivity. However, change management, when applied appropriately, can be a powerful tool in de-risking work that can’t otherwise be de-risked with other mechanisms such as testing.

Lack of change management can lead to issues such as missed steps during rollout, an inability to detect when something has gone wrong, and an inability to roll back.

What can be done?

There are two separate points to discuss: when to use a more formal change management process, and what such a process should look like. 

From my personal experience, a formal change management process is more effective and more respected by engineering teams when it is used sparingly and limited to one-off changes which are not a routine part of operations. 

Routine changes, such as software deploys, are best handled by automated tooling which has safeguards built in and takes humans out of the loop. Automation should be used wherever possible, and manual change management processes used where manual activities must be performed by humans. 

There are much richer resources out there on developing a change management process, so I will just cover the major points. 

A change management document minimally requires:

• Description of a change to be performed

• Procedure to follow to safely complete that change

• Review / approval process for the description and procedure

• Tracking of the change while it is being executed

This can be done in a dedicated tool, a Google Doc, or anything in between. 

What is most critical, however, is getting the description and procedure right. 

A description should of course discuss the change being made. However, it must also acknowledge the potential risks. It may go as far as to link to a pre-mortem exercise run by the team, where the team performing the change has brainstormed all the ways the change could go wrong. This is critical information both for the person writing the procedure, but also for anyone reviewing the procedure for soundness. 

The procedure itself must include:

• An outline of the steps required to perform the change. Enough detail should be in the procedure such that one does not need to rely on outside resources in order to execute the change. If this is not possible, necessary outside links must be provided in the procedure. This allows everyone reviewing the document to be sure that their understanding of the steps to be performed matches what actually will be done. I also recommend breaking the steps into sections or “acts”, with clear breakpoints where the procedure can be safely paused (or must be paused). 

• Rollback instructions, preferably for each “act”. The rollback instructions provide details on the steps needed to move the system back to the previous state. This is a place where it is easy to get lazy, but the rollback instructions must be written with the same rigor as the procedure. (You’ll be glad this is the case if you’re actually executing the rollback instructions.)

• Observability. For section, it should be clear what signals will be monitored to check if the system is healthy or if a rollback must be performed. The procedure should include not only the signals to be monitored, but what healthy/unhealthy signals look like. 

Lastly, if the procedure that is about to be executed is new (and especially if it involves any bespoke tooling), it should be tested in a pre-production environment. Not only should the happy-path be tested, but any rollback procedures as well. The gold-standard for a rollback procedure would be one that you’ve tested in production.1 If possible, game-day failure scenarios in the rollout and ensure the rollback procedures handle them. A rollback option that hasn’t been tested may as well not exist. 

Observability

Observability will rarely be the cause of an issue, but the ability to notice when something has gone wrong and quickly identify what that something is can certainly mean the difference between minor and major impact. 

What can be done?

Modern tools and libraries can make creating extensive dashboards and alarms easy, but it is important to build with intention, and continuously refine the signals you’re looking at. 

Dashboards should “tell a story” about what is going on. The “so what” of every graph on a dashboard should be clear; if not self-evidently, then at least by a written description. Graphs which don’t provide a useful signal should be removed or at least demoted to a less visible location where they won’t confuse someone trying to understand the state of a service. 

There should be a strong signal to noise ratio. Developers should not become accustomed to graphs which “look bad” or behave erratically, even if the service is behaving as expected, because it will significantly lower the chance that someone notices a real issue in the future. 

Ideally, your tool should allow the ability to quickly dive into the data when a deviation is apparent. Error rates up? It should be possible to relatively quickly go into the logs of a failed request. Latency up? The faster you can get to a latency breakdown, the better. 

Alarms that actively alert operators must also have a high signal to noise ratio. Folks should not become accustomed to ignoring alarms. However, alarms must reliably alert when an actual issue occurs. 

I also strongly believe the user experience of the tooling you use to implement observability is critical. If creating a dashboard or alarms is difficult or confusing, folks are much less likely to create the right observability signals or keep them up to date.2

Lastly, always have a plan B for if your monitoring itself goes down. In the worst case, this may look like “SSH to some critical services and tail logs” but have something in mind, and most importantly, write down what you would do. 

Runbooks

Runbooks in this context are procedures written for responding to alarms. Like observability, good runbooks may not prevent something from going wrong, but they can certainly help stop things from going from bad to worse. 

What can be done?

All alerts to developers must be accompanied by a runbook. 

The most important trait of a good runbook is that it provides clear, unambiguous guidance for what to do. Remember, your runbook may be followed by a relatively inexperienced developer at 1 am.3 This doesn’t mean it is possible for a runbook to identify a remediation for every single possible type of failure, but it should at the very least provide clear guidance on where to find logs and metrics. If your runbook can get you to the bottom of the stacktrace or point you at the broken dependency and the contact for that dependency, that is pretty good. 

Having a template can reduce the friction of writing runbooks while also providing some consistency. While at Stripe, I worked with another developer to develop a template to use for runbooks going forward. We came up with the following sections:

• 1-2 sentence description of the detector and what it is telling you.

• 1-2 sentence description of the potential impact, so the responding operator understands the severity. 

• Temporary alerts or callouts about ongoing work, which may impact this detector. For example, “we’re currently rolling out feature x, which could trigger this alert. If this alert fires, start by turning off y feature flag.” These are not expected to be a permanent part of the runbook. 

• Troubleshooting steps. These are expected to be run in order. Each step is also written in a clear “if this, then that” format. We also used a visual language to clearly label links to logs and graphs, so that they’d stick out, and folks would not be surprised at where a link is taking them.

• Escalation instructions for if the runbook has not resolved the issue and next steps are not otherwise clear.

Lastly, schedule time (at least annually) for the team to review its runbooks and make updates to make sure all the information in them is still accurate. 

Incident reporting

By “incident reporting” I mean the process of documenting, reviewing, and sharing lessons learned from incidents. Every incident is an opportunity to identify new safety mechanisms that need to be built, practices that need to be changed, and so on. 

What can be done?

Establish a culture of following up on incidents, identifying causes, planning remediation work, and sharing out lessons. What this exactly looks like will depend on the existing institutions of a company. Establish a bar for what constitutes an incident, and what level of review is appropriate for that level of impact.

Create a template that captures the information that should be collected for each incident. A timeline and quantified impact are great for hard data, but I have found there is a large amount of value in creating space for a team to write a narrative about what happened through the eyes of the folks that worked on the incident. A “five whys” approach can be useful for unblocking thinking, but also has the risk of being applied too rigidly. 

Incident reviews should be blameless, in that the blame should not be laid at the feet of specific engineers or teams, but should rather strive to understand the human-computer interactions which led to the incident taking place (if applicable). 

Lastly, it is important to recognize the work that goes into writing an incident report and facilitating the review around it. 

At some future date, I will write a post about the operational and incident review process I developed for my organization at Stripe. 

Conclusion

I’ve covered a lot of different topics in these two posts, so before wrapping it up, I would like to discuss a few overarching themes that I’ve alluded to. 

Normalization of deviance

The greatest risk to any of the best practices I’ve mentioned is the normalization of deviance. In short, the normalization of deviance is when the habit of skirting or not following best practices becomes culturally acceptable within an organization. 

Some examples of normalizing deviance could be not creating or updating unit tests for a change, performing risky manual actions in production without following an organization’s change management processes, or creating detectors without runbooks. This is only a partial list; normalization of deviance from any practice is possible. 

If there is one act that will torpedo any attempts at operational rigor, it is a culture of normalizing deviance. 

The most effective antidote is to quickly identify cases where a process is not being followed and then either:

1. Explicitly and visibly prioritize diligence around wherever it is you’re slipping to bring the organization habits back in line.

2. Edit the process so that what is written on paper more closely aligns with what folks believe is the more appropriate level of rigor. 

Recommended further reading. 

Slow is smooth; smooth is fast; safety begets efficiency 

While the principal topic has been reducing the occurrence and severity of incidents, the suggestions in this post and the previous one are not just about safety and reliability. A company which has built tools and processes to execute safety is also one where engineers can execute with more confidence. 

This is most evident when it comes to testing: if an engineer can edit a code base to add/remove/change functionality and have confidence that the testing infrastructure will catch any unexpected side effects, that engineer can move their code through the pipeline faster and deploy with more confidence. Reviewers of the code can also focus their attention on the intended behavior of the change, knowing that tests largely have their back. 

Similarly, mature change management practices can remove the toil and uncertainty around rolling out a big manual change. Even if the process can feel bureaucratic to teams, at least it can be of a known quantity. A predetermined approval process can also reduce churn and allow a team to move forward confidently.

All of this is to say that proper investment in the areas discussed in these last two posts can yield not only decreased downtime, but also greater developer productivity. 

Further Reading

I want to take a moment to acknowledge that this post and the preceding one have only begun to touch on everything involved in building and maintaining resilient systems. I haven’t even touched on writing software and services to be more resilient in the first place! For that topic, there are many great books out there, of which I’ve read absolutely none, but I would probably start with the classic Designing Data-Intensive Applications.

For more on operating large scale systems, Google’s Site Reliability Engineering book remains a classic, though I will warn about considering it the final word on any topic.4

If you’ve made it this far, thank you for reading! I look forward to sharing more thoughts and tales from the trenches! 

Consuming a large amount of information about how systems fail is one of the greatest privileges of working at Stripe (and formerly in Amazon Web Services). “The greatest teacher, failure is,” as Yoda says. Both my current and former employer hold an incredibly high bar for how they operated, and as a result, there is a great amount of information available on even the smallest failures; little failures that happen every day but generally go unnoticed by our users. 

This year, I set a goal for myself to sit in as many of these incident discussions as possible to try to extract some common threads to share back to the organization. As I head out on the second part of my parental leave, I wanted to pause to share some of the observations I’ve made so far that are more generalizable and appropriate to share externally. 

The need for testing was a common theme, so I will devote a first post to just testing. Part 2 will cover the remaining learnings. 

Testing a change to make sure it will work as expected is evergreen advice, and the impact that lack of testing has should not be surprising. During this most recent period in my organization, a low-double-digit percentage of incidents could be attributed in some part to a lack of testing. I will highlight some of the specific types of issues I have observed. 

Testing in the code base

The simplest form of testing failure pertains to the lack of tests in the first place. A service may have no tests at all, tests may only cover a very limited number of code paths, or provide minimal assertions around the correctness of code. 

What can be done?

“Write more tests” is an intuitive response to address these issues. But it might also be the laziest possible remediation item in these cases because it ignores why these tests are lacking in the first place. If testing is not being properly performed by engineering teams, there is likely an underlying cause, such as the friction involved in adding use cases, or the perceived or actual utility of these tests.

For example, the following may add friction to proper testing:

• There are no frameworks for running tests at the appropriate depth (if a pattern is established for unit tests, it may not be easily extendable to functional/integration1 tests).

• There are so few existing tests engineers feel it is an unacceptable cost to create tests as a part of their planned work.

• The testing framework and overall testing strategy is not well documentend.

There may also be deficiencies in the utility of tests in the code base. This may be because:

• Tests exist but assertions are superficial. They may validate that the code executes, but critical assertions may not exist to validate the behavior of the code. 

• Useful mocks for dependencies do not exist, making it difficult to have confidence in the correct behavior of code which interfaces with dependencies. 

• Failing tests are not used to gate merges or deploys.

• Test failures do not correlate well with actual issues in the code (see the section on Testing Observability below.)

Fixing testing deficiencies is not a trivial matter. For testing infrastructure to be useful (and used) it must be (and recognized to be) a tool that makes a task simpler.2 It requires investment the same way we invest in deploy and orchestration tooling (and quite possibly much more investment). This is all the more difficult in environments, such as Stripe, that require synthetic or “mock” third-party components to test against, and even more so if the ownership is not clear around who should create those mocks.3

A few approaches I have seen be useful in improving test coverage over time:

• Standardizing the testing frameworks and patterns used within a code base to reduce the cognitive load of writing new tests. 

• Requiring minimal test coverage for a change before merging code.

• Building tools which allow engineers to visually inspect test coverage for the code they’re working with. (This makes it more obvious where coverage is missing.)

I would avoid anything which makes individual engineers feel responsible for the overall code coverage for a service. This is a valuable metric for leadership to track, but is difficult for engineers to address while delivering other work. It is much more effective to show opportunities for improvement in the context of existing work. 

Testing at scale

Beyond basic functional testing, testing at scale is an additional challenge. It is particularly problematic for infrastructure teams and services that maintain stateful components. 

Periodic load testing where a call path is pushed to its limits is a critical piece of engineering for reliability. However, this can be very challenging to do exhaustively. Call patterns in production may not be easy to replicate.4 Load can also come from surprising places, especially for infrastructure teams. For example, infrastructure teams might face a thundering herd to its orchestration systems when another team attempts to scale up to mitigate its own issues.

What can be done?

Even though load testing is not a panacea, it still plays a valuable role in the overall operational readiness of a service. It is all the more effective when paired with a diligent analysis of real life use cases and pathologically bad, worst case scenarios. That is to say, testing is not as simple as pointing a load test against an API and writing down the Requests Per Second (RPS). A thorough load test should take into account:

• Current, actual customer behavior, so that the load test will accurately replicate the type of load currently placed on the system. 

• Worst-case use cases which are not commonly seen, but would stress the system. It is important to understand where rate limits and load shedding may be needed to protect the service. 

• Future business needs, in case the current customer behavior is subject to change. 

• Whether real dependencies of the service will be used in the test (and if not, an honest description of the limited usefulness of the result).

• How components behave differently under synthetic traffic versus real-life usage patterns. For example, caching (both explicitly used by the service or used implicitly by dependencies such as databases) may behave very differently depending on how synthetic traffic is constructed. 

A thorough load test should report on:

• The maximum throughput a service can handle while maintaining latency and availability targets. 

• The component or system which limits performance, so that it is clear where future investments would need to be directed in order to improve performance. 

Lastly, teams should take action on the outputs from their load tests:

• Teams should create alerts to notify them when the current load approaches the theoretical maximums. The greater the uncertainty about the theoretical maximum, or the greater the difficulty in raising the maximum, the lower the alerting threshold should be. Reviewing these metrics on a weekly basis can also provide further advance notice of potential scaling issues. 

• If appropriate, service owners should set rate limiters to protect services based on known maximum supportable throughput. 

Load tests should be performed on a regular cadence throughout the year (especially in anticipation of periods of high load), as well as after architectural changes.

Test in the right environments

Where to test can pose an interesting challenge. Commonly, a shared QA/Beta stage is established in which everyone is supposed to deploy code and perform validation of changes before deploying to production. The result is an environment that is frequently broken because it contains changes being tested that do not work as such. This can make it difficult for an engineer to be confident about any single change. Breakages can become so frequent that engineering teams become accustomed to ignoring signals from errors in this environment.

This can be especially problematic for infrastructure teams. Engineers that rely on stable deployment tooling to be productive with their own tests in QA. They may not appreciate being beta testers for new versions of tools that impact their productivity.

What can be done?

I have seen two solutions in practice. 

The first is to have a series of testing environments, where the quality ratchets up as you go along. While the first environment in the pipeline may be a free-for-all, the final environment before production is expected to be more stable and provide a high signal-to-noise ratio. In AWS I saw as many as 4 stages before production. 

The other approach I have seen (sometimes used in conjunction with the first) is to have separate testing environments per team or organization. An infrastructure team may have a QA environment, while other teams’ QA environments would use the production infrastructure and dependencies. This isolates teams from each others’ testing.5

In cases where it is not appropriate to use a production dependency for a test environment, consider a post-production environment (deployed to after production in the deployment pipelines) which exists for teams to test their candidate code against. 

Another option (which I have not seen completely implemented but have always wanted to) is to provide engineers a way to spin up bespoke testing environments which are effectively a clone of the production state of the world with just the component to be tested replaced with a candidate.

Testing observability

Lastly, for testing to be valuable, there must be confidence in the signal provided by a testing environment. Unit and integration tests must not be flaky such that engineers are used to ignoring failures. If a pre-production or staging environment is used, alerts must be in place to notice that a potentially breaking change has been introduced before it goes into production. And those alerts must again have a high signal to noise ratio such that the warnings they provide are not ignored. 

What can be done?

I will devote more in Part 2 about avoiding the normalization of deviance. For now, suffice it to say that any system which is used to gate code being deployed to production must be treated as production in terms of quality. This means having similar alerting as production, and an urgency to fix issues that is second only to an actual production issue. 

Engineers must also be given adequate alternatives for testing such that the final validation stage is not the first time when their code is being exercised. Otherwise, the constant need for urgent fixes to address recurring breakages will not be sustainable. 

Conclusion

This concludes my thoughts and observations on testing. While my recent observation of Stripe infrastructure incidents provided the inspiration for writing this essay, the experience I draw from is much greater and indeed encompasses my software development career to date. 

Please join me for Part 2, where I will cover other common contributors to incidents, as well as some other recommended best practices. 

Thank you to Yanjie Niu for editing this post and providing thoughtful feedback.

Subscribe now

For those of you who have been in the industry for a while, this probably won’t be very interesting. However, if you’ve joined since things like Slack and Zoom became commonplace, it might be a little more novel.

I joined Amazon in 2010 after graduating. Back then we didn’t have Zoom or Slack. We didn’t have Microsoft Teams. We didn’t even had Skype for Business. We had Microsoft Office Communicator. It looked a bit like the old MSN Messenger, and no one really cared for it.

But this isn’t a story about that. This is a story about:

Back when I joined Amazon, everyone was issued one of these.

Since we didn’t have Zoom (or whatever), it wasn’t uncommon when you wanted to have a synchronous conversation with an engineer in a different location, that you would do this:

1. Find their name on in internal website called the phone tool

2. Find a 5 digit extension on their page

3. Push physical buttons one at a time

4. Pick up a physical handset and proceed to hold it against your ear like a cave person for the next 30 mins or whatever.

Our phone system also supported voicemail, so sometimes you’d come back to your desk and there would be a flashing red light indicating you had a voicemail.

To listen to it, you’d again have to push a series of buttons, enter your pin, and LISTEN TO THE MESSAGES ONE AT A TIME just like in medieval times.

But all of that is just background

This isn’t a story about phones. This is a story about my phone. Or specifically, my extension.

Every employee was allocated an extension out of a (at the time) 4 digit namespace, I believe randomly.

My extension was: 0...0...0...1

That’s right. Somehow they gave me, a lowly SDE 1, extension number 1.1 Again, I believe this was completely random.

It is also important to know that all Amazon extensions mapped to a full 10 digit number, which was done by prefixing a the area code (206) plus the first 3 digits (let’s say 266 in this case, because that’s what it was). That made my phone number 206-266-0001.

Surprisingly, this did not lead to me getting lots of spam calls, as spammers crawled the Amazon corporate phone namespace.

It did however, lead to some confusion with Amazon’s main corporate phone number: 206-266-1000.

Me: “Hello?”

Them: “Uh…hello. Is this…Amazon?”

Me: “Yes, I am Amazon. Speaking?”2

I did have people call me trying to return things. Or sue me. However, all in all, it was surprisingly uncommon.

But wait, there’s more…

It wasn’t just the public that would be confused. My extension also led to some internal confusion.

On one occasion, I was hard coded as the test phone number for some automated test suite, because they thought why would extension 1 be actually allocated to anyone.

A more interesting time, for a few weeks, every time I came to the office my phone had several voicemails in Mandarin, from other Amazon extensions. This one took a while to sort out because I do not speak Mandarin. It turns out some internal wires were crossed and for a short period I had become the IT help desk for one of the offices in Beijing.

Conclusion

One of the hard parts of leaving Amazon to join Stripe was knowing that I was giving up this extension. I actually had the same desk phone for my almost 10 years at the company, and continued to use it periodically. (Most often, to call candidates during phone interviews.)

Because it was a VoIP phone, the phone (and my extension) followed me to wherever there was a PoE Ethernet jack available.

Unfortunately, I was not allowed to take it with me to Stripe.

I’ll miss you little Polycom.

Dr. Seuss's book, How the Grinch Stole Christmas!, is a cornerstone of both children's literature and Christmas lore. The book was originally published in 1957, but has since then spawned three different movies, each with their own interpretation of the original story. In this essay, I will walk you through a comparative analysis of the three different interpretations. I hope in the end I will convince you that the 2018 film, The Grinch by Illumination, is the superior Grinch adaptation.

My daughter Amelia has been a huge Grinch fan since she first read an abbreviated board-book version of the story. I have since then had ample interaction with that adaptation, the original picture-book version, and several different movie interpretations. I have also since then formed some strong opinions on the merits of each adaptation; specifically how each tackles the Grinch's backstory and the resolution of the conflict in the main plot. Excluded from this analysis is any consideration of how well any of the movies are executed upon as movies. That's a job best left up to the critics and reasonable persons may disagree on this point.

At a high level, all three movies follow the same basic structure. We learn of a people called the Whos, who live in Whoville. Christmas is culturally a very important holiday for them and great effort is expended in celebrating it. We are also introduced to a green, hairy creature called the Grinch who lives on Mount Crumpet, just north of Whoville. The Grinch does not like Christmas and despises the festivities. He resolves to stop Christmas from happening by donning a Santa Claus outfit and stealing all the gifts and decorations while the Whos sleep. On Christmas morning, as he is about to destroy his loot by pushing it off his mountain, he becomes aware of the Whos singing (and celebrating) nonetheless. Christmas comes all the same. This causes the Grinch to have a change of heart, understand the true meaning of Christmas, and return to Whoville to join in the festivities.

Diving in, the first movie adaptation was a 1966 direct-to-television adaptation of the book by CBS. This adaptation is very true to the original source material, both in terms of the animation style and plot (with only some minor, inconsequential deviations and room made for some musical numbers). For that reason, it also serves as a good baseline for my analysis. [1]

Like the book, the 1966 movie provides scant details on the origins of the Grinch. All that we know is he lives on Mount Crumpet, he hates the Whos, and most of all he hates Christmas. The narrator admits the reasons for this are unknown, and only suggests that most likely reason might be because "his heart is two sizes too small." The movie makes no effort to explain further. The two movie adaptations that followed were clearly not satisfied to leave it at that.

The 2000 film How the Grinch Stole Christmas by Universal Pictures does lead with the fact that the Grinch's heart is incorrectly sized, but the movie establishes a much more complex backstory for the character. The Grinch arrived in Whoville as an orphan. He was adopted and lovingly cared for by two women. However, in school is repeatedly bullied for his different appearance. While attempting to use Christmas to win the affections of his young love interest, further bullying by his classmates pushes him over the edge. He declares his hatred for Christmas and runs away from town to live as a recluse on Mount Crumpet, where he has been presumably living up to the present day.

Arguably the 2000 version provides the most consistent backstory for the Grinch (with the caveat that no adaptation offers an explanation for where the Grinch, adult or baby, really came from). The backstory explains both his disdain for the Whos, his loathing of the Christmas season, and his feelings toward presents.

The 2018 adaptation, The Grinch, by Illumination takes a similar approach as the 2000 version, with a slightly lighter touch. References are made to the Grinch living in an orphanage, feeling alone, and being left out of the Christmas activities. (He actually appears to be the only child in a very large orphanage, which is both confusing and sad.) No further details are provided on when or how he made it to his mountain retreat; and as for why, we're left to assume he never felt welcome in Who society.

The movies differ more in their depiction of the present day. The original 1966 version is again scant on details. We never actually see Who society outside of the Grinch's night time escapades and his eventual reconciliation with them at the end. The Grinch has some visions of how they celebrate the holiday, but it is not a very holistic view, and it isn't clear if this is reality or how he imagines it. What we do see are brief snapshots of festivities; children opening gifts, eating a feast, and lastly (and the least tolerated by the Grinch) singing carols are a group. They sing, and sing, and sing, and the more the Grinch thinks about it, the more he resolves to stop the whole thing. Lastly, it isn't clear how the Whos see the Grinch, or if they are even aware of him, until the end.

The 2000 film elaborates significantly on Whoville in the present day. The Who culture (at least as it pertains to Christmas) is deeply materialistic. Anticipation of Christmas revolves around shopping and decorating, with a significant flavor of competitions between Who families. The Whos are anxious about purchasing the right gifts, and how their outward appearances are perceived by other Whos.

In the 2000 film, we also see the relationship between the Grinch and the present-day Whos. He is generally feared and reviled by the Whos. Teenagers dare each other to visit his cave (where he is more than happy to play up his own infamy). He visits Whoville in disguise, and when he is recognized by a young girl, she reacts in fear. He does have a mailbox at the Whoville post office, but Whos remark with disdain that there are never any Christmas cards coming and going. When little Cindy Lou Who suggests that the Grinch be made the Holiday Cheermeister (because he most of all is in need of cheer), she knows it is a drastic act and it is not one without controversy. (The Grinch ultimately agrees to fulfill this role, and is briefly won over by the celebrations, but is ultimately reminded of past grievances, which is what motivates him to "steal" Christmas in this film.)

The 2018 film has a very different take; one that is much more optimistic. In the 2018 Whoville, we still see a Whoville that is infatuated with the Christmas season, and all the decorations and gifts it brings. It does, perhaps, at times even suffer from glimmers of the same tendency toward excess (e.g. the tree must be ten times bigger than last year). However, on the whole, the movie depicts a town which is much more balanced. The Whos are kind and cooperative. There is no competition. There is no one attempting to out do their neighbors.

As in the 2000 film, in the 2018 film the Grinch is shown interacting with the Whos. However, in the more recent film, the Whos do not appear to be afraid of the Grinch. He is greeted warmly and invited to participate in the festivities; it is only by his own and continual choice that he is excluded. The 2018 film goes so far as to depict a Who, living on the outskirts of town, that considers the Grinch to be his best friend (although the feeling does not appear to be mutual.)

To summarize, in the 2000 film, (regardless of who is to blame) there is mutual animosity between the Grinch and the citizens of Whoville, while in the 2018 film, the animosity is only in the Grinch's head.

Up to this point, both movies still present a coherent view (though one more negative and one more positive). The difference between the Grinch and Whoville interactions in the two later films, however, is crucial when evaluating how the story resolves in each.

This brings us to the main plot. For reasons which vary slightly between the films, the Grinch resolves to "steal" Christmas. In every movie, he dresses as Santa, sneaks into town at night, and takes all the gifts and decorations.

The 1966 and 2000 versions have the Grinch about to destroy his loot, when he hears the Whos singing. He realizes that Christmas came without "ribbons, tags, packages, boxes, or bags" and that therefore Christmas must mean something a little bit more. Upon this revelation, his heart grows three times in size, and he returns to Whoville with everything he stole. He is forgiven and joins in the festivities.

My main objection to the two early versions (and the gripe that motivated me to write this essay) is that his revelation is a complete non sequitur. At no point in either of the early two versions does it hint at the fact that the Grinch was upset or bitter because he thought Christmas was a purely materialistic holiday. Even in the 2000 film, where the Grinch is vocally critical of the materialistic nature of Who society, it was hardly his reason for becoming a recluse in the first place (which has more to do with how he was treated by his peers), so it isn't clear why this is the revelation that softens his heart.

The 2018 version of the film, however, provides a much more satisfying version of the ending, and it is preceded by an element that is unique to this film.

In the two earlier films, as in the book, the Grinch's efforts to steal Christmas are almost foiled when he is interrupted by a young Who, little Cindy Lou Who (who in the book and first movie, is no more than two, but is a bit older in the later two films). She asks why Santa is stealing the tree and he explains he is taking it back to his workshop to repair it.

In the more recent film, the pair have a more in depth interaction, where Cindy asks the Grinch (who she again believes to be Santa) if he can offer some reprieve to her overworked mother. Cindy also explains how happy and at peace she feels when the Whos all sing together on Christmas morning.

"Everyone should be happy, right?...I wish you could celebrate with us tomorrow. We all get together and sing. It's so beautiful that if you close your eyes and listen, all of your sadness just goes away."

This has no immediate impact on the Grinch's plot, but plants a seed for later.

In the 2018 film, the Grinch isn't swayed because he realizes stealing the gifts had no effect. His heart grows because he finally lets go of the animosity which lived inside himself.

As in the previous films, he climbs to the top of Mount Crumpet to dispose of the gifts, he hears the Whos singing. He looks through a spyglass and sees little Cindy singing with the rest of the Whos. However, it isn't some realization as to the meaning of Christmas that sways him. Rather, it is his decision in that moment to take a leap of faith in the little Who's advice:

As he watched the small girl, he thought he might melt.

If he did what she did, would he feel what she felt?

The luscious sound swelled, reaching up to the skies.

And the Grinch heard with his heart, and it tripled in size. [2]

As in the other films, he returns the gifts. But in this film, rather than immediately joining in the holiday celebrations, he meekly returns back to his lair. The final resolution comes when Cindy herself explains he has been forgiven and extends the invitation for him to celebrate with them, further confirming there was never really excluded.

In this version the materialism of the Whos and the attack upon it was tangential. The Grinch saw, despite his efforts to ruin the material aspects of Christmas, the Whos were still happy. His salvation came not from recognizing the true spirit of Christmas, but accepting an invitation to a moment of vulnerability. The Grinch experiences that which has been there and available to him all along.

This is a much better ending. In this latest film by Illumination, we are finally given a Grinch story with both a consistent back story and an ending that is satisfying both in the traditional sense, and also in how it reconciles with the conflict introduced earlier in the story.

In conclusion, the 2018 film The Grinch by Illumination is the only retelling of the traditional Grinchian lore that offers a consistent, intact plot, and a satisfactory ending. The original story, and 1966 movie adaptation, while no doubt classics, are riddled with inconsistencies. Both the 2000 and 2018 films get credit for providing credible backstories which at least provide a hint of character motivation. But it is only the 2018 film, over 50 years later, which finally gives us the fleshed out Grinch and Whoville universe that we have deserved all this time.

[1] It's also worth saying that if your personal criteria for "best" is "true to the original source," you can stop reading here because there is absolutely no contest on that point and I won't try to convince you otherwise.

[2] I've avoided remarking on the aesthetics of these films, but this is by far my favorite part of any Grinch film. I also appreciate Illumination's commitment in the latest film to writing new content in the Seussian style.