Today, I want to share some of the findings, and in particular which changes I introduced to SCOPE because of the insights from the study.

SCOPE: A new Review Model for the AI era

SCOPE shifts review away from one code review phase after implementation and towards distributed change oversight that happens at several stages and by several people. In SCOPE, not every change gets the same amount of oversight, and it also makes those oversight decisions on how much oversight a change gets explicit and team-informed through Proportionate Escalation.

Those two concepts behind SCOPE - Staged Change Oversight and Proportionate Escalation - proved stable and applicable in many different business and engineering contexts within the study. At the same time, the study also showed where the model needed to evolve.

Review Starts before Code Exists

SCOPE already recognized in its first version that review starts before implementation, but I did not capture this as a dedicated review stage. This is probably the biggest shift within the model introduced by the empirical findings.

Reviewing artifacts like architectural documents and design documents, or refining the ticket, was important even before AI, but it was not an official part of code review. Yet, many problems experienced during code reviews had to do with shortcomings during planning and could be fixed or improved by rethinking and reorganizing the coordination of the work that happened before implementation.

AI makes this even more crucial, as the distinction between planning and implementation blurs. When machines take over more of the writing of the code, and developers increasingly instruct machines in plain English, we have to ask ourselves: what is the work we review and assess, and when should we do so?

SCOPE’s answer is that review is a staged activity that starts before implementation, to reduce rework costs and build team consensus early on, and carries on after implementation. SCOPE therefore has four stages that a change can run through: Initial SCOPE Review, Agent Review, Developer Review, and Independent Review.

Share

Some Changes Need Significantly Less Oversight

Based on the insights from the study, SCOPE is now more radical in allowing and anticipating stages to be skipped. This is a natural next step to account for the many different settings and stages of AI adoption, but also the different business contexts and goals. The stages in SCOPE are now building blocks that can be used to ensure the right oversight per change. In some contexts, and for some well-understood and low-risk changes, an Initial SCOPE Review followed by an Agent Review can be enough. In other cases, a change might need to go through all four stages. In addition, proportionate oversight does not only refer to how many review stages are present, but also influences their intensity.

SCOPE: A Concise Introduction

Finally, I also got feedback that the original post detailing SCOPE was too long. So I wrote up the SCOPE model in a concise way to allow you to absorb the underlying ideas and concepts more quickly.

Have a look at the new to-the-point write-up here. If you are looking for the deep-dive version of the SCOPE model, then this post is for you.

Let me know what you think! I appreciate your feedback. I have so many more interesting study findings, and I will share more about them with you soon.

Interesting Reads

There is more to code review than automated detection by John Allspaw: In this post, John looks at an unpublished paper that argues that code review can be completely replaced by LLMs, and shows several functions of code review that cannot be replaced by a machine. He argues that code review is also a coordination, a sensemaking, and a governance process, and I could not agree more.

Whiteboard - An Open Source IDE: I came across this open-source tool that focuses on building a place where humans and agents can architect software together. What is cool is that you can read explanations of the code while also being able to look at the code at the same time. In my opinion, it is an excellent way to review, understand, and think about your software together with developer peers, while having the code at your fingertips, but not at the forefront.

Well, that is it for now. If you read the SCOPE article, please send me your thoughts! I really appreciate them.

Michaela

Written by me, not by AI.

Want to go deeper on code reviews?
Find practical guides, workshops, and resources at awesomecodereviews.com.

Thanks for reading Awesome Code Reviews! Subscribe for free to receive new posts and support my work.

AI-generated code comes with many challenges. Giving up on code review could be one option, but only if agents really become that good that we would never ever have to look at code anymore. Right now, although GenAI is bafflingly good at times, we are far from this. Architectural drift, wrong intent, introduced technical debt, plainly wrong code, and major incidents are all likely if you let your agents loose without strong oversight.

Yet, the role of code has definitely already shifted. I see it in my own work, but also when working with engineering teams or conducting research interviews. Engineers work at a higher level of abstraction for more of their time. But not always.

Because of all those shifts, I do believe we have to redesign code reviews to adapt to those changes, and so I started working on a new code review model. I call it SCOPE: Staged Code Oversight with Proportional Escalation.

The basic idea behind SCOPE

SCOPE goes from code review to code oversight. It shifts from a pure peer practice to a multi-stage practice, where everyone involved has a clear role and accountability. And the amount of code review and oversight increases or decreases depending on understanding needs, risk, and the assurance already established.

Sounds complicated? Only until you get the hang of it.

In a nutshell, SCOPE has three stages, each with a different focus:

Agent Review → Developer Review → Peer Review

1. Agent Review

Before asking another developer to inspect generated code, we should use automation as much as reasonably possible.

That includes traditional checks such as tests, static analysis, linters, and security tooling, but increasingly also AI-based reviewers that can inspect changes, identify suspicious areas, compare an implementation against its intent, or challenge the producing agent.

But successful automation is not proof that a change is correct.

The important question is what evidence these systems have produced, how trustworthy that evidence is, and how much confidence it should give us for this particular change.

2. Developer Review

The first human reviewer in SCOPE is usually the developer who has been steering the agent or has adopted the agent's work.

The steering developer must inspect the work deeply enough to take it over as their own.

This does not necessarily mean reading every generated line with the same level of attention. How deeply the developer needs to inspect the implementation depends on what we are changing, how difficult it is to understand, what could happen if we are wrong, and how much confidence we can reasonably place in the evidence already available.

3. Peer Review

Peers then provide something the steering developer and their agents cannot provide by themselves: an independent perspective.

They can challenge assumptions, notice implications the developer missed, bring knowledge from other parts of the system, and build enough understanding of important changes to maintain shared knowledge within the team.

Peers don’t redo the code review from stages 1 and 2. Their involvement again depends heavily on risk, understanding needs, and established assurance.

This could mean that for some changes, the peers only look at the rationale, the available evidence, and important decisions. For other changes, they might perform a detailed code review for selected parts of the change.

Proportional Escalation

This change in focus and depth is reflected in the proportional escalation part of SCOPE and is also the heart of SCOPE.

Proportional escalation means concentrating human attention where risk, understanding needs, or insufficient assurance make independent scrutiny most valuable, rather than reviewing every change and every line of code equally deeply.

Escalation in SCOPE is not limited to reviewing more code. It can also mean involving additional or more specialized reviewers, breaking work into smaller reviewable steps, or adding earlier review points before implementation proceeds.

You can read the more detailed version of SCOPE here.

Feedback on SCOPE

SCOPE is a candidate model. This means I am actively asking for feedback, and I’m also conducting a study to better understand its strengths and weaknesses, its limitations, and where it fits, or does not fit, different industry settings.

And I would really appreciate your feedback. I’d love to hear which parts resonate with you, how it would or would not fit your team’s software development life cycle, whether parts seem difficult to apply in practice, or which limitations you see. You can send me an email or use the short survey linked in the article.

Interesting Reads

• Model-Based Agentic Software Engineering (MAGE), by James C. Davis: James reached out to me because of SCOPE and shared with me his introduction to MAGE and an extensive online book about model-based agentic SE. One of his central arguments is that as implementation becomes cheaper, engineering judgment becomes comparatively scarcer. MAGE puts a strong focus on making engineering knowledge and intent explicit through models, and on turning recurring obligations into constraints, validators, gates, and other mechanisms that can be evaluated independently of the agent producing the code. This distinction between producing work and establishing sufficient confidence in the work is also important in SCOPE. I see the developments around safeguards and harnesses for agents going in a similar direction to MAGE.

• From Code Review to Code Critique by Chandra Maddila et al.: In this recent preprint, they show an interesting mismatch between what human reviewers focus on and what current AI reviewers tend to comment on. Their automated solution tries to infer the intent behind a change, detect whether the implementation has drifted from that intent, and identify a small number of regions in the diff that most deserve human attention. This is important for code review, because, as human attention becomes a scarce resource, it’s less about whether we review all code and more about deciding where human attention has the highest value.

• Code Review Tool "Ref Plans”: New tools for code review are appearing everywhere. Many of them are still experimental, so I normally don’t write much about individual tools. But Ref Plans caught my attention because it moves part of the review process earlier. The idea is simple: review the plan before the diff. Of course, reviewing specifications, designs, and implementation plans is not a new concept. Many organizations already do this for larger changes. What I find interesting here is that the tool applies this idea directly to the new developer-agent workflow.

Well, that is it for now. If you read the SCOPE article, please send me your thoughts! I really appreciate them.

Michaela

Subscribe now

Want to go deeper on code reviews?
Find practical guides, workshops, and resources at awesomecodereviews.com.

Over the last half-year, I have become deeply involved in code review research again. I have studied code review for many years, but agentic programming is changing some of its underlying assumptions. Code can now be produced at a speed and volume that was previously impossible, while the human capacity to understand, verify, and take responsibility for it has not increased at the same rate.

This raises questions that I find genuinely fascinating: Which parts of traditional code review still serve us well? Where is the process beginning to break down? And how can teams preserve quality and shared understanding without turning human review into the bottleneck for every AI-generated change?

These questions led me to work on a new code review operating model, and I'm currently engaging with several companies to evaluate, refine and improve it.

I will share what I learn along the way. There may also be opportunities for teams and individual developers to participate in the research, and I will provide more information about that soon.

A new course: Code Reviews in the AI Era

This research is also shaping a new course I am developing on code reviews in the AI era. The goal is to translate this emerging model into practical and actionable guidance for teams.

The course will explore how teams can benefit from AI-assisted and agentic development without giving up the quality, shared understanding, and responsible engineering decisions that code review is meant to support.

You can sign up on Awesome Code Reviews to be notified when the first materials become available. Many of the resources will be free, with optional paid material for individuals and teams who want to go deeper.

An Awesome Code Reviews Makeover

I also took on a big endeavour: I redesigned and reorganised awesomecodereviews.com.

I refreshed many of the existing articles (and I am still working through others), improved the navigation, and started transforming the site into a proper knowledge hub that has a large collection of free resources to improve code reviews.

The previous version contained a lot of valuable material, but it had become difficult to explore, and some of the material was rather outdated.

The new structure should make it easier to find practical guidance, research, and learning resources depending on what you want to understand or improve.

It will also become the main place where I share my current work on how agentic programming is changing code review. I already have several new articles in the pipeline, including the deeper pieces on code review surrender and code review exploitation, the concepts I outlined in the last issue. Since I started talking about these ideas, several developers and engineering leaders have told me they recognise these patterns from their own work.

I would be very happy if you had a look at the knowledge hub and let me know what you think!

Interesting resources

• Better tools made Copilot code review worse by GitHub: This article talks about migrating Copilot code review to new repository-exploration tools. Surprisingly, the supposedly better tools initially increased costs while causing the reviewer to detect fewer useful issues. The agent explored the repository too broadly instead of investigating the change itself. GitHub improved the results by redesigning the instructions around a review-specific workflow: start with the diff, formulate concrete review questions, and retrieve only the evidence needed to answer them.

• The Productivity-Experience Paradox by Annie Vella: Annie describes an interesting tension arising with GenAI: developers may feel more productive while parts of their developer experience, especially flow, gradually deteriorate. She also raises a deeper concern that AI could erode some of the joy, craft, and lived experience of being a software engineer. I personally experience this differently. For me, there has probably never been a more exciting time to be a software engineer. I enjoy the new possibilities enormously. At the same time, I see many of the problems Annie identifies. Increased output does not automatically mean better work, better systems, or a better experience for the humans responsible for them. Her article offers a thoughtful perspective on that tension.

• From Technical Debt to Triple Debt by Tech Lead Journal with Margaret-Anne Storey: This podcast explores what happens when teams produce software faster while gradually losing their understanding of the system and the reasoning behind important decisions. The episode also covers cognitive surrender, developer fatigue when supervising agents, and the idea of introducing deliberate “strategic friction” to preserve understanding. These questions are closely connected to the challenges I am currently exploring around code review: who understands a change, who verifies it, and where responsibility sits when much of the implementation was produced by an agent.

• MoTaCon 2026 is around the corner: Finally, I just saw what my friend Rosie Sherry and the Ministry of Testing community are putting together for MoTaCon 2026. MoTaCon takes place on 1 October 2026 at the Brighton Dome and covers quality engineering, AI, leadership, product, accessibility, and security. One particularly interesting aspect is that attendance is included with an active Professional, Team, or Unlimited Ministry of Testing membership. Instead of purchasing a separate conference ticket, members can register for the event while also receiving the other membership benefits.

As always, please feel free to reach out to me! I love to engage with people over my content and to hear your perspectives and ideas!

Best,
Michaela

Subscribe now

Want to go deeper on code reviews?
Find practical guides, workshops, and resources at awesomecodereviews.com.

Code review surrender is a capacity failure. Reviewers still believe review matters, but cannot keep up with the growing volume, pace, and cognitive demands of agentic code changes. Reviews become delayed, superficial, skipped, or quietly abandoned.

Code review exploitation is a responsibility failure. Reviewers become, or feel like, the first serious validators of a change because author-side verification is absent, invisible, or difficult to trust.

Both phenomena are related, but they are not the same problem.

Surrender is not a moral failure by authors or reviewers. It can happen even in strong teams with responsible engineers. But simply removing review because the queue is unmanageable is not transformation.

Exploitation is not necessarily a malicious activity. Sometimes it is caused by an inexperienced or non-technical author who creates a heavy review burden by using GenAI. The feeling of exploitation can also be aggravated by a team or company culture that values code production more than quality assurance.

You can deep dive into both failure modes in this new post.

Cognitive Debt in GenAI Driven Development

Another topic I'm actively working on is Cognitive Debt. Cognitive Debt was described by Margaret-Anne Storey as: “the accumulated gap between a system’s evolving structure and a team’s shared understanding of how and why that system works and can be changed over time.”

Over the last 4 months, I have been doing an experiment where I’m developing a software system with the help of GenAI while not looking at the code. Agents do all the work, and I try to explain and design the system with natural language via prompts and specifications. I built harnesses, made sure the agents have good instructions, skills, and feedback loops that give them insights into the system’s quality, such as tech debt.

And I’m feeling how cognitive debt builds up. The way I specify and the way the system and implementation turn out are not completely aligned. I can’t tell for sure how they diverge because I’m not looking at the code. But I know. I see it in failures, in behaviors that are not intended, even in files that are touched during a change.

Yes, maybe the problem is that I’m a bad specifier, but I ask: who is not? Haven’t engineers struggled with specifying for as long as I can think? Haven’t we started to work iteratively because thinking through everything upfront is just super hard, or at least for me, it is almost mission impossible?

I’m currently in the process of writing up my experience with this experiment. And I’m also looking more closely at how teams and engineers experience cognitive debt especially in the absence of code reviews.

If you want to share your experience with me, please get in touch or just write to me at michaela@michaelagreiler.com.

Interesting resources

• The high cost of AI immaturity by LeadDev: Recently, I was on a panel to talk about AI adoption and especially the dos and don'ts when it comes to adopting AI. It was an interesting discussion, and we touched on the topic of code reviews (starting at 17:15), where I briefly share my current thinking around code reviews. I talk about the agent-dev review cycle, AI reviews, and why slowing down is not always a bad thing.

• Agentic Code Review by Addy Osmani: Addy writes about the problem we all face: human capacity is limited, and code review is now the bottleneck because the hard part of engineering has moved to verification. He gives some practical tips, such as using risk to know where to spend human attention, keeping code changes small, reviewing tests more carefully than code, and ensuring that humans are still held responsible for the changes.

• Just One More Prompt by Leif Singer: Leif talks about how working with LLMs can make you "addicted" to one more prompt. But he also touches on cognitive debt. He makes the link that, as building is now so easy and so fun, he builds more, faster. But really engaging with the problem, taking the time to build a mental model, that is still hard. And as building is easier and gives you more dopamine, we might just build new stuff, create more code, instead of really understanding or polishing what we already have. Relatable?

Well, that is it for now. I’m currently also writing a large piece about my view on code reviews, and I hope I can share it with you soon.

As always, please feel free to reach out to me! I love to engage with people over my content and to hear your perspectives and ideas!

Michaela

Subscribe now

Want to go deeper on code reviews?
Find practical guides, workshops, and resources at awesomecodereviews.com.

For three days, we talked about how AI is actually changing software engineering, and what is relevant to understand, explore, and study.

For me, one of the best parts was reconnecting with previous colleagues and meeting many new inspiring people — all world-leading researchers. What better place to listen, learn, and discuss.

I had great conversations about the future of the developer profession, developer identity and new career paths, and how we prepare students and junior developers. Of course, we also discussed research insights and empirical findings on AI’s impact on software quality, testing, and code reviews.

Another topic that kept coming up was cognitive debt and intent debt, and how teams struggle to understand the systems they build when more and more code is AI-generated.

I wrote down the discussions that interested me most in this blog post called: What 40 Software Engineering Researchers Talk About When They Talk About AI.

Reads of the Month:

Coding Is Dead (…But It Still Smells Funny) by Adam Tornhill: I met Adam during my trip to Copenhagen and talked with him about the future of software engineering. He pointed me towards his article, where he argues that yes, AI fundamentally changes our roles as developers, but writing software, especially production-ready software, won’t become trivial because of this. Yes, generating prototypes is now easy, but maintaining reliable, evolving production systems remains hard. He also demoed CodeScene and showed empirical results indicating that code quality impacts AI performance: high-quality code leads to better AI-generated code and also reduces the tokens you spend. His advice: improve your codebase health before using AI to generate more code. You can even use AI to improve the codebase first, but make sure AI knows what “better" means, for example through concrete measurements.

How Claude Code works in large codebases by Anthropic: This week, Anthropic published an excellent piece about how to make Claude Code work for large codebases. While it is specific to Claude, some of the pain points and misconceptions, as well as the strategies, are applicable to other setups as well. The main point of the article is that the performance you see does not solely depend on the underlying model, but is significantly influenced by the “harness”: that could be CLAUDE.md, AGENTS.md, or GEMINI.md files; hooks; skills; plugins; MCP servers; LSP integrations; and so on. What was also clear from the article is that configuring and working with AI is not trivial, and current best practices are changing constantly. What worked well three months ago could now negatively impact your experience.

Spec Kit: Build high-quality software faster by Github: GitHub’s Spec Kit is an open-source toolkit for “spec-driven development,” where specifications become the central artifact that guides and generates implementation work, instead of being discarded once coding starts. The workflow starts by defining project principles, then writing a feature spec, creating a technical plan, breaking it into tasks, and letting an AI coding agent implement those tasks. It provides CLI tooling and agent commands, and is meant to reduce vague “vibe coding” by forcing clearer requirements, decisions, tasks, and validation before code generation.

LeadDev Panel: “The high cost of AI immaturity”

I’m excited to share that I’ll be joining LeadDev’s panel to discuss how engineering leaders can assess where their organizations are on the AI adoption journey and move from experimentation toward disciplined, scalable use. The conversation will focus on practical questions around readiness, governance, and where AI can realistically support engineering work today.
It takes place on Wednesday, 27 May 2026 at 14:00 Central European Summer Time

Find more information here.

That's it for this month!
Let me know your thoughts about AI!
Michaela

Subscribe now

Want to go deeper on code reviews?
Find practical guides, workshops, and resources at awesomecodereviews.com.

In the post, I dive into how GitHub Copilot supports developers during code reviews by offering explanations, suggesting improvements, and answering questions. I also showcase how Copilot works within the IDE, highlighting the new “Review and Comment” functionality, which enables a guided and structured review experience. Additionally, I examine Copilot’s beta feature for reviewing pull requests directly on GitHub, demonstrating how it can enhance the PR review process.

Subscribe now

Here are four key insights from the article:

1) Use Copilot in the IDE for self-reviews.
Copilot shines in IDEs like Visual Studio Code, helping you iteratively improve your code before submitting a PR. The new Review and Comment feature enables structured and guided code reviews, highlighting changes line by line with detailed explanations. It’s a powerful companion for self-reviews and error detection.

2) Be cautious with AI suggestions.
While Copilot enhances code comprehension and suggests solutions, its outputs are based on statistical patterns rather than true understanding. Applying these suggestions requires careful evaluation to avoid broken code and errors. Always critically assess AI-generated feedback before implementing it.

3) A pull request is too late for AI review.
Once your code reaches the code review stage, applying AI-generated improvements is not ideal. Reviewers cannot easily validate these suggestions without deeper investigation, making them impractical unless the changes are obvious to the reviewer.

4) Many workflows for AI code reviews exist.
AI can still assist during PR reviews by answering questions about the code, helping build a mental model of the changes, summarizing the PR, or clarifying potential improvements. While it’s not a replacement for human judgment, it’s a valuable tool for specific use cases.

All in all, I’m a fan of LLMs and tools like Copilot and their ability to assist developers. These are fantastic technologies that can make us happier and better developers. Yet, we must apply them thoughtfully and stay aware of their limitations.

If you’re curious about GitHub Copilot and want a comprehensive walkthrough of its review capabilities—along with my insights and experiences with AI-assisted review workflows—be sure to check out the full post: Using GitHub Copilot to Automate Code Reviews.

Cheers, Michaela

Have you ever experienced anxiety around code reviews? If so, you're not alone. Many developers feel nervous or stressed when giving or receiving feedback on their code. This is what Carol Lee, PhD, Kirsten Foster-Marks, and Cat Hicks, PhD call Code Review Anxiety, and they've written a paper and a comprehensive workbook to help us all manage it more effectively. In this article, I want to give you an overview of the steps it takes to tackle code review anxiety.

What is Code Review Anxiety?

Code Review Anxiety is the fear of judgment, criticism, and negative evaluation during code reviews. Whether you’re the one reviewing or being reviewed, this anxiety can lead to stress and avoidance. Code review anxiety doesn’t just affect junior developers. Senior developers often face pressure to "live up to" their experience, which can create its own set of challenges and stress.

While it’s common to feel nervous, it’s essential to recognize that code reviews are not meant to be a personal attack but a way to improve and grow as developers.

Why Managing Code Review Anxiety Matters

Left unchecked, this anxiety can affect the quality of your code review feedback, how you respond to code review comments, and your willingness to engage in code reviews. Instead of fostering growth and learning, it can lead to frustration, defensiveness, and avoidance—none of which help anyone. Thus, feeling anxious can have a long-lasting negative effect on your relationships with your colleagues and negatively influence team dynamics and culture.

That’s why it’s crucial to manage code review anxiety. The good news is there are actionable steps you can take to do just that.

Steps to Manage Code Review Anxiety

1. Create Awareness. The first step to reducing code review anxiety is being aware of it. Recognize the specific situations that trigger your anxiety. Are you worried about making mistakes or receiving harsh feedback? Once you identify your fears around code reviews, you’re already on the path to managing them.

2. Reduce Physiological Arousal When you feel anxious, your body reacts. High cortisol levels, rapid breathing, and muscle tension are common responses to stress. To counter this, it’s important to calm your body. Simple techniques like deep breathing, muscle relaxation, or a workout (hello, jumping jacks!) can help you lower stress and regain a sense of calm before a review.

3. Challenge Biased Thinking What beliefs do you have about yourself or your colleagues that increase your anxiety around code reviews? Maybe you think you’re not skilled enough or that others will judge you harshly. These are often biased or exaggerated thoughts like "I'm not good enough.". By acknowledging them and challenging their validity, you can see them for what they are—unfounded fears.

4. Replace Negative Thoughts with Balanced Ones Once you’ve identified negative thoughts, replace them with more realistic and balanced ones. For instance, if you’re thinking, “I’m not good enough,” challenge it by asking, “Good enough for what?” "Good enough to do your job?!" Very likely, you are good enough to do your job. So, a better thought might be, “I’m learning and improving with each review,” which is both true and empowering.

5. Proactively Engage in Code Reviews After you have gone through all the other steps, it's time to do more code reviews. Yes, you read that right. One of the best ways to overcome fear is by facing it. The more you avoid code reviews, the more anxiety grows. Instead, proactively engage in them. Ask for feedback and be specific about what kind of input you’re looking for. When giving feedback, don’t shy away from making suggestions, but ensure you do it respectfully and constructively.

Why This Process Works

Each step plays a role in gradually reducing the anxiety associated with code reviews. By being aware of your fears, reducing their physical impact, challenging negative thoughts, and proactively engaging in reviews, you build resilience and confidence. Over time, this reduces the stress you associate with code reviews and transforms them into a productive, learning-oriented process.

Take Control of Your Code Review Experience

Remember, nobody is perfect, and mistakes are part of the learning process. The key is to approach code reviews with a growth mindset and see them as opportunities for development, not judgment.

Want to learn more about managing code review anxiety? Carol Lee and Kirsten Foster-Marks have explained everything in great detail in their code review anxiety workbook.

For that, surveys are an excellent tool of choice. A good, solid survey provides you with reliable and insightful data.

But how exactly do you measure developer experience using a survey?

After my talk at GOTO Amsterdam about improving developer experience, many developers approached me, wanting to learn how to design such a developer experience survey and which questions they should ask.

So, let me give you a step-by-step guide on designing and running a DevEx survey:

Developer Experience Survey: Step-by-Step Guide

1. Define the Survey Goal/Focus
   First, you have to settle on the aspects or factors you want to measure. Our grounded theory study on DevEx identified 26 factors that influence developer experience. While we identified a large range of factors, you do not necessarily have to measure all of them, especially not all of them at once. You can make a selection based on your current understanding of your developer experience. For example, if you know code reviews and tech debt are a big headache, you might focus more on factors that cover those areas, such as communication, knowledge sharing, deadlines, or factors that have to do with tooling.

2. Question Design
   In the next step, you formulate questions or statements based on the factors you selected. You also have to decide which scale to use. When designing the survey, keep simplicity and clarity at the forefront of your thoughts. You have to use closed-ended questions, like Likert scale questions if you want to derive indexes or measure the change in developer experience over time. Open-ended questions are a wonderful and rich instrument for gaining a deeper understanding of what is going on. Analyzing and understanding open-ended questions takes more effort, though.

3. Piloting Your Survey
   Before running your survey within the whole department or company, you have to conduct a pilot test with a small, representative sample of your audience. From this pilot, you want to get as much feedback as possible: did the participant interpret the questions as you intended? Were questions ambiguous or unclear? How about the length of the survey?

4. Running the Survey
   Once you have updated the survey, you can run it with all the developers you want to participate. For that, you have to find a tool that allows for smooth data collection and, ideally, also analysis. Common choices are tools such as Google Forms, SurveyMonkey, Culture Amp, Lattice, or Officevibe.

5. Data Analysis and Interpretation
   As a first step, you have to prepare and clean the data. This includes handling missing data, outliers, or inconsistencies. For open-ended questions, you need to code the responses. Then, you can use descriptive analysis, such as summary statistics or frequency distributions, to get a first picture of the outcomes and results. If you have a skilled Analyst next to you, you can also dive into inferential analysis, such as hypothesis testing, correlations, or regressions. Finally, you have to make sense of all the results and interpret them based on the context and circumstances at your company.

6. Continuous Improvement
   Many companies like Google, LinkedIn, or Shopify run developer satisfaction, happiness, or experience surveys. One thing all of those companies have in common is the need to continuously improve and adapt their surveys to make them more reliable and valid, as well as to keep them up with the changes in technology, engineering practices, and processes.

Concrete Surveys to Measure Developer Experience

Let's look at some concrete examples of developer experience surveys. I'll show you two types of Developer Experience surveys. to show the diversity in questions that you can ask, and also to inspire you for your own developer experience survey.

Developer Experience Survey based on three DevEx Dimensions

For our study of the impact of developer experience on outcomes such as innovation or productivity, we measure three dimensions of developer experience that have proven to be very relevant in practice:

1. Flow state,

2. Feedback loops,

3. Cognitive load.

Here are the questions that we developed and used:

Flow state:

• I have a significant amount of time for deep work in my work days.

• In a typical week, how often are you interrupted to work on something else that was unplanned or suddenly requested?

• Generally speaking, the coding tasks I work on are more engaging than boring

Feedback loops:

• How often does it take more than 10 minutes to obtain an answer to an internal technical question (i.e., about code, a system, or the domain you are using)?

• Approximately what percentage of the code reviews you request are completed within four business hours?

Cognitive load:

• For the primary team you work on, how would you rate the ease of deploying changes?

• How often can you easily understand the code you work with?

• In general, the processes I need to follow to do my work are easy for me to figure out.

• In general, the developer tools I have are intuitive and easy to use.
  

The questions we used to measure developer experience were either previously validated in the literature or developed and refined over time with expert input. You can look up the origin of each question in the paper.

Atlassian's Developer Experience Survey

Atlassian has a different angle from which to measure DevEx. They use eight signs to measure developer experience:

1. Sustainable speed for shipping

2. Waiting time

3. Execution independence

4. Ways of working

5. External standards

6. Maintenance

7. Onboarding

8. Developer satisfaction

For each vital sign, they recommend to ask about the developers satisfaction, and their perceived importance of this sign.

This is the example survey including questions for each sign from their playbook:

Sustainable speed to ship

• How important is shipping high-quality code sustainably for your team?

• How satisfied are you with your team’s ability to ship high-quality code sustainably?

Waiting time

• How important is minimizing waiting time to your productivity?

• How satisfied are you with the amount of developer waiting time on your team?

Execution independence

• How important do you consider your team's ability to deliver independently of other teams?

• How satisfied are you with your team's delivery independence?

Ways of working

• How important is it for your team to discover and onboard new ways of working, including tools, processes, and practices?

• How satisfied are you with your team’s ability to discover and onboard a new way of working, including tools, processes, and practices?

External standards

• How important to your productivity is the amount of maintenance or platform work it takes to meet the externally generated company standards your team owns?

• How satisfied are you with the amount of maintenance or platform work it takes to meet the externally generated company standards your team owns?

Maintenance

• How important to your productivity is the amount of effort required of you to maintain your team’s standards with regard to code, tools, and pipelines?

• How satisfied are you with the amount of effort required for code, tools, and pipeline maintenance?

Onboarding

• How important to your productivity is the amount of time it takes new hires or internal transfers to become effective on your team?

• How satisfied are you with the amount of time it takes new hires or internal transfers to become effective on your team?

Developer satisfaction

• How important is your satisfaction to your productivity?

• How satisfied are you with your team's developer productivity?

Survey Design Pitfalls

As you see from the two examples, your developer experience survey can contain different questions and cover different factors. Yet, not every question is a good question to ask. Some survey instruments are more reliable than others. Common mistakes people make when designing their own survey include using leading questions, subjective or ambiguous terminology, or designing a too-long and/or boring survey.

It's also important to always test your survey with a small set of people first and incorporate their feedback (i.e., piloting the survey). Thoroughly reading about survey design or working with a skilled Survey Designer or Survey Researcher significantly increases your chances of getting reliable and valid data that provides accurate and actionable insights.

Using a Ready-Made DX Survey Tool

Instead of designing your own questionnaire, you can use a tool like DX, which already provides you with a solid set of questions. As DX's main value proposition is to help you measure and improve developer experience, they have a complete team dedicated to ensuring their survey instruments lead to reliable measurements and actionable improvements. Using DX is a great way to get metrics and measurements, even industry benchmarks, out of the box and to quantify your improvements over time.

In-depth Investigations

Yet measuring developer experience is not enough to improve it. Once you identify areas of concern, I recommend regularly conducting deep dives into those areas. Here, designing our own survey, running a developer experience workshop, or shadowing/ interviewing developers can give you rich insights. System data, such as error logs, test logs, commit data, or PR data, can also be an insightful source of information - especially if used in combination with insights coming directly from the developers. Only when you deeply understand the friction points, problems, and challenges developers face can you start to create an improvement plan and take actions that lead to a better developer experience.

So, now tell me, do you consider developer experience at your company? How do you measure and/or improve it, and if so?

Thanks for reading. If you know someone who might like this newsletter, consider sharing it with them:

Share Awesome Code Reviews

Thanks, Michaela

The fact is that how developers review code vastly differs from developer to developer. Even within the same team, developers rarely use the same approach. Experienced reviewers usually have a pretty systematic way of reviewing code, but it’s informal, ad hoc, and only in their heads. Yet, many developers, especially juniors or people new to the team, often wonder how to review a particular piece of code.

Unfortunately, more formal review techniques and approaches aren’t taught or shared frequently amongst developers. Yet, we can use code reading, code inspection, and even debugging approaches to improve our ad hoc code review technique. When we know about those review techniques, they help us to:

1. transform and improve our review approach to a more systematic and thorough approach,

2. streamline review approaches amongst developers in the same team or company, ensuring the same thoroughness and review quality, independent of the reviewer,

3. build a shared vocabulary and a shared understanding of code review techniques, which helps us better communicate our approaches to our team members, compare them, and improve them as a team.

That's a pretty cool outcome, right?

Still, as code review techniques are not widely known, you might wonder what I am referring to. That’s why I wrote a comprehensive article on different review approaches on my website, awesomecodereviews.com.

The article highlights the following review techniques:

1. Checklist-based Code Reviews: Use a checklist to make sure you do not forget what to look for. This is my favorite technique, as it is easy to implement and a very effective review technique.

2. Test-Driven Code Reviews: Start by reviewing the test cases first. This ensures high-quality test cases and helps with understanding the production code.

3. Bottom-up versus Top-down Code Reading: You can either start your review with low-level parts, such as single code lines or methods, or with high-level artifacts, such as specifications or tickets. Reviewers typically use both approaches intertwined during reviews.

4. Control-flow versus Data-flow Code Reviews: You either follow the execution path of a program or you follow how data is passed through the application. Again, both approaches are often combined and have pros and cons.

5. Pattern Recognition: This approach is similar to a checklist-based approach. Yet, here, the focus is on detecting common anti-patterns and bad practices, as well as best practices in code.

6. Cross-Referencing: You inspect all references, like dependencies and relationships between structures within a program. It’s a thorough approach, yet it can lead to feeling overwhelmed.

7. Change-Impact Analysis: This approach involves analyzing a change's impact on the other parts of the system. It is a very thorough analysis, yet a good understanding of the software under review is a prerequisite.

8. Trace-Based Code Reading: Instead of looking at the static code paths, like during a control-flow inspection, you examine the program during runtime, often using a debugger, helping you to find very tricky and hidden problems and side effects.

9. Abstract-Driven Code Reviews: This is an academic review approach in which you compare the code to its specification, documentation, or detailed ticket description.

10. Functionality or Use-Case-Driven Code Reviews: This is another academic review approach that uses defined use cases or scenarios that you must play through while reviewing the code.

Are you surprised that there are so many different ways to review code? You can read more about each review approach, its pros and cons, and when to use it here.

Let me know which review techniques you already use in your ad hoc approach and which are new to you!

I’ll talk to you again in a bit.

Michaela

Long turnaround times lead to large reviews.

In addition, long turnaround times often lead to large reviews. If developers know they must wait days or weeks before someone looks at their code changes, they cannot afford to submit small, coherent changes. Then, instead of breaking up user stories and working iteratively and incrementally, developers implement the whole user story and ask for feedback only once they are done. Also, they might add more and more commits to the user story while the pull request is open because time passes, nobody looks at the PR, and they have to get on with their work.

Large PRs lead to long turnaround times.

Developers have difficulty giving valuable feedback when the code review size increases. Several studies show a direct correlation between review size and feedback quality. The larger the review, the lower the review feedback quality. In large reviews, fewer bugs are detected, and fewer readability or maintainability issues are discussed.

In addition, large reviews also lead to an increase in review times because reviewers have a hard time understanding this large code change, and problems that are found might be hard to resolve. So, the vicious circle of long waiting times and large reviews continues.

So, how can we break this cycle?

5 Tips for faster code reviews

To increase the code review turnaround time and still keep your review process thorough and effective, I recommend the following five tips:

1. Break Up User Stories: In my experience, user stories tend to be too large. So, break them up into smaller pieces. Make sure each user story can be reviewed independently and, as a rule of thumb, does not take you more than two days to work on. This means that instead of waiting for an entire feature or project to be completed, break it down into smaller, more manageable user stories.

2. Code Review Checklist: Using a code review checklist does not only boost your problem-detection power but also streamlines the process of reviewing. By using a checklist vetted by the whole team, you ensure everyone knows what issues to look out for. This clarity also comes with a nice boost in code review turnaround time.

3. Integrate Linters and Static Analysis Tools: Tools like ESLint, Flake8, or SonarQube can automatically catch common mistakes and enforce coding standards. To ensure you are not cramping up your CI/CD pipeline, use pre-commit hooks to ensure that only code meeting these criteria is pushed to the repository.

4. Clarity for code review duties: All team members must clearly understand their expected role in code reviews. This fosters accountability and a collective commitment to code quality. Further, the team should work out a schedule around code reviews to help people plan and make sure they have the time to do a proper review. So, a code review should never (or very seldomly) come as a surprise.

5. Select fewer code reviewers: Yes, you heard that right. Instead of adding more potential reviewers, I’d advise selecting only a few. If too many reviewers are listed on a code review, you might find yourself experiencing the “diffusion of responsibility” phenomenon. This means that everyone thinks that someone else will do the review. Because everyone waits for another person to do the review, the code review turnaround time increases.

More stats to convince you

In a recent study, which I’ve conducted together with Nicole Forsgren, Eirini Kalliamvakou, and Brian Houck from Microsoft, Dr. Storey from the Univeristy of Victoria, and Abi Noda from DX, we could clearly show the impact a fast turnaround time has on team and business outcomes.

As you see in the illustration below, a faster turnaround time increases innovation by 20%, boosts creativity and learning by 15%, and even reduces technical debt by 10%.

Cool, ha?!

To know more about this study, read my previous issue on measuring developer experience here.

So, what are your best tips for a faster turnaround time?

Hit reply and let me know, or share your thoughts on Twitter with me: https://twitter.com/mgreiler.

Subscribe now

Developer Experience (DevX)

Over the last two years, I have been investigating developer experience - that is, how developers experience their work and how that influences outcomes such as productivity, code quality, or retention.

In our first in-depth study, we look at which factors influence developer experience. We found 25 factors that developers regularly described when they talked about developer experience and grouped them into 4 categories:

• development and release

• Product management

• Collaboration and culture

• Developer flow and fulfillment

We also identified how developer experience can be improved and which strategies developers have to improve it, as well as which challenges and barriers they face when they try to improve DevX.

Devs started to talk about DevX

Our research was published in mid-2022 (you can download the full paper here), and since then, a lot of people in the industry have started to talk about developer experience. This is awesome!

Developers all around the world could relate to our findings and echoed that they indeed need good developer experience to be productive and ship high-quality code. Developers know that they need time to focus and do deep work. They know about flow state and how fast code review turnaround times are a game changer.

DevX? No buy-in from management

But once again, business folks were too far away from programming to understand the importance of DevX. The fact is that many companies are only willing to spend money on DevX during “good“ times — when we have too much time, too much money, and no other priorities. Well, those times do not exist, especially not with a bad DevX.

When talking to engineering leaders, we realized that what is missing is data that quantifies the impact of DevX. That’s why we have worked on getting this data over the last year. Data that shows the impact of DevX on productivity, code quality, retention, innovation - well, all the outcomes that business cares about.

Hard numbers to convince business

Our research article will be published soon. The research goal was to quantify DevX's impact on developers, teams, and organizations. The picture below highlights how improvements to the flow state of devs, their cognitive load, and the feedback loops they encounter have a direct and quantifiable impact.

Code Review Related DevX

Another interesting outcome of this study is that we could show that a smooth code review process impacts productivity, retention, and learning. As you see in the quote below, code reviews also impact innovation.

If we think about it, it makes sense. Innovation suffers if we have to wait a long time to get feedback on our work or if releasing new features is painful.

How to Improve Code Review Turnaround Times

But which strategies help to increase turn-around times? Well, that depends on the underlying root cause for the slow feedback. Should we have a look into this topic in the next issue? Let me know!

Cheers, Michaela

Sign up now so you don’t miss the first issue.

Subscribe now

In the meantime, tell your friends!