A lot of teams look like this right now. Someone on theteam built a custom tool to speed upynthesis. An engineer stood up an MCP server that connects an AI assistant to internal systems. An ops lead has an agent automating a reporting workflows. And someone in another department built a skill that does almost the same thing as one of those, with different instructions and different integrations. Everyone is experimenting and building, and plenty of people are solving problems someone else already solved. Which is fine as we start to explore and learn these tools.

I think that’s a good problem to have, at least at first. It means AI adoption has moved past curiosity. People are spotting friction in their own work and fixing it themselves instead of waiting on a central tech team to get to it.

It raises a hard question: what happens when an org has hundreds of skills, agents, MCP connections, and custom workflows and no plan for how they fit together?

There’s a big upside here, and a risk of rebuilding the same fragmented tech environment that transformation teams have spent years cleaning up. How far an org gets with AI from here depends less on how many tools people create and more on how well the org finds, evaluates, connects, governs, and reuses what already exists.

More tools don’t automatically mean more efficiency

Enterprise tech leaders have spent years fighting application sprawl: cutting redundant software, standardizing integrations, consolidating overlapping capabilities.

Generative AI brings that problem back, and this time almost anyone can create the tools.

A business analyst can automate a recurring task without a formal dev cycle. A marketer can spin up a specialized assistant. A PM can build a repeatable research workflow. An engineer can expose internal capabilities through an MCP server.

The barrier to building has dropped a lot. The barrier to coordinating across the org hasn’t moved.

Here’s a rough example. Say 500 people at a company are building AI-enabled workflows, and 20% of them make something reusable. That’s 100 tools or automations, fast.

Some of those will be very very valuable. Others will overlap, run on stale instructions, duplicate something the platform already does, or depend on integrations nobody reviewed. A few will turn into things people rely on every day without anyone treating them as production dependencies.

So individuals save time while the org piles up complexity. An agent that saves one person three hours a week looks like a clear win. If six teams each build and maintain their own version of that agent, the company is paying a hidden tax in duplicated building, troubleshooting, maintenance, and governance.

Gains at the individual level don’t add up to optimization at the org level on their own. To get the bigger payoff, orgs need to manage employee-built AI as a shared portfolio of reusable assets.

Start by sorting out what’s being built

People use skill, agent, MCP, and automation pretty interchangeably, which makes this stack hard to manage. They’re related, and they solve different problems.

A skill might teach an assistant how to run a consistent competitive analysis. An MCP server might give it access to approved market intelligence sources. An agent could use the skill and those tools to research competitors, synthesize findings, and draft a report. An automation could kick that off every month.

Most of the value tends to show up in the combination. Without a shared architecture, though, each team writes its own instructions, integrations, orchestration logic, and report format, and the org ends up with several versions of one capability.

Teams can keep experimenting wherever they are. What needs a home is the stuff that works, so other teams can pick it up.

Manage capabilities alongside tools

Orgs usually manage software as applications, infrastructure, and services. Employee-built AI adds another category to track: capabilities.

A capability is a reusable function that gets you to a specific business outcome, whether it’s built as a skill, agent, API, MCP tool, or a regular automation.

Take customer feedback analysis. One team builds a skill that classifies feedback themes. Another builds an agent that pulls and summarizes customer comments. A third builds an MCP integration to the feedback repository. You could log those as three unrelated tools, or you could treat them as pieces of one customer intelligence capability.

That moves the conversation from “what tools has everyone built?” to “what business capabilities do we have now, where are the gaps, and how do we get them to more teams?”

The second question is a much better starting point for optimization, and it gives you a way to manage the portfolio without shutting down experimentation.

The missing layer: an internal AI capability registry

Most orgs already have some mix of software inventories, API catalogs, knowledge repos, and service management platforms. What’s usually missing is a searchable, maintained list of the AI capabilities employees have built.

A registry like that could connect experimentation to enterprise adoption. Think of it as a living catalog that answers practical questions:

• What’s already been built?

• What problem does it solve?

• Who owns it?

• What systems does it touch?

• Is it approved for wider use?

• How reliable is it?

• Can another team reuse it?

Every entry needs a small, consistent set of metadata. Here’s what one could look like:

It doesn’t need to start as a fancy platform. Structured metadata, clear ownership, and a way to search it are enough to begin. What matters is that it becomes part of how people share and manage AI capabilities.

Over time it could hook into source repos, access controls, evaluation results, deployment environments, and usage telemetry. Then people can find approved capabilities before building new ones, and platform teams can spot where to consolidate.

There’s a culture upside too. People who build useful things get visibility for it, and other teams get to start from proven work instead of from zero.

Build once, adapt many times

One of the more promising patterns in enterprise AI is separating reusable capabilities from the workflows that use them.

Take a common task: pulling information from an internal knowledge system. Without shared architecture, several teams each build their own integration to the same repository, and each one handles authentication, search, filtering, and errors a little differently. A better setup is one reusable, permission-aware integration that any approved assistant or agent can call.

Skills work the same way. If the company has a solid method for writing product requirements, evaluating experiments, or reviewing accessibility, that expertise can be packaged as reusable instructions and evaluation criteria. Teams adapt it to their workflows without rebuilding the method each time.

Here’s the shared architecture, top to bottom:

1. Business workflows and employee experiences: research, reporting, product delivery, customer operations

2. Skills for reusable expertise, agents for workflow execution, and automations for scheduled actions

3. A shared MCP and API integration layer with permission-aware access to approved tools and resources

4. Enterprise systems and trusted data: knowledge bases, analytics, CRM, internal services

Across every layer: registry, identity, governance, evaluation, telemetry.

This is conceptual, and every org can land on its own stack. Some workflows will stay simple automations, and others will be worth full agent orchestration.

The principle is to cut unnecessary duplication and keep the flexibility to solve different problems. Standardize the capabilities that benefit from consistency, and leave room to customize where the business context calls for it. That balance is what makes reuse work in practice.

Governance that supports building

When the number of AI tools starts growing, the instinct is to add controls: approval processes, security reviews, architecture standards, access restrictions, documentation requirements.

Those matter, especially when AI systems can pull sensitive data or take actions inside enterprise apps. If every employee-built skill goes through the same review as a production application, though, orgs will slow down the experimentation that made these tools useful in the first place.

Progressive governance works better. Low-risk capabilities stay lightweight. A skill that formats meeting notes doesn’t need the oversight of an agent that can edit customer records, start financial transactions, or change production configs. The level of governance should match what happens if the capability goes wrong.

One way to set that up is three tiers:

• Personal experimentation: sandboxed capabilities that use approved tools and non-sensitive information, with limited distribution.

• Shared team capabilities: documented, tested, and owned tools available to a defined group, with the right access and security reviews.

• Enterprise production capabilities: operationally supported services with formal evaluation, monitoring, version control, incident response, and change management.

Capabilities move up a tier based on demonstrated value, risk, and readiness.

This lines up with the NIST AI Risk Management Framework, which emphasizes governance, measurement, and ongoing risk management across the AI lifecycle.

It also builds a useful habit. Some of what gets built should stay an experiment, some should get merged with something else or retired, and some should become shared enterprise capabilities with clear owners and support.

Measure business outcomes

An easy mistake is treating tool creation as a stand-in for AI maturity. A dashboard showing 400 agents created, 250 skills published, and 60 MCP servers deployed tells you people are experimenting. It tells you very little about business value.

A more useful measurement framework looks at five dimensions:

The gap between theoretical and realized savings matters a lot here. Say an agent cuts a 60-minute reporting task to 15 minutes, and the team runs 100 reports a month. On paper that’s 75 hours saved. If people then spend time validating outputs, fixing errors, or rerunning failed workflows, the realized savings could be much lower. You have to measure the whole workflow, including the new work AI adds.

A simple way to frame value:

Realized benefits could include verified labor capacity freed up, fewer errors, faster cycle times, and measurable revenue gains. Costs should cover development, inference, integration, evaluation, human review, security, and maintenance.

Time saved and money saved are two separate numbers. Freed-up capacity becomes a financial benefit once it turns into a business outcome you can observe.

Observability matters for this. Standards like OpenTelemetry are developing conventions for measuring generative AI operations, including model usage, execution time, and agent activity. Those signals help teams understand technical performance. Business outcomes still need their own measurement.

The leadership question should move from “how much AI are we using?” to “what measurable improvement is AI producing?”

Optimize the workflow along with the tool

Orgs tend to evaluate AI tools one at a time, even though most business processes run across multiple systems, teams, and handoffs.

A research agent speeds up analysis. A documentation skill makes output more consistent. An MCP integration cuts time spent hunting for information. Each one is a local improvement. Connect the three into one workflow and you get something bigger.

Say a product development process runs like this: customer feedback gets collected, themes get identified, opportunities get prioritized, requirements get drafted, and stakeholders get a structured summary. Today those steps might involve different people, different apps, and a lot of re-sending the same information. With a coordinated AI capability stack, some of those stages can share context, reuse validated information, and produce standard outputs. People still own prioritization and approval, and AI handles the supporting work where it fits.

That could mean a shorter cycle overall, fewer handoffs, and less repeated effort. This is where workflow-level optimization starts to beat isolated task automation.

Connecting agents isn’t automatically better than one agent or a deterministic workflow, though. More orchestration can add latency, cost, failure points, and debugging headaches. The work should drive the architecture.

Sometimes the most efficient AI solution is a well-written skill. Sometimes it’s a single API call. Sometimes it’s an agent coordinating several tools. And sometimes the best optimization is cutting an unnecessary step out of the process entirely.

A practical 90-day plan

You don’t need a full enterprise AI platform before you start improving how employee-built capabilities get managed. A focused 90-day push can lay the groundwork. More of my thoughts on that soon.

Read more

Teams that track their work in Jira can show their impact with numbers, and AI makes that all the more practical.

Researchers struggle sometimes to show what they contributed. Findings land in a deck, the deck gets shared, a decision gets made, and the trail back to the research goes cold. Six months later nobody remembers which study changed the roadmap.

Jira keeps the record, and AI makes the record easy to read. Once research lives as tickets linked to product work, an AI assistant connected to Jira can count it and summarize it in a few minutes. That turns a vague sense of “research helped” into a report you can hand to a PM or a leader.

Here is how I set it up, and the AI workflows I run on top of it.

Why Jira matters

Product and engineering already plan in Jira. Their work has owners, statuses, sprints and dependencies. For us, research usually lives somewhere else: a wiki page, a spreadsheet of study ideas, a Slack thread, and sometimes sparse Jira’s.

When research sits outside the system PMs plan from, it gets treated as a service you request. Put it in Jira and PMs see the linked research epics inside their own initiatives, with status and owner. A study marked Blocked becomes a planning input they can act on before it turns into a missed date.

There’s a second payoff. Every ticket has a date, a status, a theme and a link to the product work it supports. That makes research into data, and data is something AI can read.

The structure that makes it measurable

AI counts what you record consistently. Five habits give it enough to work with, and none of them need admin rights.

1. An epic for every body of work: a survey wave, a usability round, a workshop, a measurement framework. Anything longer than a couple of days.

2. Stories for the deliverables: split a study into analysis, data work and readout so each piece moves on its own.

3. Labels for strategic themes: pick a handful that match your org’s bets, like onboarding, trust or checkout, and use them on every epic.

4. Links to the product epic: link each research epic to the PM-owned initiative it supports, using a link type like “relates to” or “accomplishes.” This link is your evidence of collaboration.

5. Current statuses: move tickets to Blocked when they stall and close them when they’re done. Stale tickets throw off every AI summary.

I also ticket the work that never makes a readout: teaching sessions, interview loops, internal tools, conference talks, writing. If it took hours, it gets a ticket.

The five habits on the left are what make the five reports on the right possible.

Here is how: five AI workflows on top of Jira

All of these run through an AI assistant with a Jira connection, either Jira’s built-in AI or a general assistant connected through an integration. The assistant runs the queries, reads the tickets and writes the summary. You check it.

1. The impact report by theme

Ask the assistant to group your closed epics by label and list the product initiatives each one linked to.

You get a table that says how many studies research delivered for each strategic bet and which product work they fed.

2. The collaboration map

Links and partner-team names in ticket titles show who you worked with.

Leaders often ask how many teams research supports. This gives you the number.

3. The review evidence doc

At review time, ask the assistant to turn your closed tickets into a list of accomplishments, each with its ticket link.

I used this for my own goals review. Every claim pointed at a closed ticket, so I spent my time deciding what mattered most.

4. The stale ticket audit

Once a month, ask for anything open that hasn’t moved.

This keeps the data clean, which keeps workflows 1 through 3 accurate.

5. The quarterly plan

Combine Jira with your planning docs and channels.

I ran this for a quarter of planning. It surfaced a blocked study that needed a go or retire decision before it could sit on the roadmap.

What to watch for

• Check every number: AI summaries can miscount or invent a link. Spot-check a few tickets before a report leaves your hands.

• Pair counts with outcomes: ticket volume shows output, and the decisions research changed still need your words.

• Keep tickets light: a title, a one-line goal and the right label is enough. Heavy tickets stop getting written.

• Mind what you share: ticket titles can carry sensitive project names, so scrub them before an AI summary goes outside your team.

Start with one epic

Create an epic for the study or project if you are not in research that you’re running now, give it a theme label and link it to your PM’s initiative. Next month, ask an AI assistant to summarize what research delivered for that theme. The first report will be small. By the end of a quarter, it may be the clearest case for research impact you’ve ever had.

I’d love to hear how your team tracks research. Leave a comment or reply to this post and let me know!

There’s something funny about the way we celebrate individuality these days.

Everyone is encouraged to be themselves, although preferably in a way other people can recognize. We have aesthetics for every personality, communities for every interest, and entire industries built around helping us stand out. You can be a minimalist, a maximalist, a free spirit, a creative rebel, or someone who has rejected modern life in favor of a simpler existence. There are probably three thousand videos explaining how to achieve whichever one appeals to you.

Even rebellion seems to come with a shopping list.

I’ve been thinking about this lately because so much of what we call individuality feels strangely familiar. The clothes, the language, the lifestyles, the opinions. We have more ways to express ourselves than previous generations could have imagined, yet we seem remarkably good at sorting ourselves into recognizable groups.

Maybe that’s simply what humans do. We’ve always looked for people who share …

Read more

I made a promise to the State of Indiana as a child. I graduated high school a year early, went to Purdue, and eventually earned my master’s at DePaul. I’m grateful for the scholarship that helped me get there. I’m also still angry about parts of the system.

I was in fifth grade when I remember signing a contract with the State of Indiana.

I was a child, making a promise about a future I could barely imagine.

The agreement was connected to Indiana’s 21st Century Scholars program, an initiative designed to help students from low-income families afford college.

I understood the opportunity was important.

I also understood that there were rules.

What I couldn’t have understood at that age was how complicated my feelings about those rules would become.

I went on to graduate from high school a year early. I attended Purdue University, where 21st Century Scholars helped pay for my undergraduate education. Later, I earned my master’s degree at DePaul University, without that scholarship paying for graduate school.

I did what I set out to do. And I’m grateful.

Truly.

But I’m also angry sometimes…..

Angry at the expectations attached to financial assistance. Frustrated by the way institutions can turn support into something that feels conditional, bureaucratic, and deeply personal. And sometimes angry when I think about Purdue Promise, the university program intended to help students like me succeed.

For years, I’ve struggled with how to reconcile those feelings.

• How can you be grateful for something that changed your life and still resent parts of the experience?

• How do you criticize a system that helped pay for your education?

• And why does receiving help sometimes make it feel harder to question the terms?

I wanted to understand whether those feelings were simply personal or whether the structure of these programs could help explain them.

So I started looking at the research, the historical requirements, and the outcomes. What emerged wasn’t a simple story of a good scholarship or a bad one.

It was a story about opportunity, accountability, poverty, and the complicated relationship between receiving assistance and feeling deserving of it.

And the more I looked, the more I realized that gratitude and anger weren’t necessarily contradictory. They were responses to different parts of the same experience.

The Good: Indiana Made College Possible

I want to begin here because I don’t want the criticism to overshadow what matters most.

Indiana paid for my education.That is an enormous thing.

The 21st Century Scholars program was established in 1990 to help income-eligible Indiana students pursue higher education. It provides up to four years of tuition assistance at eligible Indiana colleges and universities.

For students growing up in families where college costs otherwise make higher education unrealistic, that commitment can change what the future looks like.

And it changed mine.

I knew from a young age that college was something I could work toward.

I had a path.

I had a reason to take the opportunity seriously.

And I did.

I graduated from high school a year early and went to Purdue.

I don’t share that because I think academic achievement makes someone more deserving of help. It doesn’t.

A student who struggles academically deserves educational opportunity just as much as a student who excels.

I share it because my experience complicates the way we talk about scholarship recipients.

I wasn’t simply trying to meet the minimum expectations. I was moving ahead of the traditional timeline.

And the scholarship was part of the educational path that followed.

Purdue Promise added another layer of assistance for eligible 21st Century Scholars, combining financial support with individualized coaching and student success resources.

The university has reported meaningful results.

In 2018, Purdue reported that nearly 63% of the 292 Purdue Promise students who entered in fall 2014 had graduated within four years. That compared with just over 60% of Purdue students overall.

The university also reported that 60% of Purdue Promise graduates had no student debt. Those who graduated with debt owed an average of approximately $7,000.

Those are significant outcomes.

They don’t prove that Purdue Promise alone caused those results, and they don’t describe every cohort or individual experience. But they provide evidence that the program helped support a meaningful path through college.

Programs like 21st Century Scholars can change lives.

They can make college financially accessible.

They can help students graduate with less debt.

They can create opportunities that extend well beyond an undergraduate degree.

I know that because I benefited from one.

The good is genuinely good. And my criticism doesn’t make me any less grateful for it.

The Bad: The Rules Started Before Adulthood

Here’s where the story gets complicated.

When I remember signing that agreement, I remember understanding that there were expectations attached to the opportunity.

The historical rules show just how far those expectations extended.

Indiana didn’t simply ask Scholars to maintain their grades and graduate from high school.

The program’s pledge also included behavioral requirements.

Under Indiana law in 2010, participating students were expected to graduate from an eligible high school, maintain at least a 2.0 cumulative GPA, avoid illegal drug use, avoid specified criminal or delinquent conduct, and apply for college admission and financial assistance on time.

Scholarship applicants also had to certify that they had not illegally consumed alcohol before graduating from high school.

Older administrative regulations like mine, went further, including provisions related to school disciplinary records and the release of certain juvenile or criminal records upon request.

These weren’t merely suggestions about responsible behavior. They were conditions attached to a potentially life-changing educational benefit.

I understand the intention.

The state wanted students to stay in school, make responsible decisions, avoid trouble, and prepare for college. Those are reasonable goals. But the approach is not equitable.

Why was financial assistance for students from low-income families tied to a behavioral agreement that students from wealthier families didn’t need to sign to access college?

A teenager whose family can afford tuition may make a mistake without automatically putting that financial support at risk.

For a Scholar, certain mistakes could carry consequences that extended into their educational future. That’s a meaningful difference.

And it raises questions about how we define deservingness.

When financial assistance is tied to academic and behavioral compliance, we’re doing more than helping someone afford college.

We’re also establishing conditions under which we believe that person deserves continued assistance.

Some accountability is reasonable.

But the question is whether every condition is necessary, proportionate, and supported by evidence.

And whether the consequences fall differently on students who have fewer financial resources.

Indiana’s requirements have evolved over time. The minimum GPA increased to 2.5 for students expected to graduate after June 2014. The state later introduced additional college-readiness and credit-completion requirements.

Those later rules shouldn’t be confused with the rules that applied to every earlier Scholar.

But the underlying philosophy remains worth examining.

For students from financially vulnerable backgrounds, access to education was not simply a matter of admission.

It was also tied to maintaining eligibility under a set of conditions that began years before college.

One historical detail still deserves clarification: the program’s ordinary formal enrollment period was seventh or eighth grade, although older Indiana law allowed certain other-grade eligibility arrangements. I remember a fifth-grade agreement, but I haven’t established whether that was formal enrollment, a preparatory pledge, or a school-based introduction to the program. That uncertainty doesn’t erase the memory.

It does remind me how important it is to distinguish what a child remembers being promised from what the official rules required.

The Ugly: When Help Comes With Conditions That Change How It Feels

College introduced another layer.

At Purdue, the state’s scholarship and Purdue Promise were related but distinct programs.

The state provided tuition assistance under its eligibility rules.

Purdue Promise required a structured support system for qualifying Scholars. That support included individual coaching, academic guidance, and programming designed to help students graduate.

Purdue described its coaching model as a way to monitor student progress, connect students with resources, and address issues that could affect academic success or financial aid eligibility.

I can understand why the university built the program this way.

Students navigating college without substantial financial resources may benefit enormously from personalized guidance.

A good coach can help someone understand a complicated financial aid process, identify academic support, or navigate an unfamiliar institution.

But I think we need to be willing to examine the other side of structured assistance.

What happens when the people helping you are also part of a system monitoring your compliance with program expectations?

How does a support relationship change when participation is connected to a scholarship agreement?

When does accountability feel encouraging?

And when might it begin to feel like scrutiny?

I don’t think those questions have universal answers.

Some students may find mandatory coaching reassuring.

Some may experience the structure as a source of stability.

Others may feel that the support comes with another set of obligations they cannot afford to get wrong.

The existence of coaching requirements doesn’t prove that students were mistreated. And my own frustration isn’t evidence that the entire program failed.

But positive institutional outcomes don’t tell us everything about how a program feels to the people participating in it.

A graduation rate tells us whether students completed their degrees within a particular period. It doesn’t tell us whether they felt trusted.

A coaching appointment tells us that a meeting happened.

It doesn’t necessarily tell us whether the student experienced that meeting as helpful, stressful, empowering, or obligatory. For me, it was stressful. Always being questioned, monitored, even though I was excelling.

And a scholarship agreement tells us what the student was required to do. It doesn’t tell us what it felt like to depend on the assistance attached to that agreement.

Financial vulnerability can make institutional decisions feel much more consequential.

When you have limited resources to fall back on, losing assistance isn’t a minor inconvenience. It can change your entire educational plan.

That reality can make even routine administrative requirements feel higher-stakes. And I wish we talked about that more. Not because student success programs are inherently harmful.

But because a system can be designed to support students and still create pressures that deserve examination.

The Sad: Gratitude Can Become Its Own Kind of Pressure

This is the part I find hardest to articulate. When someone helps pay for your education, you’re expected to be grateful.

And I am.

But gratitude can become complicated when it starts to feel like an obligation to stay quiet.

• You received an opportunity.

• You benefited from public funding.

• You earned a degree.

So what exactly do you have to complain about?

Receiving assistance doesn’t mean giving up the right to question how the assistance is administered.

And successfully navigating a system doesn’t mean the system worked perfectly.

• I graduated from high school a year early.

• I went to Purdue.

• I earned two undergraduate degrees at the same time.

• I continued my education at DePaul and earned my master’s without 21st Century Scholars paying for that degree.

• I have a great and important job in tech that I love.

By many conventional measures, my story is a success. And I’m proud of it.

But I don’t think success should require me to turn the entire experience into an uncomplicated story of gratitude.

The uncomfortable parts still count.

The frustration still counts.

We should also listen to the people who succeeded and still have questions.

Because they can tell us something important.

They can help us understand where a program’s measurable outcomes and its lived experience don’t necessarily align.

It gives us a fuller picture of what the assistance actually meant.

What the Numbers Reveal—and What They Don’t

The more I researched 21st Century Scholars and Purdue Promise, the more interested I became in what institutions choose to measure.

• We can measure how many students enroll.

• How many maintain scholarship eligibility.

• How many complete their degrees.

• How much debt they graduate with.

Those are important metrics.

Purdue’s 2018 results are encouraging. Nearly 63% of the fall 2014 Purdue Promise cohort graduated within four years, and a majority of the graduates had no debt.

But there’s another way to read the same figures.

Approximately 37% of that entering cohort had not graduated within four years.

That doesn’t mean they all dropped out, lost their funding, or failed to earn a degree later.

It means that even in a program reporting positive outcomes, a substantial share of students did not complete their degrees on that four-year timeline.

And I want to understand more about what happened to them.

• Did they change majors?

• Did they need to work?

• Did family circumstances interrupt their education?

• Did they struggle academically?

• Did scholarship requirements become difficult to maintain?

• Or did they simply need more time?Those questions matter because college completion isn’t a single event. It’s the outcome of years of academic, financial, and personal decisions.

Indiana’s later adoption of a 30-credit annual renewal requirement illustrates the tension.

The policy was designed to encourage timely progress toward graduation. Under the rules effective for scholarship renewals beginning in the 2017–18 award year, students generally needed to earn 30 credits during the relevant academic year to renew the full 21st Century Scholarship, with other aid possibilities and eligibility provisions available in some circumstances.

That sounds straightforward.

But a student taking 15 credits each semester could fall short after failing or withdrawing from one three-credit course. I had to take courses all summer while also working two jobs just to make sure I had the scholarship for the next year.

For a student with ample financial resources, that setback may mean retaking the class.

For a student dependent on financial aid, it may create additional uncertainty about how to pay for the next year. Re: one time, I had to retake a statistics class and it almost made it so I couldn’t graduate because I did not know how I would afford it.

I don’t think that automatically makes the policy wrong. But it makes the consequences worth studying.

And it highlights a broader problem with evaluating educational programs solely through completion metrics. We can quantify academic progress.

It’s much harder to quantify the stress of maintaining eligibility, the burden of navigating institutional rules, or the feeling that a single setback might jeopardize an opportunity you’ve been working toward since childhood.

Those experiences deserve research, too.

What Came After

I think experiencing education with and without scholarship support has shaped how I look back.

I know what it means to receive significant financial assistance.

I also know what it means to pursue additional education without that assistance.

The scholarship helped with one important stage of my education, but my goals extended beyond it. That distinction makes me think about how we define the success of programs like 21st Century Scholars.

• Is success getting a student into college?

• Is it helping them earn a bachelor’s degree?

• Is it helping them graduate with less debt?

• Is it creating opportunities for graduate education, career advancement, and long-term economic mobility?

Ideally, we’d want to understand all of those outcomes.

But they’re not interchangeable.

A scholarship that reduces undergraduate tuition may create possibilities that extend well beyond the years in which the assistance is provided.

At the same time, an undergraduate scholarship cannot eliminate every financial obstacle a student will encounter afterward.

My master’s degree is part of my story.

So is the scholarship that helped pay for Purdue.

So are the expectations, frustrations, and complicated feelings I still carry from that earlier experience.

What I Wish We Would Do Differently

I don’t want Indiana to eliminate 21st Century Scholars.

I don’t want Purdue to stop supporting students through Purdue Promise.

I want more students to receive meaningful financial assistance.

I want students to have access to academic coaching, mentoring, and resources that help them succeed.

And I want the institutions providing that assistance to be willing to ask difficult questions about the experience they’ve created.

• Are the eligibility requirements supported by evidence?

• Are the consequences of noncompliance proportionate?

• Do students clearly understand the rules?

• Do support services feel supportive to the students receiving them?

• Are there meaningful ways for students to challenge decisions or provide feedback?

• And are institutions measuring the experiences of students who struggle, not just celebrating those who succeed?

I’d also like to see more attention paid to how scholarship rules affect students differently depending on their circumstances.

• The student working to support their family.

• The student navigating college without parental guidance.

• The student who experiences a personal crisis.

• The student who graduates early.

• The student who needs an extra semester.

• The student who meets every requirement but still finds the process emotionally exhausting.

These students don’t all need identical support.

But they do deserve systems designed with the complexity of their lives in mind.

And perhaps most importantly, they deserve the freedom to speak honestly about their experiences without feeling that criticism makes them ungrateful.

The Good, the Bad, the Ugly, and the Sad

I am a 21st Century Scholar.

I remember making a promise to the State of Indiana when I was in fifth grade. I graduated from high school a year early. I attended Purdue University, where the scholarship helped pay for my education.

I went on to earn my master’s degree at DePaul.

And today, I can look back on that journey with both gratitude and frustration.

1. The good is the opportunity Indiana created.

2. The bad is the complexity of the rules attached to it.

3. The ugly is the possibility that systems designed to help financially vulnerable students can also create pressures that are difficult to see in institutional success reports.

4. And the sad is that receiving life-changing assistance can make it feel harder to question the way that assistance was delivered.

I don’t think these realities contradict one another. I think they belong in the same conversation.

Indiana made an investment in my future. I took that opportunity seriously. I worked hard. Really hard. I earned my degrees.

And I’m thankful for the financial support that helped make part of that journey possible.

We can finish a checkout in a few taps now, but……I still want a second to see what I’m agreeing to.

I booked a room the other night. Two nights, nothing fancy, a city I already know pretty well.

The rate on the first screen was fine. I got through the dates and the room, and I was most of the way done before the total changed. A fee I hadn’t seen. Cleaning, or a resort fee. I don’t remember which. It wasn’t a lot of money. But it was annoying it showed up after all the checks.

So I sat there with the card fields filled in, deciding if I cared enough to start over.

Checkout used to be a slog. Name, address, the address again, the card, the little code on the back, a password, the password again. A lot of that got pulled out, and a lot of it should have. Asking me for a fax number was never doing anything useful.

An extra field and a fee on the last step are both bad friction.

When I’m the one paying, they don’t land the same way. The field is annoying. The fee has me wondering if the first price was the real one in the first place. And this is not only a booking lodging problem. We see this hidden fees bs everywhere on all consumer producta.

Baymard (https://baymard.com/blog/current-state-of-checkout-ux) updated a big checkout benchmark in November 2025, and about 64% of the desktop sites they scored were mediocre or worse. They’ve spent about 16 years testing this, and they still find that the checkout flow itself is often why someone leaves. Their cart abandonment page, updated in September 2025, puts the average across 50 different studies at 70.22%.

A chunk of that is people who were just looking. On the same page, the other reasons include extra costs that were too high, at 40%, and a checkout that was too long or complicated, at 17%. Not being able to see the total up front is on there too, at 12%.

People do leave when the form is a mess or if it is too long so a better frictionless checkout is not unwarranted. People also leave, or they finish while annoyed, when the price moves at the end. Pulling out another field doesn’t tell me the fee…..A fee that shows up after when you would expect.

Harry Brignull (https://deceptive.design/types/hidden-costs/) calls the on-purpose version hidden costs. A low price gets you in. You do the clicking. The real number shows up when starting over would cost you the time you already spent.

A 2021 paper, Price Salience and Product Choice (https://doi.org/10.1287/mksc.2020.1261), comes from a large experiment on StubHub. Tom Blake, Sarah Moshary, Kane Sweeney, and Steven Tadelis found that when the full price was on the screen up front, people bought fewer tickets, and they bought cheaper ones. Hiding the fee made comparing harder, and people spent more than they would have if they’d seen the number earlier. Of the revenue the site gave up by showing the fee up front, at least 28% came from people picking a lower-priced ticket, not only from people leaving. That’s a 2021 result, so I’m not treating the 28% like this year’s news. but the shape and flavor of it is familiar.

Kim Flaherty at Nielsen Norman Group wrote the shopper version of this back in 2018, in a piece on taxes, fees, and shipping (https://www.nngroup.com/articles/ecommerce-taxes-fees/). In their ecommerce studies, a surprise charge at the end was often enough to make someone leave, and a lot of those people read it as the site trying to trick them. Ordinary tax and ordinary shipping can wait until there’s an address. A fee that’s odd for that product should be on the page before anyone has typed a card.

I didn’t need a paper trail to feel annoyed with this but it helps.

Amazon’s 1-Click is where a lot of this speed comes from. They turned it on in 1997. The patent announcement in 1999 (press.aboutamazon.com/1999/10/amazon-com-r…es-patent-for-1-click) is pleased with itself in the way those announcements are. Shipping and billing stay stored. Buying is one click, instead of typing the same address every time.

If I already know the thing, and the price, and that I want it sent to me, that click is a kindness. I’m not nostalgic for retyping my street.

When something else can hit the button

The newer version of a fast checkout is a shopping assistant that can do the clicking for me.

I don’t have big theories on that. I have mostly… worry. If I hand it the errand, I still want to know what it picked, what it paid, and whether it can do the same thing next week without asking or without supervision.

A lab study published in March 2026 looked at a shopping assistant that could recommend, or could act. When Interfaces “Act for You” (https://www.mdpi.com/2414-4088/10/3/22), by Stefanos Balaskas, Kyriakos Komis, Ioanna Yfantidou, and Dimitra Skandali. 72 people did shopping tasks in a lab, with eye tracking. When the assistant acted for them, they finished faster. Trust went down. So did the feeling of being in control. Unease went up, and people were more likely to undo what it had just done. A preview of what was about to happen, plus a reason, softened the drop in trust and control. It didn’t reliably stop people from stepping in.

Which is what I’d want a checkout to do. The speed showed up in the timing. The wish to look still showed up in what people did. I don’t want the old checkout back. Password rules, a second address form, a box I have to notice so I don’t get added to a list. That stuff made it slow. I was still guessing about the total.

I want the total where I can see it. If the charge comes back, I want to see when. Not expertise and full transparency and checks constantly. Enough that I know what my card is about to do.

A faster checkout can have that second to pause and still be fast and the final decision should stay with me. Which is also why, for now, I will not be using ai personal assistants in this way.

I work with AI every day, and I keep running into the same two takes. It's going to solve everything. Or it's going to ruin everything.

I've said some version of that myself, so I'm not pretending it's a fresh insight. It just isn't a very useful place to start. I already did the longer pass on the good and the harm, with the studies, in AI for Good and for Harm. This is the other half. What we actually do with it once it's here.

There are real problems. Bad actors. Companies making decisions I wouldn't make. Privacy. Misinformation. Bias. Work getting used without permission. Surveillance, fraud, jobs, the energy cost. A lot of power sitting with not that many companies. I'm not waving any of that off.

What I don't hear very often, in the middle of all of that, is much about us. How we use it. What we hand it. What we stop doing ourselves.

AI didn't eliminate human agency. I think it made it matter more, honestly, because the tool will keep going if you let it. Including on things you probably should have kept.

We've been here before

The internet made information easy to get. It also made scams, junk, and losing a whole afternoon very easy.

Social media let ordinary people publish. Then we found out what happens when the thing that spreads matters more than the thing that's true.

Phones put a camera, a map, a bank, and a ridiculous amount of other stuff in a pocket. They also made it normal to spend four hours watching strangers organize refrigerators.

None of those were simply good or simply bad. They turned the volume up on whatever people were already doing. AI feels like that to me, except the tool is a lot more powerful and it changes fast. Deciding you're just not going to look at it doesn't really hold up as a plan.

What I'm actually worried about

A lot of the fear stays pretty vague. "AI is scary" doesn't tell me what to do on a Tuesday.

I'm worried about systems getting shipped before anyone has really checked them. I'm worried about fake things that are convincing, in amounts we didn't used to have. I'm worried about someone giving a real decision to a system they don't understand. Companies being sloppy with data. Governments using this to watch people or push them. A few companies owning the infrastructure, and it being hard to ask them anything.

That's a different list than the robots.

Not knowing how it works doesn't keep you safe from it. It mostly means you won't catch it when it's being used badly. Understanding a risk and being afraid of something you haven't looked at are not the same thing. The first one at least gives you somewhere to stand.

You don't need to be an expert

I don't think most people need to understand neural networks, or weights, or transformers.

Most of us drive without being able to explain the engine. We use electricity without being electricians. We take medication without becoming pharmacists. We still learn the dumb practical version. Don't text and drive. Don't stick a fork in an outlet. Taking twice the dose does not make you twice as healthy.

That's the level I mean. Not expertise. Enough that you aren't careless.

What is this good at. What is it bad at. When do I need to go check the answer. What should I not paste in. Am I using it to help me think, or am I asking it to think instead of me.

I ask myself that last one more than I'd like to admit.

The line I keep coming back to

Use it. Don't abuse it.

I know that sounds simple. It's still the line I come back to.

I use it when I can't get my head around something. I'll have it explain a thing a few different ways until one of them actually lands. I use it to poke at something I already believe, to sort a mess, to translate, to try a rough version before I spend real time on it. I use it on the tedious parts of the work so I have something left for the parts that need me.

The bad uses are usually quieter than people make them sound.

Treating a confident paragraph like it's true. Pasting in something you had no business sharing. Sounding like an expert in a field you don't know. Pretending to be someone else. Skipping the part where you decide.

The one I watch in myself is convenience turning into dependence. There's a gap that used to exist between having a question and getting an answer, and now that gap can be a few seconds. I wrote about what that does to thinking in AI, Dopamine, and the Death of the Pause.

A tool that lets you do more is not the same as a tool you can't really work without. I don't think the switch happens on a day you notice.

The companies don't get a pass

I get nervous writing about personal responsibility here, because it can sound like the people building this are done. They're not.

Testing it, saying what it can't do, being careful with people's data, treating misuse as their problem too. That sits with the companies in a way it does not sit with me when I open a window and type.

Schools have a version of this. So do employers, journalists, researchers, governments. Anyone putting it into something other people have to use.

"People should just be responsible" is not an answer to a system that was built carelessly. I also don't think a careless system means I get to stop thinking about how I use it. I can hold both of those at once. I kind of have to.

Not using it and not learning it are different

Not using AI is fine. I don't think everyone has to.

Not learning anything about it is a separate choice, and I think it's a worse one. You're going to run into it either way. In what you see online. In software at work. School, healthcare, banking, hiring, customer service, ads. A lot of it will never say AI on the label.

You can hate that and still be better off if you know it's there.

It's also easier to criticize something you've actually used, or at least looked at. I'm not interested in being less skeptical. I'd like to be skeptical about the specific thing in front of me, not the category.

You can try a tool and still not trust the company that made it. You can like one use and think another one is a bad idea. You can be impressed and uncomfortable in the same afternoon. I am, a lot of days.

"Is it good or bad" is not a question I can do much with. Who built this. What is it actually doing. What am I giving it. Who benefits if I use it. Where does it fail. What happens if it's wrong. What do I get worse at if I lean on it.

Those are slower. They're also the ones I can answer.

It's going to keep moving

The models will get better. New problems will show up. Some of what people are afraid of now will be right. Some of it will look quaint. In five years we'll probably be arguing about something that isn't even in this post.

I don't know where it goes. Staying uninformed until it settles isn't much of a strategy, because it isn't going to settle.

I want to know enough to hear it when someone is exaggerating what it can do, and when they're exaggerating what it can't. Enough to be a little harder to fool. Enough to ask the people building it a real question, and enough to decide where my own line is.

Then use it, if I'm going to use it.

It is already powerful. I don't think that part is really up for debate anymore. The part I can still do something about is whether I notice when I've handed it a decision that should have stayed mine.

I’ve been using agent-based modeling (ABM) to help my design and product teams think through our AI features and prioritization. This isn’t a new technique per se, but it rarely comes up in product and design work. I have found it incredibly helpful, especially when trying to explain complex abstract tradeoffs — “how much does trust matter vs. retention?” They can look at the model, see the levers, and react to something visible. Allowing us to actually pressure-test assumptions together.

Since I have found it useful so far, I want to share. I will walk through what it is, how I use it, and a simple example so you can see whether it’s useful for what you’re working on. I find that AI features don’t behave like static UI so ABMs can be really helpful tools. Because AI creates feedback loops that play out across many sessions over time. Trust builds or erodes. Users adapt. Behaviors compound. Most research methods are good at capturing a moment, but not a dynamic so much that is ever changing. Incoming….ABM!

ABM was designed to model this kind of tricky thing: individual agents with different traits, responding to their environment, producing patterns you didn’t explicitly program. That’s the same structure as a user base interacting with an AI product. It ends up fitting pretty directly as a good tool to have in our toolbelts.

What It Is

ABM is a simulation method. You define virtual users — agents — give them attributes and behavioral rules, and run them through many interactions with your product. You’re watching what happens in aggregate: whatever dynamics matter for the decision you’re making — trust, satisfaction, engagement, adoption, error tolerance, feature drop-off. You define what you’re watching.

For my work-most AI product decisions don’t play out in a single interaction. They create feedback loops. A user gets a wrong answer from an AI assistant. Their trust drops a bit. They start ignoring the feature. The AI gets less engagement to learn from. The next interaction is worse. That compounds across sessions, and we cannot catch it in a usability test or a one-time survey.

How I Use It

I had good data on how users felt about AI at a point in time, but not on what would happen to those feelings after weeks of use, or after a few bad experiences in a row. I needed a way to model that over time or start to explore the idea.

With an ABM-what I build is essentially a small simulation of my user population. I start by thinking about who my users actually are and the meaningful differences between them. Some people come in with high trust in AI. Some are skeptical. Some are somewhere in the middle. I give each simulated user a starting point that reflects that, along with any other attributes that matter for the question I’m trying to answer. The questions in my work include: how often they use the feature, how much they care about being in control, how tolerant they are of errors.

Then I write the behavioral rules. If I’ve done interviews, looked at survey data, and I know that users who get a wrong answer with no way to recover tend to disengage quickly, that becomes a rule. If I know that users who can give feedback and see the AI respond to it report higher satisfaction, that’s a rule too. The rules are my best translation of what I know about real user behavior into something the simulation can act on.

The last piece is the levers. These can be equiated to the actual design decisions I’m testing. Does the featureexplain the AI’s reasoning? Can users flag errors? How does it handle recovery? Each of these levers map directly to something that could change or may have an impact. It connects the simulation to real decisions the team is weighing.

Once all of that is in place, we can run it. The simulation steps through many interactions. Every step is one user session and we can watch how the things we care about shift across the population over time.

A Simple Example

Customer service AI works well as an example here because most people have encountered one, either as a user or someone who helped build it.

*This example happens to focus on trust, but you could just as easily model engagement, feature adoption, or how users respond to errors, etc. The structure is the same regardless of what you’re measuring.

Setting it up

Say you start with three user types: high-trust (30%), moderate-trust (50%), and low-trust (20%).

High-trust users engage readily and only escalate to a human when things get complex. Moderate-trust users try the AI first but ask for a human quickly if it misses. Low-trust users assume it’ll fail them and head straight for a human unless they’ve had a few good experiences.

Each agent has a trust score which is basically a number between 0 and 100 that starts at a different point depending on their type. High-trust users start around 75. Moderate around 50. Low-trust around 25. That score changes with every interaction based on the rules you write that again is based on the data you have about your users.

How it runs

Each step of the simulation represents one customer service interaction. For every step, each agent contacts the AI. The model then determines what happens. Does the AI resolve the issue, get it wrong, or partially help?

That outcome is either set as a probability (the AI gets it right 70% of the time) or drawn from your research data if you have it or even better a live data option!

Based on the outcome, the rules fire:

• Correct resolution → trust score nudges up by a small amount, satisfaction increases

• Wrong answer, no recovery path → trust drops more sharply

• Wrong answer, clear handoff to a human → trust drops less, satisfaction holds

• Trust score falls below 15 → the agent (user) stops using the AI on future visits

This repeats for every agent, every step, across 60 simulated days. You’re running potentially thousands of individual interactions which is why simulation is useful.

The levers

You test two design decisions. Version A surfaces a human escalation option after two failed attempts. Version B surfaces it after one. You run both versions with the same 500 agents and compare what happens.

What the output looks like

The output is typically a set of line charts showing how your key metric in this case, trust changes over time across the two versions. You can look at this for the whole population, or break it down by user type. What you’re looking for in this abm is any divergence: where do the lines separate, and which user segment is driving it?

You might see the population-level charts look similar between Version A and B, but when you segment by user type, low-trust users in Version A could start dropping off significantly around day 20 while Version B holds them longer.

Why It Helps

Surveys tell you how users feel at a point in time. Usability tests show behavior with a prototype. A/B tests show which version performed better, but only after real users are already in it.

ABM adds time. You can run six months of simulated use in minutes. In my examples you can watch what happens to a user who starts neutral and then hits two bad AI responses back to back. You can test whether improving accuracy matters more than improving recovery UX, or whether you need both. The rules you write are a statement of what you believe about your users and can help us in simulating different situations.

Getting Started

If you’re comfortable with Python, Mesa is a good place to start — it’s actively maintained and Mesa 3 (released in 2025) includes SolaraViz, a browser-based interactive visualization module that makes reading and sharing output a lot more straightforward. If you’re newer to code, NetLogo has a simpler interface and a large library of community examples to learn from.

For visualization specifically, a few worth knowing:

• Mesa + SolaraViz — built into Mesa 3, good for interactive browser-based exploration of your model as it runs (Mesa 3: Agent-based modeling with Python in 2025)

• GAMA Platform — strong if you need spatial or geographic modeling with rich visual output

• Repast Simphony — built by Argonne National Laboratory, supports 2D/3D visualization and scales well for larger models

• AnyLogic — commercial tool, steeper learning curve, but the most polished output if you’re presenting to stakeholders

You can also prompt an AI to write the scaffolding for you. The repeatable parts of an ABM — the agent class, the simulation loop, the output charts in Mesa or Plotly — are templatable. Describe your agents, rules, and levers in plain language and have it generate a working starting model. I’ve been building this out as a reusable AI skill. The parts that still need you are the behavioral rules — those have to come from actual research.

Keep the first model small. Two or three agent attributes, three or four rules, one lever to test. Run it, look at the output, and ask: does this match what we’d expect? Where’s the surprise?

Using AI to Help Build It

Once you have a working base model, it becomes reusable. The agent types, the trust and satisfaction mechanics can carry over. Updating the rules and levers as new questions come up becomes easier. The model becomes something you return to rather than rebuild.

I’ve also been thinking about this being something AI can help us generate. The repeatable parts of an ABM — the agent class, the simulation loop, the output charts, are templatable. I’ve been developing this as an AI skill that can be used in codex, claude, cursor, etc: basically if you describe your behavioral assumptions in plain language, it writes the model code and a simple UI for adjusting levers and reading results. The part that stays yours is knowing your users well enough to write the rules and tell the AI how to build the models and the code.

What It Doesn’t Replace

ABM is only as good as the rules you put in. If your behavioral assumptions are wrong, the simulation produces a coherent but wrong story. It doesn’t replace talking to users it depends on having done that work first and is a powerful thought tool within Product Design and Research.

I want to be clear and say that it also doesn’t replace testing. A simulation result is a reason to prioritize something in real experimentation, not a reason to skip it. Its value is in helping to potentially narrow the space of what you’re testing and surfacing assumptions before they cost us.

Happy to talk through how to set this up for a specific product or feature if you’re working on something where this might fit. Or if you have any tips and tricks you would like to share!

ABM MODELING IN PRODUCT DESIGN & RESEARCH was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.

A personalization system treats clicks, saves, filters, and bookings as signals, and one action can mean more than one thing.

I have been thinking about what happens and how we think about the process of how and when personalization systems learn from the things people do. In travel for example, a person looks at a few hotels, saves a destination, filters by price, maybe ignores a recommendation, books something, and the system starts adjusting around that. More of this. Less of that. Different rankings. Different suggestions. A slightly different version of the experience the next time they come back.

What does the system think it learned from that behavior? And did the behavior actually mean that?

Because the raw action itself is easy to identify or sort into a box or rank in theory. Someone clicked. Someone saved. Someone booked. Someone corrected the assistant. Okay, fine. But the meaning of that is not always as obvious as that.

Someone can click into five expensive hotels because they want expensive hotels. They can also click into five expensive hotels because they are trying to understand what expensive even means in a city they have never been to.

Someone could save three places because they are seriously considering them. They can also save three places because they are not ready to decide and do not want to lose them. How do we know the difference? This is not only travel specific. It is agnostic to any one industry or software.

A good starting is to think about people’s behavior. There’s always a trail or patterns. But getting from that to “the system thinks this user prefers X” is where it gets really fuzzy really fast.

Every one of those actions can mean a lot of different things depending on what was going on at the time. For example, I’ve saved things I would never actually buy, just to show someone else later. And then later, that save starts coming back around in the experience. Recommending more of that. Filtering toward it. Acting like it learned something really significant about me. In this case, it didn’t.

In that situation the moment became behavior that turned into assumption and eventually a “signal”. Because once the the system starts learning, it doesn’t forget so easily. It carries things forward. It builds on them.

So we are no longer looking at anything in isolation. We are looking at how something someone did three sessions ago or in a different part of the experience and it is shaping what is seen in the now and as the experience continually adapts for them and around them.

What I mean when I say “signal”

A signal, in a more meta sense, is something that points to something else. Smoke is a signal of fire. A pause in conversation can be a signal that someone is unsure, annoyed, thinking, distracted, or trying to be polite. What I am trying to say is, a signal can mean a many number of things.

In personalization and in the context of research, a signal is usually an observable piece of data the system can use to learn or make a decision from. It might be a click, search, save, filter, skip, booking, rating, correction, return visit, dwell time, or repeated pattern over time.

The system gets traces of behavior. Some are explicit. Like a person directly telling the system or experience something, or taking an action that states a preference like applying a filter or saying “I want this.” Others are implicit. A person may not directly state a preference, but behavior may suggest one like repeatedly choosing similar things or comparing choices.

It is hard to look at an action without context wanting to know what else was happening. Was this person actually trying to decide something, or were they just poking around? Were they doing this for themselves, or for someone else? Were they early in the process, or trying to get it done?

Were they exploring? Comparing? Narrowing? Committing? I think the point there is made.

When one thing counts more than another

As I have mentioned, not all of these signals carry the same weight. But saying that is easier than underestanding how it actually works. So I’ve been trying different mental models and approaches to this.

Because not every signal should live in the same place

Some signals will be “evergreen”. Meaning they show up across sessions or contexts. They are not based on one click or one temporary need. They repeat enough that they might actually say something meaningful about the person over time.

Other signals are more task or “situation specific”. They matter a lot, but only inside the container of that situation or task. The destination for a trip, the season for clothes shopping, the budget, the reason, etc. Something can be very true for this situation and not true generally for every situation.

And some are really just “live-moment” signals. They help the system respond to what is happening right now, but they should not necessarily follow the person around later. Like a chat sessions remembering what was said in the previous message in the same session.

Before the system decides how much to remember, how strongly to weight something, or how fresh something is-we need to ask what kind of thing we are looking at. Is this about the person more generally, about this specific situation or task, is this about the current moment, or is this just something that happened and we do not know enough about yet?

The rough model in my head right now looks something like this:

This helps me think about how fresh does not always mean more true. Old does not always mean irrelevant by default. A click from yesterday might have been curiosity, confusion, or price-checking. A repeated pattern from last month might be more meaningful because it has shown up across different contexts. So the question is what kind of signal is this, should it shape this moment, this situation, or the evergreen understanding of the person?

How I am researching this

The framework is useful as a way to think, but I also want to start to test whether it actually holds up against what people say, what they do, and where personalization seems to help or break down in real experiences.

One piece is VoT, support feedback, open-ended survey comments, and past research.

I am looking for the places where people are already describing the effects of personalization, even if they are not calling it that. I am looking specifically at how people talk about memory and preferences. What they want a product to remember, what they expect it to know, what feels useful when it carries forward, and what feels wrong when it does.

Another is a Max Differential. MaxDiff is a research method that forces people to make tradeoffs between options.

Instead of rating everything as “important,” participants choose what feels most and least important from small sets. That helps show the relative weight people place on different signals, contexts, or factors.

Then the next layer is connecting that back to the products I work on and starting to create capture blueprints in tandem.

Where are we already making assumptions from behavior? Which signals are being treated as durable when they might be temporary? Which ones are being underused because they do not fit neatly into a current model? Where might we need more transparency, correction, or expiration?

Streaming ads are often louder than the show because the ad is inserted separately. California’s SB 576 says that audio can’t be louder than the program.

I used to think the volume jump when a streaming ad starts was my TV acting up or a symptom of my sound bar. Sorry sony, I was really dogging you for a whule. It isn’t.

Let’s start with some background on this- the law. The CALM Act passed in 2010 and took effect in December 2012. It requires commercials on broadcast and cable TV to match the loudness of the show around them, measured in a unit called LKFS (loudness, K-weighted, relative to full scale). Most networks target -24 LKFS, with about 2 decibels of tolerance built into the FCC rule.

Every piece of broadcast content carries a metadata tag called dialnorm that tells your TV’s decoder how loud the dialogue is, so the decoder can adjust playback and keep everything level. This works because one party, the network, controls the entire signal from show to commercial and back.

Streaming was never built that way. Your show comes off the platform’s own servers. The ad is frequently inserted by a separate ad network through a process called server-side ad insertion, stitched into the stream seconds before it reaches your screen. The CALM Act never applied to this pipeline because the law predates streaming carrying any meaningful ad load. Nobody updated it. For over a decade there was no requirement that any party in that chain agree on a loudness target, so in practice none of them did.

There’s a second, more mechanical problem specific to short clips. Loudness normalization software needs a few seconds of audio to measure a clip and settle into its target level. Ads run 15 to 30 seconds. That leaves a real spike zone, often the first 2 to 3 seconds, where the normalizer hasn’t caught up yet and the audio can run hotter than everything around it, even on a platform that’s trying to normalize correctly.

Then there’s the part that isn’t a technical accident. Advertisers know people leave the room, mute the TV, or pick up their phone the second a show cuts to commercial. A Sharethrough study found 46% of people mute commercials specifically because they find them too loud or annoying, and a separate study put the share of viewers who actively pay attention to any streaming ad at just 24%. Loud is one of the few tools left to fight for a few seconds of attention from someone who has already mentally checked out. Mixing some ads hot is a bet that volume buys back attention the content alone wouldn’t.

But honestly, the louder-ad problem sits inside a bigger one: people are fed up with streaming ads generally. A March 2026 survey of 1,000 U.S. adults by All About Cookies found 76% think streaming platforms run too many ads, and 52% have considered canceling a subscription over it. Hulu (71%) and Peacock (68%) topped the list of platforms named specifically for ad load. Loud ads and frequent ads aren’t two separate complaints. They’re the same complaint showing up twice.

California is the first to put regulations on this in motion. Senate Bill 576 takes effect July 1, 2026, and requires streaming platforms to keep ad audio no louder than the content it interrupts, the same standard broadcast has followed since 2012. State Senator Tom Umberg wrote the bill after his own legislative director complained that loud streaming ads kept waking his newborn daughter.

Illinois has a similar bill moving through its legislature, targeting 2027. The Motion Picture Association and a coalition representing more than 20 streaming platforms opposed SB 576, arguing that platforms don’t always control the audio on ads they didn’t produce and don’t always sell directly. That’s a true statement. It’s also a description of the exact structural gap that let this go unregulated for over a decade: a lot of parties touch the signal, and none of them were required to own the problem.

None of this is really about decibels on their own. It’s also about what a startle does to trust, is it actually benefiting the products they are trying to sell. The moment something in your own living room gets measurably louder without your permission, you lose a small amount of control over your environment, and the platform gives no sign of noticing or caring that it happened. That’s a cheap trade for three extra seconds of attention on an ad that, per the numbers above, most people weren’t going to watch anyway.

So to the loud af ads on all my streaming services, I will not be buying your products. In fact, I now feel disdain for your loudness and repetitiveness. But repetitiveness is for another day.

Sources:

• California bans loud commercials on Netflix, Hulu, and other streaming services

• California Law Strikes Back Against Loud Ads

• Illinois bill would ban loud ads on streaming services starting in 2027

• SSAI as an encoding challenge

• Dialnorm — Wikipedia

• Streaming Ads: Less Than One-Quarter of Viewers Actually Pay Attention

• New Research: Understanding Consumer Behaviors During TV Commercial Breaks — Sharethrough

Deep research tools can read fifty sources in the time it takes you to open five. That’s not the hard part anymore. The hard part is what you do with fifty sources of evidence once you have them. The actual time it takes to thoughtfully review fifty sources is helpful, but honestly, that is just more sources you have to sift through.

Here’s the failure mode: you ask for “research on X,” get back a long, well-organized report, skim it, feel informed, and then make the same call you would have made anyway. The report didn’t change anything because it was never built to.

The discipline here isn’t new. Anyone who’s done real research with Google Scholar already knows the moves: narrow the query instead of typing in a general subject, use boolean operators to exclude noise, check who published a source and when, and follow citations back to the original data instead of trusting the abstract. Deep research tools run on the same logic, just with a lot more horsepower and a lot less friction.

That horsepower is why skipping the discipline part is costing you more. A bad Google Scholar search wastes ten minutes. A bad deep research prompt produces forty confident, well-cited pages built on the wrong premise and not answering the questions you need.

If you can name the decision you’re making

Compare these two prompts:

1. Research the market for project-management software.

2. We’re deciding whether to build an approval workflow for mid-market marketing teams this quarter. What evidence suggests this is a painful, underserved problem? Which existing products solve it, where do users complain, and what would make switching unlikely?

Before you run anything, write down three things:

1. The decision. What will you do differently after this research?

2. The current assumption. What do you already believe?

3. The disconfirming evidence. What would prove you wrong?

The third one is the one a lot of people skip, and it’s the one that matters most. It’s easy to collect facts that make a favorite idea sound inevitable. But watch what happens when you ask the research to go looking for reasons not to do the thing.

Ask what the research couldn’t find

A report that summarizes everything it found is not that useful on its own. A report that also tells you what it couldn’t find is.

Instead of a generic summary, ask for findings organized into categories that map to how a decision actually gets made: the underlying need, how people currently work around it, direct and adjacent alternatives, recurring complaints, real constraints (technical, legal, market), and open questions that need a real person to answer.

That last category is the payoff. A review complaining about “too many approval steps” is a lead, not an answer. Is the problem the number of steps, unclear ownership, audit requirements, slow notifications, or a process that shouldn’t exist at all? A tool can surface the complaint. Only a conversation, an observation, or product data can tell you which one it is.

Check where every claim comes from

This is the same source-weighting judgment a grad student applies to a literature review, just aimed at a business question instead of a dissertation. A vendor’s landing page tells you how they want to be seen. It doesn’t tell you whether customers are happy. A five-year-old article might be useful context in a slow category and actively misleading in a fast one.

Tell the tool how to weigh what it finds:

Prioritize primary sources: official documentation, filings, original research, public data, and direct customer feedback. Separate vendor claims from independent evidence. Flag any claim that rests on weak, old, or anecdotal sources.

Ask for a citation next to every claim that would change your decision, then actually open two or three of them. Check that the source says what the summary claims it says. This takes minutes and it catches the most expensive kind of research mistake: a confident conclusion resting on a source nobody read.

Plan the conversation that comes after

A research run that ends in a deck and no next step was a research run that didn’t need to happen.

Ask for a short validation plan instead:

• What are the riskiest assumptions in here?

• Who should we talk to first?

• What question would separate a real pain point from a minor annoyance?

• What’s the fastest way to test this with actual behavior, not a survey answer?

• What result would count as a no?

The real value of deep research and super power. If you’re evaluating an AI feature that drafts support replies, the research can surface demand, likely competitors, and the objections support teams will raise.

What it can’t do is tell you what actually slows someone down when they write a reply. For that, you still have to ask them: walk me through the last reply that took too long. What did you need that you didn’t have?

“If you’re a bird, I’m a bird” is the line from The Notebook that outlasted the movie, and this is what that line is doing.

I am making this post for my mom, she reads my substack…sometimes. Let’s see how long it takes her to find it. For everyone else, thanks for reading ;)

Growing up, we watched The Notebook over and over and over. It was my mom’s favorite, and after a while, it became all of ours too. It is one of those movies, that when we see it on we immediately think of all of us siblings and mom settling in on a Sat to watch.

For me, the part that still gets me is the ending, the two of them dying together, holding hands (sorry for the spoiler, the movie’s been out for twenty years). But there’s also a line in it that lives well beyond the movie itself, quoted at weddings and captions and just everywhere.

I thought it would be fun to dig into why that line has lived on the way it has. Not the romance of it, but the mechanics: what made it stick for twenty years across people who’ve forgotten almost everything else about the movie.

So…here is what I found. The answer runs through four separate branches of psychology, none of which were written about a movie. Staying close without disappearing into someone. Absorbing another person’s way of seeing the world as your own. One choice can cause rerouting an entire life. Closeness that doesn’t cost you your autonomy. Put that all together, and it brings about a compelling argument of why this line has so much power.

Reason one: it’s not asking you to disappear

The first branch is family systems theory, built by the psychiatrist Murray Bowen. His whole career circled one tension: every person is pulled between wanting closeness and wanting a self of their own. He called the failure mode fusion, where the boundary between two people collapses and one or both lose track of where they end and the other begins. Low differentiation of self means getting swallowed by a relationship. High differentiation means staying attached without disappearing into it (scoping review, ScienceDirect).

The movie’s supporting cast keeps pushing this version of love onto the two leads: stay still, stay familiar, don’t get complicated. It’s also the version a lot of us have lived through, where someone’s separateness gets treated like a threat instead of a fact.

The bird line is a promise of the opposite: staying close to someone while leaving them fully intact, which isn’t the natural default. It’s the harder version of love, said in seven words.

Reason two: someone else’s way of seeing sticks to you

The second branch is the self-expansion model, built by Arthur and Elaine Aron in the 1980s. The idea: people are wired to grow their own capability and perspective, and one of the main ways they do that is by absorbing another person’s resources, viewpoints, and identity into their own (self-expansion model, Wikipedia; ResearchGate). You don’t just love someone. You take in pieces of how they see the world, and those pieces stay with you.

It’s someone taking the scenic route because a person no longer in their daily life once taught them to look out the window.

It’s a literal description of what loving someone does to you: their way of seeing becomes part of the equipment you see with. It shapes the way you see, feel, live, etc.

Reason three: the movie’s whole plot is a documented pattern

The third branch is life course theory, built by the sociologist Glen Elder. Two of his core ideas explain the part of the story that has nothing to do with romance. Linked lives: a person’s choices and transitions ripple outward and change the options available to the people connected to them, for better or worse (ResearchGate). Turning points: moments that redirect an entire life’s trajectory, even though they rarely look significant while they’re happening (Turning Points in Psychology).

It’s the idea running under The Notebook’s whole plot. A letter that gets intercepted. A mother who’s sure she knows better. A single summer that ends up being the thing two people measure the rest of their lives against. Elder’s model predicts this pattern directly: a choice rarely announces itself as a turning point while it’s happening, and it’s only years later you can see it rearranged everything that came after.

By the time two people are holding hands at the end of their lives, you’re watching the sum of every one of those small moments or choices.

Reason four: it names the test for whether love is good for you

The fourth branch is self-determination theory, built by Edward Deci and Richard Ryan around three needs people share regardless of culture or age: autonomy, competence, relatedness. Meet all three and motivation and well-being hold up. Undermine any one and they don’t (SimplyPsychology). Their later work, Relationships Motivation Theory, applies that directly to partnerships: closeness that chips away at your autonomy, a partner who wants you near but wants you smaller, doesn’t support well-being no matter how connected it feels (Ryan and Deci, SDT documents).

That’s the test the bird line passes. It treats your freedom as something worth protecting.

Why the line and the ending are the same idea

Four theories, one arc. Fusion is the failure mode. Self-expansion is what happens when it goes right. Linked lives and turning points explain why none of it happens in a vacuum. Self-determination theory is the test: did the closeness leave your autonomy standing, or did it need you smaller.

The ending and the line are making the same argument from opposite ends. Two people holding hands as they die is the payoff of a life spent staying, freely, over and over, decades of turning points that could have gone another way. The bird line is the promise that makes that staying possible without either person disappearing.

If you’re a bird, I’m a bird was never about becoming the same shape as someone else. So mom-if you’re a bird, I’m a bird.

• Differentiation of self: A scoping review of Bowen Family Systems Theory’s core construct

• Self-expansion model, Wikipedia

• Love and expansion of the self: The state of the model

• The Linked Lives Principle in Life Course Studies

• Turning Points in Psychology: Therapy and Life-Course Change

• Self-Determination Theory of Motivation, SimplyPsychology

• Self-Determination Theory and the Facilitation of Intrinsic Motivation, Ryan and Deci

You can add free analytics to a GitHub project, a GitHub Pages site, a portfolio, or a small blog without building a dashboard.

I never really realized how easy it is to track analytics on a GitHub Pages site or a personal site until I looked into it recently. There are a lot of slick ways to build an AI-powered analytics dashboard now. But it doesn’t have to be complicated.

If you have a GitHub project, a GitHub Pages site, a portfolio, docs, or a small blog, you can get analytics for free without much setup. Things like: are people finding this? What did they click? Did that post send anyone over? Is the README doing its job?

That’s plenty to start with.

Before you add anything, pick one thing you want to know. Maybe it’s:

• Are people landing on the repo after I share it?

• Which page do they look at first?

• Is anyone making it past the homepage to the docs?

• Did my launch post do anything besides give me a nice dopamine hit?

For a small project, I mostly care about four numbers: visitors, where they came from, the page they visited, and one action that matters to me. That action could be a star, a docs click, an install, a signup, whatever fits your thing. You can learn a lot from just that.

Why not just use Google Analytics or Squarespace?

Squarespace analytics only works if your site is built on Squarespace. A GitHub Pages site, a docs site, or a bare HTML page isn’t on Squarespace, so that option’s off the table for most of what this post covers.

Google Analytics is the obvious universal answer, and it works fine. The reasons I’d skip it for a small project: GA4 has a steep setup curve for something this small, it pulls in a lot of tracking you don’t need, and depending on where your visitors are, it can trigger cookie consent requirements you’d otherwise avoid. For a personal site or a small repo, that’s a lot of overhead for four numbers.

GitHub Insights is already there, no setup at all. Cloudflare Web Analytics skips cookies and heavy tracking scripts entirely, which is why it’s lighter to add and to live with. If you’re already deep in the Google ecosystem and don’t mind the setup, GA is a reasonable choice too. This is just the lower-effort path for a small project.

GitHub already has some of this

If your project is in a public repo, go to Insights → Traffic, replacing OWNER/REPO with your GitHub username and repository name.

GitHub shows visitors, full clones, referring sites, and popular content for the last 14 days. After you share a project somewhere, you can usually tell whether it led to people checking it out.

After a few weeks, patterns show up. Maybe a tutorial brings in more people than a release announcement. Maybe everyone goes straight to the install docs and your README needs a clearer next step. Maybe nobody’s clicking the demo link. That’s useful, even at small numbers.

For GitHub Pages or a personal site, try Cloudflare Web Analytics

If you have a site alongside the repo, start with Cloudflare Web Analytics. It’s free, lightweight, and gives you page views, visitors, referrers, and some performance data without dragging you into a dashboard rabbit hole.

You don’t have to move your whole site to Cloudflare to use it. Create an account, add the site, and drop a small JavaScript snippet in the <head>. If you’re on Cloudflare Pages, it’s even easier.

For a Jekyll GitHub Pages site, put the snippet in your shared layout so it shows up on every page. For plain HTML, add it to the shared header or each page.

Then leave it alone for a week. You don’t need to check it every hour.

Page views are nice. Actions are better.

Someone opening a page is a start, but what you want to know is whether they did the thing the page was built for.

Pick one or two clear next steps:

• Read the quickstart

• View the demo

• Install the package

• Join the newsletter

• Open an issue

Make the links obvious. When you share the site, use a UTM link so you can tell where people came from:

https://your-site.example/docs?utm_source=linkedin&utm_medium=social&utm_campaign=launch

It looks a little nerdy, but it means you can see later whether LinkedIn, your newsletter, or a community post brought people in. Keep the labels boring and consistent. Use linkedin every time, not LinkedIn one week and LI-post the next.

If you want to host it yourself

There are open-source options too. Plausible Community Edition is free to self-host and works well if you already like running your own small services.

One caveat: free software isn’t free work. You still need a server, backups, updates, and patience when something breaks. If that sounds fun, go for it. If it sounds like a project you’ll avoid for six months, stick with GitHub Insights and Cloudflare. No shame there.

I wouldn’t self-host analytics just for a cleaner chart. Do it for the control, the privacy, or the learning experience.

Keep a small weekly record

A weekly record is what helps most.

Once a week, spend ten minutes writing down what changed:

• I published a guide.

• I fixed the install instructions.

• I shared the project in a Slack group.

• I added a demo video.

Then look at the traffic around that change. You’re not trying to prove a theory. You’re collecting enough signal to make the next call easier.

Maybe you write another guide. Maybe you move the demo higher on the page. Maybe you stop posting somewhere that never sends anyone over. Small adjustments are the whole game.

A low-effort setup

If you’re starting from nothing, here’s what I’d do in half an hour:

1. Open GitHub Insights → Traffic and write down today’s numbers.

2. Add Cloudflare Web Analytics to your site.

3. Pick one action you care about, like a docs click or signup.

4. Make a tiny weekly spreadsheet.

5. Add UTM tags to the next link you share.

That’s it. You don’t need to track every click or turn your personal site into a surveillance machine. You need enough data to see what’s landing.

And when it helps you decide between writing the next tutorial, fixing the README, or sharing the project somewhere new, it’s doing its job.

Links to get started

• GitHub repository traffic docs

• GitHub Pages

• Cloudflare Web Analytics

• Cloudflare Web Analytics setup docs

• Plausible Community Edition

• Google Sheets

When more people can make the website, the app, or the deck, polished is no longer what makes it good. Judgment, context, and responsibility are.

Making a website, dashboard, a game, an app, a slide deck, image, short video, or first-pass analysis used to require more specialized access than it does now. The tools have become easier to use, cheaper to have, and better at handling parts of the work that once took some serious skill and time to learn. As we know, now, a person with an idea can move from a blank page to something presentable in minutes.

Which has a lot of us wondering: If more people can make the thing, what becomes the definition of good?

“Polished” is no longer enough. It is easier than ever to make something look finished while leaving the important bits out.

• A dashboard can be clean and readable while answering no useful questions.

• A site can have elegant copy and no clear path for the person who needs to use it.

• An analysis can include charts, confidence, and recommendations without a sound basis for any of them.

Good begins before the output

The first question is still: what is this for?

A dashboard should help someone notice, decide, or act. If it gives a team thirty measures with no indication of what to pay attention to, it is closer to a display case than a working tool. A site should help someone complete a task, understand an offer, find an answer, or make contact. A report should make a claim that its evidence can support.

The tools can generate options, but they cannot settle the purpose without help. Someone has to decide which question is worth answering, whose needs count, what information is relevant, and what can be left out.

This is where a lot of “good” lives: in selection and omission.

Herbert Simon wrote in 1971 that an abundance of information creates a scarcity of attention.[1] His point feels useful here. More content, more charts, more variations, and more possible outputs do not automatically create more value. They make attention harder to use well. A good piece of work respects the attention of the person receiving it. It does not confuse completeness with care.

A good output has judgment behind it

Judgment is sometimes mistaken for taste, as if it only means knowing whether something looks good. It includes taste, but it is broader than that.

Judgment is knowing that a chart needs a comparison point. It is knowing when a number is too uncertain to carry the claim being made about it. It is recognizing when a “simple” workflow works only for people who already know the language, have a reliable internet connection, or can afford to make a mistake.

It is also knowing when not to build something.

A team might ask for a dashboard because they have data, when a weekly conversation and a short list of open questions would serve them better. Someone might request a new site when the more pressing problem is that customers cannot get a response from the current one. The availability of a tool can make a new object feel like progress. Sometimes it is. Sometimes it gives a more visible shape to a problem that needs different kind of work.

The context changes the result

In research, like I have touched on in the past, it is common to find a gap between the official version of a process and the version people use to get through their day. A system may have a clear flow on paper. In use, people may be interrupted, working from a phone, switching between tools, looking after a child, handling an upset customer, or trying to finish before a deadline.

Lucy Suchman’s work on situated action challenged the idea that people carry out plans in a neat, pre-scripted way.[2] What people do depends on the conditions around them. That is true for products and services, but also for dashboards, analyses, and creative work. An output that looks good in a quiet review meeting can become unusable when it reaches the person with less time, less context, or greater consequences for getting it wrong.

So, it is even more valuable when we talk about what the definition of “good” is. A good site does not assume the visitor has read the page above it. A good dashboard does not assume every viewer knows how a measure was calculated. A good analysis does not assume a neat dataset reflects a neat world. A good recommendation names what it does not know.

This kind of work can look less impressive from a distance because it spends time on things that do not always show up in the final image: testing, checking, revising labels, removing unnecessary steps, verifying a source, and asking who might be excluded. Those are not decorative touches. They are part of the work.

Faster tools do not remove responsibility

The promise of automation is often relief from repetitive work. That relief can be real, sometimes, if time is protected to allow for it. It can free people to spend more time on questions, relationships, and decisions that need attention.

It can also create a kind of distance from the work.

Lisanne Bainbridge wrote about the “ironies of automation” in 1983. In highly automated systems, people are often left with the tasks that are hardest to automate: monitoring, intervening when something goes wrong, and handling unusual cases.[3] The basic point extends beyond industrial systems. When a tool produces a first draft, a chart, a recommendation, or a whole page of code, the human role may appear lighter. In many cases, it becomes more concentrated. Someone still has to decide whether the output is sound, whether it fits the situation, and what happens if it is wrong.

That is not a reason to avoid tools by any means. I want us to be more thoughtful. It is a reason to be clear about where responsibility sits.

• A good analysis is not only one that returns an answer. It shows enough of its basis that someone can question it.

• A good dashboard does not turn a complicated situation into a score without explaining what the score leaves out.

• A good website does not make a promise that the organization behind it cannot keep.

The research on automation bias is extremely relevant here. Reviews of the field describe a tendency for people to rely too heavily on automated recommendations, especially in consequential settings.[4] When an output arrives quickly and in a confident voice, it can feel more settled than it is. That feeling is part of the design problem.

Craft still has a place

None of this means craft or expertise no longer matters. In fact, I would say, like many others, it matters no even more so.

Good visual hierarchy helps people find what they need. Good writing lowers the effort required to understand something. Good interaction design makes a service less frustrating. Good analysis can make a complex pattern easier to see. These are worthwhile skills, and tools can help more people develop them.

The difference is that craft cannot be the whole definition of quality. A beautiful thing can still be confusing. A fast thing can still waste time. A clever thing can still make a poor judgment. An output can be technically correct and still fail the person it was made for.

The more accessible production becomes, the more visible these distinctions become. The baseline rises. It becomes less remarkable that a person can make a site or chart. It becomes more noticeable whether they understood the problem, made careful choices, and left someone better able to act.

Sources

[1] Simon, Herbert A. “Designing Organizations for an Information-Rich World.” In Computers, Communications, and the Public Interest, 1971.

[2] Suchman, Lucy. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge University Press, 1987.

[3] Bainbridge, Lisanne. “Ironies of Automation.” Automatica, 1983.

[4] Langer, Mareike, and colleagues. “Exploring Automation Bias in Human–AI Collaboration: A Review and Research Agenda.” AI & Society, 2025.

People get asked to describe how they make decisions, do a task, use a product, get through their day. What they say back is useful. It tells you what they remember, what they value, what feels normal to them, how they explain their own choices to themselves. It’s also usually only half the picture.

The other half shows up when someone can show you the task. Rather than describing it from memory. A process that sounded simple over turns out to be a process that starts with a sticky note by the monitor, a spreadsheet on a personal drive, a coworker who just knows where things are, etc. People don’t hide these details on purpose. The task has just become so much muscle memory people forget. Do a workaround long enough and it stops being a workaround. It’s just how the task gets done.

Most of what people do is built from habit, time pressure, and whatever tools they have available to them. Someone can know exactly what they’re trying to accomplish and still not walk you through every step they took to get there. Lying and carelessness have nothing to do with it. When I work with AI, I often find it hard to explain in every concise detail how I got to the finish line.

Psychologists Richard Nisbett and Timothy Wilson made this case back in 1977, in a paper called “Telling More Than We Can Know,” reviewing evidence that people have limited access to the mental processes behind their own judgments.[1] Ask someone why they made a call and you’ll get an answer that sounds reasonable and sincere. Whether it’s the real answer is a different question.

Interviews still matter, though. And so does what people say. People are the best source for huge parts of the story: what they were trying to do, what felt right, what felt wrong, what they’d tried before, and what happened. Someone’s account of their own experience is real evidence. It just can’t carry the whole picture itself.

Memory adds its own distortion. One study compared cigarette use logged in the moment against reports of the same behavior given later from memory. The later reports misremembered and rounded.[2] That study was about smoking, but the same thing happens everywhere. Ask how often someone uses a service, how long a task usually takes, where they get stuck, and the answer you get is shaped by whichever example is most recent, most annoying, or easiest to turn into a story.

Take a form. Ask someone about it in an interview and they’ll probably say it’s fine. Watch them fill it out and you’ll see them pause at a term they don’t recognize, open a new tab to look something up, text a family member before they hit submit. They’ll still call it fine, because they got through it. Fine and easy aren’t the same thing, and the gap between them rarely makes it into the interview.

Or take someone who says they follow the standard process. What they usually mean is they follow it as closely as they need. Mid-shift, they’ll skip a step that feels redundant, jot something down because it’s about to disappear from the screen, ask a coworker to sanity-check a call before moving on. Training, time pressure, bad system design, old habits, a mistake from three months ago they never want to repeat again: any of these could be behind it.

Watching alone also doesn’t hand you the answer. It hands you a much better question.

People say they compare prices, then buy the first familiar brand they see. They say they’d switch to a faster service, then stay with the slow one because switching feels like more effort than it’s worth. They say they don’t use a feature, and later it turns out they tried it once, didn’t get it, and never went back.

A decision like that depends on conditions: familiarity, confidence, what’s socially expected, fear of getting it wrong. Most people don’t have language for those pressures until someone asks about it.

Observation surfaces the stuff nobody thinks to mention: interruptions, handoffs, missing information, a step repeated three times because nobody trusted it the first two. It also shows where a system is quietly asking a person to hold more than they should. A sticky note or a personal checklist is usually a sign that something important got left out of the design.

Observation has its own blind spot. You can see someone pause, look away, take a different route through a task. You can’t know why without asking. A pause might mean confusion. It might mean someone double-checking their own work. A workaround might be covering for bad design, or it might be the smartest way to handle a case nobody planned for. The person doing the work is still the only one who can tell you which.

K. Anders Ericsson and Herbert Simon drew a related line in their work on verbal reports: the difference between someone talking through a task while they’re doing it and someone explaining it afterward.[3] Timing changes what a person can tell you. Looking back, people tend to smooth events into a story that makes sense. Talking in the moment surfaces details that would otherwise get lost, though it changes how the task feels to do. Both versions are useful, for different reasons.

Interviews teach you the language people use for a task, what worries them, the history behind a choice. Observation shows how the task actually runs under real conditions. Usability testing shows where a design forces a guess or asks someone to remember too much. A follow-up conversation fills in why someone picked one path over another.

The questions don’t need to be clever. “Can you show me how you did that last time?” gets more out of someone than “How do you usually do it?” ever will. “What were you looking for there?” explains a pause. “What’s that note there to remind you of?” often surfaces a responsibility the system quietly pushed onto a person. “When does this get hard?” beats asking whether something is easy, every time.

This works anywhere people are involved. Building a product, trying to understand someone in your life, or just the type of person who can’t stop noticing how people behave. Listen to what someone tells you. Watch what they actually do. The two won’t match half the time, and that’s fine. That’s where the good stuff is.

Sources

[1] Nisbett, Richard E., and Timothy D. Wilson. “Telling More Than We Can Know: Verbal Reports on Mental Processes.” Psychological Review, 1977.

[2] Wang, Hao, Saul Shiffman, Sandra D. Griffith, and Daniel F. Heitjan. “Truth and Memory: Linking Instantaneous and Retrospective Self-Reported Cigarette Consumption.” Annals of Applied Statistics, 2012.

[3] Ericsson, K. Anders, and Herbert A. Simon. Protocol Analysis: Verbal Reports as Data. MIT Press, 1993.

UX research and human factors are related fields. Both study how people interact with products, tools, and systems, and both aim to make those interactions work better. The methods overlap substantially, but the two disciplines ask different questions, and that difference shows up fairly obviously throughout projects.

Researchers in both fields watch people do tasks, ask them questions, test designs, trace handoffs between people and systems, and look for the moments where something goes wrong or someone gets stuck.

Earlier in my career, I worked as a quant researcher on the global expansion of Stories and Reels.

The process was mostly comparative: run the same core research question in a new market, then look for where behavior was different from what we’d seen elsewhere. A format built around one audience’s habits doesn’t always translate, and the research existed to catch that gap before it turned into low adoption nobody could explain.

A few other examples make this more concrete for me. A team studying a sign-up flow might find that people drop off at the payment step, not because the price is wrong but because the form asks for a phone number before it explains why.

A team comparing subscription tiers might watch people bounce between two plans, unable to tell what they’d lose by downgrading, and end up picking the more expensive option out of caution rather than preference.

I also spent time on DOD-funded exoskeleton research at Shirley Ryan AbilityLab, and the process there ran differently from the start. It began with breaking a movement down into its pieces, whether that movement belonged to a patient relearning to walk or a soldier carrying weight over distance. From there it moved into measuring how the body actually moved, then into testing that kept inching closer to real conditions rather than staying in a lab.

The real question was whether the device kept working once someone got tired, stressed, or thrown into a situation nobody had planned for.

A hospital’s medication administration system might let a nurse select the wrong dosage from a dropdown because the units aren’t visible without scrolling, and a nurse four hours into a double shift, getting paged mid-task, is the person likely to miss it. An aircraft cockpit might have two warning lights that look nearly identical under low cabin lighting, so a pilot glancing over during a high-workload moment reads one for the other.

The methods overlap in some ways.

Interviews, watching people do the task, and testing something before it ships: both fields use all of these. What changes is what each side is looking for.

Take a payment flow. A UX researcher asks whether people know where to start, understand the labels, and feel confident before they submit. A human factors practitioner looks at the same flow for error opportunities: is the confirmation visible, does the interface make a risky action too easy to trigger, and can a distracted person catch a mistake before it goes through.

Now take a medication order entry system in a hospital, since it shows the contrast even more clearly. A UX researcher might ask whether a physician can find a patient’s chart quickly, whether the drug search returns the right results on the first try, and whether the ordering flow matches how doctors think through a prescription.

A human factors practitioner looks at the same system and asks something even deeper: if a physician is interrupted mid-order, does the system make it obvious the order is incomplete? If two drug names look alike on screen, is there a safeguard that catches the mix-up before the order reaches a pharmacist? Both researchers might sit in on the same shift and watch the same screen.

UX research usually asks whether a product fits people’s needs and expectations. Human factors asks bigger: can people do the task safely and consistently, under the real conditions they’ll be doing it in.

That difference carries into what each field measures. UX research tends to track things like whether people finish the task, how long it takes them, and how satisfied they feel afterward. Human factors tends to track things like how often people make an error, how quickly they react, and how physically demanding the task is.

The two fields show up at different points in a project and sometimes converge

UX research can shape early product decisions, test concepts, refine designs, and keep watching a product after launch. Human factors is more likely to show up when requirements are being defined, risks assessed, a design validated, or the larger system evaluated. In industries like medical devices or aviation, that work often has to be written up formally for regulators, because a mistake with a medical device can hurt someone.

Some products need input from both fields at once. A vehicle’s dashboard needs UX research to keep the layout clean enough for a driver to read at a glance. It also needs human factors to ask whether that same driver, glancing over for half a second at highway speed, will read a warning icon correctly at all. A trading platform needs order entry fast enough for people making decisions under time pressure, which is a UX question. Whether the confirmation step is strong enough to catch a mistyped order size before it executes is a human factors question.

The Reels work was mostly about whether a format matched what people expected in a given market. The exoskeleton work was about whether a device held up once someone got tired or thrown off. But there are moments in each where we could have looked both at the UX of the cite and the human factors impact.

More resources on both I recommend checking out

UX research

• Nielsen Norman Group, When to Use Which User-Experience Research Methods

• Erika Hall, Just Enough Research (2024 edition)

• Steve Portigal, Interviewing Users: How to Uncover Compelling Insights (2nd edition, 2023)

• Steve Krug, Don’t Make Me Think, Revisited and Rocket Surgery Made Easy

Human factors

• Human Factors and Ergonomics Society, What Is Human Factors and Ergonomics?

• U.S. Food and Drug Administration, Applying Human Factors and Usability Engineering to Medical Devices

• AHRQ Patient Safety Network, Human Factors Engineering primer

• FAA Human Factors Division, hf.faa.gov

• Don Norman, The Design of Everyday Things

Being the only person responsible for both research and design can feel like you need every tool available but you really don’t. Here is what you need.

You need a setup that can help:

1. Talk to users

2. Turn what you learn into product decisions

3. Design and test solutions

4. Keep the work visible to the rest of the team

Everything else is optional until it solves a problem you genuinely have and even then, there are probably a few ways to bootstrap it without another tool.

Your core stack

1. A way to talk to users

You need regular access to customers and potential customers.

Start with:

• Zoom or Google Meet for calls

• Calendly for scheduling

• A simple recruiting form

• Existing customers, sales leads, support tickets, or community members

If you work at a company with a customer-success or sales team, get to know them early. They are often the fastest route to users.

You do not need a paid research panel right away. Use one later if you cannot reach the people you need through your own network or company.

2. One place for research notes

Pick one place where every interview, usability test, survey, and insight lives.

Notion, Google Docs, Airtable, or a spreadsheet all work.

For each study, keep track of:

• The question you were trying to answer

• Who you spoke to

• The main findings

• Supporting quotes or clips

• What the team decided to do next

This is important whether you are a team of one or a team of 100. You will not remember every interview from six months ago, and you should not have to repeat research because nobody can find the previous work.

A dedicated research tool like Dovetail is useful later with large amounts of reports and data. Start with a system you will actually maintain, easily.

3. Figma

Figma is the main tool for a solo researcher-designer.

Use it for:

• Rough sketches and wireframes

• Product flows

• Final screens

• Clickable prototypes

• Design feedback

• Developer handoff

• Reusable components

Keep your files easy to navigate. You do not need to build a massive design system. Start with the components you use most often: buttons, inputs, cards, navigation, alerts, and modals. Build the design system as the product needs it.

4. A way to test designs before they ship

You can do this with a Figma prototype and a video call.

Give someone a task and watch what happens. For example:

You need to invite a teammate to this project. Show me what you would do.

Ask them to think out loud. Avoid explaining anything or rescuing them too quickly.

Look for:

• Where they hesitate

• What they expect to happen

• What they miss

• Where the language is unclear

• Whether they can complete the task

You can buy tools like Maze, Lyssna, UserTesting, or Optimal Workshop later. For now, five live sessions with the right people will get you far.

5. Analytics

You need enough data to know whether people are using the product successfully.

Pick one: PostHog, Mixpanel, Amplitude, or Google Analytics.

Start with the most important user journey. For example:

Do not try to track every click. Start with the actions that show whether users are getting value.

6. Session recordings

A session-recording tool lets you see where users are struggling without waiting for an interview. Microsoft Clarity is free. Hotjar and FullStory are paid options.

Use recordings to find things such as:

• Forms people abandon

• Buttons people repeatedly click

• Confusing navigation

• Errors you cannot reproduce

• Pages people leave quickly

Then use research to understand the reason behind the behavior.

7. A way to keep the team informed

As a solo researcher and designer, part of the job is making sure your work does not disappear. Use whatever your team already uses: Slack, Notion, Linear, Jira, or email, etc.

After a study, share a short summary:

• What we learned

• Why it matters

• What should change

• What is still unknown

Keep it short. Link the full research for anyone who wants the detail.

The setup I would start with

If I were starting as a solo researcher-designer tomorrow, I would use:

This is enough to do really strong solid work.

What can wait

You probably do not need these on day one:

• A paid participant-recruitment service

• A dedicated research repository

• A survey platform beyond Google Forms

• A formal design-system platform

• An expensive usability-testing platform

• Multiple analytics tools

• A large library of design templates

Add them when you have a specific reason.

For example, get a research repository when your notes are hard to search. Get paid recruiting when you cannot reach the right participants. Get a stronger testing tool when you need to test with many people quickly.

Opening an AI tool before the question is finished changes how long you sit with the discomfort of thinking.

Early on in my use of AI I started to notice something about the way I use AI. I’d have a question and almost immediately open AI.

Sometimes, I had not even finished thinking about the question yet.

• Is there research on this?

• What am I missing?

• Explain this another way.

• What’s the argument against it?

A few minutes later I have an answer, several new questions, and a much clearer sense of whatever I was trying to understand.

I like this about AI. I also wonder what it’s doing to my tolerance for not knowing something.

Because there used to be a gap between having a question and getting an answer. Sometimes that gap was a few minutes. Sometimes it was days. Sometimes you forgot the question entirely.

Now that gap can be seconds. And I’ve started wondering if there is an impact.

We talk about “dopamine hits” from Instagram, sugar, shopping, notifications, video games and basically anything else we enjoy doing too much.

Dopamine is involved in motivation, learning and reward. One particularly interesting part of the research involves reward prediction error, which is essentially the difference between what we expect to happen and what actually happens.

Our brains don’t only care about rewards. They also care about surprises.

• Something was better than expected.

• Something was worse.

• Something happened that we didn’t predict.

That information helps us learn. Once I started thinking about AI this way, I could see why prompting can be so compelling. You know what you asked, but you don’t know exactly what you’re going to get.

A connection. A phrase. A better question.

So you ask another one.

Question. Response. Adjustment. Another question.

The loop can continue almost as quickly as you can think.

We’ve been here before

Video games have feedback loops.

Social media has rewards and gamification.

Plus TV, phones, food, I could go on.

When the internet came around it made information available almost instantly. Google dramatically shortened the distance between wondering about something and finding out.

So saying “AI gives us quick rewards and our brains like quick rewards” doesn’t tell us very much. Of course they do.

AI didn’t invent instant gratification.

AI can meet us in the middle of thinking

TV can stop me from thinking because I’m watching TV instead.

A video game can occupy my attention for three hours.

Social media can make fifteen minutes disappear without me noticing.

AI can do all of those things too, but that’s not what makes it interesting to me.

We already outsource parts of our thinking all the time

There is a concept in psychology called cognitive offloading. It describes the way we use things outside our brains to reduce the cognitive work required to accomplish something.

Writing something down is cognitive offloading.

So is using a calendar.

A calculator.

GPS.

A grocery list.

Google.

This isn’t some modern failure of human intelligence. Humans are extremely good at building tools so we don’t have to keep everything in our heads. I don’t think my life would improve if I memorized every phone number again. The interesting question with AI is how much further cognitive offloading can go.

A calculator can do arithmetic for me.

GPS can tell me where to turn.

Google can help me find information.

AI can take something much less finished and do something with it.

The blank page is a good example

Writing is one of the places I notice this most. You have an idea. You try to write it down. It sounds wrong. You delete it. Try again. Still wrong.

You don’t actually know what you think yet. So you keep going.

Eventually the writing gets clearer because the thinking gets clearer.

That process is inefficient and annoying.

AI can be extremely good at shortening it.

But better writing and better thinking aren’t always the same thing.

That’s the friction I’m becoming more careful about removing.

Video games actually make the comparison more interesting

Good games don’t remove all friction.

They create it.

A game that’s impossibly difficult isn’t much fun. A game that’s so easy you can’t fail isn’t much fun either. The designer is constantly managing the space between what you can do and what you can’t quite do yet.

You try. You fail. You adjust.

Eventually you succeed.

Friction is part of the reward.

AI doesn’t have to remove that process either. That’s something I think gets lost when we talk about AI doing things “for us.”

I can ask AI to write something for me.

Or I can ask it to tell me what’s weak about what I wrote.

I can ask for the solution to a problem.

Or I can tell it not to give me the answer and ask for one hint.

I can ask what I should think about something.

Or I can explain what I think and ask it to find the holes.

Those are completely different relationships with the same tool. One removes friction and the other can create useful friction.

I’m less worried about intelligence than discomfort

There’s already a lot of discussion about whether AI is making us smarter or stupider. I don’t think we can answer that in any meaningful way yet.

I’m more interested in something probably harder to measure. Our tolerance for cognitive discomfort.

Being confused is uncomfortable. Not knowing how to start is uncomfortable. Reading something difficult and realizing you don’t understand it is uncomfortable. Having an idea that won’t quite come together is uncomfortable.

Historically, there wasn’t always an immediate escape from those states.

You had to stay there for a while.

Now you often don’t.

If I don’t understand something, I can ask AI to explain it.

Still don’t understand? Make it simpler. Give me an analogy. Give me an example. Explain it like I’m twelve. Try again.

This is genuinely one of the best things about AI. Being able to ask unlimited follow-up questions without worrying about looking stupid….it is an extraordinary learning tool.

Patience becomes a skill

Most AI advice right now is about becoming better at using AI.

Better prompts. Better workflows. More automation.

All of that makes sense.

But I think knowing when not to ask yet.

Not never.

Just not yet.

Try the problem.

Write the bad first paragraph.

Read the original before the summary.

Come up with three ideas before asking for thirty more.

Decide what you think before asking AI to challenge it.

Then use it.

Not because doing things the hard way is inherently virtuous. Most friction is just friction. I’m perfectly happy to let technology remove it.

I don’t want to print directions from MapQuest again.

But some friction is productive.

A video game without any difficulty isn’t much of a game.

Exercise without resistance doesn’t accomplish much.

And thinking is its own version of resistance.

What I don’t want is to lose the ability to tell the difference between friction that gets in the way of thinking and friction that is the thinking.

The better AI becomes at answering our questions, the less often we will be forced to sit with them. Which means patience becomes something we have to choose.

The same habit shows up on the product side, when a system treats a click as a preference. I wrote up how I tell a lasting signal from a moment that should expire: Show Me You Know Me.

Some research questions cannot be answered in a short study. Especially clinical research. In this article I will discuss my time working with the Shirleu Ryan AbilityLab and the National Spinal Cord Injusry Model System Database.

Spinal cord injury can affect health, work, relationships, independence, community participation, and emotional well-being across a lifetime. The questions that matter soon after an injury may look very different five, ten, or forty years later.

Since 1973, the National Spinal Cord Injury Model Systems Database has collected information about people with traumatic spinal cord injuries. Participants are followed after their initial hospitalization, at one year, five years, and every five years after that.

Shirley Ryan AbilityLab’s Midwest Regional Spinal Cord Injury Care System, which works in partnership with Northwestern University and Northwestern Memorial Hospital, was one of the original five federally designated Spinal Cord Injury Model System centers.

As of June 2022, more than 52,000 people had been registered in the national database, with follow-up data available for more than 30,000. Some participants had been followed for 45 years.

Shirley Ryan AbilityLab describes it as the world’s largest and longest active spinal cord injury research database. Midwest Regional Spinal Cord Injury Care System. I was lucky to work there.

Research questions change as people’s lives change

When the database began, many urgent questions involved survival and medical complications. People with spinal cord injuries faced repeated urinary tract infections, pressure injuries, long hospitalizations, limited access to their communities, and few opportunities to return to work.

Care has changed substantially over the past five decades. People are living longer, rehabilitation stays have become shorter, and researchers are paying more attention to aging, social participation, accessibility, employment, and the environments where people live. The questions in the database have changed along with that understanding.

Researchers did not ask much about community participation fifty years ago. They now know that people who can participate in their communities tend to report a higher quality of life. Geographic information has also been added so researchers can study how neighborhood resources and the built environment relate to health, employment, and well-being. 50 Years of Tracking People with Spinal Cord Injury

Long-term research creates a record of change, but it also has to change itself. A survey designed around the concerns of 1973 would miss many questions researchers and people with spinal cord injuries consider important today.

Quality of life is larger than a clinical outcome

Medical outcomes matter. So do the parts of life that may be harder to see during a clinical visit.

• Can someone participate in their community?

• Do they feel independent?

• How has the injury affected their relationships?

• Can they work, pursue interests, and make decisions about their own life?

• Do they experience stigma, grief, anxiety, resilience, or a sense of purpose?

Those questions are part of the reason the Spinal Cord Injury Quality of Life measurement system, known as SCI-QOL, was developed.

Many existing quality-of-life measures had been created for general medical populations and later applied to spinal cord injury research. Some included irrelevant questions, such as whether someone could run or climb several flights of stairs. Other measures focused on a narrow area of functioning and could not be applied across the range of spinal cord injuries.

The problem was not only sensitivity or statistical performance. A measure can be reliable and still fail to represent the life of the person completing it.

SCI-QOL was developed to measure physical-medical health, physical functioning, emotional health, and social participation in ways relevant to people living with spinal cord injury.

The early qualitative work included focus groups with 65 people with spinal cord injuries and 42 clinicians across four SCI Model System sites. A regional consumer advisory board contributed to the study, and 28 people with spinal cord injuries participated in cognitive interviews during the item-writing process. Developing a Contemporary Patient-Reported Outcomes Measure for Spinal Cord Injury

That work identified three areas of importance:

• Physical and medical health

• Emotional health

• Social participation

Within those areas, participants discussed pain, bowel and bladder management, resilience, grief, self-esteem, independence, relationships, leisure, work, and stigma.

These topics came from people describing what affected their quality of life. Their experiences helped determine what the measurement system needed to include.

Building a measure is only part of the work

A well-designed measure does not automatically become useful in clinical care. Clinicians need to understand the scores. They need to know what different score levels may mean for a patient. The measure has to fit into an existing workflow, and the result needs to support a conversation or decision.

CROR’s project, Clinical Adaptation of the Spinal Cord Injury Quality of Life Psychosocial Measures, focused on that.

The project validated SCI-QOL items and worked to improve how people use the scores to describe different levels of social and emotional experience. The goal was to give healthcare professionals and people with spinal cord injuries a way to assess well-being and understand social and emotional outcomes over time.

A score may show that someone’s reported well-being has changed. A clinically useful measure should also help the patient and clinician understand the change, discuss what may be contributing to it, and decide whether any response is needed. A measure reaches its full value when it can be used in the setting where care happens.

Long-running research depends on relationships

Following people for decades requires a lot more than a database. Contact information changes. People move. Their health changes. Some become tired of participating. Others may not see how another survey will help.

The Midwest Regional Spinal Cord Injury Care System has enrolled nearly 4,000 participants since 1973. Maintaining strong follow-up rates requires continued contact, including a newsletter that shares research updates, information about living with spinal cord injury, and opportunities to participate in new studies. In my time alone, I enrolled over 50 new members and followed up with countless others.

Participants who contribute information over many years are giving researchers access to a part of their lives. Research teams have a responsibility to explain what is being learned and why continued participation matters. A longitudinal study is sustained through trust, relevance, and follow-through.

Long-term data can reveal consequences that appear over time

Some outcomes take years to become visible.For example, people with paraplegia were once commonly encouraged to use manual wheelchairs to maintain strength. As people lived longer, clinicians began seeing degenerative changes in shoulders that had spent decades doing work normally carried by the lower body. That evidence has contributed to new conversations about when power-assist devices may be appropriate.

Long-term data has also shown changes in the age of people sustaining spinal cord injuries. In the 1970s, people were often in their twenties at the time of injury. The average age has since risen to 43, in part because more older adults experience fall-related injuries.

Older adults may enter rehabilitation with other health conditions and need different services and follow-up care.

Neither pattern would be easy to understand from a short study or a single clinical encounter.

What can you learn from this work

Most research programs will not run for fifty years. The principles still can apply to shorter work.

Ask whether the measure reflects the population

A standard measure may be convenient and well validated. It may also contain irrelevant questions or miss experiences that matter to the population being studied.

Before selecting a measure, ask:

• Who was involved in developing it?

• Which population was used to validate it?

• Are the questions relevant across different levels of ability?

• Are important parts of the experience missing?

• Could any items feel confusing, inappropriate, or alienating?

Involve people before the questions are finalized

People with spinal cord injuries contributed to the SCI-QOL.

As I have discussed in my article on conginitive interviews for survey work. Focus groups, advisory boards, and cognitive interviews can reveal whether researchers are asking the right questions and whether participants understand those questions as intended.

Keep the source attached to the score

A number can travel farther than its context.

Document:

• What the measure assesses

• Which population it was designed for

• How it was administered

• What a change in score may indicate

• What the score cannot establish

• Which decisions require additional clinical judgment

Review what the study is missing

The National SCI Model Systems Database has produced a rich database, but it represents fewer than 10% of people with spinal cord injuries in the United States. So, even though it is a large ongoing long-running research, it can still contain gaps.

Researchers should continue asking in any work:

• Who is represented?

• Who is difficult to reach?

• Who stops responding over time?

• Which outcomes receive the most attention?

• Which parts of life remain poorly measured?

• Have social, medical, or technological changes created new questions?

The value of continuing to ask

Fifty years of spinal cord injury research has documented major changes in survival, rehabilitation, aging, community participation, and quality of life. It has also shown a great example of why measurement cannot remain fixed. As people live longer and their circumstances change, research has to follow and adapt to that. It has to account for emotional health, relationships, work, autonomy, access, and the environments in which they are in.

Evaluating a personalized, adaptive, or agentic experience needs a test of the normal path, the mismatch, and the recovery, not only a demo of the happy path.

Individuals and teams are getting better at demonstrating the ideal version of an intelligent experience-really fast.

Here is an example of what I mean. A consumer writes, “Find me a vibey restuarant near central park for with vibey cocktails.” The system understands. The results look relevant. A page highlights the amenities that matter. An itinerary appears. Perhaps the system even prepares the reservation for them.

Traditional usability testing remains essential. Yet an experience that adapts, infers, recommends, plans, or acts introduces a different class of problems and ways we need to test. The interface may be easy to operate while the system’s assumptions are not. The happy path may be smooth while the recovery path is dangerously unclear.

The harder question is: What happens when the system is wrong?

What if it applies a preference from a previous want or need? What if it treats “close to downtown” as more important than vibey? What if a correction appears to work but does not carry into the next screen?

Not every intelligent experience is an agent

The first step in this is to stop grouping every AI-influenced feature into one category.

A personalized product page is not the same as an automated booking assistant. Natural-language search is not automatically agentic. A recommendation can influence a decision without taking an action. These experiences carry different risks and should not receive the same research.

Here is a practical distinction:

• Personalization changes an experience using information the system knows or assumes about a person.

• Natural-language interaction lets someone express a need in ordinary words rather than only through menus or filters.

• Adaptation changes the order, emphasis, wording, or presentation of information.

• Recommendation proposes an option but does not act.

• Planning combines options into a proposed sequence.

• Agentic action changes something outside the conversation or interface, such as making or changing a reservation.

• Proactive action begins without a new request at that moment.

Testing should expand as the system gains influence and authority.

The core protocol: normal, mismatch, recovery

For most adaptive and agentic concepts, the strongest qualitative study contains three conditions. This three-part structure extends familiar usability methods rather than replacing them.

Nielsen Norman Group’s research-method framework emphasizes matching the method to the question and product stage. Stanford’s design methods emphasize learning through realistic prototypes and observed behavior. Carnegie Mellon’s HCI teaching includes think-aloud and usability evaluation. Human-AI guidance from Microsoft, Google, and NIST adds the need to examine expectations, failures, reliance, transparency, and recovery.

This new article is combining those ideas into a repeatable protocol for adaptive experiences.

1. Normal

The experience behaves as intended and its assumptions fit the current needs. This reveals whether the feature creates value. Does it reduce effort? Does it help someone consider better options? Does it make important information easier to find?

2. Mismatch

The experience uses one plausible but incorrect, stale, incomplete, or conflicting assumption.

This should not be a cartoonishly bad failure. It should resemble something the live system could reasonably do.

The mismatch reveals whether people notice the problem, understand its source, and know how to respond.

3. Recovery

The participant attempts to correct, override, stop, undo, or escalate the problem.This reveals whether control is real. A correction control is not successful merely because someone can click it. The person must understand what changed, where the change applies, and whether it will persist.

Test behavior before asking about trust

Researchers often ask participants whether they trust an AI feature. The answer is easy to collect but also can be difficult to interpret.

A person may say they trust a recommendation while checking every detail. Another may express doubt while relying on it completely. A rating cannot tell us whether reliance was appropriate. Treat trust as observable behavior instead.

Watch when people:

• verify a result;

• accept it without checking;

• reject or edit a recommendation;

• seek an alternative;

• hand control to the system;

• take control back;

• etc.

Then compare that behavior with the quality and consequence of the system’s output. The goal is not maximum trust. It is calibrated reliance: using the system when it is helpful and checking or overriding it when the situation requires care.

Current intent should beat old history

A useful research scenario separates:

• Hard constraints: conditions that make an option unacceptable, unsafe, or unusable, such as dates, maximum total cost, ethics, accessibility needs, or required cancellation terms, return window,etc.

• Soft preferences: conditions that improve fit but can be traded against other benefits, such as neighborhood, view, brand, breakfast, or departure time.

During the session, record which constraints the system retains, loses, changes, or asks about. Observe whether a hard constraint remains visible when the system is uncertain. Do not assume that a relevant-looking result is a valid result.

This is also where adaptive product pages deserve careful study. Personalization may reorder or emphasize content, but it should not make price, policy, accessibility, safety, sponsorship, or important limitations harder to find.

Eight questions every study should consider

Not every feature needs every test. But a research plan should consciously include or exclude these eight areas.

Intent

Did the system understand the current needs, wants, constraints, and ambiguity?

Personalization fit

Did prior information help without overriding what matters now?

Choice integrity

Could the user still find viable alternatives and material facts, or did adaptation create tunnel vision?

Understanding

Could the user form an accurate-enough explanation of why the experience changed?

Correction

Could the user fix a wrong assumption and predict what the correction would affect?

Data expectations

Did the information used by the system match what the user reasonably expected in that context?

Authority

If the system could act, did the user understand what it could do, when it would do it, and what required approval?

Recovery

Could the user detect, contain, reverse, reconstruct, and escalate a problem?

These questions work for systems powered by rules, ranking models, machine learning, generative AI, or a combination. From the users perspective, the critical issue is not the underlying model. It is how the experience changes their decisions and what happens when the system is wrong.

A practical 60-minute session

A moderated session can follow a simple structure:

1. Recent behavior interview — 5 minutes. Ground the conversation in a real planning or booking experience.

2. Expectation check — 5 minutes. Ask what the participant thinks the feature can know and do.

3. Normal task — 15 minutes. Observe value, intent handling, and natural verification.

4. Comparison — 10 minutes. Compare the standard experience with the proposed adaptation.

5. Mismatch and correction — 10 minutes. Introduce a plausible wrong assumption and observe detection and repair.

6. Action or recovery task — 10 minutes. When applicable, test permission, action status, interruption, reversal, or support.

7. Replay and debrief — 5 minutes. Revisit critical moments, reveal intentionally altered conditions, and discuss data use.

Moderators should give goals and context rather than a list of clicks. Ask for thoughts at decision points instead of demanding constant narration. Before explaining what the system did, ask what the participant believes happened.

Useful prompts include:

• “What do you expect this experience to know about you?”

• “What do you think caused these results to appear in this order?”

• “What are you checking before you continue?”

• “What does the system seem to believe about you?”

• “What do you believe will happen if you continue?”

• “What happened, what is the consequence, and what would you do next?”

What observers should capture

Notes should preserve the chain between system behavior and human consequence.

For each important incident, record:

1. the goal and constraints;

2. the information or assumption used by the system;

3. what the system displayed or did;

4. what the user believed;

5. what the user did;

6. whether they noticed the issue without prompting;

7. the consequence;

8. the recovery effort;

9. the required change or unanswered question.

This structure prevents a common failure in reporting: reducing a system problem to a vague statement such as “participants wanted more transparency.” Teams need to know which assumption was invisible, what belief it created, how that belief changed behavior, and what design or system requirement follows.

Start with a pilot

Whenever adapting a new evaluation framework it should itself be evaluated.

For an initial pilot, run two waves of six participants. Use a prototype that combines natural-language search, personalized ranking or content, and—if relevant—an action-preparation step. Include a standard condition, an intended personalized condition, and a mismatch condition.

After the first six sessions, refine the moderator guide, scenario, and observation definitions. Double-code several sessions so researchers can compare how they classify incidents and severity.

The pilot succeeds if:

• sessions can be completed in the planned time;

• every selected evaluation area produces observable evidence;

• participants understand the scenarios and consider them realistic;

• researchers can connect incidents to system inputs or design behavior;

• mismatch conditions reveal issues missed by the normal task;

• the team can translate findings into specific product requirements.

Avoid collapsing the results into one “agent experience score.” Averages conceal exactly the failures the method is designed to expose. Report patterns, conditions, consequences, and blockers instead.

The standard we should aim for

The goal is not to make intelligent experiences appear infallible. They will not be. The goal is to create experiences in which useful adaptation is visible enough to understand, wrong assumptions are easy to detect, corrections have predictable effects, authority has clear boundaries, and recovery remains possible.

That standard applies whether the feature uses a language model, a recommendation system, a set of business rules, or all three. Users experience the outcome, not the architecture diagram.

So yes, test whether the feature works. Then test the moment when it does not.

Additional sources

• Nielsen Norman Group: When to Use Which User-Experience Research Methods

• Nielsen Norman Group: Individualized Recommendations—Users’ Expectations and Assumptions

• Nielsen Norman Group: Can Users Understand Recommendations and Personalization Driven by Machine Learning?

• Stanford d.school: Bootcamp Bootleg

• Microsoft Research: Human-AI Experience Toolkit

• Microsoft Research: Planning for Natural Language Failures with the AI Playbook

• Google Research: Take It, Leave It, or Fix It

• NIST: AI Risk Management Framework—Generative AI Profile

A startup can stay close to users without a research team if one person owns a regular cadence of contact and the record stays light.

Startups often describe themselves as user-centered. The mission includes helping a particular group, removing a frustration, or making an experience easier.

Easier said than done. The team gets busy. A large customer has a request. An investor asks about growth. Support tickets pile up. Product decisions move faster than the company can investigate them. User research begins to feel like something to add later, once there is more time, funding, or headcount.

Most early-stage companies cannot afford a full research team. That is okay. They can still create a reliable way to learn from users.

The goal of this article is not to imitate the research operation of a large organization or replace the need for research. It is an ackwnowledgment that a startup needs a lighter system that keeps evidence connected to product decisions when they cannot have research. So what can you do to keep the user center in your business?

Start with the decision

Research can quickly become too large when a team begins with too broad of question and it can feel overwhelming. For example:

What do our users want?

The question covers too much. Scoping that is nearly impossible. It can lead to unfocused interviews, large surveys, and a collection of data that does not help the team decide what to do.

Begin with the decision instead:

• Should we build the feature?

• Which part of onboarding needs attention first?

• Why are people leaving after the first week?

• Do customers understand the pricing model?

• Which problem is urgent enough for someone to pay to solve?

• What would prevent users from adopting the proposed change?

Before speaking with users, write down:

1. The decision the team needs to make

2. What the team believes so far

3. What evidence supports that belief

4. What remains uncertain

5. What the team will do differently based on the answer

If the final point is unclear, the research question may not be ready, keep going.

Create a regular cadence of contact

Teams often talk to users during a launch, a crisis, or a major redesign. Long periods can pass without direct contact in between.

A small, predictable cadence is more useful and then easier to come back to those users when you need research.

For many startups, two or three user conversations each month can create a steady flow of information without stopping product development.

The conversations do not need to be long. A focused 30-minute session can explore:

• How someone handles the problem today

• What prompted them to try the product

• Where they became confused

• What almost prevented them from continuing

• Which alternatives they considered

• What happened the last time they encountered the problem

Block a recurring time for the sessions. Invite a rotating product, design, engineering, content, or support partner to observe if you are a larger start-up and have the headcount.

The main things is: regular contact makes it harder for a single memorable conversation to define the team’s view of all its users and their needs.

Speak with more than your biggest fans

Early-stage companies often have easy access to a small group of enthusiastic customers. Those relationships are valuable, but loyal users cannot answer every question.

A person who speaks with the founder often, may understand the product better than a new user. A large customer may have needs that do not represent smaller accounts. Someone who agreed to join a community group may be more engaged than the average customer. Connect with more than your power users. Connect with folks you would want to use it. Figure out their painpoints and things that would solve something for them too.

When needing research. Who you speak to is dependent on the question, the team may need to speak with:

• New users

• Long-term users

• People who stopped using the product

• People who signed up but never became active

• Customers who contacted support

• People using a competitor

• People who experience the problem but have not looked for a product

• Users with accessibility needs

• Customers from different industries, roles, or account sizes

Watch what people do

Interviews are useful, but people cannot always describe their behavior accurately. Someone may say onboarding was easy while struggling to complete it. A customer may ask for a feature when the underlying problem could be solved another way. A user may forget the workarounds they rely on because those steps have become routine.

Whenever possible, ask people to show you.

Try prompts such as:

• “Can you walk me through the last time you did that?”

• “Show me how you handle it today.”

• “What would you click next?”

• “What information would you expect to see here?”

• “Can you explain what you think will happen after you select that option?”

Observing a real task often reveals gaps that direct questions can miss.

Avoid leading with “Would you use this?” or “Do you like this idea?” People are generous with hypothetical approval. Past behavior and real tradeoffs provide stronger evidence.

Use the evidence you already have

User research does not begin and end with interviews. Startups often have valuable evidence spread across:

• Support tickets

• Sales calls

• Cancellation reasons

• Product analytics

• Search queries

• Onboarding drop-off

• App reviews

• Community discussions

• Customer success notes

• Demo questions

• Feature requests

Each source has limitations but are still very valuable. Sales conversations reflect prospects in a buying context. Support tickets overrepresent people who encountered a problem and chose to report it. Analytics show what happened but may not explain why. Looking across several sources can reveal useful patterns.

For example, a team might notice:

• Users repeatedly ask Support how a feature works.

• Product data shows low adoption.

• Interviews reveal that people misinterpret the label.

• Sales calls show that prospects expect the feature to work differently.

Together, those give the team a problem to investigate.

Give the process an owner

User learning can be a shared responsibility. It still needs an owner or an operating model. Without one, interviews happen inconsistently, notes disappear, and insights remain with the person who heard them. The owner does not need to be a trained researcher.

The tasks might include:

• Keeping a list of current questions

• Scheduling conversations

• Making sure participants understand how their information will be used

• Storing notes

• Connecting findings to decisions

• Tracking which users the team has heard from

• Sharing patterns with the broader company

Other team members should participate. Collectively the documentation and teams keep the insights from disappearing when priorities shift.

Keep documentation light

A startup does not need a complex research repository but it does need enough documentation.

For each project, capture:

• The decision being supported

• The question explored

• Who participated

• What method was used

• The strongest patterns

• Important differences or exceptions

• Links to notes or recordings, where appropriate

• The decision made afterward

• Remaining uncertainty

A short summary is enough.

Separate observations from interpretations:

• Observation: Four of six new users looked under Settings for billing information.

• Interpretation: The current billing location may not match users’ expectations.

• Decision: Test a billing link in the account navigation.

The distinction matters. Interpretations can change as the team gathers more evidence.

Protect participant trust

Lightweight research still absolutely carries responsibilities.

Tell participants:

• Why you are speaking with them

• Whether the session will be recorded

• Who may see the recording or notes

• How their information will be used

• Whether participation affects their account or service

• How they can decline a question or stop the session

Avoid collecting sensitive information unless the project requires it. Limit access to recordings and raw notes. Remove identifying details from summaries when names are not needed.

Compensate participants when possible, especially when asking for substantial time or recruiting people who do not already have a business relationship with the company. Limited resources may require a simpler process. They do not remove the need to treat people respectfully.

Build research into existing work

Research is easier to sustain when it becomes part of product development rather than a separate event.

Add a few prompts to existing routines.

During planning

• What do we know about the problem?

• Where did that information come from?

• What are we assuming?

• Whose experience is missing?

• What do we need to learn before committing resources?

During design and development

• Have we watched someone try the flow?

• Which part carries the greatest risk?

• Are we solving the problem users described?

• Has new evidence changed the direction?

After launch

• Did people use the feature as expected?

• Where did they stop?

• What questions reached Support?

• Which users benefited?

• Who struggled?

• What should we investigate next?

Adding these questions to existing planning and review meetings can keep users present without creating another large process.

Know when scrappy research is not enough

Lightweight research works well for many early product decisions. Some questions need more care. Consider bringing in an experienced researcher or specialist when:

• The decision could cause meaningful harm

• The product involves health, finance, safety, or sensitive personal information

• The team needs to understand vulnerable or difficult-to-reach populations

• A major investment depends on the result

• The question requires a large survey or statistical analysis

• The company needs accessibility research

• Teams disagree about what the evidence means

• Informal interviews keep producing conflicting answers

• The product has grown beyond the people the founding team knows personally

Support does not always require a full-time hire. A consultant, fractional researcher, or short-term research partner may be enough for a defined project.The important part is recognizing when the decision exceeds the team’s current skills.

A minimum viable research rhythm

A small company can begin with a simple cadence.

Every quarter

• Speak with two or three users.

• Invite one teammate to observe.

• Add observations to a shared document.

• Review support and product signals related to the current question.

• Review repeated patterns across conversations and data.

• Check which types of users have been included or missed.

• Connect findings to upcoming decisions.

• Share a short summary with the team.

Every year (or less if you can)

• Revisit the company’s main assumptions about its users.

• Identify what has changed.

• Review decisions influenced by user evidence.

• Choose the next important questions.

• Decide whether any work requires outside research support.

The cadence can grow with the company. Consistency matters more than volume.

Keep the mission tied to decisions

Putting users at the center of a mission does not require a research department. It does require checking assumptions before they become product decisions.

Talk with users regularly. Observe their behavior. Look beyond the loudest customers. Preserve the source and limits of the evidence. Give someone responsibility for keeping the practice alive.

Most of all, show where user evidence changed the work.

A mission remains meaningful when we keep the user center.