Njenga Kariuki

I look for a durable customer problem, one people had before the technology and will still have after it. At Amazon’s Health Computer Vision and AI organization, every new capability had to answer such a problem. We ran customer research and capability experiments side by side, and paused further investment when either lacked proof.

• Amazon Health AI The products that shipped moved clinic measures onto the phone: body fat, waist-to-hip ratio, and mobility, stability and posture.
• Alexa Map View Phone room scanning already existed; I restated the problem as whether a customer could easily map a home.

A model’s accuracy on benchmarks and the accuracy a customer perceives are separate measures. So I ask customers directly whether a result feels accurate, means something to them and leads them to act. Then I change the model and the interface together, and measure again.

• Halo Tone Raw chart positions, first shown with two presidents’ voices, were replaced by daily bars and words such as “Hopeful.”
• Halo Movement The top of the 0 to 100 scale became a product decision: very athletic people rarely scored above 90.
• Amazon Bio Discovery Checkpoints measured the interface and how useful scientists found it, beyond the models’ scores.

Labeling shows me how a model improved, which improvement it was and how it ties to what customers notice. The scoring guide that others will label by comes from real interactions I have judged myself.

• Alexa Shopping I listened to 1,000 data samples a week and judged each request and reply as success or failure.
• Halo Tone I defined what listener panels rated, from happy to sad and low to high energy; ratings became training labels.
• Prime Now I sat in the stores, asked for whatever data they had, and designed the catalog matching around it.

Each dimension then enters the evaluation with its own measure, so the team can track it from one model version to the next. Some dimensions sit outside the model: in products with complex input steps, the input step is part of the quality customers perceive.

• Halo Tone Customers wanted to see how their voice varied, so that variation, dynamism, became a measured dimension.
• Halo Movement Customers wanted variety in live workout feedback, so dynamism became a measured dimension of the workouts.
• Halo Body The scan itself became a measured part of quality, and from August 2021 it worked in smaller rooms.

The core team first agrees on what counts as an error, and writes an example set for each one. Any ambiguity in the core team grows once a labeling team applies the rules. Written rules let labelers agree with each other, and let other teams run the same measures.

• Alexa Shopping My error definitions, each with example cases, became the procedures that a team of 10 data analysts applied.
• Amazon Health AI Body fat, mobility and waist-to-hip ratio were each tested against a named clinical reference in a published study.
• M-PESA Africa A written standard for AI in work that moves money: every check still runs.

A working prototype on a team’s own work shows what the technology can do. So I build the first version with the people who will run it. The executive work decides which chains of work change; the hands-on work shows how they can.

• M-PESA Africa I set the direction reporting to the CEO, and build the first versions with the teams that redesign each chain of work.
• Prime Now I learned to code to automate the stores’ uploads myself, before the system ran for every store.
• Alexa Shopping I listened to 1,000 shopping requests a week while leading the evaluation team.

In Amazon’s health AI teams, a product manager, a designer and a group of engineers spent each Friday on one charter: “build something that won’t work.” At AWS, I made the product organization itself AI-native, from how we defined products to how we built them.

• Amazon Health AI Ideas from the Friday builds still had to pass two proofs before more money.
• Amazon Bio Discovery Alongside the product, the organization moved to AI-native ways of defining and building products.

At Prime Now, each new country was measured on revenue, a goal that did not reward growing two-hour delivery. I proposed measuring each country on its percentage growth instead, and leadership adopted it.

• Prime Now Each country’s goal moved from revenue to percentage growth.
• M-PESA Africa Each changed workflow is measured end to end, against a comparison group working the old way.

If AI halves a two-day step in a 10-day process, the customer waits nine days instead of 10. The other eight days are handoffs, approvals and waiting. So I measure from the first request to the result, and change the steps around the AI along with the step it speeds up.

• M-PESA Africa Where the time goes in one workflow: halving the waiting saves more than halving the work.
• Amazon Bio Discovery Bench scientists had waited for computational biologists to run models; Bio Discovery put both in one application.