Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by:

1. Maven - Get discounts off Mahesh's courses with my links

2. Amplitude - The market leader in product analytics

3. Jira Product Discovery - Prioritize what matters with confidence

4. NayaOne - Airgapped cloud-agnostic sandbox to validate AI tools faster

5. Product Faculty - Get $550 off their #1 AI PM Certification with my link

Today’s episode

LinkedIn just scrapped its APM program and replaced it with an Associate Product Builder track, and introduced a Full Stack Builder career ladder alongside it.

What does it even mean to be a “builder PM”?

Well, tools only get you so far. Learning Claude Code is helpful, but means nothing if you don’t have an understanding of the underlying first principles.

That’s today’s episode.

Mahesh Yadav created one of our most popular episodes, with over 35K views on YouTube, and now he’s back. Earlier, he taught you AI agents.

Today, he’s teaching you how to become a builder PM:

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

I’m giving a free talk on how to get interviews at the top AI PM companies on Thursday (April 23rd) @ 9:00AM PDT. Grab your seat.

Newsletter deep dive

Thank you for having me in your inbox. Here’s the complete guide to “becoming a builder PM.”

1. What is a “Builder PM”

2. The Builder PM Tool Stack

   • n8n

   • Claude Code

   • OpenClaw

3. Mastering AI Agents

   • The first principles of agents

   • How to build self-improving agents

4. The 10-week roadmap to Builder PM

Subscribe now

1. What is a Builder PM

Every PM has already used Claude Code or ChatGPT to get something done. But if that’s where it stops, you are not yet a builder PM.

There are two kinds of builder PM, and they’re different jobs.

Type one ships customer-facing product without engineering handoffs. Customer to 10 paying users, solo. This is what the LinkedIn APB program is training for and what lands you the comp trajectory Mahesh describes.

Type two builds internal agents that automate their own PM work. PRD reviewers, competitive intel, data dashboards. Same skills, different trajectory: stay in your current org, ship 3x more, get promoted faster.

Most of this guide teaches type two. Type one is the aspirational endpoint.

Why this title change happened now

The title change is a response to something that actually shifted in the last six months.

Six months ago, most autonomous agents broke within minutes. METR’s latest benchmarks show frontier models now sustaining multi-hour jobs, with Opus 4.7 running coding tasks for 3 to 6 hours in practice.

When agents can only run for 3 minutes, the PM’s job is to prompt them. When agents can run for 6 hours, the PM’s job is to design the system those agents run inside.

Designing that system is what builder PM actually means.

The compensation trajectory, in case you need the motivation

Mahesh’s comp trajectory shows you the top end of what’s possible. He went from $120K at Microsoft to $1.3-1.4M in his last Google role (Senior PM, AI), with a rough doubling every two years. A few of his AI PM friends at Nvidia are at $2-2.5M.

Levels.fyi still pegs median Senior PM comp at top tech around $350-450K. What Mahesh is describing is the premium the US tech market pays specifically for builder-skill AI PMs, where companies approach you and offer 30-40% over your current number. The $2M+ tier is a small group.

International markets pay differently - builder skills still move comp in EU, India, and APAC, just with different absolute numbers and different negotiation dynamics. And if you’re in mission-driven PM work where comp isn’t the yardstick, the same builder skills still let you ship 3x more impact with the same team - which is the whole point.

You probably won’t hit $1.4M next year. The premium exists, though, it’s real, and the gate is builder skills.

One honest caveat for managers reading this. If your PMs get 10x leveraged, the obvious question is whether you need fewer of them. The early signal from LinkedIn’s restructure is that the answer is smaller pods with broader scope, not layoffs - they’re keeping headcount and pushing each builder to own more surface area. That may or may not hold elsewhere. Worth naming, because every VP Product on this list is already thinking about it.

The failure mode

Here’s the trap: PMs read about Claude Code on Twitter, try it for a weekend, get a mediocre output, and conclude “this is overhyped.”

They skipped the first principles.

That’s the answer to Gothelf. He’s right that execution capacity was never the bottleneck. Judgment was. Judgment sharpens when you ship 12 prototypes a year instead of 1. The builder PM still talks to customers. They just ship without waiting.

That’s what the rest of this guide is going to give you.

When building is the wrong bet

Not every PM job benefits from this shift. A few honest cases where the 10-week investment doesn’t pay off:

• If you work in trust & safety, healthcare decisions, financial underwriting, or legal compliance, your job is to prevent errors, not to generate throughput. An agent that ships 12 prototypes a year doesn’t help you. An agent that proposes a new UI flow in a payments product can actively hurt you. Build evals skills instead.

• If your product lives inside a regulated perimeter and your company has a year-long procurement cycle for new tools, you’ll spend your 10 weeks waiting for IT approval. Spend the time on AI literacy - reading papers, running evals on existing outputs - rather than on personal tool setup.

• If you’re a founder with zero PMs under you, skip the “automate your PRD reviews” framing entirely. Your version is shipping real product to real customers using these same tools. The skills transfer. The PM-productivity examples don’t.

Builder PM is the right bet for most mid-career PMs at product-led companies with data portability. That’s a large slice of the market, not all of it.

2. The Builder PM Tool Stack

Three tools. Three different jobs. Using the wrong one for the wrong stage of your journey will waste weeks.

Pick based on where you are in the journey, ignoring whichever tool has the most Twitter hype this week.

n8n

n8n is a visual workflow builder. You drag components onto a canvas and connect them. That’s it.

Everyone who is deep in AI dismisses n8n as “too basic.” They are wrong. n8n is irreplaceable as a learning tool because you physically see every piece of the agent architecture as separate nodes.

In the episode, Mahesh demoed building a contract analyzer in n8n:

1. Email trigger hits when someone sends a contract

2. 2. Gmail node pulls the MSA (master service agreement)

3. Data loader converts the file to text

4. Text splitter chunks it into 1,000-character pieces with 200-character overlap

5. Embedding model converts chunks to vectors

6. Vector database stores them

7. AI agent reads the playbook and flags risks

8. Gmail node emails back the analysis

9. Eval workflow runs against ground truth

Every one of those is a visible node on a canvas. When something breaks, you see exactly where.

The limitation is real. n8n has no code mode, no version control, no test suites, and no path to production beyond a simple webhook. It stops you around 10 customers.

For most builder PMs, use n8n for 2-3 weeks, then move on. The exception is simple internal workflows with clear webhooks and no collaboration needs - those can live in n8n forever and that’s fine. The “graduate from n8n” rule is about your learning trajectory, not about n8n being bad at its job.

Claude Code

Claude Code is where you build the real thing.

The same tool works for a PM with zero coding experience building skills in English, and for a senior engineer building production services on top of that PM’s work.

Mahesh showed his PRD review setup in the episode. Here is what it does:

1. He drops a PRD into a folder

2. Claude Code reads his checklist (ruthlessly specific, encodes Amazon PR/FAQ format, includes AI-specific criteria like “is this differentiated from ChatGPT or is it a commodity AI wrapper”)

3. The agent reads the document and writes inline comments back into the .docx file, using a Python library to handle the XML under the hood

4. Comments are strategic, not surface level. Things like “What prevents Datadog or the Big Four from building this?” and “How do you handle misclassification?”

That is already useful. But here is where it gets serious.

Every 30 minutes, a second agent runs. It opens the folder of recent reviews, compares the AI’s output against the version Mahesh actually shipped, and logs the deltas to a learner.md file.

When the same correction appears 5 times across 5 days, the learner emails Mahesh: “I want to update your checklist. Here is the proposed update.”

He reviews, approves, and the checklist evolves.

Every day, the reviewer is a little better than the day before. That is the real moat. Not the initial skill. The learning loop that compounds from your judgment. We’ll go deeper on this in Section 3.

The other things to build in Claude Code:

• Competitive intelligence with subagents (3 subagents researching 3 competitors in parallel, rolling up into one report)

• Prototypes from PRDs (clone competitor screens, modify them into working products, ship to customers for feedback in days instead of months)

• Data dashboards generated from your actual production data

Use Claude Code as your daily driver for weeks 4-6 of the roadmap. By the end, you should have at least 2-3 skills running on your real work, with learner loops on top of each.

OpenClaw

OpenClaw is a pattern. The pattern matters because enterprise agents will probably work this way at scale. GCP and AWS are already shipping sandboxed-agent primitives, and the architecture maps directly.

Here is what OpenClaw adds that Claude Code doesn’t have:

Delegation through existing channels. You send the agent a WhatsApp message. It goes and does the work on a separate machine. When it is done, it sends the result back through WhatsApp. You are not sitting in a terminal watching it think.

Full machine sandboxing. Instead of granting file-by-file permissions, you install the agent on a dedicated Mac mini or a VM. It controls the entire compute environment. (Side note: Mac minis with 32GB+ are 10-18 weeks out right now, and several high-memory configs are unavailable entirely. CNN reported Apple Store employees calling them “OpenClaw machines.”)

Model agnosticism. Connect any model, including open source. You are not locked into Anthropic’s rate limits, which every heavy Claude Code user complains about daily.

For PMs at big companies asking the obvious question: Google is not going to let you install OpenClaw on your work laptop. That is true. But Google is going to offer the same pattern inside GCP. AWS is going to offer it inside their infrastructure. The architecture of sending a message to a sandboxed agent that reproduces your problem, tries a solution, and returns results - that is how enterprise agents are going to work at scale.

If you’re at a regulated company (finance, health, government, legal) or your data can’t leave the corporate perimeter, the pattern still applies but the venue changes. Your version is Claude Code or an internal agent platform running inside your SSO and VPN. NayaOne-style airgapped sandboxes exist for exactly this reason. The architecture is the same. The compliance envelope is different. Build what you can inside your perimeter and push your infra team for the pattern on the outside.

Learn the OpenClaw pattern now and you’ll recognize the shape of enterprise agent platforms when they arrive.

3. Mastering AI Agents

Tools without first principles is how PMs waste 3 weeks. Here are the first principles.

The first principles of agents

Every working agent has 4 components. Every disappointing agent is missing at least one. Before you debug a prompt, debug the architecture.

Component 1 - Intelligence (The brain)

The model is the intelligence layer. My current default: Opus 4.7 for agent orchestration and skill authoring, GPT-5.4 for fast single-turn tasks where latency matters. You can run either. The rest of the architecture matters more than this choice.

On its own, it is a genius with amnesia. Ask it about neural networks, it answers perfectly. Ask it what Trump said about Iran this week, and if you picked a cheap model, it tells you its knowledge cutoff is June 2024.

The intelligence layer alone gives you knowledge cutoffs, generic answers, and zero session memory. A model without the other three components is just a more expensive way to use Google.

Component 2 - Tools (The hands)

Tools let the agent take actions. Without them, it can think but it cannot do anything.

In the demo, Mahesh added Tavily (a search tool) to the agent. Same question about Iran. This time, the agent searched, found the answer, and responded with current information. One tool turned it from useless to useful.

The essential tools for PM agents are search APIs (Tavily, Perplexity) for current information, file processors to read contracts and PRDs, MCP servers to connect to Gmail, Slack, GitHub, and your CRM, and bash execution to run scripts. PMs who skip this layer wonder why their agent cannot do anything beyond trivia.

Component 3 - Memory (The recall)

Without memory, the agent forgets everything between messages.

The demo showed this failure instantly. The agent searched and answered the Iran question. Two messages later, Mahesh asked “what conflict am I talking about?” and the agent replied that it saw no previous mention of a conflict. Adding session memory fixed it in one click.

Memory has three layers: session memory for the current conversation, long-term memory for patterns across days and weeks, and structured memory for specific facts the agent needs to retrieve reliably. Every PM agent that feels “dumb” is almost certainly missing one of these.

Component 4 - Knowledge (Your data)

This is where your agent stops being a generic chatbot and starts being valuable.

Knowledge = your company-specific context. Contracts, playbooks, competitive intel, product specs, customer research. Without it, the agent answers from the internet. With it, the agent answers from your actual data.

The RAG pipeline is straightforward: upload your documents, chunk them into 1,000-character pieces with 200-character overlap, convert to vectors, store in a database, and query when asked a question.

In the demo, Mahesh asked about payment terms and tariff impacts on company contracts. Without knowledge, the agent gave generic legal advice from the internet. With the MSA uploaded, the agent answered from the actual contract clauses.

That difference is the entire value proposition of building your own agents vs. using a generic chatbot.

How to build self-improving agents

This is the section almost nobody is doing yet. Which is exactly why it matters.

You automated one task. Your PRD review agent works. Your competitive analysis runs every morning. Then you notice the agent makes the same mistake three days in a row. It misses the same section. It applies the wrong standard.

Static skills produce static output forever. Learning systems compound daily.

Here is the learner loop architecture Mahesh runs on his PRD reviewer:

Step 1 - Every review creates a job folder

Timestamped. Contains:

• Who ran the review

• The input document

• The AI’s output

• The version you actually shipped after your corrections

Step 2 - A scheduled subagent runs every 30 minutes

It opens recent job folders. Compares the AI’s output against your modified version.

Step 3 - Differences get logged to learner.md

What the AI got right. What it missed. What you added. What you deleted.

Step 4 - A threshold triggers a proposal

When the same correction appears 5 times over 5 days, the learner proposes a specific checklist update: an actual edited version of the checklist line.

Step 5 - You review and approve

The new checklist becomes the input for the next review cycle.

The critical design decision is that the learner never updates the checklist automatically. It accumulates evidence, it proposes, you approve. You stay in the loop. The system still compounds.

After a month, your review agent catches things it missed on day one.

After three months, your checklist has evolved from your original draft into something shaped by hundreds of real reviews and your actual judgment calls.

Inside a team, you’ll want one canonical version of each skill in a shared repo, with a light review process when someone proposes a merge. Otherwise you get skill drift, five competing PRD reviewers, and nobody knows which one is current.

Static skills are where most PMs stop. The learning systems that compound from your judgment are what separate the builder PM who gets promoted from the builder PM who watched a YouTube video.

4. The 10-week roadmap to Builder PM

You have a full-time job and maybe 5-8 hours a week to learn this. Here is the sequencing, tested with real PMs in real cohorts:

• Weeks 1-3, Foundations. n8n only. Build one agent with all 4 components, run one real evaluation, build one multi-agent system. By week 3 you can explain the 4 components to a junior PM.

• Weeks 4-6, Build. Move to Claude Code. Pick one real weekly task and automate it with a skill. Add a learner loop on top. Spin up your first subagent system (3 competitors in parallel works well).

• Weeks 7-9, Delegate. OpenClaw time. Pick one complete job (not a subtask) and delegate it via WhatsApp to an agent on a VM. Iterate on the skill until the output is good enough to forward to your team unchanged.

• Weeks 9-10, See the pattern. Read the AI tool market through this lens. Every new tool: is it solving for context, actions, or evals? Is it a variant of the agentic loop? Could you build it yourself?

The biggest failure mode is skipping Phase 1. PMs who jump straight to Claude Code hit a wall around week 3, have no mental model for debugging, and conclude “Claude Code is overhyped.” Do not be that PM.

That decomposition skill from Phase 4 is also what’s showing up in L5 and L6 AI PM interviews. Based on reports from candidates I’ve coached through Land a PM Job and notes from Blind, some loops now include live building exercises where you’re expected to open Claude Code and produce a working prototype.

Candidates who default to Figma mocks when the interviewer is looking for a working prototype have lost offers. Figma is still the right tool for design work. The signal the interviewer is looking for is whether you can ship.

For hiring managers

In 6 months every PM resume will claim builder skills. Two questions cut through it:

1. “Walk me through a skill you’ve built and how the checklist evolved.” Real builders have version history. Fakes have a one-liner.

2. “Show me the learner.md.” Real builders have a file of failures. Fakes don’t.

If a candidate says they’ve used Claude Code but can’t name the four components of an agent or explain when they’d choose n8n over Claude Code, they watched a YouTube video. That’s different from building.

[Bonus] Episode Summary

Ten weeks. Three phases of building. One phase of seeing.

At the end, you are a PM who builds AI systems that get better every day.

In 2026, that is the difference between being automated and being the one doing the automating.

Get Transcript

Where to find Mahesh Yadav

• LinkedIn

• Maven Course

Related content

Podcasts:

1. Claude Code Team OS with Carl Vellotti

2. OpenClaw + Claude Code with Naman Pandey

3. Claude Code OS with Dave Killeen

Newsletters:

1. The complete context engineering guide

2. How to use Claude Code like a pro

3. Practical AI agents for PMs

PS1. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

PS2. The third cohort of my LandPMJob Program is filling up. Apply now!

Share

Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by:

1. Bolt: Ship AI-powered products 10x faster

2. Amplitude: The market-leader in product analytics

3. Pendo: The #1 software experience management platform

4. NayaOne: Airgapped cloud-agnostic sandbox

5. Product Faculty: Get $550 off their #1 AI PM Certification with my link

Today’s episode

Sometimes, I get access to the wildest guests on this podcast. Today, we get the awesome opportunity to look inside the design processes at OpenAI and Figma:

• Ed Bayes leads design on Codex at OpenAI

• And Gui Seiz leads all AI design at Figma

And they worked with me to put together a masterclass on how to design in the AI era.

If you want to design like the leading AI companies, this episode is for you: complete with screen shares and everything else you need to adopt the new AI design workflow.

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

I’m accepting applications for my third LandPMJob cohort. Join Me.

Newsletter deep dive

As a thank you for having me in your inbox, here is the complete guide to the new code-plus-canvas design workflow:

1. Why the linear design pipeline is dead

2. The code-canvas loop

   • Codex to Figma

   • Figma to Codex

3. When to use which tool

4. The 5-step adoption roadmap

5. Total football for product teams

Subscribe now

1. Why the linear design pipeline is dead

You know the process. Everyone does. It was unchanged for 15 years:

Paper sketches. Wireframes. High-fidelity mockups. Developer handoff. Engineering builds it. Design files bug tickets because the spacing is off by 4 pixels.

Every single stage in that pipeline existed for one reason: the next stage was too expensive to start with.

Constraint 1 - High fidelity required engineering

Building something interactive used to require an engineer. So designers front-loaded cheap artifacts. Wireframes exist because prototypes cost too much. Handoff specs exist because designers could not write code.

Codex changed this. A designer can now build a functional prototype in minutes. No engineer required. No sprint ticket. No two-week wait.

Constraint 2 - Design files and code were separate worlds

A designer worked in Figma. An engineer worked in VS Code. Getting them in sync required handoff documents. Redline specs. Manual transcription of padding values and border radii.

The Figma MCP changed this. Design files and codebases now read from each other directly. One click. Pixel-perfect. No translation layer.

Constraint 3 - Feedback required the finished product

You could not test interactions in a wireframe. You could not feel tap targets in a mockup. You had to wait until engineering built it. Then you filed bugs. Then engineering fixed them. Then you filed more bugs.

Now you test interactions in code from day one. Before a single engineer touches it. The feedback loop collapsed from weeks to minutes.

What this looks like in practice

1. Inside OpenAI. The Codex design team has been shipping this way since December. Something changed when the models hit a capability threshold. Content designers are submitting PRs. PMs bring working prototypes to design reviews.

2. Inside Figma. Designers are working directly in staging. Monetization designers who never wrote code before are building technically complex prototypes. The phrase I keep hearing is there are no more P2s. The cutoff between ship and backlog is arbitrary now.

I covered the foundations of this shift in my AI prototyping for PMs guide. The difference now is that the tools on both sides have converged.

The old pipeline was not a design process. It was a workaround for expensive tools. The tools got cheap. The pipeline died.

2. The code-canvas loop

The new workflow is code + canvas in a loop.

The most common question about this workflow. Will it be lossy? Will things break when you translate between tools?

Short answer: it is already remarkably high fidelity. And it only gets better with every model improvement.

Direction 1 - Codex to Figma

You have a React app running locally. You have been building a composer system in code. You need to feel how buttons morph. How permissions prompts expand. Where tap targets land. Static mockups cannot answer these questions.

Now you want to go deep on the visual layer. Get pixel-perfect on a component. Swap icons. Test type scales.

Open the Codex desktop app. Drag the pop-out window over your running app. Type this -

Import my homepage and composer permissions state into a new Figma file. @figma

Three things happen -

1. Reads your file system. Codex understands the React structure, components, CSS

2. Snapshots the running interface. The Figma MCP captures your live app state

3. Opens it in Figma. Click Open in Figma. You get a real, editable design file

The result is not a screenshot. Everything is responsive.

1. Every padding value matches your CSS exactly

2. Every border radius is one-to-one

3. Every shadow transfers without manual transcription

4. You can select specific component nodes instead of full screens

The newest MCP improvement. The use Figma tool now references your actual design library. Components map to your design tokens and local styles. You are not working on a facsimile. You are working on a representation built from your real design system.

The snapshot is not a copy. It is a living bridge between your code and your canvas.

Direction 2 - Figma to Codex

The reverse direction is just as fluid.

You have been iterating in Figma. Changed the model name in the picker. Adjusted a color. Refined spacing on a card.

Copy a link to the modified component. Paste it into Codex -

Update my code with the change I made here [Figma component link]

Codex reads the link through the MCP. Diffs against your local codebase. Generates the code change.

For non-coding designers this is the unlock. You are no longer blocked. Make changes in Figma. Your engineer pastes the link into Codex. The change propagates automatically.

One engineer recently demonstrated aligning Storybook to GitHub to Figma with AI running the loop and finding diffs between them. The maintenance overhead of keeping design systems in sync, the thing every team complains about, can now be automated.

Three foundations that make this work

1. Name components well. Good naming is table stakes for humans. Essential for agents

2. Align design tokens with CSS tokens. Border radius, spacing scale, color palette. One-to-one mapping

3. Use annotations. Read annotations into the MCP so intent travels alongside the visual output

The handoff is dead. The loop replaced it.

Where AI still falls short

AI isn’t still where it needs to be for lossless movement. These areas are limited:

1. Shader effects do not translate to a static canvas

2. Complex CSS transitions cannot be fully represented in Figma yet

3. Edge case decisions that only exist in code because an engineer solved something that never got recorded in the design file

4. Web-specific effects that Figma’s canvas does not yet support

Annotations help bridge some of these gaps. But the designer’s judgment is still critical for the last mile.

The good news. Every model improvement makes this better. OpenAI’s model 5.4 produced a material jump in quality. Internal designers say it is meaningfully better at working with the Figma MCP than anything before.

If you tried this six months ago and gave up, try again. The reliability has crossed into daily-use territory.

The tools have limits. But those limits are shrinking every month instead of staying fixed.

3. When to use which tool

The biggest mistake teams make with this workflow. Trying to do everything in code. Or everything in canvas.

Match the tool to the question you are trying to answer.

Mode 1 - Go wide on ideas

Use the canvas.

You can see the whole flow in front of you. Rearrange screens. Explore lateral directions. Print it out and put it on a wall and tear it down.

The canvas is still the gold standard for divergent exploration. No code tool matches the spatial freedom of dragging artboards around and seeing twenty concepts at once.

When to pick this -

1. You are exploring completely new interaction paradigms

2. You need multiplayer collaboration with the whole team

3. You want to rally people around a hero image

4. You are doing deep design system work with tokens and color tests

Mode 2 - Test interactions in code

Use Codex.

How does the button morph? How does the composer resize? Where are the tap targets? What happens at different breakpoints?

Static mocks cannot answer these questions. You need something running.

When to pick this -

1. You are testing responsive behavior across mobile, tablet, desktop

2. You need to feel how transitions work in real time

3. You want to stress-test an engineering concept by forking a branch

4. You are building a prototype that needs real data

Mode 3 - Ship the last mile

Use both.

Build interactions in code. Pop them into Figma through the MCP. Go deep on pixel-perfect details. Push changes back to code. Ship.

This is where the loop becomes the most powerful. The round-trip that used to take a sprint now takes ten minutes.

When to pick this -

1. You are polishing a nearly finished feature

2. You need to fix button animations, loading states, string changes

3. Engineering is waiting and you want to unblock them without filing a ticket

4. You want to submit a PR yourself

I covered the decision frameworks in my Codex PM guide. The underlying principle is simple.

The question is never which tool. It is what are you trying to learn right now.

4. The 5-step adoption roadmap

If you are at a traditional company, and your design team follows the linear pipeline, and your company has not have procured these tools yet, here is the path:

Step 1 - Just start

Download the Codex desktop app. You do not need your company’s permission. Build something for yourself.

Real examples from inside OpenAI -

1. A GTM team member built an entire iOS app with zero iOS experience

2. A comms team member designed an interactive drag-and-drop seating plan in HTML

3. A design lead and his wife are building a Japan trip planner for their honeymoon

None of these people are engineers. They downloaded the app and started.

If you do not know what to build, ask yourself one question. What would I build if I could build anything? Then build it.

Step 2 - Get the loop working once

Install the Figma MCP plugin from the Codex app. Open a project you already have.

The checklist -

1. Import one screen from code to Figma

2. Make one change in Figma

3. Push that change back to code

4. Verify it worked

Get the loop working once end to end. This proves the pipeline before you scale it.

The foundation matters. Name your components well. Align design tokens with CSS tokens. Clean up your Figma file structure. If your files are messy it is harder for both colleagues and agents to get up to speed.

Step 3 - Start with polish not process

Do not replace your existing workflow on day one. Use code for the polish stage first.

1. The button animation that never feels right

2. The loading state engineering skipped

3. The string change stuck in the sprint backlog for three weeks

4. The spacing tweak that would take 30 seconds but requires a whole ticket

This is the lowest-risk entry point. You are not changing how the team designs. You are adding a new capability where the cost of iteration was always highest.

Step 4 - Shift the starting point

Once comfortable, change where design begins.

Instead of paper sketches or low-fi Figma frames, open Codex and describe what you want. Show the team something real in the first meeting.

What changes -

1. Edge cases surface in the first conversation instead of the third sprint

2. Product strategy discussions start with working software instead of static decks

3. Your wireframe gives the team more dimensionality to the problem than any mockup

4. Feedback is immediate because the thing exists

Step 5 - Use AI as your tutor

You will hit walls. You will not understand why a React component renders the way it does.

Ask. The AI is an infinitely patient tutor that never clocks out.

Questions that build facility -

1. Start simple. Can you build this? If it does something, ask how does that work

2. Go deeper. I just inherited this system. Can you explain the data architecture

3. Find gaps. Are there redundant systems? Look through the entire codebase

4. Learn structure. What is the difference between a layout page and a normal page

The people who succeed in this era are not the ones who already know how to code. They are the ones curious enough to keep asking questions.

The roadmap is not learn to code. It is learn to be curious in a world where every question gets answered.

5. Total football for product teams

The natural question from above is: If designers can code and engineers can design and PMs can prototype, do we even need separate roles?

Yes. And understanding why is the most important takeaway.

The three questions that define each role

1. Engineers ask how do we build this well

2. Designers ask how should someone experience this

3. PMs ask why should we build this at all

A designer who ships code still self-identifies as a designer. They still care about being the voice of the user. They still think about craft and flow and emotional experience. The medium changed. The mandate did not.

The skill layer accelerates the blurring

Inside Figma, designers are writing skills that teach AI how to design well. PMs write skills for product decision-making. Anyone can use anyone else’s skill.

1. A designer uses a PM’s skill to work through a strategic decision

2. A PM uses a designer’s skill to prototype a flow

3. An engineer uses a design skill to ship polish to production

The skill layer means you are no longer limited to a third of the picture. You can lean into adjacent domains without switching careers.

Why roles survive despite tool convergence

The constraint was never the role. It was the amount of time it took to become proficient in tools outside your domain. If you wanted to learn code that gated you. If you wanted to learn design that gated you.

Now those gates are open. What remains is natural inclination. Judgment. Taste.

The mental model is total football. In 1970s Holland, every player could play every position. The goalkeeper could attack. The striker could defend. But each player still had a natural spike. The team was more dangerous because everyone could cover for each other.

The phrase inside OpenAI - prototypes, not PRDs. PMs bring working prototypes to design reviews. They ship PRs to stress-test ideas with engineers. The artifact that aligns teams is now running software, not static documents.

The bottleneck has moved. If developers have been accelerated 10x, designers have been accelerated maybe 1.5 to 2x. Design can become the bottleneck if you are not coding yourself.

The tools are the same across roles. The questions are different. And the questions are what define the role.

Get Transcript

Where to find Ed Bayes

• LinkedIn

• OpenAI

• X

Where to find Gui Seiz

• LinkedIn

• Figma

• X

Related content

Podcasts:

1. Xinran Ma - Design with AI

2. Crash Course in AI Product Design

3. Codex PM Guide with Carl Vellotti

Newsletters:

1. AI prototyping for PMs

2. The PM guide to Bolt

3. Codex PM guide

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Share

Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by:

1. Bolt: Ship AI-powered products 10x faster

2. Jira Product Discovery: Plan with purpose, ship with confidence

3. Kameleoon: Leading AI experimentation platform

4. Amplitude: The market-leader in product analytics

5. Product Faculty: Get $550 off their #1 AI PM Certification with my link

Today’s episode

If you want to run a truly AI-pilled product team, you need to get everyone on a Team OS.

When (PM @ DoorDash, Author of In The Weeds) first described this concept I had never heard of it, but after she showed it to me, I realized it’s absolutely genius.

We’ve done a lot of Claude Code episodes on this podcast, but this episodes is something genuinely new and impactful for PMs.

It’s the architecture and steps to building a Team Operating System so that you can scale your impact as a PM and product team:

• You checking in every single thing you do to a repo

• Team members checking it before checking with you

• Everyone building one shared, compounding knowledge base

If you are building a team that runs on AI, this is the episode to watch.

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

I’m starting the third cohort of my Land PM Job program soon. Join us.

Newsletter deep dive

As a thank you for having me in your inbox, here is the complete guide to building a Team OS with Claude Code:

1. The Team OS structure

2. Context management theory

3. Scaling analytics across functions

4. How to write 10x docs with planning

5. The learning flywheel

Subscribe now

1. The Team OS structure

As a PM, you are the human router. Every question goes through you. Every answer lives in your head or in a doc no one can find. That does not scale when one PM supports 20 people across five functions.

A Team OS fixes this. One shared GitHub repo. Every function checks in their work. Any coding agent traverses it. Everyone self-serves.

Grab Hannah's Template Repo

Component 1 - The root Claude MD

The root file loads every session. Keep it to three things -

1. Doc index. A map telling Claude where every type of information lives. Without it, Claude runs expensive explore agents searching your file system. With it, Claude navigates directly on the first try.

2. Team roster with handles. Every person, their role, their Slack ID, their GitHub handle. Enables natural language queries like “Slack Alex about the bug from today’s call.”

3. Key Slack channels. Channel names mapped to IDs and purposes. “Send this in the product channel” just works.

Don’t stuff too much into the root file. If it is longer than one page, you are burning context on information not needed in 80% of sessions.

I covered the foundations of building a PM operating system in my setup guide. The root Claude MD is the single most important file in that system.

Copy-pasteable template -

# Team OS

## Doc index
- product/       - PRDs, strategy, competitive research, customers
- analytics/     - Metrics, queries, schemas, dashboards, playbooks
- engineering/   - Bug investigations, RFCs, technical designs
- team/          - Onboarding, retros, team-level docs
- .claude/       - Shared agents, commands, skills

## Team
- Alex Chen (eng lead) - @alexc (Slack) / @achen (GitHub)
- Morgan Li (designer) - @morganl (Slack) / @mli (GitHub)
- Taylor Kim (analyst) - @taylork (Slack) / @tkim (GitHub)
- Jordan Wu (strategy) - @jordanw (Slack) / @jwu (GitHub)

## Channels
- #product-team    - Daily standups and feature updates
- #eng-team        - Engineering discussions and PRs
- #data-insights   - Weekly metrics and analysis reports

Component 2 - Nested doc indexes

Every major folder gets its own Claude MD. These are not content files. They are navigation maps.

In the episode, a query about customers consumed only 3% of the context window. Claude did not touch analytics. Did not read engineering docs. Navigated directly to the right files.

Without nested Claude MDs, Claude runs explore agents across your entire repo. Burns tokens. Takes longer. Leaves less thinking room.

Template for every folder -

# [Folder name]

## Doc index
- subfolder/file - What it contains and when to read it
- subfolder/file - What it contains and when to read it

## Key context
1-2 sentences needed in 80%+ of sessions in this folder

Component 3 - The folder architecture

A production Team OS has three top-level sections -

1. .claude/ folder. Shared agents, commands, skills. Customer call summary skill. PR creation command. Weekly synthesis automation. Everyone uses these.

2. Product development. Customers, competitive research, PRDs, strategy docs, launch emails, meeting summaries, analytics (metrics/queries/schemas), engineering (bugs/RFCs). Organized by function, then product area.

3. Team folder. Onboarding guides, retros, team-level docs.

The full directory tree -

team-os/
├── .claude/
│   ├── agents/
│   ├── commands/
│   └── skills/
│       └── customer-call-summary.md
├── product/
│   ├── CLAUDE.md
│   ├── customers/
│   │   ├── CLAUDE.md
│   │   ├── acme-corp/
│   │   │   ├── CLAUDE.md
│   │   │   ├── calls/
│   │   │   └── summaries/
│   │   └── forge-labs/
│   ├── competitive/
│   ├── prds/
│   ├── strategy/
│   │   ├── plans/
│   │   └── vision/
│   └── workflows/
├── analytics/
│   ├── CLAUDE.md
│   ├── billing/
│   │   ├── metrics.md
│   │   ├── queries/
│   │   └── schemas/
│   └── onboarding/
├── engineering/
│   ├── bugs/
│   └── rfcs/
└── team/
    ├── onboarding/
    └── retros/

The ownership model matters too. On the team in the episode, functional leads own their folders but the whole team agrees on structure. The data scientist owns analytics. Engineers own bugs and RFCs. The PM owns product context. Strategy partners own customer calls.

A non-technical strategy partner who had never opened GitHub two months ago now puts up PRs every day. This is not just for technical people.

Every level of nesting is a context-saving decision. The more precisely you organize, the less Claude has to read. That is the whole game.

2. Context management theory

You open Claude Code. You paste in a PRD. A competitive analysis. Three customer call transcripts. You have already consumed half your context window before you even ask a question.

No thinking room left. That is the trap.

The Team OS is built on the opposite principle. Load only what you need, when you need it.

The four pillars

1. Context. The information in a given LLM session. What the model can access right now.

2. Context window. How much it can hold. ~1M tokens today. Seven to eight novels. But the docs produced by a single team far exceed that.

3. Compaction. When the window fills, information gets compressed. You lose fidelity. Compressed summaries are much less useful than originals.

4. Thinking room. The gap between loaded info and window size. This is where the model reasons. More context loaded = less room to think.

I wrote about these foundations in my context engineering guide with an OpenAI product leader.

The token efficiency framework

Not all context is equal. A well-structured repo has three tiers -

1. Tier 1 - Always loaded. Root Claude MD. Team roster. Channel map. Under 500 tokens. Loaded every session.

2. Tier 2 - Loaded on query. Folder-level Claude MDs. Doc indexes. 200-500 tokens each. Only loaded when Claude navigates to that folder.

3. Tier 3 - Loaded on demand. Actual content files. PRDs, transcripts, SQL queries. Hundreds to thousands of tokens. Only loaded when specifically needed.

The magic is in the split. If you only need to know what a metric measures, Claude reads metrics.md and stops. Does not pull SQL queries. Does not pull table schemas.

Someone who burns hundreds of thousands of tokens and hits their usage limit in 30 minutes generally has unstructured, unoptimized context. The Team OS eliminates this.

Failure modes that burn context

1. Flat repo with no Claude MDs. Claude runs explore agents for every query. Burns thousands of tokens just navigating. I have seen teams hit usage limits in 30 minutes because of this.

2. Overstuffed root file. Everything dumped into one Claude MD. Loaded every session whether needed or not. Kills thinking room before you even start working.

3. Full transcripts instead of summaries. A one-hour customer call is 10,000+ tokens. A structured summary is 500. The repo points Claude to summaries first. Full transcripts only if the summary does not answer the question.

The skill that fixes the third failure mode - a shared customer call skill. Every team member summarizes calls in the same format. Same structure. Same fields. Cross-customer analysis becomes trivial.

Here is the pattern. You have ten people taking calls with twenty different customers. Without the skill, every summary looks different. With the skill, Claude can compare apples to apples across hundreds of calls in seconds.

Context management is not a nice-to-have. It is the difference between an AI that helps and an AI that hallucinates.

3. Scaling analytics across functions

Your engineer is on call at 2 AM. Something looks wrong with billing. They need to check a dashboard. They need the query for churn by segment. They need the table schema to validate the data.

You are asleep. Your analyst is asleep. The engineer is stuck.

This section pays for the entire Team OS investment.

Layer 1 - Metrics, queries, and schemas

Organize by product area, then by data type -

analytics/
├── billing/
│   ├── metrics.md          # Definitions only
│   ├── queries/
│   │   ├── generation_success_rate.sql
│   │   └── churn_by_segment.sql
│   └── schemas/
│       ├── billing_events.md
│       └── subscription_tables.md
├── onboarding/
│   ├── metrics.md
│   ├── queries/
│   └── schemas/
└── dashboards.md           # Links to all dashboards

The split is intentional -

1. What are our billing metrics? - Claude reads metrics.md only. Low context.

2. Show me the churn query. - Claude reads metrics + query file. Medium.

3. Full metric + query + schema. - All three files. Higher but still targeted.

4. Point Claude at the raw database. - Everything loaded. Maximum burn. Maximum hallucination risk.

The prompt that shows the power - “How do we calculate generation success rate? Show me the metric definition, the SQL query, and the table schema.” Claude navigates to the right product area. Pulls exactly three files. Gives you everything.

Hook this up to Snowflake MCP or another analytics tool. Claude does the actual analysis using verified approaches your analyst checked in.

Layer 2 - Playbooks and verified approaches

Your data scientist has a process for investigating funnel drop-off. That process belongs in the repo as a playbook.

When you ask “why are users dropping off during custom domain setup?” Claude finds the playbook. Follows the same methodology your analyst uses. Produces a result you can trust.

This is how you kill hallucinations in data analysis. Not by hoping. By giving Claude verified approaches.

The failure mode to watch for - not having the analyst audit the playbooks. If Claude follows a wrong methodology, it produces confidently wrong results. Your data scientist must own and verify the analytics folder.

Layer 3 - The feature launch gate

Make this non-negotiable. A feature is not launched until the repo is updated.

The checklist -

1. Metric definitions for the new feature checked in

2. SQL queries verified by your analyst

3. Table schemas documented

4. Dashboards linked

5. Playbooks for common investigations created

This is how shared context compounds. Every sprint, the repo gets more valuable. Every function gets more self-sufficient.

I covered the broader launch process in my product launch playbook. Add repo updates as a hard gate.

A second gate worth adding - the repo is part of the PRD itself. When you write a PRD for a new feature, include a section specifying what metrics, queries, and schemas will be checked into the repo as part of the launch.

The repo is not documentation. It is the system that makes everyone on the team faster every single day.

4. How to write 10x docs with planning

You type a prompt. Claude makes all the decisions. You get something back. You spend two hours fixing it.

That is not Claude Code’s fault. That is a planning failure.

The junior employee metaphor is perfect. You hired a brilliant junior. Then gave them zero guidance.

Three prompting tiers

1. Basic prompt. Type request. Claude decides everything. Unpredictable quality. Fine for quick lookups. Terrible for strategy docs.

2. Lightweight alignment. Add “give me a proposal first.” Claude proposes an approach. You correct direction in 30 seconds. I use this for anything with even slight ambiguity.

3. Full plan mode. Press Shift+Tab twice. Claude’s bias for action is removed. Cannot execute until you approve the plan. This is where real quality happens.

In the episode, even the lightweight alignment produced dramatically better results. Claude used the repo context to generate a competitive analysis proposal without being told to. Found the right folders. Read the right docs. Proposed the right structure.

The full planning process

Five phases for a strategy doc -

1. Phase 1 - Load context. Claude reads competitive research, vision docs, writing guides in parallel. Structured repo = only relevant files loaded.

2. Phase 2 - Ask user questions. Claude uses the ask-user-question tool. Who is the audience? What is the focus? Should we update competitive intel?

3. Phase 3 - Build the plan file. Section-by-section document structure. You read the plan. Push back. Refine.

4. Phase 4 - Push your thinking. The advanced move -

Use ask-user-question tool to push me on my thinking.
Help me consider other angles for this document.
Challenge my assumptions. Take as long as you need.

Claude interviews you. Catches gaps. Suggests sections you missed.

5. Phase 5 - Review agent prompts. For complex plans, check what each agent will be prompted with. What files will it read? What writing guide? Skills auto-invoke only ~70% of the time. Always explicitly specify in the plan.

Here is the thing. Most people rush in. Let it write a bad first draft. Then yell at it for getting things wrong. The fix is not better prompting after the fact. It is better planning before you start.

I covered the Claude Code setup foundations. Plan mode is where intermediate users become advanced.

Parallel agents and temp files

A single agent cannot read 40 context files and write a great long doc. Context fills. Quality degrades.

Fix these by doing followling-

1. Split the document across agents. Each gets specific context files and your writing guide.

2. Each agent writes to a temp file. If ten agents return work to the parent simultaneously, the context window overflows. You lose everything. You must prompt Claude to use temp files - it does not always do this automatically.

3. The orchestrating agent compiles the final document from temp files.

Two more advanced techniques -

1. Verification prompts. Tell Claude how to self-check. Require sources for claims. Use Playwright MCP to verify front-end output.

2. Save plan files to the repo. Native plan files get wiped every 24-72 hours. If you spent three hours on a plan, save it. Next time you start at 80% done. Your team can reuse it. OpenAI published that they made plan files first-class artifacts of their engineering repos.

The decision matrix for when to plan -

1. Straightforward task, clear scope - lightweight proposal. 30 seconds.

2. Complex doc, some ambiguity - full plan mode. 15-30 minutes on the plan.

3. Highly ambiguous, multi-phase research - full plan mode with thinking partnership. Could be hours spread across a day.

4. Recurring process - saved plan file in the repo. Start at 80% done every time.

Most people under-plan. That is why the output does not match what they wanted. The plan is not overhead. The plan is the work.

5. The learning flywheel

After 1,500 hours, Hannah is still iterating on her setup every single day. You should be, too.

The beginner’s mindset

Ask Claude to teach you. This one habit accelerates learning faster than anything else.

1. When you encounter a file you do not understand - explain why this is a YAML file and not markdown

2. When you download someone else’s skill - explain why this skill works and what could be improved

3. When you want to improve your repo - analyze this repository structure and suggest improvements

The prompt I run every day -

Research everything Anthropic shipped in the last 90 days. Read the top Claude Code influencers and their best posts. Compare my setup to the latest features. Tell me how to 10x my setup.

Claude’s training data gets stale. This prompt forces it to find what is new and apply it to your system.

I covered this progression in my AI PM learning roadmap. Claude Code fluency is one of the highest-leverage skills on it.

The automation-to-learning loop

If you only have two hours this weekend, ask one question - what can I automate to free up six hours next week?

The flywheel -

1. Automate one task → Free up time

2. Use freed time to learn → Improve your repo

3. Better repo → More automation possible

4. More automation → Even more time freed

Every MCP you hook up makes the system more powerful. The rule from the episode - any core software you use daily should be connected. The limit does not exist.

Shared automations compound the flywheel across the team. A weekly customer research synthesis that posts to Slack. A PR notification command that tags the right reviewer. A metrics check that runs every morning.

Mistakes that stall progress

1. Giving up after day one. Building a Team OS takes weeks. You are building infrastructure, not running a prompt. Commit to 30 days.

2. Copying without understanding. People download skills from the internet. Do not know why they work. When they break, they cannot fix them. Always ask Claude to explain first.

3. Treating it as a coding tool. The most misleading name in AI. Most PM time in Claude Code is writing docs, doing analysis, building prototypes, running automations. Not writing code.

4. Not clearing between tasks. Type clear when switching. Leftover context pollutes results.

5. Context rot. Not updating the repo. Claude uses outdated competitive intel. Outdated metrics. The repo must be a living system that every team member updates as part of their workflow.

The terminal is not scarier than a chatbot. Once you have typed into it for an hour or two, you feel comfortable. The barrier is psychological, not technical.

The PMs who build a Team OS this quarter multiply their leverage by 10x. The PMs who keep being the bottleneck for context just made themselves the slowest person on the team.

Get transcript

Where to find Hannah Stulberg

• LinkedIn

• In the Weeds Substack

• Her Github repo

Related content

Podcasts:

1. My Claude Code PM OS with Dave Killeen

2. Claude Code + Analytics with Frank Lee

3. Claude Code as PM OS with Carl Vellotti

Newsletters:

1. The ultimate guide to context engineering

2. Build your PM operating system

3. How to use Claude Code like a pro

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Share

Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by:

1. Bolt: Ship AI-powered products 10x faster

2. Amplitude: The market-leader in product analytics

3. Pendo: The #1 software experience management platform

4. NayaOne: Airgapped cloud-agnostic sandbox

5. Product Faculty: Get $550 off their #1 AI PM Certification with my link

Today’s episode

Claude Code hit $2.5 billion in annualized revenue in 12 months.

It is the fastest B2B software product ramp in history.

So why are most people still using it like a chatbot?

This is how most people use Claude Code. Type a prompt and get output. The context fills up. It compacts. You lose everything. You start over.

The top users flipped it. They built operating systems. Skills, sub-agents, file structures, hooks. Every prompt benefits from everything built before.

I sat down with Carl Vellotti for the third time. His first episode was the beginner course. His second episode was the advanced masterclass. Together they crossed over a million views across platforms.

Today is the operating system layer. If you are already an 80 out of 100 on Claude Code, this episode will bring you to a 95.

If you are living in Claude Code 8 to 10 hours a day and want to stop fighting the tool, this is the one episode to watch.

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

I’m putting on a free webinar on Behavioral and AI PM interviews. Join me.

Newsletter deep dive

As a thank you for having me in your inbox, here is the complete step-by-step guide to building your own Claude Code operating system.

Two starter repos you can fork today. Every file explained. Every technique from the episode plus what I have learned building my own PM OS.

1. Fork your starter OS

2. The CLAUDE.md file

3. The compounding file system

4. Mastering context

5. Skills that fix weaknesses

6. Data you can actually trust

7. The daily loop

1. Fork your starter OS

You do not need to build from scratch. Carl open-sourced his personal operating system at github.com/carlvellotti/carls-product-os. I open-sourced mine at github.com/aakashg/pm-claude-code-setup.

Carl’s repo comes with a BLANK-OS folder you rename and fill in, plus an EXAMPLE-OS with real content so you can see what a populated system looks like.

Step 1 - Fork and clone

# Carl's OS (lighter starting point, free)
git clone https://github.com/carlvellotti/carls-product-os.git
cp -r BLANK-OS/ my-workspace && cd my-workspace

# Or mine (41 skills, 7 sub-agents, $49)
git clone https://github.com/aakashg/pm-claude-code-setup.git my-pm-os
cd my-pm-os

Step 2 - Open in Claude Code

That is it. You are inside a pre-structured operating system. Every folder, every placeholder, every workflow is in place. Fill it with your data.

Carl draws a useful distinction. Workflows tell Claude how to create something. Templates tell Claude what the output looks like. Knowledge is reference material useful across projects. Project research is scoped to one project and gets archived with it. Your brand guidelines go in knowledge. Competitor research for a specific launch goes in the project folder.

You do not need everything on day one. Start with CLAUDE.md + GOALS.md. These two files give you 80% of the value. Add structure when friction appears.

2. The CLAUDE.md file

This is the most important file in your operating system. It is automatically loaded into context for every single message. Not just the first. Every one.

I have iterated on my CLAUDE.md over 100 times. What you think will work does not always. After lots of testing, here is what actually matters.

What goes in it

# About me
- Name: [Your name]
- Role: Senior PM at [Company]
- Team: Growth, reporting to [Manager name]
- Current quarter: Launching [feature], improving [metric]

# How I work
- Short, direct communication. No filler.
- Show reasoning before conclusions.
- When I say "spin up", delegate to a sub-agent.
- Check /knowledge/people/ before drafting any message.
- Use Jupyter notebooks for any data analysis.

# My tools
- GitHub CLI: installed (use for all git operations)
- Vercel CLI: installed (use for deployments)
- Firecrawl CLI: installed (use for web scraping)
- Tavily MCP: enabled (use for web search only)

# Current priorities
1. Ship auth redesign by March 28
2. Prepare board deck for Q2
3. Hire senior designer

# My skills
See /.claude/skills/ - auto-matching hook is active.

# Preferences
- Bold key phrases in documents
- Bullet points for 3+ items
- End sections with a punchy one-liner

The compounding trick

Update this file every week. Every time you hit an issue with Claude Code, pull the fix in. “Do not run research in the main session. Always use a sub-agent.” “When I say spin up, delegate to a separate instance.” Over months, this file becomes an incredibly precise instruction set. Claude feels like it already knows you.

I covered the CLAUDE.md deep dive in my Dave Killeen episode, where the CPO at Pendo told me his Claude Code system is better than the human EA he used to have. The CLAUDE.md was the foundation of everything he built.

If you only set up one file, make it this one.

3. The compounding file system

The fundamental difference between a Claude Code OS and just chatting with AI is that everything is a markdown file. And those files are alive.

In a normal AI conversation, everything lives in one chat window. Close it and it is gone. In a Claude Code OS, the AI reads and writes to persistent files. Every conversation makes the files better. Every file makes the next conversation better. That is the compounding loop.

The folder structure

my-workspace/
├── CLAUDE.md              ← Always in context, every message
├── GOALS.md               ← Quarterly priorities
├── .claude/
│   ├── skills/            ← Custom commands
│   └── hooks/             ← Auto-trigger scripts
├── knowledge/
│   ├── people/            ← Stakeholder dossiers
│   ├── company/           ← Strategy, positioning, competitors
│   └── research/          ← User research, industry reports
├── projects/
│   ├── [project-name]/    ← One folder per active project
│   └── _archive/          ← Completed projects
├── sub-agents/            ← Reviewer personas
├── workflows/             ← Process templates
├── templates/             ← Output templates (PRDs, OKRs)
├── data/                  ← CSVs, Jupyter notebooks
├── tasks/
│   ├── current.md
│   └── backlog.md
└── tools/                 ← Pre-built scripts

The knowledge folder

People files are where the real compounding happens. Create one for each stakeholder at knowledge/people/[name].md:

# Sarah Kim - Design Lead
- Reports to: VP Design
- Cares about: User research, accessibility, craft
- Communication style: Visual thinker. Wants mockups, not docs.
- Pet peeve: Decisions made without user data

## Recent context
- Pushing back on the auth redesign timeline
- Concerned about mobile accessibility
- Liked the competitive analysis I shared last week

## Meeting notes
### March 21 - Auth redesign sync
- Wants mobile prototypes before eng sprint
- Asked for usability report from last iteration
- Action: I owe her the report by Friday

Connect a meeting transcription tool like Granola via MCP. After every meeting, tell Claude: “Update the people file for Sarah based on today’s meeting.” Over time, when you draft a message to Sarah, Claude already knows she prefers visual communication and is concerned about mobile. The output is dramatically different from a generic draft.

Company files hold strategy, positioning, and competitive landscape. Research files hold user interviews, industry reports, and anything you reference across projects.

The projects folder

Every new task gets its own folder. When you come back the next day:

@projects/board-deck-q2/ Get up to speed on this project, 
then help me finish the revenue slide.

Claude reads every file in the folder. All the research, all the drafts, all the data. It picks up exactly where you left off.

Bootstrap fast. Do not build project folders one by one. Connect Linear or Notion via MCP and run:

Pull my last 5 active projects from Linear. For each one, 
create a folder in /projects/ with the PRD and latest status.

The people folder compounds across all projects. The project folder compounds within one. Together they mean Claude never starts from zero.

4. Mastering context

You have the file system. Now the biggest technical challenge: making sure Claude can actually use it without running out of context.

I covered the foundations in my context engineering guide. Here is how it works inside Claude Code specifically.

Step 1 - See your context

Run this right now:

/status line

Ask for a color-coded context meter. Green under 50%. Orange 50-80%. Red over 80%. Then run /context to see exactly what is eating it:

1. System prompt - ~2% (cannot change)

2. MCP servers - ~8%+ (each one adds permanent overhead)

3. Custom agents - ~4%

4. Conversation - grows with every message

Before you type a single message, 10% to 16% is already gone. Disable anything you are not actively using.

Step 2 - Delegate to sub-agents

Without sub-agents, a research task pushes context from 16% to 25%. Almost 10% of your window gone for one question.

With a sub-agent, the same task costs 0.5%. Claude spins up a separate instance. That instance does all the web searches and page reads. Your main session only gets the summary.

The numbers from the episode:

1. Without sub-agent - 16% to 25% context (+10%)

2. With sub-agent - 16% to 16.5% context (+0.5%)

3. Sub-agent used 10 tool calls and 30,000 tokens, all outside main session

Two ways to trigger this:

1. Inline - Add “spin up” or “use a sub-agent” to any prompt

2. Skill - Build a skill at .claude/skills/delegate-research.md that automatically delegates research tasks. Both repos include this.

Bonus trick: If Claude goes on a tangent, hit Escape twice. Roll back to before the bad prompt. Everything after it is erased from context completely.

Step 3 - Replace MCPs with CLIs

This is the hierarchy most people get backwards. MCPs eat context just by existing. CLIs sit on your machine with zero context overhead.

Connection Context cost When to use MCP High (always loaded) Last resort API Medium Custom integrations CLI Zero Always prefer

Andrej Karpathy confirmed it. The CLIs to install today:

1. GitHub CLI (gh) - Claude is aggressively competent at it. PRs, issues, repos, zero context cost.

2. Vercel CLI - Deploy, check logs, set env variables without leaving Claude Code.

3. Firecrawl CLI - Clean web scraping without MCP overhead.

The test: for every MCP you have enabled, ask Claude “does a CLI for this exist?” More often than you think, the answer is yes.

Every MCP you replace with a CLI is context you get back for actual work.

5. Skills that fix weaknesses

Every time someone says “Claude Code is bad at X,” the real problem is the same. Claude does not have the right tools or instructions. I covered the basics in my Claude Code guide. Here is the advanced version.

Tier 1 - Prompt-only skills

No code. No APIs. Just a well-written prompt. Anthropic has 56 official plugins you can install with /plugins. The front-end design one is the best example. All it does is tell Claude “do not look like AI.” Before-and-after quality is dramatic.

To build your own, create a file at .claude/skills/[name].md:

---
name: write-like-me
description: Use when the user asks to write, draft, or edit.
---

# Writing Style Skill

1. Short sentences. One idea per sentence.
2. Use "you" and "your" when teaching.
3. Open with the problem, not the solution.
4. Bold the key phrases.
5. End every section with a punchy one-liner.

Tier 2 - Tool-powered skills

Give Claude tools to check its own work. The make slides skill from the episode includes Puppeteer so Claude can screenshot its HTML output, measure text overflow, and fix issues automatically. Tell it “iterate 3 times before showing me” and the first result is already polished.

The web research skill replaces Claude’s unreliable default web search (basically Google first-page results) with Tavily for higher-quality search and Firecrawl for clean markdown scraping. Install Firecrawl as a CLI, not an MCP, to save context.

A key finding from a Google paper: literally pasting a prompt twice improves output. The practical version: after Claude executes any skill, tell it “double-check your output against the skill instructions.” Quality improves every time. This is the builder-validator pattern.

Whenever you catch yourself screenshotting and saying “fix this” repeatedly, that is a signal. Build a self-checking skill.

Tier 3 - Auto-invoking with hooks

Skills are supposed to auto-invoke from keywords. It does not work reliably. The fix is a user_prompt_submit hook that runs a lightweight script matching your message against skill keywords. Zero context cost. Instant. Claude still decides whether to use it.

This beats listing skills in CLAUDE.md because it adds no permanent context overhead and auto-updates when you add new skills. Both repos include the hook script ready to use.

The ask user questions tool

This was Carl’s favorite Claude Code feature, and one I had never used before this episode.

Tell Claude “use your ask user questions tool” and it generates a custom UI with checkboxes and input fields right inside the terminal. Instead of making assumptions, Claude asks you. Sometimes 10 questions. Sometimes 67.

Use it for:

1. Requirements gathering - “Use your ask user questions tool to grill me on all possible requirements for this feature”

2. Filling context gaps - “Ask me everything you need before starting this project”

3. Decision making - “Here are 10 research results. Help me decide which to implement”

The main reason people do not like AI output is assumptions. This tool eliminates them.

Skills are what you do. Hooks are how you compound. The ask user questions tool is how you eliminate assumptions.

6. Data you can actually trust

The biggest question PMs have about AI outputs: how can I trust this?

You are presenting numbers to your boss. They ask “where did this come from?” You need proof of work.

The answer is Jupyter notebooks. Tell Claude “analyze this data in a Jupyter notebook” and it creates a .ipynb file showing every query as a code cell, every result as output, every chart rendered inline. Renders natively in VS Code or Cursor.

The progression Carl demonstrated live:

1. Visualize - Load the CSV, see columns, shape, sample rows

2. Chart - Distribution charts, bar charts, segment breakdowns

3. Analyze - Correlation heatmaps, regression, statistical tests

At each step, your data scientist or manager can read every cell, verify the methodology, and check the results. The code is the proof.

The mental model: take the roles you work with and create versions of Claude Code for each one. The front-end design skill is the designer. Jupyter notebooks are the data analyst. The web research skill is the research assistant.

The difference between trusting AI and not trusting AI is visibility into the process.

7. The daily loop

This is where the operating system comes alive. Everything from sections 1 through 6 feeds into one daily workflow.

Create a standup skill that pulls from all your sources. GitHub CLI for recent commits. Tasks/current.md for priorities. Calendar for meetings. Linear for ticket status. Knowledge/people/ for meeting prep with stakeholder context already loaded.

One command. No tab switching. No manual assembly.

The compounding loop

You have a meeting with Sarah. Granola transcribes it. Your “update people” skill pulls key points into Sarah’s people file. Later, you draft a message to her about the timeline. Claude reads her file. It knows her concerns from the last meeting. The message is specific, not generic.

You finish a project. You move it to _archive/. Next quarter, you start something similar. You tell Claude “look at the archived auth-redesign project.” It reads the PRD, the user research, the launch results. Your new project starts with all the context of the old one.

You hit an issue with Claude Code. You add a rule to CLAUDE.md. Every future session inherits that rule. The same mistake never happens twice.

That is what an operating system is. Not a folder full of files. A system where every interaction makes the next one better. The compounding is the whole game.

How to start today

Fork the starter OS. Go to Carl’s product OS or mine. Clone it. Open it in Claude Code.

Fill in your CLAUDE.md and GOALS.md. These two files give you 80% of the value.

Set up the context status line. Run /status line. You will never unsee how fast your context fills up.

If you want the full system with 41 skills, 7 sub-agents, and the CLAUDE.md I have iterated 100+ times, get the PM OS here. If you want Carl’s free course that teaches Claude Code inside Claude Code, go to ccforpms.com.

Every person file you update, every project folder you populate, every skill you build makes the next prompt dramatically better. That is what turns Claude Code from a chatbot into an operating system.

Get Transcript

Where to Find Carl Vellotti

• Linkedin

• X (Twitter)

• Instagram

• Newsletter

• Claude Code for PMs course

Related content

Podcasts:

1. Claude Code Masterclass with Carl Vellotti (Ep 2)

2. Claude Code Initial Guide with Carl (Ep 1)

3. Claude Code PM OS with Dave Killeen

Newsletters:

1. Claude Cowork and Code setup guide

2. How to use Claude Code like a pro

3. My PM OS

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Share

Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by

1. Product Faculty: Get $550 off their #1 AI PM Certification with my link

2. Amplitude: The market-leader in product analytics

3. Pendo: The #1 software experience management platform

4. NayaOne: Airgapped cloud-agnostic sandbox

5. Kameleoon: Leading AI experimentation platform

Today’s episode

There’s very few AI PMs who have seen inside a single company like Netflix, Meta, or Amazon. It’s even more rare to find one who has experience at all three.

That’s who I’ve brought in today.

Jyothi Nookula breaks out the whiteboard for a full deep dive covering all aspects of AI PM fundamentals and job searching for an AI PM job.

If you are trying to break into AI PM, this is the one episode to watch.

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

I’m putting on a free webinar on Behavioral and AI PM interviews. Join me.

Newsletter deep dive

As a thank you for having me in your inbox, here is the complete guide to becoming an AI product manager in 2026.

1. Taxonomy of AI PM Roles

2. The Core AI PM fundamentals

   • When to use AI

   • Which AI to Use When

   • The 3 Building Blocks of an AI Feature

3. How to Become an AI PM

Subscribe now

1. Taxonomy of AI PM Roles

Before you write a resume, update a portfolio, or prep for a single interview, you need to answer two questions.

What type of AI PM role are you targeting? And where in the stack do you want to sit?

Get these wrong and you’ll spend months preparing for interviews that test completely different skills than what you studied.

Axis One - Traditional PM with AI features vs AI Native PM

Type 1 - Traditional PM with AI features

This is 80% of what is labelled as AI PM jobs right now. Companies adding LLM capabilities to existing products.

1. A chatbot on a customer service portal

2. AI summarization inside a document tool

3. Smart search on an existing knowledge base

The core product existed before anyone bolted an LLM onto it.

Type 2 - AI native PM

The remaining 20%. Here the product IS AI.

1. OpenAI

2. Anthropic

3. Cursor

The product is fundamentally probabilistic. The value proposition is impossible without AI. You cannot build ChatGPT without an LLM.

There are 4x more open roles in the traditional category. If you are trying to break into AI PM, that is where the volume is. But the AI native roles are where the hardest product problems live.

Know which type you are applying for before you write a single line on your resume. The skills overlap. The interview questions do not.

Axis Two - Where in the AI Stack You Fit

After the role type, the second axis is where in the stack you sit.

The deeper you go, the harder the technical bar.

Layer 1 - Application PMs (60% of roles)

Own the end-to-end user experience. How users interact with AI. How to build trust. How to make AI reliable for everyday use.

This is the easiest entry point for someone converting from a traditional PM role. It builds on existing product management skills with AI knowledge layered on top.

Layer 2 - Platform PMs (30% of roles)

Build tools that other teams use.

1. Developer platforms

2. Model orchestration systems

3. Evaluation frameworks

4. Observability tools

You are not building for end users. You are building for other builders.

Layer 3 - Infra PMs (10% of roles)

Build the foundational systems that power everything above.

1. Vector databases

2. GPU orchestration

3. Model serving optimization

4. Kernel level compilation

The good news? The hardest roles are the smallest bucket.

The layer you choose determines the interview prep, the portfolio, and the companies you target. Pick your layer before you pick your first project.

2. The Core AI PM Fundamentals

Three things separate the AI PMs who ship from the ones who flounder: knowing when to use AI, knowing which AI technique fits the problem, and understanding the technical building blocks well enough to make real product decisions.

Let’s go through each.

2a. When to use AI

Here is where most teams get it wrong. Not a technical mistake. A product decision mistake.

MIT found that 19 out of 20 AI pilots fail. A key factor? Picking the wrong problems. Teams reach for AI when a rules-based system would have solved it faster, cheaper, and more reliably.

I covered the decision frameworks in my AI product strategy post. Here is the decision tree from the episode.

AI makes sense in three patterns

1. Pattern recognition in complex data. YouTube uses ML to identify patterns across billions of viewing sessions. The relationships are multi-dimensional. No rules could capture that.

2. Prediction from historical data. At Amazon, AI forecasts inventory needs based on hundreds of variables. Seasonal trends, promotions, weather patterns. Humans cannot process that many variables.

3. Personalization at scale. Individualised experiences for millions of users. Content recommendation engines are the classic example.

Heuristics win in four situations

1. Explainability is non-negotiable. AI models still have low explainability. Regulated industries need auditability.

2. Clear domain rules exist. Tax calculation. Tax codes are complex but explicit. Perfect for rules.

3. Data is limited. New features, new markets, no historical data. Do not force-fit AI.

4. Speed is critical. AI systems take longer to build. For MVPs, ship first. Add AI later.

The best AI PMs know when to say no to AI. That judgment is more valuable than knowing how to build a RAG system.

2b. Which AI to Use When

You decided AI is the right tool. Now which AI?

Most teams jump straight to LLMs when a simple ML model would have solved the problem in a week at a fraction of the cost. I wrote about this trap in my RAG vs fine tuning vs prompt engineering guide.

Traditional ML

Regression models. Random forests. XGBoost. Mature. Reliable. Still powers most of the AI you use daily.

Choose ML when:

1. You have structured data and need to predict or classify

2. You need the model to explain its decisions

3. Speed and cost matter

The PM question: Can I put this problem in a spreadsheet with clear input columns and an output I want to predict? If yes, start with ML.

Deep learning

Neural networks. Computer vision. Speech recognition.

Choose deep learning when:

1. You are dealing with images, video, or audio

2. Humans can do the task easily but you cannot write rules for it

3. Pattern recognition is too sophisticated for traditional ML

The face recognition example from the episode is perfect. You recognise a face instantly. Writing if-then statements to replicate that? Impossible.

Gen AI

LLMs. Diffusion models.

Choose Gen AI when:

1. Users need to interact conversationally

2. You are creating new text, images, or code

3. You need reasoning and synthesis across multiple sources

The optimization hierarchy (before fine tuning)

This is the most important takeaway for anyone building with AI right now.

1. Prompt optimization - better prompts, better results

2. Context engineering - what information gets loaded into the context window

3. RAG - dynamically retrieve relevant knowledge

4. Fine tuning - only if 1-3 do not work

80% of use cases get solved with RAG. Teams that jump to fine tuning because it is in the API documentation burn months unnecessarily.

These are not competitors. They are tools in your toolkit. The best AI products combine multiple approaches.

2c. The 3 Building Blocks of an AI Feature

These are three building blocks every AI PM must know. Each builds on the last.

Building Block 1 - Agents vs workflows

An AI agent makes decisions and takes actions on its own. You do not tell it the exact order. It understands the goal and reasons its way there.

A workflow is the opposite. Predetermined sequences. Every step defined. Deterministic outcomes.

The agent architecture

Four components.

1. Perception - how it receives input (text, images, APIs)

2. Reasoning - where models live (LLMs, classifiers, planning algorithms)

3. Execution - how it takes action (text generation, API calls)

4. Learning - feedback mechanism for improving over time

Building Block 2 - Prompts and Context Engineering

Prompts are the primary interface between you and the AI system. And most teams treat them as an afterthought.

Three layers to understand:

System prompts set overall behavior and personality. For a customer service agent, this means defining empathetic personality, professional tone, identity verification requirements, and escalation rules. The system prompt is the AI’s job description. Write a bad one and you’ll get a bad employee.

Few-shot examples are the most underrated technique in production AI. Show the model concrete examples of what a good response and a bad response look like. It sounds like extra work. In production, it is the single highest-leverage investment you can make. I’ve seen teams double response quality by adding 3-5 well-crafted examples instead of writing 3 more paragraphs of instructions.

Context engineering is where the real PM skill lives. I covered the full framework in my context engineering guide, co-written with an OpenAI product leader. The key insight: context engineering is a PM problem, not an engineering problem.

Three context layers matter:

1. Immediate context — the current conversation or task. What did the user just ask?

2. Session context — recent interactions and state. What happened earlier in this conversation?

3. Knowledge context — broader reference information. This is where RAG comes in.

Here’s why this matters commercially. Claude Sonnet has a 200K token context window. That sounds like a lot. It fills fast when you’re loading a knowledge base, conversation history, real-time data, and user prompts simultaneously. Every token costs money.

The teams that overspend on AI are almost always loading too much context into every interaction. A customer asks about their order status and the system loads the entire product catalog, the user’s full purchase history, and a 50-page return policy document. That’s wasteful. Context engineering is the art of knowing what to load and when — giving the model exactly enough information to answer well, and nothing more.

Building Block 3 - RAG (Retrieval-Augmented Generation)

RAG (Retrieval Augmented Generation) is the single most important technique for enterprise AI. I covered the fundamentals in my RAG vs fine tuning guide.

1. Take your company documents

2. Chunk them into smaller pieces

3. Convert chunks into vectors using an embedding model

4. Store vectors in a vector database

5. When a user asks a question, vectorize the query

6. Find the nearest matching chunks

7. Load those chunks alongside the user input into the LLM

8. The LLM generates a response grounded in your actual documents

The result: the LLM answers using your data, not its training data. It can cite sources. It stays current because you can update the document store without retraining anything.

Why RAG wins over fine tuning for most use cases

1. Works with your existing data immediately

2. No model retraining required

3. Update documents without touching the model

4. Cost is a fraction of fine tuning

5. 80% of enterprise use cases are solved here

The failure mode to watch for: bad chunking strategy. If your chunks are too large, the model gets flooded with irrelevant context. Too small, and it loses the thread. Getting the chunk size and overlap right is where most RAG implementations succeed or fail.

RAG is not glamorous. But it solves most of the problems teams try to throw fine tuning at. Start here.

3. How to Become an AI PM

You know the role taxonomy. You understand the technical fundamentals. Now how do you get the job?

Four steps.

Step 1 - Build products, not projects

Do not stop at a demo. Launch. Get real users. When things break, fix them.

“I built this product. It broke in these ways. Here are the challenges I overcame.” That beats any course certificate in an interview.

The difference between a project and a product: a project is something you show in a portfolio. A product is something someone else uses. Hiring managers can tell the difference in 30 seconds.

Step 2 - Three portfolio artifacts

You need three things in your portfolio, each demonstrating a different AI PM competency:

1. An app solving a real problem you have. Not a tutorial clone. Not a toy. Something you actually use. This proves you can identify a problem and ship a solution.

2. An agent that demonstrates goal-oriented reasoning. Show that you understand the difference between workflows and agents. Build something where the AI makes real decisions about which tools to use and when.

3. A RAG system grounding AI in domain knowledge. Pick a domain you know well. Build a system that answers questions using your own documents. This proves you understand the most commercially relevant AI technique.

Each with real users. Not static demos.

Step 3 - Get certified

The AWS AI Practitioner certificate pairs well with hands-on portfolio work. It signals to hiring managers that your knowledge isn’t just theoretical. It covers the fundamentals - ML concepts, model selection, responsible AI - in a structured way that complements your portfolio.

Certifications alone don’t get you hired. Certifications plus shipped products do.

Step 4 - Learn the PM cultures

Each company shapes a different PM.

1. Amazon - document-driven. PRFAQs and six-pagers reviewed up to VP/CEO level. PMs spend 40-50% of their time writing. You become an exceptional writer.

2. Meta - experimentation first. Ship variants. Test against control groups. Let data decide. The most sophisticated experimentation infrastructure in the industry. You live and breathe statistical significance.

3. Netflix - context over control. No rigid process. No documentation requirements. Trust and autonomy. You define your own swim lane.

Amazon teaches thinking backwards from customer pain. Meta teaches rapid testing. Netflix teaches the power of autonomy.

The companies you work at shape who you are as a PM. The insights compound. Get the experience at even one of these environments if you can.

Get Transcript

Where to find Jyothi Nookula

• LinkedIn

• NextGen Product Manager

Related content

Podcasts:

1. AI PM: Complete Course with Pawel Huryn

2. How to Become an AI PM with Marily Nika

3. AI PM Masterclass with Ankit Shukla

Newsletters:

1. The ultimate guide to context engineering

2. RAG vs fine tuning vs prompt engineering

3. AI foundations for PMs

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by:

1. Kameleoon: Leading AI experimentation platform

2. Testkube: Leading test orchestration platform

3. Pendo: The #1 software experience management platform

4. Bolt: Ship AI-powered products 10x faster

5. Product Faculty: Get $550 off their #1 AI PM Certification with my link

Today’s episode

I have done four eval episodes now. We covered error analysis and LLM judges with Hamel and Shreya. Evals from first principles with Ankit. The PM’s role in depth.

But I kept hearing the same question: how do I actually build one from scratch?

That is what today’s episode delivers. We build an eval entirely from scratch. Live. On camera. No pre-written prompts, no pre-written data.

I sat down with Ankur Goyal, Founder and CEO of Braintrust, the eval platform behind Replit, Vercel, Airtable, Ramp, Zapier, and Notion. They just announced their Series B at an $800 million valuation.

Users are running 10x more evals than this time last year. People log more data per day now than they did in the entire first year the product existed. The episode explains why, and then we build one live.

We connect to Linear’s MCP server, generate test data, write a scoring function, and iterate until the score goes from 0 to 0.75. And along the way, Ankur drops what might be the most important reframe of the year: evals are the new PRD.

Plus, we cover the complete eval playbook for PMs:

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

If you want my coaching in your job search, apply to my cohort.

Newsletter deep dive

I took everything from the episode and combined it with lessons from four eval deep dives. Here is the complete practitioner’s playbook:

1. Why vibe checks stop scaling

2. The data-task-scores framework

3. Evals are the new PRD

4. Offline vs online evals

5. How to maintain eval culture

Subscribe now

1. Why vibe checks stop scaling

Open your AI product right now. Try three examples. Tweak the prompt. Ship it.

Two weeks later, support tickets start coming in about edge cases you never tested.

That is the vibe check trap.

When you do a vibe check, you are using your brain as a scoring function. You intuit whether the result is good or bad. That is an eval. It is just the version that does not scale.

Vibe checking is not wrong.

It is the earliest version of what eventually needs to become structured. In the error analysis process I covered in an earlier episode, the first step is reviewing 100 traces and taking notes. That is a structured vibe check.

Here is what matters: the best companies treat the transition from vibe checks to structured evals as a product decision, not an engineering one.

The wall hits when more people start using the product, more subject matter experts start contributing to quality, and more engineers start touching the prompts.

At that point, one person’s gut feeling cannot cover the surface area. You need software and process to execute at scale with predictable performance.

And here is the thing about durability. Models change every few months. Agent frameworks get replaced.

The prompt you wrote today might be irrelevant by summer. But if you invest in understanding what your users actually need and encode that as data, scores, and eval flows, that survives every model swap.

The companies that believe their agent wiring is their moat are highly likely to fail. The ones building true differentiation are the ones investing in evals.

The prompt is temporary. The eval is permanent. That is the whole game.

2. The data-task-scores framework

In the evals deep dive, we covered the full analyze-measure-improve lifecycle. That is the theory. Here is the practitioner’s shorthand: every eval, no matter how complex, is exactly three things.

Component 1 - Data

A set of inputs your AI product needs to handle.

During the episode, we generated test data for a Linear task management bot:

1. “What tasks are assigned to me?”

2. “Are there any overdue tasks?”

3. “How many tasks need to be triaged?”

You can optionally include ground truth answers. If you know the correct answer is 12, include it. If you do not, skip it.

Do not spend a month building a perfect golden data set before running your first eval. Auto-generate silly questions. Jump in. Start iterating.

This echoes what I covered on bootstrapping datasets with synthetic data. But in the live demo, we took it further. We used Opus to generate the data set in seconds, edited it by hand, and ran our first eval within minutes.

Component 2 - Task

A task takes an input and generates an output. It could be:

1. A single LLM call to GPT 5 Nano

2. A complex agent calling MCP servers, vector databases, and multiple models

3. Anything in between

At the end of the day, it produces some kind of output. That is what you evaluate.

During the demo, the first task attempt failed badly. The model just said “Happy to help with Linear” instead of answering. That zero score was expected and useful.

Here is what happened next.

We connected the Linear MCP server so the model actually had access to task data. Still failed. The model said “I am ready to help with Linear tasks” but never called a single tool.

So we changed the system prompt:

Don’t ask clarifying questions, just use the tools and figure it out.

This matters more than you’d expect. Models are post-trained to ask clarifying questions in conversational contexts. In an eval pipeline where you send one question and expect one answer, that post-training behavior works against you. You have to explicitly override it.

Then we loosened the scoring function. It was penalizing responses that referenced Braintrust tasks as sources, even though that was valid citation behavior. The scorer was too harsh for what we actually cared about.

Then we added few-shot examples to the prompt and specified which MCP tools to use.

Score went from 0 to 0.75 across the board. Each iteration touched a different part of the data-task-scores framework. That is the rhythm: identify which component is the bottleneck, fix it, rerun.

Component 3 - Scores

Scores take the input, the expected output if you have one, and the actual output. Their job: produce a number between 0 and 1.

That normalization is critical. It forces everything to be comparable. A week from now, when you run a new eval after swapping the model, you compare directly against today.

Do not overcomplicate scores. Use categorical options, not freeform numbers:

1. A = Full answer with citations

2. B = Partial answer

3. C = No answer at all

In the evals deep dive, I recommended binary Pass/Fail as the default. There is a case that not every score must be binary, but clear criteria beats freeform every time. The key is that you are not asking the LLM to generate an arbitrary number. You are giving it defined options and asking it to classify.

Data in, task runs, score comes out. Everything else is iteration.

3. Evals are the new PRD

This is the reframe that hit me hardest.

Think about a PRD from 2015. Unstructured document. A spec meant to communicate how to build something. The engineering team reads it, half follows it, the final product never matches.

The modern PRD is an eval. It is something that an engineering team who maybe does not know everything about the problem can use to quantify how well the software is solving it.

Step 1 - Encode your product intuition as data

Instead of writing paragraphs describing what the product should do, create a data set of inputs that represent what users actually need.

Step 2 - Use scores as success criteria

Instead of prose acceptance criteria, write scoring functions that quantify whether the software works.

An engineering team that does not know the domain can look at the eval and know exactly what good looks like.

When the eval passes and the product still feels wrong, that is on you. Your scoring function does not capture what matters. And that is actually a new area of leverage that PMs did not have before. You go from providing a qualitative spec nobody follows into something quantifiable.

I have been saying PMs should own error analysis for months now. This episode pushed that further: PMs should own the eval itself. The scoring function IS the modern PRD.

Step 3 - Close the distance gap

The farther you are from the end user, the more critical evals become.

Ankur framed this as a distance principle.

At Anthropic, the people training the models, building the harness, building the product, and using the product all sit inside one set of walls. Feedback circulates with almost no friction. In healthcare, engineers are not patients or doctors.

Ankur’s parents are both doctors, and he said when he talks to them about their work, he has almost no idea what they’re talking about. The jargon is specialized, the stakes are high, and the engineering team has no intuition for what matters.

Evals become the mechanism that bridges that gap.

Finance has the same problem with different jargon. Domain experts speak a language that engineers don’t, and patients become counterparties, but the structural issue is identical. PMs own the scoring function that translates between what users need and what engineering measures.

There was a Twitter controversy about Claude Code not using evals. This blew up enough that someone pinged me about it the day it happened because their boss was questioning whether evals even mattered.

Here is the reality: Claude Code is doing evals. People at Anthropic are using the product, providing feedback, and incorporating that feedback into iterations. That is a form of eval. They just don’t need formal process because the distance between builder and user is zero. That is not the case for 99% of AI products.

One more thing Ankur pointed out: a big use case for Braintrust has been helping companies collect evals they can share with labs so the labs can better support their use case.

When you have distance, you need a ledger to capture the information. Otherwise, how do you communicate it?

The best PRD in 2026 is not a document. It is a data set, a task function, and a scoring function the whole team can run.

4. Offline vs online evals

You have your eval running in a playground. Scores are improving.

That is only half the picture.

Workflow 1 - Offline evals test your hypothesis

This is what we built in the demo. A data set, a task, a scoring function. You run experiments on your laptop. You iterate on the prompt. You swap models. You see the score move from 0 to 0.75.

Fast. Cheap. Repeatable.

This is the AI equivalent of what used to require an A/B test. In the old world, non-deterministic problems meant expensive production experiments. Now you can run those experiments offline. That is why Braintrust customers average 12.8 experiments per day.

Workflow 2 - Online evals test reality

Take the same scoring functions and run them on real production logs. Every time a user asks a question, the scorer evaluates the response.

If your offline eval achieves 0.75 but your online eval consistently shows 0.3, something is not translating.

Workflow 3 - The flywheel

The real power is in the loop:

1. Online evals surface examples not performing well in production

2. Grab those examples, add them to your offline data set

3. Your offline evals now reflect real user behavior, not hypothetical test cases

4. Iterate until scores improve

5. Push changes, repeat

This is the same continuous improvement flywheel I covered in the evals deep dive. Now you can see what it looks like in practice at companies like Ramp, Notion, and Dropbox.

The best teams run this loop daily. More on the specific ritual in the eval culture section below.

Offline evals tell you what should work. Online evals tell you what actually does. The gap between them is your roadmap.

5. How to maintain eval culture

The biggest risk is not that your eval system is bad. It is that your team treats evals as a gate instead of an iterative loop.

Mistake 1 - Running evals only at the end

If you edit your prompt, test it on three examples, then run a full eval just to see if you can ship, you are doing it backwards.

Start from the eval. The best teams Ankur works with have a morning ritual. In standup, they pull up examples from the previous day’s production logs. They reconcile what they see with what their eval scores show. They find novel patterns that have emerged, add them to the data set, and grind on those failures all day. That is the job now.

Mistake 2 - Only having evals that pass

If every eval succeeds, you have blind spots. Either:

1. You do not understand what your users are hitting

2. You do not understand what is impossible today

Have failing evals. When a new model drops, rerun them first. Something interesting always happens.

And interesting does not always mean better. Gemini 3 Flash was outperforming Gemini 3 Pro on coding benchmarks but hallucinating more. Those nuances only surface with a full testing suite.

This is also how you plan product launches around model releases.

Braintrust shipped Loop this way. They built the eval for the feature before they shipped it. The eval failed on every model they tried. Then Claude 3.7 came out and there was a huge jump. The eval passed. They shipped.

Ankur said there was a watershed moment where 3.7 was the first model that could look at its own work and improve. Prior to that, he said models were like a dog looking at itself in the mirror. They didn’t recognize they were evaluating a virtual representation of themselves.

Create the eval for the feature you want. Watch the models. The moment a model hits the quality threshold, you ship.

Mistake 3 - Siloing evals to AI engineers

Evals should not be constrained to engineers. Product managers need access. Domain experts need access. The people who understand the end user should be improving the scoring functions.

Zapier was Braintrust’s first customer. Ankur told a story about their CTO Brian, who has been there for a long time and is very successful.

When Ankur met him, Brian introduced himself as a full-time AI engineer. This guy probably doesn’t have to work, but Ankur said he has never seen anyone nerd out about AI as much as Brian does. Companies like Zapier have pre-existing product-market fit.

If Ramp doesn’t work, it is very bad. They don’t have the leeway to screw things up. The quality bar is high, the scale is real, and vibe checks can’t cover the surface area. That is why these companies gravitate toward evals.

Braintrust removed user-based pricing specifically for this reason. They realized evals should not be locked behind engineering seat licenses.

Going back to the PRD analogy: if only engineers can see and edit the PRD, the product reflects what engineers think matters. Not what users need.

The teams that win are not the ones with the most sophisticated eval setup. They are the ones where evals are the first thing the PM opens every morning.

The whole game

Treat the eval as a first-class product artifact with a PM who owns the experience. The scoring function is the new PRD. The data set is the new user research. The flywheel is the new sprint cycle.

Open your platform of choice, write your first scorer, and run it on 5 real examples today. You will learn more in that first hour than in a month of vibe checking.

The prompt is temporary. The eval is permanent. That is the whole game.

Get transcript

Where to find Ankur Goyal

• X

• LinkedIn

• Braintrust

Related content

Newsletters:

1. AI Evals deep dive

2. AI observability for PMs

3. How to build AI products

Podcasts:

1. AI evals with Hamel Husain and Shreya Shankar

2. AI evals part 2 with Hamel and Shreya

3. AI Evals explained simply with Ankit

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Share

Check out the conversation on Apple, Spotify and YouTube.

Brought to you by:

1. Jira Product Discovery: Plan with purpose, ship with confidence

2. Vanta: Automate compliance, manage risk, and prove trust

3. Mobbin: Discover real-world design inspiration

4. The AI Evals Course for PMs & Engineers: You get $1250 with my link

5. Product Faculty: Get $550 off their #1 AI PM Certification with my link

Today’s Episode

OpenClaw is the hottest new AI tool that no one is helping you with. There’s zero real content on OpenClaw for PMs.

I covered what OpenClaw is and why it matters when it first went viral. But the rest of the web has been oddly quiet.

That is a big mistake. OpenClaw as software is as important as ChatGPT in 2023 or Claude Code in 2025.

It’s the software soon every PM will be using.

What I’ve found is that setup is the hardest part. So in today’s episode, I’m giving you a complete, step-by-step installation.

Plus, we cover five of the top PM use cases:

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

If you want my PM Operating System in Claude Code, click here.

Newsletter Deep Dive

As a thank you for having me in your inbox, here is the complete guide to setting up OpenClaw and building PM automations with it.

1. Why PMs should care about OpenClaw

2. How to Set Up OpenClaw

   • How to connect OpenClaw to Slack

   • The workspace file system

3. 5 of the Top PM Use Cases

   • Slack knowledge base

   • Automated stand-up summaries

   • Competitive intelligence on autopilot

   • Voice of customer reports

   • Smart bug routing by customer tier

4. Security and deployment

Subscribe now

1. Why PMs should care about OpenClaw

Go open Claude or ChatGPT right now. Ask it to check your Slack for blockers, scan your competitor’s pricing page, and post a summary to your team channel at 9 a.m. tomorrow.

It can’t do any of that.

Every LLM you use today is reactive. You ask, it answers. You close the tab, it dies. No access to your files, your tools, or your team’s communication.

OpenClaw is different in three ways.

It is proactive, not reactive. Set up a cron job and walk away. It executes at 3 a.m. when you are asleep. Scans channels, monitors websites, generates reports, posts them to Slack. No fingers lifted.

It is model agnostic. Not locked into one provider. Deep research? Plug in Claude Opus. Fast customer responses? Gemini Flash. Budget? Qwen 3.5 at 1/10th the cost. You control the model per use case.

It runs locally. Your data stays on your machine. No cloud lock-in. It reads and writes local files, which means you build a living repository of product docs, customer feedback, and competitive intelligence that grows over time.

Peter Steinberger described the difference between skills and tools perfectly. Tools are organs. Can the agent do it? Skills are textbooks. Does the agent know how to do it? OpenClaw has both.

The difference between OpenClaw and every other AI tool you use is simple. It acts. Everything else just answers.

Continue Reading Online

The three-command installation of OpenClaw

Most people who quit OpenClaw quit during setup. Not because it is hard. Because terminal commands feel unfamiliar.

The truth is, if you are not seeing red text, you are good. Yellow warnings are normal.

Two paths.

The one-click path (training wheels). Go to emergent.sh. They have a bot feature that pastes the install command for you. Replace your LLM key, hit enter, done. The limitation: no full control. You cannot use your own RAM, run a local model, or customize the workspace.

The terminal path (what most PMs should do). Three commands.

Step 1 - Install OpenClaw

Open your terminal. Paste:

npm install -g openclaw@latest

Finds the latest version. Installs it. If it fails, you probably do not have Node.js 22+ installed. Go to nodejs.org, install it, re-run.

Step 2 - Run the onboarding wizard

openclaw onboard

The wizard walks you through:

1. Accept the security warning (hit yes)

2. Choose Quick Start over Manual

3. Pick your LLM provider

4. Paste your API key

5. Choose your messaging channel

For the LLM provider, you have every option. Google, Anthropic, OpenAI, and more. If you want the best dollar-per-quality ratio, Gemini is strong. To create a key, Google “Gemini key create,” click Create, name it OpenClaw, copy, paste it into a note you won't lose.

After the provider, choose the specific model. Gemini 3 Pro for deep work. Any Flash model for speed.

Then choose a messaging channel. For PMs, pick Slack socket mode.

Step 3 - Hatch the bot

The wizard installs the gateway and hatches your bot. You see “Wake up, my friend.” You are live.

During onboarding, OpenClaw generates a soul.md file. This forces you to give your bot a name, a personality, interaction rules. Peter Steinberger designed it this way. He wanted a companion, not a disposable chatbot.

Skip skills and tools selection during setup. Easier to add later by editing the markdown files directly, or by asking the bot.

Three commands. 15 minutes. If you can install a Chrome extension, you can install OpenClaw.

How to connect OpenClaw to Slack

This is the highest-leverage setup. It puts your AI agent inside the tool you already use all day.

The terminal instructions during onboarding are bare bones. Here is the full walkthrough.

Step 1 - Create a Slack app

1. Go to api.slack.com/apps

2. Click Create an App

3. Choose From Scratch

4. Name it OpenClaw

5. Select your workspace

6. Click Create App

Step 2 - Enable Socket Mode

1. Left sidebar: Socket Mode

2. Toggle on

3. Token name: secret-token

4. Hit Generate

5. Copy and save the token

Step 3 - Add bot token scopes

Go to OAuth & Permissions. Scroll to Bot Token Scopes. Add:

1. chat:write

2. channels:history

3. channels:read

4. groups:history

5. groups:read

6. im:history

7. im:read

8. users:read

Each scope controls what your bot can see and do. Skip any your org restricts, but all matter for the use cases below.

Step 4 - Install to your workspace

Scroll to the top. Click Install to Workspace. Hit Allow. Copy the Bot User OAuth Token (starts with xoxb).

Step 5 - Paste tokens into OpenClaw

Back in your terminal, the wizard wants two things:

1. Bot Token (the xoxb one)

2. App Token (the secret-token from Socket Mode)

Paste both. Configure channel access. Allow all channels unless you have restrictions.

Once tokens are in, go to any Slack channel, invite your OpenClaw bot, mention it with @OpenClaw. It responds. You are running an AI agent inside Slack.

The workspace file system

Before building automations, understand where OpenClaw lives on your machine.

Everything is in a hidden folder at ~/.openclaw. On Mac, click Finder, right-click, Go to Folder, type a period, select .openclaw.

Inside the workspace:

1. soul.md - Bot’s personality and values

2. agents.md - Agent identity, safety rules, operational instructions

3. user.md - Your preferences (direct, no fluff, whatever you set during onboarding)

4. memory.md - Persistent memory that survives restarts

5. tools.md - Local configuration notes

6. heartbeat.md - Your cron jobs (starts empty)

All markdown files. Open in any text editor. Ask an LLM to generate content for them and paste in. Or ask OpenClaw itself to modify them.

There is also a gateway dashboard at 127.0.0.1:18789. Chat with your bot, view cron jobs, change API providers, check gateway status. Whatever you do in the dashboard reflects in the terminal and vice versa. Same brain.

If you want to swap from Gemini to Anthropic, go to Config > Secrets in the dashboard, change the model, save. Or just tell the bot in chat and it reprograms itself.

Terminal is your command center. Gateway dashboard is your control room. Both talk to the same agent.

3. 5 of the Top Use Cases for PMs

Setup done. Now let's put it to work.

The five use cases below are the ones Naman and I spent weeks building and testing on camera. Each one replaces a manual workflow you’re probably doing right now:

1. A Slack knowledge base that answers your team’s product questions from your own docs

2. Automated stand-up summaries posted before your first meeting

3. A competitive intelligence pipeline that runs while you sleep

4. Voice of customer reports pulled from every source into one place

5. Smart bug routing that triages by customer tier without you touching it

Plus the full security and deployment guide so you don’t end up WhatsApp-ing your mom a pairing code.

(paid subscribers only)

Read more

Check out the conversation on Apple, Spotify and YouTube.

Brought to you by

1. Pendo: The #1 software experience management platform

2. Jira Product Discovery: Plan with purpose, ship with confidence

3. Amplitude: The market-leader in product analytics

4. NayaOne: Airgapped cloud-agnostic sandbox

5. Product Faculty: Get $550 off their #1 AI PM Certification with my link

Today’s Episode

This episode is a real life example of how a CPO actually uses Claude Code to do everything:

• His morning daily plan

• His career planning

• His meeting prep

All with real data plugged into his real systems: CRM, calendar, etc..

He thinks it’s better than the human EA (executive assistant) he used to have. And in today’s episode, he demonstrates everything.

Dave Killeen has a 25 year career in PM, so when he embraces a new AI tool this thoroughly, it’s worth checking out:

Apple Podcast

Spotify

If your product team doesn’t have access yet, send this to your manager or skip-level.

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

If you want my PM Operating System in Claude Code, click here.

Newsletter Deep Dive

As a thank you for having me in your inbox, here is the complete guide to running your entire work life in Claude Code:

1. The daily plan command

2. How to connect everything with MCP servers

3. Skills vs MCP vs hooks

4. The compounding file system

5. From backlog to PRD to Kanban board

6. Career planning inside your OS

7. How to get started today

Subscribe now

1. The Daily Plan Command

Here is the core problem with how most PMs start their day.

You open six or seven tabs. Calendar in one. CRM in another. Meeting notes buried somewhere. LinkedIn messages you meant to respond to three days ago. By the time you've assembled the picture, you've burned 30 minutes and lost your flow state.

The fix: one slash command. Five minutes later, you have a full daily plan.

Here is what the command does under the hood:

Step 1 - Check what intel has already run

Before assembling the plan, the system checks whether all of its intelligence digests have already run for the day. Has it pulled LinkedIn data? Twitter data? Newsletter summaries? If any are missing, it executes those first.

Step 2 - Pull structured data through MCP

It pulls your calendar, weekly priorities, quarterly goals, tasks, and CRM data through MCP servers. Your specific connections will vary. Dave connects to Clari (Pendo’s sales tool), Granola (meeting notes), LinkedIn via Phantom Buster, YouTube transcripts, and 120 newsletters. You start with whatever tools you already use. Calendar is the easiest first connection.

Step 3 - Assemble and surface what matters

Everything gets assembled into a single markdown page. Three priorities for the day. Which accounts need attention, because it has been listening to customer conversations through Granola. Which LinkedIn messages are connected to existing enterprise accounts. It even writes the Slack messages to send to his team.

One command. No tab switching. No manual assembly. The AI does it all for you.

2. How to connect everything with MCP, APIs, or CLIs

The daily plan works because every tool is connected.

Most PMs hear “MCP” and “API” and assume it requires engineering work. It does not! The process is simpler than most people expect.

Step 1 - Find the documentation

Go to whatever tool you want to connect. Your calendar is the easiest starting point. Find the API, MCP, or CLI docs.

Step 2 - Tell Claude to build the server

Tell Claude through voice:

Here is the API/CLI/MCP documentation for this tool. <I have an API key for you.> Connect with it.

Claude reads the documentation, builds the server, and you are connected.

Step 3 - Point the AI at new use cases

Once the server is built, you can ask Claude:

Knowing what you know of the API documentation, how can we make this even better? What could we do?

It comes back with use cases you had not thought of, because it has ingested all the API documentation into your system.

Dave’s rule is simple. If he has access to an API, he connects it. Everything and anything gets connected. The more data flows in, the smarter every command becomes.

Why MCP instead of raw API calls? Because MCP acts as guardrails. It defines exactly how the AI should interact with each service, what data to pull, and in what format. That makes the results far more deterministic than pointing Claude at a raw API and hoping for the best.

3. Skills vs MCP vs hooks

This is where most people building Claude Code workflows get confused. There are three building blocks, and each one does something fundamentally different.

Building block 1 - Skills (now synonymous with commands)

A skill is a plain English instruction file that tells the AI what to do when you invoke it. Think of it as a job description. “When I say /daily-plan, pull my calendar, check my goals, scan my intel digests, and assemble a plan in this format.”

Skills are flexible and easy to create. Describe what you want in a conversation and Claude writes the skill file for you. Dave has about 60 in his system. You’ll probably start with five or six and grow from there.

If you want something to behave loosely and benefit from AI judgment, make it a skill.

Building block 2 - MCP servers

MCP servers are structured integrations that tell the AI how to interact with external services. Tighter than skills. They enforce guardrails around what data gets pulled, what format it comes in, and what steps the AI follows.

Example: a task MCP server ensures tasks get created in a consistent way every time, always attached to the right project and the right strategic pillar. Without the guardrails, the AI might skip the project link or categorize tasks differently each run.

If you want something to behave the same way every single time, make it an MCP server.

Building block 3 - Hooks

Hooks are triggers that fire at specific moments in your conversation with Claude. The most powerful one is the session start hook. Every time Dave opens a new Claude Code chat, the hook automatically injects his weekly priorities, quarterly goals, working preferences, and past mistakes into the context. The AI never starts from scratch.

Hooks are only available in Claude Code terminal and Claude Code desktop. Not in Cursor. This is the single biggest reason to graduate from Cursor to terminal.

Skills are what you do. MCP is how you connect. Hooks are how you compound.

4. The Compounding File System

Here is the fundamental difference between this approach and just chatting with ChatGPT.

In a normal AI conversation, everything lives in that one chat window. Maybe the AI encodes something into its memory. But there is no structured, persistent knowledge base that grows over time.

In a Claude Code operating system, everything is a markdown file. And those files are alive.

How it compounds

When a new Granola meeting transcript comes in with actions from it, or there is a new angle from a stakeholder on a call, that information gets appended to the stakeholder’s person page, to the project’s page, and to the company page if there is a company connected to it. Every time the AI later pulls on that entity, it has all that fresh context. You can ask, “What do you know of this particular project?” The AI knows where to look because the file already holds months of accumulated context.

The mistakes file and working preferences

Dave has a mistakes file that the AI writes to whenever it makes an error. That file gets injected into every new session through the session start hook, so the same mistake never happens twice. He also has a working preferences file that captures how he likes to work. Every time he says, “Hey Claude, why did you do that?” the AI picks it up and logs it.

The Claude MD file

Your Claude MD file acts as a map. (Grab my Claude.MD file here.) Keep it short. Anthropic calls the technique “progressive disclosure” - the Claude MD springboards the AI into other files where deeper context lives, rather than stuffing everything into one giant file.

One trick worth stealing from Dave’s setup: he has a section called “harsh truths for Dave” that the AI wrote after he asked it to audit his system for bloat. That section gets injected into every session.

The more you dance with the AI, the more the files get smarter. That is the whole game.

5. From backlog to PRD to Kanban board

Most PMs stop at one part of the cycle. Dave has the full loop running inside his system.

Step 1 - Collect and rank ideas

Maintain a backlog inside your system. Some ideas come from you. Some come from the AI scanning GitHub repos, Hacker News, and Reddit communities for relevant projects.

Step 2 - Generate a PRD from any idea

Pick an idea from the backlog. Tell Claude to write the PRD. The AI reads the full system context, checks for overlap with existing capabilities, recognizes dependencies, and produces a thorough first draft. Dave’s honest assessment as a CPO is that these are strong first drafts. He would want commercial context and tighter metrics before shipping at work. But for personal and open source projects, he has largely stopped editing them. He calls it “vibe CPOing.”

Step 3 - Manage everything on a Kanban board

When PRDs started piling up, Dave described the pain point to Claude. “I have too many PRDs in flight, I need a way to see what is shipped and what is next.” Claude built the entire web UI in three hours. Each card contains a PRD. Each card has a play button. The AI ranks the cards and tells him what to work on next.

That is malleable software. You have a pain point. You describe it. The AI builds the tool. If it works, keep iterating. If it does not, kill it.

6. Career planning inside your OS

Most PMs manage their product backlog with obsessive rigor and manage their own career with almost none.

Dave built a career MCP server that changes that. It does three things automatically.

Function 1 - Scan for evidence

As Dave works with the AI every week, it listens for evidence of skills demonstrated, feedback received, and outcomes delivered. Granola transcripts, conversations, project outcomes. Everything gets collected automatically.

Function 2 - Run skills gap analysis

Based on his career goals, the system identifies where he is strong and where the gaps are. It maps those gaps against his quarterly goals and weekly priorities. When he runs his weekly plan, it says, “Your thought leadership is strong, but you have not invested in expanding strategic influence across the wider business this quarter. Here are the gaps. Here is what your weekly plan should include.”

Function 3 - Calculate promotion readiness

The system calculates a score based on accumulated evidence. When review time comes, the evidence is already assembled. Not scrambled together the night before.

The career system uses the same compounding principle as everything else. Evidence accumulates over time. Gaps get narrower. And your career roadmap stays connected to your daily work instead of living in a document you never open.

7. How to get started today

Dave’s system is open source. It is called DEX, and it is on GitHub. Or you can use mine (free for founding members).

Step 1 - Clone the repo and run setup

You clone the repo, follow the step-by-step guide, and run /setup. It asks for your name, your role, your company size, and your goals. Then it scaffolds the entire system around your situation. Within five minutes, you are up and running.

Step 2 - Start in Cursor, graduate to terminal

Start in Cursor if the terminal feels intimidating. Do not worry about the messy file tree on the left. Trust the AI to organize things. Then once you are comfortable, move to Claude Code in the terminal. That is where hooks live. That is where compounding lives. On Mac, Ghostty is a cleaner terminal experience than the default.

Step 3 - Use your voice

Do not type. Use Wispr Flow or Speechify. Talking to Claude instead of typing completely changes the dynamic. And be very clear about your goal. Do not tell the AI how to get somewhere. Tell it where you want to end up. Give it a precise, sharp goal, and let it figure out the most elegant path on its own.

There has never been a better time to be a product geek. The PMs who build their own operating systems will compound their advantage every single day. The ones still switching between 15 tabs every morning will keep losing 30 minutes before they even start.

Get Transcript

[Bonus] Takeaway Visual

Where to find Dave Killeen

• LinkedIn

• Pendo

• Dex

Related Content

Newsletters:

1. Claude Cowork and Code setup guide

2. How to use Claude Code like a pro

3. My Claude Code PM OS

Podcasts:

1. Carl Vellotti on Claude Code

2. Rachel Wolan on Claude Code for CPOs

3. Frank Lee on Claude Code and MCP workflows

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Share

Check out the conversation on Apple, Spotify and YouTube.

Brought to you by - Reforge:

Get 1 month free of Reforge Build (the AI prototyping tool built for PMs) with code BUILD

Today’s Episode

Most PMs are using AI the same way they used Google in 2005.

Type something in. Get something out. Move on.

That is not how the best PMs are using it. The best PMs have stopped treating AI as a search engine and started treating it as a team member. One that already knows their product, their writing style, their strategy. One that does not need to be briefed from scratch every single time.

That shift is what today’s episode is about.

I sat down with Lisa Huang, SVP of Product at Xero, the $18 billion finance platform. She built the AI assistant for the first generation Meta RayBan smart glasses. She created Gemini Gems at Google. She has been an AI PM at Apple, Meta, and Google - three of the most demanding AI product environments in the world.

And she gave us a masterclass on three key topics:

1. How to build Gemini Gems and AI projects that actually work

2. What she learned building AI into a wearable device

3. What the future of the AI PM career looks like

Listen or watch now:

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

Newsletter Deep Dive

As a thank you for having me in your inbox, here is the complete guide to Gemini Gems, building AI features, and navigating your AI PM career.

1. Gemini Gems - the feature most PMs are ignoring

2. The three must-have Gems for every PM

3. How to build one that actually works

4. What building AI into hardware teaches you

5. Key AI PM Topics in 2026:

   • How to build good AI agents

   • How to measure AI agents

   • The future of the PM role

   • How to break into AI PM

Subscribe now

1. Gemini Gems - the feature most PMs are ignoring

Here is the core problem with how most people use LLMs.

Every time you open a new chat, you start from scratch. Your role. Your company strategy. Your writing style. Your product history. All gone. You are essentially hiring a brilliant contractor and handing them zero blueprints every single time.

Gemini Gems close that gap.

Standard Gemini is your general contractor. Capable. Powerful. But needs a full briefing every single time. A Gem is your master craftsman. It already knows your house. It knows your preferences. You ask once and it delivers exactly what you need.

The same principle applies to Claude Projects and ChatGPT custom GPTs. The name does not matter. The principle does. Stop having generic conversations. Start building personalized AI that holds your context permanently.

2. The three must-have Gems for every PM

Gem 1 - The writing clone

PMs communicate all day. To engineers. To stakeholders. To executives. To customers.

Upload your PRDs, past emails, and team Slack messages. Build a version of your LLM that sounds like you. Use it to draft the first version of everything. You will not lose your voice. You will just get it back faster.

Gem 2 - The product strategy advisor

Feed it your company strategy docs, market positioning, and competitor analysis. Use it as a thought partner when you are working through hard decisions.

It will not replace your judgment. But it will surface angles you had not thought of. That is the product strategy edge most PMs are leaving on the table.

Gem 3 - The user research synthesizer

You cannot be in every interview. You cannot read every support ticket.

Upload the raw transcripts, survey data, and customer support tickets. Ask for synthesis. Ask for key themes. This is the kind of AI-powered customer intelligence work that separates good PMs from great ones.

Your email may get cut off around this point. Continue reading online or in the app:

Continue Reading Online

3. How to Build a Gem That Actually Works

Go to Gemini. Click Gems. Create a new one.

Three things matter.

Step 1 - Write detailed instructions

Not “help me write better.” That gets you nothing.

Write a full page of context. Your role. Your audience. Your format preferences. The more specific you are, the more personalized the output. Think of it the same way you would brief a new hire. Give them everything they need to do the job without asking you twice. (Use AI to help.)

Step 2 - Upload your key context documents

What makes a Gem personalized is not the instructions alone. It is the knowledge. Upload your PRDs, past emails, competitor teardowns, roadmaps. The Gem reads all of it before responding.

One distinction worth knowing. Unlike Claude Projects where you can train the project over time through conversation, Gemini Gems work strictly off what is in the instructions and knowledge files.

So as your context changes, update those files. You can even ask the Gem how it would update its own system prompt based on your conversation. Then paste that update in.

Step 3 - Iterate

Your first version will not be perfect. That is fine. Treat it like a mini AI product you are shipping for yourself. Iterate on the instructions. Iterate on the knowledge. The Gem gets better the more you invest in it.

The biggest mistake is vague instructions. Fix that first and everything else improves.

4. What building AI into hardware teaches you

The first generation AI assistant for Meta RayBan smart glasses launched years before the product hit 4 million sales. That zero-to-one process surfaced constraints most PMs never face.

When you put AI into a wearable, the rules change completely.

Weight. Battery life. Privacy. Bystander concerns. The fact that a fashion company like Luxottica does not move like a Silicon Valley engineering team. All of these compress your design space in ways that a purely software product never would.

And then there is the processing question - cloud or on-device?

Cloud is the default today. But on-device is the future. Once you are wearing a device on your face all day, people are going to want their data staying local. As models get smaller and more efficient, the technical barriers to on-device are dropping fast. Privacy wins over performance every time when the device is that personal.

The lesson for any PM building AI features in any context: do not fall in love with the technology. Understand it deeply. The best AI products live at the exact intersection of what the user genuinely needs and what the technology can reliably do today.

Build fast. See what users do. Update your assumptions. Repeat.

5. Key AI PM Topics in 2026

Since Lisa is such a tenured AI PM leader, I had to ask her about all the hot AI PM topics right now.

5a. How to build an accurate AI agent

Here is what makes building AI agents in any high-stakes domain genuinely hard.

Accuracy is not a nice-to-have. It is the product. In finance, the decimal matters. In legal, the clause matters. In healthcare, the dosage matters. And LLMs out of the box are not naturally great at any of these.

Two things close that gap:

Advantage 1 - Domain knowledge

The companies winning at agents are not just wiring up a generic LLM. They understand every workflow. Every stakeholder. Every acceptable accuracy level at each step.

You have to craft the agent experience around your specific constraints. What are the tasks? What are the subtasks? Where does a 90% answer suffice and where does it fail the user entirely? Map that before you build.

Advantage 2 - Proprietary data

Generic agents give generic answers. The agents that win are the ones with data nobody else has.

Transaction-level data. Interaction history. Domain-specific corpora. That data lets you personalize to each user’s actual situation. It also lets you surface benchmarks and insights that a general-purpose model simply cannot.

On architecture: use a hybrid system. LLMs in multi-agent workflows where it makes sense, but programmatic code where you need tighter control over reliability. Not everything should be non-deterministic. Know when each applies.

5b. How to measure an AI agent

This is the question I get most from AI PMs right now. How do you measure something that is non-deterministic and evolving fast?

Three layers. They build on each other. Do not skip ahead.

Layer 1 - Quality

Is the AI doing what it is supposed to do?

This means evals. Human annotators. LLM judges. You need all three because none of them alone scales. Human annotators give you ground truth. LLM judges give you scale. Evals give you a consistent framework across use cases.

Track quality regularly across all use cases. Understand the gaps. Know which investments close which gaps. This is your foundation. You cannot build reliably on top of it until it is solid.

Layer 2 - Product metrics

Once quality is solid, standard product metrics apply. Adoption. Usage. Retention. CSAT. MAU or WAU depending on the use case.

Also track qualitative signals. Social media. Customer conversations. Support tickets. Users will tell you things in those channels that no dashboard will surface.

Layer 3 - Business impact

Revenue attribution. Retention influence. ARR contribution.

Every company measures this differently. What matters is that you have a system for it and that you track it consistently. AI investment needs to show up somewhere on the business scorecard.

The order matters. Quality first. Product metrics second. Business impact third. Skip to layer three without the foundation and you are measuring on sand.

5c. The future of the PM role

A lot of PMs I talk to are anxious about this. I hear it constantly.

The layoffs feel disproportionate. Junior PM roles are harder to get. The director and group PM layers are compressing. The question I keep getting is: is this career still worth pursuing?

Here’s what I’d say:

AI will not replace PMs. What it will replace is the execution work that fills most of a PM’s day today. Writing PRDs. Creating mocks. Managing roadmaps. Pulling data. All of it will be accelerated or automated.

But product judgment? The ability to look at ambiguous signals and say this is the right bet and here is why? That is not going anywhere. That is what PMs get paid for. Not the deliverables. The taste.

What is changing is the structure. PM-to-engineer ratios will compress. The pure PM role is evolving into a hybrid. The expectation is becoming that PMs also build. Not just spec and hand off, but prototype, design, and code enough to show what they mean. As Lisa said:

Now is a time of transformation and everyone has the availability to go do that.

5d. How to break into AI PM

The excuse I hear most is “I don’t work on AI features at my company.”

It is not an acceptable excuse.

You do not need your company’s permission to build AI products. You do not need a budget. You do not need a team. Claude, OpenAI, Gemini, all of them give you access to the same models that companies are building on. Most companies are not fine-tuning anything. They are using the exact same consumer tools you already have access to.

So build. Build Gems. Build projects. Build small AI products that solve problems you actually have. Use your personal data so there are no company restrictions.

How to stand out in an interview

One candidate who got hired for a senior AI role had zero AI experience going in. Lisa was not sure about them. Then in the first interview they said:

“I saw you are building financial tools for small businesses. I went and watched three hours of TikTok videos from coaches who work with small businesses. Here is what they said about what those businesses actually need financially.”

Nobody else had done that. Not one candidate. You can guess who Lisa hired.

Do the work before you are asked to. That is the whole strategy.

The roadmap to break in

Get direct AI experience in your current role if you can. If not, build on the side. Invest in your network because referrals still matter more than most people admit. Show up well in every job because today’s colleagues become tomorrow’s connections.

When you are ready to interview, treat interview prep as its own skill. Practice out loud. Get mock interviews with people who have been inside those companies. Drill until the structure is second nature.

The FAANG PM interview is a specific game. Product sense. Product execution. Behavioral. Case questions. You can be a great PM and still lose if you cannot perform in that format in that time window. Practice the format, not just the content.

Get Transcript

Where to Find Lisa Huang

• LinkedIn

• Xero

Related Content

Newsletters

1. AI product strategy

2. How to become an AI PM

3. Practical AI agents for PMs

Podcasts

1. Claude Code + Analytics - Vibe PMing with Frank Lee

2. AI evals explained simply with Ankit Shukla

3. How to become an AI PM with Marily Nika

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Share

Check out the conversation on Apple, Spotify and YouTube.

Brought to you by:

1. Pendo: The #1 software experience management platform

2. Testkube: Leading test orchestration platform

3. Gamma: Turn customer feedback into product decisions with AI

4. Product Faculty: Get $550 off the AI PM Certification with code AAKASH550C7

5. Mobbin: Discover real-world design inspiration

Today’s Episode

Every forward-thinking product team is using AI prototyping tools.

But most are using these tools wrong.

They open Bolt or Replit, type a vague prompt, get something that looks decent, show it around, and move on:

• No problem space work

• No divergent solutions

• No user testing

That’s why I’ve created today’s episode. I got Nadav Abrahami (Co-founder of $5.5B giant Wix, CEO of Dazl) to walk you through the complete workflow to use these tools to their maximum capability.

I’ve done three episodes on AI prototyping. This one builds on that compendium of knowledge to cover: when to prototype, how to prompt, when to go high fidelity, and how to hand off to engineers with zero open questions.

Listen or watch now:

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

Newsletter Deep Dive

As a thank you for having me in your inbox, here’s a complete guide to AI prototyping for PMs:

1. When to use AI prototyping

2. The ideal workflow: step-by-step

3. How to prompt without breaking things

4. When to go high fidelity vs low fidelity

5. PRD + prototype - the new standard

6. How to hand off to engineers

Subscribe now

1. When to use AI prototyping (and when not to)

Let me start with what I see happen constantly.

A PM finally gets access to an AI prototyping tool. They start using it for everything. They produce a lot of prototypes. Zero of them go anywhere.

The tool isn’t the problem. The sequencing is.

AI prototyping tools don’t replace problem space work. They accelerate solution space work. Those are two completely different things.

Before you open any prototyping tool, you need three things locked down:

1. The problem you’re solving

2. The user story you’re designing for

3. The rough shape of the solution

If you can’t write all three in a single paragraph, you’re not ready to prototype. Go back to discovery.

Once you have those three, there are two moments in the development lifecycle where AI prototyping really shines.

Moment 1 - Early ideation

This is where most teams barely prototype at all. A rare ~5% of exceptional PMs do it at the idea stage.

That’s a huge missed opportunity. Running 3-4 variations at ideation takes the same time it used to take to run one. You’ll learn more from 30 minutes of functional prototyping than from two hours of writing user stories.

Moment 2 - Post-Figma validation

After the design team has narrowed on a direction, you want to get as close to the real experience as possible before handing off to engineering.

Static Figma screens can’t tell you whether the interaction actually works. A functional prototype can. It’s also ~75% of forward-thinking teams’ most common use case for these tools right now.

Do the research first. Then prototype.

2. The workflow step by step

Here’s the exact flow we used to build a LinkedIn sentiment analysis feature from scratch. Use it as your blueprint:

Step 1 - Start from your design system

Don’t start from a blank page.

Drop a screenshot of your existing product into the tool. Ask it to recreate that page first.

Now every prototype you build from this point will look like it belongs in your product - not like a generic template from the internet.

Save this as a team template. Everyone builds from the same base. Visual fidelity stays consistent. Time to first prototype drops dramatically.

Step 2 - Explore 3 to 4 divergent solutions

This is the step almost every PM skips.

They build one version, it looks decent, they move forward. But the entire point of AI prototyping is that building a second and third version costs almost nothing now. All three variations take the same time it used to take to build one.

We built two versions of the sentiment analysis feature in the episode. One with sentiment inline on each post card. One as a separate summary section with a cohort graph below. Neither was perfect. Both were useful. That comparison is the point.

Don’t move forward until you have at least three versions to evaluate.

Step 3 - Visually edit the strongest direction

Once you’ve picked the best direction, stop prompting for every small change.

Switch to direct visual editing for fine-tuning. Move elements. Match colours with the eyedropper. Adjust spacing. This is faster than prompting because the result is immediate - you see it happen in real time, no waiting for the AI to reinterpret what you meant and get it 90% right.

Prompt for big structural changes. Visually edit for everything fine.

Step 4 - Build out the full end-to-end flow

Single-page prototypes miss too much.

The real value of prototyping is surfacing edge cases before they hit engineering. The moment you start connecting pages, edge cases appear automatically.

When we added a clickable drill-down page showing per-post sentiment detail in the episode, it immediately surfaced two edge cases - what happens with fewer than 4 posts, and what happens with too many. Both would have cost engineering time to discover in sprint. We found them in minutes.

Build the whole flow. Not just the happy path.

Step 5 - Test with your own users

Ideally not a user testing platform. Your actual users.

The ones who requested the feature. The ones who know your product well enough to have strong opinions. Get them on a video call and put the prototype in front of them.

Users who asked for a feature already know what they wanted. Watching them interact with your version tells you in 20 minutes whether you solved their actual problem - or just your interpretation of it.

3. How to prompt without breaking things

Prompting for prototyping is different from prompting for writing or analysis. The rules are different.

You don’t need a system prompt. You don’t need few-shot examples or framework instructions. What you need is clarity.

Any ambiguity in your prompt will get exploited. Not every time - but statistically, any instruction that can be read two ways will eventually be read the wrong way. When that happens in a prototyping tool, the AI runs with its wrong interpretation at full speed, confidently, until it’s done. It doesn’t stop to ask.

The fix isn’t more prompt engineering. It’s clearer language before you send the prompt. Follow these three steps:

Step 1 - Use an LLM to audit your prompt first

Before running any complex prompt, paste it into a separate chat and ask it to find the contradictions. Ask what’s ambiguous. Ask what could be understood two different ways. Fix those. Then run the real prompt.

This sounds like extra work. It saves you 30-minute regeneration loops.

Step 2 - Use discuss/plan mode for anything major

Tell the AI what you’re planning and ask how it understands your intent. Don’t ask if it can do it - that always gets a yes. Ask what it thinks the right approach is.

The difference in response quality between “can you do this?” and “what do you think of this approach?” is significant.

Step 3 - Break big prompts into smaller steps

A huge prompt covering an entire feature produces average results across every part. A focused prompt on one piece at a time produces better results on each piece.

Build in stages. You’ll get there faster.

4. When to go high fidelity

Low fidelity is for exploration. High fidelity is for two specific situations.

Situation 1 - Selling the feature internally

If your feature still needs buy-in from leadership or stakeholders, a polished prototype does more work than any deck or written spec.

People engage differently with something they can touch. If it looks rough, they focus on the roughness instead of the idea. A lot of features die in alignment meetings not because the idea was bad - but because it looked unfinished.

Make it look real when the goal is alignment.

Situation 2 - Usability testing with real users

Low fidelity and real usability testing don’t work together.

Put something rough in front of a user and ask them to evaluate the experience - you’ll get feedback about the roughness, not the experience. You need it to feel real.

The right workflow is low fidelity for internal team exploration, then one high-fidelity version for users. Not every prototype needs to reach high fidelity. But any prototype going to users does.

5. PRD plus prototype - the new standard

The PRD isn’t dead. But what it’s for has changed.

The old way

It used to be that the PRD was everything. It had to describe the feature completely, in text, for every reader - engineers, designers, stakeholders - to understand the vision.

That worked when prototypes were expensive and rare. But the flip side is that a lot of the content ended up vacuous. You’d get a PRD with every section filled in, but the actual quality of the content was weak. People skim text. Nobody skims an interactive prototype.

The new way

Most of the feature - the core flows, the interaction patterns, the visual logic - should now live in the prototype. That’s what people actually engage with.

The PRD fills in what the prototype can’t show: edge cases, empty states, error conditions, tracking requirements, rollout plan. Things that are hard to prototype exhaustively. Write those down.

Here’s a good test. If someone reads the PRD, plays with the prototype, and still has questions - something’s missing. No open questions is the goal.

The PRD should live inside the prototype project as a file in the same folder. When you do that, the AI reads it when you prompt new features. It doesn’t skim. It applies the context. Your PRD stops being documentation and starts being active instruction.

Cover your 90% flows in the prototype. Cover all edge cases in the PRD.

6. How to hand off to engineers

A functional prototype built in a tool like Dazl isn’t a throwaway demo. It’s a full server-side and client-side application. React components, standard file structure, nothing proprietary. A developer on your team picks it up and it looks like any other web project they’ve worked on.

So share the published prototype link. That alone answers ~90% of the questions that would normally fill up a handoff call.

But for anything more specific, download the project folder, drop it next to the production codebase, and tell Cursor or Claude Code to copy the interaction. Most of the implementation work gets handled automatically.

The skill to build now

It’s how to read enough of the codebase to have an informed conversation.

Open the actual project your company is building. Ask an AI to explain the architecture. Ask for a diagram. Ask what a specific component does.

Do that for a few weeks and you’ll close the communication gap with your engineering team faster than almost anything else you could do.

Get Transcript

Related content

Newsletters

1. AI PRDs: everything you need to know

2. Product Requirements Documents (PRDs): a modern guide

3. Ultimate guide to AI prototyping tools (Lovable, Bolt, Replit, v0)

Podcasts

1. Putting every AI prototyping tool to the test with Magic Patterns CEO

2. AI prototyping with Eric Simons, the CEO of Bolt

3. AI prototyping mastery with Sachin Rekhi

4. AI prototyping intro with Colin Matthews

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Check out the conversation on Apple, Spotify and YouTube.

Brought to you by:

1. Amplitude: The market-leader in product analytics

2. Pendo: The #1 software experience management platform

3. Testkube: Leading test orchestration platform

4. Product Faculty: Get $550 off the AI PM Certification with code AAKASH550C7

5. Bolt: Ship AI-powered products 10x faster

Today’s episode

There is a term Andrej Karpathy coined last year: vibe coding.

We have the same for product management: Vibe PMing.

You describe the problem. The agent pulls the data. Analyzes the chart. Synthesizes the feedback. Drafts the spec. Files the ticket.

That is not theory. That is what I walked through in today’s episode with a principal PM at Amplitude who builds MCP and agent products for a living. He showed it live, on screen, in real time.

If you tune in, you’ll learn the full end-to-end workflow:

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

Newsletter deep dive

As a thank you for having me in your inbox, here is the complete guide to vibe PMing with Claude Code and your analytics tool.

1. What vibe PMing actually means

2. How to set up Claude Code with MCP

3. Five workflows that replace hours of manual PM work

4. The biggest mistakes people make with MCP

5. How to become an AI-native PM in 2026

1. What vibe PMing actually means

You as a PMs use Claude as a writing tool. you paste in a brief, get a draft, copy it into a doc.

That is using 10% of what is available to you.

The real unlock is connecting Claude Code to the tools where your actual work lives. Your analytics platform. Your ticket system. Your customer feedback. Your meeting notes.

When you do that, something changes. The agent is not just a writing partner anymore. It is a product analyst, a spec writer, and a ticket filer, all running while you are in your next meeting.

This is what vibe PMing means. Same way vibe coding let engineers describe what they want and let the agent build it, vibe PMing lets you describe the problem and let the agent figure out what is happening in your data, what customers are saying, and what to do about it.

The term is new. The workflow is not complicated. But most PMs have not made the jump yet, and that gap is widening fast.

2. How to set up Claude Code with MCP

MCP stands for Model Context Protocol. The simplest way to think about it -

It is the easiest way to connect your AI models with any external tool, action, and data.

When you hook Claude Code up to your analytics provider via MCP, the agent can read your charts, query your dashboards, pull your customer feedback, and navigate your product taxonomy, all from terminal. Here is how to set it up.

Step 1 - Build your product repo

Start in Cursor or Claude Code. Create a folder structure for your product context. This is where you store your PRDs, your Q1 plans, your roadmap notes, your specs, all in Markdown files.

Once they are in there, you reference them with the @ command. The agent immediately pulls that context. No copy-pasting. No searching through docs.

Step 2 - Connect your MCPs

In Claude Code, you add your MCP servers. At minimum your analytics provider like Amplitude, and your ticket system like Linear.

Each MCP gives the agent access to the tools and data in that platform. Amplitude’s MCP gives Claude Code access to your charts, dashboards, feedback, experiments, and feature flags.

The key detail - the MCP works best when it is well-configured. The tool names, descriptions, and instructions all get passed as context to the model. Clear names mean fewer wrong tool calls. This is worth spending time on upfront.

Step 3 - Write your skills

Skills are the secret weapon most people skip. A skill is a Markdown file with three parts - a name, a description of when to use it, and a set of heuristics for how to execute it.

You write a skill once. From then on, you just type:

/analyze-chart or /analyze-feedback

and the agent knows exactly what to do, which tools to call, and what format to return. This is what separates a one-off experiment from a repeatable system.

Step 4 - Manage context deliberately

Claude Code shows you what percentage of your context window you are using. When you hit around 80-90%, do not wait for it to compact on its own. Run a command to write a Markdown summary of your progress and what is left to do. Then start a fresh session with that file as context.

Also be selective about which MCPs you have active at any given time. Too many loaded at once means the agent is processing irrelevant tool descriptions on every query, which slows responses and introduces noise.

3. Five workflows that replace hours of manual PM work

This is the core of today’s episode. Five workflows I now run regularly.

Workflow 1 - Deep chart analysis

When a metric spikes or drops and you do not know why, this is the workflow.

Drop a chart URL into Claude Code. Run the analyze-chart skill. The agent parses the URL, pulls the underlying data, looks at related charts and events, checks for relevant feature flag changes, and returns a structured report.

1. What happened and when

2. The most likely hypothesis

3. Supporting evidence

4. What it means for the business

What would have taken a data analyst three hours, navigating the taxonomy, building the breakdown charts, investigating multiple hypotheses, the agent does in about 90 seconds.

The part that surprised me - the agent also cross-references your customer feedback and annotations automatically. So it is not just telling you what moved. It is hypothesizing why, using qualitative context alongside quantitative data.

Workflow 2 - Automated dashboard reporting

This one hit close to home. If you have ever spent a Sunday pulling metrics together for a Monday business review, you know exactly what I am talking about. At Amazon, that was a three-to-five hour ritual every single week. At Epic Games, same thing.

Here is the new workflow. You point a dashboard agent at the four or five dashboards you care about. Every Monday morning, you get a clean report in your inbox.

1. Top three to five insights across all your metrics

2. What changed week over week

3. The one urgent thing that needs your attention

You do not analyze the dashboards anymore. You just react to what already matters. The report goes into Slack. Your whole team has context before the first meeting.

Workflow 3 - Customer feedback synthesis

Most teams have customer feedback scattered everywhere. Zendesk, Slack, Gong, app store reviews, NPS surveys. Nobody reads all of it. The stuff that reaches the roadmap is whatever was loud enough to find its way into a conversation.

With Amplitude’s AI feedback product plus MCP, all of that data gets piped into one place. Then you run your analyze-feedback skill against it. You can ask it to focus on a specific product area. The agent navigates the feedback insights, clusters them, and returns a structured report.

1. Top feature requests

2. Urgent issues

3. The number one thing customers loved that week

Nobody has to skim Slack threads anymore. The synthesis happens automatically and it is sitting in your inbox when you get to your desk.

Workflow 4 - From insight to spec

Most PMs stop at the insight. They have the chart analysis. They have the feedback summary. And then they open a blank Google Doc and start writing from scratch.

You do not have to do that anymore.

Once the analysis is done, you drop the insight into Claude Code along with a few images of the current product experience. You give it your goals and your constraints. Then you just talk to it. What should we build? What are the tradeoffs? What would we cut?

Claude Opus is the best brainstorming partner I have used for product strategy. It thinks in specifics. It pushes back when an idea is weak. It helps you get to a position, not just a list of options.

When you land on a direction, you tell it to draft the PRD using the template in your repo.

Draft this as a PRD using the template in /docs/prd-template.md

The agent pulls your PRD template from the repo and generates a first draft in your format, using the context it already has from the analysis. If the draft is off, you tell it. Too long. Wrong framing. Acceptance criteria are vague. Two or three rounds and you have something worth sending.

The whole thing takes 20 minutes instead of two hours. And you spent those 20 minutes thinking, not typing.

Workflow 5 - From spec to shipped

This is where most workflows fall apart. You have a good spec. Now what?

Two paths, depending on what it is.

If it is a small fix, a UI tweak, a copy change, a minor behavior adjustment, you drop it straight into Claude Code or Cursor and point it at your repo. The agent picks it up and starts working while you go to your next meeting. You come back, review the diff, and either ship it or give feedback.

If it needs the team, you hit the Linear MCP from the same terminal:

File this under the AI capabilities project, assign it to Richard,
add the feedback analysis from this session as context

Done. You can also skip the ticket entirely for simpler things and pipe it straight to the engineer via Slack MCP.

You are not writing tickets anymore. You are routing decisions. The agent handles the formatting, the filing, the context. You just tell it where things should go.

That is what it means to vibe PM. Not doing less thinking, doing more of it, and letting the agent handle everything else.

4. The biggest mistakes people make with MCP

Two that come up constantly.

Wrong expectations

People assume MCP can orchestrate complex multi-step workflows out of the box. It cannot. MCP is the easiest way to connect your AI to external data and actions. That is it. You still have to write the skills and prompts that tell the agent what to do with that access.

“Set the right expectations for yourself on what MCP is used for. It is by far the easiest way to connect external systems with most of the AI clients you are using.”

Too many MCPs loaded at once

Every MCP you have connected adds tool descriptions to your context window on every query, even when those tools are not relevant. This slows responses and can confuse the model.

Be surgical. Only keep MCPs active that are relevant to your current workflow. And when you are building an MCP yourself, invest time in the tool names and descriptions. The clearer they are, the fewer wrong tool calls you get. Think of it like eval-driven development. When you find an edge case, you optimize the descriptor, not just the prompt.

The good news - most clients like Cursor and Claude now use dynamic tool calling, so they do not load all tools into context on every query. The context rot problem is largely being solved at the client level.

5. How to become an AI-native PM in 2026

The simplest version of the advice: default to AI on most tasks.

Treat it as a thought partner. Kick off every analysis, every spec, every strategy session by firing a prompt first and seeing what comes back. You can participate in every part of the process now, pulling data, building analysis, generating prototypes, making recommendations on a draft PR, routing tickets to your team. Most PMs are still only doing one or two of these. The ones doing all five are operating at a different level.

A few tactical things worth building into your week:

1. Spend a few hours each weekend looking at what shipped in the last seven days. New model releases, new agent capabilities, new MCP integrations. The surface area of what is possible changes weekly right now. Staying current is not optional if you want to stay relevant.

2. Learn the AI-specific frameworks. Understanding how evals work. Knowing when you are managing a traditional product process versus when you need to think about edge cases and model behavior. Everyone has an AI feature now. Not everyone knows how to manage one properly.

3. If your org is not giving you access to Claude Code, Cursor, and GitHub, make the request. The productivity delta is not subtle. Show them this episode.

The PMs who figure this out in the next six months will look like wizards to everyone still writing tickets manually.

The modern AI PM stack

For reference, the full stack that makes this work:

1. Claude Code - your primary agent harness and terminal interface

2. Cursor - for IDE workflows, direct file edits, faster model switching

3. Amplitude MCP - to pull charts, dashboards, feedback, and session data

4. Linear MCP - to route specs and tickets without leaving terminal

5. Granola - for meeting notes that feed directly into your product repo

6. Wisprflow - to dictate instructions and orchestrate agents hands-free

The real power is not any single tool. It is the connections between them. Context from your meetings flows into your repo. Your repo feeds your agent. Your agent queries your analytics. Your analytics surfaces insights. Those insights become specs. Those specs become tickets or code.

When that loop is running, you stop feeling like a coordinator and start feeling like an operator.

Get Transcript

Related content

Newsletters:

1. How to use Claude Code like a pro

2. Steal 6 of my Claude skills

3. Context engineering

4. The AI stack for PMs

5. Practical AI agents for PMs

Podcasts:

1. How to build an AI-native PM operating system with Mike Bal

2. AI evals explained simply with Ankit Shukla

3. Advanced guide to AI prototyping with Sachin Rekhi

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Check out the conversation on Apple, Spotify and YouTube.

Brought to you by:

1. NayaOne: Airgapped cloud-agnostic sandbox

2. Pendo: The #1 software experience management platform

3. Maven: Get 15% off Xinran’s course with my link

4. Bolt: Ship AI-powered products 10x faster

5. Gamma: Turn customer feedback into product decisions with AI

Today’s Episode

The role of PM is taking on more and more of design and engineering. But there are literally no good guides on designing with AI for PMs.

So I brought in an expert.

runs Design with AI, one of the top newsletters on AI design. He’s been studying AI design tools for three years. And he hasn’t shared most of this information publicly before.

In today’s episode, we going live. We building real prototypes. We showing you the exact workflows that top 1% designers use.

If you watch, you’ll learn the entire workflow from PRD to prototype to product:

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

Newsletter Deep Dive

As a thank you for having me in your inbox, I’ve written up the complete guide to designing with AI for PMs:

1. What designing with AI means

2. The mind map of AI design workflows

3. The exact workflow in Google AI Studio

4. How to go from idea to prototype in Lovable

5. The 5 best tools for different design use cases

6. How to know when an AI-generated design is actually good

Subscribe now

1. What Designing with AI Means

Designing with AI sounds broad because it is. But there’s structure to it.

Xinran breaks down the entire universe into a mind map with clear categories.

The Five Core Categories

Xinran’s mind map breaks AI design into five main areas.

Category 1 - Prompting

How to prompt better to get better results. This includes prompt engineering, context setting, and iteration techniques.

Most people stop here. Prompting is just the starting point.

Category 2 - Ideation

Using AI as an assistant to generate ideas that humans find hard to think about. This is about divergent thinking at scale.

Category 3 - Design and Prototyping

Using AI tools to actually create designs and prototypes.

We’ll go deep on this with live demos in sections 2 and 3.

Category 4 - Workflows

The systems and processes for how you actually work with AI design tools day-to-day. This is about becoming fast and effective.

Xinran has workflows for different scenarios - quick prototypes, high-fidelity designs, full-stack experiences.

Category 5 - Staying Conscious

The part fewer people talk about. How to bring intention and thoughtfulness to AI design. This includes awareness of risks, biases, and unintended consequences.

You can’t just prompt and ship. You need to think about what you’re building and who it impacts.

Why Designing with AI Matters for PMs

The role of PM, designer, and engineer are merging.

You don’t need to become a professional designer. But you do need to understand design workflows.

With AI tools, the barrier to creating good designs dropped dramatically. A PM can now prototype experiences that used to require a designer and weeks of time.

Only if you understand the full system, not just prompting.

2. Generating Divergent Design Ideas with Google Stitch

The first workflow uses Google Stitch for rapid design exploration.

In the demo, Xinran took a screenshot of Redfin’s “Ask Redfin” AI chat section on a home detail page.

The challenge: given this existing experience, what are the other design variants we could explore?

This is different from starting with a blank canvas. You have an existing design. You want to see what else is possible.

Step 1 - Provide Context and a Visual Reference

In Stitch, paste a screenshot of the existing design.

Then add a prompt with three parts: the context (what the section does and who uses it), the business goal, and your ask.

Xinran’s prompt:

Step 2 - Generate Initial Variations

Stitch generates two or three design variants by default.

Each one takes a different approach to layout, spacing, and information hierarchy while keeping the core functionality.

Review them. See which direction has potential.

Step 3 - Go YOLO for Divergent Exploration

This is where Stitch gets interesting. There’s a creative range slider that goes from refined to YOLO. Set it to YOLO.

Specify what you want to vary: layout, color schemes, text content. Set the output to three or four variants. Hit generate.

The results go wide.

Some will look nothing like the original. Some will break conventions you assumed were fixed. That’s the point. You’re using AI to explore directions you wouldn’t have reached on your own in a 30-minute brainstorm.

Step 4 - Export to Google AI Studio

Pick the variant with the most potential. Export it directly to Google AI Studio.

AI Studio receives the design as an HTML reference file with a prompt. From here, you can turn the static design into an interactive prototype.

You can add system instructions for style consistency.

You can use the annotate feature to add comments and request specific changes, similar to how you’d collaborate in Figma.

One tip: if you selected multiple screens in Stitch before exporting, you get a multi-screen prototype in AI Studio rather than a single page.

3. From Idea to Prototype

The second workflow is more ambitious. We’re going from a blank idea to a fully functional prototype you can click through and test with users.

The trick is you don’t start in Lovable. You start in a custom GPT.

Step 1 - Use a Custom GPT to Get Clarity

Xinran built a custom GPT that asks you a sequence of focused questions before generating any design spec.

The questions force you to define three things: who you’re designing for, what their core need is, and which specific experience to build first.

In the demo, it looked like this.

The GPT asked: What is the main goal of your product? He typed “help users track expenses.”

It asked: Who are the intended users? He typed “freelancers.”

It asked: What platform? He selected “responsive web.”

Then it generated the key user flows to choose from.

One detail worth stealing: his custom GPT is programmed to never suggest login or signup as a key user flow.

His reasoning is that login doesn’t solve the core user problem. It’s standard everywhere. If you figure out the core flow first, auth becomes trivial to add later.

Most generic ChatGPT prompts will recommend sign-up as step one. His GPT skips it entirely.

The GPT then generates a lightweight spec in markdown format.

Not a full PRD. Just enough to define the front-end screens, components, and interactions.

No backend logic. No success metrics. No user personas. Just what the prototype needs to render.

Step 2 - Run a Mock Check in Claude

Before pasting into Lovable, Xinran pastes the spec into Claude as a sanity check. He doesn’t expect great design from Claude.

He uses it as a quick visual preview to see if the prompt produces roughly the right screens and flows.

If something looks dramatically off in Claude’s output, the problem is the spec, not the tool. He goes back and revises before spending Lovable credits.

And yes, he uses ChatGPT for the prompt and Claude for the mock run specifically to save Claude tokens.

His words:

I like to shift it to ChatGPT in order to save tokens for Claude.

Step 3 - Paste Into Lovable for the Real Prototype

Copy the same markdown spec into Lovable. Lovable reads it and generates a working prototype in about 60 seconds.

For the expense tracker demo, it created the add expense screen, the summary view, confirmation states, and navigation between them.

Compared to Claude’s output, the Lovable version had refined spacing, proper dropdown components, arrow states, and a confirmation page that actually looked finished.

You now have a clickable prototype. You can tap through the flow. You can see how it feels.

Step 4 - Iterate on What You See

The first version won’t be perfect. That’s expected.

Now you refine. You iterate.

Look at what Lovable generated. What’s wrong? What’s missing?

In the demo, the expense tracker had the right screens but needed refinement.

You tell Lovable what to change in plain language. Each iteration takes 20-30 seconds.

Five to ten iterations gets you to something polished enough to share with stakeholders or test with users

Step 5 - Add Interactivity and Logic

Here’s where it gets powerful. You can add real logic to your prototype.

• Want the expense categories to auto-suggest based on past entries? Tell Lovable.

• Want the summary view to update totals in real time as you add expenses? Tell Lovable.

• Want to connect a bank API for transaction imports? Tell Lovable.

The tool generates working code. Each iteration makes the prototype more functional.

Step 6 - Export and Share

When you’re happy with the prototype, export it.

Lovable gives you clean React code. You can hand this to engineers. You can deploy it as a test version. You can share it with stakeholders.

Engineers don't need to start from scratch. They have working code to build on.

4. The Best Tools for Different Use Cases

Not every tool is best for every job. You need the right tool for what you’re trying to do.

Xinran uses different tools for different scenarios. Here’s his stack.

For Generating Effective Prompts - Custom GPT

The best tool for creating good prompts is a custom version of ChatGPT.

Build a custom GPT that understands your design system, your product, and your user base.

Feed it your brand guidelines, your existing designs, and examples of good outputs.

Now when you ask for design ideas, it generates suggestions that fit your context.

This is better than generic ChatGPT because it knows your specific situation.

For High-Quality Prototypes - Lovable

Lovable ranks first in Xinran’s stack for design quality.

The output is polished, the code is clean, and it ships with features like visual editing (still catching up to V0 and Figma Make, but improving fast). If price isn’t a factor, this is his pick.

For a Close Second - v0

v0 produces similar quality to Lovable with a different design aesthetic. Lovable leans glamorous and vibrant. v0 is more restrained. v0 also lets you edit code without upgrading to a paid plan, which makes it more accessible.

Personal preference decides this one.

For Design Variations - Magic Patterns

Magic Patterns (free in my bundle) excels at generating multiple design variations quickly.

Give it a single design. Ask for 5 variations. It generates 5 completely different visual directions.

This is perfect for early exploration. You want to see options. You want diversity.

Magic Patterns gives you that faster than any other tool.

For more on AI prototyping tools, check out the AI prototyping guide.

For Free Prototyping - Google AI Studio

Google AI Studio is the best free option.

You don’t need to pay extra. You get solid quality output. You can generate designs, variations, and basic prototypes.

Use this if you’re just getting started or if budget is a constraint.

The quality isn’t quite as good as Lovable. But it’s free and it’s surprisingly capable.

For Full-Stack Experiences - Cursor

Cursor is the best tool if you need backend functionality or complex interactions.

It’s more technical than the other tools. You need some development knowledge.

But if you can write basic code, Cursor is incredibly powerful. You can build complete applications with real databases, APIs, and complex logic.

Use this for serious prototypes that need to feel production-grade.

Where Claude Fits In

Xinran doesn't rank Claude against Lovable or v0.

He puts it in a different category. Claude is his mock-run tool: paste a spec in, get a fast visual check, verify the prompt makes sense before moving to a dedicated prototyping tool.

The design quality isn't as refined as Lovable or V0, but that's not the point. It's a sanity check, not the final output

5. How to Know When an AI Design is Actually Good

Generating designs is easy. Knowing if they’re good is hard.

Here’s how to evaluate AI-generated designs.

Layer 1 - Visual Representation

The first layer is what people see immediately. Does the design look good?

Is it visually pleasant? Does it resonate with users? Is the aesthetic appropriate for your brand and audience?

This is important. But it’s only part of the equation.

Many AI designs look great visually but fail at deeper levels.

Layer 2 - Problem Solving

Does the design actually solve the user problem?

Forget about AI tools for a second. Does this experience address what users need?

You need to validate with real users. Show them the design. Watch them try to use it. See if they can accomplish their goals.

AI can generate pretty designs all day. Only users can tell you if they work.

For more on AI product validation, check out how to do AI evals.

Layer 3 - Design Principles

Is the design accessible? Is it readable for different groups of people? Does it follow established interaction patterns?

These fundamentals still matter.

AI tools don’t automatically produce accessible designs. You need to check:

1. Color contrast ratios

2. Text readability

3. Screen reader compatibility

4. Keyboard navigation

5. Touch target sizes

Run the design through accessibility checkers. Test it with assistive technologies.

Don’t skip this. Accessibility isn’t optional.

Layer 4 - Implementation Feasibility

Can engineers actually build this? Is the design technically feasible within your constraints?

Even though AI prototyping tools generate code, that code might not match your tech stack or architecture.

Review the design with your engineering team. Make sure it’s buildable.

There’s no point creating elaborate designs that can’t ship.

Most people stop after layer 1. They see something pretty and call it done. Run all four checks before you ship anything.

6. The Skills You Need to Master AI Design

If you want to get better at designing with AI, focus on these five skills.

Skill 1 - Prompting with Clarity

The first skill is writing clear, specific prompts.

Good prompts are rooted in clarity about what you want to build. If you can’t clearly describe what you want, AI can’t build it.

Practice being specific. Include context. Define constraints. Describe the user and their needs.

Vague prompts get vague results. Clear prompts get clear results.

For more on prompt engineering, check out the complete guide to prompt engineering.

Skill 2 - Providing Context

Context is what separates good AI outputs from great ones.

The necessary context includes:

1. Who are the users?

2. What problem are you solving?

3. What constraints matter (technical, brand, accessibility)?

4. What’s the broader product or feature this fits into?

More context = better outputs.

Don’t just say “design a button.” Say “design a primary CTA button for a B2B SaaS onboarding flow where users need to connect their calendar. The brand is professional and trustworthy.”

That’s specific. That’s context-rich. That’s what gets you quality.

Skill 3 - Effective References

Visual references, code references, and text references all help AI understand what you want.

The key is balance. Too few references and AI has nothing to work with. Too many references and you overwhelm it.

Use 2-4 references per prompt. Make them spot-on for what you’re trying to achieve.

A screenshot of a similar UI. A link to a design system component. A snippet of code showing interaction logic.

These references anchor AI’s output to something concrete.

Skill 4 - Iteration Speed

Getting good at AI design means getting fast at iteration.

The magic isn’t in the first output. It’s in the 10th iteration after you’ve refined and tweaked.

Practice iterating quickly. Review, identify what’s wrong, tell AI how to fix it, repeat.

The faster you can iterate, the better your final output.

Speed comes from experience. The more you use these tools, the faster you get.

Skill 5 - User Empathy

This is the most important skill. If you’re empathetic toward users, you’ll go the extra mile to validate what they need.

AI tools make it easy to generate designs. But only human empathy tells you if those designs actually help users.

Talk to users. Watch them use your prototypes. Listen to their frustrations. Understand their context.

All the AI design tools in the world won’t help if you’re building the wrong thing.

This is still product work at its core. AI just makes the execution faster.

7. The Biggest Workflow Changes

Xinran’s workflow has completely transformed since AI design tools became good.

Here’s what changed and why it matters.

Before AI - Linear and Slow

The old workflow was linear.

You’d start with user research. Then create rough sketches. Then build wireframes. Then design high-fidelity mockups in Figma. Then connect screens together. Then create a clickable prototype.

Each step depended on the previous step. You couldn’t skip ahead. You couldn’t explore alternatives without going back to earlier stages.

This took weeks.

After AI - Parallel and Fast

The new workflow is parallel.

You can describe what you want in text. Add a visual or code reference. Generate a proof of concept in minutes.

You can explore multiple directions simultaneously. You can iterate on any part of the experience without redoing everything.

The workflow became non-linear. You can jump between ideation and prototyping freely.

This is powerful, especially in early stages.

The Alignment Advantage

Early alignment matters more than perfect details.

With AI tools, you don’t need every screen designed in Figma to convey your idea.

You can generate a rough prototype. Show it to stakeholders. Get aligned on direction.

Then go back and refine the details.

This is faster and more effective than spending days on pixel-perfect mockups only to learn stakeholders wanted something different.

The ability to get aligned quickly transforms product development.

What This Means for You

If you’re still designing the old way - linear, Figma-first, perfect pixels before validation - you’re too slow.

Adopt the new workflow:

1. Describe what you want to build

2. Generate a quick prototype with AI

3. Share with stakeholders and users

4. Iterate based on feedback

5. Refine the details once direction is validated

This is how top designers work now. It’s how you should work too.

Get Transcript

Where to Find Xinran

• LinkedIn

• Newsletter

• Maven course

Related Content

Newsletters:

• AI Prototyping Tutorial

• AI Prototype to Production

• How to Pair with Design as a PM

Podcasts:

• Advanced Guide to AI Prototyping with Sachin Rekhi

• 25 Designs to Steal

• AI Design

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Check out the conversation on Apple, Spotify and YouTube.

Brought to you by - Reforge:

Get 1 month free of Reforge Build (the AI prototyping tool built for PMs) with code BUILD

Today’s Episode

Ankit Shukla is BACK after his gangbusters episode, that is my #2 most popular of all time. This time he's diving deep on one of the most important new AI skills for PMs: Evals.

Whether you're working on AI features now or not, this is a skill you want to have an intuitive understanding of. So, I'm building on my library of eval episodes with today's drop.

I've never heard someone explain evals from first principles as intuitively as Ankit has with this one. Hope you enjoy as much as I did!

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

Newsletter Deep Dive

As a thank you for having me in your inbox, I’ve written up a complete guide to AI evals.

1. Why AI evals matter for all PMs

2. The fundamental nature of LLMs

3. How to build your evaluation rubric

4. The complete eval metrics framework

5. How to build LLM judges step-by-step

6. Production monitoring that actually works

Subscribe now

1. Why AI Evals Matter for Every PM

AI evals matter for every PM because AI is just another API you might call. And once you call that API, you need to know if it’s working.

When you build a traditional feature, you know what’s happening. The code is deterministic. If you write “2 + 2”, you get 4. Every single time. AI doesn’t work like that.

The same prompt can give you different answers. The same user query can produce wildly different results. You don’t control the output the way you control traditional code.

This creates a fundamental problem for product management. You can’t just “test” an AI feature the way you test a normal feature. You need a completely different approach.

The Three-Part Eval System

There are three types of evaluations you need:

1. Offline evals - Testing before launch

2. Online evals - Monitoring after launch

3. Human evals - Spot-checking quality

Most teams only do offline evals. Some do online evals. Almost nobody does human evals properly.

You need all three. Offline evals tell you if the feature is ready to ship. Online evals tell you if it’s working in production. Human evals tell you if users actually like it.

Skip any of these and your feature will fail.

Why PMs Should Own This

Product leaders like Todd Olson (CEO of Pendo) and Rachel Wolan (CPO of Webflow) all say the same thing.

AI evals is the most important new skill for PMs.

Why?

Because PMs sit at the intersection of business, customer, and technology. You understand what success looks like. You understand what the customer needs. You understand how the business measures value.

Engineers understand the code. Data scientists understand the models. But PMs understand the outcome.

That’s why AI product managers need to own evals.

The Cost of Not Doing Evals

Here’s what happens when you skip evals.

Your prototype works great in the demo. You show it to stakeholders. Everyone loves it. You get approval to ship.

Then you launch.

Users start complaining. The AI hallucinates. It gives wrong answers. It misunderstands simple queries. Your support tickets explode.

You scramble to fix it. But you don’t know what’s broken. You don’t have metrics. You don’t have baselines. You’re flying blind.

Eventually you roll back the feature. Six months of work down the drain.

2. The Fundamental Nature of LLMs

Before you can build good evals, you need to understand how LLMs actually work.

Most PMs don’t understand this. They treat LLMs like a black box. Type in a prompt, get an answer. Magic.

But LLMs aren’t magic. They’re statistical models. And understanding how they work changes how you evaluate them.

LLMs Are Probabilistic

LLMs don’t “know” anything. They predict the next token based on probability distributions.

When you ask “What’s the capital of France?”, the LLM doesn’t “look up” the answer. It predicts that the most probable next tokens are “The capital of France is Paris.”

This seems like a small distinction. But it matters. Because sometimes the LLM predicts wrong. Not because it’s “broken”. But because the probability distribution led it to a different answer.

This is why the same prompt can give different results. The model samples from a distribution. Different samples = different outputs.

The Temperature Problem

Temperature controls randomness.

Temperature = 0 → Deterministic outputs (always picks highest probability token) Temperature = 1 → More random outputs (samples from full distribution)

Most products use temperature between 0.3 and 0.7.

Why does this matter for evals?

Because you need to test at the temperature you’ll use in production. If you eval at temperature 0 but ship at temperature 0.7, your evals are meaningless.

The Context Window Problem

LLMs have limited context windows. GPT-4 has 128K tokens. Claude has 200K tokens.

But here’s what most PMs don’t realize.

The quality of responses degrades as you fill the context window. The model “pays attention” to the beginning and end of the context more than the middle.

This creates the “lost in the middle” problem.

If you put critical information in the middle of a long prompt, the LLM might miss it. Even though it’s technically within the context window.

This means your evals need to test different context lengths. A prompt that works at 1K tokens might fail at 50K tokens.

The Prompt Sensitivity Problem

Small changes in prompts create large changes in outputs.

Change “Please summarize this document” to “Summarize this document” and you get different results. Add “Be concise” and you get different results again.

This is called prompt sensitivity.

It makes AI products incredibly fragile. A tiny change breaks everything.

Your evals need to test prompt variations. Not just one canonical prompt. Multiple variations that users might actually type.

The Hallucination Problem

LLMs hallucinate. They make up facts. They cite sources that don’t exist. They invent details.

Why?

Because they’re trained to predict plausible text. Not accurate text. Plausible.

If the most plausible completion is a fake citation, the model will generate it. Confidently.

You can’t just check if the output “looks good”. You need to verify it’s actually correct.

3. How to Build Your Evaluation Rubric

Now let’s build an evaluation rubric step by step.

A rubric defines what “good” looks like. Without a rubric, you can’t measure quality. You’re just guessing.

Start With User Scenarios

Don’t start with the AI. Start with the user.

What are the top 10 scenarios your users will encounter?

For a customer support chatbot:

1. User asks about return policy

2. User asks about shipping times

3. User asks about product specifications

4. User asks about account issues

5. User asks unclear questions

For a code generation tool:

1. User requests a simple function

2. User requests a complex algorithm

3. User requests refactoring

4. User requests bug fixes

5. User requests tests

Write these down. These become your test cases.

Define Success Criteria

For each scenario, define what success looks like.

Bad success criteria: “The response is helpful” Good success criteria: “The response contains the correct return window (30 days) and includes the return portal link”

Bad success criteria: “The code works” Good success criteria: “The code passes all test cases, follows project style guide, and includes error handling”

Specific. Measurable. Unambiguous.

Build Your Rubric Categories

A good rubric has 4-6 categories:

1. Correctness - Is the answer factually correct?

2. Completeness - Does it address all parts of the query?

3. Clarity - Is it easy to understand?

4. Tone - Does it match brand voice?

5. Safety - Does it avoid harmful content?

6. Efficiency - Is it concise without being terse?

Each category gets a 1-5 scale:

1. Completely fails the criteria

2. Mostly fails with some success

3. Partially succeeds

4. Mostly succeeds with minor issues

5. Fully succeeds

Define what each score means for each category. Don’t leave it ambiguous.

Create Reference Examples

This is the part most teams skip. Don’t.

For each category and score level, create a reference example.

Example for “Correctness” in a support chatbot:

Score 5: “Our return policy allows returns within 30 days of purchase. Visit returns.company.com to start your return. You’ll receive a full refund within 5-7 business days.”

Score 3: “You can return items within 30 days. Check our website for details.”

Score 1: “We don’t accept returns.” (Factually wrong)

These examples become your ground truth. They show evaluators (human or LLM) what quality looks like.

Test Your Rubric

Have 2-3 people independently grade the same 10 outputs using your rubric.

Calculate inter-rater reliability. If people disagree on scores, your rubric is ambiguous.

Refine the rubric until multiple evaluators give similar scores.

This step is critical. A bad rubric produces bad evals. A good rubric produces reliable evals.

4. The Complete Evaluation Metrics Framework

Now let’s talk metrics.

You need different metrics for different use cases. There’s no “one size fits all” metric for AI evaluation.

Retrieval Metrics

If your AI retrieves information (like RAG systems), you need retrieval metrics.

1. Precision - Of the documents retrieved, how many are relevant?

2. Recall - Of all relevant documents, how many did you retrieve?

3. F1 Score - Harmonic mean of precision and recall

4. MRR (Mean Reciprocal Rank) - How quickly do you surface the right document?

5. NDCG (Normalized Discounted Cumulative Gain) - Weighted scoring based on position

For most AI PM use cases, F1 and NDCG matter most.

Generation Metrics

If your AI generates text, you need generation metrics.

1. BLEU - Compares generated text to reference text (good for translation) ROUGE - Measures overlap between generated and reference text (good for summarization)

2. METEOR - Accounts for synonyms and stemming (better than BLEU for most uses)

3. BERTScore - Uses embeddings to measure semantic similarity (best for most products)

Most AI products should use BERTScore. It’s the most robust.

Task-Specific Metrics

Some tasks need custom metrics.

For code generation:

1. Does it compile?

2. Does it pass tests?

3. Does it follow style guide?

4. What’s the cyclomatic complexity?

For customer support:

1. Does it contain the correct information?

2. Does it include required links?

3. Does it match brand tone?

4. Does it resolve the issue?

For summarization:

1. Does it capture key points?

2. Does it omit irrelevant details?

3. Is it the right length?

4. Is it coherent?

Define the metrics that matter for your specific product.

LLM-as-Judge Metrics

This is where it gets powerful.

Instead of using traditional metrics, use an LLM to grade outputs.

Ask GPT-4 or Claude: “On a scale of 1-5, how helpful is this response?”

This works surprisingly well. LLM judges often correlate better with human judgment than traditional metrics.

The key is giving the judge a good rubric. Feed it your scoring criteria. Give it reference examples.

Then let it grade outputs automatically.

Choosing the Right Metrics

Use this decision tree:

If you’re retrieving documents → Precision/Recall/F1 If you’re generating text similar to references → BERTScore If you’re doing a specific task → Task-specific metrics
If you’re measuring quality holistically → LLM-as-judge

Most products need multiple metrics. Don’t rely on just one.

5. How to build LLM judges step-by-step

Now let’s build an LLM judge from scratch.

This is where most teams get stuck. They understand the concept. They don’t know how to implement it.

Here’s the step-by-step process.

Step 1 - Define Your Evaluation Prompt

Your evaluation prompt needs four components:

1. The rubric (scoring criteria)

2. Reference examples

3. The input query

4. The output to evaluate

Here’s a template:

You are an expert evaluator for a customer support chatbot.

Evaluate the following response on these criteria:

1. Correctness (1-5): Is the information factually accurate?
2. Completeness (1-5): Does it fully address the query?
3. Clarity (1-5): Is it easy to understand?
4. Tone (1-5): Does it match our friendly, helpful brand voice?

Reference examples:
[Insert your 5-point and 1-point reference examples here]

User query: {query}
Assistant response: {response}

Provide scores for each criterion and a brief justification.

Step 2 - Test on Known Examples

Before automating, test manually.

Take 10 outputs where you know the correct score. Run them through your LLM judge. Compare the judge’s scores to your ground truth.

If the judge scores match your expectations, great. If not, refine your prompt.

Common fixes:

1. Add more reference examples

2. Make criteria more specific

3. Add chain-of-thought reasoning (”Explain your reasoning before scoring”)

4. Use a better model (GPT-4 vs GPT-3.5)

Iterate until the judge is reliable.

Step 3 - Implement with Claude Code

You don’t need to write this yourself. Take your evaluation prompt from Step 1, your test dataset, and give Claude Code a prompt like:

Here’s a template:

"Build me an eval pipeline that takes a CSV of test cases (columns: query, response, expected_score), runs each one through Claude as a judge using this evaluation prompt [paste prompt], parses the scores, and outputs a summary CSV with per-criterion averages and a flagged list of any responses scoring below 3.5 on any dimension."

That gets you a working eval pipeline in minutes. You can iterate from there - add visualizations, connect it to your production logs, set up scheduled runs.

The PM’s job is defining the rubric, curating the test cases, and interpreting the results. The implementation is a solved problem now.

Step 4 — Read the Results and Find the Gaps

Once your pipeline runs, you need to know what to look at.

Tell Claude Code:

“Generate a summary dashboard from my eval results. Show me mean score per criterion, the distribution of scores, the 10 worst-performing examples, and the 10 best-performing examples.”

Then do your PM job: look at the worst performers. Why did they fail? Is there a pattern? Are certain query types consistently weak? Is one criterion dragging everything down?

This is where your product sense matters. The numbers tell you where to look. Your judgment tells you what to do about it.

Step 5 - Set Quality Thresholds

This is a pure product decision. No tool makes this call for you.

Define minimum acceptable scores for each criterion. Example:

• Correctness: Must be ≥4.0 average

• Completeness: Must be ≥4.0 average

• Clarity: Must be ≥3.5 average

• Tone: Must be ≥3.5 average

If any criterion falls below threshold, the feature isn’t ready to ship.

Once you’ve decided your thresholds, add them as pass/fail gates in your pipeline. Tell Claude Code:

“Add pass/fail checks to my eval pipeline. Flag the run as failed if any criterion average drops below these thresholds: [paste thresholds]. Output a clear pass/fail summary at the end of each run.”

Step 6 - Run Evals Continuously

Don’t eval once. Eval continuously.

Run your eval suite before every release, daily in production on a sample of traffic, after every prompt change, and after every model update. This catches regressions before users see them.

Tell Claude Code:

“Set up my eval pipeline to run nightly against a random 1% sample of yesterday’s production traffic. Send me a Slack notification if any criterion drops below threshold.”

Now you have a system, not a one-off test.

Common LLM Judge Pitfalls

• Pitfall 1 - Using the same model as your judge and your product Solution: Use a stronger model as your judge (e.g., GPT-4 to judge GPT-3.5 outputs)

• Pitfall 2 - Not calibrating your judge Solution: Regularly compare judge scores to human scores and adjust prompts

• Pitfall 3 -Judging too many dimensions at once Solution: Break complex rubrics into multiple judge calls

• Pitfall 4 -Not using temperature=0 for judges Solution: Always use temperature=0 for evaluation (you want deterministic scores)

6. Production Monitoring That Actually Works

Evals don’t stop at launch. Production monitoring is where most teams fail.

You shipped the feature. Users are using it. Now what?

The Three Layers of Production Monitoring

Layer 1 - System Metrics These are the basic health metrics:

1. Latency (p50, p95, p99)

2. Error rate

3. Token usage

4. API costs

5. Timeout rate

If these metrics spike, something’s wrong with your infrastructure.

Layer 2 - Quality Metrics

These measure if your AI is performing well:

1. Average LLM judge scores

2. Human feedback scores (thumbs up/down)

3. Task success rate (did the user get what they wanted?)

4. Hallucination rate (how often does it make things up?)

These tell you if your feature is actually working.

Layer 3 - Business Metrics These measure if your AI is driving value:

1. Feature adoption rate

2. User retention

3. Customer satisfaction (CSAT/NPS)

4. Support ticket deflection

5. Revenue impact

These tell you if the feature is worth keeping.

You need all three layers. System metrics without quality metrics = blind to bad outputs. Quality metrics without business metrics = no idea if it matters.

Setting Up Automatic Alerts

Don’t just monitor. Alert.

Set up alerts for:

1. Quality scores drop below threshold

2. Error rates spike above 1%

3. Latency exceeds 3 seconds for p95

4. Hallucination rate exceeds 5%

When alerts trigger, investigate immediately.

The Human Review Queue

Sample 1% of production traffic for human review.

Every day, have someone on your team review 10-20 real user interactions. Grade them manually using your rubric.

This catches things LLM judges miss. It keeps you connected to real user experiences.

If human scores diverge from LLM judge scores, recalibrate your judge.

The Feedback Loop

Production monitoring should feed back into your eval dataset.

When you find bad outputs in production:

1. Add them to your test dataset

2. Label them with correct scores

3. Rerun evals to confirm your system would catch them now

This creates a virtuous cycle. Your evals get better over time.

When to Rollback

Define rollback criteria ahead of time.

Example criteria:

1. Quality score drops >10% from baseline

2. Error rate exceeds 5%

3. More than 3 critical bugs reported in first 24 hours

4. Business metrics show negative impact

If any criterion is met, rollback immediately. Don’t wait to see if it “gets better”.

Final Words

AI evals isn’t just testing with a new name.

It’s a fundamentally new skill that combines product sense, technical understanding, and statistical thinking.

Here’s what you need to do:

1. Build a comprehensive evaluation rubric for your product

2. Implement the right metrics (retrieval, generation, task-specific, LLM-as-judge)

3. Create an LLM judge that automates evaluation

4. Set quality thresholds before launch

5. Monitor quality continuously in production

6. Feed production learnings back into your eval dataset

And most importantly: don’t ship without evals.

Where to find Ankit Shukla

• HelloPM

• Twitter (X)

• LinkedIn

• YouTube

Related Content

Newsletters:

• AI Evals

• AI Testing

• LLM Judges

Podcasts:

• How to Do AI Evals Step-by-Step with Real Production Data

• The PM’s role in AI Evals

• AI Evals Live

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Check out the conversation on Apple, Spotify and YouTube.

Brought to you by:

1. Maven: Get 15% off Caitlin’s courses with code AAKASHxMAVEN

2. Pendo: The #1 software experience management platform

3. Kameleoon: AI experimentation platform

4. Amplitude: The market-leader in product analytics

Today’s Episode

Discovery might be the most important core PM skill for building great products.

But most PMs are unprepared to do discovery in AI. PMs run surveys incorrectly, conduct interviews poorly, and end up with poor insights.

Today will give you the roadmap to avoid all those mistakes.

Caitlin Sullivan is a user research expert who runs courses teaching PMs how to do AI-powered discovery. And in today’s episode, she shows you exactly how she does it.

We’re talking live demos. Step-by-step workflows. Real survey data. Real interview transcripts.

This is a masterclass in discovery. The kind that moves the needle.

Apple Podcast

Spotify

Your Newsletter Subscriber Bonus

As a thank you for having me in your inbox, I’ve written up a complete guide to building your AI native PM operating system. It’s the ultimate composable stack playbook:

Today’s guide covers:

1. Why AI Changes Discovery Completely

2. The Step-by-Step Survey Analysis Workflow

3. The Step-by-Step Interview Analysis Workflow

4. How to Build Discovery Agents with Claude Code

5. Advanced Techniques for Production-Grade Analysis

Subscribe now

1. Why AI Changes Discovery Completely

Most PMs treat discovery as a checkbox.

Run some user interviews. Send out a survey. Check the box. Move on to building.

The problem is the analysis never actually happens.

You end up with:

1. Survey responses sitting in a spreadsheet

2. Interview recordings you never transcribe

3. Notes scattered across documents

4. Zero actionable insights

I’ve seen this pattern on every product team I’ve worked with.

The issue isn’t that PMs don’t care about discovery. It’s that traditional discovery takes too much time.

A proper interview analysis used to require:

1. Transcribing the recording (30+ minutes per interview)

2. Reading through the entire transcript

3. Extracting key themes and quotes

4. Organizing findings by topic

5. Synthesizing across multiple interviews

For a standard discovery sprint with 10 interviews, that’s 10+ hours of analysis work.

Most PMs don’t have 10+ hours. So the analysis doesn’t happen.

AI Solves the Time Problem

With AI, the same analysis takes minutes instead of hours. (If you haven't set up your AI stack yet, start with my bundle.)

Here’s what the workflow looks like for analyzing a single interview:

1. Upload transcript to Claude

2. Run structured analysis prompt

3. Get organized findings with timestamps

4. Export to markdown file

Total time: 3-5 minutes per interview.

For 10 interviews, that’s 30-50 minutes instead of 10+ hours.

This changes the economics of discovery completely.

When analysis takes 10 hours, you only do it for the most critical projects. When analysis takes 30 minutes, you can do it for everything.

The Three Discovery Use Cases

AI-powered discovery works for three main scenarios:

Use Case 1 - Feature Validation

You’ve built a prototype. You need to validate it with users before investing engineering time.

• Traditional approach: Run 5-10 user tests, take notes, report findings verbally to the team.

• AI approach: Record every session, get transcripts analyzed automatically, share structured findings document with timestamps.

The difference is you actually have data to reference later. When someone asks “why did we build it this way?” you have timestamped quotes to point to.

Use Case 2 - Retention Analysis

Your product has a retention problem. You need to understand why users churn.

• Traditional approach: Look at analytics, make guesses about why people leave.

• AI approach: Interview churned users, analyze transcripts for patterns, identify specific friction points with evidence.

In the episode, we walked through this exact process using interviews with meditation app users to understand retention drivers.

The output included:

1. Value anchors (what users care about most)

2. Friction points (where the experience breaks down)

3. Competitive gaps (what alternatives provide that you don’t)

4. Risk segments (which user groups are likely to churn)

All with timestamped evidence from actual user conversations.

Use Case 3 - Market Research

You’re exploring a new product area. You need to understand user needs and competitive landscape.

• Traditional approach: Read competitor websites, run a few interviews, make educated guesses.

• AI approach: Systematic survey analysis, interview synthesis, competitive analysis with structured outputs.

This is where AI really shines. You can process hundreds of survey responses and dozens of interviews to find patterns that would be impossible to spot manually.

The key insight is that AI doesn’t replace the need for good discovery questions. It just removes the friction from analysis.

You still need to know your users. You still need to ask good questions. But now you can actually act on what you learn.

Continue Reading Online

2. The Step-by-Step Survey Analysis Workflow

Let’s get tactical.

Setup

You’ve just finished a survey. You have responses in a spreadsheet.

For the demo, Caitlin used a meditation app retention survey with these questions:

1. How long have you been using the app?

2. What made you start using it?

3. What keeps you coming back?

4. What would make you stop using it?

5. Have you considered alternatives?

Standard retention research questions. Nothing fancy.

The key is how you analyze the responses.

Step 1 - Export Your Data

Get your survey data into a clean format.

Google Forms automatically exports to Sheets. Typeform exports to CSV. Whatever tool you use, get a clean spreadsheet with:

1. One row per response

2. One column per question

3. Timestamps if you have them

For Caitlin’s workflow, she just copies this data directly. No need to sanitize or clean it first.

Step 2 - Open Claude Projects

Here’s the first key decision point.

You can analyze surveys in two ways:

Option A - Single conversation in regular Claude

Good for: One-off analysis, quick insights, learning the workflow

Downside: No memory between sessions, have to reupload data each time

Option B - Claude Projects with uploaded files

Good for: Ongoing research, multiple rounds of analysis, building institutional knowledge

Downside: Requires paid Claude Pro subscription

The better path is Claude Projects. The advantage is you can upload all your survey data and interview transcripts once, then reference them in multiple conversations.

Create a new project called something like “Meditation App Retention Study.”

Upload your survey responses spreadsheet.

Step 3 - Write the Analysis Prompt

This is where most people struggle.

Generic prompts like “analyze this survey” produce generic outputs. You need structure.

Here’s the exact prompt structure from the episode:

I have survey responses from [product] users about [topic].

Please analyze the data and provide:

1. Overall patterns and themes
2. Key insights by question
3. User segments (if patterns emerge)
4. Actionable recommendations

For each insight, include:
- The finding
- Supporting evidence (quotes)
- Implications for product decisions

The key elements:

1. Context - Tell Claude what this is (product, topic, goal)

2. Structure - Define exactly what outputs you want

3. Evidence requirement - Ask for quotes to back up claims

4. Actionability - Force conclusions that inform decisions

This structure ensures you get usable output instead of generic observations.

Step 4 - Review Initial Analysis

Claude processes the data and returns structured findings.

For the meditation app survey, it identified:

1. 3 stable subscriber segments (daily meditators, therapy seekers, sleep users)

2. 2 at-risk segments (price-conscious users, feature explorers)

3. 4 key retention drivers (habit formation, therapy integration, sleep quality, guided content)

4. 3 churn risk factors (price increases, feature removal, competitor features)

Each finding included supporting quotes from actual survey responses.

This initial analysis takes about 2-3 minutes to generate.

Step 5 - Dig Deeper with Follow-Up Prompts

The real power comes from iterating.

The follow-up prompt:

For the "therapy seekers" segment, can you:

1. Extract all relevant quotes
2. Identify their specific pain points
3. Compare what they get from our app vs traditional therapy
4. Recommend features that would increase their retention

This produces a focused deep-dive on one segment.

The beauty of Claude Projects is you can come back tomorrow and ask “What did we learn about therapy seekers?” and Claude remembers the analysis.

You’re building institutional knowledge that persists.

Step 6 - Export to Markdown

Once you have findings you want to share, export them.

The approach that works best: Ask Claude to format everything as a markdown document with:

1. Executive summary at the top

2. Detailed findings by section

3. All supporting quotes included

4. Recommendations clearly called out

Copy the markdown. Paste into your note-taking system. Share with your team.

Now you have a permanent record of the research, not just scattered conversation history.

Common Survey Analysis Mistakes

Mistake 1 - Not providing enough context

If you just upload a spreadsheet with no explanation, Claude has to guess what it’s looking at.

Always include: What product, what goal, what decisions this research informs.

Mistake 2 - Accepting the first output

The initial analysis is never complete. Iterate. Ask follow-ups. Dig into specific segments or questions.

Mistake 3 - Not exporting findings

If you leave everything in the chat, you’ll forget it. Export to markdown, save to your docs, make it permanent.

Mistake 4 - Analyzing surveys in isolation

Surveys tell you what people say. Interviews tell you what they mean. Usage data tells you what they do.

The best discovery processes combine all three sources.

3. The Step-by-Step Interview Analysis Workflow

Survey analysis is useful. But interview analysis is where AI really shines.

Interviews are rich, unstructured conversations. Extracting insights manually is time-consuming and inconsistent.

With AI, you can systematically analyze every interview the same way.

The Setup

You’ve conducted user interviews. You have transcripts.

For the demo, Caitlin analyzed meditation app retention interviews. The interviews explored:

1. Why users started using the app

2. What value they get from it

3. What would cause them to stop using it

4. How it compares to alternatives

Standard retention interview questions.

Step 1 - Get Clean Transcripts

Record your interviews (with permission). Use a transcription service.

Options:

1. Otter.ai (automatic transcription during Zoom calls)

2. Descript (upload recordings, get transcripts)

3. Riverside.fm (records and transcribes automatically)

4. Claude itself (upload audio files directly)

The transcript quality matters. Clean transcripts with speaker labels produce better analysis.

Format should be:

Interviewer: Why did you start using the meditation app?

User: I was going through a really stressful period at work...

Interviewer: What keeps you coming back?

User: Honestly, the therapy chat feature...

Step 2 - Define Your Analysis Framework

Before analyzing, decide what you’re looking for.

The framework that produces the best results is value anchors:

Value anchors are the core reasons users care about your product. The things that, if you removed them, users would leave.

For each interview, extract:

1. Value anchors - What specifically provides value

2. Supporting evidence - Exact quotes with timestamps

3. Competitive context - How this compares to alternatives

4. Retention risk - What might cause this user to churn

This framework ensures consistent analysis across all interviews.

You can use different frameworks depending on your goal. For feature validation, you might focus on usability issues and feature requests instead.

Step 3 - Create the Analysis Prompt

Here’s the exact interview analysis prompt:

I'm analyzing user interviews for retention research.

Please analyze this interview transcript and provide:

## Phase 1: Retention Assessment
- Retention segment (stable, moderate risk, high risk)
- Key retention drivers
- Churn risk factors

## Phase 2: Value Anchors
For each value anchor:
- Specific feature or capability they value
- Why it matters to them (quote with timestamp)
- How it compares to alternatives
- Risk if this value anchor were removed

## Phase 3: Recommendations
- What would increase this user's retention
- What product changes would reduce churn risk
- What competitive gaps we should address

Include exact quotes with timestamps for all findings.

This prompt produces a structured analysis document for every interview.

Step 4 - Analyze One Interview

Upload the first transcript. Run the analysis prompt.

Claude processes the transcript and returns:

Retention Assessment: “Stable subscriber - high engagement, multiple value anchors, low churn risk”

Value Anchors:

1. Therapy chat feature (”This therapy chat feature directly addresses a gap that traditional therapy couldn’t fill” - 00:08:32)

2. Daily habit formation (”I’ve built it into my morning routine” - 00:14:12)

3. Sleep tracking integration (”The sleep quality improvements are measurable” - 00:21:45)

Recommendations:

1. Protect therapy chat feature (core retention driver)

2. Build on habit formation (push notifications, streaks)

3. Expand sleep analytics (more detailed insights)

Total analysis time: 3-5 minutes.

Compare that to the 30-60 minutes it would take to manually review the transcript, extract quotes, and organize findings.

Step 5 - Analyze All Interviews

Repeat for every interview transcript.

In the demo, we analyzed 5 interviews. That’s 15-25 minutes total.

Each interview gets its own structured analysis document.

The consistency is the key benefit here. Every interview is analyzed using the exact same framework, making patterns easier to spot.

Step 6 - Synthesize Across Interviews

Once you’ve analyzed all individual interviews, synthesize across them.

Prompt:

I've analyzed 5 interviews. Please synthesize findings across all of them:

1. What value anchors appear in multiple interviews?
2. What retention patterns are consistent across segments?
3. What churn risks are most common?
4. What recommendations apply across user types?

Organize findings by frequency (how many users mentioned each theme).

This produces your executive summary.

The output shows you which findings are one-offs vs which are patterns that matter.

For the meditation app:

1. Therapy chat - Mentioned by 4/5 users (critical feature)

2. Habit formation - Mentioned by 5/5 users (universal need)

3. Price sensitivity - Mentioned by 2/5 users (segment-specific)

Now you know where to focus.

Step 7 - Export Everything

Create a master document with:

1. Executive summary (synthesized findings)

2. Individual interview analyses (detailed breakdowns)

3. Supporting evidence (quotes with timestamps)

4. Recommendations (prioritized by frequency)

This becomes your research deliverable.

Share it with your team. Reference it in product strategy discussions. Use it to inform roadmap decisions.

The key is you have real evidence behind every decision. When someone questions why you’re prioritizing the therapy chat feature, you can point to 4 separate interviews with timestamped quotes.

4. How to Build Discovery Agents with Claude Code

Manual analysis works. But agents are better.

The difference between manual and agent-based:

1. Manual approach - You run prompts, review outputs, iterate, export results

2. Agent approach - You define the workflow once, the agent runs it automatically on all your data

For ongoing research programs, agents are worth the setup time.

When to Build Discovery Agents

Build agents when:

1. You’re running regular user research (weekly, monthly)

2. You need consistent analysis across many interviews

3. Multiple people on your team do research

4. You want to scale research without hiring more researchers

Don’t build agents when:

1. This is a one-time research project

2. You’re still figuring out your analysis framework

3. You don’t have technical resources to maintain the agent

Start with manual workflows. Once you’ve done 10+ interviews the same way, automate it.

The Architecture

Discovery agents need three components:

Component 1 - Data Pipeline

The agent needs access to your research data.

For surveys: Google Sheets integration For interviews: Transcript files in a shared folder

Google Sheets works well for survey data. Local files work for interview transcripts.

Component 2 - Analysis Logic

This is where your prompts live.

The agent needs instructions for:

1. How to process survey responses

2. How to analyze interview transcripts

3. What outputs to generate

4. Where to save results

You’re essentially converting your manual workflow into code.

Component 3 - Output Generation

The agent needs to save results somewhere useful.

The agent outputs markdown files with:

1. Executive summaries

2. Detailed findings

3. Supporting quotes

4. Recommendations

These files automatically save to a shared folder where her team can access them.

Step 1 - Set Up the Project File

Create a file called instructions.md that defines your agent’s behavior.

Here’s what the instructions file looks like:

# Discovery Analysis Agent

## Purpose
Analyze user research data (surveys and interviews) for the meditation app retention study.

## Data Sources
- Survey responses: [Google Sheets link]
- Interview transcripts: /transcripts folder

## Analysis Framework

### For Surveys
1. Read all responses
2. Identify patterns and themes
3. Segment users by behavior
4. Extract actionable insights
5. Output to survey-analysis.md

### For Interviews
1. Process each transcript individually
2. Extract value anchors with timestamps
3. Assess retention risk
4. Provide recommendations
5. Output to interview-analysis-[name].md

## Output Format
All outputs should be markdown files with:
- Executive summary
- Detailed findings
- Supporting evidence
- Recommendations

This file becomes the agent’s operating manual.

Step 2 - Build the Survey Analysis Script

Claude Code can write the script for you.

Prompt:

Based on the instructions.md file, write a Python script that:

1. Connects to the Google Sheets survey data
2. Analyzes responses using the Claude API
3. Generates a structured markdown report
4. Saves the report to survey-analysis.md

Use the analysis prompt from my previous manual workflow.

Claude Code generates the script.

The script uses:

1. Google Sheets API to read survey data

2. Claude API to analyze responses

3. File system operations to save results

You don’t need to write any of this yourself. Claude Code handles the implementation.

Step 3 - Build the Interview Analysis Script

Same process for interviews.

Prompt:

Write a Python script that:

1. Reads all transcript files from /transcripts folder
2. Analyzes each interview using the Claude API
3. Generates individual analysis reports
4. Saves reports to interview-analysis-[name].md

Use the value anchors framework from my previous manual workflow.

Claude Code generates another script.

This one loops through all transcript files and processes them automatically.

Step 4 - Test the Agent

Run both scripts on your existing data.

Check the outputs:

1. Do the survey findings match your manual analysis?

2. Are interview analyses using the correct framework?

3. Are quotes properly extracted with timestamps?

4. Are markdown files properly formatted?

Iterate on the prompts and scripts until outputs are consistent with manual analysis.

This testing phase is critical. You’re essentially training the agent to think like you.

Step 5 - Schedule Automatic Runs

Once the agent works reliably, schedule it.

Options:

1. Run manually when new data arrives

2. Schedule weekly runs (for ongoing research programs)

3. Trigger runs when new files appear in the transcripts folder

The simplest approach: Run the agent whenever you complete a new batch of interviews.

Drop transcript files in the folder, run the script, get analysis reports in minutes.

The Results

In the demo, the agent analyzed:

• 1 survey dataset with 20+ responses

• 5 interview transcripts

Total runtime: Less than 2 minutes.
Total output: 6 markdown files with structured findings.

The analysis quality matched the manual workflow. But it happened automatically.

This is the power of AI agents for discovery. You define the workflow once, then scale it infinitely.

5. Advanced Techniques for Production-Grade Analysis

The basic workflows work. But here are the advanced techniques that separate good analysis from great analysis.

Technique 1 - Comparative Analysis

Don’t just analyze your own data. Compare it to competitive alternatives.

The advanced version of the survey prompt includes:

For users who mentioned considering alternatives:

1. What alternatives did they consider?
2. What features attracted them to those alternatives?
3. What kept them with our product instead?
4. What would cause them to switch?

Organize findings as a competitive feature matrix.

This produces a structured comparison of your product vs alternatives, based on what actual users say.

You can use this to inform product strategy and identify competitive gaps.

Technique 2 - Longitudinal Tracking

If you run regular research, track how findings change over time.

Create a master synthesis document that combines:

1. This month’s survey findings

2. Last month’s survey findings

3. Trend analysis (what’s changing)

Prompt:

Compare the survey results from [current month] to [previous month]:

1. Which themes are strengthening?
2. Which themes are weakening?
3. Are new patterns emerging?
4. What retention risks have increased?

Highlight any significant changes.

This helps you spot trends before they become problems.

Technique 3 - Sentiment Analysis

Beyond extracting what users said, analyze how they said it.

Advanced prompt:

For each interview, assess:

1. Overall sentiment (positive, neutral, negative)
2. Emotional intensity (calm, frustrated, excited)
3. Confidence level (certain, uncertain, conflicted)

Use this to prioritize which feedback matters most.

A user who’s calmly explaining a minor annoyance is different from a user who’s frustrated about a major blocker.

Sentiment analysis helps you prioritize.

Technique 4 - Hypothesis Testing

If you have specific hypotheses, test them against the data.

Hypothesis: “Users who integrate the app into their morning routine have higher retention.”

Prompt:

Analyze all interviews for evidence supporting or refuting this hypothesis:

"Users who integrate the app into their morning routine have higher retention."

For each interview:
- Does the user have a morning routine?
- Is the app part of it?
- What's their retention segment?

Calculate the correlation.

Claude can process your research data to validate or invalidate specific hypotheses.

This is similar to how you’d use product analytics to test quantitative hypotheses. But you’re doing it with qualitative data.

Technique 5 - Risk Scoring

Build a simple scoring system to prioritize users by retention risk.

Scoring framework:

Assign each user a retention risk score (0-10) based on:

- Number of value anchors (more = lower risk)
- Price sensitivity mentioned (yes = higher risk)
- Alternative consideration (yes = higher risk)
- Feature dependency (fewer dependencies = higher risk)

Output a ranked list from highest to lowest risk.

This helps you identify which users to focus retention efforts on.

Technique 6 - Feature Prioritization Matrix

Combine research findings with usage data to prioritize features.

Prompt:

Create a feature prioritization matrix:

Columns:
- Feature name
- Number of users who mentioned it
- User segment requesting it
- Retention impact (high/medium/low)
- Competitive gap (yes/no)
- Recommended priority (P0/P1/P2)

Sort by recommended priority.

Now you have a data-backed roadmap prioritization framework.

Final Words

Bad discovery happens when you treat it as a phase. Good discovery happens when you treat it as a habit.

The best PMs continuously gather evidence, analyze patterns, and adjust course.

AI makes this possible.

Start with the manual workflows Caitlin showed. Analyze one survey. Process one interview. Get comfortable with the prompts.

Then build it into your regular rhythm. Weekly surveys. Monthly interviews. Continuous synthesis.

You’ll have superpowers soon enough.

Where to Find Caitlin Sullivan

• LinkedIn

• Maven Course (use code AAKASHxMAVEN for 15% off)

Related Content

Newsletters:

• How to Do Product Discovery Right

• Advanced Techniques: Continuous Discovery

• Customer Interviews: Advanced Techniques

Podcasts:

• Teresa Torres’ Guide to AI Discovery

• Complete Course: AI Product Discovery

• Ultimate Guide to Knowing Your Users as a PM