Read more

Then I partnered with PostHog to analyze 4.2 million survey responses across nearly 6,000 in-app surveys. What we found upends most of what PMs think they know about surveys.

This Reddit post captures what most teams experience:

106 upvotes. 17 comments. Near unanimous agreement.

The problem was never surveys. The problem was how they ran them.

They send surveys. They collect data. Nothing changes. Users stop responding. The PM concludes “surveys don’t work” and goes back to guessing.

Meanwhile, every PM I know is obsessed with shipping AI features. Prototyping with Claude Code, building agents, vibe coding products in an afternoon.

Most of these AI features will fail. But the teams shipping them won’t know why until their retention curves tell them months later. By then the feature has shipped, the PR cycle is done, and the data is pointing at a problem they can’t diagnose.

Catching it early requires talking to users at the right moment. In-app surveys do that better than any other tool in a PM’s kit.

Today’s Post

Today’s post is a collaboration with Jina Yoon, Adam Bowker, and Cory Slater at PostHog. We’ve jointly put together the ultimate guide to in-app surveys:

1. The 5 Survey Mistakes Exposed by 4 Million Responses

2. Real Survey Teardowns from 6 Companies

3. The In-App Survey Playbook

4. Two Claude Skills You Can Download Today

5. How to Analyze Your Survey Data with AI

Subscribe now

1. The 5 Survey Mistakes Exposed by 4 Million Responses

Mistake 1 - You’re Not Surveying Users Who Are Leaving

We often assume users who are leaving are already upset. Why would they take time to give feedback?

Wrong. Exit surveys got the highest response rate (15.5%) of any survey intent type. Compare that to general satisfaction surveys at 8.4%.

PostHog gets an impressive 42% response rate on exit surveys for their Session Replay product. The feedback is specific enough to act on:

1. “I wish I could set it to sample only 1% of sessions.”

2. “It would be awesome to limit the total amount of recordings per month.”

3. “We’d definitely continue using if we were sure this is HIPAA and SOC2 compliant.”

Each of those is a product decision you can make tomorrow. Compare that to “Great product, love it!” from a satisfaction survey.

Reed Hastings writes about Netflix’s feedback culture in No Rules Rules: the ratio inside the company is 25% positive, 75% developmental. Negative feedback is always more actionable.

The fix: Always have an exit survey. If you’re only going to run one survey, make it an exit survey. Trigger it on the cancellation or downgrade flow. You’ll catch users at the moment they’ve made their decision, when the feedback is freshest. This is the highest-signal input you can get for your retention strategy.

Mistake 2 - Your Timing Is Wrong

Have you ever been hit with a “How can we improve?” popup right after visiting a website for the first time?

That’s URL-based targeting. Surveys triggered when a user lands on a page. Many teams use it because it’s easy to deploy.

Event-triggered surveys get 1.6x more responses than URL-based targeting (11.7% vs 8.8%). Users who get surveyed on page load are in a background tab, mid-task, or never actually see it.

This is what Sprig figured out when they rebuilt on event-driven architecture. Surveys fire based on what the user just did, not what URL they’re on. That single shift is what let them scale their infrastructure to over 10 billion API interactions per month.

The fix: Trigger your surveys right after the user does something relevant. After they complete a product tour. After they use a feature. After they hit a milestone. If you need a refresher on where those moments sit in the user journey, my onboarding and activation guides map them out.

And your first question’s format alone creates a 3.6x response rate gap. More on that below.

Mistake 3 - Your First Question Is Wrong

First impressions matter. Your survey’s first question is the whole ballgame.

Surveys that started with a single-choice question get 3.6x more responses than those that started with an open-ended one.

The full ranking:

1. Single choice: 15.6%

2. Multiple choice: 14.4%

3. Ratings: 8.8%

4. Link: 8.6%

5. Open-ended: 4.3%

This tracks with behavioral psychology. Cialdini’s commitment and consistency principle says small, low-cost actions create momentum. Same reason Yelp asks you to write a review before asking you to sign up.

The fix: Lead with single or multiple choice. Escalate to ratings. End with open-ended. Shallow end to deep end.

The format sequence is what the data validates. The specific questions are yours to customize.

Mistake 4 - Your Open-Ended Questions Are Too Open

Open-ended questions are low response rate by nature (4.3%). Typing takes more effort than clicking.

But you can do much better by asking them well.

The default “What can we do to improve our product?” in PostHog only gets 3.2%. Questions that connect to the user’s context get 6%. Nearly 2x from framing alone.

Generic questions fail because they ask the user to scan every feature, every frustration, every wish, then prioritize, then articulate. That’s a 30-second mental task. They close the popup.

Three better approaches (adapt to your product):

1. Anchor to what they just did: “What did you think about that template creation process?”

2. Use dead time: “Before we load your dashboard, what metrics matter most to you?”

3. Give something back: “Want to earn a certificate for completing this training? Tell us your email.”

Another option: the foot-in-the-door technique for recruiting interview candidates. Launch a single-choice question first. People who respond are psychologically primed to say yes to “Would you be open to a 15-minute call?” Your survey becomes your interview recruitment pipeline. One of the best ways to feed your continuous discovery practice.

Mistake 5 - Your PMF Survey Is Diluted

Sean Ellis coined the famous PMF question: “How would you feel if you could no longer use this product?” If 40%+ say very disappointed, you’ve hit product-market fit.

But PMF surveys only get a 13.1% response rate. Buffer’s team wrote about this on their open blog: you need 40-50 active user responses before the signal stabilizes. That means surveying 300-400 active users.

At a startup, you might not have 300 active users. You survey everyone, including yesterday’s signups and free trial lurkers. Their responses dilute your signal.

At a big company, you have plenty of responses but from a mix of power users, casual browsers, enterprise admins, and freemium tire-kickers. The aggregate describes nobody.

Superhuman’s fix (from Rahul Vohra’s First Round Review piece): add a second question: “What type of people do you think would most benefit from [product]?”

Happy users almost always answer by describing themselves. Those self-descriptions become your ICP. Superhuman went from 22% “very disappointed” to above the 40% threshold by narrowing their focus based on these responses.

The fix: Only trigger PMF surveys after activation milestones. If your aha moment is “collaborate on a board with two people” (Miro), only survey users who’ve crossed that threshold. Users who haven’t activated will say “somewhat disappointed” and that’s noise.

And contextualize with the self-description question. Your PMF score and your ICP definition come from the same survey.

2. Real Survey Teardowns from 6 Companies

The 5 mistakes tell you what not to do. The 6 companies below show what running this well looks like end to end. Six different survey types, six different decisions they drove.

Superhuman’s PMF Segmentation Engine

Source: Rahul Vohra, First Round Review

Section 1 covered the segmentation move. Here’s what Section 1 didn’t cover: the fourth question.

Superhuman’s survey had four questions, not three. The fourth was “How can we improve?” and it’s where the roadmap came from.

“Very disappointed” users told them what was already working. “Somewhat disappointed” users who valued speed told them what was missing. The top request from that second group was a mobile app. Superhuman built it.

That’s the dual job of the Superhuman survey. The first three questions tell you who to build for. The fourth tells you what to build next. The survey told them exactly what to build and exactly what to ignore.

Slack’s 3-Question Onboarding Personalization

Source: Appcues evolution teardown, Userpilot teardown

Slack asks three survey questions during workspace creation: team name, what you’ll use Slack for, and invite teammates. The answer to “what will you use Slack for” gets reflected back immediately in the workspace setup.

The user’s answer is immediately reflected in the product. You see your own words in the channel name. Onboarding feels tailored from minute one. And Slack gets free segmentation data: they now know what problems each workspace is trying to solve.

Slack’s onboarding has changed at least 3 times since 2014. Each version reduced friction. But the 3-question survey persisted across every iteration. That tells you something.

PostHog’s Slack Response Pipeline

Every survey response at PostHog pipes into a dedicated Slack channel instantly. Someone responds within minutes. Not automated. Human.

Their numbers: 42% exit survey response rate on Session Replay. Users share detailed, actionable feedback because they know someone is reading.

Teams that treat survey responses as the start of a conversation keep getting responses. Teams that don’t watch response rates decay. (If you want more data like this from the PostHog team, their newsletter is here.)

Three patterns connect Superhuman, Slack, and PostHog. Context beats generic timing. Surveys feed product decisions, not spreadsheets. And closing the loop with human follow-up is the difference between 4% and 42% response rates. Three more companies show different angles on the same principles.

🔒 The full guide continues for paid subscribers.

Below: The complete In-App Survey Playbook with four survey templates, conditional branching flows, and 12 bad/good/awesome mockups showing exactly what to ship for every survey type. Two downloadable Claude Skills (a Survey Analyzer that scores your draft survey and rewrites weak questions, and a Survey Generator that builds complete surveys from a goal). A 30-minute AI analysis workflow that turns a CSV export into a 1-page action brief, replacing 10+ hours of manual synthesis. Three more teardowns (Amplitude, Miro, Sprig). And a printable poster with the complete framework.

Read more

The first is personal: what you need to know before every decision and every meeting. Competitor moves, user feedback, stakeholder context. When this breaks, you’re the last person in the room to know things you’re supposed to know first.

The second is your team’s: the systems that serve your users, surface …

Read more

Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by:

1. Maven - Get discounts off Mahesh's courses with my links

2. Amplitude - The market leader in product analytics

3. Jira Product Discovery - Prioritize what matters with confidence

4. NayaOne - Airgapped cloud-agnostic sandbox to validate AI tools faster

5. Product Faculty - Get $550 off their #1 AI PM Certification with my link

Today’s episode

LinkedIn just scrapped its APM program and replaced it with an Associate Product Builder track, and introduced a Full Stack Builder career ladder alongside it.

What does it even mean to be a “builder PM”?

Well, tools only get you so far. Learning Claude Code is helpful, but means nothing if you don’t have an understanding of the underlying first principles.

That’s today’s episode.

Mahesh Yadav created one of our most popular episodes, with over 35K views on YouTube, and now he’s back. Earlier, he taught you AI agents.

Today, he’s teaching you how to become a builder PM:

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

I’m giving a free talk on how to get interviews at the top AI PM companies on Thursday (April 23rd) @ 9:00AM PDT. Grab your seat.

Newsletter deep dive

Thank you for having me in your inbox. Here’s the complete guide to “becoming a builder PM.”

1. What is a “Builder PM”

2. The Builder PM Tool Stack

   • n8n

   • Claude Code

   • OpenClaw

3. Mastering AI Agents

   • The first principles of agents

   • How to build self-improving agents

4. The 10-week roadmap to Builder PM

Subscribe now

1. What is a Builder PM

Every PM has already used Claude Code or ChatGPT to get something done. But if that’s where it stops, you are not yet a builder PM.

There are two kinds of builder PM, and they’re different jobs.

Type one ships customer-facing product without engineering handoffs. Customer to 10 paying users, solo. This is what the LinkedIn APB program is training for and what lands you the comp trajectory Mahesh describes.

Type two builds internal agents that automate their own PM work. PRD reviewers, competitive intel, data dashboards. Same skills, different trajectory: stay in your current org, ship 3x more, get promoted faster.

Most of this guide teaches type two. Type one is the aspirational endpoint.

Why this title change happened now

The title change is a response to something that actually shifted in the last six months.

Six months ago, most autonomous agents broke within minutes. METR’s latest benchmarks show frontier models now sustaining multi-hour jobs, with Opus 4.7 running coding tasks for 3 to 6 hours in practice.

When agents can only run for 3 minutes, the PM’s job is to prompt them. When agents can run for 6 hours, the PM’s job is to design the system those agents run inside.

Designing that system is what builder PM actually means.

The compensation trajectory, in case you need the motivation

Mahesh’s comp trajectory shows you the top end of what’s possible. He went from $120K at Microsoft to $1.3-1.4M in his last Google role (Senior PM, AI), with a rough doubling every two years. A few of his AI PM friends at Nvidia are at $2-2.5M.

Levels.fyi still pegs median Senior PM comp at top tech around $350-450K. What Mahesh is describing is the premium the US tech market pays specifically for builder-skill AI PMs, where companies approach you and offer 30-40% over your current number. The $2M+ tier is a small group.

International markets pay differently - builder skills still move comp in EU, India, and APAC, just with different absolute numbers and different negotiation dynamics. And if you’re in mission-driven PM work where comp isn’t the yardstick, the same builder skills still let you ship 3x more impact with the same team - which is the whole point.

You probably won’t hit $1.4M next year. The premium exists, though, it’s real, and the gate is builder skills.

One honest caveat for managers reading this. If your PMs get 10x leveraged, the obvious question is whether you need fewer of them. The early signal from LinkedIn’s restructure is that the answer is smaller pods with broader scope, not layoffs - they’re keeping headcount and pushing each builder to own more surface area. That may or may not hold elsewhere. Worth naming, because every VP Product on this list is already thinking about it.

The failure mode

Here’s the trap: PMs read about Claude Code on Twitter, try it for a weekend, get a mediocre output, and conclude “this is overhyped.”

They skipped the first principles.

That’s the answer to Gothelf. He’s right that execution capacity was never the bottleneck. Judgment was. Judgment sharpens when you ship 12 prototypes a year instead of 1. The builder PM still talks to customers. They just ship without waiting.

That’s what the rest of this guide is going to give you.

When building is the wrong bet

Not every PM job benefits from this shift. A few honest cases where the 10-week investment doesn’t pay off:

• If you work in trust & safety, healthcare decisions, financial underwriting, or legal compliance, your job is to prevent errors, not to generate throughput. An agent that ships 12 prototypes a year doesn’t help you. An agent that proposes a new UI flow in a payments product can actively hurt you. Build evals skills instead.

• If your product lives inside a regulated perimeter and your company has a year-long procurement cycle for new tools, you’ll spend your 10 weeks waiting for IT approval. Spend the time on AI literacy - reading papers, running evals on existing outputs - rather than on personal tool setup.

• If you’re a founder with zero PMs under you, skip the “automate your PRD reviews” framing entirely. Your version is shipping real product to real customers using these same tools. The skills transfer. The PM-productivity examples don’t.

Builder PM is the right bet for most mid-career PMs at product-led companies with data portability. That’s a large slice of the market, not all of it.

2. The Builder PM Tool Stack

Three tools. Three different jobs. Using the wrong one for the wrong stage of your journey will waste weeks.

Pick based on where you are in the journey, ignoring whichever tool has the most Twitter hype this week.

n8n

n8n is a visual workflow builder. You drag components onto a canvas and connect them. That’s it.

Everyone who is deep in AI dismisses n8n as “too basic.” They are wrong. n8n is irreplaceable as a learning tool because you physically see every piece of the agent architecture as separate nodes.

In the episode, Mahesh demoed building a contract analyzer in n8n:

1. Email trigger hits when someone sends a contract

2. 2. Gmail node pulls the MSA (master service agreement)

3. Data loader converts the file to text

4. Text splitter chunks it into 1,000-character pieces with 200-character overlap

5. Embedding model converts chunks to vectors

6. Vector database stores them

7. AI agent reads the playbook and flags risks

8. Gmail node emails back the analysis

9. Eval workflow runs against ground truth

Every one of those is a visible node on a canvas. When something breaks, you see exactly where.

The limitation is real. n8n has no code mode, no version control, no test suites, and no path to production beyond a simple webhook. It stops you around 10 customers.

For most builder PMs, use n8n for 2-3 weeks, then move on. The exception is simple internal workflows with clear webhooks and no collaboration needs - those can live in n8n forever and that’s fine. The “graduate from n8n” rule is about your learning trajectory, not about n8n being bad at its job.

Claude Code

Claude Code is where you build the real thing.

The same tool works for a PM with zero coding experience building skills in English, and for a senior engineer building production services on top of that PM’s work.

Mahesh showed his PRD review setup in the episode. Here is what it does:

1. He drops a PRD into a folder

2. Claude Code reads his checklist (ruthlessly specific, encodes Amazon PR/FAQ format, includes AI-specific criteria like “is this differentiated from ChatGPT or is it a commodity AI wrapper”)

3. The agent reads the document and writes inline comments back into the .docx file, using a Python library to handle the XML under the hood

4. Comments are strategic, not surface level. Things like “What prevents Datadog or the Big Four from building this?” and “How do you handle misclassification?”

That is already useful. But here is where it gets serious.

Every 30 minutes, a second agent runs. It opens the folder of recent reviews, compares the AI’s output against the version Mahesh actually shipped, and logs the deltas to a learner.md file.

When the same correction appears 5 times across 5 days, the learner emails Mahesh: “I want to update your checklist. Here is the proposed update.”

He reviews, approves, and the checklist evolves.

Every day, the reviewer is a little better than the day before. That is the real moat. Not the initial skill. The learning loop that compounds from your judgment. We’ll go deeper on this in Section 3.

The other things to build in Claude Code:

• Competitive intelligence with subagents (3 subagents researching 3 competitors in parallel, rolling up into one report)

• Prototypes from PRDs (clone competitor screens, modify them into working products, ship to customers for feedback in days instead of months)

• Data dashboards generated from your actual production data

Use Claude Code as your daily driver for weeks 4-6 of the roadmap. By the end, you should have at least 2-3 skills running on your real work, with learner loops on top of each.

OpenClaw

OpenClaw is a pattern. The pattern matters because enterprise agents will probably work this way at scale. GCP and AWS are already shipping sandboxed-agent primitives, and the architecture maps directly.

Here is what OpenClaw adds that Claude Code doesn’t have:

Delegation through existing channels. You send the agent a WhatsApp message. It goes and does the work on a separate machine. When it is done, it sends the result back through WhatsApp. You are not sitting in a terminal watching it think.

Full machine sandboxing. Instead of granting file-by-file permissions, you install the agent on a dedicated Mac mini or a VM. It controls the entire compute environment. (Side note: Mac minis with 32GB+ are 10-18 weeks out right now, and several high-memory configs are unavailable entirely. CNN reported Apple Store employees calling them “OpenClaw machines.”)

Model agnosticism. Connect any model, including open source. You are not locked into Anthropic’s rate limits, which every heavy Claude Code user complains about daily.

For PMs at big companies asking the obvious question: Google is not going to let you install OpenClaw on your work laptop. That is true. But Google is going to offer the same pattern inside GCP. AWS is going to offer it inside their infrastructure. The architecture of sending a message to a sandboxed agent that reproduces your problem, tries a solution, and returns results - that is how enterprise agents are going to work at scale.

If you’re at a regulated company (finance, health, government, legal) or your data can’t leave the corporate perimeter, the pattern still applies but the venue changes. Your version is Claude Code or an internal agent platform running inside your SSO and VPN. NayaOne-style airgapped sandboxes exist for exactly this reason. The architecture is the same. The compliance envelope is different. Build what you can inside your perimeter and push your infra team for the pattern on the outside.

Learn the OpenClaw pattern now and you’ll recognize the shape of enterprise agent platforms when they arrive.

3. Mastering AI Agents

Tools without first principles is how PMs waste 3 weeks. Here are the first principles.

The first principles of agents

Every working agent has 4 components. Every disappointing agent is missing at least one. Before you debug a prompt, debug the architecture.

Component 1 - Intelligence (The brain)

The model is the intelligence layer. My current default: Opus 4.7 for agent orchestration and skill authoring, GPT-5.4 for fast single-turn tasks where latency matters. You can run either. The rest of the architecture matters more than this choice.

On its own, it is a genius with amnesia. Ask it about neural networks, it answers perfectly. Ask it what Trump said about Iran this week, and if you picked a cheap model, it tells you its knowledge cutoff is June 2024.

The intelligence layer alone gives you knowledge cutoffs, generic answers, and zero session memory. A model without the other three components is just a more expensive way to use Google.

Component 2 - Tools (The hands)

Tools let the agent take actions. Without them, it can think but it cannot do anything.

In the demo, Mahesh added Tavily (a search tool) to the agent. Same question about Iran. This time, the agent searched, found the answer, and responded with current information. One tool turned it from useless to useful.

The essential tools for PM agents are search APIs (Tavily, Perplexity) for current information, file processors to read contracts and PRDs, MCP servers to connect to Gmail, Slack, GitHub, and your CRM, and bash execution to run scripts. PMs who skip this layer wonder why their agent cannot do anything beyond trivia.

Component 3 - Memory (The recall)

Without memory, the agent forgets everything between messages.

The demo showed this failure instantly. The agent searched and answered the Iran question. Two messages later, Mahesh asked “what conflict am I talking about?” and the agent replied that it saw no previous mention of a conflict. Adding session memory fixed it in one click.

Memory has three layers: session memory for the current conversation, long-term memory for patterns across days and weeks, and structured memory for specific facts the agent needs to retrieve reliably. Every PM agent that feels “dumb” is almost certainly missing one of these.

Component 4 - Knowledge (Your data)

This is where your agent stops being a generic chatbot and starts being valuable.

Knowledge = your company-specific context. Contracts, playbooks, competitive intel, product specs, customer research. Without it, the agent answers from the internet. With it, the agent answers from your actual data.

The RAG pipeline is straightforward: upload your documents, chunk them into 1,000-character pieces with 200-character overlap, convert to vectors, store in a database, and query when asked a question.

In the demo, Mahesh asked about payment terms and tariff impacts on company contracts. Without knowledge, the agent gave generic legal advice from the internet. With the MSA uploaded, the agent answered from the actual contract clauses.

That difference is the entire value proposition of building your own agents vs. using a generic chatbot.

How to build self-improving agents

This is the section almost nobody is doing yet. Which is exactly why it matters.

You automated one task. Your PRD review agent works. Your competitive analysis runs every morning. Then you notice the agent makes the same mistake three days in a row. It misses the same section. It applies the wrong standard.

Static skills produce static output forever. Learning systems compound daily.

Here is the learner loop architecture Mahesh runs on his PRD reviewer:

Step 1 - Every review creates a job folder

Timestamped. Contains:

• Who ran the review

• The input document

• The AI’s output

• The version you actually shipped after your corrections

Step 2 - A scheduled subagent runs every 30 minutes

It opens recent job folders. Compares the AI’s output against your modified version.

Step 3 - Differences get logged to learner.md

What the AI got right. What it missed. What you added. What you deleted.

Step 4 - A threshold triggers a proposal

When the same correction appears 5 times over 5 days, the learner proposes a specific checklist update: an actual edited version of the checklist line.

Step 5 - You review and approve

The new checklist becomes the input for the next review cycle.

The critical design decision is that the learner never updates the checklist automatically. It accumulates evidence, it proposes, you approve. You stay in the loop. The system still compounds.

After a month, your review agent catches things it missed on day one.

After three months, your checklist has evolved from your original draft into something shaped by hundreds of real reviews and your actual judgment calls.

Inside a team, you’ll want one canonical version of each skill in a shared repo, with a light review process when someone proposes a merge. Otherwise you get skill drift, five competing PRD reviewers, and nobody knows which one is current.

Static skills are where most PMs stop. The learning systems that compound from your judgment are what separate the builder PM who gets promoted from the builder PM who watched a YouTube video.

4. The 10-week roadmap to Builder PM

You have a full-time job and maybe 5-8 hours a week to learn this. Here is the sequencing, tested with real PMs in real cohorts:

• Weeks 1-3, Foundations. n8n only. Build one agent with all 4 components, run one real evaluation, build one multi-agent system. By week 3 you can explain the 4 components to a junior PM.

• Weeks 4-6, Build. Move to Claude Code. Pick one real weekly task and automate it with a skill. Add a learner loop on top. Spin up your first subagent system (3 competitors in parallel works well).

• Weeks 7-9, Delegate. OpenClaw time. Pick one complete job (not a subtask) and delegate it via WhatsApp to an agent on a VM. Iterate on the skill until the output is good enough to forward to your team unchanged.

• Weeks 9-10, See the pattern. Read the AI tool market through this lens. Every new tool: is it solving for context, actions, or evals? Is it a variant of the agentic loop? Could you build it yourself?

The biggest failure mode is skipping Phase 1. PMs who jump straight to Claude Code hit a wall around week 3, have no mental model for debugging, and conclude “Claude Code is overhyped.” Do not be that PM.

That decomposition skill from Phase 4 is also what’s showing up in L5 and L6 AI PM interviews. Based on reports from candidates I’ve coached through Land a PM Job and notes from Blind, some loops now include live building exercises where you’re expected to open Claude Code and produce a working prototype.

Candidates who default to Figma mocks when the interviewer is looking for a working prototype have lost offers. Figma is still the right tool for design work. The signal the interviewer is looking for is whether you can ship.

For hiring managers

In 6 months every PM resume will claim builder skills. Two questions cut through it:

1. “Walk me through a skill you’ve built and how the checklist evolved.” Real builders have version history. Fakes have a one-liner.

2. “Show me the learner.md.” Real builders have a file of failures. Fakes don’t.

If a candidate says they’ve used Claude Code but can’t name the four components of an agent or explain when they’d choose n8n over Claude Code, they watched a YouTube video. That’s different from building.

[Bonus] Episode Summary

Ten weeks. Three phases of building. One phase of seeing.

At the end, you are a PM who builds AI systems that get better every day.

In 2026, that is the difference between being automated and being the one doing the automating.

Get Transcript

Where to find Mahesh Yadav

• LinkedIn

• Maven Course

Related content

Podcasts:

1. Claude Code Team OS with Carl Vellotti

2. OpenClaw + Claude Code with Naman Pandey

3. Claude Code OS with Dave Killeen

Newsletters:

1. The complete context engineering guide

2. How to use Claude Code like a pro

3. Practical AI agents for PMs

PS1. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

PS2. The third cohort of my LandPMJob Program is filling up. Apply now!

Share

Aman Goyal just went through an AI PM job search successfully. You know what question he didn’t get? Product design.

Design a pencil for the blind.
How would you create a dating app for Facebook?

These questions used to be the bread and butter of PM interviews. He didn’t get asked a single one. Instead, he did face one round several times: AI system design.

I’ve covered AI product sense, product design, success metrics, and behavioral interviews. Today, I cover AI system design.

Why Now

Companies want to know if you can architect an intelligent system end to end. Not wireframes. Not feature lists. Data pipelines. Model selection trade-offs. Orchestration layers. Agent architectures. Failure modes.

And the stakes are absurd.

At OpenAI, stock-based compensation averages $1.5 million per employee per year. Google and Meta are competing with equally aggressive equity packages for AI talent. Senior AI PM roles at these companies pay $500K-$800K+ in total compensation, with staff-level and above clearing $1M when equity is included.

But here’s the gap that blows my mind. There are hundreds of guides on product sense interviews. Same for system design. But AI system design for PMs? Zero resources exist.

That’s wild. Because many of the top AI companies are now asking it.

So I built the guide myself.

Today’s Post

This is the first and only comprehensive PM AI system design interview guide on the internet:

1. What they evaluate

2. The DASME framework

3. Practice AI tools + mock takeaways

4. Company-by-company breakdowns

5. Anti-patterns to avoid and how to fix them

6. 64-question practice bank + 2 worked examples

Subscribe now

Want live coaching through the AI PM job search? Apply for my Land PM Job cohort. Next Cohort starts May 4. Both of the prior cohorts sold out, so secure your seat.

I’m giving a free webinar on getting AI PM interviews next week. Join us.

1. What They Evaluate

This is not an engineering system design interview.

You are not being compared against software engineers. You don’t need to whiteboard load balancers or discuss database sharding algorithms.

But you absolutely need to go deeper than a standard product design answer.

Technical Fluency - 30-40% of the score

Can you speak intelligently about model selection? Do you know when to use an LLM versus a traditional ML model? Can you articulate why an XGBoost classifier might be better than GPT-4 for churn prediction?

This is the dimension where most PM candidates fail. Not because they need to write code. But because they freeze when the interviewer asks “are you going to use an LLM or an ML model for this?”

One candidate I coached told me the interviewer asked about F1 scores. She said she’d have to check. Interview was over in their minds.

System Architecture Thinking - 25-30%

Can you draw a coherent diagram showing how data flows through the system? Can you identify the distinct components and explain how they connect?

Product Judgment Within Technical Constraints - 20-25%

Your user segmentation and pain point analysis still matter. But they need to serve the system design, not replace it.

Trade-off Articulation - 10-15%

Can you proactively surface trade-offs without being asked? Latency versus accuracy. Cloud versus on-premise. LLM versus ML model. The best candidates name these trade-offs before the interviewer has to prompt them.

How it differs from other cases

In a product design or product sense interview, you focus on users, pain points, solutions, and success metrics. The deliverable is a product concept.

In an AI system design interview, you still start with users and pain points. But the deliverable is an architecture diagram showing data flows, model choices, agent roles, orchestration patterns, and failure handling. You spend 60% of your time on the technical system.

The candidates who get rejected treat it like a product design question. They spend 30 minutes on user personas and pain points, then sketch a vague “AI layer” at the end.

The ratio should be flipped. 30-40% product framing. 60-70% system architecture.

Quote me on this: “If you spend more time on personas than on your system diagram, you will not pass this round.”

🔒 The rest of this post is for paid subscribers only. You’ll get -

1. The DASME framework with architecture diagrams, model selection table, and exact time allocations

2. 3 practice tools I built to help you prepare (Claude Skill, Custom GPT, Gemini Gem)

3. How 5 top AI companies ask this question differently

4. 7 anti-patterns with exact fixes

5. 64 practice questions with category guidance and 2 full worked examples (search/retrieval and content moderation)

Read more

Three bullets. Three shifts. All already happening at one of the largest product orgs in tech.

And he’s not alone.

Garry Tan, CEO of Y Combinator, open-sourced his entire Claude Code setup (gstack). It hit 33K GitHub stars in a week (now past 65K). He’s personally shipping 100 PRs/week while running YC.

Boris Cherny, creator of Claude Code at Anthropic, ships 20-30 PRs a day. 100% AI-written. He uninstalled his IDE.

Jiaona Zhang, CPO at Laurel, Stanford lecturer, ex-Airbnb: her PMs are “going beyond prototyping to production.” Her designers push code too.

OpenAI’s Codex team shipped an entire product with 1,500 merged PRs and zero manually written code. Superhuman uses Codex so PMs can “contribute lightweight code changes without pulling in an engineer, except for code review.”

A startup CTO posted yesterday: “Last week, one of our PMs built and shipped a feature. Not spec’d it. Built it, tested it, shipped it to production. In a day.”

These aren’t engineers. They’re PMs, CPOs, and CEOs.

This piece is the step-by-step manual for joining them.

Today’s Post

1. The New Map: PM vs Design vs Eng (start here if this is new to you)

2. Planning in Git: The Open-Source Planning System

3. Your First Pull Request and How to Earn Eng Trust (start here if you’re ready to ship)

4. What PMs Ship and How to Test It (start here if you’re already pushing PRs)

5. Real-Time Customer Issue Monitoring

6. Real-World Teardown

Plus 4 downloadable tools: The Open-Source PM Planning System (GitHub repo), The PM PR Skill File, The “Make the Case” Pitch Doc (with enterprise variant), and The Customer Issue Monitoring Playbook.

Subscribe now

1. The New Map: PM vs Design vs Engineering

Every PM reading this is asking the same question: Do I need to become an engineer?

No. The boundaries shifted. The roles didn’t merge.

But here’s the part most people miss. Shipping code makes you a better PM.

When you can test a copy change in an afternoon instead of theorizing about it for a quarter, your strategy gets sharper. Your user research gets faster feedback. Your product sense sharpens because you see the consequences of your decisions in hours instead of months.

Nothing in this piece replaces strategy, user research, or stakeholder alignment. Those skills matter more than ever because the cost of building the wrong thing just dropped to near zero. The PM who ships has tighter feedback loops on everything. Shipping is additive. It amplifies every other PM skill.

Here’s how the map actually changed.

The Old Map

• PM: specs, tickets, prioritization

• Design: mockups, prototypes, handoffs

• Engineering: all code, all deploys, all monitoring

Thick walls between each role. PM couldn’t touch code. Engineer waited for the spec. Designer handed off a Figma file and hoped for the best.

The New Map

• PM zone expanded into: copy, config, AI prompts, planning docs in git, small front-end changes, production monitoring. The stuff where the PM has the most context and an engineer was just translating PM intent.

• Design zone expanded into: coded prototypes (v0, Cursor), design system changes shipped directly, visual QA against real user sessions.

• Eng core concentrated on: architecture, infrastructure, security, complex logic, performance, code review. The hard problems. The things that actually need senior engineering judgment.

This makes engineers happier. The Zenflow CTO who posted about his PM shipping code wrote:

“When PMs build their own ideas, their specifications get sharper, because they now understand what the agent needs to execute well. Sharper specs produce better agent output.”

The engineer’s life got better because the PM got closer to the work.

At Anthropic, Cat Wu (head of product) told TechCrunch that code output per engineer grew 200%. The bottleneck shifted from writing code to reviewing it. That’s engineers telling you they need help with the volume. PMs shipping the small stuff is part of how you solve that.

If you’re at a company where this hasn’t started yet: Later in this piece (paid), there’s a pitch doc you can hand your eng director — with a variant for regulated environments.

2. Planning in Git: The Open-Source Planning System

MacInnis said it plainly:

“No more planning decks, only markdown pushed to a git repo.”

The Old Way

PM writes a Google Doc or Notion page. Shares the link. Comments scatter across three tools. Review meeting happens. Engineers can’t find the spec during implementation because it’s buried in someone’s Drive folder. The doc lives in a silo, disconnected from the code.

The New Way

PM writes a markdown file. Pushes it to the same repo where the code lives. Engineers and Claude Code both reference it directly. When you want to know why something shipped, git log shows you. Version control gives you diffs, history, and accountability.

The content is the same as a good PRD. The location changed. And the location matters because it puts the spec where the builder — human or AI — can actually use it.

What a PLANNING.md Looks Like

Here’s a preview. The full worked examples are in the downloadable repo.

# PLANNING: Smart Notification Batching

## Problem
Users receive 12+ notifications/day. "Notification fatigue" tickets up 34% last
quarter. 8% of users muted all notifications.

## Hypothesis
Batching low-priority notifications into a daily digest will reduce mute rates
by 15%+ while maintaining engagement with high-priority alerts.

## Success Metrics
- Primary: notification mute rate drops ≥15% vs control
- Guardrail: high-priority CTR doesn't drop >2%
- Guardrail: DAU doesn't decrease >1%

## Rollout
- 10% of users, 2 weeks, user-level randomization
- Kill if mute rate doesn't improve ≥5% after 2 weeks

That’s it. That replaced a 15-page Google Doc and a 30-minute review meeting.

CLAUDE.md: Your Brief to the AI

CLAUDE.md is a persistent instruction file in the project root. Claude Code reads it at the start of every session. It encodes: product context, coding standards for PM-scoped changes, review expectations, and what the PM should and shouldn’t touch. You write it once and update as the product evolves.

Your PM OS is the deeper version of this. For today, what matters is: if you’re going to ship code as a PM, CLAUDE.md is where you encode your judgment so the AI follows it every time.

You Can Start Today Without a Terminal

Everything in this section works from GitHub’s web interface. Create a repo. Click “Add file.” Write your PLANNING.md in the browser. Commit it. You just shipped a spec to git. No terminal. No Claude Code.

If you’re preparing for PM interviews, fork the repo below and write a PLANNING.md for a case study - it’s a better portfolio artifact than a PDF.

Downloadable #1: The Open-Source PM Planning System

A public GitHub repo you can fork. Designed for product directors to deploy across their team and for individual PMs to use solo.

Get the Planning System

What’s inside:

• PLANNING-TEMPLATE.md — fill-in template with every section

• Two worked examples — one for a non-AI feature, one for an AI feature with behavior contract

• CLAUDE.md — PM-specific Claude Code config with product context sections, review checklist, and PM-zone boundaries

• Team rollout playbook — how a director deploys this system, measures it, and iterates

• Pilot measurement template — metrics for the first sprint pilot

• Weekly review cadence doc — how to run the weekly planning review

• Planning review skill — a Claude Code skill that reviews your doc and flags gaps before you share it

The README addresses product directors: “You’re deploying a planning system for your team. Here’s how to set it up, measure it, and iterate.” When they join a new company, they fork this in week one.

🔒 The rest of this post is for paid subscribers.

Next: the skill file that auto-reviews your PRs before engineers see them (so your first merge doesn’t embarrass you). A copy slop audit where one PM changed two words on a CTA and got a 14% lift in trial signups. Step-by-step monitoring that catches production issues in minutes, not weeks. And a full teardown showing how PM #2 shipped the same feature 8.5 weeks faster than PM #1, using this exact workflow.

Plus 3 more downloadable tools, including the pitch doc that gets your eng director to approve the pilot.

Read more

Read more

Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by:

1. Bolt: Ship AI-powered products 10x faster

2. Amplitude: The market-leader in product analytics

3. Pendo: The #1 software experience management platform

4. NayaOne: Airgapped cloud-agnostic sandbox

5. Product Faculty: Get $550 off their #1 AI PM Certification with my link

Today’s episode

Sometimes, I get access to the wildest guests on this podcast. Today, we get the awesome opportunity to look inside the design processes at OpenAI and Figma:

• Ed Bayes leads design on Codex at OpenAI

• And Gui Seiz leads all AI design at Figma

And they worked with me to put together a masterclass on how to design in the AI era.

If you want to design like the leading AI companies, this episode is for you: complete with screen shares and everything else you need to adopt the new AI design workflow.

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

I’m accepting applications for my third LandPMJob cohort. Join Me.

Newsletter deep dive

As a thank you for having me in your inbox, here is the complete guide to the new code-plus-canvas design workflow:

1. Why the linear design pipeline is dead

2. The code-canvas loop

   • Codex to Figma

   • Figma to Codex

3. When to use which tool

4. The 5-step adoption roadmap

5. Total football for product teams

Subscribe now

1. Why the linear design pipeline is dead

You know the process. Everyone does. It was unchanged for 15 years:

Paper sketches. Wireframes. High-fidelity mockups. Developer handoff. Engineering builds it. Design files bug tickets because the spacing is off by 4 pixels.

Every single stage in that pipeline existed for one reason: the next stage was too expensive to start with.

Constraint 1 - High fidelity required engineering

Building something interactive used to require an engineer. So designers front-loaded cheap artifacts. Wireframes exist because prototypes cost too much. Handoff specs exist because designers could not write code.

Codex changed this. A designer can now build a functional prototype in minutes. No engineer required. No sprint ticket. No two-week wait.

Constraint 2 - Design files and code were separate worlds

A designer worked in Figma. An engineer worked in VS Code. Getting them in sync required handoff documents. Redline specs. Manual transcription of padding values and border radii.

The Figma MCP changed this. Design files and codebases now read from each other directly. One click. Pixel-perfect. No translation layer.

Constraint 3 - Feedback required the finished product

You could not test interactions in a wireframe. You could not feel tap targets in a mockup. You had to wait until engineering built it. Then you filed bugs. Then engineering fixed them. Then you filed more bugs.

Now you test interactions in code from day one. Before a single engineer touches it. The feedback loop collapsed from weeks to minutes.

What this looks like in practice

1. Inside OpenAI. The Codex design team has been shipping this way since December. Something changed when the models hit a capability threshold. Content designers are submitting PRs. PMs bring working prototypes to design reviews.

2. Inside Figma. Designers are working directly in staging. Monetization designers who never wrote code before are building technically complex prototypes. The phrase I keep hearing is there are no more P2s. The cutoff between ship and backlog is arbitrary now.

I covered the foundations of this shift in my AI prototyping for PMs guide. The difference now is that the tools on both sides have converged.

The old pipeline was not a design process. It was a workaround for expensive tools. The tools got cheap. The pipeline died.

2. The code-canvas loop

The new workflow is code + canvas in a loop.

The most common question about this workflow. Will it be lossy? Will things break when you translate between tools?

Short answer: it is already remarkably high fidelity. And it only gets better with every model improvement.

Direction 1 - Codex to Figma

You have a React app running locally. You have been building a composer system in code. You need to feel how buttons morph. How permissions prompts expand. Where tap targets land. Static mockups cannot answer these questions.

Now you want to go deep on the visual layer. Get pixel-perfect on a component. Swap icons. Test type scales.

Open the Codex desktop app. Drag the pop-out window over your running app. Type this -

Import my homepage and composer permissions state into a new Figma file. @figma

Three things happen -

1. Reads your file system. Codex understands the React structure, components, CSS

2. Snapshots the running interface. The Figma MCP captures your live app state

3. Opens it in Figma. Click Open in Figma. You get a real, editable design file

The result is not a screenshot. Everything is responsive.

1. Every padding value matches your CSS exactly

2. Every border radius is one-to-one

3. Every shadow transfers without manual transcription

4. You can select specific component nodes instead of full screens

The newest MCP improvement. The use Figma tool now references your actual design library. Components map to your design tokens and local styles. You are not working on a facsimile. You are working on a representation built from your real design system.

The snapshot is not a copy. It is a living bridge between your code and your canvas.

Direction 2 - Figma to Codex

The reverse direction is just as fluid.

You have been iterating in Figma. Changed the model name in the picker. Adjusted a color. Refined spacing on a card.

Copy a link to the modified component. Paste it into Codex -

Update my code with the change I made here [Figma component link]

Codex reads the link through the MCP. Diffs against your local codebase. Generates the code change.

For non-coding designers this is the unlock. You are no longer blocked. Make changes in Figma. Your engineer pastes the link into Codex. The change propagates automatically.

One engineer recently demonstrated aligning Storybook to GitHub to Figma with AI running the loop and finding diffs between them. The maintenance overhead of keeping design systems in sync, the thing every team complains about, can now be automated.

Three foundations that make this work

1. Name components well. Good naming is table stakes for humans. Essential for agents

2. Align design tokens with CSS tokens. Border radius, spacing scale, color palette. One-to-one mapping

3. Use annotations. Read annotations into the MCP so intent travels alongside the visual output

The handoff is dead. The loop replaced it.

Where AI still falls short

AI isn’t still where it needs to be for lossless movement. These areas are limited:

1. Shader effects do not translate to a static canvas

2. Complex CSS transitions cannot be fully represented in Figma yet

3. Edge case decisions that only exist in code because an engineer solved something that never got recorded in the design file

4. Web-specific effects that Figma’s canvas does not yet support

Annotations help bridge some of these gaps. But the designer’s judgment is still critical for the last mile.

The good news. Every model improvement makes this better. OpenAI’s model 5.4 produced a material jump in quality. Internal designers say it is meaningfully better at working with the Figma MCP than anything before.

If you tried this six months ago and gave up, try again. The reliability has crossed into daily-use territory.

The tools have limits. But those limits are shrinking every month instead of staying fixed.

3. When to use which tool

The biggest mistake teams make with this workflow. Trying to do everything in code. Or everything in canvas.

Match the tool to the question you are trying to answer.

Mode 1 - Go wide on ideas

Use the canvas.

You can see the whole flow in front of you. Rearrange screens. Explore lateral directions. Print it out and put it on a wall and tear it down.

The canvas is still the gold standard for divergent exploration. No code tool matches the spatial freedom of dragging artboards around and seeing twenty concepts at once.

When to pick this -

1. You are exploring completely new interaction paradigms

2. You need multiplayer collaboration with the whole team

3. You want to rally people around a hero image

4. You are doing deep design system work with tokens and color tests

Mode 2 - Test interactions in code

Use Codex.

How does the button morph? How does the composer resize? Where are the tap targets? What happens at different breakpoints?

Static mocks cannot answer these questions. You need something running.

When to pick this -

1. You are testing responsive behavior across mobile, tablet, desktop

2. You need to feel how transitions work in real time

3. You want to stress-test an engineering concept by forking a branch

4. You are building a prototype that needs real data

Mode 3 - Ship the last mile

Use both.

Build interactions in code. Pop them into Figma through the MCP. Go deep on pixel-perfect details. Push changes back to code. Ship.

This is where the loop becomes the most powerful. The round-trip that used to take a sprint now takes ten minutes.

When to pick this -

1. You are polishing a nearly finished feature

2. You need to fix button animations, loading states, string changes

3. Engineering is waiting and you want to unblock them without filing a ticket

4. You want to submit a PR yourself

I covered the decision frameworks in my Codex PM guide. The underlying principle is simple.

The question is never which tool. It is what are you trying to learn right now.

4. The 5-step adoption roadmap

If you are at a traditional company, and your design team follows the linear pipeline, and your company has not have procured these tools yet, here is the path:

Step 1 - Just start

Download the Codex desktop app. You do not need your company’s permission. Build something for yourself.

Real examples from inside OpenAI -

1. A GTM team member built an entire iOS app with zero iOS experience

2. A comms team member designed an interactive drag-and-drop seating plan in HTML

3. A design lead and his wife are building a Japan trip planner for their honeymoon

None of these people are engineers. They downloaded the app and started.

If you do not know what to build, ask yourself one question. What would I build if I could build anything? Then build it.

Step 2 - Get the loop working once

Install the Figma MCP plugin from the Codex app. Open a project you already have.

The checklist -

1. Import one screen from code to Figma

2. Make one change in Figma

3. Push that change back to code

4. Verify it worked

Get the loop working once end to end. This proves the pipeline before you scale it.

The foundation matters. Name your components well. Align design tokens with CSS tokens. Clean up your Figma file structure. If your files are messy it is harder for both colleagues and agents to get up to speed.

Step 3 - Start with polish not process

Do not replace your existing workflow on day one. Use code for the polish stage first.

1. The button animation that never feels right

2. The loading state engineering skipped

3. The string change stuck in the sprint backlog for three weeks

4. The spacing tweak that would take 30 seconds but requires a whole ticket

This is the lowest-risk entry point. You are not changing how the team designs. You are adding a new capability where the cost of iteration was always highest.

Step 4 - Shift the starting point

Once comfortable, change where design begins.

Instead of paper sketches or low-fi Figma frames, open Codex and describe what you want. Show the team something real in the first meeting.

What changes -

1. Edge cases surface in the first conversation instead of the third sprint

2. Product strategy discussions start with working software instead of static decks

3. Your wireframe gives the team more dimensionality to the problem than any mockup

4. Feedback is immediate because the thing exists

Step 5 - Use AI as your tutor

You will hit walls. You will not understand why a React component renders the way it does.

Ask. The AI is an infinitely patient tutor that never clocks out.

Questions that build facility -

1. Start simple. Can you build this? If it does something, ask how does that work

2. Go deeper. I just inherited this system. Can you explain the data architecture

3. Find gaps. Are there redundant systems? Look through the entire codebase

4. Learn structure. What is the difference between a layout page and a normal page

The people who succeed in this era are not the ones who already know how to code. They are the ones curious enough to keep asking questions.

The roadmap is not learn to code. It is learn to be curious in a world where every question gets answered.

5. Total football for product teams

The natural question from above is: If designers can code and engineers can design and PMs can prototype, do we even need separate roles?

Yes. And understanding why is the most important takeaway.

The three questions that define each role

1. Engineers ask how do we build this well

2. Designers ask how should someone experience this

3. PMs ask why should we build this at all

A designer who ships code still self-identifies as a designer. They still care about being the voice of the user. They still think about craft and flow and emotional experience. The medium changed. The mandate did not.

The skill layer accelerates the blurring

Inside Figma, designers are writing skills that teach AI how to design well. PMs write skills for product decision-making. Anyone can use anyone else’s skill.

1. A designer uses a PM’s skill to work through a strategic decision

2. A PM uses a designer’s skill to prototype a flow

3. An engineer uses a design skill to ship polish to production

The skill layer means you are no longer limited to a third of the picture. You can lean into adjacent domains without switching careers.

Why roles survive despite tool convergence

The constraint was never the role. It was the amount of time it took to become proficient in tools outside your domain. If you wanted to learn code that gated you. If you wanted to learn design that gated you.

Now those gates are open. What remains is natural inclination. Judgment. Taste.

The mental model is total football. In 1970s Holland, every player could play every position. The goalkeeper could attack. The striker could defend. But each player still had a natural spike. The team was more dangerous because everyone could cover for each other.

The phrase inside OpenAI - prototypes, not PRDs. PMs bring working prototypes to design reviews. They ship PRs to stress-test ideas with engineers. The artifact that aligns teams is now running software, not static documents.

The bottleneck has moved. If developers have been accelerated 10x, designers have been accelerated maybe 1.5 to 2x. Design can become the bottleneck if you are not coding yourself.

The tools are the same across roles. The questions are different. And the questions are what define the role.

Get Transcript

Where to find Ed Bayes

• LinkedIn

• OpenAI

• X

Where to find Gui Seiz

• LinkedIn

• Figma

• X

Related content

Podcasts:

1. Xinran Ma - Design with AI

2. Crash Course in AI Product Design

3. Codex PM Guide with Carl Vellotti

Newsletters:

1. AI prototyping for PMs

2. The PM guide to Bolt

3. Codex PM guide

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Share

I’ve coached 200+ PM candidates. 30+ landed AI PM offers in the last 12 months at OpenAI, Anthropic, Google, Meta, and Amazon.

And what I’ve watched happen to the interview process is wild.

Candidates who would have cruised through a PM loop in 2023 are getting rejected.

The AI PM interview in 2026 tests a completely different set of skills. Most candidates are studying for the old test.

Mock Example

To show you what the new test looks like, I recorded a full, unedited mock interview with Dr. Bart Jaworski (12,000+ PMs helped, former Microsoft AI PM):

Watch it. Then read the breakdown below. (I break down the 10 specific things I did in this mock that most candidates miss, with a pinnable infographic, in the paid section.)

Want live coaching like this three times a week? Apply for my Land PM Job cohort (Cohort 3, May 4, 75 seats).

Why Become an AI PM

The competition is for modern AI PM roles is brutal because the roles pay so well:

If you want to break in, you can’t bring the old playbook.

Today’s Post

I’ve put together the ultimate guide to PM behavioral interviews in 2026:

1. What’s Changed and Why the Old Prep Fails

2. How to Pass the New Interview:

   • A custom GPT, Claude Skill, and Gemini Gem to grade your responses

   • 10 takeaways from my mock to emulate

   • The 6 questions you must nail

3. The Reference Vault:

   • Real questions by company

   • Interview prep bank

   • Cheat sheet

Subscribe now

1. What’s Changed and Why the Old Prep Fails

I’ve seen five big shifts in working with candidates.

Change 1 - They test whether you’ve built AI, not whether you understand it

The old interview accepted “I understand how transformers work.”

The new one asks: Tell me about a time your model degraded in production and what you did.

They’ll probe technical details to verify you drove the work. If you can’t name the architecture, the eval metrics, and the business impact, they assume you were adjacent. Not driving.

One candidate I coached at Google told me:

“They asked me what the F1 score was. I said I’d have to check. Interview was over in their minds.”

Change 2 - Vibe coding is a real round now

Google India. Figma. Perplexity. v0. Netflix. Stripe.

PMs are being asked to build prototypes in 45 minutes using Cursor, Bolt, or Lovable. I wrote the web’s first vibe coding interview guide because this caught so many candidates off guard. It’s also been confirmed on Blind.

If you’ve never opened these tools, you will fail this round. No framework saves you. You need reps.

Change 3 - AI product sense replaced traditional product sense

Google removed the standalone technical interview for PMs entirely. OpenAI made AI product sense a required round.

The questions sound similar (”design a feature for X”) but the evaluation is different. They want you to separate model-layer from application-layer problems, address safety without being prompted, and prioritize with real math. Not “high/medium/low.” Actual estimates with numbers attached.

Change 4 - Behavioral questions got harder

“Tell me about a time you failed” became “Walk me through an AI product decision you made that seemed right but you’d approach differently now.”

“Tell me about a trade-off” became “Describe a time you chose between model accuracy and serving latency, and explain the technical reasoning.”

Generic STAR stories don’t survive these questions. You need AI-specific depth in every answer.

Change 5 - Safety is tested everywhere

Anthropic has a dedicated AI safety & ethics round. OpenAI embeds it throughout. Google tests it inside “Googleyness.”

If you hit minute 40 of a case without mentioning safety, you’ve told them you don’t understand AI product management. You don’t need a separate safety answer. You need safety woven into everything.

These five shifts add up to one thing: the prep that worked in 2023 gets you rejected in 2026.

I built 3 practice tools and broke down the 6 questions that actually decide AI PM offers. Here's the full system.

The rest of this post is for paid subscribers only. You’ll get:

1. The 6 behavioral questions every AI PM company asks, with fresh answers for 3 candidate archetypes (career changer, lateral mover, engineer pivot)

2. Real questions by company (OpenAI, Anthropic, Google, Meta, Amazon, Netflix, Nvidia)

3. 3 practice tools I built: a Claude Skill, Custom GPT, and Gemini Gem that grade your responses

4. The complete 84-question bank (downloadable)

Read more

Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by:

1. Bolt: Ship AI-powered products 10x faster

2. Jira Product Discovery: Plan with purpose, ship with confidence

3. Kameleoon: Leading AI experimentation platform

4. Amplitude: The market-leader in product analytics

5. Product Faculty: Get $550 off their #1 AI PM Certification with my link

Today’s episode

If you want to run a truly AI-pilled product team, you need to get everyone on a Team OS.

When (PM @ DoorDash, Author of In The Weeds) first described this concept I had never heard of it, but after she showed it to me, I realized it’s absolutely genius.

We’ve done a lot of Claude Code episodes on this podcast, but this episodes is something genuinely new and impactful for PMs.

It’s the architecture and steps to building a Team Operating System so that you can scale your impact as a PM and product team:

• You checking in every single thing you do to a repo

• Team members checking it before checking with you

• Everyone building one shared, compounding knowledge base

If you are building a team that runs on AI, this is the episode to watch.

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

I’m starting the third cohort of my Land PM Job program soon. Join us.

Newsletter deep dive

As a thank you for having me in your inbox, here is the complete guide to building a Team OS with Claude Code:

1. The Team OS structure

2. Context management theory

3. Scaling analytics across functions

4. How to write 10x docs with planning

5. The learning flywheel

Subscribe now

1. The Team OS structure

As a PM, you are the human router. Every question goes through you. Every answer lives in your head or in a doc no one can find. That does not scale when one PM supports 20 people across five functions.

A Team OS fixes this. One shared GitHub repo. Every function checks in their work. Any coding agent traverses it. Everyone self-serves.

Grab Hannah's Template Repo

Component 1 - The root Claude MD

The root file loads every session. Keep it to three things -

1. Doc index. A map telling Claude where every type of information lives. Without it, Claude runs expensive explore agents searching your file system. With it, Claude navigates directly on the first try.

2. Team roster with handles. Every person, their role, their Slack ID, their GitHub handle. Enables natural language queries like “Slack Alex about the bug from today’s call.”

3. Key Slack channels. Channel names mapped to IDs and purposes. “Send this in the product channel” just works.

Don’t stuff too much into the root file. If it is longer than one page, you are burning context on information not needed in 80% of sessions.

I covered the foundations of building a PM operating system in my setup guide. The root Claude MD is the single most important file in that system.

Copy-pasteable template -

# Team OS

## Doc index
- product/       - PRDs, strategy, competitive research, customers
- analytics/     - Metrics, queries, schemas, dashboards, playbooks
- engineering/   - Bug investigations, RFCs, technical designs
- team/          - Onboarding, retros, team-level docs
- .claude/       - Shared agents, commands, skills

## Team
- Alex Chen (eng lead) - @alexc (Slack) / @achen (GitHub)
- Morgan Li (designer) - @morganl (Slack) / @mli (GitHub)
- Taylor Kim (analyst) - @taylork (Slack) / @tkim (GitHub)
- Jordan Wu (strategy) - @jordanw (Slack) / @jwu (GitHub)

## Channels
- #product-team    - Daily standups and feature updates
- #eng-team        - Engineering discussions and PRs
- #data-insights   - Weekly metrics and analysis reports

Component 2 - Nested doc indexes

Every major folder gets its own Claude MD. These are not content files. They are navigation maps.

In the episode, a query about customers consumed only 3% of the context window. Claude did not touch analytics. Did not read engineering docs. Navigated directly to the right files.

Without nested Claude MDs, Claude runs explore agents across your entire repo. Burns tokens. Takes longer. Leaves less thinking room.

Template for every folder -

# [Folder name]

## Doc index
- subfolder/file - What it contains and when to read it
- subfolder/file - What it contains and when to read it

## Key context
1-2 sentences needed in 80%+ of sessions in this folder

Component 3 - The folder architecture

A production Team OS has three top-level sections -

1. .claude/ folder. Shared agents, commands, skills. Customer call summary skill. PR creation command. Weekly synthesis automation. Everyone uses these.

2. Product development. Customers, competitive research, PRDs, strategy docs, launch emails, meeting summaries, analytics (metrics/queries/schemas), engineering (bugs/RFCs). Organized by function, then product area.

3. Team folder. Onboarding guides, retros, team-level docs.

The full directory tree -

team-os/
├── .claude/
│   ├── agents/
│   ├── commands/
│   └── skills/
│       └── customer-call-summary.md
├── product/
│   ├── CLAUDE.md
│   ├── customers/
│   │   ├── CLAUDE.md
│   │   ├── acme-corp/
│   │   │   ├── CLAUDE.md
│   │   │   ├── calls/
│   │   │   └── summaries/
│   │   └── forge-labs/
│   ├── competitive/
│   ├── prds/
│   ├── strategy/
│   │   ├── plans/
│   │   └── vision/
│   └── workflows/
├── analytics/
│   ├── CLAUDE.md
│   ├── billing/
│   │   ├── metrics.md
│   │   ├── queries/
│   │   └── schemas/
│   └── onboarding/
├── engineering/
│   ├── bugs/
│   └── rfcs/
└── team/
    ├── onboarding/
    └── retros/

The ownership model matters too. On the team in the episode, functional leads own their folders but the whole team agrees on structure. The data scientist owns analytics. Engineers own bugs and RFCs. The PM owns product context. Strategy partners own customer calls.

A non-technical strategy partner who had never opened GitHub two months ago now puts up PRs every day. This is not just for technical people.

Every level of nesting is a context-saving decision. The more precisely you organize, the less Claude has to read. That is the whole game.

2. Context management theory

You open Claude Code. You paste in a PRD. A competitive analysis. Three customer call transcripts. You have already consumed half your context window before you even ask a question.

No thinking room left. That is the trap.

The Team OS is built on the opposite principle. Load only what you need, when you need it.

The four pillars

1. Context. The information in a given LLM session. What the model can access right now.

2. Context window. How much it can hold. ~1M tokens today. Seven to eight novels. But the docs produced by a single team far exceed that.

3. Compaction. When the window fills, information gets compressed. You lose fidelity. Compressed summaries are much less useful than originals.

4. Thinking room. The gap between loaded info and window size. This is where the model reasons. More context loaded = less room to think.

I wrote about these foundations in my context engineering guide with an OpenAI product leader.

The token efficiency framework

Not all context is equal. A well-structured repo has three tiers -

1. Tier 1 - Always loaded. Root Claude MD. Team roster. Channel map. Under 500 tokens. Loaded every session.

2. Tier 2 - Loaded on query. Folder-level Claude MDs. Doc indexes. 200-500 tokens each. Only loaded when Claude navigates to that folder.

3. Tier 3 - Loaded on demand. Actual content files. PRDs, transcripts, SQL queries. Hundreds to thousands of tokens. Only loaded when specifically needed.

The magic is in the split. If you only need to know what a metric measures, Claude reads metrics.md and stops. Does not pull SQL queries. Does not pull table schemas.

Someone who burns hundreds of thousands of tokens and hits their usage limit in 30 minutes generally has unstructured, unoptimized context. The Team OS eliminates this.

Failure modes that burn context

1. Flat repo with no Claude MDs. Claude runs explore agents for every query. Burns thousands of tokens just navigating. I have seen teams hit usage limits in 30 minutes because of this.

2. Overstuffed root file. Everything dumped into one Claude MD. Loaded every session whether needed or not. Kills thinking room before you even start working.

3. Full transcripts instead of summaries. A one-hour customer call is 10,000+ tokens. A structured summary is 500. The repo points Claude to summaries first. Full transcripts only if the summary does not answer the question.

The skill that fixes the third failure mode - a shared customer call skill. Every team member summarizes calls in the same format. Same structure. Same fields. Cross-customer analysis becomes trivial.

Here is the pattern. You have ten people taking calls with twenty different customers. Without the skill, every summary looks different. With the skill, Claude can compare apples to apples across hundreds of calls in seconds.

Context management is not a nice-to-have. It is the difference between an AI that helps and an AI that hallucinates.

3. Scaling analytics across functions

Your engineer is on call at 2 AM. Something looks wrong with billing. They need to check a dashboard. They need the query for churn by segment. They need the table schema to validate the data.

You are asleep. Your analyst is asleep. The engineer is stuck.

This section pays for the entire Team OS investment.

Layer 1 - Metrics, queries, and schemas

Organize by product area, then by data type -

analytics/
├── billing/
│   ├── metrics.md          # Definitions only
│   ├── queries/
│   │   ├── generation_success_rate.sql
│   │   └── churn_by_segment.sql
│   └── schemas/
│       ├── billing_events.md
│       └── subscription_tables.md
├── onboarding/
│   ├── metrics.md
│   ├── queries/
│   └── schemas/
└── dashboards.md           # Links to all dashboards

The split is intentional -

1. What are our billing metrics? - Claude reads metrics.md only. Low context.

2. Show me the churn query. - Claude reads metrics + query file. Medium.

3. Full metric + query + schema. - All three files. Higher but still targeted.

4. Point Claude at the raw database. - Everything loaded. Maximum burn. Maximum hallucination risk.

The prompt that shows the power - “How do we calculate generation success rate? Show me the metric definition, the SQL query, and the table schema.” Claude navigates to the right product area. Pulls exactly three files. Gives you everything.

Hook this up to Snowflake MCP or another analytics tool. Claude does the actual analysis using verified approaches your analyst checked in.

Layer 2 - Playbooks and verified approaches

Your data scientist has a process for investigating funnel drop-off. That process belongs in the repo as a playbook.

When you ask “why are users dropping off during custom domain setup?” Claude finds the playbook. Follows the same methodology your analyst uses. Produces a result you can trust.

This is how you kill hallucinations in data analysis. Not by hoping. By giving Claude verified approaches.

The failure mode to watch for - not having the analyst audit the playbooks. If Claude follows a wrong methodology, it produces confidently wrong results. Your data scientist must own and verify the analytics folder.

Layer 3 - The feature launch gate

Make this non-negotiable. A feature is not launched until the repo is updated.

The checklist -

1. Metric definitions for the new feature checked in

2. SQL queries verified by your analyst

3. Table schemas documented

4. Dashboards linked

5. Playbooks for common investigations created

This is how shared context compounds. Every sprint, the repo gets more valuable. Every function gets more self-sufficient.

I covered the broader launch process in my product launch playbook. Add repo updates as a hard gate.

A second gate worth adding - the repo is part of the PRD itself. When you write a PRD for a new feature, include a section specifying what metrics, queries, and schemas will be checked into the repo as part of the launch.

The repo is not documentation. It is the system that makes everyone on the team faster every single day.

4. How to write 10x docs with planning

You type a prompt. Claude makes all the decisions. You get something back. You spend two hours fixing it.

That is not Claude Code’s fault. That is a planning failure.

The junior employee metaphor is perfect. You hired a brilliant junior. Then gave them zero guidance.

Three prompting tiers

1. Basic prompt. Type request. Claude decides everything. Unpredictable quality. Fine for quick lookups. Terrible for strategy docs.

2. Lightweight alignment. Add “give me a proposal first.” Claude proposes an approach. You correct direction in 30 seconds. I use this for anything with even slight ambiguity.

3. Full plan mode. Press Shift+Tab twice. Claude’s bias for action is removed. Cannot execute until you approve the plan. This is where real quality happens.

In the episode, even the lightweight alignment produced dramatically better results. Claude used the repo context to generate a competitive analysis proposal without being told to. Found the right folders. Read the right docs. Proposed the right structure.

The full planning process

Five phases for a strategy doc -

1. Phase 1 - Load context. Claude reads competitive research, vision docs, writing guides in parallel. Structured repo = only relevant files loaded.

2. Phase 2 - Ask user questions. Claude uses the ask-user-question tool. Who is the audience? What is the focus? Should we update competitive intel?

3. Phase 3 - Build the plan file. Section-by-section document structure. You read the plan. Push back. Refine.

4. Phase 4 - Push your thinking. The advanced move -

Use ask-user-question tool to push me on my thinking.
Help me consider other angles for this document.
Challenge my assumptions. Take as long as you need.

Claude interviews you. Catches gaps. Suggests sections you missed.

5. Phase 5 - Review agent prompts. For complex plans, check what each agent will be prompted with. What files will it read? What writing guide? Skills auto-invoke only ~70% of the time. Always explicitly specify in the plan.

Here is the thing. Most people rush in. Let it write a bad first draft. Then yell at it for getting things wrong. The fix is not better prompting after the fact. It is better planning before you start.

I covered the Claude Code setup foundations. Plan mode is where intermediate users become advanced.

Parallel agents and temp files

A single agent cannot read 40 context files and write a great long doc. Context fills. Quality degrades.

Fix these by doing followling-

1. Split the document across agents. Each gets specific context files and your writing guide.

2. Each agent writes to a temp file. If ten agents return work to the parent simultaneously, the context window overflows. You lose everything. You must prompt Claude to use temp files - it does not always do this automatically.

3. The orchestrating agent compiles the final document from temp files.

Two more advanced techniques -

1. Verification prompts. Tell Claude how to self-check. Require sources for claims. Use Playwright MCP to verify front-end output.

2. Save plan files to the repo. Native plan files get wiped every 24-72 hours. If you spent three hours on a plan, save it. Next time you start at 80% done. Your team can reuse it. OpenAI published that they made plan files first-class artifacts of their engineering repos.

The decision matrix for when to plan -

1. Straightforward task, clear scope - lightweight proposal. 30 seconds.

2. Complex doc, some ambiguity - full plan mode. 15-30 minutes on the plan.

3. Highly ambiguous, multi-phase research - full plan mode with thinking partnership. Could be hours spread across a day.

4. Recurring process - saved plan file in the repo. Start at 80% done every time.

Most people under-plan. That is why the output does not match what they wanted. The plan is not overhead. The plan is the work.

5. The learning flywheel

After 1,500 hours, Hannah is still iterating on her setup every single day. You should be, too.

The beginner’s mindset

Ask Claude to teach you. This one habit accelerates learning faster than anything else.

1. When you encounter a file you do not understand - explain why this is a YAML file and not markdown

2. When you download someone else’s skill - explain why this skill works and what could be improved

3. When you want to improve your repo - analyze this repository structure and suggest improvements

The prompt I run every day -

Research everything Anthropic shipped in the last 90 days. Read the top Claude Code influencers and their best posts. Compare my setup to the latest features. Tell me how to 10x my setup.

Claude’s training data gets stale. This prompt forces it to find what is new and apply it to your system.

I covered this progression in my AI PM learning roadmap. Claude Code fluency is one of the highest-leverage skills on it.

The automation-to-learning loop

If you only have two hours this weekend, ask one question - what can I automate to free up six hours next week?

The flywheel -

1. Automate one task → Free up time

2. Use freed time to learn → Improve your repo

3. Better repo → More automation possible

4. More automation → Even more time freed

Every MCP you hook up makes the system more powerful. The rule from the episode - any core software you use daily should be connected. The limit does not exist.

Shared automations compound the flywheel across the team. A weekly customer research synthesis that posts to Slack. A PR notification command that tags the right reviewer. A metrics check that runs every morning.

Mistakes that stall progress

1. Giving up after day one. Building a Team OS takes weeks. You are building infrastructure, not running a prompt. Commit to 30 days.

2. Copying without understanding. People download skills from the internet. Do not know why they work. When they break, they cannot fix them. Always ask Claude to explain first.

3. Treating it as a coding tool. The most misleading name in AI. Most PM time in Claude Code is writing docs, doing analysis, building prototypes, running automations. Not writing code.

4. Not clearing between tasks. Type clear when switching. Leftover context pollutes results.

5. Context rot. Not updating the repo. Claude uses outdated competitive intel. Outdated metrics. The repo must be a living system that every team member updates as part of their workflow.

The terminal is not scarier than a chatbot. Once you have typed into it for an hour or two, you feel comfortable. The barrier is psychological, not technical.

The PMs who build a Team OS this quarter multiply their leverage by 10x. The PMs who keep being the bottleneck for context just made themselves the slowest person on the team.

Get transcript

Where to find Hannah Stulberg

• LinkedIn

• In the Weeds Substack

• Her Github repo

Related content

Podcasts:

1. My Claude Code PM OS with Dave Killeen

2. Claude Code + Analytics with Frank Lee

3. Claude Code as PM OS with Carl Vellotti

Newsletters:

1. The ultimate guide to context engineering

2. Build your PM operating system

3. How to use Claude Code like a pro

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Share

Today’s Post

I tested all of Claude's team features and ranked the ones that matter for PMs and builders:

1. The Full Tier List

   • Deep dives into every S and A tier feature

   • Overview of the other features

2. My Claude Setup

3. 4 PM Workflows That Chain These Together

4. Lessons from how Anthropic Builds Product

Plus downloadable poster + Claude.MD file for PMs:

Subscribe now

Before we get in, here’s a roadmap for reading this. S tier features changed how I work daily. A tier features save me measurable time most weeks. B tier features are solid and worth learning when you hit the right scenario. C and D tier exist but I won’t waste your time on them here.

If you’re short on time, read S tier and Section 2 (My Setup). If you have 20 minutes, read the whole thing. If you want to go deep on Claude Code specifically, I wrote a full masterclass and a complete Cowork guide that pair with this piece.

1. The Full Tier List

S Tier: Game Changers

These changed how I work. Adopt them now. Here’s how:

Read more

But here’s the thing: Bolt.new is the best harness for that power when it comes to AI prototyping for PMs.

$40M ARR in 5 months. Over 7 million users. A $700M valuation. And a $105M Series B. Those numbers are a signal: AI prototyping has changed how modern PMs work.

I sat down with Eric Simons, the CEO, on my podcast last year. What he told me changed how I think about prototyping, PRDs, and the entire product development lifecycle:

Since that conversation, I’ve spent months using Bolt.new, building prototypes, testing with users, and comparing it head-to-head against every major competitor.

This newsletter is the result.

For the full interactive experience -- with comparison tools, a tool picker quiz, animated walkthroughs, and more -- check out the companion Learn Bolt.new Guide we built entirely inside Bolt. Read the Learn Bolt.new Guide

Read the Learn Bolt.new Guide

Today’s Post

Here’s everything we’re covering:

1. How to Understand Bolt.new

2. The Role of AI Prototyping for Modern PMs

3. How Bolt.new Fits Into Your Product Development Process

4. The PM’s Tactical Guide to Bolt.new

5. Advanced Bolt.new Tips & Techniques

6. Bolt.new vs. The Competition

7. Real-World Case Study: landpmjob.com

8. Building Your PM Portfolio with Bolt.new

9. Prototype to Production

Subscribe now

1. How to Understand Bolt.new

Let’s start with what Bolt.new actually is.

Bolt.new is a cloud development environment that lets you go from a natural language prompt to a full-stack, working web application in minutes. You describe what you want in plain English. Bolt.new generates the frontend, backend, and database. You iterate by chatting with it. And you deploy with one click.

That sounds like every other AI coding tool. Here’s what makes it different.

The entire environment runs in your browser

Bolt.new uses WebContainers - a technology built by the StackBlitz team that runs a full Node.js development environment directly in your browser tab. No waiting for servers to provision. Everything is instant.

This is a meaningful technical distinction. It’s why Bolt.new feels fast.

The key facts:

• Founded by: Eric Simons and Albert Pai (born out of StackBlitz, which built WebContainers)

• AI backbone: Claude Opus/Sonnet power the code generation

• Pricing: Token-based (free tier available, Pro ~$20/mo)

• Deployment: One-click to Netlify, Vercel, or Bolt.new Cloud

• Integrations: Supabase, Figma, GitHub, Stripe, plus 170+ MCP integrations (via Pica) for connecting to external services

• What it generates: Full-stack web apps (Vite by default, with support for React, Next.js, Astro, and other popular JS frameworks) with backend, database, and auth

For PMs, the relevant framing is this: Bolt.new is the tool that lets you build the prototype yourself, in minutes, without asking engineering for help.

Continue Reading Online

2. The Role of AI Prototyping for Modern PMs

Before we dive into tactics, let’s talk about why AI prototyping matters and how to think about it strategically.

Good Ways to Use AI Prototyping

Start with Clear Product Problems. Use prototypes to explore solutions to well-defined problems, not to figure out what problem you’re solving.

Good: “Users abandon onboarding at step 3. Let’s prototype 3 different onboarding flows.”
Bad: “Let’s build something cool and see what happens.”

Prototype to Learn, Not to Build. The goal is learning and validation, not production code. Optimize for speed of learning, not code quality. Build 3 rough prototypes in a day to test different approaches, then pick the winner to refine.

Test with Real Users Early. Put prototypes in front of actual users within 24-48 hours. Don’t polish in isolation. Ship a prototype to 10 users by end of day, collect feedback, iterate tomorrow.

Use Prototypes in PRDs. Attach working prototypes to your PRDs. Show, don’t just tell. This eliminates ambiguity. Instead of writing “The dashboard will show KPIs”, link to a prototype: “Here’s what I mean: [prototype]”

Bad Ways to Use AI Prototyping

Treating Prototypes as Production Code. Prototypes are disposable. Over-investing in code quality slows learning velocity. Instead: Build fast, learn fast, throw away. Only productionize after validation.

Endless Iteration Without User Feedback. Polishing in isolation leads to solutions that miss the mark. You need user signal. Instead: Ship rough, get feedback, iterate. Repeat daily.

Using AI to Avoid Thinking. AI accelerates execution, not strategy. You still need to define what to build and why. Instead: Do the hard thinking first. Use AI to execute faster, not to decide for you.

The tool makes you faster at testing hypotheses. It doesn’t generate the hypotheses for you.

3. How Prototyping Fits Into A Modern Product Development Process

Forward-thinking teams are moving beyond documents and closer to the “bare metal” of the pixels that actually define a product. In the old way, prototypes were rare and came late. In the new way, prototypes are everywhere from day one.

The Old Way

• Ideation: Most teams barely prototype. A rare few exceptional PMs do (~5%).

• Planning: Some teams use prototypes, but sketches and mockups are far more common (>75%).

• Discovery: In empowered companies, many teams test prototypes (~50%). In feature factories, this step is skipped entirely.

• PM Handoff: It’s rare for PMs to include prototypes in their PRDs -- often met with side-eyes from designers (5%).

• Design Exploration: This is where most prototypes finally appear (~75%).

• Engineering Start: Engineers typically start from scratch with a final design.

The New Way

• Ideation: You use AI prototyping to work out product problems. You build 3-4 quick prototypes of different approaches in an afternoon.

• Planning: You pair roadmap discussions with working demos. “Here’s what I’m proposing” goes from a paragraph in a doc to a clickable app.

• Discovery: You put functional prototypes in front of real customers the same day you build them.

• PM Handoff: You attach working prototypes to your PRDs. Engineers can see exactly what you mean. Designers can see your intent and elevate it.

• Design Exploration: Designers go from your PM-fidelity prototype to a design-level prototype that fits into the design system.

• Engineering Start: Engineers can use the prototype code as a starting point, then polish it with tools like Cursor or Windsurf for production.

In other words: all three of PM, design, and engineering are now using these tools. And the PM is leading the charge.

Anthropic does this internally. They build prototypes for every customer problem they’re considering. They launch them internally. They see what people actually use. Then they decide what to productionize.

4. The PM’s Tactical Guide to Prototyping

Now let’s get tactical. Here’s exactly how to use Prototyping effectively as a PM.

The 4-Step Workflow

Step 1 -- Start with Context. Give Bolt.new a clear picture of what you want. You can describe the design you’re after in your prompt, use Bolt.new’s Design agent to generate a beautiful starting point automatically, or import a Figma mockup for pixel-level precision. The more visual context you provide, the fewer assumptions the AI makes.

Step 2 -- Create a PRD. Before diving in, create a concise PRD that outlines core functionality, user flows, and critical requirements. You can ask Bolt.new itself to help generate this.

Step 3 -- Build. Start by describing the entire feature at a high level, then break it down into components. Be explicit about functionality and interactions.

Step 4 -- Iterate, One Step at a Time. Change one thing at a time. Create a copy before significant changes. Provide specific feedback about what is and isn’t working.

Good vs. Bad Prompts

The difference between a mediocre prototype and a great one often comes down to prompt quality. Here are real examples:

Building a Feature:

• Bad: “Create a task management app”

• Good: “Create a task management app with a sidebar showing projects, a main area showing task lists grouped by status (To Do, In Progress, Done), and a detail panel on the right. Use a clean minimal design with a blue accent color.”

Fixing a Bug:

• Bad: “The button doesn’t work”

• Good: “I want the Save button to store the form data to Supabase and then redirect to the dashboard page. Currently nothing happens when I click it.”

Design Feedback:

• Bad: “Make it look better”

• Good: “The card spacing is too tight. Add 16px gap between cards, round the corners to 12px, and add a subtle box shadow on hover.”

The pattern: specificity wins. The AI can’t read your mind. Tell it exactly what you want.

The Debugging Playbook

Debugging can feel like the scariest part for non-technical PMs. It doesn’t have to be. Follow these four principles:

Principle 1: Use a System Prompt. When starting a new project, define not just what your app should do, but how it should be built. System prompts should focus on architecture and development guidelines.

Principle 2: Create Instructions First. Before fixing an issue, explicitly state what you’re trying to accomplish. Don’t say “the button doesn’t work.” Say “I want the Save button to store the form data and redirect to the dashboard.”

Principle 3: Plan Before Code. When facing a complex bug, ask the AI to make a plan first without writing any code. This forces systematic thinking rather than jumping to solutions.

Principle 4: Learn Code Patterns. You don’t need to become a developer. But learning to recognize basic patterns in HTML structure, React components, and database calls helps tremendously.

5. Advanced Bolt.new Tips & Techniques

Once you’ve mastered the basics, get good at these 6 techniques:

Advanced Tip 1 - Use Different Models for Different Tasks

Switch between Claude Opus for complex architecture, Sonnet for balanced speed/quality, and Haiku for simple UI tweaks. Each model has different strengths and costs.

Example: Use Opus to design your data model and core logic, then switch to Sonnet for implementing features.

Advanced Tip 2 - Leverage Plan Mode

Before implementing, ask Bolt.new to create a plan. Review the plan, provide feedback, then execute. This prevents wasted iterations.

Example: Say “Create a plan for adding user roles and permissions” before diving into implementation.

Advanced Tip 3 - Use Design Context Strategically

Bolt.new’s Design agent can generate polished, on-brand UIs from a prompt alone. For teams with existing Figma files, importing high-fidelity mockups gives Bolt.new precise visual targets and improves first-pass quality even further.

Example: Describe your design direction in the prompt and let the Design agent handle the aesthetics, or import Figma screens when you need exact design-system alignment.

Advanced Tip 4 - Checkpoint Before Major Changes

Create explicit save points before attempting risky refactors. Use Bolt’s version history to roll back if needed.

Example: Say “Save current state as checkpoint” before major architectural changes.

Advanced Tip 5 - Publish with Built-in Security Audit

When deploying via Bolt.new Cloud, it automatically runs security checks for common vulnerabilities, exposed secrets, and auth issues.

Example: Bolt.new will warn you if API keys are hardcoded or if authentication is misconfigured before going live.

Advanced Tip 6 - Create Progress Tracker MD Files

Use markdown files to track progress, decisions, and next steps. Bolt.new can read these files to maintain context across sessions.

Example: Create a PROGRESS.md file: “Built user auth | Working on dashboard | Next: Add analytics”

6. Bolt.new vs. The Competition

The Big Four

• Bolt.new -- Quick prototypes with flexible iteration power to expand

• Lovable -- Easy to use for non-technical users

• Replit -- Products that require working with persistent data or internal tools

• v0 -- Beautiful front-end design by default

Bolt’s strength is iteration speed. Entire environment runs in the browser (instant, no VM spin-up). Most flexible -- you pick the tools and libraries. Best for PMs who want speed and control.

Lovable has surged to $400M ARR. Its selector tool and Supabase sync make it friendlier for non-technical PMs than any other option. If your team has zero technical background, start here.

Replit, valued at $1.16B, dominates with full-stack power. Built-in databases, built-in auth. The pick for PMs building lasting tools that need to scale into real products. It sacrifices speed for depth.

v0, by Vercel (valued at $3.25B), shines at crafting polished front-end prototypes. If visual design quality is your top priority, v0 will give you the most beautiful output by default.

Additional Specialized Tools

• Reforge Build -- Product education platform with hands-on AI prototyping curriculum and cohort-based learning, available in my bundle (Education)

• Dazl -- Enterprise-focused AI app builder with compliance and governance features (Enterprise)

• Magic Patterns -- Design-first prototyping tool with beautiful defaults and component libraries, also in my bundle (Design-First)

• Base44 -- Purpose-built for building an entire app on top (Startup)

Adjacent & Complementary Tools

Claude Code (Highly Recommended) -- The most powerful AI coding assistant. It reads your entire codebase and makes complex multi-file edits autonomously. While Bolt.new gives you the prototype, Claude Code helps you take it to production with intelligent refactoring.

Cursor -- VS Code fork with built-in AI for inline editing and chat. Complements Bolt.new for local development and production code editing.

Windsurf -- Cursor alternative with strong debugging capabilities. Complements Bolt.new for complex debugging workflows.

Figma -- Industry-standard design tool with Bolt.new integration. Import Figma designs directly into Bolt.new to jumpstart prototypes.

ChatGPT -- Good for brainstorming and one-off prototypes. Complements Bolt.new for ideation phase before building.

Supabase -- Postgres database with auth, real-time, and storage. One-click integration with Bolt.new for instant backend.

My take: Use Bolt.new for rapid prototyping and iteration. Once you’re ready to take a prototype to production, use Claude Code to refactor, optimize, and scale the codebase. This combination is unbeatable.

7. Real-World Case Study: landpmjob.com

I host my entire landpmjob.com cohort platform on Bolt.new:

It’s a >$1M/year business hosted entirely on Bolt.new.

Key Features I’ve built in Bolt.new:

• SEO Optimization -- Custom meta tags, structured data, and optimized loading for search rankings

• AI Applications -- Resume analyzer, interview prep bot, and job matching algorithm built into the platform

• Free Tools -- Publicly accessible tools (resume checker, salary calculator) that drive top-of-funnel traffic

• Comparison Pages -- Dynamic comparison tables (PM bootcamps, certifications) that rank well in search and convert visitors

• Payment Integration -- Stripe integration for course payments with automatic access provisioning

• Content Delivery -- Video hosting, progress tracking, and certificate generation -- all built in Bolt

Bolt.new isn’t just for throwaway prototypes. With the right approach, it can power real businesses at scale.

8. Building Your PM Portfolio with Bolt.new

One of the highest-ROI uses of Bolt.new: creating a professional PM portfolio that showcases your work and helps you land interviews.

More than 80% of PMs don’t have a portfolio. But the 18% who do are seeing outsized success in the job market. With Bolt.new, you can build a portfolio in 30 minutes that would have taken days before.

Why a PM Portfolio Matters

• Get Discovered -- Publish on the open web so recruiters and hiring managers can find you

• Stand Out -- Put it atop your resume so applicants can learn more about your work in detail

• Convert Inbound -- Feature it on LinkedIn to convert more profile visits into conversations

Quick Start Prompt for Bolt

“Create a professional PM portfolio website with a hero section, career highlights showcasing 4 achievements with metrics, detailed case study pages, skills section, and contact page. Use a clean, modern design with a professional color scheme. Make it mobile responsive.”

The AI PM Portfolio is especially powerful. If you’re targeting AI PM roles but don’t have AI features on your resume, build AI prototypes as side projects and showcase them in your portfolio. This flips a weakness into a strength.

What to Include

• Hero Section -- Your unique positioning, professional photo, contact info

• Career Highlights -- 4-5 achievements with metrics showing your impact

• Case Studies -- Detailed stories of 2-3 projects with data and work products

• Skills & Tools -- Technical skills, PM tools, domain expertise

• About Page -- Your story, background, what makes you unique

9. Prototype to Production

Eventually, some prototypes need to graduate to production apps (like my cohort). Here’s how to make that transition smoothly:

1. Security & Secrets. Don’t hardcode credentials. Always use environment variables. Move all sensitive information to proper secrets management. Bolt.new Cloud automatically checks for vulnerabilities.

2. Authentication. Most prototypes use simplified auth. For production, implement proper user management -- Supabase Auth, Auth0, or Clerk.

3. Scaling Servers. Prototyping environments have limited resources. Migrate to Vercel, Netlify, or AWS Amplify for real traffic.

4. Custom Domains. Give your app a professional appearance. Configure through Bolt.new Cloud or your deployment platform.

5. Refactor with Claude Code. Use Claude Code to refactor your Bolt.new prototype into production-grade code. It can read your entire codebase and make complex improvements.

Remember: scaling is optional. Many internal tools can live permanently as prototypes. Only invest in scaling when you’re confident in product-market fit. The best product decision is often knowing when a prototype is “good enough.”

[Bonus] Takeaways Poster

Final Words

AI prototyping is redefining the PM role. Not replacing it. Redefining it.

The PMs who will win in 2026 and beyond are the ones who think in interactions rather than features. Who embrace experimentation over specification. Who seek early disconfirmation rather than building false certainty.

You don’t need to become a developer. You don’t need permission from engineering. You need a problem worth solving, a clear prompt, and 30 minutes.

Now go build something.

For the full interactive version with comparison tools, tool picker quiz, animated walkthroughs, and step-by-step workflows to build your first prototype or PM portfolio, check out the Learn Bolt.new Guide.

Cheers,

Aakash

Share

Thanks to Bolt for making today’s post free for everyone. Hope you enjoyed yesterday’s Job Search OS drop. Tomorrow for paid subscribers I’ll have an awesome walkthrough on Anthropic’s latest features. And on Friday, a podcast with Designers at OpenAI and Figma.

“I think we’ve achieved AGI.”

Whether or not you agree with his definition, something real has shifted. AI agents can now do work that creates economic value. They can research, write, build, and execute multi-step workflows autonomously.

So the question I’ve been asking myself: if AGI is here, what does that mean for job search?

I think the answer is that we can build the first product in the world that actually automates a job search end to end. Not a chatbot. Not a resume template. A system that does the work while you sleep.

I started building one. Then I saw this tweet blow up:

I had to investigate. Spent the last week running Cowork through a full job search workflow.

The tweet is technically accurate. You can point Cowork at job boards with your resume, and it fills forms and submits.

But it’s the wrong strategy, and the wrong tool.

Mass-applying with AI doesn’t work. And Claude Cowork isn’t the right harness for the job (yet).

You need a system that helps you apply to a few targeted roles. And the harness needs to be customizable.

That harness is Claude Code - with a Job Search OS.

The people getting hired in 2026 look different. They apply to fewer roles, but each one is surgically targeted. Their resumes never contain a fabricated bullet. They have referrals stacked before they submit. And they spend 20 minutes a day on their search instead of 3 hours.

I know this because I’ve coached hundreds of candidates through job searches. I run a job search cohort (Cohort 3 starts May 4, 75 seats). I’ve placed people at OpenAI, Anthropic, Meta AI, and Google in the last year.

Over two thousand people now use my PM Operating System. I spent the last 6 months encoding the same depth of knowledge into a system for the full job search funnel: from discovering roles to landing offers.

What Your Morning Looks Like

30-minute one-time setup. Then:

• You open your laptop. Your daily briefing is waiting. The OS scanned your target companies for new postings in the last 24 hours. The top roles that match your profile are scored and ranked. A customized resume for each one, built from your real experience. Already reviewed and corrected. Ready to submit.

• Below the roles: your outreach for the day. Draft messages personalized to each person. Follow-ups for recent connections. Referral requests for roles you applied to. Nudges for referrals that have gone quiet.

• Prep for your upcoming interviews. The questions that company actually asks (from web research plus insider data from hundreds of candidate debriefs), the interviewer’s background, recent product news, and a game plan for your specific candidacy gaps. Mock interviews grade your answers and tell you which stories to swap in.

• After the interview. Every question categorized. Every answer scored. The exact moments where you left value on the table, with rewrites. A personalized thank-you email referencing what you actually discussed. Each interview makes the next one sharper.

• Your daily time: 20-30 minutes. The system handles the rest. The OS can apply to as many roles as match your criteria. The difference is that every single application is tailored, reviewed, and backed by a referral strategy. Volume AND precision.

If you’re full-time job searching after a layoff, this is a no-brainer. The OS does in 20 minutes what used to take 3 hours. The math on ROI is simple: the average job switch leads to a 10-15% raise. At a $200K salary, that’s $20-30K. This costs $49.

Introducing the Job Search OS

I want everyone using this to be operating at the level of the best job searchers I’ve coached. So I’ve put together a copy-paste system to go from zero to a fully automated job search.

This is more than a collection of prompts. This is months of figuring out what actually works. Hundreds of iterations. Testing, improving, learning.

It’s the actual system I use with my coaching clients and cohort members. Not theory. Not templates from the internet. Not bloat.

18 skills. That’s it. Not 200 skills with filler, not a pile of files you’ll never open. 18 skills, each one mapping to a specific step in my system for getting interviews. Every file earns its place. Nothing extra. The bare minimum you need and nothing else.

How to Get It

You have two options to get the Job Search OS:

1. Buy it one-time on Gumroad for $49

2. Join my Founders Plan for $250

   • Get access to all of my digital products (including PM OS and prompt library)

   • Plus a year of the newsletter

   • And my AI bundle

Get the Job Search OS ($49)

Upgrade to Founding Plan

What You Get

What’s Inside

Here’s a small preview:

Today’s Deep Dive

I’m covering everything you need to use the Job Search OS:

1. How to install the Job Search OS

2. Using the Job Search OS (every skill, every workflow, with screenshots)

3. Pro tips, mistakes, and FAQ

Putting it all together, it’s the most comprehensive job search system for Claude Code + Cowork available anywhere.

This post is for subscribers in the Founding Member plan

Upgrade to Founding Plan

Already in the Founding Member plan? Sign in.

Read more

Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by:

1. Bolt: Ship AI-powered products 10x faster

2. Amplitude: The market-leader in product analytics

3. Pendo: The #1 software experience management platform

4. NayaOne: Airgapped cloud-agnostic sandbox

5. Product Faculty: Get $550 off their #1 AI PM Certification with my link

Today’s episode

Claude Code hit $2.5 billion in annualized revenue in 12 months.

It is the fastest B2B software product ramp in history.

So why are most people still using it like a chatbot?

This is how most people use Claude Code. Type a prompt and get output. The context fills up. It compacts. You lose everything. You start over.

The top users flipped it. They built operating systems. Skills, sub-agents, file structures, hooks. Every prompt benefits from everything built before.

I sat down with Carl Vellotti for the third time. His first episode was the beginner course. His second episode was the advanced masterclass. Together they crossed over a million views across platforms.

Today is the operating system layer. If you are already an 80 out of 100 on Claude Code, this episode will bring you to a 95.

If you are living in Claude Code 8 to 10 hours a day and want to stop fighting the tool, this is the one episode to watch.

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

I’m putting on a free webinar on Behavioral and AI PM interviews. Join me.

Newsletter deep dive

As a thank you for having me in your inbox, here is the complete step-by-step guide to building your own Claude Code operating system.

Two starter repos you can fork today. Every file explained. Every technique from the episode plus what I have learned building my own PM OS.

1. Fork your starter OS

2. The CLAUDE.md file

3. The compounding file system

4. Mastering context

5. Skills that fix weaknesses

6. Data you can actually trust

7. The daily loop

1. Fork your starter OS

You do not need to build from scratch. Carl open-sourced his personal operating system at github.com/carlvellotti/carls-product-os. I open-sourced mine at github.com/aakashg/pm-claude-code-setup.

Carl’s repo comes with a BLANK-OS folder you rename and fill in, plus an EXAMPLE-OS with real content so you can see what a populated system looks like.

Step 1 - Fork and clone

# Carl's OS (lighter starting point, free)
git clone https://github.com/carlvellotti/carls-product-os.git
cp -r BLANK-OS/ my-workspace && cd my-workspace

# Or mine (41 skills, 7 sub-agents, $49)
git clone https://github.com/aakashg/pm-claude-code-setup.git my-pm-os
cd my-pm-os

Step 2 - Open in Claude Code

That is it. You are inside a pre-structured operating system. Every folder, every placeholder, every workflow is in place. Fill it with your data.

Carl draws a useful distinction. Workflows tell Claude how to create something. Templates tell Claude what the output looks like. Knowledge is reference material useful across projects. Project research is scoped to one project and gets archived with it. Your brand guidelines go in knowledge. Competitor research for a specific launch goes in the project folder.

You do not need everything on day one. Start with CLAUDE.md + GOALS.md. These two files give you 80% of the value. Add structure when friction appears.

2. The CLAUDE.md file

This is the most important file in your operating system. It is automatically loaded into context for every single message. Not just the first. Every one.

I have iterated on my CLAUDE.md over 100 times. What you think will work does not always. After lots of testing, here is what actually matters.

What goes in it

# About me
- Name: [Your name]
- Role: Senior PM at [Company]
- Team: Growth, reporting to [Manager name]
- Current quarter: Launching [feature], improving [metric]

# How I work
- Short, direct communication. No filler.
- Show reasoning before conclusions.
- When I say "spin up", delegate to a sub-agent.
- Check /knowledge/people/ before drafting any message.
- Use Jupyter notebooks for any data analysis.

# My tools
- GitHub CLI: installed (use for all git operations)
- Vercel CLI: installed (use for deployments)
- Firecrawl CLI: installed (use for web scraping)
- Tavily MCP: enabled (use for web search only)

# Current priorities
1. Ship auth redesign by March 28
2. Prepare board deck for Q2
3. Hire senior designer

# My skills
See /.claude/skills/ - auto-matching hook is active.

# Preferences
- Bold key phrases in documents
- Bullet points for 3+ items
- End sections with a punchy one-liner

The compounding trick

Update this file every week. Every time you hit an issue with Claude Code, pull the fix in. “Do not run research in the main session. Always use a sub-agent.” “When I say spin up, delegate to a separate instance.” Over months, this file becomes an incredibly precise instruction set. Claude feels like it already knows you.

I covered the CLAUDE.md deep dive in my Dave Killeen episode, where the CPO at Pendo told me his Claude Code system is better than the human EA he used to have. The CLAUDE.md was the foundation of everything he built.

If you only set up one file, make it this one.

3. The compounding file system

The fundamental difference between a Claude Code OS and just chatting with AI is that everything is a markdown file. And those files are alive.

In a normal AI conversation, everything lives in one chat window. Close it and it is gone. In a Claude Code OS, the AI reads and writes to persistent files. Every conversation makes the files better. Every file makes the next conversation better. That is the compounding loop.

The folder structure

my-workspace/
├── CLAUDE.md              ← Always in context, every message
├── GOALS.md               ← Quarterly priorities
├── .claude/
│   ├── skills/            ← Custom commands
│   └── hooks/             ← Auto-trigger scripts
├── knowledge/
│   ├── people/            ← Stakeholder dossiers
│   ├── company/           ← Strategy, positioning, competitors
│   └── research/          ← User research, industry reports
├── projects/
│   ├── [project-name]/    ← One folder per active project
│   └── _archive/          ← Completed projects
├── sub-agents/            ← Reviewer personas
├── workflows/             ← Process templates
├── templates/             ← Output templates (PRDs, OKRs)
├── data/                  ← CSVs, Jupyter notebooks
├── tasks/
│   ├── current.md
│   └── backlog.md
└── tools/                 ← Pre-built scripts

The knowledge folder

People files are where the real compounding happens. Create one for each stakeholder at knowledge/people/[name].md:

# Sarah Kim - Design Lead
- Reports to: VP Design
- Cares about: User research, accessibility, craft
- Communication style: Visual thinker. Wants mockups, not docs.
- Pet peeve: Decisions made without user data

## Recent context
- Pushing back on the auth redesign timeline
- Concerned about mobile accessibility
- Liked the competitive analysis I shared last week

## Meeting notes
### March 21 - Auth redesign sync
- Wants mobile prototypes before eng sprint
- Asked for usability report from last iteration
- Action: I owe her the report by Friday

Connect a meeting transcription tool like Granola via MCP. After every meeting, tell Claude: “Update the people file for Sarah based on today’s meeting.” Over time, when you draft a message to Sarah, Claude already knows she prefers visual communication and is concerned about mobile. The output is dramatically different from a generic draft.

Company files hold strategy, positioning, and competitive landscape. Research files hold user interviews, industry reports, and anything you reference across projects.

The projects folder

Every new task gets its own folder. When you come back the next day:

@projects/board-deck-q2/ Get up to speed on this project, 
then help me finish the revenue slide.

Claude reads every file in the folder. All the research, all the drafts, all the data. It picks up exactly where you left off.

Bootstrap fast. Do not build project folders one by one. Connect Linear or Notion via MCP and run:

Pull my last 5 active projects from Linear. For each one, 
create a folder in /projects/ with the PRD and latest status.

The people folder compounds across all projects. The project folder compounds within one. Together they mean Claude never starts from zero.

4. Mastering context

You have the file system. Now the biggest technical challenge: making sure Claude can actually use it without running out of context.

I covered the foundations in my context engineering guide. Here is how it works inside Claude Code specifically.

Step 1 - See your context

Run this right now:

/status line

Ask for a color-coded context meter. Green under 50%. Orange 50-80%. Red over 80%. Then run /context to see exactly what is eating it:

1. System prompt - ~2% (cannot change)

2. MCP servers - ~8%+ (each one adds permanent overhead)

3. Custom agents - ~4%

4. Conversation - grows with every message

Before you type a single message, 10% to 16% is already gone. Disable anything you are not actively using.

Step 2 - Delegate to sub-agents

Without sub-agents, a research task pushes context from 16% to 25%. Almost 10% of your window gone for one question.

With a sub-agent, the same task costs 0.5%. Claude spins up a separate instance. That instance does all the web searches and page reads. Your main session only gets the summary.

The numbers from the episode:

1. Without sub-agent - 16% to 25% context (+10%)

2. With sub-agent - 16% to 16.5% context (+0.5%)

3. Sub-agent used 10 tool calls and 30,000 tokens, all outside main session

Two ways to trigger this:

1. Inline - Add “spin up” or “use a sub-agent” to any prompt

2. Skill - Build a skill at .claude/skills/delegate-research.md that automatically delegates research tasks. Both repos include this.

Bonus trick: If Claude goes on a tangent, hit Escape twice. Roll back to before the bad prompt. Everything after it is erased from context completely.

Step 3 - Replace MCPs with CLIs

This is the hierarchy most people get backwards. MCPs eat context just by existing. CLIs sit on your machine with zero context overhead.

Connection Context cost When to use MCP High (always loaded) Last resort API Medium Custom integrations CLI Zero Always prefer

Andrej Karpathy confirmed it. The CLIs to install today:

1. GitHub CLI (gh) - Claude is aggressively competent at it. PRs, issues, repos, zero context cost.

2. Vercel CLI - Deploy, check logs, set env variables without leaving Claude Code.

3. Firecrawl CLI - Clean web scraping without MCP overhead.

The test: for every MCP you have enabled, ask Claude “does a CLI for this exist?” More often than you think, the answer is yes.

Every MCP you replace with a CLI is context you get back for actual work.

5. Skills that fix weaknesses

Every time someone says “Claude Code is bad at X,” the real problem is the same. Claude does not have the right tools or instructions. I covered the basics in my Claude Code guide. Here is the advanced version.

Tier 1 - Prompt-only skills

No code. No APIs. Just a well-written prompt. Anthropic has 56 official plugins you can install with /plugins. The front-end design one is the best example. All it does is tell Claude “do not look like AI.” Before-and-after quality is dramatic.

To build your own, create a file at .claude/skills/[name].md:

---
name: write-like-me
description: Use when the user asks to write, draft, or edit.
---

# Writing Style Skill

1. Short sentences. One idea per sentence.
2. Use "you" and "your" when teaching.
3. Open with the problem, not the solution.
4. Bold the key phrases.
5. End every section with a punchy one-liner.

Tier 2 - Tool-powered skills

Give Claude tools to check its own work. The make slides skill from the episode includes Puppeteer so Claude can screenshot its HTML output, measure text overflow, and fix issues automatically. Tell it “iterate 3 times before showing me” and the first result is already polished.

The web research skill replaces Claude’s unreliable default web search (basically Google first-page results) with Tavily for higher-quality search and Firecrawl for clean markdown scraping. Install Firecrawl as a CLI, not an MCP, to save context.

A key finding from a Google paper: literally pasting a prompt twice improves output. The practical version: after Claude executes any skill, tell it “double-check your output against the skill instructions.” Quality improves every time. This is the builder-validator pattern.

Whenever you catch yourself screenshotting and saying “fix this” repeatedly, that is a signal. Build a self-checking skill.

Tier 3 - Auto-invoking with hooks

Skills are supposed to auto-invoke from keywords. It does not work reliably. The fix is a user_prompt_submit hook that runs a lightweight script matching your message against skill keywords. Zero context cost. Instant. Claude still decides whether to use it.

This beats listing skills in CLAUDE.md because it adds no permanent context overhead and auto-updates when you add new skills. Both repos include the hook script ready to use.

The ask user questions tool

This was Carl’s favorite Claude Code feature, and one I had never used before this episode.

Tell Claude “use your ask user questions tool” and it generates a custom UI with checkboxes and input fields right inside the terminal. Instead of making assumptions, Claude asks you. Sometimes 10 questions. Sometimes 67.

Use it for:

1. Requirements gathering - “Use your ask user questions tool to grill me on all possible requirements for this feature”

2. Filling context gaps - “Ask me everything you need before starting this project”

3. Decision making - “Here are 10 research results. Help me decide which to implement”

The main reason people do not like AI output is assumptions. This tool eliminates them.

Skills are what you do. Hooks are how you compound. The ask user questions tool is how you eliminate assumptions.

6. Data you can actually trust

The biggest question PMs have about AI outputs: how can I trust this?

You are presenting numbers to your boss. They ask “where did this come from?” You need proof of work.

The answer is Jupyter notebooks. Tell Claude “analyze this data in a Jupyter notebook” and it creates a .ipynb file showing every query as a code cell, every result as output, every chart rendered inline. Renders natively in VS Code or Cursor.

The progression Carl demonstrated live:

1. Visualize - Load the CSV, see columns, shape, sample rows

2. Chart - Distribution charts, bar charts, segment breakdowns

3. Analyze - Correlation heatmaps, regression, statistical tests

At each step, your data scientist or manager can read every cell, verify the methodology, and check the results. The code is the proof.

The mental model: take the roles you work with and create versions of Claude Code for each one. The front-end design skill is the designer. Jupyter notebooks are the data analyst. The web research skill is the research assistant.

The difference between trusting AI and not trusting AI is visibility into the process.

7. The daily loop

This is where the operating system comes alive. Everything from sections 1 through 6 feeds into one daily workflow.

Create a standup skill that pulls from all your sources. GitHub CLI for recent commits. Tasks/current.md for priorities. Calendar for meetings. Linear for ticket status. Knowledge/people/ for meeting prep with stakeholder context already loaded.

One command. No tab switching. No manual assembly.

The compounding loop

You have a meeting with Sarah. Granola transcribes it. Your “update people” skill pulls key points into Sarah’s people file. Later, you draft a message to her about the timeline. Claude reads her file. It knows her concerns from the last meeting. The message is specific, not generic.

You finish a project. You move it to _archive/. Next quarter, you start something similar. You tell Claude “look at the archived auth-redesign project.” It reads the PRD, the user research, the launch results. Your new project starts with all the context of the old one.

You hit an issue with Claude Code. You add a rule to CLAUDE.md. Every future session inherits that rule. The same mistake never happens twice.

That is what an operating system is. Not a folder full of files. A system where every interaction makes the next one better. The compounding is the whole game.

How to start today

Fork the starter OS. Go to Carl’s product OS or mine. Clone it. Open it in Claude Code.

Fill in your CLAUDE.md and GOALS.md. These two files give you 80% of the value.

Set up the context status line. Run /status line. You will never unsee how fast your context fills up.

If you want the full system with 41 skills, 7 sub-agents, and the CLAUDE.md I have iterated 100+ times, get the PM OS here. If you want Carl’s free course that teaches Claude Code inside Claude Code, go to ccforpms.com.

Every person file you update, every project folder you populate, every skill you build makes the next prompt dramatically better. That is what turns Claude Code from a chatbot into an operating system.

Get Transcript

Where to Find Carl Vellotti

• Linkedin

• X (Twitter)

• Instagram

• Newsletter

• Claude Code for PMs course

Related content

Podcasts:

1. Claude Code Masterclass with Carl Vellotti (Ep 2)

2. Claude Code Initial Guide with Carl (Ep 1)

3. Claude Code PM OS with Dave Killeen

Newsletters:

1. Claude Cowork and Code setup guide

2. How to use Claude Code like a pro

3. My PM OS

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Share

If you’ve tried OpenClaw or Agent Zero, you know the pain. Local installs, environment configs, dependency hell. Computer runs entirely in the cloud and routes your prompts across 19+ AI models automatically.

I spent weeks testing it. Here’s the web’s first PM guide.

Today’s Post

1. What Is Perplexity Computer

   • How It Works, Remembers, and Runs

   • Setting Up (Connectors, Skills, Custom Instructions)

2. How to Use Perplexity Computer

   • Top 6 Use Cases for PMs

   • How to Prompt Without Wasting Credits

3. Honest Comparison and Limitations

Subscribe now

1. What Is Perplexity Computer

Perplexity CEO Aravind Srinivas calls it a “general-purpose digital worker.”

How It Works

Computer sits on top of 19+ frontier AI models. Claude Opus 4.6 handles core reasoning. Gemini runs deep research. ChatGPT 5.3 assists with long-context recall. Grok takes lightweight tasks. Nano Banana generates images. Veo 3.1 produces videos.

You write one prompt. Computer picks the best model combo for the job.

Tip: You can manually assign which models handle which parts of your task. Most people skip this and let Computer pick the most expensive option. Learning to control model routing is the fastest way to cut costs.

How It Remembers

Computer maintains context across sessions and connects to your services and files. Over time, it learns your preferences and standard workflows without you re-specifying them. Start a new task when the objective changes. Continue an existing task when you’re iterating on the same deliverable. Continuing is always cheaper.

How It Runs

Most AI tools give you suggestions. Summaries. Plans. Outlines you then have to go build yourself.

Computer delivers the finished thing. Research reports with source citations. Deployed dashboards with shareable links. Cleaned datasets with charts. Launch kits with positioning docs and email drafts.

And it runs them in parallel. Start a task. Immediately start another. They execute independently in the cloud. Close your browser. Go to dinner. When you come back, everything is done.

Scheduled tasks work the same way. Your machine doesn’t need to be on. Set it up once. Computer runs it on a recurring schedule whether your laptop is open, closed, or sitting in a drawer.

You can choose which models handle which sub-tasks. You can force the system to check in with you before executing. Most people never touch these controls. I’ll show you exactly how in the prompting section.

Setting Up Perplexity Computer

You need a Perplexity Pro subscription ($20/month), Max ($200/month), or Enterprise Max ($325/seat/month). Pro gets 4,000 credits per month. Max gets 10,000.

Getting started takes 30 seconds. Go to perplexity.ai, log in, click the Computer icon in the left sidebar. You’re in.

Everything is organized around Tasks. A Task is not a conversation. It’s a job. When you submit a prompt, the orchestrator (powered by Claude Opus 4.6) reads it, breaks the objective into subtasks, assigns each one to the best-suited AI model, runs sub-agents in parallel, and compiles results in the right panel.

Tip: Start a new task when the objective changes. Continue an existing task when you’re iterating on the same deliverable. Continuing is always cheaper because of persistent memory

Then, do three things:

Step 1 - Connect Everything

Connectors let Computer access your actual data and take real actions in your existing software. Not summaries. Real read-and-write access.

Setup takes about a minute per app. Click Connectors in the sidebar, browse the list, click Enable, complete OAuth. Done.

Computer has 400+ built-in integrations, and the interesting ones go way beyond Slack and Google Drive:

• Through Plaid, you can link your actual brokerage and Computer builds you a live portfolio dashboard. Daily P&L, position cards, news feed, earnings calendar, price alerts. A personal Bloomberg terminal on a private URL. Full setup and prompt in Section 6.4.

• Research access is wild. CB Insights, PitchBook, Statista. Computer bypasses the paywalls. A $5,000 market sizing report? Computer often pulls it.

• The Snowflake connector is powerful. Type “What were the top 10 customers by revenue last quarter?” Computer writes the SQL, runs it, hands you charts. Plain English in, data out. Same with Salesforce, HubSpot, Databricks.

• GitHub lets Computer write, refactor, and push code to your repos. Gmail, Slack, LinkedIn go deeper than you’d expect. I had it scan my inbox, summarize my newsletters, draft unsubscribe requests, and calculate my SaaS burn rate from receipt emails. Tag Computer in a Slack channel and it runs research without you switching tabs.

• For SEO, Ahrefs, Google Search Console, Google Analytics, WordPress, Cloudflare. Full audit, competitor analysis, keyword mapping, content drafting, and live site updates.

• And the one that surprised me most. Computer connects to Amazon’s Selling Partner API and Shopify natively.

You can also bring your own tools. Provide an MCP (Model Context Protocol) server URL and Computer connects to proprietary CRMs, custom analytics servers, or private APIs. Enterprise admins can share custom connectors across the organization.

Step 2 - Create Skills

A Skill is a saved set of instructions that auto-activates when Computer recognizes a matching task. Same concept as Claude Skills or Cowork plugins.

Without Skills, you’re re-explaining your brand guidelines, formatting preferences, and reporting structure on every task. With Skills, you explain it once.

Computer ships with built-in Skills for Slides, Research, Research Report, and Chart. To create a custom one, click Skills in the sidebar, click + Create skill, and upload a .md file. Skills stack. A research Skill can hand off to a report-formatting Skill, which hands off to a slides Skill. One prompt, full pipeline.

Tip: If you’ve explained the same thing to Computer twice, it should be a Skill. Brand guidelines. Formatting rules. Research methodology. Build it once, never repeat yourself.

Step 3 - Custom Instructions

Skills activate for specific task types. Custom Instructions apply to ALL tasks, all the time. Keep them under 1,500 characters. The single most effective one I've found:

“Always come back to me and clarify any misunderstandings and challenge my thinking to make sure you’re very clear on the stated outcome, and let’s create a brief short plan before we build anything.”

That one line forces Computer to verify what you want before spending credits. Massive savings.

2. How to Use Perplexity Computer as a PM

I’ve identified 6 particularly powerful use cases you should steal. Here’s everything you need to get them set up:

Read more

Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by

1. Product Faculty: Get $550 off their #1 AI PM Certification with my link

2. Amplitude: The market-leader in product analytics

3. Pendo: The #1 software experience management platform

4. NayaOne: Airgapped cloud-agnostic sandbox

5. Kameleoon: Leading AI experimentation platform

Today’s episode

There’s very few AI PMs who have seen inside a single company like Netflix, Meta, or Amazon. It’s even more rare to find one who has experience at all three.

That’s who I’ve brought in today.

Jyothi Nookula breaks out the whiteboard for a full deep dive covering all aspects of AI PM fundamentals and job searching for an AI PM job.

If you are trying to break into AI PM, this is the one episode to watch.

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

I’m putting on a free webinar on Behavioral and AI PM interviews. Join me.

Newsletter deep dive

As a thank you for having me in your inbox, here is the complete guide to becoming an AI product manager in 2026.

1. Taxonomy of AI PM Roles

2. The Core AI PM fundamentals

   • When to use AI

   • Which AI to Use When

   • The 3 Building Blocks of an AI Feature

3. How to Become an AI PM

Subscribe now

1. Taxonomy of AI PM Roles

Before you write a resume, update a portfolio, or prep for a single interview, you need to answer two questions.

What type of AI PM role are you targeting? And where in the stack do you want to sit?

Get these wrong and you’ll spend months preparing for interviews that test completely different skills than what you studied.

Axis One - Traditional PM with AI features vs AI Native PM

Type 1 - Traditional PM with AI features

This is 80% of what is labelled as AI PM jobs right now. Companies adding LLM capabilities to existing products.

1. A chatbot on a customer service portal

2. AI summarization inside a document tool

3. Smart search on an existing knowledge base

The core product existed before anyone bolted an LLM onto it.

Type 2 - AI native PM

The remaining 20%. Here the product IS AI.

1. OpenAI

2. Anthropic

3. Cursor

The product is fundamentally probabilistic. The value proposition is impossible without AI. You cannot build ChatGPT without an LLM.

There are 4x more open roles in the traditional category. If you are trying to break into AI PM, that is where the volume is. But the AI native roles are where the hardest product problems live.

Know which type you are applying for before you write a single line on your resume. The skills overlap. The interview questions do not.

Axis Two - Where in the AI Stack You Fit

After the role type, the second axis is where in the stack you sit.

The deeper you go, the harder the technical bar.

Layer 1 - Application PMs (60% of roles)

Own the end-to-end user experience. How users interact with AI. How to build trust. How to make AI reliable for everyday use.

This is the easiest entry point for someone converting from a traditional PM role. It builds on existing product management skills with AI knowledge layered on top.

Layer 2 - Platform PMs (30% of roles)

Build tools that other teams use.

1. Developer platforms

2. Model orchestration systems

3. Evaluation frameworks

4. Observability tools

You are not building for end users. You are building for other builders.

Layer 3 - Infra PMs (10% of roles)

Build the foundational systems that power everything above.

1. Vector databases

2. GPU orchestration

3. Model serving optimization

4. Kernel level compilation

The good news? The hardest roles are the smallest bucket.

The layer you choose determines the interview prep, the portfolio, and the companies you target. Pick your layer before you pick your first project.

2. The Core AI PM Fundamentals

Three things separate the AI PMs who ship from the ones who flounder: knowing when to use AI, knowing which AI technique fits the problem, and understanding the technical building blocks well enough to make real product decisions.

Let’s go through each.

2a. When to use AI

Here is where most teams get it wrong. Not a technical mistake. A product decision mistake.

MIT found that 19 out of 20 AI pilots fail. A key factor? Picking the wrong problems. Teams reach for AI when a rules-based system would have solved it faster, cheaper, and more reliably.

I covered the decision frameworks in my AI product strategy post. Here is the decision tree from the episode.

AI makes sense in three patterns

1. Pattern recognition in complex data. YouTube uses ML to identify patterns across billions of viewing sessions. The relationships are multi-dimensional. No rules could capture that.

2. Prediction from historical data. At Amazon, AI forecasts inventory needs based on hundreds of variables. Seasonal trends, promotions, weather patterns. Humans cannot process that many variables.

3. Personalization at scale. Individualised experiences for millions of users. Content recommendation engines are the classic example.

Heuristics win in four situations

1. Explainability is non-negotiable. AI models still have low explainability. Regulated industries need auditability.

2. Clear domain rules exist. Tax calculation. Tax codes are complex but explicit. Perfect for rules.

3. Data is limited. New features, new markets, no historical data. Do not force-fit AI.

4. Speed is critical. AI systems take longer to build. For MVPs, ship first. Add AI later.

The best AI PMs know when to say no to AI. That judgment is more valuable than knowing how to build a RAG system.

2b. Which AI to Use When

You decided AI is the right tool. Now which AI?

Most teams jump straight to LLMs when a simple ML model would have solved the problem in a week at a fraction of the cost. I wrote about this trap in my RAG vs fine tuning vs prompt engineering guide.

Traditional ML

Regression models. Random forests. XGBoost. Mature. Reliable. Still powers most of the AI you use daily.

Choose ML when:

1. You have structured data and need to predict or classify

2. You need the model to explain its decisions

3. Speed and cost matter

The PM question: Can I put this problem in a spreadsheet with clear input columns and an output I want to predict? If yes, start with ML.

Deep learning

Neural networks. Computer vision. Speech recognition.

Choose deep learning when:

1. You are dealing with images, video, or audio

2. Humans can do the task easily but you cannot write rules for it

3. Pattern recognition is too sophisticated for traditional ML

The face recognition example from the episode is perfect. You recognise a face instantly. Writing if-then statements to replicate that? Impossible.

Gen AI

LLMs. Diffusion models.

Choose Gen AI when:

1. Users need to interact conversationally

2. You are creating new text, images, or code

3. You need reasoning and synthesis across multiple sources

The optimization hierarchy (before fine tuning)

This is the most important takeaway for anyone building with AI right now.

1. Prompt optimization - better prompts, better results

2. Context engineering - what information gets loaded into the context window

3. RAG - dynamically retrieve relevant knowledge

4. Fine tuning - only if 1-3 do not work

80% of use cases get solved with RAG. Teams that jump to fine tuning because it is in the API documentation burn months unnecessarily.

These are not competitors. They are tools in your toolkit. The best AI products combine multiple approaches.

2c. The 3 Building Blocks of an AI Feature

These are three building blocks every AI PM must know. Each builds on the last.

Building Block 1 - Agents vs workflows

An AI agent makes decisions and takes actions on its own. You do not tell it the exact order. It understands the goal and reasons its way there.

A workflow is the opposite. Predetermined sequences. Every step defined. Deterministic outcomes.

The agent architecture

Four components.

1. Perception - how it receives input (text, images, APIs)

2. Reasoning - where models live (LLMs, classifiers, planning algorithms)

3. Execution - how it takes action (text generation, API calls)

4. Learning - feedback mechanism for improving over time

Building Block 2 - Prompts and Context Engineering

Prompts are the primary interface between you and the AI system. And most teams treat them as an afterthought.

Three layers to understand:

System prompts set overall behavior and personality. For a customer service agent, this means defining empathetic personality, professional tone, identity verification requirements, and escalation rules. The system prompt is the AI’s job description. Write a bad one and you’ll get a bad employee.

Few-shot examples are the most underrated technique in production AI. Show the model concrete examples of what a good response and a bad response look like. It sounds like extra work. In production, it is the single highest-leverage investment you can make. I’ve seen teams double response quality by adding 3-5 well-crafted examples instead of writing 3 more paragraphs of instructions.

Context engineering is where the real PM skill lives. I covered the full framework in my context engineering guide, co-written with an OpenAI product leader. The key insight: context engineering is a PM problem, not an engineering problem.

Three context layers matter:

1. Immediate context — the current conversation or task. What did the user just ask?

2. Session context — recent interactions and state. What happened earlier in this conversation?

3. Knowledge context — broader reference information. This is where RAG comes in.

Here’s why this matters commercially. Claude Sonnet has a 200K token context window. That sounds like a lot. It fills fast when you’re loading a knowledge base, conversation history, real-time data, and user prompts simultaneously. Every token costs money.

The teams that overspend on AI are almost always loading too much context into every interaction. A customer asks about their order status and the system loads the entire product catalog, the user’s full purchase history, and a 50-page return policy document. That’s wasteful. Context engineering is the art of knowing what to load and when — giving the model exactly enough information to answer well, and nothing more.

Building Block 3 - RAG (Retrieval-Augmented Generation)

RAG (Retrieval Augmented Generation) is the single most important technique for enterprise AI. I covered the fundamentals in my RAG vs fine tuning guide.

1. Take your company documents

2. Chunk them into smaller pieces

3. Convert chunks into vectors using an embedding model

4. Store vectors in a vector database

5. When a user asks a question, vectorize the query

6. Find the nearest matching chunks

7. Load those chunks alongside the user input into the LLM

8. The LLM generates a response grounded in your actual documents

The result: the LLM answers using your data, not its training data. It can cite sources. It stays current because you can update the document store without retraining anything.

Why RAG wins over fine tuning for most use cases

1. Works with your existing data immediately

2. No model retraining required

3. Update documents without touching the model

4. Cost is a fraction of fine tuning

5. 80% of enterprise use cases are solved here

The failure mode to watch for: bad chunking strategy. If your chunks are too large, the model gets flooded with irrelevant context. Too small, and it loses the thread. Getting the chunk size and overlap right is where most RAG implementations succeed or fail.

RAG is not glamorous. But it solves most of the problems teams try to throw fine tuning at. Start here.

3. How to Become an AI PM

You know the role taxonomy. You understand the technical fundamentals. Now how do you get the job?

Four steps.

Step 1 - Build products, not projects

Do not stop at a demo. Launch. Get real users. When things break, fix them.

“I built this product. It broke in these ways. Here are the challenges I overcame.” That beats any course certificate in an interview.

The difference between a project and a product: a project is something you show in a portfolio. A product is something someone else uses. Hiring managers can tell the difference in 30 seconds.

Step 2 - Three portfolio artifacts

You need three things in your portfolio, each demonstrating a different AI PM competency:

1. An app solving a real problem you have. Not a tutorial clone. Not a toy. Something you actually use. This proves you can identify a problem and ship a solution.

2. An agent that demonstrates goal-oriented reasoning. Show that you understand the difference between workflows and agents. Build something where the AI makes real decisions about which tools to use and when.

3. A RAG system grounding AI in domain knowledge. Pick a domain you know well. Build a system that answers questions using your own documents. This proves you understand the most commercially relevant AI technique.

Each with real users. Not static demos.

Step 3 - Get certified

The AWS AI Practitioner certificate pairs well with hands-on portfolio work. It signals to hiring managers that your knowledge isn’t just theoretical. It covers the fundamentals - ML concepts, model selection, responsible AI - in a structured way that complements your portfolio.

Certifications alone don’t get you hired. Certifications plus shipped products do.

Step 4 - Learn the PM cultures

Each company shapes a different PM.

1. Amazon - document-driven. PRFAQs and six-pagers reviewed up to VP/CEO level. PMs spend 40-50% of their time writing. You become an exceptional writer.

2. Meta - experimentation first. Ship variants. Test against control groups. Let data decide. The most sophisticated experimentation infrastructure in the industry. You live and breathe statistical significance.

3. Netflix - context over control. No rigid process. No documentation requirements. Trust and autonomy. You define your own swim lane.

Amazon teaches thinking backwards from customer pain. Meta teaches rapid testing. Netflix teaches the power of autonomy.

The companies you work at shape who you are as a PM. The insights compound. Get the experience at even one of these environments if you can.

Get Transcript

Where to find Jyothi Nookula

• LinkedIn

• NextGen Product Manager

Related content

Podcasts:

1. AI PM: Complete Course with Pawel Huryn

2. How to Become an AI PM with Marily Nika

3. AI PM Masterclass with Ankit Shukla

Newsletters:

1. The ultimate guide to context engineering

2. RAG vs fine tuning vs prompt engineering

3. AI foundations for PMs

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Check out the conversation on Apple, Spotify, and YouTube.

Brought to you by:

1. Kameleoon: Leading AI experimentation platform

2. Testkube: Leading test orchestration platform

3. Pendo: The #1 software experience management platform

4. Bolt: Ship AI-powered products 10x faster

5. Product Faculty: Get $550 off their #1 AI PM Certification with my link

Today’s episode

I have done four eval episodes now. We covered error analysis and LLM judges with Hamel and Shreya. Evals from first principles with Ankit. The PM’s role in depth.

But I kept hearing the same question: how do I actually build one from scratch?

That is what today’s episode delivers. We build an eval entirely from scratch. Live. On camera. No pre-written prompts, no pre-written data.

I sat down with Ankur Goyal, Founder and CEO of Braintrust, the eval platform behind Replit, Vercel, Airtable, Ramp, Zapier, and Notion. They just announced their Series B at an $800 million valuation.

Users are running 10x more evals than this time last year. People log more data per day now than they did in the entire first year the product existed. The episode explains why, and then we build one live.

We connect to Linear’s MCP server, generate test data, write a scoring function, and iterate until the score goes from 0 to 0.75. And along the way, Ankur drops what might be the most important reframe of the year: evals are the new PRD.

Plus, we cover the complete eval playbook for PMs:

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

If you want my coaching in your job search, apply to my cohort.

Newsletter deep dive

I took everything from the episode and combined it with lessons from four eval deep dives. Here is the complete practitioner’s playbook:

1. Why vibe checks stop scaling

2. The data-task-scores framework

3. Evals are the new PRD

4. Offline vs online evals

5. How to maintain eval culture

Subscribe now

1. Why vibe checks stop scaling

Open your AI product right now. Try three examples. Tweak the prompt. Ship it.

Two weeks later, support tickets start coming in about edge cases you never tested.

That is the vibe check trap.

When you do a vibe check, you are using your brain as a scoring function. You intuit whether the result is good or bad. That is an eval. It is just the version that does not scale.

Vibe checking is not wrong.

It is the earliest version of what eventually needs to become structured. In the error analysis process I covered in an earlier episode, the first step is reviewing 100 traces and taking notes. That is a structured vibe check.

Here is what matters: the best companies treat the transition from vibe checks to structured evals as a product decision, not an engineering one.

The wall hits when more people start using the product, more subject matter experts start contributing to quality, and more engineers start touching the prompts.

At that point, one person’s gut feeling cannot cover the surface area. You need software and process to execute at scale with predictable performance.

And here is the thing about durability. Models change every few months. Agent frameworks get replaced.

The prompt you wrote today might be irrelevant by summer. But if you invest in understanding what your users actually need and encode that as data, scores, and eval flows, that survives every model swap.

The companies that believe their agent wiring is their moat are highly likely to fail. The ones building true differentiation are the ones investing in evals.

The prompt is temporary. The eval is permanent. That is the whole game.

2. The data-task-scores framework

In the evals deep dive, we covered the full analyze-measure-improve lifecycle. That is the theory. Here is the practitioner’s shorthand: every eval, no matter how complex, is exactly three things.

Component 1 - Data

A set of inputs your AI product needs to handle.

During the episode, we generated test data for a Linear task management bot:

1. “What tasks are assigned to me?”

2. “Are there any overdue tasks?”

3. “How many tasks need to be triaged?”

You can optionally include ground truth answers. If you know the correct answer is 12, include it. If you do not, skip it.

Do not spend a month building a perfect golden data set before running your first eval. Auto-generate silly questions. Jump in. Start iterating.

This echoes what I covered on bootstrapping datasets with synthetic data. But in the live demo, we took it further. We used Opus to generate the data set in seconds, edited it by hand, and ran our first eval within minutes.

Component 2 - Task

A task takes an input and generates an output. It could be:

1. A single LLM call to GPT 5 Nano

2. A complex agent calling MCP servers, vector databases, and multiple models

3. Anything in between

At the end of the day, it produces some kind of output. That is what you evaluate.

During the demo, the first task attempt failed badly. The model just said “Happy to help with Linear” instead of answering. That zero score was expected and useful.

Here is what happened next.

We connected the Linear MCP server so the model actually had access to task data. Still failed. The model said “I am ready to help with Linear tasks” but never called a single tool.

So we changed the system prompt:

Don’t ask clarifying questions, just use the tools and figure it out.

This matters more than you’d expect. Models are post-trained to ask clarifying questions in conversational contexts. In an eval pipeline where you send one question and expect one answer, that post-training behavior works against you. You have to explicitly override it.

Then we loosened the scoring function. It was penalizing responses that referenced Braintrust tasks as sources, even though that was valid citation behavior. The scorer was too harsh for what we actually cared about.

Then we added few-shot examples to the prompt and specified which MCP tools to use.

Score went from 0 to 0.75 across the board. Each iteration touched a different part of the data-task-scores framework. That is the rhythm: identify which component is the bottleneck, fix it, rerun.

Component 3 - Scores

Scores take the input, the expected output if you have one, and the actual output. Their job: produce a number between 0 and 1.

That normalization is critical. It forces everything to be comparable. A week from now, when you run a new eval after swapping the model, you compare directly against today.

Do not overcomplicate scores. Use categorical options, not freeform numbers:

1. A = Full answer with citations

2. B = Partial answer

3. C = No answer at all

In the evals deep dive, I recommended binary Pass/Fail as the default. There is a case that not every score must be binary, but clear criteria beats freeform every time. The key is that you are not asking the LLM to generate an arbitrary number. You are giving it defined options and asking it to classify.

Data in, task runs, score comes out. Everything else is iteration.

3. Evals are the new PRD

This is the reframe that hit me hardest.

Think about a PRD from 2015. Unstructured document. A spec meant to communicate how to build something. The engineering team reads it, half follows it, the final product never matches.

The modern PRD is an eval. It is something that an engineering team who maybe does not know everything about the problem can use to quantify how well the software is solving it.

Step 1 - Encode your product intuition as data

Instead of writing paragraphs describing what the product should do, create a data set of inputs that represent what users actually need.

Step 2 - Use scores as success criteria

Instead of prose acceptance criteria, write scoring functions that quantify whether the software works.

An engineering team that does not know the domain can look at the eval and know exactly what good looks like.

When the eval passes and the product still feels wrong, that is on you. Your scoring function does not capture what matters. And that is actually a new area of leverage that PMs did not have before. You go from providing a qualitative spec nobody follows into something quantifiable.

I have been saying PMs should own error analysis for months now. This episode pushed that further: PMs should own the eval itself. The scoring function IS the modern PRD.

Step 3 - Close the distance gap

The farther you are from the end user, the more critical evals become.

Ankur framed this as a distance principle.

At Anthropic, the people training the models, building the harness, building the product, and using the product all sit inside one set of walls. Feedback circulates with almost no friction. In healthcare, engineers are not patients or doctors.

Ankur’s parents are both doctors, and he said when he talks to them about their work, he has almost no idea what they’re talking about. The jargon is specialized, the stakes are high, and the engineering team has no intuition for what matters.

Evals become the mechanism that bridges that gap.

Finance has the same problem with different jargon. Domain experts speak a language that engineers don’t, and patients become counterparties, but the structural issue is identical. PMs own the scoring function that translates between what users need and what engineering measures.

There was a Twitter controversy about Claude Code not using evals. This blew up enough that someone pinged me about it the day it happened because their boss was questioning whether evals even mattered.

Here is the reality: Claude Code is doing evals. People at Anthropic are using the product, providing feedback, and incorporating that feedback into iterations. That is a form of eval. They just don’t need formal process because the distance between builder and user is zero. That is not the case for 99% of AI products.

One more thing Ankur pointed out: a big use case for Braintrust has been helping companies collect evals they can share with labs so the labs can better support their use case.

When you have distance, you need a ledger to capture the information. Otherwise, how do you communicate it?

The best PRD in 2026 is not a document. It is a data set, a task function, and a scoring function the whole team can run.

4. Offline vs online evals

You have your eval running in a playground. Scores are improving.

That is only half the picture.

Workflow 1 - Offline evals test your hypothesis

This is what we built in the demo. A data set, a task, a scoring function. You run experiments on your laptop. You iterate on the prompt. You swap models. You see the score move from 0 to 0.75.

Fast. Cheap. Repeatable.

This is the AI equivalent of what used to require an A/B test. In the old world, non-deterministic problems meant expensive production experiments. Now you can run those experiments offline. That is why Braintrust customers average 12.8 experiments per day.

Workflow 2 - Online evals test reality

Take the same scoring functions and run them on real production logs. Every time a user asks a question, the scorer evaluates the response.

If your offline eval achieves 0.75 but your online eval consistently shows 0.3, something is not translating.

Workflow 3 - The flywheel

The real power is in the loop:

1. Online evals surface examples not performing well in production

2. Grab those examples, add them to your offline data set

3. Your offline evals now reflect real user behavior, not hypothetical test cases

4. Iterate until scores improve

5. Push changes, repeat

This is the same continuous improvement flywheel I covered in the evals deep dive. Now you can see what it looks like in practice at companies like Ramp, Notion, and Dropbox.

The best teams run this loop daily. More on the specific ritual in the eval culture section below.

Offline evals tell you what should work. Online evals tell you what actually does. The gap between them is your roadmap.

5. How to maintain eval culture

The biggest risk is not that your eval system is bad. It is that your team treats evals as a gate instead of an iterative loop.

Mistake 1 - Running evals only at the end

If you edit your prompt, test it on three examples, then run a full eval just to see if you can ship, you are doing it backwards.

Start from the eval. The best teams Ankur works with have a morning ritual. In standup, they pull up examples from the previous day’s production logs. They reconcile what they see with what their eval scores show. They find novel patterns that have emerged, add them to the data set, and grind on those failures all day. That is the job now.

Mistake 2 - Only having evals that pass

If every eval succeeds, you have blind spots. Either:

1. You do not understand what your users are hitting

2. You do not understand what is impossible today

Have failing evals. When a new model drops, rerun them first. Something interesting always happens.

And interesting does not always mean better. Gemini 3 Flash was outperforming Gemini 3 Pro on coding benchmarks but hallucinating more. Those nuances only surface with a full testing suite.

This is also how you plan product launches around model releases.

Braintrust shipped Loop this way. They built the eval for the feature before they shipped it. The eval failed on every model they tried. Then Claude 3.7 came out and there was a huge jump. The eval passed. They shipped.

Ankur said there was a watershed moment where 3.7 was the first model that could look at its own work and improve. Prior to that, he said models were like a dog looking at itself in the mirror. They didn’t recognize they were evaluating a virtual representation of themselves.

Create the eval for the feature you want. Watch the models. The moment a model hits the quality threshold, you ship.

Mistake 3 - Siloing evals to AI engineers

Evals should not be constrained to engineers. Product managers need access. Domain experts need access. The people who understand the end user should be improving the scoring functions.

Zapier was Braintrust’s first customer. Ankur told a story about their CTO Brian, who has been there for a long time and is very successful.

When Ankur met him, Brian introduced himself as a full-time AI engineer. This guy probably doesn’t have to work, but Ankur said he has never seen anyone nerd out about AI as much as Brian does. Companies like Zapier have pre-existing product-market fit.

If Ramp doesn’t work, it is very bad. They don’t have the leeway to screw things up. The quality bar is high, the scale is real, and vibe checks can’t cover the surface area. That is why these companies gravitate toward evals.

Braintrust removed user-based pricing specifically for this reason. They realized evals should not be locked behind engineering seat licenses.

Going back to the PRD analogy: if only engineers can see and edit the PRD, the product reflects what engineers think matters. Not what users need.

The teams that win are not the ones with the most sophisticated eval setup. They are the ones where evals are the first thing the PM opens every morning.

The whole game

Treat the eval as a first-class product artifact with a PM who owns the experience. The scoring function is the new PRD. The data set is the new user research. The flywheel is the new sprint cycle.

Open your platform of choice, write your first scorer, and run it on 5 real examples today. You will learn more in that first hour than in a month of vibe checking.

The prompt is temporary. The eval is permanent. That is the whole game.

Get transcript

Where to find Ankur Goyal

• X

• LinkedIn

• Braintrust

Related content

Newsletters:

1. AI Evals deep dive

2. AI observability for PMs

3. How to build AI products

Podcasts:

1. AI evals with Hamel Husain and Shreya Shankar

2. AI evals part 2 with Hamel and Shreya

3. AI Evals explained simply with Ankit

PS. Please subscribe on YouTube and follow on Apple & Spotify. It helps!

Share

Andrej Karpathy built a system that runs the other 500 rounds for you while you sleep. He called it autoresearch.

42,000 GitHub stars. Fortune called it “The Karpathy Loop.” Shopify CEO Tobi Lutke pointed it at Shopify’s templating engine and got 53% faster rendering from 93 automated commits.

Everyone covered the ML side. Most PMs closed the tab thinking it wasn’t for them. That’s a mistake. The pattern underneath has nothing to do with GPUs or neural networks. It works on anything you can score.

I’ve spent two weeks pulling apart Karpathy’s repo, the community forks, and the real-world applications people are building on top of it.

Today’s Post

I’ve built the PM’s Ultimate Guide to Autoresearch, covering:

1. What autoresearch actually does

2. Why this matters if you’re a PM

3. How to set it up

4. Six high-value use cases

5. Why the experiment log matters

6. Where Karpathy is taking this next

7. The complete toolkit: skill, eval templates, and analyzer

Subscribe now

Quick setup if you want to follow along: install Claude Code, clone the autoresearch repo with git clone https://github.com/karpathy/autoresearch, and pick the skill or prompt that frustrates you most. That’s your starting point.

1. What autoresearch actually does

Let me ground you in the original before I show you the PM version.

Karpathy trains small language models as side projects. Like every ML researcher, he was stuck in the same grind: tweak the training script, run an experiment, wait for results, check if the model got better, decide whether to keep the change or throw it away, then start the whole cycle again. A productive day might get through 8-10 of these rounds, and most of that time is spent waiting for the GPU.

So he built a system that does the entire cycle autonomously.

The repo has three files. train.py is the training script and the only file the agent is allowed to modify. prepare.py is the evaluation harness that scores the model, and the agent cannot touch it (if it could, it would just make the test easier instead of making the model better). And program.md is the instruction file that tells the agent how to behave, what to try, when to keep changes, and when to revert.

The loop works like this. The agent reads the code, forms a hypothesis about what might improve the model, makes a change to train.py, runs a 5-minute training experiment, and checks the metric (validation bits per byte, a single number where lower means better). If the score improved, the change gets committed to git and becomes the new baseline. If it didn’t improve, git reset wipes the change instantly. Then the agent starts the next round.

That gives you about 12 experiments per hour, roughly 100 overnight.

Karpathy left it running for two days. The agent found 20 improvements on code he’d already hand-tuned for months, including a bug in his attention implementation he’d missed entirely. All 20 stacked and transferred to a larger model, producing an 11% speedup.

Shopify CEO Tobi Lutke tried it the same night. He ran 37 experiments overnight and woke up to a 0.8B parameter model outperforming his hand-tuned 1.6B model. Half the parameters, better results.

Then he pointed the pattern at Liquid, Shopify’s templating engine, and got 53% faster rendering and 61% fewer memory allocations from 93 automated commits.

This is a new way to optimize anything.

2. Why this matters if you’re a PM

My run on a landing page skill: 41% to 92% in 4 rounds. Three changes kept, one auto-reverted.

The pattern works because it removes the bottleneck every PM actually faces: you know the prompt could be better, but you'll never run 50 iterations manually.

It only works when three things are true:

1. A clear metric. Score the output as a number, not a feeling. “Is this good?” doesn’t work. “Does the headline include a specific number?” does. Yes or no. Add up the yeses across 30 test runs and you have a score the agent can optimize against.

2. A measurement tool that runs without you. Claude Code builds an evaluation script that generates outputs, scores them against your criteria, and prints the result. No human in the loop. The scoring runs programmatically so the loop runs overnight.

3. One file the agent can change. Your skill markdown, your system prompt, your email template. The agent edits this one file per round. Everything else is read-only.

All three present, it works. Any one missing, it doesn’t. That’s your filter.

The human never touches train.py. Karpathy writes program.md and the agent handles execution. That’s product management. You define what “better” means. The agent runs the 50 rounds you’d never have time for.

I use Claude Code for this. But it works on any coding agent that can read files, edit files, and use git. Cursor, Windsurf, Codex, Antigravity.

🔒 Below: the 3-step setup, 6 use cases with copy-paste prompts and eval criteria for each, the experiment log walkthrough that Karpathy says matters more than the result itself, and a downloadable toolkit with a skill improver, 10 eval templates, and a results analyzer.

Read more

Check out the conversation on Apple, Spotify and YouTube.

Brought to you by:

1. Jira Product Discovery: Plan with purpose, ship with confidence

2. Vanta: Automate compliance, manage risk, and prove trust

3. Mobbin: Discover real-world design inspiration

4. The AI Evals Course for PMs & Engineers: You get $1250 with my link

5. Product Faculty: Get $550 off their #1 AI PM Certification with my link

Today’s Episode

OpenClaw is the hottest new AI tool that no one is helping you with. There’s zero real content on OpenClaw for PMs.

I covered what OpenClaw is and why it matters when it first went viral. But the rest of the web has been oddly quiet.

That is a big mistake. OpenClaw as software is as important as ChatGPT in 2023 or Claude Code in 2025.

It’s the software soon every PM will be using.

What I’ve found is that setup is the hardest part. So in today’s episode, I’m giving you a complete, step-by-step installation.

Plus, we cover five of the top PM use cases:

Apple Podcast

Spotify

If you want access to my AI tool stack - Dovetail, Arize, Linear, Descript, Reforge Build, DeepSky, Relay.app, Magic Patterns, Speechify, and Mobbin - grab Aakash’s bundle.

If you want my PM Operating System in Claude Code, click here.

Newsletter Deep Dive

As a thank you for having me in your inbox, here is the complete guide to setting up OpenClaw and building PM automations with it.

1. Why PMs should care about OpenClaw

2. How to Set Up OpenClaw

   • How to connect OpenClaw to Slack

   • The workspace file system

3. 5 of the Top PM Use Cases

   • Slack knowledge base

   • Automated stand-up summaries

   • Competitive intelligence on autopilot

   • Voice of customer reports

   • Smart bug routing by customer tier

4. Security and deployment

Subscribe now

1. Why PMs should care about OpenClaw

Go open Claude or ChatGPT right now. Ask it to check your Slack for blockers, scan your competitor’s pricing page, and post a summary to your team channel at 9 a.m. tomorrow.

It can’t do any of that.

Every LLM you use today is reactive. You ask, it answers. You close the tab, it dies. No access to your files, your tools, or your team’s communication.

OpenClaw is different in three ways.

It is proactive, not reactive. Set up a cron job and walk away. It executes at 3 a.m. when you are asleep. Scans channels, monitors websites, generates reports, posts them to Slack. No fingers lifted.

It is model agnostic. Not locked into one provider. Deep research? Plug in Claude Opus. Fast customer responses? Gemini Flash. Budget? Qwen 3.5 at 1/10th the cost. You control the model per use case.

It runs locally. Your data stays on your machine. No cloud lock-in. It reads and writes local files, which means you build a living repository of product docs, customer feedback, and competitive intelligence that grows over time.

Peter Steinberger described the difference between skills and tools perfectly. Tools are organs. Can the agent do it? Skills are textbooks. Does the agent know how to do it? OpenClaw has both.

The difference between OpenClaw and every other AI tool you use is simple. It acts. Everything else just answers.

Continue Reading Online

The three-command installation of OpenClaw

Most people who quit OpenClaw quit during setup. Not because it is hard. Because terminal commands feel unfamiliar.

The truth is, if you are not seeing red text, you are good. Yellow warnings are normal.

Two paths.

The one-click path (training wheels). Go to emergent.sh. They have a bot feature that pastes the install command for you. Replace your LLM key, hit enter, done. The limitation: no full control. You cannot use your own RAM, run a local model, or customize the workspace.

The terminal path (what most PMs should do). Three commands.

Step 1 - Install OpenClaw

Open your terminal. Paste:

npm install -g openclaw@latest

Finds the latest version. Installs it. If it fails, you probably do not have Node.js 22+ installed. Go to nodejs.org, install it, re-run.

Step 2 - Run the onboarding wizard

openclaw onboard

The wizard walks you through:

1. Accept the security warning (hit yes)

2. Choose Quick Start over Manual

3. Pick your LLM provider

4. Paste your API key

5. Choose your messaging channel

For the LLM provider, you have every option. Google, Anthropic, OpenAI, and more. If you want the best dollar-per-quality ratio, Gemini is strong. To create a key, Google “Gemini key create,” click Create, name it OpenClaw, copy, paste it into a note you won't lose.

After the provider, choose the specific model. Gemini 3 Pro for deep work. Any Flash model for speed.

Then choose a messaging channel. For PMs, pick Slack socket mode.

Step 3 - Hatch the bot

The wizard installs the gateway and hatches your bot. You see “Wake up, my friend.” You are live.

During onboarding, OpenClaw generates a soul.md file. This forces you to give your bot a name, a personality, interaction rules. Peter Steinberger designed it this way. He wanted a companion, not a disposable chatbot.

Skip skills and tools selection during setup. Easier to add later by editing the markdown files directly, or by asking the bot.

Three commands. 15 minutes. If you can install a Chrome extension, you can install OpenClaw.

How to connect OpenClaw to Slack

This is the highest-leverage setup. It puts your AI agent inside the tool you already use all day.

The terminal instructions during onboarding are bare bones. Here is the full walkthrough.

Step 1 - Create a Slack app

1. Go to api.slack.com/apps

2. Click Create an App

3. Choose From Scratch

4. Name it OpenClaw

5. Select your workspace

6. Click Create App

Step 2 - Enable Socket Mode

1. Left sidebar: Socket Mode

2. Toggle on

3. Token name: secret-token

4. Hit Generate

5. Copy and save the token

Step 3 - Add bot token scopes

Go to OAuth & Permissions. Scroll to Bot Token Scopes. Add:

1. chat:write

2. channels:history

3. channels:read

4. groups:history

5. groups:read

6. im:history

7. im:read

8. users:read

Each scope controls what your bot can see and do. Skip any your org restricts, but all matter for the use cases below.

Step 4 - Install to your workspace

Scroll to the top. Click Install to Workspace. Hit Allow. Copy the Bot User OAuth Token (starts with xoxb).

Step 5 - Paste tokens into OpenClaw

Back in your terminal, the wizard wants two things:

1. Bot Token (the xoxb one)

2. App Token (the secret-token from Socket Mode)

Paste both. Configure channel access. Allow all channels unless you have restrictions.

Once tokens are in, go to any Slack channel, invite your OpenClaw bot, mention it with @OpenClaw. It responds. You are running an AI agent inside Slack.

The workspace file system

Before building automations, understand where OpenClaw lives on your machine.

Everything is in a hidden folder at ~/.openclaw. On Mac, click Finder, right-click, Go to Folder, type a period, select .openclaw.

Inside the workspace:

1. soul.md - Bot’s personality and values

2. agents.md - Agent identity, safety rules, operational instructions

3. user.md - Your preferences (direct, no fluff, whatever you set during onboarding)

4. memory.md - Persistent memory that survives restarts

5. tools.md - Local configuration notes

6. heartbeat.md - Your cron jobs (starts empty)

All markdown files. Open in any text editor. Ask an LLM to generate content for them and paste in. Or ask OpenClaw itself to modify them.

There is also a gateway dashboard at 127.0.0.1:18789. Chat with your bot, view cron jobs, change API providers, check gateway status. Whatever you do in the dashboard reflects in the terminal and vice versa. Same brain.

If you want to swap from Gemini to Anthropic, go to Config > Secrets in the dashboard, change the model, save. Or just tell the bot in chat and it reprograms itself.

Terminal is your command center. Gateway dashboard is your control room. Both talk to the same agent.

3. 5 of the Top Use Cases for PMs

Setup done. Now let's put it to work.

The five use cases below are the ones Naman and I spent weeks building and testing on camera. Each one replaces a manual workflow you’re probably doing right now:

1. A Slack knowledge base that answers your team’s product questions from your own docs

2. Automated stand-up summaries posted before your first meeting

3. A competitive intelligence pipeline that runs while you sleep

4. Voice of customer reports pulled from every source into one place

5. Smart bug routing that triages by customer tier without you touching it

Plus the full security and deployment guide so you don’t end up WhatsApp-ing your mom a pairing code.

(paid subscribers only)

Read more

Not because the code was bad. Because he wrote it by hand.

His ramp buddy told him to use Clyde, the janky internal predecessor to Claude Code. Boris spent half a day figuring out the tool. Then it one-shotted a working PR.

That was September 2024.

By December, Opus 4.5 wrote 100% of his code. He didn’t edit a single line manually. He uninstalled his IDE.

Today Boris ships 20-30 PRs a day running 5 parallel Claude instances. His team built Cowork, a full product for non-engineers, in about 10 days. They prototyped agent teams for months, trying hundreds of versions before shipping.

No PRDs. No Figma.

Meanwhile, most PM teams are still spending 2 weeks getting a spec through review for a feature that could be prototyped, tested, and killed in an afternoon.

The gap between these two realities is compounding every week.

And the skill that separates PMs on either side of that gap is what I’m calling taste at speed: the ability to evaluate working software fast, kill most of it, and ship the survivors.

I’ve been thinking about this since my LinkedIn post on Boris went viral (215K+ impressions):

The response told me something. This topic hit a nerve.

PMs know the ground is shifting. They just don’t know what to do about it.

Today, I’m going deep.

Today’s Post

I’ve written the web’s first guide to Taste at Speed:

1. Why Taste at Speed Is the Defining PM Skill

2. How the Best Teams Actually Ship Now

3. The Taste at Speed Framework

4. How to Build This Skill

5. Where the PRD Fits Now

6. Real-World Teardown

Plus 4 downloadable tools: a Prototype Evaluation Scorecard, a Skill-Building Roadmap, a Prototype-First PRD Template, and a Divergent Prototyping Prompt Template.

Subscribe now

1. Why Taste at Speed Is the Defining PM Skill

Boris shared an analogy recently that I haven’t been able to stop thinking about.

In the 1400s, less than 1% of Europe’s population was literate. There was a class of scribes who spent years in training. They were employed by kings and lords who were themselves often illiterate.

Then the printing press came along. The cost of printed material dropped 100x over the next 30-50 years. The quantity went up 10,000x over the next century.

“If you think about what happened to the scribes, they ceased to become scribes, but now there’s a category of writers and authors. These people now exist. And the reason they exist is because the market for literature just expanded a ton.”

Now think about this through the lens of product teams.

Software engineers are today’s scribes. PMs are the kings who “employed” them, often unable to write the code themselves.

The disconnect between “what I want built” and “who can build it” is the fundamental coordination problem that PRDs, design reviews, sprint planning, and every alignment meeting existed to solve.

When building costs near zero, that coordination layer compresses. The bottleneck moves from “can we build it” to “should we ship it.”

PRDs existed because building was expensive and you needed sign-off before committing resources. When a prototype takes 45 minutes instead of 6 weeks, nobody needs a document to authorize exploration.

They need someone who can look at working software and say “this one, not that one” in real time.

If anything, the spec now comes after the prototype.

This is taste at speed.

And it’s a filtering function, not an acceleration function. The 80% kill rate is the whole point.

“Half my ideas are bad and you just have to try stuff. You try a thing, you give it to users, you talk to users, you learn, and then eventually you might end up at a good idea. Sometimes you don’t.”

Without taste, speed just means you build the wrong thing faster. That’s a feature factory on steroids.

I think the best articulation of why judgment becomes the long-term moat is Shreyas Doshi’s recent piece on product sense. His logic: AI tools will commoditize. When everyone has roughly equivalent AI capabilities, the only differentiator is the human judgment applied on top of AI outputs. He breaks that into five skills: empathy, simulation, strategic thinking, taste, and creative execution.

Taste at speed activates all five simultaneously. You’re staring at working software and running empathy (does this solve the real problem?), simulation (what breaks at scale?), strategy (does this fit where we’re going?), taste (is this the best of the options?), and creative execution (can I see a version 2x better?) all at once.

Why Speed Compounds

The speed component matters because it creates a flywheel.

A PM who evaluates 15 prototypes a week builds judgment faster than a PM who reviews one spec a month. After 6 months, that volume of pattern-matching reps creates a taste gap that keeps widening.

The experience gap becomes a taste gap.

The taste gap becomes a career gap.

And it compounds every single week.

2. How the Best Teams Actually Ship Now

Let me walk through exactly how the Claude Code team operates.

The Old Model vs. The New Model

The traditional flow was linear:

Idea → PRD → Design → Eng builds → QA → Ship (8-12 weeks)

The AI-era flow is cyclical:

Idea → 5 prototypes → Evaluate → Kill 4 → Spec the survivor → Ship (1-2 weeks)

The spec didn’t disappear. It moved from step 2 to step 6.

It comes after you know what you’re building instead of before.

How Boris Actually Works

Five terminal tabs. Each one has a parallel checkout of the repository. He starts Claude Code in plan mode in each one. Round-robins between them.

Goes back and forth on the plan until it’s right. Then with Opus 4.6, the implementation one-shots almost every time.

He also starts agents from his phone every morning before he’s at his desk. A third of his code might come from the iOS app.

He was candid about plan mode’s shelf life:

“Plan mode probably has a limited lifespan. Maybe in a month, no more need for plan mode.”

That’s wild to hear from the guy who built it. But it tracks with his core philosophy: don’t build for the model of today. Build for the model 6 months from now.

“All of Claude Code has just been written and rewritten and rewritten over and over and over. There is no part of Claude Code that was around 6 months ago.”

The numbers are real:

• Claude Code writes ~80% of code at Anthropic on average

• Boris writes 100% of his with Claude Code

• He ships 10-30 PRs daily, every day

• Opus introduced maybe 2 bugs in an entire month of coding. He estimates he would have introduced ~20 writing by hand.

• Productivity per engineer has increased 200% since Claude Code launched, even as Anthropic has tripled in headcount

To put that in context, Boris ran code quality at Meta. A 2% productivity gain there was “a year of work by hundreds of people.”

The Team Culture

Everyone at Anthropic has the same title: Member of Technical Staff. PMs code. Data scientists code. Designers code. The finance guy codes.

This happened organically. Boris walked into the office one day and saw a data scientist with Claude Code up on his monitor. Using it to run SQL queries with ASCII visualizations in the terminal.

Boris asked if he was dogfooding.

“No, I’m using it to run queries.”

The next week, the entire row of data scientists had Claude Code running. Then half the sales team. Then the finance team. Boris calls this latent demand, and he thinks it’s the single most important principle in product.

“You can never get people to do something they do not yet do. The thing you can do is find the intent that they have and then steer it to let them better capitalize on that intent.”

No PRDs on the Claude Code team. Cat Woo, who runs product, is extremely technical. The culture is “better send a PR.”

And the prototyping volume is staggering:

• Agent teams: “probably hundreds of versions” before shipping

• Condensed file view: ~30 prototypes, then a month of internal dogfooding

• Terminal spinner: “probably 50 maybe 100 iterations, and 80% of those didn’t ship”

• Plugins: built by Daisy running swarms over a weekend. She told the swarm to build plugins, come up with a spec, make an Asana board, split into tasks. It spawned a couple hundred agents. They made 100 tasks. Then they implemented it. “That’s pretty much the version of plugins that we shipped.”

Code Review Isn’t Dead

This isn’t a free-for-all.

Every PR at Anthropic is code-reviewed by Claude Code first, catching about 80% of bugs. Then a human engineer does the second pass. There are type checkers, linters, and the build runs. Claude Code even tests itself by launching itself in a subprocess.

Boris still automates lint rules in real time. Early in his career at Meta, he kept a spreadsheet: every time he left a code review comment about a recurring pattern, he tallied it. When a row hit 3-4 instances, he wrote a lint rule. Now he does the same thing but faster: he tags Claude directly on a coworker’s PR and asks it to write the lint rule.

“The more general model will always outperform the more specific model. We have a framed copy of the bitter lesson on the wall where the Claude Code team sits.”

There’s always an engineer approving the change before anything goes to production.

The Honest Caveat

I want to be straight about why this works at Anthropic specifically:

• Small, senior team where everyone shares deep context

• The product IS the AI tool they’re building with

• Boris built his taste over 15+ years shipping at Meta scale. He was one of the most prolific code authors and code reviewers at Instagram.

• Extraordinary hiring density. Dario’s own words: “You want a relatively small set of people where almost everyone you hire is really, really good.”

The prototype-first evaluation loop, the discipline of trying dozens or hundreds of versions, the culture of showing instead of writing, the code review automation layer. All of that translates to other companies.

But most teams aren’t 100% senior generalists at an AI company building their own AI tool. The no-PRD approach requires a team small enough to align through conversation and prototypes. So the full Anthropic playbook won’t work everywhere yet.

But here’s my take: this is where most fast-moving teams will be within 18 months. The tools are getting accessible enough. Cowork is Anthropic’s bet on bringing this to non-engineers. v0, Cursor, Replit, and Lovable are all pushing in the same direction.

The question isn’t if this workflow becomes standard. It’s when.

And Boris thinks coding is already largely solved.

“I think by the end of the year, everyone’s going to be a product manager and everyone codes. The title software engineer is going to start to go away. It’s just going to be replaced by builder.”

The PMs who start building these reps now will have a massive head start.

🔒 The rest of this post is for paid subscribers. I’m going deep on the Taste at Speed Framework (5 Lenses), how to build this skill regardless of your starting point, where the PRD fits now, and a full real-world teardown showing good vs. bad prototype evaluation side by side. Plus 4 downloadable templates.

Read more