--- title: "Sonnet 5 review: I ran 64 generations to find out if it's worth it" author: "Claire Vo" published_at: "2026-06-30T23:22:23+00:00" link: "https://www.lennysnewsletter.com/p/sonnet-5-review-i-ran-64-generations" feed: "Lenny's Newsletter" clawfeed: "https://agent.clawfeeds.com/feed/nix6-rmxo-oukf.md" feed_url: "https://agent.clawfeeds.com/feed/nix6-rmxo-oukf.md" --- # Sonnet 5 review: I ran 64 generations to find out if it's worth it I’ve been testing every major frontier model release since the start of the year, and when Anthropic dropped Sonnet 5, I wanted more than a vibe check. I got tired of one-off tests I couldn’t repeat or compare over time, so I built something better: the How I AI Bench, a repeatable eval harness I constructed live using Claude Code while recording this episode. I ran Sonnet 5 blind against four other frontier models (Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro) across PRD quality, prototype generation, agentic task completion, and agent personality. The results were not what I expected. **Listen or watch on [YouTube](https://youtu.be/yJ-1LB2hF-Q), [Spotify](https://open.spotify.com/show/4aRP2XSavdtrLG5FZoonOK), or [Apple Podcasts](https://podcasts.apple.com/us/podcast/how-i-ai/id1809663079)** ### What you’ll learn: 1. What Anthropic claims Sonnet 5 improves over Sonnet 4.6, and where the benchmark data actually backs that up 2. How I built the How I AI Bench in under 45 minutes using Claude Code, starting from my own stored session history 3. Why I combined human vibe scoring (70%) with LLM as judge scoring (30%) instead of trusting either alone 4. How to set up a local HTML scoring page so you can rate AI outputs on gut feel and export those scores as JSON 5. Which model I recommend for PRDs, which for complex prototypes, and which for chatting with an agent daily --- ### Brought to you by: [![](https://substackcdn.com/image/fetch/%24s_!nBz4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff53d9bc9-15aa-47f7-942a-2d095fa18c69_1600x114.png)](https://substackcdn.com/image/fetch/%24s_!nBz4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff53d9bc9-15aa-47f7-942a-2d095fa18c69_1600x114.png) **[Runway](https://runwayml.com/howIAI)**—The creative AI platform for images, video and more **[Hyperagent](https://hyperagent.com/howiai)**—Deploy fleets of agents that handle real work ### In this episode, we cover: ([00:00](https://www.youtube.com/watch?v=yJ-1LB2hF-Q)) Sonnet 5 is out ([01:55](https://www.youtube.com/watch?v=yJ-1LB2hF-Q&t=115s)) What Anthropic claims ([04:02](https://www.youtube.com/watch?v=yJ-1LB2hF-Q&t=242s)) Why I’m done with one-off vibe checks ([05:05](https://www.youtube.com/watch?v=yJ-1LB2hF-Q&t=305s)) Building the How I AI Bench live with Claude Code ([07:42](https://www.youtube.com/watch?v=yJ-1LB2hF-Q&t=462s)) The scoring system ([10:43](https://www.youtube.com/watch?v=yJ-1LB2hF-Q&t=643s)) Agent voice eval ([11:57](https://www.youtube.com/watch?v=yJ-1LB2hF-Q&t=717s)) Quick recap ([13:58](https://www.youtube.com/watch?v=yJ-1LB2hF-Q&t=838s)) Results: The How I AI index leaderboard ([21:21](https://www.youtube.com/watch?v=yJ-1LB2hF-Q&t=1281s)) What I’m improving for the next run ([22:16](https://www.youtube.com/watch?v=yJ-1LB2hF-Q&t=1336s)) Generating a Claire-weighted index ([23:53](https://www.youtube.com/watch?v=yJ-1LB2hF-Q&t=1433s)) Model-by-task recommendations ### Tools referenced: • Claude Sonnet 5: • Claude Opus 4.8: • GPT-5.5 (OpenAI): • Gemini 3 Pro (Google DeepMind): • Cursor: ### Other references: • SWE-bench Pro (agentic coding benchmark referenced): ### Where to find Claire Vo: ChatPRD: Website: LinkedIn: X: Production and marketing by . For inquiries about sponsoring the podcast, email jordan@penname.co.