Essays on AI, data, systems, and product, from someone who ships them.https://soumyo.com/en-ushttps://soumyo.com/blog/making-hawkeye-claude-ready/https://soumyo.com/blog/making-hawkeye-claude-ready/Claude Haiku 5.5 couldn't run our Slack agent until we fixed one tool schema, and cost 4× until we turned on caching. Fixed, it still lost the classifier eval, and on a billed rerun it claimed work it never did.Wed, 28 Oct 2026 00:00:00 GMTaillmevaluationagentshttps://soumyo.com/blog/why-we-stayed-on-luna-5-6/https://soumyo.com/blog/why-we-stayed-on-luna-5-6/The newer, cheaper model lost on our real-traffic evals. How we pick the model behind Hawkeye, our Slack PM agent, and the setting that beat switching.Fri, 09 Oct 2026 00:00:00 GMTaillmevaluationagentshttps://soumyo.com/blog/beyond-95-accuracy-medical-transcription/https://soumyo.com/blog/beyond-95-accuracy-medical-transcription/Whisper looked solved on benchmarks. Then doctors used it. What it took to make medical transcription hold up in noisy clinics and ORs.Tue, 06 Jan 2026 00:00:00 GMTaihealthcarespeech-recognitionmachine-learningvoice-agentshttps://soumyo.com/blog/facts-hypotheses-and-the-art-of-writing-prds/https://soumyo.com/blog/facts-hypotheses-and-the-art-of-writing-prds/Every sentence in a PRD should either be a Fact or a Hypothesis. Avoid making Statements at all costs.Sat, 29 Nov 2025 00:00:00 GMTproduct-managementwritingmentorshipprdshttps://soumyo.com/blog/beyond-accuracy-rethinking-llm-evaluations/https://soumyo.com/blog/beyond-accuracy-rethinking-llm-evaluations/Why evaluating LLM apps takes more than accuracy scores: trustworthiness, safety, reliability, and the habit of improving them continuously.Mon, 07 Jul 2025 00:00:00 GMTaillmmachine-learningproduct-managementevaluationhttps://soumyo.com/blog/the-secret-to-mcp-servers-that-llms-love-using/https://soumyo.com/blog/the-secret-to-mcp-servers-that-llms-love-using/LLMs don't use APIs the way developers do. Why most MCP servers frustrate agents, and how product thinking builds tools they actually use well.Sun, 08 Jun 2025 00:00:00 GMTaimcpproduct-managementllmapi-designhttps://soumyo.com/blog/beyond-the-vacuum/https://soumyo.com/blog/beyond-the-vacuum/Exploring how original ideas emerge not from isolation, but from the rich interplay of diverse knowledge and experiencesFri, 14 Feb 2025 00:00:00 GMTcreativitywritingpersonal-growthoriginality