Curated resources: AI Product x How to eval?

List of resouces for building a solid eval pipeline for your AI product

UpdatedMarch 9, 2025

•2 min read•View as Markdown

Curated resources: AI Product x How to eval?

D

Dhaval Singh

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

Current Agentic Development Workflow(June)

How my development setup looks like using agents

Jun 3, 20262 min read217

Current Agentic Development Workflow(June)

The cost of doing evals has gone down substantially...

How to use coding agents to accelerate your evals

May 19, 20264 min read66

The cost of doing evals has gone down substantially...

K.I.S.S(Keep It Static, Stupid): System prompt ft. caching

How to build your system prompt to optimise for prompt caching & practical insights

Apr 22, 20265 min read339

K.I.S.S(Keep It Static, Stupid): System prompt ft. caching

Not All Caches Are Equal: Claude, OpenAI, and Gemini

We focus quite a bit on prompt caching @LittlebirdAI to ensure lower latencies and cost. But it's very tricky to get it right, esp when you deal with multiple providers. There are quite a few really g

Apr 22, 20262 min read120

Not All Caches Are Equal: Claude, OpenAI, and Gemini

Some notes on Agentic search & Turbopuffer

Practical pointers for doing Agentic, Hybrid search & Turbopuffer quirks

Aug 31, 20256 min read600

Some notes on Agentic search & Turbopuffer

Dhaval Singh's Blog

46 posts

Founding Engineer @ Seezo.io

We at Seezo have been trying our best to make evals a first class citizen in our SDLC of our LLM backed product. The end goal is to have a robust pipeline that integrates seamlessly in your normal developemnt cycle. This is even more difficult if you have a non-chatbot usecase, since most of the ready to use OSS/Paid tools cater to this specific usecase and it’s pretty hard to customize them for your needs.

From annotating to dataset managment to prompt versioning to analyzing eval results over time, there are a lot of moving parts and it can be very confusing for anyone to wrap their head around how to think about it, what tools to use, how to integrate them, etc. Hence I keep reading about evals quite a bit on a regular basis. There are a lot of articles and guides out there, but very few of them are non-AI slop and useful.

Hence here is a curated list of resouces you should focus on if you are learning or trying to build eval for your own usecase :)

PS- This is mostly focused on evals for products using LLMs and not evaluating LLMs for different tasks.