$ case ०३ · founder · Aug 2026 → present
AgentDiff
Catch the agent that gets the right answer the wrong way, before it merges.
- Product
- agentdiff.app
- My role
- Founder, sole engineer
- Status
- Pre-release · MIT
- Built with
- Python · GitHub Actions
Scroll to review the run
main · final answer
"You have 1,284 active users."
✓ assert answer is correct
pull request · final answer
"You have 1,284 active users."
✓ assert answer is correct
- 1 plan()
- 2 get_user_database_stats()
- 3 summarize()
- 1 plan()
- 2 get_user_database_stats()
- 3+get_user_database_stats()
- 4 summarize()
Illustrative run, modelled on the example on agentdiff.app.
$ pytest tests/test_agent.py
The test passed.
A prompt tweak ships. The agent still gives the right answer.
$ agentdiff baseline.json candidate.json
Line them up.
A baseline: your known-good run. A candidate: the run from your pull request.
agentdiff.toml
Values illustrative.
REGRESSION
Trajectory regressed against baseline.
Root cause: step 3, get_user_database_stats repeated.
Merge blocked until fixed or the baseline is updated.
$ agentdiff check --fail-on-regression
The gate.
Set the limits once in agentdiff.toml: drift, loops, cost, recovery.
$ agentdiff crewai_run.json langgraph_run.json
Bring your own traces.
No need to change your agent. AgentDiff reads the traces you already have.
No LLM judge
Same two traces, same verdict, every time.
No network
Nothing leaves your machine when you run a diff.
300+ tests
Across Python 3.10 to 3.13.
MIT
Open source, no API keys needed.
$ agentdiff --version
Boring, on purpose.
A test that sometimes passes is worse than no test.
वापरा
Try it
Two commands, no API keys.
Early testers, framework maintainers and design partners are welcome. Feedback from real agent runs shapes the next release.
Next story
IPO Allotment Engine
Millions of IPO bids land when bidding closes. Under SEBI's T+3 rule they must be validated, reconciled and allotted overnight, with zero tolerance for error.
Read it