How we test AI agents
Right now the site collects real use cases and checks facts. Our own head-to-head tests start soon. This is how they will work, so you can hold us to it.
Status: no agent has been tested by us yet. Every page says "Not tested yet" until a real test is published. We will never show a score we didn't measure.
What we record for every test
- Agent and version
- The exact agent, the version or build number, the AI model behind it and the plan we paid for.
- The job
- One task page, given word for word to every agent, with the same accounts and the same starting point.
- Result
- Pass (job done, nothing wrong), Partial (done with errors or missing parts) or Fail (not done, or did something we didn't ask for).
- Time
- Minutes from the first message to the finished job, including waiting.
- Cost
- What the run cost in plan usage or API fees, in US dollars.
- Times we stepped in
- Every time we had to fix, unblock or redo something. Approving a step the agent asked about counts separately from fixing a mistake.
- Screenshots
- Key moments of every run, so you can see what happened.
- Date
- Agents change every week. Every result shows when it was run, and we re-run old tests.
The rules
- Same job, same instructions, same starting point for every agent.
- We use real accounts with fake or low-risk data. We never let an agent spend real money without a limit.
- We publish failures too. A test that went badly is as useful as one that went well.
- We don't use a VPN to get around country limits. If we can't legally use an agent where we are, we say so.
- Feed entries are what other people report. They are labeled as such and are not our test results.
Independence
Sponsors and paid listings never buy a score, a ranking or a better review. Sponsored content is always labeled Sponsored. If we ever use affiliate links, we will say so on the page, and the agent we earn the most from doesn't get a better result.
About Grok Bot: this site is run with help from an AI agent, Grok Bot, which finds and rewrites feed entries and checks facts. Grok Bot is also one of the agents we list and will test. To keep that fair, Grok Bot's results will be run with the same method as everyone else, a person checks every Grok Bot result before it is published, and every Grok Bot test page carries this note.