What Is the Difference Between Oathe and Promptfoo?
Promptfoo red-teams your model before deploy. Oathe checks what agents actually did against what they said — at the level of tasks, while the work happens.
Promptfoo tests your model before you ship. Red teaming, evals, guardrails, CI. Good tools. Use them.
They test the agent. They can’t check the work.
An agent says it deployed the fix. Did it? Did it run the test it claims it ran? Is it about to run the deploy a second time because the retry forgot the first one? No pre-deploy scan answers these — they happen at the level of work and tasks, this afternoon, on your repo.
That’s the question Oathe answers: how do you know you can trust what an agent says it did vs. what it actually did?
Rules the work enforces
Oathe’s checks aren’t advice to the model. They’re enforced in the substrate, where nothing can route around them:
- Work is claimed before it’s done. A second claimant is refused.
- “Done” is an assertion. It settles only after a non-author verifies it.
- Self-verification is refused. The agent that did the work never judges it.
- Consequential actions carry receipts. A deploy or payment that already happened is refused on retry, not repeated.
- Rejected work stays open. Nothing quietly becomes true.
A prompt can’t argue with any of this. No model reads the request and decides. The state of the work decides.
The check itself
The definition of done is recorded before the agent starts. When the agent asserts completion, a verifier that didn’t write the code reads the recorded trajectory — what the agent actually did — and checks it against the task intent. It can rerun the deciding test. It can be a different harness than the one that did the work.
Pass, and the task settles. Fail, and it reopens. Either way, the answer is on the record.
We publish our own limits the same way: the roadmap says exactly which checks are enforced today, which are partial, and which are planned.
Before the work starts
Oathe Scan covers the other end: it runs a third-party skill in a sandbox and checks its behavior before your agent installs it. We audited 1,620 skills that way.
Why open source
Agents can’t trust each other’s work through a black box. Verification only means something when anyone can read how it works — every agent, every harness, every team. That’s why the runtime is open.
Test your model. Then verify the work.
npm install -g @oathe/oatheApache-2.0: github.com/oathe-ai/oathe. What Oathe is.