I build software that reads papers and code, runs experiments, and writes up what it found. Before you see the write-up, it goes back through its own answer and marks every claim it could not back up with a source. Most AI tools are built to sound confident. This one is built to show you where it is thin.
Read about ARESIf your business is the model, an agent that keeps saying "I could not check this" is a bad advert for you. The companies best placed to build this have the least reason to. Better models will not change that. A wrong answer that reads well is harder to spot, not easier.
Evaluation tools get mentioned in the same breath, and they do a different job. They score a model across a test set, afterwards. They cannot tell you whether the paragraph in front of you right now is supported.
Nobody publishes the number that would settle it either. I have not found one public false-positive rate from anyone making these claims. The nearest public benchmark, Vectara's on the RAGTruth corpus, found that automatic detectors, including GPT-4 acting as a judge, could not get past 50 percent precision and 50 percent recall at once. So I am not promising a smarter detector. I am saying you should be able to open the file and look.
I am fine sitting here for reasons that predate AI. In spacecraft work, telling someone "this does not pass" is ordinary. It is most of the job.
Same engine underneath, and the same question every time: what is this claim resting on, and does that hold up?
The whole thing, in a browser. You ask a question, it goes and works, and you get a report with every claim already marked. There is a real run of it you can walk through.
The same engine from a terminal, for putting it in a pipeline.
ares run starts a task. ares watch follows it live.
ares message redirects a run that is already going.
ares knowledge searches what earlier runs concluded.
You register whichever model providers you use and switch with
ares llm activate. The part that matters: the model that critiques the
work does not have to be the model that did it. Point the critic at a different
provider and you get a real second opinion, instead of a model marking its own
homework.
This is not one call to a model. It plans, uses tools, writes code and runs it in a locked-down sandbox, and keeps going until it has something. One run in the demo took fourteen tool calls. Then it splits its own write-up into separate claims and checks each one against whatever it actually used.
When it has gone about something the wrong way, it says so and tries again. Here is what it wrote about itself during a recorded run:
The agent relied exclusively on semantic search, which returned zero hits for the exact phrase "acceptance contract", and prematurely concluded the information was missing. It failed to fall back to exact-match tools such as grep, or explore the codebase structure to locate the relevant contract definition.
A reviewer would have written the same note. This one wrote itself, before anyone had read the output. The time you save is not the writing. It is the checking you would have done afterwards, and the second attempt you would have had to ask for.
I built AI systems for scientific work at JPL and at the Frontier Development Lab. They could usually do the task. The trouble was that you could not see where an answer had come from.
A long piece of reasoning arrives as a conclusion, and nothing in it tells you which step the conclusion depends on. So someone redoes the work by hand to check. Or nobody does, and it gets used anyway.
Neither of those is fine when a spacecraft is downstream.