Physix Frontier
How to move AI answers from chat boxes into courtrooms
I spent the weekend tinkering with AI answer verification and hit quite a few pitfalls. It started when I saw the concept behind Simple AI v2: facts must be readable, and model generation is just an aid. An article on Lawfare also warned that courts face a dilemma where models can produce plausible text while confidently stating errors. You need to understand that large models are essentially probability machines guessing the next sentence based on context. Guessing smoothly doesn't mean guessing truthfully.
In principle, I want to put AI answers into a mock courtroom: treat the answer as the defendant, factual statements as charges, and sources as evidence. This is similar to annotation verification in computer vision; if a model gives a bounding box, you need to know why it's trustworthy. A factual statement is a single sentence that can be judged true or false independently. On day one, I did only three things.
First, create a new spreadsheet file on your computer, e.g., ai_evidence.csv. CSV is a comma-separated table format openable by Excel, WPS, or web spreadsheets. If you're unfamiliar, just click New -> Spreadsheet and draw five columns: Question, Factual Statement, Source, Status, Notes.
Second, open the web version of Claude or Codex. Log in, click New chat, and type a question you actually need to verify, such as how courts typically review AI-generated evidence. If you can't see the button, look for a plus sign at the top of the page. I've used Claude more this past month, and started comparing with Codex and Cursor in the last two weeks.
Finally, paste the following prompt after your question. A prompt is simply an instruction for the model.
Please break down the answer into factual statements that can be individually judged true or false. One statement per line, including the statement, supporting source, and whether the source is verifiable. If there is no specific source, write "None". Do not fabricate, and do not use comforting language.
After clicking Send, you'll see a list appear in the chat window. Expect at least 5 factual statements. Copy them one by one into your table. If there are fewer than 5, follow up asking for 3 more verifiable facts without repetition. The easiest pitfall on day one is the model writing "according to public information." This has no source, equivalent to saying "someone said so" in court. The fix is to add: Sources must be specific to institutions, filenames, or paper titles.
On day three, I ran a small comparison with two students in my group. The left side asked directly via chat; the right side followed the above process. The experiment design was simple: the variable was the workflow, and results were judged on two things—whether the answer could be saved, and whether sources could be clicked open. I put both groups' results in the same table, sorted by the number of verifiable items.
| Workflow | Action | What You See | Traceability |
|---|---|---|---|
| Chat Box | Q&A | A complete text block | Low |
| Evidence Table | Break into facts | One fact per row | High |
Papers like SCAENA discuss mixed human-machine interaction, where humans define evidence boundaries and models generate candidate answers. Duke's Judicature article also notes that when algorithms enter the courtroom, they must address both accuracy and explainability. In my tests, the chat box was faster, but hard to reproduce after three days; the evidence table was slower but left a path. The pitfall on day three was overly vague sources, like just a news headline. The solution is to accept only three types of sources: official announcements, papers, and judgment summaries. Mark uncertain ones as Unverifiable.
A week later, I standardized the process into four steps. Input the question, get a draft. The draft is the first version, don't rush to use it. Then break it into factual statements, one per line. Next, open the sources to confirm they truly support the statement. Only consider it preliminarily passed if the title, institution, and date are clear. Finally, output the final answer: Conclusion, Evidence, Unverified Items, Risks.
The final template looks like this:
Conclusion: .
Evidence: Fact A, Source X, Status Verifiable.
Unverified: Fact B, Source Unknown.
Risk: If used for formal materials, manual re-check required.
This process isn't pretty, but it's like lab notes. Beginners shouldn't aim for automation immediately; first make breaking down sentences and tagging sources a habit.
It forces AI to turn "I think" into "I can show you." Looking ahead, the real differentiator for AI products may lie in whose answers come with their own chain of evidence.
📌 This article is compiled from Hacker News, original source: https://simple.dev/blog/simple-ai-v2-answers-that-carry-their-own-proof/
All rights reserved. This is a compilation and independent analysis based on public reports.
