Using AI to read closed-door meeting notes: don't treat summaries as conclusions

Over the weekend, I experimented with using AI to read Druckenmiller's closed-door meeting notes and fell into quite a few traps. For the past month, I've been running text tasks with OpenAI and DeepSeek, trying to quickly break down this transcript sourced from Huxiu into key arguments. Summarization went smoothly, but the pitfalls were numerous.

On day one, I copied the original text into the chat box and asked it to extract views on the Fed and AI. I didn't enable retrieval or source highlighting. It quickly gave me three sentences: rate cuts are no longer necessary, rising yields mean bond interest rates are going up, and AI may have a profit bubble where earnings growth can't keep up with valuations. The direction aligned with the original text.

The problems were in the details. The model described "reduced AI-related holdings to 20% of six months ago" as nearly liquidating everything, and added "the bubble is about to burst." The original text was actually more restrained, merely stating that the hype would eventually end. Summaries tend to compress tone, turning cautious opinions into strong judgments.

On day three, I changed my approach, asking it to do a comparison: what was the original quote, what was the paraphrase, and where might over-inference occur. This idea is common in open-source communities, similar to code reviews—don't merge directly, check the diff first. DeepSeek could pick out discrepancies in position ratios and tone, while OpenAI was better at adding background context, though that added context wasn't necessarily in the original text.

I also had it convert viewpoints into checklist items: dovish means leaning towards loose monetary policy/rate cuts—it's a policy judgment, not market consensus; a profit bubble is just a valuation reminder, not a denial of technology; specific info like "20% of six months ago" must be preserved and not eaten by the summary. Tools help me break judgments down into verifiable points, but they can't make the judgments for me.

A week later, my conclusion is: it depends. If you just want to read long transcripts faster, AI summaries are worth using. They're beginner-friendly and can translate terms like "dovish" or "yields" into plain language. But using them for trading or pasting them directly into reports carries high risk. A summary without cross-referencing the original shouldn't be merged directly into conclusions. If made open-source, this project might attract maintainers on GitHub, given the good community vibe. I'll continue using it for rough screening, but key numbers and tones still need verification against the original text.