Can LLMs Learn Expert Investment Judgment From Human Annotations?

An investor’s edge comes from judgment, not raw information, and that judgment is hard to teach to humans or AI. Thinking Machines focused on one slice of the problem: filtering and processing financial documents to surface what’s relevant to investment decisions. Their finding: with high-quality human annotations, LLMs can be trained to interpret text with expert-level taste, with a proprietary model outperforming the frontier models they tested on information accuracy and recall at a fraction of the cost.

Why investing is harder than it looks

Outperforming the market is hard. When every investor has access to the same sources of public information, alpha must come from unique insight built on taste and judgment. A strong investor’s judgment is difficult to articulate and teach directly, whether the audience is human or AI. It comes from experience.

Even when you decompose an investor’s job into its simplest constituent tasks, those tasks turn out to be surprisingly difficult for LLMs. Thinking Machines focused on one specific special case: filtering and processing financial documents to surface information relevant to investment decisions.

Investors are bombarded every day with news articles, research reports, company documents, emails, internal write-ups, and more. Reading is the easy part. The real work happens in the small, repeated judgments layered over reading: filtering, interpreting, segmenting, and identifying where the useful signal lies. These judgments are embedded throughout an investor’s daily workflow and consume substantial time.

Can LLMs be taught financial judgment?

The team asked a focused question: can you teach LLMs financial judgment? Their finding is that with high-quality human annotations, LLMs can be trained to interpret text with expert-level taste and judgment. Their proprietary model outperforms all frontier models they tested on information accuracy and recall, at a fraction of the cost.

They describe the training process and results on a subset of data cleared for public release, then sketch a broader vision of differentiated intelligence, with models tuned for specific organizational needs.

How were the models tested?

Models were evaluated on six information filtering tasks drawn from investors’ daily workflows. Beyond those six, Thinking Machines reports that they have many other internal tasks showing similar patterns: frontier models underperform compared to their internally trained models.

Accuracy was measured as the percentage of documents correctly labeled according to the firm’s investors. For classification tasks, they also calculated the F1 score.

The benchmark task structure

• 01 Financial Article Relevancy: Given a financial article, classify whether it is relevant to a C-suite investment professional. Evaluated with F1 score and accuracy.

The article excerpt outlines the setup and the first task definition; additional task definitions and full results appear in the source post. Thinking Machines’ central argument is that expert-annotated data can move LLM performance past generic frontier baselines on narrow, high-judgment workflows like financial document triage.

FAQ

What did Thinking Machines build?

Thinking Machines trained an LLM on high-quality human annotations from investors to interpret financial text with expert-level judgment, and the proprietary model outperformed every frontier model they tested on accuracy and recall for a fraction of the cost.

What tasks did they test the models on?

Six information filtering tasks drawn from investors’ daily workflows, with accuracy measured as the percentage of documents correctly labeled according to the firm’s investors and F1 score calculated for classification tasks.

Why focus on document filtering instead of broader investing tasks?

Document filtering is one of the simplest, most repeated judgments in an investor’s workflow, yet even this task proved surprisingly difficult for general-purpose LLMs, which is why Thinking Machines isolated it to demonstrate the value of expert annotation.