An investor’s edge comes from judgment, not raw information, and that judgment is hard to teach to humans or AI. Thinking Machines focused on one slice of the problem: filtering and processing financial documents to surface what’s relevant to investment decisions. Their finding: with high-quality human annotations, LLMs can be trained to interpret text with expert-level taste, with a proprietary model outperforming the frontier models they tested on information accuracy and recall at a fraction of the cost.<\/p>\n
Experience shapes an investor’s judgment over time.<\/p>\n
Even when you decompose an investor’s job into its simplest constituent tasks, those tasks turn out to be surprisingly difficult for LLMs. Thinking Machines focused on one specific special case: filtering and processing financial documents to surface information relevant to investment decisions.<\/p>\n
These judgments take up significant time in an investor’s daily workflow.<\/p>\n
The team asked whether LLMs can be taught financial judgment. LLMs trained on high-quality human annotations can interpret financial text with expert-level judgment. Their proprietary model outperformed every frontier model they tested on accuracy and recall, costing less than each of them.<\/p>\n
They describe training and results on a subset of data cleared for public release, then outline their approach to tailoring models to specific organizational needs.<\/p>\n
Models were evaluated on six information filtering tasks drawn from investors’ daily workflows. Beyond those six, Thinking Machines reports that they have many other internal tasks showing similar patterns: frontier models underperform compared to their internally trained models.<\/p>\n
Accuracy was measured as the percentage of documents correctly labeled according to the firm’s investors. For classification tasks, they also calculated the F1 score.<\/p>\n
The article excerpt outlines the setup and the first task definition; additional task definitions and full results appear in the source post. Thinking Machines’ central argument is that expert-annotated data can move LLM performance past generic frontier baselines on narrow, high-judgment workflows like financial document triage.<\/p>\n
Thinking Machines trained an LLM on high-quality human annotations from investors to interpret financial text with expert-level judgment, and the proprietary model outperformed every frontier model they tested on accuracy and recall for a fraction of the cost.<\/p>\n
Six information filtering tasks drawn from investors’ daily workflows, with accuracy measured as the percentage of documents correctly labeled according to the firm’s investors and F1 score calculated for classification tasks.<\/p>\n
Document filtering is one of the simplest, most repeated judgments in an investor’s workflow, yet even this task proved surprisingly difficult for general-purpose LLMs, which is why Thinking Machines isolated it to demonstrate the value of expert annotation.<\/p>\n