Business intelligence (BI) underpins enterprise decision-making and is widely used in tools such as Power BI and Tableau. In traditional workflows, users must first identify relevant tables, perform data transformations, and build join relationships before they can answer a business question — steps that are complex and time-consuming.
To study whether large language models (LLMs) can handle this process end-to-end, researchers harvested a large collection of real-world BI projects from public sources and manually extracted pairs of questions and ground-truth answers from real user dashboards. The result is BI-Bench, described as the first benchmark to systematically evaluate LLMs on end-to-end BI.
Even frontier LLMs perform poorly on BI-Bench, achieving less than 50% accuracy. To address these limitations, the team designed BI-Agent, a tool-augmented agent that decomposes BI workflows into subtasks on structured data — such as search, join, and transform — and orchestrates specialized data management methods across BI stages.
The researchers also developed a post-training framework that synthesizes training trajectories from real BI projects, enabling BI-Agent to be further trained with supervised fine-tuning (SFT) and reinforcement learning (RL).
According to the paper, BI-Agent delivers accuracy gains of up to 40 percentage points with vanilla LLMs, while the post-trained version yields gains of up to 30 points. The authors conclude that combining tool-augmented reasoning with domain-specific post-training is key to tackling complex BI workflows, and point to promising directions for future research.