// Generative AI · Agents · 2024
Most "chat with your data" demos look magical and quietly get the numbers wrong. This project takes the opposite stance: build an agentic analysis system where accuracy is the headline metric — combining multi-agent LangChain workflows, Tree-of-Thought prompting, and a careful benchmark of open-source models on real analysis tasks.
01 — The problem
Asking an LLM to "compute last quarter's churn" is easy to demo and hard to trust. The model may hallucinate a column, silently coerce a type, or answer confidently from a misread table. For a system feeding insights to managers and directors, a plausible-but-wrong number is worse than no answer at all.
So the design goal was inverted from the usual one: instead of maximizing how impressive the agent sounds, maximize how often its computed answer is verifiably correct — and make failures visible rather than fluent.
02 — Approach
Rather than asking a model to reason about data in its head, the system routes every quantitative question through a code-interpreter loop: the agent writes Python against the actual dataframe, executes it, and reads the real result back before answering. The natural-language layer never invents numbers — it only narrates what the code returned.
A planner decomposes the request, a coder writes and runs pandas/SQL, and a reviewer checks the output against the question before it reaches the user.
For ambiguous asks the agent expands several candidate interpretations, evaluates them, and keeps the branch whose executed result is internally consistent.
All math happens in executed code on the real data, so answers are reproducible and auditable rather than generated free-hand by the model.
Several open models were scored on identical analysis tasks to find the best accuracy-per-cost trade-off, instead of defaulting to the largest proprietary API.
03 — What it showed
The clearest finding was that the loop matters more than the model. Forcing every numeric claim through executed, reviewed code closed most of the gap between small open-source models and much larger ones — because the hard part was never the arithmetic, it was preventing the model from answering before it had verified anything.
In its applied form at Mercado Libre, the same pattern powered an agent that answered metric questions for managers, directors and above using advanced prompting (Tree-of-Thought) and multi-agent orchestration over LangChain.
04 — Stack