Building a Data Agent — Part 1: I Thought the Agent Needed More Structure

I started these experiments wanting to understand how much of an AI agent's performance came from the language model and how much came from the code around it. If I gave a model a difficult data question, what would I need to provide for it to work out a reliable answer? An agent, in this project, could do more than reply with text. It could inspect a database, query it with SQL (a language for working with databases) or run a Python program, look at the output and choose another action. The harness was the code that made that possible. It decided which tools the model could use, what information came back from them, what work stayed available and how the agent could finish.

My first instinct was to give it more structure. A planner could decide how to approach the question, an analyst could investigate the data, and a reviewer could check the answer. I expected that arrangement to prevent mistakes. As I tried different versions, though, I kept seeing investigations that repeated themselves or spent a long time on answers that were still wrong.

A way to check the answers

I used DataAgentBench, a benchmark with 54 questions across 12 datasets. All of its questions require combining information from different databases. The labels used to identify the same records don't always match cleanly, and most questions also involve recovering facts from ordinary written text before using them in a calculation. The authors took public datasets and altered them to recreate problems described in enterprise customer interviews. Because they kept the original data, they could calculate expected answers and check the agent's output automatically. A pass in the results below means the answer met that benchmark's checks.

Here are three of the questions, slightly shortened, to give an idea of the work involved:

• Patents: Which technology areas in the Cooperative Patent Classification (CPC) have the highest exponential moving average of filings each year, with a smoothing factor of 0.2? Return only level-5 group codes whose best year is 2022. The agent has to recover dates from prose, connect patents to the classification hierarchy and calculate a smoothed filing trend, including years with no filings.

• Cancer research: Among living PanCancer Atlas patients with breast invasive carcinoma (BRCA), which three histological types have the highest percentage of CDH1 gene mutations? This means matching clinical and molecular records, grouping patients by their tissue-based diagnosis and calculating mutation percentages for the right patient group.

• Sales records: As of 25 November 2022, which agent had the highest sales for orders in the past five months? Calculate sales as quantity × unit price; an opportunity qualifies only if its contract's company signing date falls within the period, and return only the agent's ID. The agent has to connect orders, opportunities and contracts across databases, using the policy's signing date to decide which sales belong in the total.

This gave me a way to test the harness on questions with known answers, while leaving the agent to work out how to reach them. A wrong result could come from choosing the wrong records, misunderstanding a date or making an incorrect calculation. I could look through the run to see where it had gone wrong, rather than judge it just by how convincing the final response sounded. Alongside the official set of 54 questions, I used smaller local test sets to compare revisions. The main comparison in this post uses the same 17 local questions across successive versions; its scores belong to that test set. I'll cover the full official benchmark runs later in the series.

Why I started with separate agents

My early experiments used DeepSeek V4 Pro and Flash. In the earlier design, the planner started a separate data analyst agent to investigate, and a reviewer checked the proposed answer against the question and evidence. This seemed like a reasonable way to divide the work: deciding what to do, doing the analysis and checking whether it answered the question. It also meant thinking about what each agent needed from the previous one. The analyst needed enough context to carry out the plan, and the reviewer needed more than a final number to check the result. Those handoffs were part of the harness too.

The earlier workflow separated planning, analysis and review. This reconstruction shows those roles and handoffs; the benchmark results below belong to the later ReAct versions.

I also tried putting the investigation under one main analyst, which chose its next action after each tool result. This followed a ReAct loop: decide what to do, use a tool, look at the result and decide again. It could still ask a helper agent to do a specific job, but the helper couldn't create further agents. The measurements in this post start with these later ReAct versions. There was still plenty of supporting code. A component called the cartographer prepared a map of the available data, so the analyst could see what it had to work with. Specialised tools let it inspect databases, run SQL, use Python within limits set by the harness and extract information from text. The harness saved results and summaries, checked how records from different tables were joined, and let the agent ask a helper to challenge a claim.

The project was in Python, and the agent used Python and SQL for analysis too. LangChain provided the model integrations and tool interfaces, LangGraph ran the execution loop, and Deep Agents connected the model, tools and helpers through the managed runtime I was using. I used LangSmith to inspect the model and tool calls afterwards.

Each addition had a purpose. A data map could save exploration time, saved findings could prevent repeated work, and another agent might catch a misunderstanding. But the analyst also had to navigate all of this while answering the question: choose a specialised tool, save a finding in the expected way, request a check and eventually submit. I started to suspect that some of those requirements were giving it more ways to get stuck.

On the 17-question local set, changes intended to help the DeepSeek version finish reduced the main analyst's model calls from 820 to 535. Each model call is another turn in which the model receives the available information and decides what to do. The median time per question—the middle value when the times are put in order—fell from about 28 minutes to 14, but the pass rate stayed at 52.9%. Both versions also had runs hit the one-hour deadline. I had shortened many investigations without improving the overall score.

Getting a correct answer submitted

I still wanted to see whether a different model would get more of the questions right, so I chose GPT-5.6 Luna. The next few experiments used Luna while I continued changing the harness. Some were small tests on a single question, which made it easier to see what the agent was doing. One of those questions asked for the maximum adjusted closing price in 2020 for The RealReal, Inc. The agent needed to identify the company, select its prices for that year and calculate the maximum. An early Luna run answered correctly in 24 model calls and about 3.4 minutes.

I tried bringing the completion checks into a single readiness step. The agent could ask what still needed checking before it submitted. I wanted that to make finishing clearer, especially when the agent had run a calculation but still needed to show where the result came from. On the stock-price question, it saved the correct draft almost immediately. Then it spent much of the run trying to satisfy the readiness checks. It restated the SQL slightly differently from the SQL saved alongside the result, and the harness treated those differences as a problem. The answer hadn't changed, but the checks kept sending the agent back to repair its submission. The harness repeated its checks on the unchanged draft answer 12 times. The run took 51 model calls and about 6.2 minutes, compared with the earlier run's 24 calls and 3.4 minutes.

A simpler ReAct revision let the model finish without that sequence of completion steps. It answered the same question in seven calls and about a minute, including a small helper check. All three attempts got the answer right. On this question, the extra checking had added work after the correct answer was already available. That made me look more closely at what happened between finding a result and returning it. The model could have enough information to answer and still spend most of its time dealing with the way I'd asked it to finish.

The answer stayed the same across these three attempts. Tokens are the units of text models read and generate; the counts include information sent again on later calls. Time and token counts cover each whole run, including the work needed to finish and submit.

Letting the agent see its earlier work

The simpler version exposed another problem. It saved earlier results, but on each turn the model mainly saw the question, summaries of its findings and the latest tool output. The complete sequence of its investigation wasn't in view. This became noticeable on a music-revenue question: how much revenue in USD did Apple Music make from Beyoncé's song Get Me Bodied in Canada? The agent had to identify the song and connect it to the relevant sales, platform and country records. It needed to use information gathered through several queries to calculate the result. Instead, it kept asking for its query history and reading saved outputs to recover information it had already seen. I stopped that attempt after 33 model calls and about five minutes, with no answer submitted.

I changed the harness to keep the conversation and tool history in view. Large results appeared as short previews, and older tool outputs were shortened as the conversation approached the model's context limit—the amount of text it could receive in a call. The agent could now follow the sequence of its work without repeatedly asking to read it back. On the same music question, the full-history version reached a correct answer in eight model calls and about 51 seconds. That was one attempt per version, and the earlier one had been stopped, so I couldn't treat it as a general speed comparison. It did show a specific problem with how the harness had been presenting the investigation. I kept the fuller history for the next comparison. By then the harness had fewer completion steps, and I'd also added clearer instructions for accessing tables across databases.

The first run was stopped, so its bar shows how long I let it continue, not how long it would have needed to finish. Each version had one attempt at this question.

Did the changes improve the answers?

I ran that Luna version on the same 17 local questions I'd used with DeepSeek V4 Pro, without extra benchmark hints in either run. The chart below brings together the two DeepSeek revisions described earlier and this later Luna version. Average time per question fell from about 26 minutes to under four, and the median went from 14 minutes to about 80 seconds. I could get through experiments much faster. Token use fell too. For the 12 question attempts with token counts in both runs, average usage dropped from about 938,000 to 194,000 per task. Each was the first question from a different dataset. The older run was missing counts for five questions, so that comparison covers 12 matched attempts rather than all 17.

The pass rate, however, was still 52.9%. Luna got some questions right that the earlier version had missed, and missed others it had previously got right. The agent was doing less work overall, but on this set I wasn't getting more correct answers.

Time and pass rate cover all 17 questions; tokens cover 12 matched attempts. DeepSeek's questions ran in parallel, while Luna's ran one at a time through a different service. The model and harness changed too, so I can't give one change all the credit.

Some of Luna's runs were still slow. On a news question, part of the investigation involved assigning categories to 5,000 articles. Luna did that in batches of 20, taking 246 model calls and about 36 minutes to get a passing answer. Much of this work happened through helpers, so counting only the main analyst's calls would have missed a large part of what the run consumed.

Another dataset failed before the model had a turn. A preparation step was meant to summarise the available data, but the summary included large chunks of document text and grew into a 15.2 MB file. Saving it failed, so the investigation never started. On a ratings question, the agent averaged each business's average rating equally. That differed from the calculation behind the benchmark's expected answer, even with the fuller history. This time the investigation had finished; it had chosen the wrong calculation. Faster runs and better access to previous results hadn't solved every kind of failure.

Where I went next

I made another attempt to simplify the tools available to the agent. A rebuilt ReAct version offered seven broader tools for finding, inspecting, transforming and submitting data. The idea was to give the analyst fewer tool choices to navigate. On eight selected questions run three times each, with extra benchmark hints and Luna's reasoning setting on low, it passed 29.2%. This was a separate test set, so that score isn't directly comparable with the 17-question results. Those attempts used 547 model calls across the analyst and text-processing work, and about nine million tokens. The agent was still struggling to load and interpret the data, even with a smaller tool menu.

I had started by dividing planning, analysis and review into separate roles. Working through these runs made that division feel less straightforward. To check a calculation, the agent often needed to inspect the data again; what it found could change the approach it had planned to take. The stock and music questions also showed how much the harness could affect an investigation through its submission rules and the information it kept in view. I wanted to try giving the model more control over that back and forth. My next experiment was a persistent workspace with useful context, Python, SQL, a command line and the packages it needed. A single tool would let it run code in that workspace, with files and results that stayed available as it worked. The model could write an analysis, inspect the output and choose its next step. That was the idea behind my first one-tool agent, which I explore in Part 2: Giving the Model a Workspace.