Agent Benchmarking: Search + Calculator

Here we go over how to benchmark performance of an agent on tasks where it has access to a calculator and a search tool.

It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.

# Comment this out if you are NOT using tracing
import os
os.environ["LANGCHAIN_HANDLER"] = "langchain"

Loading the data

First, let’s load the data.

from langchain.evaluation.loading import load_dataset
dataset = load_dataset("agent-search-calculator")

Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--agent-search-calculator-8a025c0ce5fb99d2/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)

Setting up a chain

Now we need to load an agent capable of answering these questions.

from langchain.llms import OpenAI
from langchain.chains import LLMMathChain
from langchain.agents import initialize_agent, Tool, load_tools

tools = load_tools(['serpapi', 'llm-math'], llm=OpenAI(temperature=0))
agent = initialize_agent(tools, OpenAI(temperature=0), agent="zero-shot-react-description")

Make a prediction

First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints

agent.run(dataset[0]['question'])

'38,630,316 people live in Canada as of 2023.'

Make many predictions

Now we can make predictions

predictions = []
predicted_dataset = []
error_dataset = []
for data in dataset:
    new_data = {"input": data["question"], "answer": data["answer"]}
    try:
        predictions.append(agent(new_data))
        predicted_dataset.append(new_data)
    except Exception:
        error_dataset.append(new_data)

Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised APIConnectionError: Error communicating with OpenAI: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer')).

Evaluate performance

Now we can evaluate the predictions. The first thing we can do is look at them by eye.

predictions[0]

{'input': 'How many people live in canada as of 2023?',
 'answer': 'approximately 38,625,801',
 'output': '38,630,316 people live in Canada as of 2023.',
 'intermediate_steps': [(AgentAction(tool='Search', tool_input='Population of Canada 2023', log=' I need to find population data\nAction: Search\nAction Input: Population of Canada 2023'),
   '38,630,316')]}

Next, we can use a language model to score them programatically

from langchain.evaluation.qa import QAEvalChain

llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(dataset, predictions, question_key="question", prediction_key="output")

We can add in the graded output to the predictions dict and then get a count of the grades.

for i, prediction in enumerate(predictions):
    prediction['grade'] = graded_outputs[i]['text']

from collections import Counter
Counter([pred['grade'] for pred in predictions])

Counter({' CORRECT': 4, ' INCORRECT': 6})

We can also filter the datapoints to the incorrect examples and look at them.

incorrect = [pred for pred in predictions if pred['grade'] == " INCORRECT"]

incorrect[0]

{'input': "who is dua lipa's boyfriend? what is his age raised to the .43 power?",
 'answer': 'her boyfriend is Romain Gravas. his age raised to the .43 power is approximately 4.9373857399466665',
 'output': "Isaac Carew, Dua Lipa's boyfriend, is 36 years old and his age raised to the .43 power is 4.6688516567750975.",
 'intermediate_steps': [(AgentAction(tool='Search', tool_input="Dua Lipa's boyfriend", log=' I need to find out who Dua Lipa\'s boyfriend is and then calculate his age raised to the .43 power\nAction: Search\nAction Input: "Dua Lipa\'s boyfriend"'),
   'Dua and Isaac, a model and a chef, dated on and off from 2013 to 2019. The two first split in early 2017, which is when Dua went on to date LANY ...'),
  (AgentAction(tool='Search', tool_input='Isaac Carew age', log=' I need to find out Isaac\'s age\nAction: Search\nAction Input: "Isaac Carew age"'),
   '36 years'),
  (AgentAction(tool='Calculator', tool_input='36^.43', log=' I need to calculate 36 raised to the .43 power\nAction: Calculator\nAction Input: 36^.43'),
   'Answer: 4.6688516567750975\n')],
 'grade': ' INCORRECT'}