Agent VectorDB Question Answering Benchmarking#
Here we go over how to benchmark performance on a question answering task using an agent to route between multiple vectordatabases.
It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.
# Comment this out if you are NOT using tracing
import os
os.environ["LANGCHAIN_HANDLER"] = "langchain"
Loading the data#
First, letβs load the data.
from langchain.evaluation.loading import load_dataset
dataset = load_dataset("agent-vectordb-qa-sota-pg")
Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--agent-vectordb-qa-sota-pg-d3ae24016b514f92/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)
dataset[0]
{'question': 'What is the purpose of the NATO Alliance?',
'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',
'steps': [{'tool': 'State of Union QA System', 'tool_input': None},
{'tool': None, 'tool_input': 'What is the purpose of the NATO Alliance?'}]}
dataset[-1]
{'question': 'What is the purpose of YC?',
'answer': 'The purpose of YC is to cause startups to be founded that would not otherwise have existed.',
'steps': [{'tool': 'Paul Graham QA System', 'tool_input': None},
{'tool': None, 'tool_input': 'What is the purpose of YC?'}]}
Setting up a chain#
Now we need to create some pipelines for doing question answering. Step one in that is creating indexes over the data in question.
from langchain.document_loaders import TextLoader
loader = TextLoader("../../modules/state_of_the_union.txt")
from langchain.indexes import VectorstoreIndexCreator
vectorstore_sota = VectorstoreIndexCreator(vectorstore_kwargs={"collection_name":"sota"}).from_loaders([loader]).vectorstore
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.
Now we can create a question answering chain.
from langchain.chains import RetrievalQA
from langchain.llms import OpenAI
chain_sota = RetrievalQA.from_chain_type(llm=OpenAI(temperature=0), chain_type="stuff", retriever=vectorstore_sota, input_key="question")
Now we do the same for the Paul Graham data.
loader = TextLoader("../../modules/paul_graham_essay.txt")
vectorstore_pg = VectorstoreIndexCreator(vectorstore_kwargs={"collection_name":"paul_graham"}).from_loaders([loader]).vectorstore
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.
chain_pg = RetrievalQA.from_chain_type(llm=OpenAI(temperature=0), chain_type="stuff", retriever=vectorstore_pg, input_key="question")
We can now set up an agent to route between them.
from langchain.agents import initialize_agent, Tool
tools = [
Tool(
name = "State of Union QA System",
func=chain_sota.run,
description="useful for when you need to answer questions about the most recent state of the union address. Input should be a fully formed question."
),
Tool(
name = "Paul Graham System",
func=chain_pg.run,
description="useful for when you need to answer questions about Paul Graham. Input should be a fully formed question."
),
]
agent = initialize_agent(tools, OpenAI(temperature=0), agent="zero-shot-react-description", max_iterations=3)
Make a prediction#
First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints
agent.run(dataset[0]['question'])
'The purpose of the NATO Alliance is to promote peace and security in the North Atlantic region by providing a collective defense against potential threats.'
Make many predictions#
Now we can make predictions
predictions = []
predicted_dataset = []
error_dataset = []
for data in dataset:
new_data = {"input": data["question"], "answer": data["answer"]}
try:
predictions.append(agent(new_data))
predicted_dataset.append(new_data)
except Exception:
error_dataset.append(new_data)
Evaluate performance#
Now we can evaluate the predictions. The first thing we can do is look at them by eye.
predictions[0]
Next, we can use a language model to score them programatically
from langchain.evaluation.qa import QAEvalChain
llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(predicted_dataset, predictions, question_key="input", prediction_key="output")
We can add in the graded output to the predictions
dict and then get a count of the grades.
for i, prediction in enumerate(predictions):
prediction['grade'] = graded_outputs[i]['text']
from collections import Counter
Counter([pred['grade'] for pred in predictions])
Counter({' CORRECT': 19, ' INCORRECT': 14})
We can also filter the datapoints to the incorrect examples and look at them.
incorrect = [pred for pred in predictions if pred['grade'] == " INCORRECT"]
incorrect[0]
{'input': 'What is the purpose of the Bipartisan Innovation Act mentioned in the text?',
'answer': 'The Bipartisan Innovation Act will make record investments in emerging technologies and American manufacturing to level the playing field with China and other competitors.',
'output': 'The purpose of the Bipartisan Innovation Act is to promote innovation and entrepreneurship in the United States by providing tax incentives and other support for startups and small businesses.',
'grade': ' INCORRECT'}