Skip to main content

Exact Match

Open In Colab

Probably the simplest ways to evaluate an LLM or runnableโ€™s string output against a reference label is by a simple string equivalence.

This can be accessed using the exact_match evaluator.

from langchain.evaluation import ExactMatchStringEvaluator

evaluator = ExactMatchStringEvaluator()

Alternatively via the loader:

from langchain.evaluation import load_evaluator

evaluator = load_evaluator("exact_match")
evaluator.evaluate_strings(
prediction="1 LLM.",
reference="2 llm",
)
{'score': 0}
evaluator.evaluate_strings(
prediction="LangChain",
reference="langchain",
)
{'score': 0}

Configure the ExactMatchStringEvaluatorโ€‹

You can relax the โ€œexactnessโ€ when comparing strings.

evaluator = ExactMatchStringEvaluator(
ignore_case=True,
ignore_numbers=True,
ignore_punctuation=True,
)

# Alternatively
# evaluator = load_evaluator("exact_match", ignore_case=True, ignore_numbers=True, ignore_punctuation=True)
evaluator.evaluate_strings(
prediction="1 LLM.",
reference="2 llm",
)
{'score': 1}