JsonValidityEvaluator#

class langchain.evaluation.parsing.base.JsonValidityEvaluator(**kwargs: Any)[source]#

Evaluate whether the prediction is valid JSON.

This evaluator checks if the prediction is a valid JSON string. It does not

require any input or reference.

Parameters:

kwargs (Any)

requires_input#

Whether this evaluator requires an input string. Always False.

Type:

bool

requires_reference#

Whether this evaluator requires a reference string. Always False.

Type:

bool

evaluation_name#

The name of the evaluation metric. Always “json”.

Type:

str

Examples

>>> evaluator = JsonValidityEvaluator()
>>> prediction = '{"name": "John", "age": 30, "city": "New York"}'
>>> evaluator.evaluate(prediction)
{'score': 1}

>>> prediction = '{"name": "John", "age": 30, "city": "New York",}'
>>> evaluator.evaluate(prediction)
{'score': 0, 'reasoning': 'Expecting property name enclosed in double quotes'}

Attributes

evaluation_name

The name of the evaluation.

requires_input

Whether this evaluator requires an input string.

requires_reference

Whether this evaluator requires a reference label.

Methods

__init__(**kwargs)

aevaluate_strings(*, prediction[, ...])

Asynchronously evaluate Chain or LLM output, based on optional input and label.

evaluate_strings(*, prediction[, reference, ...])

Evaluate Chain or LLM output, based on optional input and label.
__init__(**kwargs: Any) → None[source]#
Parameters:

kwargs (Any)

Return type:

None

async aevaluate_strings(*, prediction: str, reference: str | None = None, input: str | None = None, **kwargs: Any) → dict#

Asynchronously evaluate Chain or LLM output, based on optional input and label.

Parameters:

  • prediction (str) – The LLM or chain prediction to evaluate.

  • reference (Optional[str], optional) – The reference label to evaluate against.

  • input (Optional[str], optional) – The input to consider during evaluation.

  • kwargs (Any) – Additional keyword arguments, including callbacks, tags, etc.

Returns:

The evaluation results containing the score or value.

Return type:

dict

evaluate_strings(*, prediction: str, reference: str | None = None, input: str | None = None, **kwargs: Any) → dict#

Evaluate Chain or LLM output, based on optional input and label.

Parameters:

  • prediction (str) – The LLM or chain prediction to evaluate.

  • reference (Optional[str], optional) – The reference label to evaluate against.

  • input (Optional[str], optional) – The input to consider during evaluation.

  • kwargs (Any) – Additional keyword arguments, including callbacks, tags, etc.

Returns:

The evaluation results containing the score or value.

Return type:

dict