VertexPairWiseStringEvaluator

class langchain_google_vertexai.evaluators.evaluation.VertexPairWiseStringEvaluator(metric: str, **kwargs)

Evaluate the perplexity of a predicted string.

Attributes

requires_input	Whether this evaluator requires an input string.
requires_reference	Whether this evaluator requires a reference label.

Methods

__init__(metric, **kwargs)
aevaluate_string_pairs(*, prediction, ...[, ...])	Asynchronously evaluate the output string pairs.
evaluate_string_pairs(*, prediction, ...[, ...])	Evaluate the output string pairs.

Parameters:

metric (str) –

__init__(metric: str, **kwargs)

Parameters:

metric (str) –

async aevaluate_string_pairs(*, prediction: str, prediction_b: str, reference: str | None = None, input: str | None = None, **kwargs: Any) → dict

Asynchronously evaluate the output string pairs.

Parameters:

• prediction (str) – The output string from the first model.

• prediction_b (str) – The output string from the second model.

• reference (Optional[str], optional) – The expected output / reference string.

• input (Optional[str], optional) – The input string.

• **kwargs – Additional keyword arguments, such as callbacks and optional reference strings.

Returns:

A dictionary containing the preference, scores, and/or other information.

Return type:

dict

evaluate_string_pairs(*, prediction: str, prediction_b: str, reference: str | None = None, input: str | None = None, **kwargs: Any) → dict

Evaluate the output string pairs.

Parameters:

• prediction (str) – The output string from the first model.

• prediction_b (str) – The output string from the second model.

• reference (Optional[str], optional) – The expected output / reference string.

• input (Optional[str], optional) – The input string.

• **kwargs – Additional keyword arguments, such as callbacks and optional reference strings.

Returns:

A dictionary containing the preference, scores, and/or other information.

Return type:

dict