ChatModelUnitTests#

class langchain_tests.unit_tests.chat_models.ChatModelUnitTests[source]#

Base class for chat model unit tests.

Test subclasses must implement the chat_model_class and chat_model_params properties to specify what model to test and its initialization parameters.

Example:

from typing import Type

from langchain_tests.unit_tests import ChatModelUnitTests
from my_package.chat_models import MyChatModel


class TestMyChatModelUnit(ChatModelUnitTests):
    @property
    def chat_model_class(self) -> Type[MyChatModel]:
        # Return the chat model class to test here
        return MyChatModel

    @property
    def chat_model_params(self) -> dict:
        # Return initialization parameters for the model.
        return {"model": "model-001", "temperature": 0}

Note

API references for individual test methods include troubleshooting tips.

Test subclasses must implement the following two properties:

chat_model_class

The chat model class to test, e.g., ChatParrotLink.

Example:

@property
def chat_model_class(self) -> Type[ChatParrotLink]:
    return ChatParrotLink

chat_model_params

Initialization parameters for the chat model.

Example:

@property
def chat_model_params(self) -> dict:
    return {"model": "bird-brain-001", "temperature": 0}

In addition, test subclasses can control what features are tested (such as tool calling or multi-modality) by selectively overriding the following properties. Expand to see details:

enable_vcr_tests

Property controlling whether to enable select tests that rely on VCR caching of HTTP calls, such as benchmarking tests.

To enable these tests, follow these steps:

Override the enable_vcr_tests property to return True:

@property
def enable_vcr_tests(self) -> bool:
    return True

Configure VCR to exclude sensitive headers and other information from cassettes.

Important

VCR will by default record authentication headers and other sensitive information in cassettes. Read below for how to configure what information is recorded in cassettes.

To add configuration to VCR, add a conftest.py file to the tests/ directory and implement the vcr_config fixture there.

langchain-tests excludes the headers "authorization", "x-api-key", and "api-key" from VCR cassettes. To pick up this configuration, you will need to add conftest.py as shown below. You can also exclude additional headers, override the default exclusions, or apply other customizations to the VCR configuration. See example below:

tests/conftest.py#

import pytest
from langchain_tests.conftest import _base_vcr_config as _base_vcr_config

_EXTRA_HEADERS = [
    # Specify additional headers to redact
    ("user-agent", "PLACEHOLDER"),
]


def remove_response_headers(response: dict) -> dict:
    # If desired, remove or modify headers in the response.
    response["headers"] = {}
    return response


@pytest.fixture(scope="session")
def vcr_config(_base_vcr_config: dict) -> dict:  # noqa: F811
    """Extend the default configuration from langchain_tests."""
    config = _base_vcr_config.copy()
    config.setdefault("filter_headers", []).extend(_EXTRA_HEADERS)
    config["before_record_response"] = remove_response_headers

    return config

Run tests to generate VCR cassettes.
Example:
uv run python -m pytest tests/integration_tests/test_chat_models.py::TestMyModel::test_stream_time
This will generate a VCR cassette for the test in tests/integration_tests/cassettes/.

Important

You should inspect the generated cassette to ensure that it does not contain sensitive information. If it does, you can modify the vcr_config fixture to exclude headers or modify the response before it is recorded.

You can then commit the cassette to your repository. Subsequent test runs will use the cassette instead of making HTTP calls.

Testing initialization from environment variables

Some unit tests may require testing initialization from environment variables. These tests can be enabled by overriding the init_from_env_params property (see below):

Attributes

`chat_model_class`	The chat model class to test, e.g., `ChatParrotLink`.
`chat_model_params`	Initialization parameters for the chat model.
`enable_vcr_tests`	(bool) whether to enable VCR tests for the chat model.
`has_structured_output`	(bool) whether the chat model supports structured output.
`has_tool_calling`	(bool) whether the model supports tool calling.
`has_tool_choice`	(bool) whether the model supports tool calling.
`init_from_env_params`	(tuple) environment variables, additional initialization args, and expected instance attributes for testing initialization from environment variables.
`returns_usage_metadata`	(bool) whether the chat model returns usage metadata on invoke and streaming responses.
`structured_output_kwargs`	If specified, additional kwargs for with_structured_output.
`supported_usage_metadata_details`	(dict) what usage metadata details are emitted in invoke and stream.
`supports_anthropic_inputs`	(bool) whether the chat model supports Anthropic-style inputs.
`supports_audio_inputs`	(bool) whether the chat model supports audio inputs, defaults to `False`.
`supports_image_inputs`	(bool) whether the chat model supports image inputs, defaults to `False`.
`supports_image_tool_message`	(bool) whether the chat model supports ToolMessages that include image content.
`supports_image_urls`	(bool) whether the chat model supports image inputs from URLs, defaults to `False`.
`supports_json_mode`	(bool) whether the chat model supports JSON mode.
`supports_pdf_inputs`	(bool) whether the chat model supports PDF inputs, defaults to `False`.
`supports_video_inputs`	(bool) whether the chat model supports video inputs, defaults to `False`.
`tool_choice_value`	(None or str) to use for tool choice when used in tests.

Methods

`test_bind_tool_pydantic`(model, my_adder_tool)	Test that chat model correctly handles Pydantic models that are passed into `bind_tools`.
`test_init`()	Test model initialization.
`test_init_from_env`()	Test initialization from environment variables.
`test_init_streaming`()	Test that model can be initialized with `streaming=True`.
`test_init_time`(benchmark)	Test initialization time of the chat model.
`test_serdes`(model, snapshot)	Test serialization and deserialization of the model.
`test_standard_params`(model)	Test that model properly generates standard parameters.
`test_with_structured_output`(model, schema)	Test `with_structured_output` method.

test_bind_tool_pydantic( model: BaseChatModel, my_adder_tool: BaseTool, ) → None[source]#

Test that chat model correctly handles Pydantic models that are passed into bind_tools. Test is skipped if the has_tool_calling property on the test class is False.

Parameters:

model (BaseChatModel)
my_adder_tool (BaseTool)

Return type:

None

test_init() → None[source]#

Test model initialization. This should pass for all integrations.

Return type:: None

test_init_from_env() → None[source]#

Test initialization from environment variables. Relies on the init_from_env_params property. Test is skipped if that property is not set.

Return type:: None

test_init_streaming() → None[source]#

Test that model can be initialized with streaming=True. This is for backward-compatibility purposes.

Return type:: None

test_init_time( benchmark: BenchmarkFixture, ) → None[source]#

Test initialization time of the chat model. If this test fails, check that we are not introducing undue overhead in the model’s initialization.

Parameters:: benchmark (BenchmarkFixture)
Return type:: None

test_serdes( model: BaseChatModel, snapshot: SnapshotAssertion, ) → None[source]#

Test serialization and deserialization of the model. Test is skipped if the is_lc_serializable property on the chat model class is not overwritten to return True.

Parameters:

model (BaseChatModel)
snapshot (SnapshotAssertion)

Return type:

None

test_standard_params( model: BaseChatModel, ) → None[source]#

Test that model properly generates standard parameters. These are used for tracing purposes.

Parameters:: model (BaseChatModel)
Return type:: None

test_with_structured_output( model: BaseChatModel, schema: Any, ) → None[source]#

Test with_structured_output method. Test is skipped if the has_structured_output property on the test class is False.

Parameters:

model (BaseChatModel)
schema (Any)

Return type:

None