cache#

Warning

Beta Feature!

Cache provides an optional caching layer for LLMs.

Cache is useful for two reasons:

  • It can save you money by reducing the number of API calls you make to the LLM provider if you’re often requesting the same completion multiple times.

  • It can speed up your application by reducing the number of API calls you make to the LLM provider.

Cache directly competes with Memory. See documentation for Pros and Cons.

Class hierarchy:

BaseCache --> <name>Cache  # Examples: InMemoryCache, RedisCache, GPTCache

Classes

cache.AsyncRedisCache(redis_, *[, ttl])

Cache that uses Redis as a backend.

cache.AzureCosmosDBSemanticCache(...[, ...])

Cache that uses Cosmos DB Mongo vCore vector-store backend

cache.CassandraCache([session, keyspace, ...])

Cache that uses Cassandra / Astra DB as a backend.

cache.CassandraSemanticCache([session, ...])

Cache that uses Cassandra as a vector-store backend for semantic (i.e. similarity-based) lookup.

cache.FullLLMCache(**kwargs)

SQLite table for full LLM Cache (all generations).

cache.FullMd5LLMCache(**kwargs)

SQLite table for full LLM Cache (all generations).

cache.GPTCache([init_func])

Cache that uses GPTCache as a backend.

cache.InMemoryCache()

Cache that stores things in memory.

cache.MemcachedCache(client_)

Cache that uses Memcached backend through pymemcache client lib

cache.MomentoCache(cache_client, cache_name, *)

Cache that uses Momento as a backend.

cache.OpenSearchSemanticCache(...[, ...])

Cache that uses OpenSearch vector store backend

cache.RedisCache(redis_, *[, ttl])

Cache that uses Redis as a backend.

cache.RedisSemanticCache(redis_url, embedding)

Cache that uses Redis as a vector-store backend.

cache.SQLAlchemyCache(engine, cache_schema)

Cache that uses SQAlchemy as a backend.

cache.SQLAlchemyMd5Cache(engine, cache_schema)

Cache that uses SQAlchemy as a backend.

cache.SQLiteCache([database_path])

Cache that uses SQLite as a backend.

cache.SingleStoreDBSemanticCache(embedding, *)

Cache that uses SingleStore DB as a backend

cache.UpstashRedisCache(redis_, *[, ttl])

Cache that uses Upstash Redis as a backend.

Deprecated classes

cache.AstraDBCache(*[, collection_name, ...])

cache.AstraDBSemanticCache(*[, ...])