BigQueryVectorStore#

class langchain_google_community.bq_storage_vectorstores.bigquery.BigQueryVectorStore[source]#

Bases: BaseBigQueryVectorStore

A vector store implementation that utilizes BigQuery and BigQuery Vector Search.

This class provides efficient storage and retrieval of documents with vector embeddings within BigQuery. It is particularly indicated for prototyping, due the serverless nature of BigQuery, and batch retrieval. It supports similarity search, filtering, and batch operations through batch_search method. Optionally, this class can leverage a Vertex AI Feature Store for online serving through the to_vertex_fs_vector_store method.

embedding#: Embedding model for generating and comparing embeddings.

project_id#: Google Cloud Project ID where BigQuery resources are located.

dataset_name#: BigQuery dataset name.

table_name#: BigQuery table name.

location#: BigQuery region/location.

content_field#: Name of the column storing document content (default: “content”).

embedding_field#: Name of the column storing text embeddings (default: “embedding”).

doc_id_field#: Name of the column storing document IDs (default: “doc_id”).

credentials#: Optional Google Cloud credentials object.

embedding_dimension#: Dimension of the embedding vectors (inferred if not provided).

distance_type#

The distance metric used for similarity search. Defaults to “EUCLIDEAN”.

Type:: Literal[“COSINE”, “EUCLIDEAN”]

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

param content_field: str = 'content'#

param credentials: Any | None = None#

param dataset_name: str [Required]#

param distance_type: Literal['COSINE', 'EUCLIDEAN'] = 'EUCLIDEAN'#

param doc_id_field: str = 'doc_id'#

param embedding: Embeddings [Required]#

param embedding_dimension: int | None = None#

param embedding_field: str = 'embedding'#

param extra_fields: Dict[str, str] | None = None#

param location: str [Required]#

param project_id: str [Required]#

param table_name: str [Required]#

param table_schema: Any = None#

async aadd_documents(documents: list[Document], **kwargs: Any) → list[str]#

Async run more documents through the embeddings and add to the vectorstore.

Parameters:

documents (list[Document]) – Documents to add to the vectorstore.
kwargs (Any) – Additional keyword arguments.

Returns:

List of IDs of the added texts.

Raises:

ValueError – If the number of IDs does not match the number of documents.

Return type:

list[str]

async aadd_texts(texts: Iterable[str], metadatas: list[dict] | None = None, **kwargs: Any) → list[str]#

Async run more texts through the embeddings and add to the vectorstore.

Parameters:

texts (Iterable[str]) – Iterable of strings to add to the vectorstore.
metadatas (list[dict] | None) – Optional list of metadatas associated with the texts. Default is None.
**kwargs (Any) – vectorstore specific parameters.

Returns:

List of ids from adding the texts into the vectorstore.

Raises:

ValueError – If the number of metadatas does not match the number of texts.
ValueError – If the number of ids does not match the number of texts.

Return type:

list[str]

add_documents(documents: list[Document], **kwargs: Any) → list[str]#

Add or update documents in the vectorstore.

Parameters:

documents (list[Document]) – Documents to add to the vectorstore.
kwargs (Any) – Additional keyword arguments. if kwargs contains ids and documents contain ids, the ids in the kwargs will receive precedence.

Returns:

List of IDs of the added texts.

Raises:

ValueError – If the number of ids does not match the number of documents.

Return type:

list[str]

add_texts(texts: List[str], metadatas: List[dict] | None = None, **kwargs: Any) → List[str]#

Run more texts through the embeddings and add to the vectorstore.

Parameters:

texts (List[str]) – List of strings to add to the vectorstore.
metadatas (List[dict] | None) – Optional list of metadata records associated with the texts. (ie [{“url”: “www.myurl1.com”, “title”: “title1”}, {“url”: “www.myurl2.com”, “title”: “title2”}])
kwargs (Any)

Returns:

List of ids from adding the texts into the vectorstore.

Return type:

List[str]

add_texts_with_embeddings(texts: List[str], embs: List[List[float]], metadatas: List[dict] | None = None) → List[str]#

Add precomputed embeddings and relative texts / metadatas to the vectorstore.

Parameters:

ids – List of unique ids in string format
texts (List[str]) – List of strings to add to the vectorstore.
embs (List[List[float]]) – List of lists of floats with text embeddings for texts.
metadatas (List[dict] | None) – Optional list of metadata records associated with the texts. (ie [{“url”: “www.myurl1.com”, “title”: “title1”}, {“url”: “www.myurl2.com”, “title”: “title2”}])

Returns:

List of ids from adding the texts into the vectorstore.

Return type:

List[str]

async adelete(ids: List[str] | None = None, **kwargs: Any) → bool | None#

Delete by vector ID or other criteria.

Parameters:

ids (List[str] | None) – List of ids to delete.
**kwargs (Any) – Other keyword arguments that subclasses might use.

Returns:

True if deletion is successful, False otherwise, None if not implemented.

Return type:

Optional[bool]

async classmethod afrom_documents(documents: list[Document], embedding: Embeddings, **kwargs: Any) → VST#

Async return VectorStore initialized from documents and embeddings.

Parameters:

documents (list[Document]) – List of Documents to add to the vectorstore.
embedding (Embeddings) – Embedding function to use.
kwargs (Any) – Additional keyword arguments.

Returns:

VectorStore initialized from documents and embeddings.

Return type:

VectorStore

async classmethod afrom_texts(texts: list[str], embedding: Embeddings, metadatas: list[dict] | None = None, **kwargs: Any) → VST#

Async return VectorStore initialized from texts and embeddings.

Parameters:

texts (list[str]) – Texts to add to the vectorstore.
embedding (Embeddings) – Embedding function to use.
metadatas (list[dict] | None) – Optional list of metadatas associated with the texts. Default is None.
kwargs (Any) – Additional keyword arguments.

Returns:

VectorStore initialized from texts and embeddings.

Return type:

VectorStore

async aget_by_ids(ids: Sequence[str], /) → list[Document]#

Async get documents by their IDs.

The returned documents are expected to have the ID field set to the ID of the document in the vector store.

Fewer documents may be returned than requested if some IDs are not found or if there are duplicated IDs.

Users should not assume that the order of the returned documents matches the order of the input IDs. Instead, users should rely on the ID field of the returned documents.

This method should NOT raise exceptions if no documents are found for some IDs.

Parameters:: ids (Sequence[str]) – List of ids to retrieve.
Returns:: List of Documents.
Return type:: list[Document]

Added in version 0.2.11.

async amax_marginal_relevance_search(query: str, k: int = 4, fetch_k: int = 20, lambda_mult: float = 0.5, **kwargs: Any) → list[Document]#

Async return docs selected using the maximal marginal relevance.

Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents.

Parameters:

query (str) – Text to look up documents similar to.
k (int) – Number of Documents to return. Defaults to 4.
fetch_k (int) – Number of Documents to fetch to pass to MMR algorithm. Default is 20.
lambda_mult (float) – Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5.
kwargs (Any)

Returns:

List of Documents selected by maximal marginal relevance.

Return type:

list[Document]

async amax_marginal_relevance_search_by_vector(embedding: list[float], k: int = 4, fetch_k: int = 20, lambda_mult: float = 0.5, **kwargs: Any) → list[Document]#

Async return docs selected using the maximal marginal relevance.

Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents.

Parameters:

embedding (list[float]) – Embedding to look up documents similar to.
k (int) – Number of Documents to return. Defaults to 4.
fetch_k (int) – Number of Documents to fetch to pass to MMR algorithm. Default is 20.
lambda_mult (float) – Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5.
**kwargs (Any) – Arguments to pass to the search method.

Returns:

List of Documents selected by maximal marginal relevance.

Return type:

list[Document]

as_retriever(**kwargs: Any) → VectorStoreRetriever#

Return VectorStoreRetriever initialized from this VectorStore.

Parameters:

**kwargs (Any) –

Keyword arguments to pass to the search function. Can include: search_type (Optional[str]): Defines the type of search that

the Retriever should perform. Can be “similarity” (default), “mmr”, or “similarity_score_threshold”.

search_kwargs (Optional[Dict]): Keyword arguments to pass to the

search function. Can include things like:

k: Amount of documents to return (Default: 4) score_threshold: Minimum relevance threshold

for similarity_score_threshold

fetch_k: Amount of documents to pass to MMR algorithm: (Default: 20)
lambda_mult: Diversity of results returned by MMR;: 1 for minimum diversity and 0 for maximum. (Default: 0.5)

filter: Filter by document metadata

Returns:

Retriever class for VectorStore.

Return type:

VectorStoreRetriever

Examples:

# Retrieve more documents with higher diversity
# Useful if your dataset has many similar documents
docsearch.as_retriever(
    search_type="mmr",
    search_kwargs={'k': 6, 'lambda_mult': 0.25}
)

# Fetch more documents for the MMR algorithm to consider
# But only return the top 5
docsearch.as_retriever(
    search_type="mmr",
    search_kwargs={'k': 5, 'fetch_k': 50}
)

# Only retrieve documents that have a relevance score
# Above a certain threshold
docsearch.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={'score_threshold': 0.8}
)

# Only get the single most similar document from the dataset
docsearch.as_retriever(search_kwargs={'k': 1})

# Use a filter to only retrieve documents from a specific paper
docsearch.as_retriever(
    search_kwargs={'filter': {'paper_title':'GPT-4 Technical Report'}}
)

async asearch(query: str, search_type: str, **kwargs: Any) → list[Document]#

Async return docs most similar to query using a specified search type.

Parameters:

query (str) – Input text.
search_type (str) – Type of search to perform. Can be “similarity”, “mmr”, or “similarity_score_threshold”.
**kwargs (Any) – Arguments to pass to the search method.

Returns:

List of Documents most similar to the query.

Raises:

ValueError – If search_type is not one of “similarity”, “mmr”, or “similarity_score_threshold”.

Return type:

list[Document]

async asimilarity_search(query: str, k: int = 4, **kwargs: Any) → list[Document]#

Async return docs most similar to query.

Parameters:

query (str) – Input text.
k (int) – Number of Documents to return. Defaults to 4.
**kwargs (Any) – Arguments to pass to the search method.

Returns:

List of Documents most similar to the query.

Return type:

list[Document]

async asimilarity_search_by_vector(embedding: list[float], k: int = 4, **kwargs: Any) → list[Document]#

Async return docs most similar to embedding vector.

Parameters:

embedding (list[float]) – Embedding to look up documents similar to.
k (int) – Number of Documents to return. Defaults to 4.
**kwargs (Any) – Arguments to pass to the search method.

Returns:

List of Documents most similar to the query vector.

Return type:

list[Document]

async asimilarity_search_with_relevance_scores(query: str, k: int = 4, **kwargs: Any) → list[tuple[Document, float]]#

Async return docs and relevance scores in the range [0, 1].

0 is dissimilar, 1 is most similar.

Parameters:

query (str) – Input text.
k (int) – Number of Documents to return. Defaults to 4.
**kwargs (Any) –
kwargs to be passed to similarity search. Should include: score_threshold: Optional, a floating point value between 0 to 1 to

filter the resulting set of retrieved docs

Returns:

List of Tuples of (doc, similarity_score)

Return type:

list[tuple[Document, float]]

async asimilarity_search_with_score(*args: Any, **kwargs: Any) → list[tuple[Document, float]]#

Async run similarity search with distance.

Parameters:

*args (Any) – Arguments to pass to the search method.
**kwargs (Any) – Arguments to pass to the search method.

Returns:

List of Tuples of (doc, similarity_score).

Return type:

list[tuple[Document, float]]

batch_search(embeddings: List[List[float]] | None = None, queries: List[str] | None = None, filter: Dict[str, Any] | str | None = None, k: int = 5, expire_hours_temp_table: int = 12) → List[List[List[Any]]][source]#

Multi-purpose batch search function. Accepts either embeddings or queries: but not both. Optionally returns similarity scores and/or matched embeddings

Args: embeddings: A list of embeddings to search with. If provided, each

embedding represents a query vector.

queries: A list of text queries to search with. If provided, each

query represents a query text.

filter: A dictionary of filters to apply to the search. The keys

of the dictionary should be field names, and the values should be the: values to filter on. (e.g., {“category”: “news”})

k: The number of top results to return per query. Defaults to 5. with_scores: If True, returns the relevance scores of the results along with

the documents

with_embeddings: If True, returns the embeddings of the results along with: the documents

Parameters:

embeddings (List[List[float]] | None)
queries (List[str] | None)
filter (Dict[str, Any] | str | None)
k (int)
expire_hours_temp_table (int)

Return type:

List[List[List[Any]]]

delete(ids: List[str] | None = None, **kwargs: Any) → bool | None#

Delete documents by record IDs

Parameters:

ids (List[str] | None) – List of ids to delete.
**kwargs (Any) – Other keyword arguments that subclasses might use.

Returns:

True if deletion is successful, False otherwise, None if not implemented.

Return type:

Optional[bool]

classmethod from_documents(documents: list[Document], embedding: Embeddings, **kwargs: Any) → VST#

Return VectorStore initialized from documents and embeddings.

Parameters:

documents (list[Document]) – List of Documents to add to the vectorstore.
embedding (Embeddings) – Embedding function to use.
kwargs (Any) – Additional keyword arguments.

Returns:

VectorStore initialized from documents and embeddings.

Return type:

VectorStore

classmethod from_texts(texts: List[str], embedding: Embeddings, metadatas: List[dict] | None = None, **kwargs: Any) → BigQueryVectorStore[source]#

Return VectorStore initialized from input texts

Parameters:

texts (List[str]) – List of strings to add to the vectorstore.
embedding (Embeddings) – An embedding model instance for text to vector transformations.
metadatas (List[dict] | None) – Optional list of metadata records associated with the texts. (ie [{“url”: “www.myurl1.com”, “title”: “title1”}, {“url”: “www.myurl2.com”, “title”: “title2”}])
kwargs (Any)

Returns:

List of ids from adding the texts into the vectorstore.

Return type:

BigQueryVectorStore

get_by_ids(ids: Sequence[str], /) → list[Document]#

Get documents by their IDs.

The returned documents are expected to have the ID field set to the ID of the document in the vector store.

Fewer documents may be returned than requested if some IDs are not found or if there are duplicated IDs.

Users should not assume that the order of the returned documents matches the order of the input IDs. Instead, users should rely on the ID field of the returned documents.

This method should NOT raise exceptions if no documents are found for some IDs.

Parameters:: ids (Sequence[str]) – List of ids to retrieve.
Returns:: List of Documents.
Return type:: list[Document]

Added in version 0.2.11.

get_documents(ids: List[str] | None = None, filter: Dict[str, Any] | str | None = None, **kwargs: Any) → List[Document][source]#

Search documents by their ids or metadata values.

Parameters:

ids (List[str] | None) – List of ids of documents to retrieve from the vectorstore.
filter (Dict[str, Any] | str | None) –
Filter on metadata properties, e.g. {

”str_property”: “foo”, “int_property”: 123

}
kwargs (Any)

Returns:

List of ids from adding the texts into the vectorstore.

Return type:

List[Document]

job_stats(job_id: str) → Dict[source]#

Return the statistics for a single job execution.

Parameters:: job_id (str) – The BigQuery Job id.
Returns:: A dictionary of job statistics for a given job. You can check out more details at [BigQuery Jobs] (https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobStatistics2).
Return type:: Dict

max_marginal_relevance_search(query: str, k: int = 5, fetch_k: int = 25, lambda_mult: float = 0.5, **kwargs: Any) → List[Document]#

Return docs selected using the maximal marginal relevance.

Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents.

Parameters:

**kwargs (Any)
query (str) – search query text.
filter –
Filter on metadata properties, e.g. {

”str_property”: “foo”, “int_property”: 123

}
k (int) – Number of Documents to return. Defaults to 5.
fetch_k (int) – Number of Documents to fetch to pass to MMR algorithm.
lambda_mult (float) – Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5.

Returns:

List of Documents selected by maximal marginal relevance.

Return type:

List[Document]

max_marginal_relevance_search_by_vector(embedding: List[float], k: int = 5, fetch_k: int = 25, lambda_mult: float = 0.5, **kwargs: Any) → List[Document]#

Return docs selected using the maximal marginal relevance.

Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents.

Parameters:

embedding (List[float]) – Embedding to look up documents similar to.
filter –
Filter on metadata properties, e.g. {

”str_property”: “foo”, “int_property”: 123

}
k (int) – Number of Documents to return. Defaults to 5.
fetch_k (int) – Number of Documents to fetch to pass to MMR algorithm.
lambda_mult (float) – Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5.
kwargs (Any)

Returns:

List of Documents selected by maximal marginal relevance.

Return type:

List[Document]

search(query: str, search_type: str, **kwargs: Any) → list[Document]#

Return docs most similar to query using a specified search type.

Parameters:

query (str) – Input text
search_type (str) – Type of search to perform. Can be “similarity”, “mmr”, or “similarity_score_threshold”.
**kwargs (Any) – Arguments to pass to the search method.

Returns:

List of Documents most similar to the query.

Raises:

ValueError – If search_type is not one of “similarity”, “mmr”, or “similarity_score_threshold”.

Return type:

list[Document]

similarity_search(query: str, k: int = 5, **kwargs: Any) → List[Document]#

Search for top k docs most similar to input query.

Parameters:

query (str) – search query to search documents with.
filter – (Optional) A dictionary specifying filtering criteria for the documents. Ie. {“title”: “mytitle”}
k (int) – (Optional) The number of top-ranking similar documents to return per embedding. Defaults to 5.
kwargs (Any)

Returns:

Return docs most similar to input query.

Return type:

List[Document]

similarity_search_by_vector(embedding: List[float], k: int = 5, **kwargs: Any) → List[Document]#

Return docs most similar to embedding vector.

Parameters:

embedding (List[float]) – Embedding to look up documents similar to.
filter – (Optional) A dictionary specifying filtering criteria for the documents. Ie. {“title”: “mytitle”}
k (int) – (Optional) The number of top-ranking similar documents to return per embedding. Defaults to 5.
kwargs (Any)

Returns:

Return docs most similar to embedding vector.

Return type:

List[Document]

similarity_search_by_vector_with_score(embedding: List[float], filter: Dict[str, Any] | None = None, k: int = 5) → List[Tuple[Document, float]]#

Return docs most similar to embedding vector with scores.

Parameters:

embedding (List[float]) – Embedding to look up documents similar to.
filter (Dict[str, Any] | None) – (Optional) A dictionary specifying filtering criteria for the documents. Ie. {“title”: “mytitle”}
k (int) – (Optional) The number of top-ranking similar documents to return per embedding. Defaults to 5.

Returns:

Return docs most similar to embedding vector.

Return type:

List[Tuple[Document, float]]

similarity_search_by_vectors(embeddings: List[List[float]], filter: Dict[str, Any] | None = None, k: int = 5, with_scores: bool = False, with_embeddings: bool = False, **kwargs: Any) → Any#

Core similarity search function. Handles a list of embedding vectors,: optionally returning scores and embeddings.

Parameters:

embeddings (List[List[float]]) – A list of embedding vectors, where each vector is a list of floats.
filter (Dict[str, Any] | None) – (Optional) A dictionary specifying filtering criteria for the documents. Ie. {“title”: “mytitle”}
k (int) – (Optional) The number of top-ranking similar documents to return per embedding. Defaults to 5.
with_scores (bool) – (Optional) If True, include similarity scores in the result for each matched document. Defaults to False.
with_embeddings (bool) – (Optional) If True, include the matched document’s embedding vector in the result. Defaults to False.
kwargs (Any)

Returns:

A list of k documents for each embedding in embeddings

Return type:

Any

similarity_search_with_relevance_scores(query: str, k: int = 4, **kwargs: Any) → list[tuple[Document, float]]#

Return docs and relevance scores in the range [0, 1].

0 is dissimilar, 1 is most similar.

Parameters:

query (str) – Input text.
k (int) – Number of Documents to return. Defaults to 4.
**kwargs (Any) –
kwargs to be passed to similarity search. Should include: score_threshold: Optional, a floating point value between 0 to 1 to

filter the resulting set of retrieved docs.

Returns:

List of Tuples of (doc, similarity_score).

Return type:

list[tuple[Document, float]]

similarity_search_with_score(query: str, filter: Dict[str, Any] | None = None, k: int = 5, **kwargs: Any) → List[Tuple[Document, float]]#

Search for top k docs most similar to input query, returns both docs and: scores.

Parameters:

query (str) – search query to search documents with.
filter (Dict[str, Any] | None) – (Optional) A dictionary specifying filtering criteria for the documents. Ie. {“title”: “mytitle”}
k (int) – (Optional) The number of top-ranking similar documents to return per embedding. Defaults to 5.
kwargs (Any)

Returns:

Return docs most similar to input query along with scores.

Return type:

List[Tuple[Document, float]]

sync_data() → None[source]#

Return type:: None

to_vertex_fs_vector_store(**kwargs: Any) → Any[source]#

Creates and returns a VertexFSVectorStore instance based on configuration.

This method merges the base BigQuery vector store configuration with provided: keyword arguments,

then uses the combined parameters to instantiate a VertexFSVectorStore.

Parameters:: **kwargs (Any) – Additional keyword arguments to override or extend the base configuration. These are directly passed to the VertexFSVectorStore constructor.
Returns:: A fully initialized VertexFSVectorStore instance ready for use.
Return type:: VertexFSVectorStore
Raises:: ImportError – If the required LangChain Google Community feature store module is not available.

property embeddings: Embeddings | None#: Access the query embedding object if available.

property full_table_id: str#