create_spark_dataframe_agent#

langchain_experimental.agents.agent_toolkits.spark.base.create_spark_dataframe_agent(llm: BaseLLM, df: Any, callback_manager: BaseCallbackManager | None = None, prefix: str = '\nYou are working with a spark dataframe in Python. The name of the dataframe is `df`.\nYou should use the tools below to answer the question posed of you:', suffix: str = '\nThis is the result of `print(df.first())`:\n{df}\n\nBegin!\nQuestion: {input}\n{agent_scratchpad}', input_variables: List[str] | None = None, verbose: bool = False, return_intermediate_steps: bool = False, max_iterations: int | None = 15, max_execution_time: float | None = None, early_stopping_method: str = 'force', agent_executor_kwargs: Dict[str, Any] | None = None, allow_dangerous_code: bool = False, **kwargs: Any) → AgentExecutor[source]#

Construct a Spark agent from an LLM and dataframe.

Security Notice:

This agent relies on access to a python repl tool which can execute arbitrary code. This can be dangerous and requires a specially sandboxed environment to be safely used. Failure to run this code in a properly sandboxed environment can lead to arbitrary code execution vulnerabilities, which can lead to data breaches, data loss, or other security incidents.

Do not use this code with untrusted inputs, with elevated permissions, or without consulting your security team about proper sandboxing!

You must opt in to use this functionality by setting allow_dangerous_code=True.

Parameters:

  • allow_dangerous_code (bool) – bool, default False This agent relies on access to a python repl tool which can execute arbitrary code. This can be dangerous and requires a specially sandboxed environment to be safely used. Failure to properly sandbox this class can lead to arbitrary code execution vulnerabilities, which can lead to data breaches, data loss, or other security incidents. You must opt in to use this functionality by setting allow_dangerous_code=True.

  • llm (BaseLLM) –

  • df (Any) –

  • callback_manager (BaseCallbackManager | None) –

  • prefix (str) –

  • suffix (str) –

  • input_variables (List[str] | None) –

  • verbose (bool) –

  • return_intermediate_steps (bool) –

  • max_iterations (int | None) –

  • max_execution_time (float | None) –

  • early_stopping_method (str) –

  • agent_executor_kwargs (Dict[str, Any] | None) –

  • kwargs (Any) –

Return type:

AgentExecutor

On this page