Skip to main content


MosaicML offers a managed inference service. You can either use a variety of open-source models, or deploy your own.

This example goes over how to use LangChain to interact with MosaicML Inference for text completion.

# sign up for an account:

from getpass import getpass

import os

from langchain.chains import LLMChain
from langchain_community.llms import MosaicML
from langchain_core.prompts import PromptTemplate
template = """Question: {question}"""

prompt = PromptTemplate.from_template(template)
llm = MosaicML(inject_instruction_format=True, model_kwargs={"max_new_tokens": 128})
llm_chain = LLMChain(prompt=prompt, llm=llm)
question = "What is one good reason why you should train a large language model on domain specific data?"

Help us out by providing feedback on this documentation page: