DeepInfra

LangChain supports LLMs hosted by Deep Infra through the DeepInfra wrapper. First, you'll need to install the @langchain/community package:

tip

See this section for general instructions on installing integration packages.

npm

npm install @langchain/community

Yarn

yarn add @langchain/community

pnpm

pnpm add @langchain/community

You'll need to obtain an API key and set it as an environment variable named DEEPINFRA_API_TOKEN (or pass it into the constructor), then call the model as shown below:

import { DeepInfraLLM } from "@langchain/community/llms/deepinfra";

const apiKey = process.env.DEEPINFRA_API_TOKEN;
const model = "meta-llama/Meta-Llama-3-70B-Instruct";

const llm = new DeepInfraLLM({
  temperature: 0.7,
  maxTokens: 20,
  model,
  apiKey,
  maxRetries: 5,
});

const res = await llm.invoke(
  "What is the next step in the process of making a good game?"
);

console.log({ res });

API Reference:

• DeepInfraLLM from @langchain/community/llms/deepinfra

Related

• LLM conceptual guide
• LLM how-to guides