How to Use LangChain in Java: A Practical Guide for Developers

If you want to build LLM-powered applications on the JVM, learning how to use LangChain in Java is one of the fastest ways to get productive. While LangChain started as a Python and JavaScript framework, the Java ecosystem now has solid options for the same core ideas: chaining prompts, calling models, retrieving documents, and orchestrating agents without writing every API call by hand. This guide walks through what LangChain-style development looks like in Java, which libraries to use, and how to build a small but realistic application step by step.

The main option for Java developers is LangChain4j, an open-source community project that brings the LangChain programming model to the JVM. It integrates with popular models such as OpenAI, Anthropic's Claude, and local models served through Ollama, and it works naturally with Spring Boot, Quarkus, and plain Java applications.

Prerequisites and Project Setup

Before writing any code, make sure you have the basics in place:

• Java 17 or later (LangChain4j targets modern Java versions)
• A build tool such as Maven or Gradle
• An API key for at least one LLM provider, or a local model via Ollama if you prefer not to send data to a cloud service

To get started with LangChain4j, add the core dependency plus the module for your chosen model provider. This guide deliberately uses LangChain4j 1.0.0 as a reproducible API baseline, not as a claim about the latest release. Its BOM (bill of materials) manages compatible module versions, which need not all share the same version number. The official 1.0.0 release notes confirm that version 1.0.0 was released for langchain4j-core, langchain4j, the HTTP client artifacts (langchain4j-http-client, langchain4j-http-client-jdk), the OpenAI module (langchain4j-open-ai), and the BOM; the rest of the modules are released under 1.0.0-beta5, and the BOM includes the latest versions of all modules, many of which are at 1.0.0-beta5. Add the following to the corresponding sections of your Maven project:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>dev.langchain4j</groupId>
            <artifactId>langchain4j-bom</artifactId>
            <version>1.0.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
<dependencies>
    <dependency>
        <groupId>dev.langchain4j</groupId>
        <artifactId>langchain4j</artifactId>
    </dependency>
    <dependency>
        <groupId>dev.langchain4j</groupId>
        <artifactId>langchain4j-open-ai</artifactId>
    </dependency>
</dependencies>

The code below uses ChatModel, which exists in the official 1.0.0 source and provides the default chat(String) method used in the first example. Use the ChatModel import shown here with these pinned dependencies; check the migration notes before adopting a different release. Keep dependencies aligned through the BOM and consult the LangChain4j documentation when upgrading. For production, evaluate a maintained release and its migration notes rather than treating this tutorial's baseline as an update policy. The library is organized into modules per provider, so you only need integrations your application uses.

One common early mistake is adding every available integration module "just in case." Resist that temptation. Each provider module brings its own transitive dependencies, and unused modules can cause classpath conflicts, especially in application servers.

Your First LLM Chain in Java

The simplest useful program is a model call with a prompt. LangChain4j models use a builder pattern, which makes configuration explicit and testable:

import dev.langchain4j.model.chat.ChatModel;
import dev.langchain4j.model.openai.OpenAiChatModel;

public class SimpleChain {
    public static void main(String[] args) {
        ChatModel model = OpenAiChatModel.builder()
                .apiKey(System.getenv("OPENAI_API_KEY"))
                .modelName(System.getenv("OPENAI_MODEL"))
                .temperature(0.7)
                .build();

        String answer = model.chat(
                "Explain what a REST API is in two sentences for a junior developer.");
        System.out.println(answer);
    }
}

Set OPENAI_API_KEY and OPENAI_MODEL in your process environment before running the example. Choose a chat model available to your account and compatible with this provider integration; the model identifier is configuration rather than a fixed library dependency. This maps to the "Model I/O" concept from LangChain: a prompt goes in, a completion comes out. The real power comes when you start combining steps. A chain is a sequence of transformations — format a prompt, call the model, parse the output — expressed as one unit of work.

LangChain4j takes a distinctly Java-flavored approach to this through declarative AI Services. Instead of assembling chains programmatically, you define an interface and annotate it:

import dev.langchain4j.service.SystemMessage;
import dev.langchain4j.service.UserMessage;
import dev.langchain4j.service.V;

interface CodeReviewer {
    @SystemMessage("You are a senior Java reviewer. Be concise and specific.")
    @UserMessage("Review this code and list the top issues:\n{{code}}")
    String review(@V("code") String code);
}

The @V annotation binds the method parameter to the {{code}} placeholder in the prompt template, which keeps the mapping explicit regardless of compiler settings. You then build the implementation with an AiServices builder, wiring in the model. This pattern is a major reason Java developers enjoy the framework: your LLM interactions become typed interfaces you can mock in unit tests, inject with dependency injection, and evolve like any other part of your codebase.

Adding RAG: Retrieval-Augmented Generation in Java

The most common production use case for LangChain-style frameworks is RAG — answering questions over your own documents rather than relying purely on model knowledge. The workflow has two parts: ingesting documents into a searchable index, and retrieving relevant chunks at query time to include in the prompt.

LangChain4j supports this with an EmbeddingStore abstraction. You embed document chunks with an embedding model and store them in a vector database such as PgVector, Chroma, Milvus, or an in-memory store for prototyping:

import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.document.splitter.DocumentSplitters;
import dev.langchain4j.model.openai.OpenAiEmbeddingModel;

// Prepare document chunks and an embedding client; no embedding request yet
// In the 1.0.0 API, the recursive() overload takes sizes in characters:
// maxSegmentSizeInChars = 500, maxOverlapSizeInChars = 50
var splitter = DocumentSplitters.recursive(500, 50);
var embeddingModel = OpenAiEmbeddingModel.builder()
        .apiKey(System.getenv("OPENAI_API_KEY"))
        .modelName("text-embedding-3-small")
        .build();

Document doc = Document.from(
        "Our refund policy allows returns within 30 days of purchase...");
var segments = splitter.split(doc);

The version-pinned splitter source defines the two-argument overload with maxSegmentSizeInChars and maxOverlapSizeInChars parameters, measured in characters. Its three-argument overload accepts a TokenCountEstimator and measures the limits in tokens. Therefore, recursive(500, 50) does not mean 500 tokens with 50 tokens of overlap, and no fixed character-to-token conversion should be assumed. This snippet prepares segments and constructs an embedding client; it does not yet call the embedding API or populate a store. Match the chunking units to your embedding model and evaluate retrieval quality with representative documents.

Once your documents are embedded and stored, you attach a ContentRetriever to your AI service. At query time, the framework retrieves the most relevant chunks, injects them into the prompt, and lets the model ground its answer in your content. This dramatically reduces hallucination for domain-specific questions — but it is not magic. Retrieval quality depends heavily on how you chunk documents. A frequent mistake is using one giant chunk per page: embeddings become too diluted, and retrieval matches everything vaguely and nothing precisely. Chunk sizes in the few-hundred-token range with slight overlap are a sensible starting point, and you should tune this per content type.

Another practical caveat: RAG adds latency and cost. Every query triggers an embedding call plus vector search before the main model call. For a small internal tool this is negligible; for a high-traffic API, budget for it.

Agents, Tools, and When to Reach for Them

Beyond chains and RAG, LangChain4j supports tool use, letting the model call your Java functions to answer questions or take actions. You describe a method with annotations, and the framework handles serializing the model's tool-call requests back into actual Java invocations:

import dev.langchain4j.agent.tool.Tool;

class InventoryTools {
    @Tool("Look up current stock for a product")
    int stockLevel(String productId) {
        // call your inventory service here
        return 42;
    }
}

The version-pinned Tool annotation source defines the dev.langchain4j.agent.tool package and a value attribute for the tool description (the method name is used as the tool name when no name is given). Its documented contract sends String results to the model as-is, handles void returns by sending a "Success" acknowledgement, and serializes other return values into a JSON string sent to the LLM. Design returned objects so they can be serialized and avoid exposing internal or sensitive fields. Register an InventoryTools instance through an AI Services builder with .tools(new InventoryTools()); the annotation alone does not attach the tool to the earlier direct chat call. Tools are what turn a chatbot into an agent that can act — checking a database, calling an internal API, or triggering a workflow. They are also where risk concentrates. Any tool the model can invoke is effectively an API surface exposed to natural-language input. Apply the same discipline you would to a public endpoint: validate inputs, restrict scopes, and avoid giving the model destructive capabilities it does not need.

A sound rule of thumb: use plain chains when the task is predictable, RAG when the model needs your knowledge, and agents only when the sequence of steps genuinely cannot be determined in advance. Agents are harder to test and debug, and many teams reach for them prematurely.

Production Considerations for Java LLM Applications

Moving from a demo to production is where most of the real work happens, and a few JVM-specific concerns stand out.

First, treat model calls like any other unreliable network dependency. Add timeouts, retries with backoff, and fallback behavior. LangChain4j lets you configure these on the model builders; your application should also degrade gracefully when the provider is down — a cached answer or a clear error message beats a hung request.

Second, manage observability deliberately. LangChain4j integrates with popular Java observability stacks, and you can also plug in a listener to log prompts and responses. Be careful what you log: prompts frequently contain customer data, and prompt-response logs are a compliance and privacy liability if left unmanaged.

Third, control cost at the architectural level. A common production mistake is sending entire conversation histories on every turn without trimming. Older context is often irrelevant; summarize or truncate histories for long sessions, and cache responses for repeated identical prompts where appropriate.

Finally, pin model versions rather than relying on a floating alias when reproducibility matters. Model providers update their models over time, and silent behavior changes are painful to diagnose in production systems.

Frequently Asked Questions

Conclusion

Working with LangChain concepts in Java is no longer a compromise — LangChain4j and the surrounding JVM ecosystem let you build chains, retrieval pipelines, and tool-using agents with the type safety, tooling, and operational maturity Java developers expect. Start small: wire up a single chat model, convert one business function into an annotated tool or an AI service interface, and add retrieval only once you have a real knowledge-grounding need. From there, the path to production is mostly familiar engineering discipline — timeouts, observability, and cost control — applied to a new kind of dependency. Pick a small internal use case this week and build your first chain; the fastest way to learn this stack is to ship something modest that real users touch.

---

Transparency Note: Links in this article are not affiliate links. I recommend these tools and platforms based on their relevance to the topic, and no compensation is received for including them.