Agent Lightning: Flexible RL Training for Any AI Agent

Agent Lightning is a novel framework enabling reinforcement learning (RL) training of large language models (LLMs) for any AI agent. It fully decouples agent execution from RL training, allowing seamless integration with existing agents built via LangChain, OpenAI Agents SDK, AutoGen, or from scratch, with almost zero code changes.

The framework models agent execution as a Markov decision process (MDP), defining states as snapshots of execution and actions as LLM outputs. This abstraction supports a unified data interface that captures agent trajectories as sequences of transitions, enabling RL to optimize complex, dynamic workflows including multi-agent scenarios.

Agent Lightning introduces LightningRL, a hierarchical RL algorithm with a credit assignment module that decomposes trajectories into training transitions. This approach allows the use of existing single-turn RL methods without modification, overcoming challenges of long context sequences and masking in multi-turn agent interactions.

The system design features a Training-Agent Disaggregation architecture with a Lightning Server managing RL training and a Lightning Client running agents and collecting data. This separation ensures trainer-agnostic agents and agent-agnostic training, facilitating scalability, robustness, and integration with observability frameworks like OpenTelemetry.

Automatic Intermediate Rewarding (AIR) in Agent Lightning converts system monitoring signals into intermediate rewards, mitigating sparse reward issues common in RL. This mechanism is customizable, enhancing learning efficiency by providing richer feedback during agent execution.

Agent Lightning demonstrates stable, continuous improvements across diverse tasks: text-to-SQL with LangChain, retrieval-augmented generation using OpenAI Agents SDK, and math question answering with tool usage via AutoGen. These results highlight its potential for real-world agent training and deployment.

In the text-to-SQL task, a multi-agent system jointly optimizes SQL writing and rewriting agents to generate and correct queries, showcasing selective optimization within multi-agent workflows. The reward is based on final answer accuracy, with Agent Lightning enabling stable performance gains.

For retrieval-augmented generation, a single LLM formulates queries and decides when to refine or answer based on retrieved documents from a large Wikipedia corpus. Agent Lightning optimizes query formulation and reasoning, improving performance on the challenging MuSiQue multi-hop QA dataset.

In math QA, the agent uses a calculator tool to solve arithmetic and symbolic problems. Agent Lightning trains the LLM to generate correct tool calls and integrate outputs into reasoning chains, yielding consistent accuracy improvements on the Calc-X dataset.

Compared to prior RL methods that concatenate multi-turn interactions and rely on masking, Agent Lightning’s transition-based modeling better captures complex agent logic, avoids long context issues, and simplifies implementation. This flexibility supports diverse agent architectures and advanced RL algorithms.

Agent Lightning’s decoupling of RL training from agent execution addresses challenges in existing RL frameworks that require tight coupling and code rewrites. It enables easy adoption across heterogeneous agent ecosystems without specialized RL system expertise.

Future directions include extending support beyond RL to other optimization methods like prompt tuning, advancing RL algorithms for better credit assignment and exploration, and improving system infrastructure for scalability and efficiency in large-scale agent training.

Overall, Agent Lightning offers a unified, extensible solution for training AI agents with RL, unlocking adaptive, learning-based agent capabilities that better handle real-world complexity and dynamic environments.