_1722956449156001fz1R.png "Anyscale Logo")
Large Language Models (LLM) achieve tremendous success in natural language and multimodal tasks. Reinforcement Learning from Human Feedback (RLHF) is the key step to prevent LLMs from producing harmful and toxic contents. Training LLMs requires complicated model and data parallelism strategies. Programming RL dataflows requires flexible abstractions. Existing Ray-based RL frameworks (RLLib and RLlib-Flow) fail to train large models and existing RLHF frameworks fail to be flexible for conducting research.
The lack of flexible and efficient training infrastructures to easily experiment with RL algorithms enforces researchers and practitioners to seek alternative approaches such as Direct Preference Optimization (DPO) with less system complexity but potentially lower algorithmic performance.
To tackle the challenges posed by implementing RLHF systems, we propose veRL, a framework of flexible and efficient programming abstractions for RLHF research and production. We observe that most existing LLM training/inference infrastructures (e.g. DeepSpeed, FSDP, Megatron-LM) adopt the SPMD (Single Process Multiple Data) programming paradigm. To enable them in the Ray ecosystem, we create "WorkerGroup" that allows encapsulation of LLM training/inference into primitive APIs that can be invoked by the Ray driver program. Each component of the RLHF system including rollout, actor, critic, reward model and reference policy is then implemented as a WorkerGroup with several primitive APIs. veRL is efficient as existing LLM training/inference infrastructures can be easily incorporated. veRL is flexible as it encapsulates LLM training/inference inside WorkerGroup and provides a single process abstraction to implement RLHF dataflows. We hope that veRL can advance RLHF research and production in the future.
The lack of flexible and efficient training infrastructures to easily experiment with RL algorithms enforces researchers and practitioners to seek alternative approaches such as Direct Preference Optimization (DPO) with less system complexity but potentially lower algorithmic performance.
To tackle the challenges posed by implementing RLHF systems, we propose veRL, a framework of flexible and efficient programming abstractions for RLHF research and production. We observe that most existing LLM training/inference infrastructures (e.g. DeepSpeed, FSDP, Megatron-LM) adopt the SPMD (Single Process Multiple Data) programming paradigm. To enable them in the Ray ecosystem, we create "WorkerGroup" that allows encapsulation of LLM training/inference into primitive APIs that can be invoked by the Ray driver program. Each component of the RLHF system including rollout, actor, critic, reward model and reference policy is then implemented as a WorkerGroup with several primitive APIs. veRL is efficient as existing LLM training/inference infrastructures can be easily incorporated. veRL is flexible as it encapsulates LLM training/inference inside WorkerGroup and provides a single process abstraction to implement RLHF dataflows. We hope that veRL can advance RLHF research and production in the future.
