OpenEnv Integration for Training LLMs with Environments

Summary

Living TRL docs (main). White-box path: environment_factory plus GRPOTrainer auto-discovers public tool methods, runs the multi-turn loop, and reads env.reward (Echo, Wordle, multi-env Wordle+Catch). Black-box/harness path (experimental): HarnessRolloutWorker plus AsyncGRPOTrainer lets a production agent such as opencode own planner, tools, and stop condition inside an OpenEnv ResourceSession; a transparent_proxy records token ids and logprobs; verify() plus optional rollout_reward_fn / train_turn_fn / agent_turn_fn score the outcome and filter turns. Reference script examples/scripts/openenv/opencode.py on agentica-org/DeepCoder-Preview-Dataset; sibling HF-sandbox script for multi-node rollouts. OpenEnv itself is Gymnasium-style step/reset/state with Hub-hosted Spaces. Discord posted the opencode harness hash.

Keywords

openenv · trl · grpo · opencode · harness · rl · huggingface · blog

Topics

RL post-training, agent environments, coding agents

Research notes

• Primary: TRL OpenEnv docs (posted link, including #training-on-harnesses-training-a-real-coding-agent-opencode). OpenEnv https://github.com/huggingface/OpenEnv (BSD-3-Clause). TRL https://github.com/huggingface/trl (Apache-2.0). OpenEnv docs https://huggingface.co/docs/openenv/. Example dataset DeepCoder-Preview-Dataset is prior, not a new corpus here. Docs/integration, not a hosted corpus, so no datasets_local row.
• 2026-10-01: capture proxy release (OpenEnv PR #1036) announced via @ben_burtenshaw X post (https://x.com/ben_burtenshaw/status/2105681647018315903): proxy sits between harness and model, records exact token ids and logprobs of every call, appears to harness as another model provider; typed TrainingTrace API (OpenEnv PR #1280); TRL async GRPO HarnessRolloutWorker (trl PR #6947)