Allen Institute for AI (AI2) Researchers introduce SERA, Comfortable Verified Environment friendly Repository Brokers, as a coding agent household that goals to match a lot bigger closed techniques utilizing solely supervised coaching and artificial trajectories.
What’s SERA?
SERA is the primary launch in AI2’s Open Coding Brokers sequence. The flagship mannequin, SERA-32B, is constructed on the Qwen 3 32B structure and is educated as a repository degree coding agent.
On SWE bench Verified at 32K context, SERA-32B reaches 49.5 p.c resolve fee. At 64K context it reaches 54.2 p.c. These numbers place it in the identical efficiency band as open weight techniques akin to Devstral-Small-2 with 24B parameters and GLM-4.5 Air with 110B parameters, whereas SERA stays totally open in code, information, and weights.
The sequence consists of 4 fashions at the moment, SERA-8B, SERA-8B GA, SERA-32B, and SERA-32B GA. All are launched on Hugging Face beneath an Apache 2.0 license.
Comfortable Verified Era
The coaching pipeline depends on Comfortable Verified Era, SVG. SVG produces agent trajectories that appear like life like developer workflows, then makes use of patch settlement between two rollouts as a comfortable sign of correctness.
The method is:
- First rollout: A operate is sampled from an actual repository. The instructor mannequin, GLM-4.6 within the SERA-32B setup, receives a bug model or change description and operates with instruments to view information, edit code, and run instructions. It produces a trajectory T1 and a patch P1.
- Artificial pull request: The system converts the trajectory right into a pull request like description. This textual content summarizes intent and key edits in a format much like actual pull requests.
- Second rollout: The instructor begins once more from the unique repository, however now it solely sees the pull request description and the instruments. It produces a brand new trajectory T2 and patch P2 that tries to implement the described change.
- Comfortable verification: The patches P1 and P2 are in contrast line by line. A recall rating r is computed because the fraction of modified strains in P1 that seem in P2. When r equals 1 the trajectory is tough verified. For intermediate values, the pattern is comfortable verified.
The important thing consequence from the ablation examine is that strict verification just isn’t required. When fashions are educated on T2 trajectories with totally different thresholds on r, even r equals 0, efficiency on SWE bench Verified is analogous at a set pattern rely. This implies that life like multi step traces, even when noisy, are useful supervision for coding brokers.
Knowledge scale, coaching, and price
SVG is utilized to 121 Python repositories derived from the SWE-smith corpus. Throughout GLM-4.5 Air and GLM-4.6 instructor runs, the complete SERA datasets comprise greater than 200,000 trajectories from each rollouts, making this one of many largest open coding agent datasets.
SERA-32B is educated on a subset of 25,000 T2 trajectories from the Sera-4.6-Lite T2 dataset. Coaching makes use of commonplace supervised high quality tuning with Axolotl on Qwen-3-32B for 3 epochs, studying fee 1e-5, weight decay 0.01, and most sequence size 32,768 tokens.
Many trajectories are longer than the context restrict. The analysis staff outline a truncation ratio, the fraction of steps that match into 32K tokens. They then choose trajectories that already match, and for the remainder they choose slices with excessive truncation ratio. This ordered truncation technique clearly outperforms random truncation after they evaluate SWE bench Verified scores.
The reported compute funds for SERA-32B, together with information technology and coaching, is about 40 GPU days. Utilizing a scaling legislation over dataset dimension and efficiency, the analysis staff estimated that the SVG method is round 26 instances cheaper than reinforcement studying based mostly techniques akin to SkyRL-Agent and 57 instances cheaper than earlier artificial information pipelines akin to SWE-smith for reaching comparable SWE-bench scores.
Repository specialization
A central use case is adapting an agent to a selected repository. The analysis staff research this on three main SWE-bench Verified tasks, Django, SymPy, and Sphinx.
For every repository, SVG generates on the order of 46,000 to 54,000 trajectories. As a consequence of compute limits, the specialization experiments prepare on 8,000 trajectories per repository, mixing 3,000 comfortable verified T2 trajectories with 5,000 filtered T1 trajectories.
At 32K context, these specialised college students match or barely outperform the GLM-4.5-Air instructor, and in addition evaluate nicely with Devstral-Small-2 on these repository subsets. For Django, a specialised pupil reaches 52.23 p.c resolve fee versus 51.20 p.c for GLM-4.5-Air. For SymPy, the specialised mannequin reaches 51.11 p.c versus 48.89 p.c for GLM-4.5-Air.
Key Takeaways
- SERA turns coding brokers right into a supervised studying downside: SERA-32B is educated with commonplace supervised high quality tuning on artificial trajectories from GLM-4.6, with no reinforcement studying loop and no dependency on repository take a look at suites.
- Comfortable Verified Era removes the necessity for checks: SVG makes use of two rollouts and patch overlap between P1 and P2 to compute a comfortable verification rating, and the analysis staff present that even unverified or weakly verified trajectories can prepare efficient coding brokers.
- Massive, life like agent dataset from actual repositories: The pipeline applies SVG to 121 Python tasks from the SWE smith corpus, producing greater than 200,000 trajectories and creating one of many largest open datasets for coding brokers.
- Environment friendly coaching with specific price and scaling evaluation: SERA-32B trains on 25,000 T2 trajectories and the scaling examine exhibits that SVG is about 26 instances cheaper than SkyRL-Agent and 57 instances cheaper than SWE-smith at comparable SWE bench Verified efficiency.
Try the Paper, Repo and Mannequin Weights. Additionally, be happy to observe us on Twitter and don’t neglect to hitch our 100k+ ML SubReddit and Subscribe to our Publication. Wait! are you on telegram? now you’ll be able to be part of us on telegram as nicely.
