ByteDance Seed and Tsinghua AI Lab close performance gap in GPU kernel generation

ByteDance Seed and the AIR lab at Tsinghua University released CUDA Agent, an agentic reinforcement-learning (RL) system that trains a large language model to write GPU kernels that surpass the compiler. The narrow but persistent problem they address is that front-end models already generate correct CUDA code, yet the code runs slowly. On the KernelBench benchmark, which includes 20 tasks, the baseline Seed1.6 model solves 74 % of tasks but is faster than torch.compile on only 27.2 % of them, with an average geometric speedup of 0.69×, meaning its kernels are slower than the default compiler.

The research team conducted an operational sweep of the torch and transformers libraries. The language model was trained to classify up to five operator classes and map them to a single unified layer; a filter retained only operators that run both in eager and compile mode, are deterministic, produce non-constant output, and execute in 1–100 ms in eager. Samples with AST similarity above 0.9 for KernelBench tasks were removed. The resulting dataset contains 6,000 samples, 83.77 % of which are compositions of two operators. The agent loop mirrors tools from OpenHands, Bash, read/write, edit, Glob, Grep, NotebookEdit, Bash output, KillBash, following a ReAct pattern. CUDA instructions are supplied as “Agent Skills”; the SKILL.md file directs the model to profile the PyTorch model, rewrite model_new.py with tuned kernels, compile in a GPU sandbox, and iterate until the kernel is at least 5 % faster than torch.compile with atol = 1e-2 and rtol = 1e-2.

Reward exploitation is guarded by five layers: validation and profiling scripts with locked permissions, context managers that forbid fallback to torch.nn.functional, tests against five random inputs, profiling with device synchronization and warm-up, and the absence of network-search tools. The reward is discrete, not a raw acceleration ratio: r ∈ {−1, 1, 2, 3}, −1 for correctness failure, 3 if the kernel passes both eager and torch.compile by more than 5 %, 2 if it passes only eager, and 1 otherwise. Table 1 reports a 98.8 % pass rate, 98.4 % faster than eager, 96.8 % faster than torch.compile, with geometric speedups of 2.60× and 2.11× respectively. One inconsistency: the abstract and introduction claim 100 % / 100 % / 92 % acceleration, which does not match the full table.

Relevant industries include AI infrastructure and inference serving, GPU clouds, autonomous driving, quantitative trading, medical simulation and recommendation systems, wherever fused kernels sit on a latency-critical path. Use cases span merging operator sequences that torch.compile struggles with, reducing token-level cost, and retuning kernels across GPU generations. For most teams, immediate value resides in the open components—the data, reward structure and technical specification—not the closed model that requires an H20 lab to reproduce.