Can you beat NVCC with a Rust PTX compiler?
Demystifying NVCC optimizations

Emmy is our open-source compiler for LLM inference. It traces a model, fuses it into kernels, picks each kernel's schedule (tiling, staging, splits) from measured evidence, and writes the result as a typed loop language we call Kernel IR.
Until this month, the last step was printing Kernel IR as CUDA C++ and calling NVCC. That worked, but it had two problems: slow compile time and a lack of control over the PTX we got. NVCC has to parse C++ and applies a lot of optimization that our compiler already did. It needs to do more work to apply certain optimizations as it doesn't know the kernel's structure. Thus, it took minutes to compile certain large kernels with -O3 which slowed down auto-tuning considerably.
So we replaced it. A Rust crate now compiles Kernel IR straight to PTX, and PTXAS assembles it.
Hackathon
We were worried about the performance of the resulting kernels. There is a belief that you can't beat NVCC as it applies some optimizations based on the HW knowledge that NVIDIA does not expose. We wanted to see whether that was true, so we ran a student hackathon to see if anyone could beat NVCC. Entrants wrote a PTX compiler for Emmy's Kernel IR, and we measured it against NVCC on hidden tests. We got 57 entries in total.
The two winners:
- shubham-jadhav-FiatLux/emmy-ptx-compiler: 1.34× faster than our NVCC build on the hidden tests, 1.54× on large shapes.
- N00bcak/emmy-hackathon-2026: 1.24× and 1.53×.
The 50% performance gain seemed suspicious, so we looked into it and realized it came from schedule changes, not the code generator. They recognized kernels underneath and applied high-level optimizations: shared work across lanes, re-tiled attention, fused sibling loops. If you take those out, the winners were 1.01× and 1.02× faster than NVCC on the 257 hidden tests, and 1.03× and 1.04× on large shapes as measured on RTX 5090. Most other entries were slower than NVCC.
The entrant compilers were also slower on the long-tail of kernels that were not part of the hidden test set, so when we reimplemented the compiler in Emmy we optimized it on a much larger corpus of kernels and measured every kernel on V100, A100, H100, L40S, RTX 4090 and RTX 5090. The result is a compiler that matches NVCC at the median, with a few kernels faster or slower, and compiles a kernel in 13 ms instead of 173 ms.
We also decided not to go through LLVM. One entrant compiled Kernel IR to LLVM IR and then to PTX, but some instructions like cp.async or ldmatrix are not in LLVM yet, so the author had to write a fair amount of inline assembly. That entry did not pass every hidden test, and its compile time was not great either, so we stayed with a small Rust crate that prints PTX directly.
What Your Compiler Needs to Do to Match NVCC
As you might guess, matching NVCC performance on the long-tail was the hard part. Every slow kernel was investigated tediously, but in the end, just a handful of changes covered nearly all of it.
Optimized divide. This was the biggest single loss. A kernel decodes its position from the block and thread index with floor divisions like (128·block + thread) // (12288·tokens). We wrote each one as a signed 64-bit divide plus a sign fix-up, nine of them in one attention kernel. CICC writes three unsigned divides, each with a branch to a 32-bit divide when both operands fit. To do the same, the crate has to prove the operands are non-negative. Once it could, the kernel matched NVCC.
Build addresses in 64 bits. PTXAS folds an address into [base + constant] across unrolled trips only if the address is built in 64 bits from a 64-bit copy of the loop counter. If you leave indices as 32-bit, PTXAS assumes they might wrap and doesn't perform the optimization. That fix took a split-K matmul from 46.2 to 37.4 µs.
Unroll budget. CICC unrolls a static loop fully up to a size budget and makes two copies of some others. PTXAS then unrolls further on its own. We now write every one of CICC's copies in Kernel IR and mark the loop nounroll. If you leave a compact loop unmarked, PTXAS unrolls it fully: one SDPA kernel went to 11,344 machine instructions and half a second of assembly. Marked, it was 2,216 instructions and 0.06 s, and twice as fast.
Load wide. Fewer, wider loads are faster. Reading two fp8 bytes as one 16-bit load and converting them as a pack matched NVCC on the NVFP4 GEMM. On V100, reading a fragment from shared memory as one 128-bit load instead of eight 16-bit ones took us past NVCC.
Keep a compare as a predicate. When a kernel uses a bool as a number, we keep the predicate register the compare produced instead of turning it into an integer early. That closed the gap on the MoE router.
Two smaller things complete the list. Inputs the kernel never writes are read with ld.global.nc. And we copied CICC's expf sequence, so our exp gives NVCC's bits on every f32 input. Our other math functions stay within CUDA's documented bounds, measured over every f32 bit pattern.
The table shows what each change bought. Every number is in µs, measured on the same card in the same session.
| What we changed | Kernel | NVCC (µs) | Ours before (µs) | Ours after (µs) |
|---|---|---|---|---|
| Unsigned grid-decode division, with CICC's 32-bit fast path | 5090 attention | 97.6 | 478 | 99.4 |
| CICC's unroll rule for strided lane loops, written in Kernel IR | A100 softmax | 18.6 | 29.8 | 17.9 |
.pragma "nounroll" on loops we already unrolled | 5090 SDPA | 88 | 194 | 89 |
| Two fp8 bytes as one 16-bit load, then a packed convert | 5090 NVFP4 GEMM | 126.6 | 237.5 | 126.6 |
| Volta fragments as one 128-bit shared load instead of eight 16-bit ones | V100 Llama-3.1 piece | 146 | 501 | 76 |
.minnctapersm 1 on single-wave launches | 5090 matmul | – | 18.3 | 10.9 |
| A predicate kept for a bool used as a number | 5090 MoE router | 43.9 | 64.2 | 44.1 |
What NVCC Doesn't Do
The .minnctapersm row is different: it is not something we copied from NVCC. The directive tells PTXAS how many blocks of the kernel must fit on one multiprocessor at the same time. By default, PTXAS assumes several, and holds back on registers so they all fit. Many of our launches are a single wave: so few blocks that no two ever share a multiprocessor. For those we write .minnctapersm 1, and PTXAS is free to use every register it wants. In CUDA C++ the same hint is the second argument of __launch_bounds__, but NVCC never adds it on its own, because it cannot know how the kernel will be launched.
NVCC also does not fuse a difference of two products. It leaves mul, mul, sub, and PTXAS fuses them in its own way.
Where We Are Ahead and Behind
A few outlier kernels are still noticeably faster or slower than NVCC.
Ahead:
- V100 Llama-3.1-8B pieces up to 1.9× faster (76 vs 146 µs).
- H100 512×512 matmul at 10.3 vs 11.5 µs.
- Six 5090 lane kernels more than 5% faster.
Behind:
- One V100 GEMM at 1.11×.
- An H100 fp32 matmul with a dynamic size at 1.08×.
- A handful of kernels 1.1 to 1.2× slower where PTXAS picks a different load schedule from nearly identical PTX. Every respelling we tried won one case and lost another, so we left them as known costs.
Compile Time
We timed every kernel of our test corpus NVCC can still build: 269 kernels on an RTX 5090 dev box with CUDA 13.0, four at a time.
| Median per kernel | p90 | Whole set | |
|---|---|---|---|
| Our compiler (Kernel IR to PTX) | 0.9 ms | 4.9 ms | 0.6 s |
| PTXAS | 12 ms | 48 ms | 6.7 s |
| Ours, total | ~13 ms | 7.3 s | |
NVCC --cubin | 173 ms | 334 ms | 70 s |
That is about 10× faster. Apparently, most of the NVCC's time goes to its C++ front end parsing the CUDA headers, and various brute-force optimizations that our compiler stack already did.
How to Use the Compiler?
The code is in cloudrift-ai/emmy, and it is included in the latest Emmy release on PyPI.
Here is a sum over each row of a 256×128 matrix, compiled for an RTX 5090:
pip install emmy-ml
emmy compile -c "torch.randn(256, 128).sum(dim=-1)" --ir ptx --target sm_120
.version 8.7
.target sm_120
.address_size 64
.shared .align 4 .b8 k_9bd737c909a3_shared_0[16];
.visible .entry k_9bd737c909a3(
.param .u64 .ptr .align 16 k_9bd737c909a3_param_0,
.param .u64 .ptr .align 16 k_9bd737c909a3_param_1
)
.maxntid 128, 1, 1
{
.reg .pred %p<4>;
.reg .b32 %r<24>;
.reg .b64 %rd<8>;
.reg .f32 %f<17>;
griddepcontrol.wait;
ld.param.u64 %rd0, [k_9bd737c909a3_param_0];
ld.param.u64 %rd1, [k_9bd737c909a3_param_1];
// Grid(axes=(a0 in 0..256, a1_co in 0..128), decode=(block_idx.x, thread_idx.x), block_threads=128):
mov.u32 %r0, %ctaid.x;
mov.u32 %r1, %tid.x;
// let i32 lane = thread_idx.x % 32
and.b32 %r2, %r1, 31;
// let i32 warp = thread_idx.x / 32
shr.s32 %r3, %r1, 5;
// init f32 acc0 = 0
mov.f32 %f0, 0f00000000;
// f32 in0__u0 = load buf0[128 * a0 + a1_co]
mul.wide.s32 %rd2, %r0, 512;
add.s64 %rd3, %rd0, %rd2;
mul.wide.s32 %rd4, %r1, 4;
add.s64 %rd5, %rd3, %rd4;
ld.global.nc.f32 %f1, [%rd5];
// f32 acc0 <- add(acc0, in0__u0) over a1_co
add.rn.ftz.f32 %f0, %f0, %f1;
// WarpShuffle(state=(acc0), state_b=(acc0__o), length=32):
activemask.b32 %r4;
mov.b32 %r5, %f0;
shfl.sync.bfly.b32 %r6, %r5, 16, 31, %r4;
mov.b32 %f2, %r6;
// acc0 = acc0 + acc0__o
add.rn.ftz.f32 %f0, %f0, %f2;
mov.b32 %r7, %f0;
shfl.sync.bfly.b32 %r8, %r7, 8, 31, %r4;
mov.b32 %f4, %r8;
// acc0 = acc0 + acc0__o
add.rn.ftz.f32 %f0, %f0, %f4;
mov.b32 %r9, %f0;
shfl.sync.bfly.b32 %r10, %r9, 4, 31, %r4;
mov.b32 %f6, %r10;
// acc0 = acc0 + acc0__o
add.rn.ftz.f32 %f0, %f0, %f6;
mov.b32 %r11, %f0;
shfl.sync.bfly.b32 %r12, %r11, 2, 31, %r4;
mov.b32 %f8, %r12;
// acc0 = acc0 + acc0__o
add.rn.ftz.f32 %f0, %f0, %f8;
mov.b32 %r13, %f0;
shfl.sync.bfly.b32 %r14, %r13, 1, 31, %r4;
mov.b32 %f10, %r14;
// acc0 = acc0 + acc0__o
add.rn.ftz.f32 %f0, %f0, %f10;
// Smem(name=acc0_smem, extents=(4))
mov.u32 %r15, k_9bd737c909a3_shared_0;
// if 0 == lane:
setp.eq.s32 %p0, %r2, 0;
@!%p0 bra L1;
// f32 acc0_smem[warp] = acc0
shl.b32 %r16, %r3, 2;
add.s32 %r17, %r15, %r16;
st.shared.f32 [%r17], %f0;
L1:
// Sync
bar.sync 0;
// TreeHalve(bufs=(acc0_smem), state=(acc0), state_b=(acc0__o), length=4, tid_var=warp, lane=lane):
setp.lt.s32 %p1, %r1, 4;
@!%p1 bra L2;
shl.b32 %r18, %r1, 2;
add.s32 %r19, %r15, %r18;
ld.shared.f32 %f12, [%r19];
mov.b32 %r20, %f12;
shfl.sync.bfly.b32 %r21, %r20, 2, 31, 15;
mov.b32 %f13, %r21;
// acc0 = acc0 + acc0__o
add.rn.ftz.f32 %f12, %f12, %f13;
mov.b32 %r22, %f12;
shfl.sync.bfly.b32 %r23, %r22, 1, 31, 15;
mov.b32 %f15, %r23;
// acc0 = acc0 + acc0__o
add.rn.ftz.f32 %f12, %f12, %f15;
setp.eq.s32 %p2, %r1, 0;
@!%p2 bra L3;
st.shared.f32 [%r15], %f12;
L3:
L2:
bar.sync 0;
ld.shared.f32 %f0, [%r15];
// if 0 == a1_co:
setp.eq.s32 %p3, %r1, 0;
@!%p3 bra L4;
// f32 buf1[a0] = acc0
mul.wide.s32 %rd6, %r0, 4;
add.s64 %rd7, %rd1, %rd6;
st.global.f32 [%rd7], %f0;
L4:
ret;
}
Each comment is the Kernel IR statement the lines below it came from. One block sums one row, and each of its 128 threads loads one value. A warp adds up its 32 values with five shuffles. The four warp sums then meet in shared memory, where two more shuffles finish the job and thread 0 writes the result.

