Entry AStatus: Internal · In active use
Local inference runtime
- Problem
- Large language models are normally served from datacenter GPUs with far more memory than any consumer card. We wanted large models running on hardware we own, without renting capacity or moving data off-premises.
- Approach
- A CUDA C++ runtime written from scratch rather than assembled from existing serving stacks. Its memory architecture is what allows mixture-of-experts models far larger than the card to run at all. We describe the results here and the method under NDA.
- Result
- Measured resultsOwn hardware
- 86.8 tok/sDense 8B model. RTX 5090, Q8 quantization.
- 52.2 tok/s30B mixture-of-experts model served entirely from one consumer GPU.
- 23.3 tok/s397B mixture-of-experts model across an RTX 5090 and an RTX 4090, with the active expert set fully resident.
- 12.8 tok/sSustained throughput at 128K context.
A 671B model runs on a single RTX 5090. Throughput is storage-limited at that size, but the run is functional.
The runtime loads and serves a hybrid SSM+MoE checkpoint on consumer hardware.
Measured on ALIA hardware between June and July 2026. These are recorded results from those sessions, not a standing performance figure: throughput depends on the model, the quantization and how much of the expert set is resident, and each of those differs per run.
- Status
- The runtime is internal and in daily use as the inference layer for our own tooling.