Engineering Home

Our company specializes in accelerating Large Language Models (LLMs) on NVIDIA GPUs ranging from consumer grade to entry level professional cards. To achieve high inference performance, we employ a set of algorithmic innovations, including: • DynamicSparse – an advanced approach that leverages NVIDIA’s hardware support for sparse computations. Unlike conventional sparse methods, our technique does not require additional fine tuning or model conversion. Sparsification is performed on the fly within custom CUDA kernels of our software stack, using the inherent activation sparsity present in LLMs, thereby preserving output quality. • Block wise Quantization for Communication – our software implements an original block quantization algorithm for data exchanged between multiple GPUs. This enables efficient multi GPU scaling without relying on NVLink, which is often absent in low end accelerators. • Diffusion MTP – we employ proprietary Multi Token Prediction (MTP) blocks based on diffusion models. These blocks significantly reduce memory footprint, which is especially crucial for accelerators that do not feature HBM memory. • CPU Offloading – our runtime automatically offloads certain operations and low activity experts in Mixture of Experts (MoE) architectures to the host CPU, thereby boosting overall server throughput. • Low Precision Formats – the software adaptively utilizes reduced precision data types, including NV/MXFP4, NF4, and others, to optimize data transfers and further increase performance. All of the above techniques are integrated as a custom compiler layer within the vLLM framework, enabling seamless execution of virtually any modern LLM. Our core mission is to develop cost effective inference solutions for smaller models on affordable hardware.