Faster inference through hardware acceleration. ThinkLocAI supports NVIDIA CUDA, AMD ROCm, and Apple Metal.
Tokens per second (t/s) - CPU vs GPU
Real-time monitoring of your GPU
NVIDIA RTX 4090
24GB VRAM • CUDA 12.1
Supported hardware:
Transparency notice: this demo is a simulation using prepared sample data and does not show output from a live AI model. Labelled in line with Art. 50 of Regulation (EU) 2024/1689.
Support for NVIDIA GPUs from the GTX series to the H100 for data centers.
Support for AMD Radeon and Instinct GPUs, so you are not tied to one GPU vendor.
Optimized for Macs with M1, M2, or M3 and their unified memory.
Model layers are split between GPU and RAM when VRAM is not sufficient.
Ongoing monitoring of GPU utilization, temperature, and VRAM.
Batch size and context length are tuned automatically.
Process hundreds of requests at once, including across a GPU cluster.
Analyze large document volumes without long waits for results.
Short response times keep conversations with the assistant fluid.
Together we work out which models you want to run and which hardware makes sense for them.
