Hydra
- Own hardware
- No cloud
- LLMs
my own AI server, no cloud
For your organisation: AI on your own hardware: your data stays in, and you do not depend on a cloud provider's prices and terms.
Runs the LLMs for my other projects on my own hardware, more than 3 times as fast as the well-known open-source engine on the same cards, and trains speech models too. A job scheduler queues GPU work, and a card that is free picks it up by itself.
Tech stack: An AMD EPYC 7443 with 24 cores and 256 GB of memory, two RTX 3090s at 250 W and a Radeon RX 6600 XT for the Mac VMs; Proxmox runs the virtual machines.
more than 3x as fast as llama.cpp
More about HydraClose
What it does
My own models on my own hardware, more than 3 times as fast
Hydra runs, trains and tests the models for my other projects locally, without the cloud. The large language model runs on my own inference setup, on an open-source engine with my own changes, including a GPU kernel of my own (3 to 7% faster). That is more than 3 times as fast as llama.cpp with its best settings, the well-known open-source engine that tools like Ollama and LM Studio build on. Proxmox splits the machine into virtual machines; a job scheduler queues GPU work, and a card that is free picks up the next job by itself.
- for one conversation with Qwen 3.8 Flash Next (125 billion parameters), on short text and long code; up to 209 when the answer copies from the prompt
- 136 tokens/s
- as fast as llama.cpp with its best settings, the engine under Ollama and LM Studio, on the same two cards
- over 3x
- in total with four conversations at once
- about 200 tokens/s
- when reading in a prompt (up to 2,600); at 30,000 tokens the first word comes after about 12 s
- 2,500 tokens/s
My own measurement, 6 November 2026, on my own server, normal settings.
One conversation, tokens per second: llama.cpp against my own inference setup
Short prompt
- llama.cpp
- llama.cpp, n-gram
- Open-source engine
- + own kernel
30,000-token code prompt
- llama.cpp
- llama.cpp, n-gram
- Open-source engine
- + own kernel
Answer that copies from the prompt
- llama.cpp
- llama.cpp, n-gram
- Open-source engine
- + own kernel
My own measurement, 6 November 2026, on my own server, normal settings; median of 512 generated tokens per prompt, run-to-run noise about 6%. llama.cpp with the best offload over both cards, and with n-gram speculation. Open-source engine: with my own changes; + own kernel: adding a GPU kernel of my own, 3 to 7% faster, with answers just as good. Against llama.cpp with its best settings that is about 3.4x on short text, 4.5x on long code and 2.2x when the answer copies from the prompt. Several conversations at once: total tokens per second
No batching
- 1 conversation
- 2 conversations
- 4 conversations
Batched, up to 4 at once
- 1 conversation
- 2 conversations
- 4 conversations
My own measurement, 6 November 2026, on my own server, normal settings. Batched with four conversations: the best of three runs (192, 197 and 206), usually about 200. Batching turns more users into more throughput: with four conversations about 50 tokens per second each. Tokens per second: roughly words per second, much faster than you read. What runs where
Proxmox: the virtual machines
- AMD EPYC 744324 cores, 256 GB of memoryjob scheduler, virtual machines and the experts that are needed less often
- RTX 3090limited to 250 Wfirst part of the layers, plus the most-used experts
- RTX 3090limited to 250 Wsecond part of the layers, plus the most-used experts
- Radeon RX 6600 XTfor the Mac VMsdisplay for the virtual Macs
Schematic. The language model's layers are split across the two 3090s; the most-used experts live on the GPUs, the rest in system RAM, and 99.1 to 99.8% of the work hits the GPUs. The machine
What is inside
- Motherboard
- ASRock Rack ROMED8-2T, two 10 Gb ports
- Processor
- AMD EPYC 7443, 24 cores, Noctua NH-U14S
- Memory
- 8 × 32 GB ECC DDR4-2666, all 8 channels
- GPUs
- 2 × RTX 3090, each limited to 250 W
- Mac VMs
- AMD Radeon RX 6600 XT
- Power
- be quiet! Dark Power Pro 13, 1600 W
- Virtualisation
- Proxmox VE
- Management
- fully remote, like a real server
- Network
- 10 Gb
A smaller model, twice
Two coding assistants at once
For coding work a 27B model (Qwen 3.8 27B) runs on each 3090 separately: two conversations at once, each with up to about 237,000 tokens of context. Measured on real coding work between 22 October and 6 November 2026.
- per card on longer answers (median); short answers about 110
- 70 tokens/s
- when the answer copies a lot from the prompt
- up to 309 tokens/s
- of real coding work in just over two weeks
- over 6,000 requests
Live log of the coding service, over 6,000 requests, 22 October to 6 November 2026.
Also for training
37 speech recognition adapters in one campaign
Hydra also trains models. For the Dutch speech recognition in Scribe, a campaign in August 2026 trained 37 adapters (fine-tuned add-ons to a model): for Whisper turbo and small, for Parakeet, and one probe with Whisper large-v3. What came of it, and why the app still runs stock Whisper, is under Scribe.