Private Open-Weight AI Deployment
Full-scale open-weight models — the Llama, Qwen, DeepSeek, Mistral, and gpt-oss class — deployed for internal organisational use on your hardware, on your data, under your control. Sized honestly, engineered like the production workload it is, operated with NOC discipline, and handed over to a team we have trained.
The weights are downloadable. The capability is engineered.
01 — Why Private
Why Run Models Inside Your Walls
In 2026, the strongest open-weight models sit within a few points of frontier API systems on the benchmarks that predict real work — drafting, summarisation, extraction, retrieval-grounded question answering, code assistance — and several of the best families ship under Apache 2.0 or MIT licences that permit exactly this: download the weights, run them on your own servers, fine-tune them on your own data, and owe nobody a per-token fee. What used to require a hyperscaler contract is now an infrastructure project. That is the category of project we have delivered for two decades.
Four drivers make private deployment the right answer for a growing class of organisations. Sovereignty: prompts, documents, and embeddings never leave your jurisdiction — decisive under the Kenya Data Protection Act, central-bank outsourcing and cloud guidance, and the widening reach of AI-specific regulation. Confidentiality: credit files, legal opinions, board papers, patient records, and network configurations can be put in front of a model that has no outbound network path. Cost predictability: a served model is capex plus power plus operations — a budget line, not a metered bill that scales with every employee who discovers it is useful. Control: the model does not change under you at a vendor’s release cadence; you pin versions, evaluate upgrades on your own tests, and roll back on your own authority.
And the honest counterweight: sometimes the API is the better engineering answer, and we will say so in the first meeting. If your volume is low or spiky, if the task genuinely needs frontier-model reasoning, if no data-residency constraint applies, or if nobody in the building will own the platform, a commercial API behind a data-processing agreement beats an under-used GPU node that nobody patches. Published break-even analyses consistently put the crossover at sustained, high daily token volumes or at hard regulatory constraints — not at enthusiasm. Phase 0 exists to make that call with arithmetic instead of sentiment.
02 — Model Selection
Model Selection Discipline
The model is a component, chosen against your tasks — not a brand allegiance
Families Worth Shortlisting
- Qwen family — strong general capability, wide size range, Apache 2.0, good multilingual behaviour
- DeepSeek family — MIT-licensed, strong reasoning variants, aggressive efficiency
- Llama family — the broadest tooling and ecosystem support, community licence terms to read carefully
- Mistral — efficient dense models, the notable European option where that matters
- gpt-oss class — Apache 2.0 mixture-of-experts models sized for single-node serving
- Gemma/Phi-class small models — the honest choice for constrained tiers and edge cases
Dense vs Mixture-of-Experts
- MoE models compute like their active parameters but must hold total parameters in memory — cheap tokens, expensive VRAM
- Dense models are simpler to quantize, shard, and reason about under load
- Long context is a memory bill of its own: KV cache grows with context length × concurrent users, and it is the line item most sizing exercises forget
- Reasoning variants trade latency and token volume for quality — right for analysis queues, wrong for interactive chat
Licence Reading Before Commitment
- Apache 2.0 / MIT: use, modify, fine-tune, deploy commercially without royalty
- Community licences (Llama-style): acceptable-use policies, attribution, and threshold clauses that legal must actually read
- Fine-tune and derivative terms: who owns your LoRA adapters, and what may be redistributed
- No open-weight licence ships an indemnity — governance has to come from your controls, not the licence text
Evaluation Before Love
- A golden set built from your documents and tasks — not public leaderboards
- Task-level scoring: extraction accuracy, summary faithfulness, refusal behaviour, tone
- Quantized-vs-full comparisons on that same set, so precision trade-offs are measured, not assumed
- Language coverage checks for the languages your staff actually write in
- Adversarial prompts and injection attempts run before go-live, not after an incident
A model shortlist is an engineering decision with a licence attached — we treat both parts seriously.
03 — Hardware Sizing
Hardware Sizing Without Wishful Thinking
The arithmetic is not mysterious: weight memory ≈ parameters × bytes per parameter (≈2 bytes at FP16, ≈1 at FP8, ≈0.5–0.6 at 4-bit), plus KV cache that grows with context length and concurrency, plus headroom the serving stack needs to batch efficiently. We size against that arithmetic — and against your power and cooling reality — then recommend the smallest tier that honestly serves the workload.
Pilot — Single-GPU Workstation
One consumer or workstation GPU (RTX 4090/5090 class) serving 7–32B models in 4-bit or FP8. Enough for a handful of concurrent users, a first RAG index, and a real evaluation baseline. Runs on office single-phase power with a line-interactive UPS. Its job is to prove the use case before serious money moves.
≈24–32 GB VRAMDepartmental — Multi-GPU Server
Two to four workstation/server-class GPUs in a proper chassis. Serves 70B-class dense models quantized, or mid-size MoE, to tens of concurrent users with production RAG. Sustained draw in the low kilowatts: this tier needs a dedicated circuit, provisioned cooling, and online double-conversion UPS — which is where our facility background starts earning its keep.
≈48–192 GB VRAMProduction — HBM Accelerator Nodes
H100/H200/MI300-class accelerators serving large MoE models at FP8 to hundreds of users, with tensor parallelism and N+1 thinking. An eight-GPU node draws ten-plus kilowatts before you cool it: three-phase power, precision cooling, generator-coordinated UPS runtime, and per-rack power monitoring are requirements, not options — especially on African grids where ride-through is a design input.
80–192 GB HBM per GPUCPU-Only — The Honest Niche
Small models in GGUF running on server CPUs. Legitimate for air-gapped enclaves, batch classification, and embedded tooling where minutes are fine. Interactive assistant latency on larger models is poor, and we say so — this tier is for the workloads that fit it, not a way to avoid buying a GPU.
No GPU04 — Engineering
Deployment Engineering
Everything between "the weights downloaded" and "a service your organisation trusts"
Serving Stack & Quantization
High demandThe runtime decisions that determine throughput per watt
Engineering Scope
- vLLM as the production default — paged attention and continuous batching for multi-user throughput
- SGLang where workloads share prefixes — RAG services and agentic pipelines benefit from radix caching
- llama.cpp/GGUF for CPU-only, edge, and heterogeneous hardware
- Ollama confined to pilots and single-user desks — it is not a multi-tenant server
- TensorRT-LLM when squeezing maximum throughput from NVIDIA silicon justifies the build complexity
- FP8 serving on Hopper-class and newer; AWQ/GPTQ INT4 when VRAM is the binding constraint
- KV-cache quantization and context budgeting per use case
- Speculative decoding and prefix caching where the workload actually pays for them
Why It Matters
The same hardware can serve double the users — or half — depending on these choices. Tokens per second per watt is the real unit economics of a private deployment.
RAG Over Internal Knowledge
The model answers from your documents without training on them
Engineering Scope
- Embedding models running inside the enclave — retrieval that phones home breaks the entire premise
- Ingestion pipelines from document stores, file shares, and wikis with scheduled refresh
- Chunking, metadata, and hybrid (vector + keyword) retrieval tuned on your corpus
- Permission-aware retrieval: document ACLs enforced at query time, so answers only cite what the asking user may read
- Citation-grounded responses with source links, so every answer is checkable
- Retrieval quality metrics tracked as a first-class operational signal
Why It Matters
RAG is how a general model becomes your model — current, sourced, and revocable — without the cost and governance weight of training.
Fine-Tuning When Justified
LoRA on private data — after RAG, not instead of it
Engineering Scope
- LoRA/QLoRA adapter training for format, tone, and domain-language conformance
- Training-data governance: provenance, consent, and exclusion of records that must not be learned
- Eval-gated promotion — an adapter ships only when it beats the base model on the golden set
- Adapter versioning and instant detachment, keeping the base weights pristine
Why It Matters
Most "we need fine-tuning" requests are retrieval problems. When tuning is genuinely warranted, it should be cheap, reversible, and measured — LoRA on your own hardware is all three.
Identity, Access & Isolation
High demandThe controls that make a regulated deployment defensible
Engineering Scope
- SSO/OIDC integration with your directory — every prompt attributable to a person
- Role-based access per model, per RAG collection, per department
- Network segmentation and explicit no-egress firewall policy for inference and retrieval segments
- Air-gapped patterns: offline model and update transfer with hash verification for the environments that require it
- Full prompt/response audit logging with retention aligned to your regulatory obligations
- Security hardening of hosts, containers, and the model gateway itself
Why It Matters
When the auditor or the regulator asks who accessed what, through which model, against which documents — the deployment either answers in minutes or fails the question.
Integration APIs
The model as an internal service, not a destination
Engineering Scope
- OpenAI-compatible endpoints inside your LAN, so existing tooling and SDKs work unmodified
- A gateway layer with per-team quotas, rate limits, and token accounting
- Structured-output contracts (JSON schema enforcement) for downstream systems that consume model output
- Integration into the productivity estate, intranet, and ticketing workflows staff already live in
Why It Matters
Adoption follows the path of least friction. A model reachable from the tools people already use gets used; a separate portal gets a demo and then silence.
Platform, Power & Facility Engineering
High demandThe layer most AI consultancies subcontract — and we consider the foundation
Engineering Scope
- GPU serving inside your virtualization estate (passthrough/SR-IOV) or on bare metal, decided on evidence
- Linux host builds, driver and CUDA/ROCm lifecycle management, and configuration baselines
- Thermal design honest about high-ambient conditions and real airflow, not datasheet assumptions
- UPS runtime sizing, generator coordination, and graceful-degradation behaviour on power events
- Per-rack power monitoring so the electricity bill of inference is measured, not guessed
- Capacity planning as usage grows — the same discipline as any production estate
Why It Matters
We have run server virtualization estates, facility power projects, and NOC operations for two decades. A GPU node is a server with an aggressive power budget, and it fails for the same unglamorous reasons servers always have — power, heat, and neglect.
05 — Integration Map
Where the Platform Plugs In
A private model earns its keep by joining the systems you already run. These are the standing connections we engineer — each one inside your network perimeter:
Identity provider (AD / Entra / Keycloak)Inference gateway
Single sign-on and role mapping — access per model and per knowledge collection, every request attributable to a named user
Document stores (DMS, file shares, intranet, wikis)RAG indexing pipeline
Scheduled ingestion with permissions captured alongside content, so retrieval respects the same ACLs as the source system
Line-of-business systems (core records, ERP, ticketing)Integration API
Schema-validated calls for summarisation, drafting, and classification — the model as a callable internal service
Inference clusterMonitoring, SIEM & NOC
GPU thermals and power draw, latency and queue depth, token accounting, and audit events into the observability stack your operations team already watches
No parallel universe, no new silo — the model becomes another well-behaved service in the estate.
06 — Operations
Operations After Go-Live
Day two is the actual product — we build the deployment to be operated, then operate or co-operate it
Observability
- GPU utilisation, memory pressure, thermals, and power draw on dashboards, not folklore
- Latency percentiles, queue depth, and tokens/second per model
- Per-department token accounting — who uses what, at what cost
- Alert thresholds wired into your existing NOC and on-call practice
Evaluation & Drift
- The golden set from selection re-run on a schedule, not abandoned at go-live
- Quality regression gates before any model, adapter, or prompt change ships
- Hallucination and refusal-rate tracking on production traffic samples
- RAG retrieval quality monitored as the index and corpus evolve
Model Lifecycle
- Deliberate upgrade cadence — new releases evaluated on your tests, adopted on merit
- Shadow deployment: candidate model served side-by-side before it takes traffic
- Versioned model registry with rollback measured in minutes
- Quantization re-validated on every upgrade — precision behaviour does not carry over on trust
Resilience & Incident Handling
- Backup of configurations, adapters, prompts, and indexes — with restore actually drilled
- Defined degraded modes: smaller fallback model, queueing, or graceful refusal — never silent failure
- Incident runbooks and escalation paths written against your environment
- Support arrangements with named response expectations, from advisory to managed operation
A model you cannot monitor, evaluate, and roll back is not deployed — it is merely running.
Sovereignty is not where the model was made. It is where your data goes.
A GPU node is a server with an aggressive power budget. Treat it like one.
The weights are free. The capability is engineered.
08 — Enablement
Training & Handover
The deployment is not finished until your people can run it without us
Platform Administrators
Your operations team trained hands-on: serving stack, monitoring, model lifecycle, backup and rollback — with runbooks written against your estate, not generic vendor documentation.
End Users
Practical sessions on what the system can and cannot do: prompt discipline, when to trust and when to verify, data-handling rules, and how to report a bad answer so it improves the system.
Internal Champions
Power users in each department who field first-line questions, gather real usage feedback, and feed the evaluation loop — the difference between a tool that spreads and a tool that stalls.
We have trained technical and business teams across African markets for two decades — at DSI, at SkyTrend, and through ISP capacity-building programmes across the region. Enablement is a deliverable with a curriculum, not a slide at the end.
09 — Engagement
The Engagement: Assess to Enable
- 01
Assess
Use cases, data estate, regulatory constraints, and existing infrastructure — including the honest API-versus-private arithmetic. If private deployment is not justified, the engagement says so and stops here.
- 02
Size
Model shortlist evaluated on your tasks; VRAM, context, and concurrency arithmetic; hardware tier selection; and a facility survey covering power capacity, cooling, and UPS runtime before anything is ordered.
- 03
Pilot
A single-node deployment with real users and real documents. Golden-set baseline established, usage observed, and a go/no-go decision made on evidence rather than enthusiasm.
- 04
Harden
SSO and access control, network isolation, audit logging, backup and rollback, load testing, and a security review — the pilot becomes a system a regulator could inspect.
- 05
Operate
Monitoring wired into your NOC practice, evaluation and lifecycle cadence established, and a support arrangement agreed — from periodic advisory to fully managed operation.
- 06
Enable
Administrators and end users trained, runbooks handed over, champions established. The system, and the competence to run it, both stay with you.
Each phase gates the next. You can stop after any of them and keep everything produced so far.
The honest 2026 read: open-weight models are no longer the compromise option — the Qwen, DeepSeek, Llama, and gpt-oss class sit within a few points of frontier APIs on work-predictive tasks, several under MIT or Apache 2.0, and the serving layer has consolidated hard around vLLM and SGLang (Hugging Face archived TGI this spring) with FP8 now the default precision on Hopper-class hardware. What has not improved is the pilot-to-production rate — by most published counts barely one enterprise AI pilot in ten reaches production at scale, and the failures are operational, not model-quality: nobody sized the KV cache, nobody owned the evaluation set, nobody planned the rollback. The differentiator this year is not model access — everyone has that — it is whether the organisation can run inference like production infrastructure.
Find Out What Your Estate Can Actually Serve
A Phase 0 Review puts arithmetic behind the decision — your use cases, your data, your power and cooling reality — before a single GPU is ordered.