So you’ve decided you want to run AI in-house. Good decision. But now comes the part nobody writes about: actually setting it up in a South African office — with our power situation, our connectivity landscape, and our specific compliance requirements.
This isn’t a theoretical guide written by someone in Amsterdam or Austin. This is a practical walkthrough for deploying a private AI server in a Johannesburg office environment, with all the load shedding, Telkom outages, and budget realities that entails.
Let’s get into it.
Why “Run It Yourself” Makes Sense Here
Businesses in the UK or US can rely on cloud AI somewhat carelessly. Connectivity is cheap, stable, and fast. Power rarely goes out. Dollar-to-local-currency conversion isn’t a concern.
In South Africa, every one of those assumptions breaks down:
Power: Load shedding has settled into a new normal. Even in lighter schedules (Stage 2–3), you’re losing power multiple times per week. Stage 4–6 means your cloud-connected AI tools are down for hours at a time — exactly when you most need continuity.
Connectivity: South Africa’s internet infrastructure is improving, but international bandwidth remains expensive and occasionally unreliable. Every query to OpenAI or Anthropic crosses the ocean — adding latency, cost, and a dependency on undersea cables.
Currency: API costs are priced in USD. At current exchange rates, what looks like a modest API bill in dollars becomes a significant rand commitment — one that grows whenever the rand weakens.
Data sovereignty: POPIA is clear that organisations are responsible for data they hand to third parties. Sending sensitive business documents to a foreign AI API means accepting terms of service written to favour US law.
A private AI server solves all four problems simultaneously. It runs on your power supply, uses no international bandwidth after initial setup, costs rands not dollars, and keeps your data in your building.
Choosing Your Hardware
This is where most guides get vague. Let me be specific.
The GPU Is Everything
Modern AI models run on GPUs, not CPUs. When people talk about “AI hardware,” they’re primarily talking about VRAM — the memory on the graphics card. The more VRAM, the larger the model you can run, and the faster you can run it.
Here’s a practical breakdown for a Johannesburg office context:
Entry-level: NVIDIA RTX 4070 Ti or RTX 4080 (16 GB VRAM)
Approximate cost: R15,000–R22,000 for the GPU; R35,000–R50,000 for a complete workstation
What you can run: 7B–13B parameter models comfortably. Mistral 7B, Llama 3 8B, Qwen 7B — all at good speed. Suitable for a small team of 5–15 users doing general-purpose tasks: drafting documents, Q&A, summarisation.
This is the right starting point for an SME that wants to test local AI before committing to larger infrastructure.
Mid-range: NVIDIA RTX 4090 (24 GB VRAM)
Approximate cost: R28,000–R35,000 for the GPU; R60,000–R80,000 for a complete workstation
What you can run: Mixtral 8x7B at good speed, Llama 3 70B quantised at moderate speed. Serves 15–40 concurrent users for typical office tasks. This is the sweet spot for most Johannesburg SMEs and professional services firms.
Professional: Dual RTX 4090 or AI Workstation with 48–80 GB combined VRAM
Approximate cost: R150,000–R250,000 all-in
What you can run: Llama 3 70B at full speed, Qwen 72B quantised, or simultaneously running multiple smaller models for different departments. Suitable for mid-size businesses with heavy AI usage or compliance requirements around response speed.
Enterprise: NVIDIA DGX Spark
Approximate cost: R89,900–R99,000 (available from Evetech and Wootware in SA)
NVIDIA’s DGX Spark is worth a separate mention. It’s a purpose-built AI computer the size of a small speaker — 1.2 kg, 150mm × 150mm × 50mm — that delivers 1 petaFLOP of AI performance with 128 GB of unified memory. It can run models up to 200B parameters (or 405B when two units are linked via NVLink). It runs NVIDIA’s DGX OS with enterprise support.
For a business that wants a professionally supported, purpose-built AI appliance rather than a DIY workstation, the DGX Spark is a compelling option at its price point — especially since the rand cost is competitive with what you’d spend on a high-end consumer workstation build.
Power: The Load Shedding Problem
This section doesn’t exist in guides written for other markets. It should.
Sizing Your UPS
A GPU workstation doing AI inference draws 400–700W under load. Your UPS needs to handle this comfortably. A standard 1500VA UPS will give you roughly 20–30 minutes on a mid-range workstation — enough to gracefully shut down but not enough to ride out load shedding.
For genuine resilience, you need one of the following:
Option 1: High-capacity UPS (3–5 kVA) A 3kVA UPS with additional battery capacity can run a mid-range AI workstation through a 2-hour Stage 4 slot. Brands like Mecer, Victron, and Voltronic have proven themselves in the SA market. Budget R15,000–R35,000 for a setup that handles a 4-hour outage. This is the minimum viable solution for offices that can’t do solar.
Option 2: Inverter + battery bank A properly sized inverter installation (3–5 kW inverter with 100–200Ah lithium battery bank) will handle extended outages and pairs well with solar. If your office building already has solar panels, this is the path of least resistance. Cost varies widely but budget R40,000–R80,000 for a quality installation.
Option 3: Solar + battery (the permanent solution) A rooftop or carport solar installation with battery backup is increasingly common in Johannesburg commercial properties. If you’re in a building where this is feasible, this is the long-term answer — your AI server runs on solar during the day, battery at night, and you only notice load shedding when other people complain about it.
Practical tip: Configure your AI server for graceful shutdown at a battery threshold (say, 20% remaining) rather than a hard cutoff. Sudden power loss during model inference won’t damage the model weights (they’re files on disk) but can corrupt active conversation state and, in worst cases, database files if you’re running a RAG system.
Connectivity
Your private AI server doesn’t need internet connectivity to answer queries — that’s the point. But you do need connectivity for:
- Initial model downloads (large — 4B–70B files, many gigabytes)
- Software updates
- (Optionally) connecting remote staff via VPN
Fibre first, LTE backup
For a Johannesburg office, the standard recommendation is business fibre as primary (10–100 Mbps symmetric; providers like Vox, Frogfoot, and Openserve have good coverage in the JHB metro) with a 4G/5G LTE router as failover.
Your AI server’s internal network traffic — staff querying the model — doesn’t touch the internet at all. The only internet requirement is occasional model updates, which can be scheduled for off-peak hours.
Network switching matters more than internet speed
Staff querying your AI server over the office LAN need fast local network performance. A gigabit switch and Cat6 cabling are minimum. If your office is running on an old 100Mbps network, this is worth upgrading — not just for AI but for general productivity.
The Software Stack
Hardware is the foundation, but the software stack is what makes it usable. Here’s what we recommend for most Johannesburg office deployments:
Model Management: Ollama
Ollama is the simplest way to download, manage, and serve AI models locally. It handles the complexity of model quantisation, GPU layer management, and API serving behind a straightforward interface. One command downloads a model; another starts the server.
It runs on Linux, macOS, and Windows, and exposes an API that’s compatible with the OpenAI format — meaning most applications built for OpenAI will work with your local server without modification.
# Example: install Llama 3 70B on your server
ollama pull llama3:70b
# Start serving
ollama serve
User Interface: Open WebUI
Open WebUI (formerly Ollama WebUI) gives your staff a browser-based chat interface that looks and feels like ChatGPT but points at your local server. It supports multiple users, conversation history, document uploads, and model switching.
Deploy it on the same server as Ollama and give staff the local IP address — they open a browser tab and they’re running private AI. No app installation, no accounts, no data leaving the building.
For Developers: Continue or LM Studio
If your team includes developers, the Continue extension for VS Code connects to your local Ollama instance and gives them AI code assistance that works identically to GitHub Copilot — but private, local, and running entirely on your infrastructure.
(Optional) Document Intelligence: AnythingLLM
If you want to build a system that can answer questions based on your own documents — contracts, policies, reports, product manuals — AnythingLLM is an excellent addition. It handles document ingestion, vectorisation, and retrieval, and connects to your Ollama backend. This is what’s called a RAG system (more on that in our next post), and AnythingLLM makes it accessible without deep technical expertise.
Security Considerations
A private AI server has a fundamentally better security posture than cloud AI — your data never leaves the building. But there are still things to get right:
Network segmentation: Your AI server should be on your internal network, not exposed to the internet. If remote staff need access, implement a VPN rather than port-forwarding the AI server directly.
Access controls: Open WebUI supports user accounts and administrative controls. Create accounts for staff rather than sharing a single login. This lets you audit usage and revoke access when someone leaves.
Model storage: AI model weights are large files (4–40GB each) stored on your server’s hard drive. These files are not themselves sensitive (they’re the same files anyone can download), but your conversation history and uploaded documents are. Ensure your server storage is encrypted and that your backup strategy covers this data.
Staff awareness: The most common security failure in AI deployments isn’t technical — it’s staff pasting sensitive client data into public AI tools out of habit. Clear policy and accessible internal tooling (so staff have a good private alternative) is as important as the technical implementation.
What Does It Cost to Run?
Let’s make this concrete. Here’s a realistic cost model for a mid-size Johannesburg professional services firm:
One-time costs:
- AI workstation with RTX 4090: R70,000–R85,000
- UPS / power backup: R20,000–R35,000
- Network equipment (if upgrade needed): R5,000–R15,000
- Setup and configuration (professional): R15,000–R25,000
- Total one-time: ~R110,000–R160,000
Ongoing costs:
- Electricity: approximately R800–R1,500/month for a workstation running business hours
- Maintenance and updates: included in a support contract or minimal DIY time
- Total ongoing: ~R1,000–R2,000/month
Versus cloud AI (20-person team): A 20-person team using GPT-4 or Claude at moderate usage typically generates R8,000–R20,000+ per month in API costs (at current exchange rates). The break-even on a private server is 8–16 months — after which the marginal cost is essentially just electricity.
And that calculation doesn’t account for the rand continuing to weaken, which would make the cloud cost comparison increasingly favourable over time.
Getting Started: The Practical First Step
If you’re evaluating whether this is right for your business before committing to full infrastructure, there’s a sensible approach: start with a single workstation on your existing network.
A capable gaming PC with a modern GPU (even a secondhand RTX 3090 for R12,000–R18,000) can run Mistral 7B or Llama 3 8B competently enough for a pilot with a small team. Run it for 60 days. Measure actual usage patterns. Then scale your infrastructure decision on real data rather than projections.
Most businesses that run this pilot don’t go back to cloud-only AI. The combination of privacy, reliability, and cost control is too compelling — especially in the South African context.
Summary: The Load-Shedding-Resilient AI Stack
Here’s the full picture in one place:
| Component | Recommendation | Approximate Cost |
|---|---|---|
| AI Server (mid-range) | Workstation with RTX 4090 | R70,000–R85,000 |
| Power backup | 3–5 kVA UPS or inverter setup | R20,000–R40,000 |
| Network | Business fibre + LTE failover | R1,500–R3,000/month |
| Model software | Ollama (free, open-source) | — |
| User interface | Open WebUI (free, open-source) | — |
| Document AI (optional) | AnythingLLM (free, open-source) | — |
| Professional setup | Local AI specialist | R15,000–R25,000 |
The result: an AI platform that works during Stage 6, keeps your data in South Africa, and stops sending money offshore every time someone asks it a question.
We set up and support private AI infrastructure for Johannesburg businesses. If you’d like a site assessment and a hardware recommendation tailored to your specific team size and use cases, reach out here.
Photo by Google DeepMind