GUIDE 3 OF 5

Software Stack for Long-Running Private AI

Configuring the operating system, serving frameworks, and persistence layer for continuous inference operations.

Foundation: Operating System Configuration

An always-on AI server starts with a properly configured operating system. Ubuntu Server 24.04 LTS provides the most extensive compatibility with AI frameworks while maintaining long-term support cycles that match production deployment timelines. The foundational principles from the Linux uptime guide apply directly: disable unnecessary services, configure automatic security updates (without automatic reboots), and set up kernel panic handling.

Critical kernel parameters for inference workloads include increasing the maximum number of memory map areas (vm.max_map_count), tuning the I/O scheduler for NVMe drives, and configuring swap behavior. For dedicated AI hardware running exclusively as inference nodes, you can safely disable power management features that might throttle compute under sustained load.

For edge deployments like persistent Jetson AI nodes, use JetPack 6.x with the minimal L4T (Linux for Tegra) configuration. Strip the desktop environment and graphical services — every watt saved goes to inference throughput. This is especially important for ClawBox 24/7 operation where power constraints are tight.

Model Serving Frameworks

The serving framework is the heart of your long-running private AI stack. The right choice depends on your hardware, model size, and throughput requirements:

Ollama — Best for Getting Started

One-command model pulling and serving. REST API. Automatic GPU detection. Ideal for individual developers and small teams. Not designed for high-concurrency production serving but perfectly adequate for dozens of concurrent requests on capable hardware. Supports GGUF quantized models out of the box, making it the fastest path to a working always-on AI server.

vLLM — Best for Production Throughput

PagedAttention for efficient KV cache management. Continuous batching. OpenAI-compatible API. Supports tensor parallelism across multiple GPUs. The gold standard for production inference serving at scale. Requires more configuration than Ollama but delivers significantly higher throughput under concurrent load.

llama.cpp — Best for Edge and Flexibility

Runs on practically anything — CUDA, Metal, Vulkan, ROCm, CPU. Supports every quantization format. The server mode provides an OpenAI-compatible API. Indispensable for persistent Jetson AI deployments where CUDA APIs may be limited and for Apple Silicon where Metal acceleration is key.

LocalAI — Best for Multi-Model Environments

OpenAI-compatible API with support for LLMs, embeddings, image generation, audio transcription, and TTS — all locally. Container-native design. Ideal when you need a unified API surface for multiple AI capabilities running on a single 24/7 AI hardware node.

Service Management & Persistence

Running a model interactively is trivial. Keeping it running through crashes, reboots, and resource exhaustion is where persistence engineering begins. Use systemd service units with Restart=always and RestartSec=5 to ensure automatic recovery. For the model server process, configure memory limits (MemoryMax) and watchdog timers to catch hung processes.

Complementary services that make an always-on AI server production-ready include:

  • NGINX Reverse Proxy: Terminates TLS, rate-limits requests, and provides access logging. Essential for exposing the inference API securely.
  • Prometheus + Grafana: Metrics collection for GPU utilization, VRAM usage, request latency, throughput, and error rates. Alert when inference performance degrades.
  • Vector Database: ChromaDB or Qdrant for RAG (Retrieval-Augmented Generation) workloads. Run alongside the model server for local document grounding.
  • Watchdog Timer: Hardware-level monitoring that reboots the system if the OS becomes unresponsive. Critical for unattended reliable always-on computing deployments.

Containerization Strategy

Docker with the NVIDIA Container Toolkit provides reproducible, isolated inference environments. For long-running private AI deployments, containerization ensures that dependency conflicts between frameworks do not destabilize the host system. Use Docker Compose for multi-service stacks (model server + vector DB + monitoring). 

# Example docker-compose.yml for a private AI stack
services:
  ollama:
    image: ollama/ollama:latest
    runtime: nvidia
    volumes:
      - ollama_data:/root/.ollama
    restart: always
    ports:
      - "11434:11434"

  chromadb:
    image: chromadb/chroma:latest
    volumes:
      - chroma_data:/chroma/chroma
    restart: always
    ports:
      - "8000:8000"

For Jetson platforms and other ARM64 edge devices, ensure all container images have ARM64 builds available. The ecosystem for persistent Jetson AI containers is growing rapidly, with most major serving frameworks now offering native ARM64 images.

Model Lifecycle Management

An always-on AI server needs a strategy for model updates. Pulling a new model version should not interrupt ongoing inference. Techniques include:

  • Blue-Green Model Serving: Run two instances of the model server (old and new versions), switch traffic gradually. Requires sufficient VRAM for both versions simultaneously.
  • Rolling Updates on Multi-Node Clusters: Update nodes one at a time behind a load balancer. Documented extensively by the STH always-on lab community.
  • Model Registry: Use a local registry (Harbor, Zot) to cache model images and avoid pulling from external sources on every restart.

For truly reliable always-on computing, automate model health checks. A simple script that sends a test prompt and validates the response format can catch corrupted weights or configuration drift before users notice.

← Previous: Hardware Selection Next: Performance Optimization →