SAIG – SlewsIT AI Inference Gateway

Architecture Walkthrough

Northbound Interface: REST/HTTPS APIs for application, SDK, and client integration.

Gateway Data Plane: Stateless request processing, routing, load balancing, and streaming inference.

Control Plane: Centralized configuration, service discovery, and traffic policy management.

AI Inference Plane: GPU/CPU-based model execution for LLMs, embeddings, and ML pipelines.

Service Mesh: Internal gRPC communication between gateway and backend services.

Core Capabilities

• AI-aware API gateway with model-based routing
• Token-aware load balancing for GPU optimization
• GPU memory-aware request scheduling
• Streaming inference (SSE, WebSockets, gRPC)
• Prompt caching for cost and latency reduction
• Multi-tenant isolation with quotas and rate limiting
• Integrated AI guardrails (PII masking, prompt filtering)

Deployment Evolution

Phase 1 – CPU PoC: Go-based gateway with HTTP-to-gRPC bridging and CPU inference backends.

Phase 2 – Scaled Cluster: Horizontally scalable gateway instances with distributed service registry.

Phase 3 – Hardware Accelerated: DPU offload for networking and GPU-based inference for large-scale AI workloads.

AI Routing & Inference Model

• Model-based routing (LLMs, embeddings, search)
• Region and tenant-aware routing policies
• Dynamic routing based on GPU availability and queue depth
• Support for heterogeneous compute (CPU, GPU, DPU)

Use Case – Enterprise LLM Gateway

An enterprise deploying multiple AI models requires unified access, cost control, and performance optimization.

• Applications send inference requests through SAIG.
• Gateway routes requests to optimal GPU clusters based on model and load.
• Token-aware balancing maximizes GPU throughput.
• Observability tracks latency, tokens, and utilization.

Outcome: Reduced AI infrastructure cost with improved latency and centralized governance.

Observability & Control

• Real-time metrics (latency, tokens/sec, throughput)
• Distributed tracing across inference pipeline
• GPU utilization and queue monitoring
• Centralized configuration and policy management

Technology Stack

• Gateway: Go (high-performance networking)
• RPC: gRPC service mesh
• Config & Registry: etcd / Consul
• Metrics: Prometheus
• Tracing: OpenTelemetry
• Cache: Redis
• Orchestration: Kubernetes

Business Impact

• Optimize GPU utilization and reduce AI costs
• Enable scalable enterprise AI deployments
• Centralize AI access and governance
• Improve inference latency and throughput
• Accelerate AI platform standardization