Operating infrastructure for visual search and diffusion models requires rethinking traditional web-service SRE assumptions. When individual inferences consume multiple seconds of GPU compute, typical load balancers fall short.

1. The Bottleneck: Head-of-Line Blocking

In standard round-robin ingress, long-running batch generative requests easily block lightweight feature-extraction queries if routed to the same pod instance.

Mitigation:

  • Decoupled traffic at the ingress gateway (Apache APISIX) using header-based routing.
  • Separated synchronous search endpoints (SLO: <400ms) from asynchronous generative workloads running behind Redis/Kafka worker queues.

2. KV-Cache & VRAM Management

Unlike CPU workloads, out-of-memory (OOM) on a GPU crashes the entire inference daemon (CUDA OOM), dumping all inflight connections.

Remedy:

  • Implemented admission control in front of the inference fleet to reject requests when memory limits reach 92%.
  • Integrated KEDA autoscalers monitoring actual model queue depths rather than host-level resource utilization.

3. Cost Governance

GPU instances are cost-prohibitive when idle. Leveraging custom operators with Karpenter/GKE node autoprovisioning cut idle spend by 30% without violating P99 SLA guarantees.