GPU infrastructure glossary

GPU infrastructure specifically optimized and sized for running trained models to serve predictions or generate outputs, as distinct from training infrastructure.

Autoscaling automatically adds or removes GPU capacity in response to real-time demand, so infrastructure grows and shrinks with workload rather than sitting at a fixed size.

Capacity planning is the process of forecasting future GPU demand and provisioning hardware or contracts ahead of time to meet it without overbuying or running short.

Cluster management is the operational discipline of keeping a group of GPU servers healthy, updated, and correctly configured as a single managed unit.

Cost optimization (or rightsizing) is the ongoing practice of matching GPU allocation to actual workload needs, eliminating overprovisioned capacity without hurting performance.

Data residency is the requirement that data be stored and processed within a specific geographic or legal jurisdiction, often driven by national regulation.

Fractional GPU (or GPU slicing) is the allocation of less than one full GPU's compute or memory to a workload, so multiple lightweight jobs can share a single card.

The frameworks operators use to understand and forecast the profitability of GPU infrastructure, factoring in hardware cost, overcommit ratio, and utilization.

The business models and mechanisms an infrastructure owner uses to turn GPU hardware into revenue, whether direct rental, wholesale, or white-label resale.

GPU orchestration is the automated allocation, scheduling, and lifecycle management of GPU resources across a cluster, matching workloads to available capacity without manual intervention.

Allocating more virtual GPU capacity to tenants than physically exists on the underlying hardware, on the expectation not all tenants demand their full allocation at once.

GPU pooling is the aggregation of physical GPUs across servers, racks, or sites into a single logical resource pool that workloads can draw from on demand.

A GPU scheduler decides which workload runs on which GPU (or GPU slice) and when, based on priority, availability, and resource requirements.

GPU utilization is the percentage of a GPU's available compute or memory capacity that's actively being used by workloads at any given time.

GPU virtualization is the practice of abstracting a physical GPU into multiple logical instances so several workloads or tenants can share the same hardware safely and efficiently.

A cloud delivery model where GPU compute is offered as a metered, on-demand service rather than sold or leased as fixed hardware.

Hybrid cloud GPU management is the coordination of GPU workloads across a mix of on-prem hardware and public cloud capacity, typically through a single control plane.

Idle GPU capacity is compute or memory on a provisioned GPU that isn't being used by any workload, representing a direct loss of potential revenue or productivity.

A Kubernetes GPU operator is software that automates the deployment, configuration, and lifecycle management of GPU drivers, monitoring, and device plugins within a Kubernetes cluster.

MIG is an NVIDIA hardware feature that physically partitions a supported GPU into up to seven fully isolated instances, each with its own memory and compute cores.

Multi-tenancy is the ability to safely run multiple customers' or teams' workloads on shared GPU infrastructure, with strict isolation of compute, memory, and data between them.

A GPU-focused cloud provider, typically newer and narrower in scope than a hyperscaler, built specifically around GPU compute for AI workloads.

A node pool is a group of servers (nodes) with similar hardware specifications — such as the same GPU model — managed together as a single scalable unit.

On-prem AI infrastructure refers to GPU compute, storage, and networking hardware owned and operated within an organization's own data center rather than rented from a cloud provider.

A private AI cloud is a GPU infrastructure environment dedicated to a single organization, offering cloud-like elasticity and self-service without sharing hardware with outside tenants.

Resource quotas are limits set on how much GPU compute, memory, or storage a user, team, or project can consume, preventing any one group from monopolizing shared infrastructure.

RBAC is an access-control model that grants users permissions based on their assigned role rather than managing permissions individually, simplifying governance at scale.

Self-service provisioning lets end users request and launch GPU instances on demand through a portal or API, without filing a ticket or waiting on an infrastructure team.

Sovereign AI refers to a country or region's ability to develop, host, and control AI infrastructure and models within its own jurisdiction, independent of foreign cloud providers.

Telecommunications companies and CSPs launching GPU-based AI cloud services, typically leveraging existing data center, network, and customer relationships.

Tenant isolation is the guarantee that one tenant's workload cannot access, interfere with, or observe another tenant's data or compute on shared infrastructure.

In AI inference, the distinction between input tokens (the prompt) and output tokens (the generated response), often metered or priced differently.

A vGPU is a virtual instance of a physical GPU created through virtualization software, giving a VM or container dedicated GPU compute without owning the whole card.

Charging customers based on the amount of GPU memory they consume over time, rather than a flat per-GPU or per-instance rate.

GPU infrastructure that a service provider operates and brands as its own, even though the underlying orchestration software may come from a third party.

Workload orchestration is the process of deciding where, when, and how a compute job runs across available infrastructure, based on priority, resource needs, and constraints.