GPU infrastructure glossary

The control plane is the centralized management layer that handles provisioning, scheduling, monitoring, and policy enforcement across all GPU infrastructure, regardless of where it physically sits.

Autoscaling automatically adds or removes GPU capacity in response to real-time demand, so infrastructure grows and shrinks with workload rather than sitting at a fixed size.

Bin packing is a scheduling strategy that places workloads as tightly as possible onto the fewest GPUs necessary, minimizing fragmented, unusable leftover capacity.

Capacity planning is the process of forecasting future GPU demand and provisioning hardware or contracts ahead of time to meet it without overbuying or running short.

Chargeback and showback are cost-allocation models that attribute GPU infrastructure spend back to the teams, projects, or customers consuming it. Chargeback bills them; showback just reports it.

Cluster management is the operational discipline of keeping a group of GPU servers healthy, updated, and correctly configured as a single managed unit.

Cost optimization (or rightsizing) is the ongoing practice of matching GPU allocation to actual workload needs, eliminating overprovisioned capacity without hurting performance.

Data residency is the requirement that data be stored and processed within a specific geographic or legal jurisdiction, often driven by national regulation.

Fractional GPU (or GPU slicing) is the allocation of less than one full GPU's compute or memory to a workload, so multiple lightweight jobs can share a single card.

A GPU hypervisor is software that sits between physical GPU hardware and virtual machines or containers, mediating access so multiple isolated workloads can share the same GPU.

GPU orchestration is the automated allocation, scheduling, and lifecycle management of GPU resources across a cluster, matching workloads to available capacity without manual intervention.

GPU pooling is the aggregation of physical GPUs across servers, racks, or sites into a single logical resource pool that workloads can draw from on demand.

A GPU scheduler decides which workload runs on which GPU (or GPU slice) and when, based on priority, availability, and resource requirements.

GPU utilization is the percentage of a GPU's available compute or memory capacity that's actively being used by workloads at any given time.

GPU virtualization is the practice of abstracting a physical GPU into multiple logical instances so several workloads or tenants can share the same hardware safely and efficiently.

Hybrid cloud GPU management is the coordination of GPU workloads across a mix of on-prem hardware and public cloud capacity, typically through a single control plane.

Idle GPU capacity is compute or memory on a provisioned GPU that isn't being used by any workload, representing a direct loss of potential revenue or productivity.

Job queuing holds workloads that can't run immediately until capacity frees up, while priority scheduling decides which queued job runs next based on rules like urgency or tenant tier.

A Kubernetes GPU operator is software that automates the deployment, configuration, and lifecycle management of GPU drivers, monitoring, and device plugins within a Kubernetes cluster.

MIG is an NVIDIA hardware feature that physically partitions a supported GPU into up to seven fully isolated instances, each with its own memory and compute cores.

Multi-tenancy is the ability to safely run multiple customers' or teams' workloads on shared GPU infrastructure, with strict isolation of compute, memory, and data between them.

A node pool is a group of servers (nodes) with similar hardware specifications — such as the same GPU model — managed together as a single scalable unit.

On-prem AI infrastructure refers to GPU compute, storage, and networking hardware owned and operated within an organization's own data center rather than rented from a cloud provider.

Passthrough gives a single VM or container direct, exclusive access to a physical GPU, while a virtualized GPU is shared across multiple tenants through a hypervisor or partitioning layer.

Preemption is the forced suspension or eviction of a lower-priority workload so a higher-priority one can access GPU resources immediately.

A private AI cloud is a GPU infrastructure environment dedicated to a single organization, offering cloud-like elasticity and self-service without sharing hardware with outside tenants.

Resource quotas are limits set on how much GPU compute, memory, or storage a user, team, or project can consume, preventing any one group from monopolizing shared infrastructure.

RBAC is an access-control model that grants users permissions based on their assigned role rather than managing permissions individually, simplifying governance at scale.

Self-service provisioning lets end users request and launch GPU instances on demand through a portal or API, without filing a ticket or waiting on an infrastructure team.

An SLA (service-level agreement) is a formal commitment on GPU infrastructure performance or availability, such as uptime, provisioning time, or guaranteed compute access, with defined consequences if it isn't met.

SOC 2 is an auditing standard that evaluates a service provider's controls around security, availability, processing integrity, confidentiality, and privacy of customer data.

Sovereign AI refers to a country or region's ability to develop, host, and control AI infrastructure and models within its own jurisdiction, independent of foreign cloud providers.

Telemetry is the continuous collection of performance and usage data, including GPU utilization, memory pressure, temperature, and job duration, from running workloads and infrastructure.

Tenant isolation is the guarantee that one tenant's workload cannot access, interfere with, or observe another tenant's data or compute on shared infrastructure.

A vGPU is a virtual instance of a physical GPU created through virtualization software, giving a VM or container dedicated GPU compute without owning the whole card.

Workload orchestration is the process of deciding where, when, and how a compute job runs across available infrastructure, based on priority, resource needs, and constraints.

Zero-trust architecture is a security model that assumes no user, workload, or network segment is inherently trusted, requiring continuous verification for every access request.