GPU cloud margins: why most neoclouds fail and how GPU overcommit fixes it
Why do most GPU cloud startups fail to make profit?
Most GPU cloud startups fail to make profit because their GPU utilisation rate is too low. The average utilisation across non-optimised GPU cloud deployments is 20% to 40%, according to industry estimates cited in hosted·ai's own deployment data. At that utilisation rate, the revenue generated per GPU does not cover the combined cost of hardware depreciation, power, cooling, and operations. The fix is not more customers. It is software that allows operators to sell more effective capacity from the same physical hardware through GPU overcommit, combined with a customer mix that keeps utilisation high through diverse workload timing. Neoclouds that have solved the utilisation problem through overcommit ratios of 2x to 5x, combined with inference-heavy customer bases, achieve gross margins of 40% to 60% on GPU cloud. Those that have not are competing on price in a market where the most capitalised operators will always win.
The P&L waterfall: what the numbers actually look like
Take a concrete example. An operator buys 10 H100 80GB nodes.
- Hardware cost: $280,000 (at $28,000 per node, a reasonable 2026 price).
- Depreciation over three years: $7,800/month.
- Power and cooling: $3,500/month (8 GPUs per node, 10 nodes, approximately $0.05/hr per GPU in power cost).
- Total fixed cost: roughly $11,300/month before staff and software.
At 20% utilisation, charging $2.50/hr per GPU, the 80 GPUs generate $2.88/month in revenue. That is a $8,400/month loss before staff costs.
At 80% utilisation with the same pricing, revenue is $11,520/month. Roughly breakeven.
With 5x overcommit at 80% effective utilisation, you are selling the equivalent of 400 GPU slots from 80 physical GPUs. Revenue at $2.50/hr and 80% effective utilisation: $57,600/month against $11,300/month fixed costs. Gross margin: roughly 80%.
The overcommit ratio is not magic. It works because GPU workloads are bursty. Inference jobs run for seconds to minutes, not continuously at 100% load. A pool of 80 GPUs with 5x overcommit can serve 400 concurrent inference slots because at any given moment, only a fraction of those slots are actively consuming GPU cycles. The scheduler manages the queue and ensures each job gets the resources it needs when it needs them.
Why the utilisation problem is a software problem
Low utilisation is not primarily a sales problem. Most neoclouds that struggle have customers. Their customers just are not generating enough continuous load to keep utilisation high.
The structural cause is customer mix. If all your customers are running training jobs, utilisation follows the training cycle: high for days or weeks during a training run, then near zero between runs. One customer finishing a training run can drop your cluster utilisation by 30 points overnight.
Inference customers are different. They run continuously at lower and more consistent load. A model serving production traffic runs 24 hours a day. Inference customers generate more predictable utilisation curves and respond better to overcommit-based pricing because they care more about latency consistency than raw throughput.
The operators with the best unit economics typically have a customer mix that is at least 60% inference by workload hours. That mix is more predictable, more amenable to overcommit, and more stable as a revenue base. For example, packet.ai uses hosted.ai tech to overcommit and had reached to $1M ARR in just a few month and is profitable from day 1.
How GPU overcommit works technically
GPU overcommit in the hosted·ai platform works through a combination of temporal and spatial scheduling.
Temporal scheduling context-switches GPU workloads in and out of physical GPU time slots. Each tenant gets full isolated access to the GPU during their allocated slot. No two tenants co-exist on the GPU simultaneously. This is the highest-security mode and works well for workloads where isolation is the priority, even at some performance overhead from context-switching.
Spatial scheduling fits multiple tenant workloads onto the GPU simultaneously, allocating VRAM and compute dynamically based on available resources. Workloads co-exist. This delivers better performance for latency-sensitive inference because there is no context-switching delay, but tenants share the physical GPU at the same time.
The overcommit ratio, from 2x to 10x per pool, controls how many virtual GPU slots are sold per physical GPU. A 5x ratio on 10 H100s creates 50 sellable slots. The scheduler ensures that the aggregate load stays within what the physical hardware can handle without degrading performance below the service level the operator has committed to.
The platform allows different pools to have different overcommit ratios and scheduling modes. A neocloud might run a security-optimised pool at 2x overcommit for enterprise customers with strict isolation requirements, and a performance-optimised pool at 5x overcommit for inference customers who care more about cost than isolation.
What does a profitable neocloud customer mix look like?
The operators running profitable GPU clouds at scale share a few characteristics.
First, their inference-to-training revenue ratio is at least 60:40. Training revenue is lumpy; inference revenue is recurring and predictable.
Second, they have at least one anchor customer who runs consistent 24/7 workloads. That anchor customer creates a utilisation floor that makes overcommit economics work for the rest of the pool.
Third, their pricing is not purely time-based. Charging by the GPU-hour is fine, but charging by VRAM allocation, TFLOPs consumed, or on a subscription basis gives more pricing flexibility and better aligns price with value for different workload types.
Fourth, they are not trying to undercut AWS on price. The operators who compete on price alone against hyperscalers are in a permanent race to the bottom. The ones who survive long enough to be profitable are competing on something hyperscalers cannot easily match: compliance, latency, support, or specific GPU types at specific price points.
Running the numbers for your situation
The economics vary significantly based on hardware cost, power cost, customer mix, and the overcommit ratio the workloads will support. The hosted·ai team can walk through the P&L for a specific infrastructure configuration in a 30-minute call, using actual deployment data rather than market averages.