FIELD NOTE / TIKTOK
Eight idle GPUs. Zero usable slots.
The complete written thought and the evidence behind it. The video edition will follow its public release.
The written argument is here.
This approved TikTok edition is on the journal now. Its video player and original platform link will appear after each public release is verified.
Eight idle GPUs. Zero usable slots.
Day 23 · 2026-10-16 · TikTok
Short video caption
Eight GPUs idle across two domains of four do not make one eight-GPU placement. In Kueue, a required topology can constrain all pods to one domain; cohort quota borrowing is a separate question. This is a simulated cluster, not a provider measurement. Ask for topology-valid queue time, fabric test results and completed useful GPU-hours by job shape before trusting a fleet utilization headline. #EricFieldNotes
Full written post / accessible read
A neocloud dashboard can show eight idle GPUs while your eight-GPU job waits. In this simulated cluster, four free devices sit in one fabric domain and four in another. The job needs eight together. Available inventory is not a valid placement.
The customer sees idle hardware and a growing queue. The operator sees stranded fragments. Adding every free card together will not answer when this job can start, and a high fleet utilization chart will not explain why another job is late.
Kueue can require a pod set to fit in one rack or block and calculates free capacity per topology domain. A queue may borrow quota from a cohort, but borrowed quota cannot create an eight-device domain where only two fragments of four exist. The scheduler needs both policy and placement evidence.
For each workload class, report time to a topology-valid allocation, fabric performance and completed useful GPU-hours. Rehearse your actual job size and placement rule before buying more capacity or promising a start time. Do this because utilization without topology can hide the very shortage customers pay for.
#EricFieldNotes
Evidence and boundary
On-screen boundary: SIMULATED PLACEMENT · KUEUE DOCUMENTED. The sources below support documented mechanisms and specifications; illustrative scenarios are not presented as measured incidents.
- Kubernetes Kueue introduction (S14)
- Kueue topology-aware scheduling (S59)
- NVIDIA NCCL troubleshooting (S60)
A GPU is not available to a job merely because it is idle
A neocloud can report idle devices while a customer waits for capacity. That is not necessarily a broken dashboard or a scheduler bug. Capacity is shaped by the request: device type and memory, the number of ranks that must start together, the fabric domain they need, queue quota, data location and recovery behavior.
Take a deliberately simple example. A fleet has eight idle GPUs: four in domain A and four in domain B. A distributed job requires eight within one domain. The fleet total is eight, but the largest valid placement is four. Adding the two fragments does not create the communication path the job requested. This is a simulated topology, not a measurement of a named provider.
Kueue's topology-aware scheduling makes this distinction explicit. It can require all pods of a set to fit within a rack or block, and its capacity calculation works by topology domain rather than only by fleet total. Kueue's ClusterQueue also has cohort borrowing: a queue may borrow resource quota from others. Borrowed quota can solve a policy shortage, but it cannot manufacture a contiguous topology domain where none exists. Kueue topology-aware scheduling · Kueue ClusterQueue.
Even a placement that passes the scheduler is not yet a fabric-performance proof. NVIDIA's NCCL troubleshooting guidance notes that GPU peer-to-peer communication depends on topology and configuration, and points to topology and peer-to-peer checks. For a real buying decision, I would ask for a representative collective-communication run on the offered placement, not just a list of GPUs and a nominal network speed. NVIDIA NCCL troubleshooting.
The useful metrics should follow the customer's job shape: queue wait until a topology-valid allocation, p95 start time for each job size, completed useful GPU-hours, preemption and recovery cost, and fabric performance under the actual collective pattern. The provider also needs to know which work can backfill small fragments without making a promised large job late. A high utilization target that ignores these constraints can worsen customer wait time while making the fleet chart look better.
For operators, the practical move is to replay actual workload shapes against topology and queue policy before buying or advertising capacity. For buyers, ask what happens to a job that needs a whole island, what queue and placement evidence you can inspect, and how charges treat retries, preemption and failed starts. “How many GPUs do you have?” is the beginning of the conversation. “How quickly can my shape of work start and finish?” is the question that decides value.
Evidence boundary: Kueue and NCCL documentation support the scheduling and fabric mechanisms. The eight-device 4+4 example and buying criteria are illustrative analysis, not provider performance data.