GPU Orchestration Lab
I split model serving, image generation, translation, and embedding work into GPU pools that can scale and fail independently.
What needed to change
One monolithic GPU image makes unrelated workloads slow to ship and expensive to run. Orchestration state, asset handoff, and GPU-heavy execution need different lifecycles.
The constraint shaping the build
Worker pools must scale independently, tolerate interruption, and pass large artifacts without forcing the controller to become a data plane.
See whether this starting point fits your workflow
The matching system explains who it is for, what a first release includes, and the rough cost. Supporting links show the implementation evidence.
How I built it
- Keep a small Prefect control layer and isolate workload-specific Docker images.
- Use separate pools for embeddings, vLLM serving, tagging, generation, and translation.
- Move artifacts through S3-compatible storage rather than the orchestration database.
- Tie RunPod worker scaling to queue demand and the hardware profile of each stage.
What I verified
- Workload-specific worker images
- Independent GPU pools
- Object-storage handoff
- Queue-driven scaling controls
Current state: The controller, worker boundaries, and artifact handoff are implemented. No public uptime or cost-saving claim is attached.