Principal Model Optimization Engineer at Roblox in San Mateo, CA

Job description

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. ML Platform at Roblox supports hundreds of ML use cases and billions of inferences per day across Discovery, Safety, Engine and much more. We build what ML engineers use to go from idea to production: data and training pipelines, the SDKs and tools they build with, the GPU fleet and the scheduling that runs their jobs. As a Principal Engineer on ML Platform, you will set the technical direction for these systems. You will find what slows ML engineers down or wastes compute and design durable fixes. You will measure the results and keep improving them until they show up as faster iteration, more reliable jobs and better use of every GPU. You Will: Design platform capabilities that many teams adopt, so that one investment improves hundreds of use cases instead of one. Build GPU and compute management that keeps a scarce, mixed fleet busy. This covers quotas and priority tiers, preemption, scheduling across clusters, and capacity planning with Infrastructure. Define the platform’s success metrics, instrument them, and improve them quarter over quarter. Examples include time from code to first training run, job success rate, GPU utilization, and cost per training run or per inference. Lead deep investigations when jobs are slow or failing, from pipeline and scheduler behavior down to GPU profiles and hardware errors. Turn the findings into fixes that stay fixed. Work closely with ML engineers and data scientists to learn how they work, and make the paved path the easiest one to take. Partner with Infrastructure, Data Platform and Security on shared foundations such as storage, networking, data access and compliance. Raise the technical bar through design reviews, written proposals and code, and mentor senior engineers. You Have: 6+ years of professional experience and a tool chest of system design experience upon which to draw to build performant systems for all of Roblox. A track record of building ML infrastructure end to end. Examples include training pipelines, workflow orchestration (e.g. Kubeflow, Airflow, Ray), data or feature pipelines, experiment tracking and model registries, or model serving. Experience designing SDKs, APIs or developer tools that were widely adopted, and a strong sense of what makes a platform easy to use. Hands-on experience running GPU workloads at scale on Kubernetes. That includes scheduling, quotas, multi-cluster dispatch, autoscaling, and debugging GPU and node failures (e.g. Xid errors, NCCL hangs). Enough depth in training and inference performance (distributed training, mixed precision, batching, quantization) to know where the platform should do the work for its users. A habit of measuring: you set clear success metrics, ship in steps, and keep iterating until the numbers move. Experience leading cross-team technical work without direct authority, and writing clearly for both engineers and leaders. A Bachelor’s degree in Computer Science, Computer Engineering or a related field, or equivalent experience. For roles that are based at our headquarters in San Mateo, CA: The starting base pay for this position is as shown below. The actual base pay is dependent upon a variety of job-related factors such as professional background, tr