Operate and optimize large-scale LLM pre-training on 1,000+ GPU clusters using PyTorch, DeepSpeed, or Megatron-LM. Improve networking (InfiniBand/RDMA), memory management, checkpointing, and failure recovery. Manage SLURM/Kubernetes GPU clusters and apply systems engineering (C++, CUDA, Python) and 3D parallelism techniques.
We are seeking a highly skilled LLM Pre-training & Distributed Systems Engineer. This role is essential for orchestrating large-scale machine learning training runs and optimizing distributed infrastructure. The ideal candidate will have a deep understanding of GPU clusters and extensive experience in system engineering to ensure efficient and reliable training processes.
Responsibilities:
- Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
- Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
- Automate checkpointing and failure recovery during month-long training runs.
Required Skills:
- Deep expertise in 3D parallelism (Data, Tensor, Pipeline).
- Experience managing SLURM or Kubernetes-based GPU clusters.
- Strong systems engineering background (C++, CUDA, Python).
Similar Jobs
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Lead the strategic and operational direction of a specialized casualty claims team managing severe-exposure portfolios. Oversee claims managers, complex litigation, coverage strategy, reserves, settlements, legal vendors, and portfolio analytics. Collaborate with legal, underwriting, actuarial, reinsurance, and executive stakeholders to manage emerging risks, litigation trends, loss costs, and claim outcomes. Develop protocols and best practices for environmental, mass tort, toxic tort, latent injury, and other complex claims.
Top Skills:
Guidewire Claims System
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Build and operate AWS GovCloud data infrastructure across development, preproduction, and production environments. Establish PostgreSQL platforms, infrastructure-as-code with Chef, BI infrastructure, identity integrations, monitoring, observability, CI/CD, and disaster recovery. Support data pipelines, troubleshoot incidents, optimize databases, and ensure FedRAMP and FISMA compliance. Collaborate with security, compliance, DevOps, and data engineering teams while delivering greenfield infrastructure from architecture through production.
Top Skills:
AWSAws GovcloudBashChefCi/CdCloudwatchDatadogEltETLGitGitlabGoogle SamlJenkinsNagiosPostgresPythonRubySsoTableau ServerVpc
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Design, implement, optimize, and maintain scalable hybrid multi-cloud Kubernetes platforms operating at massive scale. Integrate open-source technologies, improve platform reliability, provide technical direction, mentor engineers, and participate in on-call support. The role requires expertise managing large Kubernetes clusters, observability tools, Linux environments, public clouds, custom data centers, and AI-enabled workflow improvements.
Top Skills:
AlertmanagerAWSGCPGoGrafanaHybrid Multi-CloudKubernetesKubernetes OperatorsLinuxOciPrometheusThanos
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
