Software Engineer, Training Infrastructure
Thinking Machines Lab is hiring for Software Engineer, Training Infrastructure in San Francisco. Mid level, $300k–$400k a year stated. First seen November 17, 2026; open as of November 18, 2026. Jobs taken straight from company career pages — no aggregator in between.
Save to applyFree forever. No commitment, no card required.
- Company
- Thinking Machines Lab
- Where
- San Francisco
- Type
- Full-time
- Level
- Mid
- Pay stated
- $300k–$400k a year
- First seen
- November 17, 2026
- Listed via
- Ashby
- Open for
- 1 day, as of November 18, 2026
- Experience asked
- 3 to 5 years
- Visa sponsorship
- Offered
About this job
This position is responsible for developing and managing the distributed training systems that support the company's frontier models and its fine-tuning API. You will ensure the platform remains fast, stable, and scalable as new models and users are introduced. The role is located in San Francisco or New York.
Responsibilities
- Develop, implement, and manage the distributed training infrastructure for the fine-tuning API and internal model training, including scheduling, orchestration, checkpointing, and recovery.
- Enhance training speed and reduce costs through profiling, parallelism strategies, memory optimization, and kernel- and communication-level tuning.
- Establish and refine training support for new open-weights and proprietary model architectures, including mixture-of-experts models, and accelerate the delivery of new models to users.
- Integrate reliability into the platform via automated failure detection, job recovery, multi-tenant isolation, and observability to ensure long-running jobs withstand hardware and network failures.
- Facilitate efficient multi-tenant LoRA fine-tuning and tightly coupled sampling and training for reinforcement learning workloads.
- Collaborate with the API, frameworks, and product teams to maintain consistency and proper behavior across the infrastructure and user-facing SDK.
Requirements
- A minimum of 4 years in software engineering, including the development and operation of production distributed systems.
- Proficiency in Python, along with experience in a systems language such as C++, Rust, or Go.
- Practical experience with large-scale model training or inference infrastructure, utilizing frameworks like PyTorch.
- Demonstrated ability to debug complex performance and reliability issues in distributed environments.
Nice to have
- Experience in training or serving large language models at scale, including mixture-of-experts architectures.
- Familiarity with LoRA and other parameter-efficient fine-tuning methods, or with supervised and reinforcement learning pipelines for LLMs.
- Experience with cluster orchestration and scheduling tools like Kubernetes or Slurm, and with fault-tolerant checkpointing.
- Experience in developing observability and profiling tooling.
- Contributions to open-source ML systems projects (e.g., PyTorch, Megatron, vLLM, SGLang, or similar).
Benefits
- Comprehensive health, dental, and vision coverage.
- Flexible time off.
- Paid parental leave.
- Relocation assistance when required.
Read more and apply on Thinking Machines Lab's site
Questions about this job
When was this Software Engineer, Training Infrastructure job at Thinking Machines Lab posted?
Destava first saw it on November 17, 2026, on Thinking Machines Lab's own careers page (Ashby).
Is Thinking Machines Lab still accepting applications for it?
It was open on Thinking Machines Lab's own site when Destava last read it, on November 18, 2026, 1 day after we first saw it. When the company takes it down, this page says the job has closed.
How do I apply for Software Engineer, Training Infrastructure at Thinking Machines Lab?
On Thinking Machines Lab's own site, at jobs.ashbyhq.com: open the posting. Destava does not take the application. Save the job to your Destava board to track it, see how well you fit and tailor your résumé to it.
Is this Thinking Machines Lab job posting legit?
Destava read it on Thinking Machines Lab's own careers page at jobs.ashbyhq.com, run on Ashby, not on a job board. First seen November 17, 2026. Apply on the company's own site, and never pay to apply for a job.
What does this job pay?
The posting states $300k–$400k a year.
Does Thinking Machines Lab sponsor a visa for this job?
The posting says visa sponsorship is offered or possible.
How much experience does it ask for?
3 to 5 years.
What other jobs is Thinking Machines Lab hiring for?
Thinking Machines Lab has 58 open postings on its own site: see all jobs at Thinking Machines Lab.
How many software developers jobs are open in San Francisco Bay Area?
5,137 open, 654 of them new this week, at 1,110 companies: see them all.
More jobs like this
Internship-Generative AI developer
Save to applySoftware Engineer III, Apple
Save to applyCustomer Application Engineer
Save to applySoftware Engineer, Backend - Government Products
Save to apply
See all 5,137 Software Developers jobs in San Francisco Bay Area
A search agent finds the jobs this list misses: the ones that mean the same without the same words. Have Destava watch for software engineer, training infrastructure jobs in San Francisco, new matches every morning on your own account.
Applying to this one?
See how well you fit, before you apply
Save it to your board. Add your résumé once, and Destava shows where you are strong, what is missing and how to pitch yourself, then tailors your résumé to this posting.
- Found it
- Save it
- See your fit
Free forever. No commitment, no card required.
With Destava: Save to apply · Tailor my résumé · Write a cover letter
This page includes information from the O*NET 31.0 Database by the U.S. Department of Labor, Employment and Training Administration (USDOL/ETA). Used under the CC BY 4.0 license. O*NET® is a trademark of USDOL/ETA.