LLM Training Dataset and Checkpoint Optimization Engineer
Together AISan Francisco, California, United States
Posted August 12, 2025
160000 - 230000 USD year
Full Time
About the Company
Together AI is a research-driven AI cloud infrastructure provider offering a purpose-built GPU platform for training and running advanced AI models. Serving leading SaaS companies and pioneering startups, Together AI champions open source AI and decentralized computing, advocating for transparency to drive innovation and societal benefits.
Job Description
Together.ai is seeking a Training Dataset and Checkpoint Acceleration Engineer to optimize data pipelines and checkpoint mechanisms for large-scale machine learning workloads. The role involves designing and optimizing data workflows for streaming and processing massive datasets, implementing checkpointing strategies, and ensuring high performance and reliability. They will work at the intersection of data engineering and distributed systems.
Requirements
- 5+ years of experience in data engineering, distributed systems, or ML infrastructure
- Expertise in high-performance data processing libraries (e.g., PyTorch DataLoader, TensorFlow Data, DALI)
- Proficiency in distributed storage systems and data formats (e.g., Parquet, HDF5)
- Strong understanding of checkpointing frameworks and file systems (e.g., POSIX, Lustre, GPFS)
- Proficient in Python, C++, or Go
- Experience with I/O optimization techniques
- Familiarity with compression and serialization
- Analytical and problem-solving mindset
- Strong communication and collaboration skills
Benefits
- Competitive compensation
- Equity
- Health insurance