AI Engineer
Responsibilities
- Design pre-training technical strategies, including data composition, training hyperparameters (LR schedule, batch size ramp-up, progressive sequence length strategies), etc.
- Lead pre-training data engineering, including data cleaning (deduplication, PII filtering, quality classification), tokenization, data mixing strategies, and curriculum learning.
- Configure and optimize large-scale distributed training, including 3D parallelism, GPU memory optimization, and communication optimization.
- Develop and implement long-context training solutions, including RoPE scaling, NTK-aware interpolation, and YaRN.
- Monitor and optimize training performance, including loss curve analysis, gradient anomaly detection, training stability, and checkpoint management.
- Conduct regular intermediate evaluations and adjust data composition and training strategies based on evaluation results.
Responsibilities
- Bachelor’s degree or above in AI, NLP, Computer Science, Systems, or a related field.
- Proven end-to-end experience in large language model (LLM) pre-training projects, with hands-on experience pre-training models of 7B+ parameters.
- Strong expertise in large-scale distributed training frameworks, such as Megatron-LM, DeepSpeed, or FSDP, with hands-on experience training on 64+ GPUs.
- Strong understanding of pre-training data engineering, including data cleaning, deduplication, quality filtering, tokenization, and data mixture optimization.
- Familiarity with training monitoring and debugging, including loss analysis, gradient monitoring, and training stability management.
- Familiarity with long-context training techniques, including RoPE scaling, NTK-aware interpolation, and YaRN.