• Parallelism
  • Tensor
    • Megatron
    • Alpa
  • Pipeline
    • GPipe
• Attention
  • PagedAttention
  • FlashAttention
• Scheduling
  • iteration level Scheduling: OCRA