Distributed LLM Pretraining: TorchTitan
Sets up distributed LLM pretraining at scale using torchtitan's 4D parallelism.
Test report
- Verdict
- Works with setup
- Score
- Tested
- Jul 12, 2026
- Environment
- Claude Code 2.x (agent harness)
- Upstream re-checked
- Aug 10, 2026 · 7b08ed5
Real use hard-requires a multi-GPU cluster (8-512+ GPUs, ideally H100) plus a Hugging Face token for tokenizer download — prerequisites the SKILL.md never flags upfront despite its turnkey 'launch training' framing — so testing was scoped to the config artifact rather than actual training. Install was clean (single cp, strict-YAML frontmatter, all 4 reference files present). A/B on a Llama3-8B config task: both arms produced valid FSDP2 + Float8 + compile + selective-AC TOMLs, but the skill-armed config wired the float8 converter into [model] converters (so float8 actually activates), while the base config configured [quantize.linear.float8] yet omitted the converter registration, meaning its float8 would silently never apply.
Scored on four weighted criteria — install, triggering, output vs. baseline, docs. How scoring works
- Installs cleanly 5/5
- Triggers reliably 4/5
- Output vs. baseline 7/10
- Docs & honesty 3/5
What Distributed LLM Pretraining: TorchTitan does
Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.
How to install Distributed LLM Pretraining: TorchTitan
git clone https://github.com/Orchestra-Research/AI-Research-SKILLs
cd AI-Research-SKILLs
mkdir -p ~/.claude/skills
cp -r 01-model-architecture/torchtitan ~/.claude/skills/distributed-llm-pretraining-torchtitan
Skills live in ~/.claude/skills/ (global) or .claude/skills/
(per-project). Restart Claude Code after installing.
Commands — how to trigger Distributed LLM Pretraining: TorchTitan
-
/distributed-llm-pretraining-torchtitanSets up distributed LLM pretraining at scale using torchtitan's 4D parallelism.
It also activates on plain-language prompts like these:
-
Pretrain Llama 3.1 across 64 GPUs with torchtitan -
Set up FSDP2 and tensor parallelism for pretraining -
Configure distributed checkpointing for a large training run
Frequently asked questions
- Is the Distributed LLM Pretraining: TorchTitan skill free?
- Yes. The skill itself is free from Orchestra-Research/AI-Research-SKILLs. SkillProof publishes the install command and an independent test verdict at no cost.
- Does Distributed LLM Pretraining: TorchTitan work with Claude Code?
- We tested it with Claude Code 2.x (agent harness) on Jul 12, 2026. Verdict: Works with setup. Real use hard-requires a multi-GPU cluster (8-512+ GPUs, ideally H100) plus a Hugging Face token for tokenizer download — prerequisites the SKILL.md never flags upfront despite its turnkey 'launch training' framing — so testing was scoped to the config artifact rather than actual training. Install was clean (single cp, strict-YAML frontmatter, all 4 reference files present). A/B on a Llama3-8B config task: both arms produced valid FSDP2 + Float8 + compile + selective-AC TOMLs, but the skill-armed config wired the float8 converter into [model] converters (so float8 actually activates), while the base config configured [quantize.linear.float8] yet omitted the converter registration, meaning its float8 would silently never apply.
- What is the Distributed LLM Pretraining: TorchTitan SkillProof Score?
- 7.6/10 — installs cleanly 5/5, triggers reliably 4/5, output vs. baseline 7/10, docs & honesty 3/5.
- How do I install Distributed LLM Pretraining: TorchTitan?
- Copy the install command from this page, run it in your terminal, and restart Claude Code. Skills live in ~/.claude/skills/ (global) or .claude/skills/ inside a project.
- Can I use Distributed LLM Pretraining: TorchTitan with Cursor, Copilot, Gemini CLI, Codex or other AI tools?
- The SKILL.md format is native to Claude (Claude Code, Desktop, claude.ai). The instructions inside adapt to other assistants: Cursor rules, GitHub Copilot instructions, Windsurf rules, Custom GPTs, AGENTS.md for OpenAI Codex, and GEMINI.md for Google Gemini CLI — our conversion guides cover each, and the free converter on the tools page does the wrapping for you.