History

Merge pull request #1 from GaoYusong/claude/add-claude-documentation-sJh2l

Yusong Gao committed 8 months ago

Commit d69ea922 parents d0d49a9 5f64af51 file · +170 −0Browse files

Unified · Split

CLAUDE.md+170−0
@@ -0,0 +1,170 @@
1+# CLAUDE.md
2+
3+## Project Overview
4+
5+**llm.cpp** is a C++ port of [llm.c](https://github.com/karpathy/llm.c) that trains GPT-2 (124M parameters) from scratch. It features **tinytorch**, a single-header tensor library with automatic differentiation, inspired by [ggml](https://github.com/ggerganov/ggml).
6+
7+## Build & Run
8+
9+### Prerequisites
10+
11+- **Compiler**: `clang` with C++20 support
12+- **OpenMP** (optional but recommended): `brew install libomp` (macOS) or `sudo apt-get install libomp-dev` (Ubuntu)
13+- **Data files**: Run `./dev/download_starter_pack.sh` to download GPT-2 weights and training data
14+
15+### Build Commands
16+
17+```bash
18+make all # Build everything (default)
19+make train_gpt2 # Build GPT-2 training binary
20+make test_gpt2 # Build GPT-2 tests
21+make test_tensor # Build tensor operation tests
22+make clean # Remove compiled binaries
23+```
24+
25+### Running
26+
27+```bash
28+OMP_NUM_THREADS=8 ./train_gpt2 # Train GPT-2
29+./test_tensor # Run tensor tests
30+./test_gpt2 # Run GPT-2 tests (requires data files)
31+cd example && make && ./example # Run tinytorch example
32+```
33+
34+### Performance Benchmarking
35+
36+```bash
37+./run_perf.sh # Runs test_gpt2, saves timestamped results to perf/
38+```
39+
40+## Project Structure
41+
42+```
43+tinytorch.hpp # Core single-header tensor library (~1400 lines)
44+train_gpt2.cpp # GPT-2 training pipeline with AdamW optimizer
45+test_gpt2.cpp # GPT-2 forward/backward pass tests
46+test_tensor.cpp # Comprehensive tensor operation tests
47+Makefile # Build configuration (clang, C++20, OpenMP)
48+
49+llmc/ # LLM utility headers
50+ dataloader.h # Distributed batched data loader
51+ tokenizer.h # GPT-2 tokenizer (decode-only)
52+ utils.h # Error-checked file I/O macros
53+ rand.h # Mersenne Twister RNG (PyTorch-compatible)
54+
55+example/ # Standalone tinytorch usage example
56+ tinytorch_example.cpp # Fruit classification MLP
57+ Makefile
58+
59+dev/ # Development scripts
60+ download_starter_pack.sh # Downloads model weights and training data
61+ gen_tensor_test.py # Generates test data for tensor tests
62+
63+boost/
64+ ut.hpp # Boost.UT testing framework (header-only)
65+```
66+
67+## Architecture
68+
69+### Computation Graph Pattern
70+
71+tinytorch uses a DAG-based computation graph with explicit forward/backward passes:
72+
73+1. Build graph by chaining tensor operations (returns references to new tensors)
74+2. Call `.Forward()` on the output to evaluate the graph
75+3. Call `.ZeroGrad()` then `.Backward()` for gradient computation
76+4. Update parameters manually (e.g., AdamW in `train_gpt2.cpp`)
77+
78+### Tensor Operations (12 total)
79+
80+- **Element-wise**: Add (`+`), Mul (`*`), Norm (LayerNorm)
81+- **Linear algebra**: MatMul
82+- **Reshaping**: View, Transpose, Split, Lookup, Broadcast
83+- **Activations**: GELU
84+- **Loss**: Softmax, CrossEntropy
85+
86+### Memory Management
87+
88+- `TensorContext` uses an arena allocator with a single pre-allocated buffer
89+- 16-byte aligned allocations (`kTensorMemAlign`) for SIMD compatibility
90+- Objects tracked via a linked list of `Object` structs
91+- No manual pointer management in application code; context owns all memory
92+
93+### GPT-2 Model Structure (`train_gpt2.cpp`)
94+
95+- `GPT2Config` — model hyperparameters
96+- `Embedding` — token + position embeddings
97+- `Block` — transformer block (LayerNorm, Multi-head Attention, FFN)
98+- `LMHead` — final layer norm
99+- `GPT2` — full model with parameters, optimizer state, and context
100+
101+## Coding Conventions
102+
103+### Style & Formatting
104+
105+- **Formatter**: clang-format with Google base style, 4-space indent, 100-char column limit
106+- **Linter**: clang-tidy (warnings as errors), based on gemma.cpp configuration
107+- **Standard**: C++20
108+- **Header guards**: `#pragma once`
109+
110+### Naming (enforced by clang-tidy)
111+
112+| Element | Convention | Example |
113+|---------|-----------|---------|
114+| Classes/Structs | `CamelCase` | `TensorContext`, `GPT2Config` |
115+| Enums/Constants | `kCamelCase` | `kF32`, `kOpAdd`, `kMaxTensorDims` |
116+| Namespaces | `lower_case` | `tinytorch` |
117+| Functions | Flexible (any case) | `MatMul()`, `add_forward()`, `safe_printf()` |
118+| Class members | `lower_case_` (trailing underscore) | `mem_buffer_`, `type_`, `data_` |
119+
120+### Patterns
121+
122+- **Error handling**: Macro-based with file/line context (`fopenCheck`, `freadCheck`, `mallocCheck` in `llmc/utils.h`); `assert()` for invariants; `std::runtime_error` for allocation failures
123+- **Includes**: Standard library first, then system headers, then project headers
124+- **File extensions**: `.hpp` for C++ headers (tinytorch), `.h` for C-style headers (llmc/)
125+- **Namespace**: Core library lives in `namespace tinytorch`; application code uses `using namespace tinytorch; // NOLINT`
126+- **NOLINT markers**: Used where clang-tidy exceptions are intentional
127+
128+### Adding a New Tensor Operation
129+
130+1. Add enum value to `TensorOp` in `tinytorch.hpp`
131+2. Implement `<op>_forward()` and `<op>_backward()` functions
132+3. Add cases to the `Forward()` and `Backward()` switch dispatchers
133+4. Add a method on the `Tensor` class that creates the graph node
134+5. Write tests in `test_tensor.cpp` using Boost.UT
135+
136+## Testing
137+
138+### Framework
139+
140+[Boost.UT](https://github.com/boost-ext/ut) — header-only, included in `boost/ut.hpp`.
141+
142+### Test Structure
143+
144+```cpp
145+ut::test("TestName") = [&ctx] {
146+ // Arrange: create tensors, fill with data
147+ auto &a = *ctx.NewTensor(dims)->Fill(data);
148+ // Act: build and evaluate computation graph
149+ auto &c = a + b;
150+ c.Forward();
151+ // Assert: check results
152+ ut::expect(CheckVectorEqual(c.Flatten(), expected));
153+};
154+```
155+
156+### Running Tests
157+
158+```bash
159+make test_tensor && ./test_tensor # Tensor operation tests
160+make test_gpt2 && ./test_gpt2 # GPT-2 model tests (needs data files)
161+```
162+
163+`test_gpt2` requires downloading data first via `./dev/download_starter_pack.sh`.
164+
165+## Key Files to Read First
166+
167+1. `tinytorch.hpp` — the core library; understand `TensorContextT`, `Tensor`, and the operation dispatch
168+2. `train_gpt2.cpp` — see how the library is used to build and train GPT-2
169+3. `example/tinytorch_example.cpp` — minimal example of the tinytorch API
170+4. `test_tensor.cpp` — reference for all supported tensor operations