Unified · Split
| @@ -0,0 +1,170 @@ | |||
| 1 | # CLAUDE.md | ||
| 2 | |||
| 3 | ## Project Overview | ||
| 4 | |||
| 5 | **llm.cpp** is a C++ port of [llm.c](https://github.com/karpathy/llm.c) that trains GPT-2 (124M parameters) from scratch. It features **tinytorch**, a single-header tensor library with automatic differentiation, inspired by [ggml](https://github.com/ggerganov/ggml). | ||
| 6 | |||
| 7 | ## Build & Run | ||
| 8 | |||
| 9 | ### Prerequisites | ||
| 10 | |||
| 11 | - **Compiler**: `clang` with C++20 support | ||
| 12 | - **OpenMP** (optional but recommended): `brew install libomp` (macOS) or `sudo apt-get install libomp-dev` (Ubuntu) | ||
| 13 | - **Data files**: Run `./dev/download_starter_pack.sh` to download GPT-2 weights and training data | ||
| 14 | |||
| 15 | ### Build Commands | ||
| 16 | |||
| 17 | ```bash | ||
| 18 | make all # Build everything (default) | ||
| 19 | make train_gpt2 # Build GPT-2 training binary | ||
| 20 | make test_gpt2 # Build GPT-2 tests | ||
| 21 | make test_tensor # Build tensor operation tests | ||
| 22 | make clean # Remove compiled binaries | ||
| 23 | ``` | ||
| 24 | |||
| 25 | ### Running | ||
| 26 | |||
| 27 | ```bash | ||
| 28 | OMP_NUM_THREADS=8 ./train_gpt2 # Train GPT-2 | ||
| 29 | ./test_tensor # Run tensor tests | ||
| 30 | ./test_gpt2 # Run GPT-2 tests (requires data files) | ||
| 31 | cd example && make && ./example # Run tinytorch example | ||
| 32 | ``` | ||
| 33 | |||
| 34 | ### Performance Benchmarking | ||
| 35 | |||
| 36 | ```bash | ||
| 37 | ./run_perf.sh # Runs test_gpt2, saves timestamped results to perf/ | ||
| 38 | ``` | ||
| 39 | |||
| 40 | ## Project Structure | ||
| 41 | |||
| 42 | ``` | ||
| 43 | tinytorch.hpp # Core single-header tensor library (~1400 lines) | ||
| 44 | train_gpt2.cpp # GPT-2 training pipeline with AdamW optimizer | ||
| 45 | test_gpt2.cpp # GPT-2 forward/backward pass tests | ||
| 46 | test_tensor.cpp # Comprehensive tensor operation tests | ||
| 47 | Makefile # Build configuration (clang, C++20, OpenMP) | ||
| 48 | |||
| 49 | llmc/ # LLM utility headers | ||
| 50 | dataloader.h # Distributed batched data loader | ||
| 51 | tokenizer.h # GPT-2 tokenizer (decode-only) | ||
| 52 | utils.h # Error-checked file I/O macros | ||
| 53 | rand.h # Mersenne Twister RNG (PyTorch-compatible) | ||
| 54 | |||
| 55 | example/ # Standalone tinytorch usage example | ||
| 56 | tinytorch_example.cpp # Fruit classification MLP | ||
| 57 | Makefile | ||
| 58 | |||
| 59 | dev/ # Development scripts | ||
| 60 | download_starter_pack.sh # Downloads model weights and training data | ||
| 61 | gen_tensor_test.py # Generates test data for tensor tests | ||
| 62 | |||
| 63 | boost/ | ||
| 64 | ut.hpp # Boost.UT testing framework (header-only) | ||
| 65 | ``` | ||
| 66 | |||
| 67 | ## Architecture | ||
| 68 | |||
| 69 | ### Computation Graph Pattern | ||
| 70 | |||
| 71 | tinytorch uses a DAG-based computation graph with explicit forward/backward passes: | ||
| 72 | |||
| 73 | 1. Build graph by chaining tensor operations (returns references to new tensors) | ||
| 74 | 2. Call `.Forward()` on the output to evaluate the graph | ||
| 75 | 3. Call `.ZeroGrad()` then `.Backward()` for gradient computation | ||
| 76 | 4. Update parameters manually (e.g., AdamW in `train_gpt2.cpp`) | ||
| 77 | |||
| 78 | ### Tensor Operations (12 total) | ||
| 79 | |||
| 80 | - **Element-wise**: Add (`+`), Mul (`*`), Norm (LayerNorm) | ||
| 81 | - **Linear algebra**: MatMul | ||
| 82 | - **Reshaping**: View, Transpose, Split, Lookup, Broadcast | ||
| 83 | - **Activations**: GELU | ||
| 84 | - **Loss**: Softmax, CrossEntropy | ||
| 85 | |||
| 86 | ### Memory Management | ||
| 87 | |||
| 88 | - `TensorContext` uses an arena allocator with a single pre-allocated buffer | ||
| 89 | - 16-byte aligned allocations (`kTensorMemAlign`) for SIMD compatibility | ||
| 90 | - Objects tracked via a linked list of `Object` structs | ||
| 91 | - No manual pointer management in application code; context owns all memory | ||
| 92 | |||
| 93 | ### GPT-2 Model Structure (`train_gpt2.cpp`) | ||
| 94 | |||
| 95 | - `GPT2Config` — model hyperparameters | ||
| 96 | - `Embedding` — token + position embeddings | ||
| 97 | - `Block` — transformer block (LayerNorm, Multi-head Attention, FFN) | ||
| 98 | - `LMHead` — final layer norm | ||
| 99 | - `GPT2` — full model with parameters, optimizer state, and context | ||
| 100 | |||
| 101 | ## Coding Conventions | ||
| 102 | |||
| 103 | ### Style & Formatting | ||
| 104 | |||
| 105 | - **Formatter**: clang-format with Google base style, 4-space indent, 100-char column limit | ||
| 106 | - **Linter**: clang-tidy (warnings as errors), based on gemma.cpp configuration | ||
| 107 | - **Standard**: C++20 | ||
| 108 | - **Header guards**: `#pragma once` | ||
| 109 | |||
| 110 | ### Naming (enforced by clang-tidy) | ||
| 111 | |||
| 112 | | Element | Convention | Example | | ||
| 113 | |---------|-----------|---------| | ||
| 114 | | Classes/Structs | `CamelCase` | `TensorContext`, `GPT2Config` | | ||
| 115 | | Enums/Constants | `kCamelCase` | `kF32`, `kOpAdd`, `kMaxTensorDims` | | ||
| 116 | | Namespaces | `lower_case` | `tinytorch` | | ||
| 117 | | Functions | Flexible (any case) | `MatMul()`, `add_forward()`, `safe_printf()` | | ||
| 118 | | Class members | `lower_case_` (trailing underscore) | `mem_buffer_`, `type_`, `data_` | | ||
| 119 | |||
| 120 | ### Patterns | ||
| 121 | |||
| 122 | - **Error handling**: Macro-based with file/line context (`fopenCheck`, `freadCheck`, `mallocCheck` in `llmc/utils.h`); `assert()` for invariants; `std::runtime_error` for allocation failures | ||
| 123 | - **Includes**: Standard library first, then system headers, then project headers | ||
| 124 | - **File extensions**: `.hpp` for C++ headers (tinytorch), `.h` for C-style headers (llmc/) | ||
| 125 | - **Namespace**: Core library lives in `namespace tinytorch`; application code uses `using namespace tinytorch; // NOLINT` | ||
| 126 | - **NOLINT markers**: Used where clang-tidy exceptions are intentional | ||
| 127 | |||
| 128 | ### Adding a New Tensor Operation | ||
| 129 | |||
| 130 | 1. Add enum value to `TensorOp` in `tinytorch.hpp` | ||
| 131 | 2. Implement `<op>_forward()` and `<op>_backward()` functions | ||
| 132 | 3. Add cases to the `Forward()` and `Backward()` switch dispatchers | ||
| 133 | 4. Add a method on the `Tensor` class that creates the graph node | ||
| 134 | 5. Write tests in `test_tensor.cpp` using Boost.UT | ||
| 135 | |||
| 136 | ## Testing | ||
| 137 | |||
| 138 | ### Framework | ||
| 139 | |||
| 140 | [Boost.UT](https://github.com/boost-ext/ut) — header-only, included in `boost/ut.hpp`. | ||
| 141 | |||
| 142 | ### Test Structure | ||
| 143 | |||
| 144 | ```cpp | ||
| 145 | ut::test("TestName") = [&ctx] { | ||
| 146 | // Arrange: create tensors, fill with data | ||
| 147 | auto &a = *ctx.NewTensor(dims)->Fill(data); | ||
| 148 | // Act: build and evaluate computation graph | ||
| 149 | auto &c = a + b; | ||
| 150 | c.Forward(); | ||
| 151 | // Assert: check results | ||
| 152 | ut::expect(CheckVectorEqual(c.Flatten(), expected)); | ||
| 153 | }; | ||
| 154 | ``` | ||
| 155 | |||
| 156 | ### Running Tests | ||
| 157 | |||
| 158 | ```bash | ||
| 159 | make test_tensor && ./test_tensor # Tensor operation tests | ||
| 160 | make test_gpt2 && ./test_gpt2 # GPT-2 model tests (needs data files) | ||
| 161 | ``` | ||
| 162 | |||
| 163 | `test_gpt2` requires downloading data first via `./dev/download_starter_pack.sh`. | ||
| 164 | |||
| 165 | ## Key Files to Read First | ||
| 166 | |||
| 167 | 1. `tinytorch.hpp` — the core library; understand `TensorContextT`, `Tensor`, and the operation dispatch | ||
| 168 | 2. `train_gpt2.cpp` — see how the library is used to build and train GPT-2 | ||
| 169 | 3. `example/tinytorch_example.cpp` — minimal example of the tinytorch API | ||
| 170 | 4. `test_tensor.cpp` — reference for all supported tensor operations | ||