CLAUDE.md

Project Overview

llm.cpp is a C++ port of llm.c that trains GPT-2 (124M parameters) from scratch. It features tinytorch, a single-header tensor library with automatic differentiation, inspired by ggml.

Build & Run

Prerequisites

• Compiler: clang with C++20 support
• OpenMP (optional but recommended): brew install libomp (macOS) or sudo apt-get install libomp-dev (Ubuntu)
• Data files: Run ./dev/download_starter_pack.sh to download GPT-2 weights and training data

Build Commands

make all            # Build everything (default)
make train_gpt2     # Build GPT-2 training binary
make test_gpt2      # Build GPT-2 tests
make test_tensor    # Build tensor operation tests
make clean          # Remove compiled binaries

Running

OMP_NUM_THREADS=8 ./train_gpt2    # Train GPT-2
./test_tensor                      # Run tensor tests
./test_gpt2                        # Run GPT-2 tests (requires data files)
cd example && make && ./example    # Run tinytorch example

Performance Benchmarking

./run_perf.sh    # Runs test_gpt2, saves timestamped results to perf/

Project Structure

tinytorch.hpp              # Core single-header tensor library (~1400 lines)
train_gpt2.cpp             # GPT-2 training pipeline with AdamW optimizer
test_gpt2.cpp              # GPT-2 forward/backward pass tests
test_tensor.cpp            # Comprehensive tensor operation tests
Makefile                   # Build configuration (clang, C++20, OpenMP)

llmc/                      # LLM utility headers
  dataloader.h             #   Distributed batched data loader
  tokenizer.h              #   GPT-2 tokenizer (decode-only)
  utils.h                  #   Error-checked file I/O macros
  rand.h                   #   Mersenne Twister RNG (PyTorch-compatible)

example/                   # Standalone tinytorch usage example
  tinytorch_example.cpp    #   Fruit classification MLP
  Makefile

dev/                       # Development scripts
  download_starter_pack.sh #   Downloads model weights and training data
  gen_tensor_test.py       #   Generates test data for tensor tests

boost/
  ut.hpp                   # Boost.UT testing framework (header-only)

Architecture

Computation Graph Pattern

tinytorch uses a DAG-based computation graph with explicit forward/backward passes:

1. Build graph by chaining tensor operations (returns references to new tensors)
2. Call .Forward() on the output to evaluate the graph
3. Call .ZeroGrad() then .Backward() for gradient computation
4. Update parameters manually (e.g., AdamW in train_gpt2.cpp)

Tensor Operations (12 total)

• Element-wise: Add (+), Mul (*), Norm (LayerNorm)
• Linear algebra: MatMul
• Reshaping: View, Transpose, Split, Lookup, Broadcast
• Activations: GELU
• Loss: Softmax, CrossEntropy

Memory Management

• TensorContext uses an arena allocator with a single pre-allocated buffer
• 16-byte aligned allocations (kTensorMemAlign) for SIMD compatibility
• Objects tracked via a linked list of Object structs
• No manual pointer management in application code; context owns all memory

GPT-2 Model Structure (train_gpt2.cpp)

• GPT2Config — model hyperparameters
• Embedding — token + position embeddings
• Block — transformer block (LayerNorm, Multi-head Attention, FFN)
• LMHead — final layer norm
• GPT2 — full model with parameters, optimizer state, and context

Coding Conventions

Style & Formatting

• Formatter: clang-format with Google base style, 4-space indent, 100-char column limit
• Linter: clang-tidy (warnings as errors), based on gemma.cpp configuration
• Standard: C++20
• Header guards: #pragma once

Naming (enforced by clang-tidy)

Patterns

• Error handling: Macro-based with file/line context (fopenCheck, freadCheck, mallocCheck in llmc/utils.h); assert() for invariants; std::runtime_error for allocation failures
• Includes: Standard library first, then system headers, then project headers
• File extensions: .hpp for C++ headers (tinytorch), .h for C-style headers (llmc/)
• Namespace: Core library lives in namespace tinytorch; application code uses using namespace tinytorch; // NOLINT
• NOLINT markers: Used where clang-tidy exceptions are intentional

Adding a New Tensor Operation

1. Add enum value to TensorOp in tinytorch.hpp
2. Implement <op>_forward() and <op>_backward() functions
3. Add cases to the Forward() and Backward() switch dispatchers
4. Add a method on the Tensor class that creates the graph node
5. Write tests in test_tensor.cpp using Boost.UT

Testing

Framework

Boost.UT — header-only, included in boost/ut.hpp.

Test Structure

ut::test("TestName") = [&ctx] {
    // Arrange: create tensors, fill with data
    auto &a = *ctx.NewTensor(dims)->Fill(data);
    // Act: build and evaluate computation graph
    auto &c = a + b;
    c.Forward();
    // Assert: check results
    ut::expect(CheckVectorEqual(c.Flatten(), expected));
};

Running Tests

make test_tensor && ./test_tensor   # Tensor operation tests
make test_gpt2 && ./test_gpt2      # GPT-2 model tests (needs data files)

test_gpt2 requires downloading data first via ./dev/download_starter_pack.sh.

Key Files to Read First

1. tinytorch.hpp — the core library; understand TensorContextT, Tensor, and the operation dispatch
2. train_gpt2.cpp — see how the library is used to build and train GPT-2
3. example/tinytorch_example.cpp — minimal example of the tinytorch API
4. test_tensor.cpp — reference for all supported tensor operations