Andrej Karpathyfix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank0a00a3a82 years agoHistory
NameLast commit messageLast commit date
..
adamw.cufix the constness and the warning C4305: 'initializing': truncation from 'double' to 'float'2 years ago
attention_backward.cuone more version of attention backward for another few percent perf gains2 years ago
attention_forward.cuslow attention kernel that demonstrates a direct translation of the CPU code2 years ago
classifier_fused.cuadd kernel3, which is like kernel 2 but no float42 years ago
common.hadded NaN-based makes for reference checks2 years ago
crossentropy_forward.cunicer printing2 years ago
crossentropy_softmax_backward.cufix the bug that yields cpu, gpu results mismatch2 years ago
encoder_backward.cuencoder backward add, we want to get rid of atomics, hmm2 years ago
encoder_forward.curename positional_forward to encoder_forward as it should have been2 years ago
gelu_forward.cunicer printing2 years ago
layernorm_backward.cuincorporate layernorm backward kernel 2, we are now down to 73.1ms/iter on average2 years ago
layernorm_forward.cufix kernel 5 of layernorm, bug from previous copy paste with loop bounds2 years ago
matmul_backward.cufix the 0.0f initialization for double2 years ago
matmul_backward_bias.cunice new kernel for bias reduce, we are down by ~1ms/iter, to 76.13ms/iter on average2 years ago
matmul_forward.cufix indexing in matmul_forward_kernel12 years ago
nccl_all_reduce.cufix bug with all_hostsname_hashes missing, re-arrange code a bit, make sure we only print on rank02 years ago
README.mdmight as well push the few kernels that i feel ok about so far2 years ago
residual_forward.cunicer printing2 years ago
softmax_forward.cufix the 0.0f initialization for double2 years ago
trimat_forward.curename to _forward for the kernel, and small fixes to docs, and avoid auto2 years ago
README.md

dev/cuda

This directory is scratch space for developing various versions of the needed CUDA kernels. Each file develops a kernel, see the top of each file for instructions on how to compile and run each one.