Code

History

Commits on move/encoder_kernel to dev/cuda/common.h26
  1. utilities for Packed and more vectorizationErik Schultheis committed 2 years agocb8cc25
  2. remove legacy commentAndrej Karpathy committed 2 years ago2d43e5b
  3. update dev/cuda/layernorm_backward and improve `validate_result` to take into account fp epsilon when comparing resultsErik Schultheis committed 2 years agoe553e2f
  4. remove warning noisengc92 committed 2 years ago5920143
  5. fused layernorm+residualngc92 committed 2 years ago691c1df
  6. moved bf16 boilerplate to common.hngc92 committed 2 years ago99e765d
  7. tiny irrelevant optimisation to final unaligned fused_classifier loop + add missing common.h changesademeure committed 2 years ago1ea7f9b
  8. mixed precision utilities for dev/cudaErik Schultheis committed 2 years ago06980b6
  9. small tweaksAndrej Karpathy committed 2 years ago769c911
  10. fixed potential error and generalized gelu forwardErik Schultheis committed 2 years agoc3515cf
  11. delete packed128 .fp32, use (float) okAndrej Karpathy committed 2 years ago7ba1ed8
  12. clear the L2 cache between consecutive invokations of our microbenchmarks to get reliable resultsErik Schultheis committed 2 years agod7813d2
  13. remove forceinline. blame @arund42 if this ever does not get inlined by the compilerAndrej Karpathy committed 2 years agoaf2bc47
  14. move packed128 to commonAndrej Karpathy committed 2 years ago5e12c66
  15. Fix sizeof(T) -> sizeof(D) (was overallocating memory)ademeure committed 2 years ago49ffd68
  16. Use kernel6 for train_gpt2, and add all other kernels to /dev/cuda (+bugfixes)ademeure committed 2 years ago82fef03
  17. added NaN-based makes for reference checksErik Schultheis committed 2 years ago732a8b4
  18. towards an even better backward attention kernelErik Schultheis committed 2 years ago16123c3
  19. size_t instead of int. everywhere. this will come to bite us at one point...Andrej Karpathy committed 2 years agoc393d82
  20. push my attention_backward fileAndrej Karpathy committed 2 years ago8a673ed
  21. add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so farAndrej Karpathy committed 2 years ago6be8080
  22. make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in commonAndrej Karpathy committed 2 years agoc5ee371
  23. fix crossentropy forward, loss requires probabilities so these floats must be 0..1 instead of -1..1 or we get nansAndrej Karpathy committed 2 years ago8414541
  24. fixed re-initialization of device memory after testErik Schultheis committed 2 years agof459bfa
  25. improved testingErik Schultheis committed 2 years ago59bada9
  26. extracted common testing functions into their own headerErik Schultheis committed 2 years ago7e8e038