Code

History

Commits on move/encoder_kernel to dev245+
  1. Merge branch 'master' of github.com:karpathy/llm.cAndrej Karpathy committed 2 years ago66a0b52
  2. push my attention_backward fileAndrej Karpathy committed 2 years ago8a673ed
  3. add slightly more optimized adamw kernelnopperl committed 2 years ago97e61e2
  4. add adamw cuda kernelnopperl committed 2 years ago5684ff7
  5. ensure valid test cases and incorporate fixes by @ademeureErik Schultheis committed 2 years ago43bf9eb
  6. optimised version of the cool new fused classifier + hopefully bugfixes?ademeure committed 2 years agob225501
  7. fix the backward dbias precision issuelancer committed 2 years ago6ff4ad6
  8. add naive backward for layernorm too. continue backward pass for the full model. adjust gpt2_testcu, we can finally check our first parameter gradients! technically this means that we could now train some of the parameters of the model, jut the final layernorm haha. but the kernels are really inefficient so farAndrej Karpathy committed 2 years ago6be8080
  9. minor fixes to matmul backwardAndrej Karpathy committed 2 years ago3de5077
  10. add matmul backward kernel, problem is the bias backward, ironically. which i thought i could cublas at firstAndrej Karpathy committed 2 years ago9560b1a
  11. Merge pull request #117 from ngc92/fused-classifierAndrej committed 2 years ago3c6dafe
  12. fixed numerics for online kernel, and an existing race condition in kernel 2Erik Schultheis committed 2 years agoc73fa88
  13. kernel 1 should also accumulate in double to ensure stabilityErik Schultheis committed 2 years ago8fb966b
  14. use V dimension in softmax tests adjusted tests to be more challengingErik Schultheis committed 2 years ago60cf281
  15. nicer printingErik Schultheis committed 2 years ago99b42c1
  16. fix timing outputErik Schultheis committed 2 years agofe1d8cf
  17. initial versionErik Schultheis committed 2 years ago780d8bd
  18. erase the old kernel to save LOC and do some renaming and adjust dev/cuda code to add the new largeC kernel as kernel 7Andrej Karpathy committed 2 years ago534b93a
  19. make the tests own their cublas cublaslt handles and settings, instead of putting that stuff in commonAndrej Karpathy committed 2 years agoc5ee371
  20. fix comment block of gelu forward kernelAndrej Karpathy committed 2 years ago8262351
  21. remove unused variable CAndrej Karpathy committed 2 years ago02b5b33
  22. fix crossentropy forward, loss requires probabilities so these floats must be 0..1 instead of -1..1 or we get nansAndrej Karpathy committed 2 years ago8414541
  23. add comments and handling of kernels and their correctnessAndrej Karpathy committed 2 years ago69f0385
  24. fixed re-initialization of device memory after testErik Schultheis committed 2 years agof459bfa
  25. mark problematic testsErik Schultheis committed 2 years agod035725
  26. improved testingErik Schultheis committed 2 years ago59bada9
  27. const'ness fixesErik Schultheis committed 2 years ago7e7460c
  28. extracted common testing functions into their own headerErik Schultheis committed 2 years ago7e8e038
  29. incorporate the fastest kernels and adjust readmeAndrej Karpathy committed 2 years ago6b49ed1
  30. few commentsAndrej Karpathy committed 2 years ago6f9e28a
  31. Merge branch 'attention-softmax' of https://github.com/ngc92/llm.c into ngc92-attention-softmax-2Andrej Karpathy committed 2 years ago2f99b3e
  32. small changes to comments, and very minor code modifications and movementsAndrej Karpathy committed 2 years ago40b78be
  33. Merge branch 'online_softmax' of https://github.com/lancerts/llm.c into lancerts-online_softmaxAndrej Karpathy committed 2 years ago6250336
  34. alright let's start the backward pass. crossentropy_softmax_backward first, ez oneAndrej Karpathy committed 2 years agoa18e517
  35. fix handles in cublas in attention forward as wellAndrej Karpathy committed 2 years ago59b9e8b