Code

History

Commits on feature/random_init to dev35+
  1. make hellaswag optional eval yayAndrej Karpathy committed 2 years agoda59861
  2. fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be readyAndrej Karpathy committed 2 years ago67239d9
  3. first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batchAndrej Karpathy committed 2 years ago051f3ca
  4. continued changes for sharded dataloaderAndrej Karpathy committed 2 years agoedb0df9
  5. extend dataloader to be shardedAndrej Karpathy committed 2 years ago3131028
  6. the write_datafile function accepts python list, which i think is faster but i didn't checkAndrej Karpathy committed 2 years agod536088
  7. adjust the dataloader to load the new .bin data files, and both prod datasets to use it insteadAndrej Karpathy committed 2 years ago1defbd4
  8. add fineweb, and add the first version of a new write_shard function that contains a header properlyAndrej Karpathy committed 2 years ago4bbd01f
  9. i also forgot to include the readme file for the new dev/data dirAndrej Karpathy committed 2 years ago7d11b79
  10. oops forgot to include data_common.pyAndrej Karpathy committed 2 years ago71774b3
  11. refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scaleAndrej Karpathy committed 2 years ago722e5b2
  12. small touchups to grad clipAndrej Karpathy committed 2 years agoc2d12f7
  13. Merge branch 'gradient-clipping' of https://github.com/ngc92/llm.c into ngc92-gradient-clippingAndrej Karpathy committed 2 years agoda9f4e2
  14. fix a bug, we have to be careful to make sure we evaluate loss at the token just before the first completion token, because that is the prediction for the first token, and its accuracyAndrej Karpathy committed 2 years ago8d55c4a
  15. report both acc and acc_normAndrej Karpathy committed 2 years ago4e14b52
  16. add mmlu as well and refine both a bitAndrej Karpathy committed 2 years ago7bd2389
  17. move hellaswag file to devAndrej Karpathy committed 2 years agoaec9ce5
  18. fixed up dev/cudaErik Schultheis committed 2 years ago66ce576
  19. added a useful mixed precision utility for dev/cudaErik Schultheis committed 2 years agod7a81ef
  20. Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgsAndrej committed 2 years ago109f516
  21. Merge pull request #416 from azret/feature/cpu_devAndrej committed 2 years ago9e04dea
  22. deterministic kernelErik Schultheis committed 2 years ago858c6e6
  23. general cleanupErik Schultheis committed 2 years ago2ccdfb7
  24. Setting up dev/CPU area with the first matmul_forward.cAzret Botash committed 2 years ago160b3bd
  25. remove legacy commentAndrej Karpathy committed 2 years ago2d43e5b
  26. Merge pull request #408 from ngc92/layernorm-bw-dev-cudaAndrej committed 2 years ago222d59f
  27. update dev/cuda/layernorm_backward and improve `validate_result` to take into account fp epsilon when comparing resultsErik Schultheis committed 2 years agoe553e2f
  28. fix layernorm backward: accumulate weight gradientErik Schultheis committed 2 years agodd8c9f5
  29. fixup commentErik Schultheis committed 2 years agoc66e48c
  30. fix CI compile by disabling kernel 5Erik Schultheis committed 2 years ago65727d5
  31. fix non-atomic version: * accumulate instead of assign * need dedicated argument to correctly handle the floatX == float caseErik Schultheis committed 2 years ago49ee3c8
  32. automatically switch to buffer-less version if that can fill up the GPUErik Schultheis committed 2 years ago081d224
  33. new kernel version with fewer atomicsErik Schultheis committed 2 years agoc0329eb
  34. enable bf16Erik Schultheis committed 2 years ago2287da0
  35. added current backward bias kernel to dev cudaErik Schultheis committed 2 years agoec92368