Code

History

Commits on move/encoder_kernel to dev70+
  1. update fine-web preprocessing: faster, and much less RAM consumptionErik Schultheis committed 2 years agob66eb66
  2. make hellaswag optional eval yayAndrej Karpathy committed 2 years agoda59861
  3. fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be readyAndrej Karpathy committed 2 years ago67239d9
  4. first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batchAndrej Karpathy committed 2 years ago051f3ca
  5. continued changes for sharded dataloaderAndrej Karpathy committed 2 years agoedb0df9
  6. added new layernorm backward to /dev/cuda/ademeure committed 2 years ago7cbeefc
  7. extend dataloader to be shardedAndrej Karpathy committed 2 years ago3131028
  8. the write_datafile function accepts python list, which i think is faster but i didn't checkAndrej Karpathy committed 2 years agod536088
  9. adjust the dataloader to load the new .bin data files, and both prod datasets to use it insteadAndrej Karpathy committed 2 years ago1defbd4
  10. add fineweb, and add the first version of a new write_shard function that contains a header properlyAndrej Karpathy committed 2 years ago4bbd01f
  11. i also forgot to include the readme file for the new dev/data dirAndrej Karpathy committed 2 years ago7d11b79
  12. oops forgot to include data_common.pyAndrej Karpathy committed 2 years ago71774b3
  13. refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scaleAndrej Karpathy committed 2 years ago722e5b2
  14. update the utils function and assertlancer committed 2 years ago2b0667a
  15. fix the unsupported block_sizelancer committed 2 years ago6348d41
  16. small touchups to grad clipAndrej Karpathy committed 2 years agoc2d12f7
  17. Merge branch 'gradient-clipping' of https://github.com/ngc92/llm.c into ngc92-gradient-clippingAndrej Karpathy committed 2 years agoda9f4e2
  18. fix a bug, we have to be careful to make sure we evaluate loss at the token just before the first completion token, because that is the prediction for the first token, and its accuracyAndrej Karpathy committed 2 years ago8d55c4a
  19. report both acc and acc_normAndrej Karpathy committed 2 years ago4e14b52
  20. add mmlu as well and refine both a bitAndrej Karpathy committed 2 years ago7bd2389
  21. move hellaswag file to devAndrej Karpathy committed 2 years agoaec9ce5
  22. fixed up dev/cudaErik Schultheis committed 2 years ago66ce576
  23. added a useful mixed precision utility for dev/cudaErik Schultheis committed 2 years agod7a81ef
  24. Merge pull request #383 from KarhouTam/feature/online-softmax-forward-without-cgsAndrej committed 2 years ago109f516
  25. Merge pull request #416 from azret/feature/cpu_devAndrej committed 2 years ago9e04dea
  26. deterministic kernelErik Schultheis committed 2 years ago858c6e6
  27. general cleanupErik Schultheis committed 2 years ago2ccdfb7
  28. Setting up dev/CPU area with the first matmul_forward.cAzret Botash committed 2 years ago160b3bd
  29. remove legacy commentAndrej Karpathy committed 2 years ago2d43e5b
  30. Merge pull request #408 from ngc92/layernorm-bw-dev-cudaAndrej committed 2 years ago222d59f
  31. update dev/cuda/layernorm_backward and improve `validate_result` to take into account fp epsilon when comparing resultsErik Schultheis committed 2 years agoe553e2f
  32. fix layernorm backward: accumulate weight gradientErik Schultheis committed 2 years agodd8c9f5
  33. fixup commentErik Schultheis committed 2 years agoc66e48c
  34. fix CI compile by disabling kernel 5Erik Schultheis committed 2 years ago65727d5
  35. fix non-atomic version: * accumulate instead of assign * need dedicated argument to correctly handle the floatX == float caseErik Schultheis committed 2 years ago49ee3c8