Code

History

Commits on refactor/code-to-zerocuh to dev/data19
  1. add edu fineweb sh download script, change paths to my HF repo, add hellaswag to starter packAndrej Karpathy committed 2 years ago48fee04
  2. small cosmetic fixes to fineweb data fileAndrej Karpathy committed 2 years agoea36d04
  3. Merge branch 'add_edu_fineweb_support' of https://github.com/eliebak/llm.c into eliebak-add_edu_fineweb_supportAndrej Karpathy committed 2 years agoff7a90b
  4. add shell script that downloads the .bin files of FineWeb100B directlyAndrej Karpathy committed 2 years ago2677d82
  5. add edu fineweb support, with 10B and 1000B version“eliebak” committed 2 years ago67379a2
  6. add ability to export fineweb100B as wellAndrej Karpathy committed 2 years agod0c0143
  7. change default shard size for fineweb10BAndrej Karpathy committed 2 years ago08fe3ee
  8. update fine-web preprocessing: faster, and much less RAM consumptionErik Schultheis committed 2 years agob66eb66
  9. make hellaswag optional eval yayAndrej Karpathy committed 2 years agoda59861
  10. fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be readyAndrej Karpathy committed 2 years ago67239d9
  11. first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batchAndrej Karpathy committed 2 years ago051f3ca
  12. continued changes for sharded dataloaderAndrej Karpathy committed 2 years agoedb0df9
  13. extend dataloader to be shardedAndrej Karpathy committed 2 years ago3131028
  14. the write_datafile function accepts python list, which i think is faster but i didn't checkAndrej Karpathy committed 2 years agod536088
  15. adjust the dataloader to load the new .bin data files, and both prod datasets to use it insteadAndrej Karpathy committed 2 years ago1defbd4
  16. add fineweb, and add the first version of a new write_shard function that contains a header properlyAndrej Karpathy committed 2 years ago4bbd01f
  17. i also forgot to include the readme file for the new dev/data dirAndrej Karpathy committed 2 years ago7d11b79
  18. oops forgot to include data_common.pyAndrej Karpathy committed 2 years ago71774b3
  19. refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scaleAndrej Karpathy committed 2 years ago722e5b2