Code

History

Commits on master to dev/data29
  1. fix tokenizer omgAndrej Karpathy committed 2 years ago755458d
  2. attempt to fix PRAndrej Karpathy committed 2 years ago16635d4
  3. Refactor tokenize, delegate insteadAleksa Gordic committed 2 years ago68624cc
  4. Refactor magic/version header dictAleksa Gordic committed 2 years agoe8739c8
  5. Replace fire with argparseAleksa Gordic committed 2 years agoeb1b359
  6. Add fineweb llama 3 supportAleksa Gordic committed 2 years ago5076563
  7. Use fire module, update readmeAleksa Gordic committed 2 years agof8c1ff5
  8. Add tiny shakespeareAleksa Gordic committed 2 years ago15b76bf
  9. Update header descriptionAleksa Gordic committed 2 years ago71419cb
  10. Add llama 3 tokenization and data loaderAleksa Gordic committed 2 years agod3151c4
  11. add edu fineweb sh download script, change paths to my HF repo, add hellaswag to starter packAndrej Karpathy committed 2 years ago48fee04
  12. small cosmetic fixes to fineweb data fileAndrej Karpathy committed 2 years agoea36d04
  13. Merge branch 'add_edu_fineweb_support' of https://github.com/eliebak/llm.c into eliebak-add_edu_fineweb_supportAndrej Karpathy committed 2 years agoff7a90b
  14. add shell script that downloads the .bin files of FineWeb100B directlyAndrej Karpathy committed 2 years ago2677d82
  15. add edu fineweb support, with 10B and 1000B version“eliebak” committed 2 years ago67379a2
  16. add ability to export fineweb100B as wellAndrej Karpathy committed 2 years agod0c0143
  17. change default shard size for fineweb10BAndrej Karpathy committed 2 years ago08fe3ee
  18. update fine-web preprocessing: faster, and much less RAM consumptionErik Schultheis committed 2 years agob66eb66
  19. make hellaswag optional eval yayAndrej Karpathy committed 2 years agoda59861
  20. fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be readyAndrej Karpathy committed 2 years ago67239d9
  21. first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batchAndrej Karpathy committed 2 years ago051f3ca
  22. continued changes for sharded dataloaderAndrej Karpathy committed 2 years agoedb0df9
  23. extend dataloader to be shardedAndrej Karpathy committed 2 years ago3131028
  24. the write_datafile function accepts python list, which i think is faster but i didn't checkAndrej Karpathy committed 2 years agod536088
  25. adjust the dataloader to load the new .bin data files, and both prod datasets to use it insteadAndrej Karpathy committed 2 years ago1defbd4
  26. add fineweb, and add the first version of a new write_shard function that contains a header properlyAndrej Karpathy committed 2 years ago4bbd01f
  27. i also forgot to include the readme file for the new dev/data dirAndrej Karpathy committed 2 years ago7d11b79
  28. oops forgot to include data_common.pyAndrej Karpathy committed 2 years ago71774b3
  29. refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scaleAndrej Karpathy committed 2 years ago722e5b2