Code

History

Commits on feature/libdir to dev/data14
  1. add ability to export fineweb100B as wellAndrej Karpathy committed 2 years agod0c0143
  2. change default shard size for fineweb10BAndrej Karpathy committed 2 years ago08fe3ee
  3. update fine-web preprocessing: faster, and much less RAM consumptionErik Schultheis committed 2 years agob66eb66
  4. make hellaswag optional eval yayAndrej Karpathy committed 2 years agoda59861
  5. fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be readyAndrej Karpathy committed 2 years ago67239d9
  6. first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batchAndrej Karpathy committed 2 years ago051f3ca
  7. continued changes for sharded dataloaderAndrej Karpathy committed 2 years agoedb0df9
  8. extend dataloader to be shardedAndrej Karpathy committed 2 years ago3131028
  9. the write_datafile function accepts python list, which i think is faster but i didn't checkAndrej Karpathy committed 2 years agod536088
  10. adjust the dataloader to load the new .bin data files, and both prod datasets to use it insteadAndrej Karpathy committed 2 years ago1defbd4
  11. add fineweb, and add the first version of a new write_shard function that contains a header properlyAndrej Karpathy committed 2 years ago4bbd01f
  12. i also forgot to include the readme file for the new dev/data dirAndrej Karpathy committed 2 years ago7d11b79
  13. oops forgot to include data_common.pyAndrej Karpathy committed 2 years ago71774b3
  14. refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scaleAndrej Karpathy committed 2 years ago722e5b2