Code

History

Commits on master to dev/data/data_common.py9
  1. attempt to fix PRAndrej Karpathy committed 2 years ago16635d4
  2. Refactor magic/version header dictAleksa Gordic committed 2 years agoe8739c8
  3. Add llama 3 tokenization and data loaderAleksa Gordic committed 2 years agod3151c4
  4. update fine-web preprocessing: faster, and much less RAM consumptionErik Schultheis committed 2 years agob66eb66
  5. first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batchAndrej Karpathy committed 2 years ago051f3ca
  6. the write_datafile function accepts python list, which i think is faster but i didn't checkAndrej Karpathy committed 2 years agod536088
  7. adjust the dataloader to load the new .bin data files, and both prod datasets to use it insteadAndrej Karpathy committed 2 years ago1defbd4
  8. add fineweb, and add the first version of a new write_shard function that contains a header properlyAndrej Karpathy committed 2 years ago4bbd01f
  9. oops forgot to include data_common.pyAndrej Karpathy committed 2 years ago71774b3