Code

History

Commits on move/encoder_kernel to dev/data/fineweb.py7
  1. add ability to export fineweb100B as wellAndrej Karpathy committed 2 years agod0c0143
  2. change default shard size for fineweb10BAndrej Karpathy committed 2 years ago08fe3ee
  3. update fine-web preprocessing: faster, and much less RAM consumptionErik Schultheis committed 2 years agob66eb66
  4. continued changes for sharded dataloaderAndrej Karpathy committed 2 years agoedb0df9
  5. extend dataloader to be shardedAndrej Karpathy committed 2 years ago3131028
  6. adjust the dataloader to load the new .bin data files, and both prod datasets to use it insteadAndrej Karpathy committed 2 years ago1defbd4
  7. add fineweb, and add the first version of a new write_shard function that contains a header properlyAndrej Karpathy committed 2 years ago4bbd01f