History

Commits on move/encoder_kernel to dev/data/tinystories.py3

the write_datafile function accepts python list, which i think is faster but i didn't checkAndrej Karpathy committed 2 years agod536088
adjust the dataloader to load the new .bin data files, and both prod datasets to use it insteadAndrej Karpathy committed 2 years ago1defbd4
refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scaleAndrej Karpathy committed 2 years ago722e5b2