Code

History

Commits on feature/random_init35+
  1. fix small bug on eval loggingAndrej Karpathy committed 2 years ago1f91bfc
  2. llm.c matches pytorch init from scratch exactly nowAndrej Karpathy committed 2 years ago86682af
  3. use pytorch rand and fix dumb bug lolAndrej Karpathy committed 2 years ago70a9c75
  4. Merge branch 'feature/rand' of https://github.com/azret/llm.c into feature/random_initAndrej Karpathy committed 2 years agobc1ebc1
  5. allow the python script to also init from random and save those weights, so it's a good reference for our C implementationAndrej Karpathy committed 2 years agoe6a7d1d
  6. first draft of random init, crashes with some cuBLAS error, debuggingAndrej Karpathy committed 2 years agod096318
  7. fix bug free only if inittedAndrej Karpathy committed 2 years agod3cf9e2
  8. Merge pull request #447 from karpathy/feature/hellaswagcAndrej committed 2 years agoeda0c2f
  9. make hellaswag optional eval yayAndrej Karpathy committed 2 years agoda59861
  10. fixed a bunch of small issues and we now utilize the full batch dimension, this is very close to ready to merge, might even be readyAndrej Karpathy committed 2 years ago67239d9
  11. first draft, apparently this works. needs cleanups, and also we are not yet utilizing the full batch dimension. we actually have to load in multiple examples and fully utilize batchAndrej Karpathy committed 2 years ago051f3ca
  12. Merge pull request #444 from karpathy/feature/sharded_dataAndrej committed 2 years ago69f1221
  13. add a super small crappy glob for windows that only matches a single unique file. this will make CI happy but we can't train on sharded data on windows until this is improvedAndrej Karpathy committed 2 years ago099d30f
  14. readme changesAndrej Karpathy committed 2 years ago05be4f6
  15. continued changes for sharded dataloaderAndrej Karpathy committed 2 years agoedb0df9
  16. extend dataloader to be shardedAndrej Karpathy committed 2 years ago3131028
  17. fix print format warning for size_t vs intAndrej Karpathy committed 2 years ago967420d
  18. torch tensor can't handle uint16 so let's convert to int32, which is silly because we'll convert to .long right after but okAndrej Karpathy committed 2 years ago587506d
  19. Merge pull request #440 from karpathy/feature/organize_dataAndrej committed 2 years ago170c92b
  20. docs on master-breaking changes around dataset file representationAndrej Karpathy committed 2 years ago54ccbd3
  21. adjust py file as well and make the errors betterAndrej Karpathy committed 2 years ago7d58fd2
  22. docs on master-breaking change around how we store data .bin filesAndrej Karpathy committed 2 years agof7cb77f
  23. make comment more helpfulAndrej Karpathy committed 2 years agoccc240c
  24. the write_datafile function accepts python list, which i think is faster but i didn't checkAndrej Karpathy committed 2 years agod536088
  25. also fix the python file. that should be it now, w.r.t. the new token format .bin filesAndrej Karpathy committed 2 years ago9bad49a
  26. also fix the .c file and fp32 fileAndrej Karpathy committed 2 years ago666145e
  27. adjust the dataloader to load the new .bin data files, and both prod datasets to use it insteadAndrej Karpathy committed 2 years ago1defbd4
  28. step 2 of dataloader refactor: separate out the dataloader to its own file, change its signature a little bit, and (notably) change from cudaMallocHost to a simple malloc, so that this file doesn't have to be cuda awareAndrej Karpathy committed 2 years agof7cbb30
  29. add fineweb, and add the first version of a new write_shard function that contains a header properlyAndrej Karpathy committed 2 years ago4bbd01f
  30. fix a slip in gitignore, i think i am getting tired todayAndrej Karpathy committed 2 years agoc51cd70
  31. adjust gitignoreAndrej Karpathy committed 2 years agobced34d
  32. i also forgot to include the readme file for the new dev/data dirAndrej Karpathy committed 2 years ago7d11b79
  33. oops forgot to include data_common.pyAndrej Karpathy committed 2 years ago71774b3
  34. more changes, trying to help people out because when this merges to master it will brick everyone's code...Andrej Karpathy committed 2 years agof671cf9
  35. refactor how we treat datasets, because we're about to have more of them and we don't want them to clutter up root dir etc. this is only step 1, i'm about to refactor a bunch of the dataloading, how the .bin files work and are loaded, how the DataLoader works, etc. This is all needed to support good evals and training at scaleAndrej Karpathy committed 2 years ago722e5b2