Reminds me of a new PM believing that "arabic numerals" mean arabic script instead of real every day numbers because AI told them so. Despite the entire team trying to explain it to them.
Some people would just rather believe a computer than the evidence right in front of them.
??? The only configuration that will allow disk reads at 10 GB/s is if you're using PCIe 5.0. PCIe 4.0 or lower, and SATA will not drop out long before that.
He's very clearly hauling data straight from the page cache.
The problem is dram bandwidth to the cpu. Each token costs roughly 20gb of traffic and ddr5 is roughly 50-80gb/s, plus you still have to run the compute sequentially. That’s your limit.
We have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy -- it cannot be compressed further. On all tested large models, it's a ratio of 1.34-1.23 -- and smaller models up to 3.76x. It also increases the effective bandwidth by the same rate.
Interesting. Why is that? I would have expected the opposite, since larger models have to try less hard to fit the training data. Or maybe this leaves more parameters with random initialization, resulting in higher entropy for larger models?
I honestly don't know... I didn't train the models, so I can't tell why the math works out that way. It just does. I suspect it has to do with the fact that all the small models I've tested have been quantized. I don't know of any small model trained from scratch. If you know of any, I'd be happy to encode it and see what it looks like.
reply