>> Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers.
So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU.
Sure the GPU i not fast, but if memory is important.....
Most CPU RAM is much slower than GPU RAM. GPUs typically pack RAM 2 generations ahead with a wider bus than anything you'd find on a consumer motherboard.
For reference, DDR4-3200 in quad channel is ~100 GB/s while a 3090's VRAM is 960 GB/s. Of course, most consumers only have dual channel.
M1 Pro is 200 and M1 Max is 400. Which is slow for GPU memory, but incredible for main memory -- although I'm not sure how much of that a single core can actually pull.
You're thinking about the wrong bandwidth here. The article is talking about going from the GPU's RAM <-> GPU cores (i.e. through load/store instructions in a cuda kernel), not from CPU's RAM <-> GPU's RAM. That kind of bandwidth is still important but usually not the bottleneck on most ML workloads.
The author (correctly) made a distinction about the two like you said, but at the end when talking about Raspberry Pi 4 they use a number (~4GB/s of memory bandwidth) from an article [1] which I can only assume is NOT about graphics memory or its bandwidth (do Raspberry PIs even have it?).
And how exactly is the bandwidth counted if I use integrated GPU (like i5 13600K)? Or pure CPU?
CPUs and GPUs both interface with their own memory, and those memories have a certain bandwidth. Generally, CPU memory has relatively little bandwidth, but relatively good latency. (For example, an i9-13900K supports memory up to around 100 GB/s, while even my previous GPU, a midrange Radeon HD 7850 from 2012, has over 150 GB/s of bandwidth).
An integrated GPU shares memory with the CPU, so at best it gets the same amount of bandwidth assuming the CPU is not using any (which is rather unlikely).
A dedicated GPU has its own private high-bandwidth memory (an RTX 4090 has a memory bandwidth of over 1000 GB/s), but to get anything in there it needs to be loaded over the PCIe bus (which has a measly 32 GB/s bandwidth for PCIe 4.0 x16).
That said, CPU memory does have one big advantage: it tends to be much larger. You can pair up to 192 GB with a regular Ryzen 7000 CPU, while consumer GPUs don't go above 24 GB of memory (RTX 4090, RX 7900 XTX). (There are bigger GPUs out there, but those are generally intended for datacenters, and if you go that route, an Epyc or Xeon CPU can also support much more memory than a plain desktop Ryzen, although you can also slot multiple GPUs into a single server.)
I believe that for LLM performance, memory bandwidth is key, because all the neural network layers need to be streamed from memory, and very little work is done with it each time, although I guess batching operations could help if you're working at scale, since each weight would be applied multiple times then.
Raspberry Pi's do have a usable GPU, but using it for computation is not a particularly well-traveled path. I think that's a shame. The pre-4 models have a different Broadcom graphics core to the 4, and it looks like you can get useful work out of both, but they are different enough that it's a rebuild between the generations.
This is basically the appeal of the apple chips in this domain. Apple have fuck-you money so they have a bunch of high-bandwidth decent-latency soldered onto the chip.
So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU.
Sure the GPU i not fast, but if memory is important.....