How to Choose Near-Memory Computing Chips: Start from Zero with One Table

Spent the weekend messing with Micro-Nano Core's DRAM 3D near-memory computing, hit quite a few pits. I'm not from a chip background — I do design systems, and only started touching deep learning stuff two months ago. So the process below is for outsiders like me; I'll break down the jargon into plain language as much as possible.

First, the goal. What you actually need to deliver is just a table. Once it's filled in, you can take it to ask your hardware colleagues, instead of getting ground into the dirt by a pile of terms.

In the prep stage, first understand three words — no need to memorize definitions, just remember three sentences.

DRAM is the memory in your computer, the place for temporarily holding data, forgotten when power goes off. 3D near-memory computing breaks into two things. 3D means stacking originally flat chips layer by layer, like building a tower; near-memory computing means moving the compute units next to the storage, so data travels less. As for why you have to move it, it's because compute power grows faster than data movement, and if data can't be moved over, no matter how fast you compute, you're just waiting.

More and more conceptual confusion has also appeared in the industry — many people equate 3D stacking, HBM-PIM, and near-memory computing all with compute-in-memory.

I fell into this one at first too. 3D stacking is a packaging process, near-memory computing is putting the two close together, compute-in-memory is computing directly inside the storage cells. Three different layers of things — mix them up and you'll definitely crash later.

Next is that five-row table. Open any document tool and create a table: rows are four hard metrics — bandwidth, latency, power consumption, maturity — and leave two columns, the left called "solution claims," the right called "is it enough for my scenario." First row, bandwidth: the material says at equal capacity, bandwidth can reach 5 times HBM3E, and some architectures claim up to the 30TB/s level — don't get excited, just copy it into the left column. Second row, latency: this column is the easiest to leave blank; I went through several materials, bandwidth numbers all over the screen, latency almost nobody mentions — blank is blank, don't fill it in yourself. Third row, power consumption: near-memory computing saves the energy of the movement part, not the whole chip's energy — don't mix these two columns together. Fourth row, maturity: hybrid bonding, this process was used on image sensors around 2016, on flash memory in 2018, and those two have always been the main battlefield; moving it to high-compute chips hasn't been long.

By now you should have a feeling — this table can't be fully filled. Not filling it is normal; it's meant to expose information gaps.

Now three pits I personally fell into.

First, treating peak bandwidth as usable bandwidth. I initially copied the marketing number straight into the right column, then got stumped by a colleague's one question: under what access pattern was this bandwidth measured? How pseudo-channels are scheduled, how banks are mapped — all affect how much you actually get. Claims and real measurements have a gap in between.

Second, ignoring heat. The denser the stacking, the harder heat escapes; once heat rises, the refresh strategy has to change, and once changed, latency changes again. These three things are linked — don't look at them separately.

Third, comparing it directly with HBM. I saw a domestic 14nm+ chip, 520 TFLOPS, 6.4TB/s — nice-looking specs. But that's another company's route, with different process and supply chain. Copying numbers across solutions equals not doing an evaluation.

My final table had two and a half of four rows blank. But it wasn't wasted effort. Before, when I saw "bandwidth is several times HBM," I'd just note it down; now I first ask three questions: under what access pattern was it measured, what's the latency, where's the failure boundary. A few days ago I wrote a piece on world models, saying it's too early to talk about capability, and we should first draw the failure boundaries. Applied to storage, the logic is the same.

For the edge scenario, a paper achieved a density of 12.77GB/s/mm² — the direction works. Whether it works for my scenario is another matter.

You can swap this table for your own scenario's fields and refill it, like how big your model is, how fast it needs to run, whether you can accept slowdowns. Where you can't fill it in is where you need to go ask people.