Lecture 6: Where Should My Data Live? Taming Heterogeneous Memory in HPC

Author: Dr. Antonio J. Peña, Barcelona Supercomputing Center (BSC), Accelerators and Communications for HPC.

Heterogeneous computing usually means CPUs, GPUs and accelerators, but memory has quietly gone heterogeneous too. A single node can now mix DRAM with High Bandwidth Memory (Sapphire Rapids Xeon Max, GPUs), and until recently with persistent memory like Intel Optane. CXL is bringing memory expansion and disaggregated pools into racks.

In my talk at the BSC course "It's the Memory, Stupid!" I asked a simple question: with several kinds of memory, where should each piece of data live?

Deep memory is not heterogeneous memory

HBM brings bandwidth but limited capacity and slightly higher latency than DRAM; Optane and CXL DIMMs bring capacity at the cost of latency.

Software sees this in two ways. In deep memory (Intel's "memory mode"), the fast tier is a hardware cache for the slow one: applications run unmodified, but you only see the bigger tier's capacity, and misses cost two accesses. In heterogeneous memory (flat mode), every tier is addressable and software decides where allocations go.

On Linux, flat-mode tiers show up as NUMA nodes: bind a whole app with numactl, enable kernel tiering (Linux 5.5+), or allocate per object with Intel's Unified Memory Framework (UMF). OpenMP (#pragma omp allocate) and MPI (mpi_memory_alloc_kinds) offer standard hooks too.

The hard part isn't the API, it's the decision

The mechanisms exist; guidance doesn't. numactl is first-come-first-served for the whole process, kernel page migration is reactive and page-granular, and per-allocation APIs leave the decision to developers who rarely know which allocations matter.

With DRAM among a node's priciest parts, placement must be automated, or at least guided, if heterogeneity is to cut cost rather than add complexity.

ecoHMEM: profile once, place smartly

Enter ecoHMEM, our BSC framework (with FlexMalloc contributed by Intel):

1. Profile with Extrae, BSC's sampling-based tracer, attributing last-level cache misses to memory objects rather than code lines. (Full instrumentation with our EVOP tool gives similar placements at far higher overhead.)
2. Decide with hmem_advisor, which treats placement as a multiple knapsack problem: tiers are knapsacks, objects are items weighted by size and valued by stall cycles, rescaled by each tier's latency. A bandwidth-aware variant covers bandwidth-bound codes.
3. Run with FlexMalloc, which intercepts malloc/free and redirects allocations, matched by call stack, to their tier.

No source changes or recompilation, just one profiling run.

The results (Jordà et al., IEEE Cluster 2022; Jordà and Peña, SBAC-PAD 2026): on DRAM + Optane, gains of 0–2x over memory mode and kernel page migration with no meaningful slowdowns, and a quarter of the DRAM costing little or nothing. The bandwidth-aware algorithm turned a 35% OpenFOAM overhead into a 6% speedup, and on Sapphire Rapids, HBM placement sped up LULESH by 12.4% and miniFE by 11.3%.

When deep memory shines: encrypted AI

Flat mode isn't always the answer. Homomorphic encryption lets a provider run inference on encrypted data with an encrypted model, protecting both sides' IP, but inflates memory and compute by over 100x. DRAM capacity had capped HE inference at MobileNetV2 with expansion factor 0.35.

With Optane in memory mode, we ran full-size MobileNetV2 (about 3x larger) and ResNet-50, the first non-toy DNN under HE (Lloret-Talavera et al., IEEE Transactions on Computers, 2022). With good locality and a need for capacity, a hardware-cached tier just works.

Takeaways

Heterogeneity is here to stay, in memory as much as in processors. Production APIs give us the how; tools like ecoHMEM help with the where.

ecoHMEM is open source on GitHub, with FlexMalloc from Intel, and the AccelCom group at BSC welcomes theses, internships and collaborations.

Psssttt... don't miss the 7th Workshop on Heterogeneity and Memory Systems (HMEM) at SC26!