High Performance Container Workshop

https://haampie.github.io/isc-2025

ISC 2025, Hamburg — Harmen Stoppels

This talk

  1. Quick Spack intro/refresher
  2. Share binaries with Spack users
  3. Share binaries with everyone else

Spack refresher

  • What is Spack?
  • Installing packages
  • Environments
  • Build from source vs. binaries

What is Spack?

  • Package manager for HPC & scientific computing
  • Builds packages from source
  • Handles complex dependency graphs
  • Multiple versions/variants of same package
  • Works on Linux, macOS, Windows

Installing packages


							# Basic package
							spack install python
							
							# Specific version
							spack install python@3.11
							
							# With variants
							spack install python@3.11+optimizations~debug
							
							# With dependencies
							spack install numpy ^python@3.11 ^openblas

Environments


							# spack.yaml
							spack:
							  specs:
							    - python@3.11
							    - numpy
							    - scipy
						

							$ spack env create myenv
							$ spack env activate myenv
							$ spack install

Source vs binaries

  • Default: Build everything from source
  • Binary caches: Pre-built packages
  • Speed: Hours → minutes
  • Reproducibility: Same binaries everywhere
  • Today's focus: OCI-based binary caches

Part 1: Binary Caches

Sharing Spack builds with other Spack users

Try out the build cache

  • spack install should use it

Part 2: Container Images

The same binaries, now for everyone

Who can use this?

  • Spack users can install from the cache
  • DevOps teams benefit from faster builds in CI/CD
  • Reproducible environments across teams
  • But what about users who don't know Spack?
  • The goal: Share your HPC software stack with anyone
  • Container images that "just work"
  • No Spack knowledge required

Turns out...

  • We can spack install from the cache
  • But we can also docker run them!
  • Same binaries, different interface
  • No rebuilding required!
  • Spack is a container builder

Making containers runnable

Turns your existing binaries into runnable containers:


						$ spack -e . buildcache push --base-image ubuntu:24.04
  • Any base image works (if glibc is compatible)
  • Gives you a basic shell
  • No Dockerfile needed!
  • No rebuilding needed!

Why is this special?

Traditional approach:


								# Dockerfile
								FROM ubuntu:24.04
								RUN apt-get update && ...
								RUN spack install python numpy

Build locally → Write Dockerfile → docker build → Build again

Spack approach:


								$ spack install python numpy    # Build locally
								$ spack buildcache push --base-image ubuntu:24.04 ...

Build once → Containerize existing binaries → Done!

Container composition made easy

  • Traditional containers: hard to combine
  • Try merging official CUDA + MKL + Python images
  • Spack: trivial due to single prefix per package
  • Just add more specs to your environment!

Multi-package container


							$ spack -e . add julia vim
							$ spack -e . install
							$ spack -e . buildcache push \
							    --tag julia-and-vim \
							    --base-image ubuntu:24.04 \
							    ghcr

The best of both worlds

For Spack users:

  • spack install from fast binary caches
  • Share builds across teams and CI/CD
  • Reproducible environments

For everyone else:

  • docker run the same binaries
  • No Spack installation required
  • Minimal, efficient containers
  • Easy composition of complex software stacks
  • Perfect for sharing HPC applications