Using Containers to Accelerate HPC
Containers Tutorial Session @ Supercomputing Conference 2024 (SC24)
Date: Monday, 18 November 2024 8:30am - 5pm EST
Location: Altanta, GA, USA
Link: SC 2024 Tutorial Details
Keywords: Containerized HPC, System Software and Runtime Systems, Scientific Software Development, DevOps
Prerequisites
This is a hands-on tutorial. Participants should bring a laptop and load or pre-install a terminal and/or ssh client in advance to make best use of time during the tutorial.
See also the Setup page.
Questions
You can ask questions verbally or with this Google Doc. Please append your question below the others in the document.
We have also created a Slack Team for this. The invitation link is here.
Schedule
| Setup | ||
| 08:30 | 1. Introduction and update on Linux containers |
What is a container and how is it different from a virtual machine?
What are some of the benefits of containers and how can containers help me do my job better? |
| 08:50 | 2. Building and running Docker containers |
How can I use Docker to run a container?
How do I create an image from scratch? How can I share an image between systems? Is Docker the only tool for building containers? |
| 09:20 | 3. Container images best practices |
Are there any practices I should avoid?
Are there practices I should try to follow? How can I keep my images small? How can I speed up iterative updates and testing? |
| 09:30 | 4. HPC and containers | What are the limitations of industry-standard container approaches in an HPC environment? |
| 09:50 | 5. Installing a container engine |
What does the procedure to install a container engine looks like, and what are its main caveats?
What are the key aspects that can be configured by system administrators for container engine deployments? |
| 10:00 | 6. MORNING BREAK | |
| 10:30 | 7. Running HPC jobs with containers |
How can I execute commands in a container with Singularity or Shifter?
How are variables and directories shared between host and container? Is it possible to simplify the container user experience? |
| 11:50 | 8. Q & A session | |
| 12:00 | 9. LUNCH BREAK | |
| 13:30 | 10. Container services and Kubernetes |
How should services like databases or portals be deployed?
What is different about deploying services versus HPC applications? |
| 14:10 | 11. Containers with E4S |
How can I use the E4S full featured containers?
What packages can I load using Spack? How can I compile and test an MPI application? |
| 15:00 | 12. AFTERNOON BREAK | |
| 15:30 | 13. Advanced HPC use cases |
How can I run an MPI enabled application in a container using a bind approach?
How can I run a GPU enabled application in a container? How can I store files inside a container filesystem? How can I use workflow engines in conjunction with containers? |
| 16:20 | 14. Success stories and use cases | What are some science use cases that benefit from containers? |
| 16:40 | 15. Final Q & A and wrap-up | Any final questions |
| 17:00 | Finish |
The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.