Re: [HTCondor-users] Where are container images cached/stored?

[Steven C Timm <timm@xxxxxxxx>]

I answered part of this in a Slack discussion with Matthew earlier but will give a few more details.

We actually have a double container layer.. a Docker-based layer that is managed by the site and is used for scheduling and managing cgroups, and then the user jobs which run in singularity/apptainer.  The local-managed docker containers we store in our own Harbor docker registry, not on Dockerhub.  But in general whenever a container is launched in the Docker universe (now the container universe) it is cached on the local execute node so it only has to be pulled once.

The user-based singularity containers are distributed via CVMFS.

Steve

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Matthew T West via HTCondor-users <htcondor-users@xxxxxxxxxxx>
Sent: Sunday, November 19, 2023 5:06 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Cc: Matthew T West <m.t.west@xxxxxxxxxxxx>
Subject: [HTCondor-users] Where are container images cached/stored?

 

For those folks that support containers as run-time environments on their clusters, where are those images cached or stored?

1. When pulling from a remote repo like docker-hub in a htcondor submit file, is the image ultimately cached on the particular execute node where the job runs?
   
2. For those hosting/storing images locally, where do you usually put them?
1. Local storage on the Access Point
2. Shared (networked) central file system
3. The OSDF
4. Some bespoke local solution
   

Cheers,
Matt

-- 
Matthew T. West
DevOps & HPC SysAdmin
University of Exeter, Research IT
www.exeter.ac.uk/research/researchcomputing/support/researchit
57 Laver Building, North Park Road, Exeter, EX4 4QE, United Kingdom