|
Thanks very much for this info, Jamie. This was super helpful.
Best,
Matthew
--
Matthew Feickert
University of Wisconsin-Madison
From: Jaime Frey <jfrey@xxxxxxxxxxx>
Sent: Tuesday, June 17, 2025 15:15
To: Matthew Feickert <matthew.feickert@xxxxxxxx>
Cc: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] condor_submit pull policy for container universe vs. docker universe
Yes, use âdocker_pull_policyâ. Itâll be used on EPs that use Docker and ignored on EPs that use Singularity/Apptainer.
HTCondor does not provide its own cache management for container images. It can transfer a SIF file or exploded SIF directory from the AP on request.
- Jaime
On Jun 16, 2025, at 4:05âPM, Matthew Feickert <matthew.feickert@xxxxxxxx> wrote:
Thanks, Jamie. This is super helpful.
> The docker_pull_policy submit command influences how a Docker container image is retrieved on the EP. It is consulted both when the job universe is âdocker' and when the job universe is âsingularity' using a Docker repository path for the container_image.
So, if I understand this correctly, I should switch this back to using the "docker_pull_policy" with the container universe in my HTCondor submit description files and I should still be able to have this work, correct?
> Docker is the only supported containerization system under the container universe that includes an image cache, so we didnât add a separate 'container_pull_policyâ submit command.
This last sentence surprised me. Does that mean that if I'm using a .sif Apptainer container
```
```
that container image isn't cached at all, and so is pulled from GitHub Container Registry every time?
Best,
Matthew
--
Matthew Feickert
University of Wisconsin-Madison
From: Jaime Frey <jfrey@xxxxxxxxxxx>
Sent: Monday, June 16, 2025 15:19
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Cc: Matthew Feickert <matthew.feickert@xxxxxxxx>
Subject: Re: [HTCondor-users] condor_submit pull policy for container universe vs. docker universe
The docker_pull_policy submit command influences how a Docker container image is retrieved on the EP. It is consulted both when the job universe is âdocker' and when the job universe is âsingularity' using a Docker repository path for the container_image.
Docker is the only supported containerization system under the container universe that includes an image cache, so we didnât add a separate 'container_pull_policyâ submit command.
- Jaime
On Jun 12, 2025, at 4:47âPM, Matthew Feickert via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:
Hi All,
```
WARNING: the line 'container_pull_policy = missing' was unused by condor_submit. Is it a typo
```
> `docker_pull_policy = < always >`:
> if `docker_pull_policy` is set to _always_, when a docker universe job starts on a execution point, the option "âpull always" will be passed to the docker run command. This only impacts EPs which already have a locally cached version of the image. With this
option, docker will always check with the repo to see if the cached version is out of date. This requires more network connectivity, and may cause docker hub to throttle future pull requests. It is generally recommended to never mutate docker image tag name,
and avoid needing this option.
but then for the container universe set of commands there is no `container_pull_policy`. There is
> `transfer_container = < True | False >`
> A boolean value that defaults to True. When `false`, sif container images and expanded directories are assumed to be pre-staged on the EP, and HTCondor will not attempt to transfer them.
but that sounds like it might be different(?).
Do you know if this means that there is no way to overwrite the container cache if the container universe is used?
Best,
Matthew
--
Matthew Feickert
University of Wisconsin-Madison
_______________________________________________
HTCondor-users
mailing list
To
unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with
a
subject:
Unsubscribe
Join
us in June at Throughput Computing 25: https://urldefense.com/v3/__https://osg-htc.org/htc25__;!!Mak6IKo!Po7dl7Yc8bx5BSDLs5aGKxVM0IBYKc1CjNnSNgx8jE-NcS2F44Kxl9zmwXDPSOc5sSat7soo583XjpOwwlyw3lLyylIFTA$
The
archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/
|
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}
