Re: [HTCondor-users] MAX_CONCURRENT_DOWNLOADS not working?

[Angel de Vicente <angel.de.vicente@xxxxxx>]

Hello Tim,

Tim Theisen via HTCondor-users
<htcondor-users@xxxxxxxxxxx> writes:

> I have not had any time to try an reproduce this. We did discuss this at our
> team meeting. It was pointed out the MAX_CONCURRENT_DOWNLOADS only limits
> downloads handled by HTCondor itself. Anything that is downloaded by the osdf or
> curl plugins would not be limited by this knob. They mentioned that another knob
> would limit those downloads.
>
> Are you using plugins for downloads?

Not as far as I know. At least I haven't installed any plugins.

I have now upgraded the HTCondor version in all the clients to 24.0.10,
but I see more or less the same behaviour.

The submit script I have is simply:

,----
| Executable = /usr/bin/fallocate
| Arguments  = -l 10G test_$(ClusterId)_$(Process).dat
| Log        = download.$(ClusterId).log
| Output     = download.$(ClusterId).$(Process).out
| Error      = download.$(ClusterId).$(Process).err
| 
| Queue 20
`----

Now I don't see any jobs in the transfer queue, they immediately start
transferring.

But even more disconcerting is the issue that, as with version 23.0.21,
when removing the jobs via "condor_rm" the output file transfer
continues regardless, until the transfer is finished and then later
deleted, and the condor_q command shows that the transfer is
progressing, despite the jobs being removed:

,----
| $ condor_q -nobatch
| 
| -- Schedd: xxxxx.es : <xxxxx:9618?... @ 08/22/25 20:06:39
|  ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
| 1522.0   angelv          8/22 20:06   0+00:00:11  > 0    0.0 fallocate -l 10G test_1522_0.dat
| 1522.1   angelv          8/22 20:06   0+00:00:11  > 0    0.0 fallocate -l 10G test_1522_1.dat
| 1522.2   angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_2.dat
| 1522.3   angelv          8/22 20:06   0+00:00:11  > 0    0.0 fallocate -l 10G test_1522_3.dat
| 1522.4   angelv          8/22 20:06   0+00:00:10  > 0    0.0 fallocate -l 10G test_1522_4.dat
| 1522.5   angelv          8/22 20:06   0+00:00:11  > 0    0.0 fallocate -l 10G test_1522_5.dat
| 1522.6   angelv          8/22 20:06   0+00:00:10  > 0    0.0 fallocate -l 10G test_1522_6.dat
| 1522.7   angelv          8/22 20:06   0+00:00:10  > 0    0.0 fallocate -l 10G test_1522_7.dat
| 1522.8   angelv          8/22 20:06   0+00:00:10  > 0    0.0 fallocate -l 10G test_1522_8.dat
| 1522.9   angelv          8/22 20:06   0+00:00:10  > 0    0.0 fallocate -l 10G test_1522_9.dat
| 1522.10  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_10.dat
| 1522.11  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_11.dat
| 1522.12  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_12.dat
| 1522.13  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_13.dat
| 1522.14  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_14.dat
| 1522.15  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_15.dat
| 1522.16  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_16.dat
| 1522.17  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_17.dat
| 1522.18  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_18.dat
| 1522.19  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_19.dat
| 
| Total for query: 20 jobs; 0 completed, 20 removed, 0 idle, 0 running, 0 held, 0 suspended 
| Total for angelv: 20 jobs; 0 completed, 20 removed, 0 idle, 0 running, 0 held, 0 suspended 
| Total for all users: 230 jobs; 0 completed, 20 removed, 104 idle, 106 running, 0 held, 0 suspended
| 
`----

Cheers,
-- 
Ãngel de Vicente  
 Research Software Engineer (Supercomputing and BigData)
 Instituto de AstrofÃsica de Canarias (https://www.iac.es/en)