Re: [HTCondor-users] MAX_CONCURRENT_DOWNLOADS not working?

[Tim Theisen <tim@xxxxxxxxxxx>]

That does seem like a defect. I'll reproduce this locally and get back 
to you.

...Tim

On 8/22/25 14:18, Angel de Vicente via HTCondor-users wrote:
> Hello Tim,
>
> Tim Theisen via HTCondor-users
> <htcondor-users@xxxxxxxxxxx> writes:
>
> > I have not had any time to try an reproduce this. We did discuss this at our
> > team meeting. It was pointed out the MAX_CONCURRENT_DOWNLOADS only limits
> > downloads handled by HTCondor itself. Anything that is downloaded by the osdf or
> > curl plugins would not be limited by this knob. They mentioned that another knob
> > would limit those downloads.
> >
> > Are you using plugins for downloads?
> Not as far as I know. At least I haven't installed any plugins.
>
> I have now upgraded the HTCondor version in all the clients to 24.0.10,
> but I see more or less the same behaviour.
>
> The submit script I have is simply:
>
> ,----
> | Executable = /usr/bin/fallocate
> | Arguments  = -l 10G test_$(ClusterId)_$(Process).dat
> | Log        = download.$(ClusterId).log
> | Output     = download.$(ClusterId).$(Process).out
> | Error      = download.$(ClusterId).$(Process).err
> |
> | Queue 20
> `----
>
> Now I don't see any jobs in the transfer queue, they immediately start
> transferring.
>
> But even more disconcerting is the issue that, as with version 23.0.21,
> when removing the jobs via "condor_rm" the output file transfer
> continues regardless, until the transfer is finished and then later
> deleted, and the condor_q command shows that the transfer is
> progressing, despite the jobs being removed:
>
> ,----
> | $ condor_q -nobatch
> |
> | -- Schedd: xxxxx.es : <xxxxx:9618?... @ 08/22/25 20:06:39
> |  ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
> | 1522.0   angelv          8/22 20:06   0+00:00:11  > 0    0.0 fallocate -l 10G test_1522_0.dat
> | 1522.1   angelv          8/22 20:06   0+00:00:11  > 0    0.0 fallocate -l 10G test_1522_1.dat
> | 1522.2   angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_2.dat
> | 1522.3   angelv          8/22 20:06   0+00:00:11  > 0    0.0 fallocate -l 10G test_1522_3.dat
> | 1522.4   angelv          8/22 20:06   0+00:00:10  > 0    0.0 fallocate -l 10G test_1522_4.dat
> | 1522.5   angelv          8/22 20:06   0+00:00:11  > 0    0.0 fallocate -l 10G test_1522_5.dat
> | 1522.6   angelv          8/22 20:06   0+00:00:10  > 0    0.0 fallocate -l 10G test_1522_6.dat
> | 1522.7   angelv          8/22 20:06   0+00:00:10  > 0    0.0 fallocate -l 10G test_1522_7.dat
> | 1522.8   angelv          8/22 20:06   0+00:00:10  > 0    0.0 fallocate -l 10G test_1522_8.dat
> | 1522.9   angelv          8/22 20:06   0+00:00:10  > 0    0.0 fallocate -l 10G test_1522_9.dat
> | 1522.10  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_10.dat
> | 1522.11  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_11.dat
> | 1522.12  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_12.dat
> | 1522.13  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_13.dat
> | 1522.14  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_14.dat
> | 1522.15  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_15.dat
> | 1522.16  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_16.dat
> | 1522.17  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_17.dat
> | 1522.18  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_18.dat
> | 1522.19  angelv          8/22 20:06   0+00:00:09  > 0    0.0 fallocate -l 10G test_1522_19.dat
> |
> | Total for query: 20 jobs; 0 completed, 20 removed, 0 idle, 0 running, 0 held, 0 suspended
> | Total for angelv: 20 jobs; 0 completed, 20 removed, 0 idle, 0 running, 0 held, 0 suspended
> | Total for all users: 230 jobs; 0 completed, 20 removed, 104 idle, 106 running, 0 held, 0 suspended
> |
> `----
>
> Cheers,

--
Tim Theisen (he, him, his)
Release Manager
Center for High Throughput Computing
Department of Computer Sciences
University of Wisconsin - Madison
4261 Computer Sciences and Statistics
1210 W Dayton St
Madison, WI 53706-1685
+1 608 265 5736