Re: [HTCondor-users] MAX_CONCURRENT_DOWNLOADS not working?

[Tim Theisen <tim@xxxxxxxxxxx>]

Hello,

I have confirmed that it does wait for the output transfer to complete 
before removing the job. The code is designed to work in this way and 
changing it would require significant effort.

...Tim

On 8/22/25 14:33, Tim Theisen via HTCondor-users wrote:
> That does seem like a defect. I'll reproduce this locally and get back 
> to you.
>
> ...Tim
>
> On 8/22/25 14:18, Angel de Vicente via HTCondor-users wrote:
> > Hello Tim,
> >
> > Tim Theisen via HTCondor-users
> > <htcondor-users@xxxxxxxxxxx> writes:
> >
> > > I have not had any time to try an reproduce this. We did discuss 
> > > this at our
> > > team meeting. It was pointed out the MAX_CONCURRENT_DOWNLOADS only 
> > > limits
> > > downloads handled by HTCondor itself. Anything that is downloaded by 
> > > the osdf or
> > > curl plugins would not be limited by this knob. They mentioned that 
> > > another knob
> > > would limit those downloads.
> > >
> > > Are you using plugins for downloads?
> > Not as far as I know. At least I haven't installed any plugins.
> >
> > I have now upgraded the HTCondor version in all the clients to 24.0.10,
> > but I see more or less the same behaviour.
> >
> > The submit script I have is simply:
> >
> > ,----
> > | Executable = /usr/bin/fallocate
> > | Arguments = -l 10G test_$(ClusterId)_$(Process).dat
> > | LogÂÂÂÂÂÂÂ = download.$(ClusterId).log
> > | OutputÂÂÂÂ = download.$(ClusterId).$(Process).out
> > | ErrorÂÂÂÂÂ = download.$(ClusterId).$(Process).err
> > |
> > | Queue 20
> > `----
> >
> > Now I don't see any jobs in the transfer queue, they immediately start
> > transferring.
> >
> > But even more disconcerting is the issue that, as with version 23.0.21,
> > when removing the jobs via "condor_rm" the output file transfer
> > continues regardless, until the transfer is finished and then later
> > deleted, and the condor_q command shows that the transfer is
> > progressing, despite the jobs being removed:
> >
> > ,----
> > | $ condor_q -nobatch
> > |
> > | -- Schedd: xxxxx.es : <xxxxx:9618?... @ 08/22/25 20:06:39
> > |Â IDÂÂÂÂÂ OWNERÂÂÂÂÂÂÂÂÂÂÂ SUBMITTEDÂÂÂÂ RUN_TIME ST PRI SIZE CMD
> > | 1522.0ÂÂ angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:11Â > 0 0.0 fallocate 
> > -l 10G test_1522_0.dat
> > | 1522.1ÂÂ angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:11Â > 0 0.0 fallocate 
> > -l 10G test_1522_1.dat
> > | 1522.2ÂÂ angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:09Â > 0 0.0 fallocate 
> > -l 10G test_1522_2.dat
> > | 1522.3ÂÂ angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:11Â > 0 0.0 fallocate 
> > -l 10G test_1522_3.dat
> > | 1522.4ÂÂ angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:10Â > 0 0.0 fallocate 
> > -l 10G test_1522_4.dat
> > | 1522.5ÂÂ angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:11Â > 0 0.0 fallocate 
> > -l 10G test_1522_5.dat
> > | 1522.6ÂÂ angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:10Â > 0 0.0 fallocate 
> > -l 10G test_1522_6.dat
> > | 1522.7ÂÂ angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:10Â > 0 0.0 fallocate 
> > -l 10G test_1522_7.dat
> > | 1522.8ÂÂ angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:10Â > 0 0.0 fallocate 
> > -l 10G test_1522_8.dat
> > | 1522.9ÂÂ angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:10Â > 0 0.0 fallocate 
> > -l 10G test_1522_9.dat
> > | 1522.10Â angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:09Â > 0 0.0 fallocate 
> > -l 10G test_1522_10.dat
> > | 1522.11Â angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:09Â > 0 0.0 fallocate 
> > -l 10G test_1522_11.dat
> > | 1522.12Â angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:09Â > 0 0.0 fallocate 
> > -l 10G test_1522_12.dat
> > | 1522.13Â angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:09Â > 0 0.0 fallocate 
> > -l 10G test_1522_13.dat
> > | 1522.14Â angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:09Â > 0 0.0 fallocate 
> > -l 10G test_1522_14.dat
> > | 1522.15Â angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:09Â > 0 0.0 fallocate 
> > -l 10G test_1522_15.dat
> > | 1522.16Â angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:09Â > 0 0.0 fallocate 
> > -l 10G test_1522_16.dat
> > | 1522.17Â angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:09Â > 0 0.0 fallocate 
> > -l 10G test_1522_17.dat
> > | 1522.18Â angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:09Â > 0 0.0 fallocate 
> > -l 10G test_1522_18.dat
> > | 1522.19Â angelvÂÂÂÂÂÂÂÂÂ 8/22 20:06ÂÂ 0+00:00:09Â > 0 0.0 fallocate 
> > -l 10G test_1522_19.dat
> > |
> > | Total for query: 20 jobs; 0 completed, 20 removed, 0 idle, 0 
> > running, 0 held, 0 suspended
> > | Total for angelv: 20 jobs; 0 completed, 20 removed, 0 idle, 0 
> > running, 0 held, 0 suspended
> > | Total for all users: 230 jobs; 0 completed, 20 removed, 104 idle, 
> > 106 running, 0 held, 0 suspended
> > |
> > `----
> >
> > Cheers,
--
Tim Theisen (he, him, his)
Release Manager
Center for High Throughput Computing
Department of Computer Sciences
University of Wisconsin - Madison
4261 Computer Sciences and Statistics
1210 W Dayton St
Madison, WI 53706-1685
+1 608 265 5736