Mailing List Archives Authenticated access	UW Madison Computer Sciences Department Computer Systems Lab

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs

Date: Fri, 9 Aug 2024 15:27:38 +0000
From: Marco Mambelli <marcom@xxxxxxxx>
Subject: Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs

I think I found the problem
maxMemory is 1 and somehow it is too small for the CE to be considered:

maxMemory = ((RequestMemory ?: 1) > 100 ? RequestMemory : 100)
MemoryProvisioned = 128
RequestMemory = ifthenelse(MemoryUsage =!= undefined,MemoryUsage,(ImageSize + 1023) / 1024)
Requirements = (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory)

[root@ce-workspace /]# condor_ce_q -af maxMemory RequestMemory ImageSize MemoryUsage
1 1 100 undefined
1 1 100 undefined
1 1 100 undefined

Replacing maxMemory with:
maxMemory = ((RequestMemory ?: 1) > 100 ? RequestMemory : 100)

Gets maxMemory evaluated to 100 and allows jobs to run

Marco



> On Aug 9, 2024, at 10:11âAM, Jaime Frey <jfrey@xxxxxxxxxxx> wrote:
> 
> [EXTERNAL] â This message is from an external sender
> 
> You are looking at the unrouted jobs in the CE queue. You need to look at the routed jobs in the site queue (i.e. run condor_q).
> 
> - Jaime
> 
>> On Aug 8, 2024, at 11:49âPM, Marco Mambelli <marcom@xxxxxxxx> wrote:
>> 
>> Initially I has a problem because I was omitting the plus (maxMemory instead of +maxMemory)
>> But also after setting +maxMemory=RequestMemory jobs are still not matching
>> 
>> [root@ce-workspace /]# condor_ce_q
>> 
>> 
>> -- Schedd: ce-workspace.glideinwms.org : <10.89.0.2:41921?... @ 08/09/24 04:22:29
>> OWNER    BATCH_NAME    SUBMITTED   DONE   RUN    IDLE  TOTAL JOB_IDS
>> fermilab ID: 1        8/9  03:55      _      _      1      1 1.0
>> fermilab ID: 2        8/9  03:57      _      _      1      1 2.0
>> fermilab ID: 3        8/9  04:05      _      _      1      1 3.0
>> 
>> Total for query: 3 jobs; 0 completed, 0 removed, 3 idle, 0 running, 0 held, 0 suspended
>> Total for all users: 3 jobs; 0 completed, 0 removed, 3 idle, 0 running, 0 held, 0 suspended
>> 
>> [root@ce-workspace /]# condor_ce_q -l | grep -i mem
>> maxMemory = RequestMemory
>> RequestMemory = ifthenelse(MemoryUsage =!= undefined,MemoryUsage,(ImageSize + 1023) / 1024)
>> Requirements = (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory)
>> maxMemory = RequestMemory
>> RequestMemory = ifthenelse(MemoryUsage =!= undefined,MemoryUsage,(ImageSize + 1023) / 1024)
>> Requirements = (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory)
>> maxMemory = RequestMemory
>> RequestMemory = ifthenelse(MemoryUsage =!= undefined,MemoryUsage,(ImageSize + 1023) / 1024)
>> Requirements = (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory)
>> [root@ce-workspace /]# condor_ce_q -l | grep -i size
>> ExecutableSize = 100
>> ExecutableSize_RAW = 87
>> ImageSize = 100
>> ImageSize_RAW = 87
>> RequestMemory = ifthenelse(MemoryUsage =!= undefined,MemoryUsage,(ImageSize + 1023) / 1024)
>> TransferInputSizeMB = 0
>> ExecutableSize = 100
>> ExecutableSize_RAW = 87
>> ImageSize = 100
>> ImageSize_RAW = 87
>> RequestMemory = ifthenelse(MemoryUsage =!= undefined,MemoryUsage,(ImageSize + 1023) / 1024)
>> TransferInputSizeMB = 0
>> ExecutableSize = 100
>> ExecutableSize_RAW = 87
>> ImageSize = 100
>> ImageSize_RAW = 87
>> RequestMemory = ifthenelse(MemoryUsage =!= undefined,MemoryUsage,(ImageSize + 1023) / 1024)
>> TransferInputSizeMB = 0
>> [root@ce-workspace /]# condor_ce_q
>> 
>> 
>> -- Schedd: ce-workspace.glideinwms.org : <10.89.0.2:41921?... @ 08/09/24 04:23:53
>> OWNER    BATCH_NAME    SUBMITTED   DONE   RUN    IDLE  TOTAL JOB_IDS
>> fermilab ID: 1        8/9  03:55      _      _      1      1 1.0
>> fermilab ID: 2        8/9  03:57      _      _      1      1 2.0
>> fermilab ID: 3        8/9  04:05      _      _      1      1 3.0
>> 
>> Total for query: 3 jobs; 0 completed, 0 removed, 3 idle, 0 running, 0 held, 0 suspended
>> Total for all users: 3 jobs; 0 completed, 0 removed, 3 idle, 0 running, 0 held, 0 suspended
>> 
>> [root@ce-workspace /]# condor_ce_q -better 1.0
>> 
>> 
>> -- Schedd: ce-workspace.glideinwms.org : <10.89.0.2:41921?...
>> The Requirements expression for job 1.000 is
>> 
>>   (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory)
>> 
>> Job 1.000 defines the following attributes:
>> 
>>   DiskUsage = 100
>>   ImageSize = 100
>>   RequestDisk = DiskUsage (kb)
>>   RequestMemory = ifthenelse(MemoryUsage =!= undefined,MemoryUsage,(ImageSize + 1023) / 1024) (mb)
>> 
>> The Requirements expression for job 1.000 reduces to these conditions:
>> 
>>        Slots
>> Step    Matched  Condition
>> -----  --------  ---------
>> [0]           0  TARGET.Arch == "X86_64"
>> [1]           0  TARGET.OpSys == "LINUX"
>> [3]           0  TARGET.Disk >= RequestDisk
>> [5]           0  TARGET.Memory >= RequestMemory
>> 
>> 
>> [root@ce-workspace /]# condor_status
>> Name                              OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime
>> 
>> slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxx LINUX      X86_64 Unclaimed Idle      0.000 1763  0+00:29:22
>> 
>>              Total Owner Claimed Unclaimed Matched Preempting  Drain Backfill BkIdle
>> 
>> X86_64/LINUX     1     0       0         1       0          0      0        0      0
>> 
>>        Total     1     0       0         1       0          0      0        0      0
>> [root@ce-workspace /]# condor_status -l grep Memory
>> condor_status: unknown host grep
>> [root@ce-workspace /]# condor_status -l | grep Memory
>> ChildMemory = {  }
>> DetectedMemory = 1763
>> MachineResources = "Cpus Memory Disk Swap GPUs"
>> Memory = 1763
>> TotalMemory = 1763
>> TotalSlotMemory = 1763
>> TotalVirtualMemory = 6690348
>> VirtualMemory = 0
>> WithinResourceLimits = (MY.Cpus > 0 && TARGET.RequestCpus <= MY.Cpus && MY.Memory > 0 && TARGET.RequestMemory <= MY.Memory && MY.Disk > 0 && TARGET.RequestDisk <= MY.Disk && (TARGET.RequestGPUs =?= undefined || MY.GPUs >= TARGET.RequestGPUs))
>> [root@ce-workspace /]# condor_status -l | grep Arch
>> Arch = "X86_64"
>> [root@ce-workspace /]# condor_status -l | grep OpSys
>> OpSys = "LINUX"
>> OpSysAndVer = "AlmaLinux9"
>> OpSysLegacy = "LINUX"
>> OpSysLongName = "AlmaLinux release 9.4 (Seafoam Ocelot)"
>> OpSysMajorVer = 9
>> OpSysName = "AlmaLinux"
>> OpSysShortName = "AlmaLinux"
>> OpSysVer = 904
>> [root@ce-workspace /]# condor_status -l | grep Disk
>> ChildDisk = {  }
>> Disk = 19881824
>> MachineResources = "Cpus Memory Disk Swap GPUs"
>> TotalDisk = 19881824
>> TotalSlotDisk = 19881824.0
>> WithinResourceLimits = (MY.Cpus > 0 && TARGET.RequestCpus <= MY.Cpus && MY.Memory > 0 && TARGET.RequestMemory <= MY.Memory && MY.Disk > 0 && TARGET.RequestDisk <= MY.Disk && (TARGET.RequestGPUs =?= undefined || MY.GPUs >= TARGET.RequestGPUs))
>> [root@ce-workspace /]# condor_ce_q -l | grep -i disk
>> DiskUsage = 100
>> DiskUsage_RAW = 89
>> RequestDisk = DiskUsage
>> Requirements = (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory)
>> DiskUsage = 100
>> DiskUsage_RAW = 89
>> RequestDisk = DiskUsage
>> Requirements = (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory)
>> DiskUsage = 100
>> DiskUsage_RAW = 89
>> RequestDisk = DiskUsage
>> Requirements = (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory)
>> 
>>> On Aug 8, 2024, at 1:37âPM, Jaime Frey <jfrey@xxxxxxxxxxx> wrote:
>>> 
>>> [EXTERNAL] â This message is from an external sender
>>> 
>>> The CE will look at maxMemory in the incoming job ad and default_maxMemory of the route in order to set RequestMemory of the routed job. If neither is set, then the value 2000 is used. Does that match your expectation? Note that the RequestMemory attribute of the incoming job is ignored (though if maxMemory references it, that is respected).
>>> 
>>> If you send me the incoming and routed job ads and the route configuration, I can take a look to see whatâs happening.
>>> 
>>> - Jaime
>>> 
>>>> On Aug 7, 2024, at 7:14âPM, Marco Mambelli via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:
>>>> 
>>>> I think the GPU is a red herring,
>>>> The problem is the HTCondor-CE bug/feature where also a sleep job asking basically no memory is bumped by the CE to request 2 GB of memory.
>>>> I was setting maxMemory in the glidein ( <submit_attr name="maxMemory" value="RequestMemoryâ/>) but something is going wrong
>>>> 
>>>>> On Aug 7, 2024, at 6:05âPM, Marco Mambelli via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:
>>>>> 
>>>>> [EXTERNAL] â This message is from an external sender
>>>>> 
>>>>> Greetings,
>>>>> HTCondor-CE  23.0.8 (htcondor 23.8.1) seems to add a GPU requirement that causes all jobs to stay idle.
>>>>> There is no GPU on the host and no GPU is mentioned in the job classed on the submit host.
>>>>> On the HTCondor-CE there seems to be a GPU requirement that is not matched and causes the jobs to stay idle.
>>>>> 
>>>>> Follow below the outputs of:
>>>>> - condor_q  -better -reverse
>>>>> - condor_q  -better
>>>>> - condor_q -l | grep -i gpu
>>>>> 
>>>>> All jobs submitted stay idle
>>>>> Any suggestion on how to troubleshoot this?
>>>>> Any recent change involving GPU requirements?
>>>>> 
>>>>> Thank you,
>>>>> Marco
>>>>> 
>>>>> 
>>>>> Command outputs:
>>>>> 
>>>>> 
>>>>> [root@ce-workspace /]# condor_q -all -better -reverse -machine slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxx 4
>>>>> 
>>>>> 
>>>>> -- Schedd: ce-workspace.glideinwms.org : <10.89.0.35:46367?...
>>>>> 
>>>>> -- Slot: slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxx : Analyzing matches for 1 Jobs in 1 autoclusters
>>>>> 
>>>>> The Requirements expression for this slot is
>>>>> 
>>>>> START &&
>>>>> (WithinResourceLimits)
>>>>> 
>>>>> START is
>>>>> true
>>>>> 
>>>>> WithinResourceLimits is
>>>>> (MY.Cpus > 0 &&
>>>>>   TARGET.RequestCpus <= MY.Cpus && MY.Memory > 0 &&
>>>>>   TARGET.RequestMemory <= MY.Memory && MY.Disk > 0 &&
>>>>>   TARGET.RequestDisk <= MY.Disk && (TARGET.RequestGPUs is undefined ||
>>>>>     MY.GPUs >= TARGET.RequestGPUs))
>>>>> 
>>>>> This slot defines the following attributes:
>>>>> 
>>>>> Cpus = 1
>>>>> Disk = 17978392
>>>>> GPUs = 0
>>>>> Memory = 1763
>>>>> 
>>>>> Job 4.0 has the following attributes:
>>>>> 
>>>>> TARGET.RequestCpus = 1
>>>>> TARGET.RequestDisk = 100
>>>>> TARGET.RequestGPUs = undefined
>>>>> TARGET.RequestMemory = 2000
>>>>> 
>>>>> The Requirements expression for this slot reduces to these conditions:
>>>>> 
>>>>>    Clusters
>>>>> Step    Matched  Condition
>>>>> -----  --------  ---------
>>>>> [6]           0  TARGET.RequestMemory <= MY.Memory
>>>>> [12]          1  TARGET.RequestGPUs is undefined
>>>>> 
>>>>> slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxx: Run analysis summary of 1 jobs.
>>>>> 0 (0.00 %) match both slot and job requirements.
>>>>> 0 match the requirements of this slot.
>>>>> 1 have job requirements that match this slot.
>>>>> [root@ce-workspace /]# condor_q -all -better 4
>>>>> 
>>>>> 
>>>>> -- Schedd: ce-workspace.glideinwms.org : <10.89.0.35:46367?...
>>>>> The Requirements expression for job 4.000 is
>>>>> 
>>>>> (RequestGpus ?: 0) >= (TARGET.Gpus ?: 0)
>>>>> 
>>>>> Job 4.000 defines the following attributes:
>>>>> 
>>>>> GlideinCpusIsGood =  !isUndefined(MATCH_EXP_JOB_GLIDEIN_Cpus) && (int(MATCH_EXP_JOB_GLIDEIN_Cpus) =!= error)
>>>>> JobGPUs = JobIsRunning ? int(MATCH_EXP_JOB_GLIDEIN_GPUs) : OriginalGPUs
>>>>> JobIsRunning = (JobStatus =!= 1) && (JobStatus =!= 5) && GlideinCpusIsGood
>>>>> JobStatus = 1
>>>>> OriginalGPUs = undefined
>>>>> RequestGpus = ifThenElse((WantWholeNode =?= true && OriginalGPUs =!= undefined),( !isUndefined(TotalGPUs) && TotalGPUs > 0) ? TotalGPUs : JobGPUs,OriginalGPUs)
>>>>> 
>>>>> slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxx has the following attributes:
>>>>> 
>>>>> TARGET.Gpus = 0
>>>>> TARGET.TotalGPUs = 0
>>>>> 
>>>>> The Requirements expression for job 4.000 reduces to these conditions:
>>>>> 
>>>>>      Slots
>>>>> Step    Matched  Condition
>>>>> -----  --------  ---------
>>>>> [0]           0  TARGET.Gpus ?: 0
>>>>> [1]           1  (RequestGpus ?: 0) >= (TARGET.Gpus ?: 0)
>>>>> 
>>>>> No successful match recorded.
>>>>> Last failed match: Wed Aug  7 22:45:27 2024
>>>>> 
>>>>> Reason for last match failure: no match found
>>>>> 
>>>>> 004.000:  Run analysis summary ignoring user priority.  Of 1 machines,
>>>>>   0 are rejected by your job's requirements
>>>>>   1 reject your job because of their own requirements
>>>>>   0 match and are already running your jobs
>>>>>   0 match but are serving other users
>>>>>   0 are able to run your job
>>>>> 
>>>>> WARNING:  Be advised:
>>>>> Job did not match any machines's constraints
>>>>> To see why, pick a machine that you think should match and add
>>>>>  -reverse -machine <name>
>>>>> to your query.
>>>>> 
>>>>> [root@ce-workspace /]# condor_q -all -l 4 | grep -i gpu
>>>>> AutoClusterAttrs = "MachineLastMatchTime,Offline,RemoteOwner,RequestCpus,RequestDisk,RequestGPUs,RequestMemory,TotalJobRuntime,ConcurrencyLimits,FlockTo,Rank,Requirements,DiskUsage,GlideinCpusIsGood,JobCpus,JobGPUs,JobIsRunning,JobMemory,JobStatus,MATCH_EXP_JOB_GLIDEIN_Cpus,MATCH_EXP_JOB_GLIDEIN_GPUs,MATCH_EXP_JOB_GLIDEIN_Memory,OriginalCpus,OriginalGPUs,OriginalMemory,TotalCpus,TotalGPUs,TotalMemory,WantWholeNode"
>>>>> GlideinGPUsIsGood =  !isUndefined(MATCH_EXP_JOB_GLIDEIN_GPUs) && (int(MATCH_EXP_JOB_GLIDEIN_GPUs) =!= error)
>>>>> JOB_GLIDEIN_GPUs = "$$(ifThenElse(WantWholeNode is true, !isUndefined(TotalGPUs) ? TotalGPUs : JobGPUs, OriginalGPUs))"
>>>>> JobGPUs = JobIsRunning ? int(MATCH_EXP_JOB_GLIDEIN_GPUs) : OriginalGPUs
>>>>> OriginalGPUs = undefined
>>>>> RequestGPUs = ifThenElse((WantWholeNode =?= true && OriginalGPUs =!= undefined),( !isUndefined(TotalGPUs) && TotalGPUs > 0) ? TotalGPUs : JobGPUs,OriginalGPUs)
>>>>> Requirements = (RequestGpus ?: 0) >= (TARGET.Gpus ?: 0)
>>>>> [root@ce-workspace /]# condor_ce_version
>>>>> $HTCondorCEVersion: 23.0.8 $
>>>>> $CondorVersion: 23.8.1 2024-06-27 BuildID: 742100 PackageID: 23.8.1-1 GitSHA: 8cf018d1 $
>>>>> $CondorPlatform: x86_64_AlmaLinux9 $
>>>>> [root@ce-workspace /]#
>>>>> _______________________________________________
>>>>> HTCondor-users mailing list
>>>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>>>> subject: Unsubscribe
>>>>> You can also unsubscribe by visiting
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.cs.wisc.edu_mailman_listinfo_htcondor-2Dusers&d=DwIFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EF06-Wh4L9CNLgD8bnIjNQ&m=p0P_yFwa6Gf99JwL4Sgp1E33007ZsGmps9icaOG7j1wLA1LIjyqSILgof0vfB1hE&s=4z99ZPvqqUGaL9Mu02EwmlMB1GXnpnc90qd7dQOsyug&e=
>>>>> 
>>>>> The archives can be found at:
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.cs.wisc.edu_archive_htcondor-2Dusers_&d=DwIFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EF06-Wh4L9CNLgD8bnIjNQ&m=p0P_yFwa6Gf99JwL4Sgp1E33007ZsGmps9icaOG7j1wLA1LIjyqSILgof0vfB1hE&s=qRvMMDCB5KF-T04yiJiOjBnBTKVfMk1ZVckSSzjZDKY&e=
>>>> 
>>>> 
>>>> _______________________________________________
>>>> HTCondor-users mailing list
>>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>>> subject: Unsubscribe
>>>> You can also unsubscribe by visiting
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.cs.wisc.edu_mailman_listinfo_htcondor-2Dusers&d=DwIGaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EF06-Wh4L9CNLgD8bnIjNQ&m=6NA-IR9Hv5jfUZzSVhCLgx-GxtnxmsUc9iGcRoqS3wNZJsHWuRYDSNhV98-d1EGV&s=8v2IJLNY0SK3I7ZLF8DI6kpZRL3S8cntIxpb9jPaOtY&e=
>>>> 
>>>> The archives can be found at:
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.cs.wisc.edu_archive_htcondor-2Dusers_&d=DwIGaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EF06-Wh4L9CNLgD8bnIjNQ&m=6NA-IR9Hv5jfUZzSVhCLgx-GxtnxmsUc9iGcRoqS3wNZJsHWuRYDSNhV98-d1EGV&s=x6m8L0s4H3rcMOMpJSHOjbdA6Wme429KkMemXAluHNg&e=

Follow-Ups:
- Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs
  - From: Jaime Frey
- Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs
  - From: Marco Mambelli

References:
- [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs
  - From: Marco Mambelli
- Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs
  - From: Marco Mambelli
- Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs
  - From: Jaime Frey
- Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs
  - From: Marco Mambelli
- Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs
  - From: Jaime Frey

Prev by Date: Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs
Next by Date: Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs
Previous by thread: Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs
Next by thread: Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs
Index(es):
- Date
- Thread

Mailing List Archives

Authenticated access

Re: [HTCondor-users] All jobs stay idle in HTCondor-CE because of GPU requirement - but host of jobs have no GPUs