[HTCondor-users] HTCondor can't execute the job with error: Error: can't find resource with ClaimId

Mailing List Archives Authenticated access	UW Madison Computer Sciences Department Computer Systems Lab

Date: Thu, 17 Jun 2021 13:32:43 +0300 (MSK)

From: "Dmitry A. Golubkov" <dmitry.golubkov@xxxxxxxxxxxxxx>

Subject: [HTCondor-users] HTCondor can't execute the job with error: Error: can't find resource with ClaimId

Dear all,

I have the htcondor cluster configured to use partitionable slots. After job submit, htcondor creates dynamic slots and trying to execute the job, but fails immediately with the error "Error: can't find resource with ClaimId (...) for 444 (ACTIVATE_CLAIM)" (please take a look at the log in attachment). After some time it re-creates dynamic slots and passes the execution succecefully, but why? Is it a known issue? This situation happens very often and slows down the execution of the jobs. Any ideas, how this can be solved? Or it is my own specific issue? The latest version of htcondor behaves the same way.

Thanks in advance,

Dmitry.

2021-06-17T10:06:27.179552492Z condor_shared_port[124]: ForkedChildrenCurrent = 0#012ForkedChildrenPeak = 0#012MyAddress = "<10.42.0.201:32819?addrs=10.42.0.201-32819&alias=pseven-htcondorexecute-deploy-68b66d9c55-nvrwz.pseven-htcondor&noUDP>"#012RequestsBlocked = 0#012RequestsFailed = 0#012RequestsPendingCurrent = 0#012RequestsPendingPeak = 1#012RequestsSucceeded = 11#012SharedPortCommandSinfuls = "<10.42.0.201:32819?alias=pseven-htcondorexecute-deploy-68b66d9c55-nvrwz.pseven-htcondor>" 2021-06-17T10:06:27.182125814Z condor_procd[120]: ProcAPI: read 8 pid entries out of 71 total entries in /proc 2021-06-17T10:06:27.182131828Z condor_procd[120]: ProcAPI: new boottime = 1621268011; old_boottime = 1621268011; /proc/stat boottime = 1621268011; /proc/uptime boottime = 1621268011 2021-06-17T10:06:27.182641701Z condor_procd[120]: ...snapshot complete 2021-06-17T10:06:27.183508281Z condor_shared_port[121]: About to update statistics in shared_port daemon ad file at /var/lock/condor/shared_port_ad : 2021-06-17T10:06:27.183528683Z condor_shared_port[121]: ForkedChildrenCurrent = 0#012ForkedChildrenPeak = 0#012MyAddress = "<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP>"#012RequestsBlocked = 0#012RequestsFailed = 0#012RequestsPendingCurrent = 0#012RequestsPendingPeak = 4#012RequestsSucceeded = 24#012SharedPortCommandSinfuls = "<10.42.0.202:38495?alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor>" 2021-06-17T10:06:27.783478999Z condor_procd[123]: taking a snapshot... 2021-06-17T10:06:27.783509562Z condor_procd[123]: ProcAPI: read 8 pid entries out of 71 total entries in /proc 2021-06-17T10:06:27.783669969Z condor_procd[123]: ProcAPI: new boottime = 1621268011; old_boottime = 1621268010; /proc/stat boottime = 1621268011; /proc/uptime boottime = 1621268011 2021-06-17T10:06:27.784338111Z condor_procd[123]: ...snapshot complete 2021-06-17T10:06:36.188645789Z condor_startd[125]: Swap space: 48075796 2021-06-17T10:06:36.189293248Z condor_startd[125]: slot1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:36.189568700Z condor_startd[125]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:36.189578441Z condor_startd[125]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:36.190057135Z condor_startd[125]: slot1: Sent update to 1 collector(s) 2021-06-17T10:06:36.191389053Z condor_startd[122]: Swap space: 48075796 2021-06-17T10:06:36.192068486Z condor_startd[122]: slot1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:36.192361412Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:36.192386593Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:36.192827978Z condor_startd[122]: slot1: Sent update to 1 collector(s) 2021-06-17T10:06:37.762162200Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:38.782230507Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:38.782660296Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 411 2021-06-17T10:06:38.782670107Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:39.801495576Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:39.801912087Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 416 2021-06-17T10:06:39.801923034Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:40.818181617Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:40.818205256Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:40.818231155Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:40.818349689Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000030; send_time=0.000058; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:40555>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:40.819427800Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:40.819965614Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 421 2021-06-17T10:06:40.819972487Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:40.827063801Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:40.827337912Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:40.828013418Z condor_collector[275]: (Sending 2 ads in response to query) 2021-06-17T10:06:40.828021085Z condor_collector[51]: QueryWorker: forked new worker with id 275 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:40.828126377Z condor_collector[275]: Query info: matched=2; skipped=0; query_time=0.000245; send_time=0.000183; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:37457>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:40.828520093Z condor_collector[51]: QueryWorker: Child 275 done 2021-06-17T10:06:41.836682288Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:41.837092996Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:41.837124266Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:41.837151246Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:41.837170070Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000052; send_time=0.000079; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:41545>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:41.838220417Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:41.838943467Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 426 2021-06-17T10:06:41.838964443Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:41.846459656Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:41.846751333Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:41.847406110Z condor_collector[276]: (Sending 2 ads in response to query) 2021-06-17T10:06:41.847416449Z condor_collector[51]: QueryWorker: forked new worker with id 276 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:41.847578440Z condor_collector[276]: Query info: matched=2; skipped=0; query_time=0.000290; send_time=0.000189; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:45141>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:41.847952448Z condor_collector[51]: QueryWorker: Child 276 done 2021-06-17T10:06:42.841112013Z condor_schedd[129]: Clearing userlog file cache 2021-06-17T10:06:42.841206226Z condor_schedd[129]: JobsRunning = 0 2021-06-17T10:06:42.841230338Z condor_schedd[129]: JobsIdle = 0 2021-06-17T10:06:42.841248959Z condor_schedd[129]: JobsHeld = 0 2021-06-17T10:06:42.841276071Z condor_schedd[129]: JobsRemoved = 0 2021-06-17T10:06:42.841293360Z condor_schedd[129]: LocalUniverseJobsRunning = 0 2021-06-17T10:06:42.841310901Z condor_schedd[129]: LocalUniverseJobsIdle = 0 2021-06-17T10:06:42.841327051Z condor_schedd[129]: SchedUniverseJobsRunning = 0 2021-06-17T10:06:42.841343734Z condor_schedd[129]: SchedUniverseJobsIdle = 0 2021-06-17T10:06:42.841359403Z condor_schedd[129]: NumSubmitters = 0 2021-06-17T10:06:42.841380883Z condor_schedd[129]: MaxJobsRunning = 10000 2021-06-17T10:06:42.841399596Z condor_schedd[129]: MaxRunningSchedulerJobsPerOwner = 200 2021-06-17T10:06:42.841861469Z condor_schedd[129]: TransferQueueManager stats: active up=0/100 down=0/100; waiting up=0 down=0; wait time up=0s down=0s 2021-06-17T10:06:42.841904250Z condor_schedd[129]: TransferQueueManager upload 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load 2021-06-17T10:06:42.841919453Z condor_schedd[129]: TransferQueueManager download 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load 2021-06-17T10:06:42.844344646Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:42.844403368Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:42.844921671Z condor_schedd[129]: Sent HEART BEAT ad to 1 collectors. Number of active submittors=0 2021-06-17T10:06:42.845709125Z condor_collector[51]: ScheddAd : Updating ... "< parallel_schedd@xxxxxxxxxxxxxxxxxxxxxx , 10.42.0.213 >" 2021-06-17T10:06:42.847540572Z condor_collector[51]: SubmittorAd : Updating ... "< user20001@pseven-htcondorparallel_schedd@submit.pseven-htcondor , 10.42.0.213 >" 2021-06-17T10:06:42.848195724Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:42.848250043Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:42.848274051Z condor_schedd[129]: Sent ad to 1 collectors for user20001 Hit=0 Tot=0 Idle=0 Run=0 2021-06-17T10:06:42.848290996Z condor_schedd[129]: Sent owner (0 jobs) ad to schedd plugins 2021-06-17T10:06:42.848306591Z condor_schedd[129]: ============ Begin clean_shadow_recs ============= 2021-06-17T10:06:42.848348061Z condor_schedd[129]: ============ End clean_shadow_recs ============= 2021-06-17T10:06:42.848364764Z condor_schedd[129]: Sending RESCHEDULE command to negotiator(s) 2021-06-17T10:06:42.849373549Z condor_schedd[129]: Will use TCP to update collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:42.849419860Z condor_schedd[129]: Trying to query collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:42.851514101Z condor_collector[51]: Got QUERY_NEGOTIATOR_ADS 2021-06-17T10:06:42.851556368Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:42.851799433Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:42.852444001Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000104; send_time=0.000567; type=Negotiator; requirements={true}; locate=0; limit=0; from=SCHEDD; peer=<10.42.0.213:42875>; projection={}; filter_private_ads=0 2021-06-17T10:06:42.855972302Z condor_schedd[129]: -------- Begin starting jobs -------- 2021-06-17T10:06:42.856031530Z condor_schedd[129]: -------- Done starting jobs -------- 2021-06-17T10:06:42.856055225Z condor_schedd[129]: SharedPortClient: sent connection request to negotiator negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx for shared port id negotiator_13_cf39 2021-06-17T10:06:42.856081683Z condor_schedd[129]: Completed RESCHEDULE to negotiator negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:42.857524069Z condor_negotiator[52]: ---------- Started Negotiation Cycle ---------- 2021-06-17T10:06:42.857578055Z condor_negotiator[52]: Phase 1: Obtaining ads from collector ... 2021-06-17T10:06:42.857598552Z condor_negotiator[52]: Getting startd private ads ... 2021-06-17T10:06:42.858364817Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:06:42.858984756Z condor_collector[51]: Got QUERY_STARTD_PVT_ADS 2021-06-17T10:06:42.860564841Z condor_collector[51]: QueryWorker: forked new high priority worker with id 277 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:42.860616497Z condor_collector[277]: (Sending 2 ads in response to query) 2021-06-17T10:06:42.867308637Z condor_collector[277]: Query info: matched=2; skipped=0; query_time=0.000727; send_time=0.004303; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:43253>; projection={}; filter_private_ads=0 2021-06-17T10:06:42.868061766Z condor_collector[51]: QueryWorker: Child 277 done 2021-06-17T10:06:42.868104398Z condor_negotiator[52]: Getting Scheduler, Submitter and Machine ads ... 2021-06-17T10:06:42.868127191Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:06:42.869585675Z condor_collector[51]: Got QUERY_ANY_ADS 2021-06-17T10:06:42.871275282Z condor_collector[51]: QueryWorker: forked new high priority worker with id 278 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:42.871517770Z condor_collector[278]: (Sending 4 ads in response to query) 2021-06-17T10:06:42.873738838Z condor_collector[278]: Query info: matched=4; skipped=9; query_time=0.000985; send_time=0.001999; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:43531>; projection={}; filter_private_ads=0 2021-06-17T10:06:42.874960630Z condor_collector[51]: QueryWorker: Child 278 done 2021-06-17T10:06:42.878752692Z condor_negotiator[52]: Sorting 4 ads ... 2021-06-17T10:06:42.879980377Z condor_negotiator[52]: Ignoring submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor from schedd at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> with no requested jobs 2021-06-17T10:06:42.880010484Z condor_negotiator[52]: Ignoring submitter user20001@pseven-htcondor from schedd at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> with no requested jobs 2021-06-17T10:06:42.880026361Z condor_negotiator[52]: Got ads: 4 public and 2 private 2021-06-17T10:06:42.880037768Z condor_negotiator[52]: Public ads include 0 submitter, 2 startd 2021-06-17T10:06:42.880048845Z condor_negotiator[52]: Phase 2: Performing accounting ... 2021-06-17T10:06:42.880060457Z condor_negotiator[52]: Entering compute_significant_attrs() 2021-06-17T10:06:42.880070806Z condor_negotiator[52]: Leaving compute_significant_attrs() - result=_cp_orig_RequestCpus,_cp_orig_RequestDisk,_cp_orig_RequestMemory,KFlops,MachineLastMatchTime,Offline,RemoteOwner,RequestCpus,RequestDisk,RequestMemory,Scheduler,TotalJobRuntime 2021-06-17T10:06:42.880083275Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:42.880120536Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:42.880129526Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:42.880239022Z condor_collector[51]: (Sending 0 ads in response to query) 2021-06-17T10:06:42.880256411Z condor_collector[51]: Query info: matched=0; skipped=1; query_time=0.000082; send_time=0.000101; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:46003>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:42.882848015Z condor_negotiator[52]: Phase 3: Sorting submitter ads by priority ... 2021-06-17T10:06:42.882873888Z condor_negotiator[52]: Starting prefetch round; 0 potential prefetches to do. 2021-06-17T10:06:42.882885694Z condor_negotiator[52]: Assigned 0 units of work for prefetching. 2021-06-17T10:06:42.882987472Z condor_negotiator[52]: Prefetch summary: 0 attempted, 0 successful. 2021-06-17T10:06:42.883211606Z condor_negotiator[52]: Phase 4.1: Negotiating with schedds ... 2021-06-17T10:06:42.883250436Z condor_negotiator[52]: numSlots = 2 (after trimming=2) 2021-06-17T10:06:42.883258450Z condor_negotiator[52]: slotWeightTotal = 200.000000 2021-06-17T10:06:42.883264010Z condor_negotiator[52]: minSlotWeight = 100.000000 2021-06-17T10:06:42.883268985Z condor_negotiator[52]: pieLeft = 0.000 2021-06-17T10:06:42.883273852Z condor_negotiator[52]: NormalFactor = 0.000000 2021-06-17T10:06:42.883278766Z condor_negotiator[52]: MaxPrioValue = 0.000000 2021-06-17T10:06:42.883283622Z condor_negotiator[52]: NumSubmitterAds = 0 2021-06-17T10:06:42.883288464Z condor_negotiator[52]: resources used scheddUsed= 0.000000 2021-06-17T10:06:42.883293516Z condor_negotiator[52]: negotiateWithGroup resources used submitterAds length 0 2021-06-17T10:06:42.883300508Z condor_negotiator[52]: ---------- Finished Negotiation Cycle ---------- 2021-06-17T10:06:42.883305734Z condor_negotiator[52]: Updating collector with accounting information 2021-06-17T10:06:42.883374056Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:06:42.883386823Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:06:42.883559129Z condor_negotiator[52]: Done Updating collector with accounting information 2021-06-17T10:06:42.883725609Z condor_collector[51]: AccountingAd : Updating ... "< <none>negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:06:42.891606427Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:42.892000722Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:42.892535933Z condor_collector[51]: QueryWorker: forked new worker with id 279 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:42.892587529Z condor_collector[279]: (Sending 2 ads in response to query) 2021-06-17T10:06:42.892836544Z condor_collector[279]: Query info: matched=2; skipped=0; query_time=0.000319; send_time=0.000170; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:40757>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:42.893410266Z condor_collector[51]: QueryWorker: Child 279 done 2021-06-17T10:06:43.918399572Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:43.919605053Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:43.919664159Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:43.919688320Z condor_collector[51]: (Sending 0 ads in response to query) 2021-06-17T10:06:43.919963738Z condor_collector[51]: Query info: matched=0; skipped=1; query_time=0.000125; send_time=0.000143; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:41997>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:43.943880204Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:43.945023552Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:43.946664978Z condor_collector[51]: QueryWorker: forked new worker with id 280 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:43.946705609Z condor_collector[280]: (Sending 2 ads in response to query) 2021-06-17T10:06:43.947445745Z condor_collector[280]: Query info: matched=2; skipped=0; query_time=0.000989; send_time=0.000544; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:37777>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:43.949105997Z condor_collector[51]: QueryWorker: Child 280 done 2021-06-17T10:06:44.957672324Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:44.957985589Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:44.957999747Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:44.958005391Z condor_collector[51]: (Sending 0 ads in response to query) 2021-06-17T10:06:44.958049558Z condor_collector[51]: Query info: matched=0; skipped=1; query_time=0.000041; send_time=0.000031; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:34207>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:44.965554096Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:44.965776376Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:44.966388469Z condor_collector[51]: QueryWorker: forked new worker with id 281 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:44.966398587Z condor_collector[281]: (Sending 2 ads in response to query) 2021-06-17T10:06:44.966499364Z condor_collector[281]: Query info: matched=2; skipped=0; query_time=0.000294; send_time=0.000135; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:39885>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:44.966890450Z condor_collector[51]: QueryWorker: Child 281 done 2021-06-17T10:06:45.322486194Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:45.323177148Z condor_collector[51]: QueryWorker: forked new worker with id 282 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:45.323194660Z condor_collector[282]: (Sending 2 ads in response to query) 2021-06-17T10:06:45.323756335Z condor_collector[282]: Query info: matched=2; skipped=0; query_time=0.000267; send_time=0.000688; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.207:34077>; projection={}; filter_private_ads=0 2021-06-17T10:06:45.324187111Z condor_collector[51]: QueryWorker: Child 282 done 2021-06-17T10:06:45.975003424Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:45.975390155Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:45.975408232Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:45.975412907Z condor_collector[51]: (Sending 0 ads in response to query) 2021-06-17T10:06:45.975532409Z condor_collector[51]: Query info: matched=0; skipped=1; query_time=0.000029; send_time=0.000040; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:46217>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:45.984280701Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:45.984591639Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:45.985184716Z condor_collector[51]: QueryWorker: forked new worker with id 283 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:45.985235321Z condor_collector[283]: (Sending 2 ads in response to query) 2021-06-17T10:06:45.985473040Z condor_collector[283]: Query info: matched=2; skipped=0; query_time=0.000325; send_time=0.000157; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:36389>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:45.986060643Z condor_collector[51]: QueryWorker: Child 283 done 2021-06-17T10:06:46.064376154Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:46.064389372Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:46.064392225Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:46.064444887Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000016; send_time=0.000058; type=Scheduler; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.207:44031>; projection={MyAddress AddressV1 CondorVersion CondorPlatform Name Machine}; filter_private_ads=0 2021-06-17T10:06:46.066922816Z condor_schedd[129]: OwnerCheck retval 1 (success),no ad 2021-06-17T10:06:46.067018305Z condor_schedd[129]: schedd: NewCluster rval 2 errno 0 2021-06-17T10:06:46.067550047Z condor_schedd[129]: OwnerCheck retval 1 (success),no ad 2021-06-17T10:06:46.067558659Z condor_schedd[129]: schedd: NewProc rval 0 errno 0 2021-06-17T10:06:46.079818308Z condor_schedd[129]: OwnerCheck retval 1 (success),no ad 2021-06-17T10:06:46.079833086Z condor_schedd[129]: schedd: NewProc rval 1 errno 0 2021-06-17T10:06:46.082291573Z condor_schedd[129]: OwnerCheck retval 1 (success),no ad 2021-06-17T10:06:46.082331651Z condor_schedd[129]: schedd: NewProc rval 2 errno 0 2021-06-17T10:06:46.085458165Z condor_schedd[129]: OwnerCheck retval 1 (success),no ad 2021-06-17T10:06:46.085468606Z condor_schedd[129]: schedd: NewProc rval 3 errno 0 2021-06-17T10:06:46.087546475Z condor_schedd[129]: OwnerCheck retval 1 (success),no ad 2021-06-17T10:06:46.087562408Z condor_schedd[129]: schedd: NewProc rval 4 errno 0 2021-06-17T10:06:46.095452232Z condor_schedd[129]: New job: 2.0 2021-06-17T10:06:46.095502668Z condor_schedd[129]: New job: 2.0, Duplicate Keys: 2, Total Keys: 3 2021-06-17T10:06:46.095508475Z condor_schedd[129]: New job: 2.1 2021-06-17T10:06:46.095512375Z condor_schedd[129]: New job: 2.1, Duplicate Keys: 2, Total Keys: 12 2021-06-17T10:06:46.095515931Z condor_schedd[129]: New job: 2.2 2021-06-17T10:06:46.095521365Z condor_schedd[129]: New job: 2.2, Duplicate Keys: 2, Total Keys: 11 2021-06-17T10:06:46.095525394Z condor_schedd[129]: New job: 2.3 2021-06-17T10:06:46.095528910Z condor_schedd[129]: New job: 2.3, Duplicate Keys: 2, Total Keys: 12 2021-06-17T10:06:46.095542861Z condor_schedd[129]: New job: 2.4 2021-06-17T10:06:46.095595347Z condor_schedd[129]: New job: 2.4, Duplicate Keys: 2, Total Keys: 12 2021-06-17T10:06:46.096530305Z condor_schedd[129]: QMGR Connection closed 2021-06-17T10:06:46.097054249Z condor_schedd[129]: -------- Begin starting jobs -------- 2021-06-17T10:06:46.097065880Z condor_schedd[129]: -------- Done starting jobs -------- 2021-06-17T10:06:46.099446854Z condor_collector[51]: ScheddAd : Updating ... "< parallel_schedd@xxxxxxxxxxxxxxxxxxxxxx , 10.42.0.213 >" 2021-06-17T10:06:46.099478601Z condor_collector[51]: SubmittorAd : Updating ... "< user20001@pseven-htcondorparallel_schedd@submit.pseven-htcondor , 10.42.0.213 >" 2021-06-17T10:06:46.102883395Z condor_collector[51]: Got QUERY_NEGOTIATOR_ADS 2021-06-17T10:06:46.102925282Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:46.102932193Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:46.102935579Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000086; send_time=0.000447; type=Negotiator; requirements={true}; locate=0; limit=0; from=SCHEDD; peer=<10.42.0.213:44893>; projection={}; filter_private_ads=0 2021-06-17T10:06:46.103832963Z condor_schedd[129]: Found idle MPI cluster 2 2021-06-17T10:06:46.103881297Z condor_schedd[129]: Started timer (55) to call handleDedicatedJobs() in 0 secs 2021-06-17T10:06:46.103888438Z condor_schedd[129]: JobsRunning = 0 2021-06-17T10:06:46.103892587Z condor_schedd[129]: JobsIdle = 5 2021-06-17T10:06:46.103895915Z condor_schedd[129]: JobsHeld = 0 2021-06-17T10:06:46.103899044Z condor_schedd[129]: JobsRemoved = 0 2021-06-17T10:06:46.103903156Z condor_schedd[129]: LocalUniverseJobsRunning = 0 2021-06-17T10:06:46.103907058Z condor_schedd[129]: LocalUniverseJobsIdle = 0 2021-06-17T10:06:46.103910713Z condor_schedd[129]: SchedUniverseJobsRunning = 0 2021-06-17T10:06:46.103916215Z condor_schedd[129]: SchedUniverseJobsIdle = 0 2021-06-17T10:06:46.103919912Z condor_schedd[129]: NumSubmitters = 1 2021-06-17T10:06:46.103923904Z condor_schedd[129]: MaxJobsRunning = 10000 2021-06-17T10:06:46.103927536Z condor_schedd[129]: MaxRunningSchedulerJobsPerOwner = 200 2021-06-17T10:06:46.103931750Z condor_schedd[129]: TransferQueueManager stats: active up=0/100 down=0/100; waiting up=0 down=0; wait time up=0s down=0s 2021-06-17T10:06:46.103936041Z condor_schedd[129]: TransferQueueManager upload 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load 2021-06-17T10:06:46.103939520Z condor_schedd[129]: TransferQueueManager download 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load 2021-06-17T10:06:46.103943010Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:46.103948642Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:46.103952975Z condor_schedd[129]: Sent HEART BEAT ad to 1 collectors. Number of active submittors=1 2021-06-17T10:06:46.103956757Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:46.103961122Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:46.103966049Z condor_schedd[129]: Sent ad to 1 collectors for user20001 Hit=5 Tot=5 Idle=0 Run=0 2021-06-17T10:06:46.103970940Z condor_schedd[129]: ============ Begin clean_shadow_recs ============= 2021-06-17T10:06:46.103975554Z condor_schedd[129]: ============ End clean_shadow_recs ============= 2021-06-17T10:06:46.104009101Z condor_schedd[129]: Sending RESCHEDULE command to negotiator(s) 2021-06-17T10:06:46.104014858Z condor_schedd[129]: Will use TCP to update collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:46.104018694Z condor_schedd[129]: Trying to query collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:46.104021975Z condor_schedd[129]: SharedPortClient: sent connection request to negotiator negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx for shared port id negotiator_13_cf39 2021-06-17T10:06:46.104068167Z condor_negotiator[52]: ---------- Started Negotiation Cycle ---------- 2021-06-17T10:06:46.104086423Z condor_negotiator[52]: Phase 1: Obtaining ads from collector ... 2021-06-17T10:06:46.104087802Z condor_schedd[129]: Completed RESCHEDULE to negotiator negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:46.104089871Z condor_negotiator[52]: Getting startd private ads ... 2021-06-17T10:06:46.104093601Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:06:46.104108692Z condor_schedd[129]: Starting DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:06:46.104113565Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:06:46.104116453Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:06:46.104119740Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:06:46.104123135Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:06:46.104127342Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:06:46.104130605Z condor_schedd[129]: Found 1 idle dedicated job(s) 2021-06-17T10:06:46.104133784Z condor_schedd[129]: DedicatedScheduler: Listing all dedicated jobs - 2021-06-17T10:06:46.104136725Z condor_schedd[129]: Dedicated job: 2.0 user20001 2021-06-17T10:06:46.104139553Z condor_schedd[129]: Trying to query collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:46.105083910Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:46.105224801Z condor_collector[51]: QueryWorker: forked new worker with id 284 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:46.105337342Z condor_collector[284]: (Sending 2 ads in response to query) 2021-06-17T10:06:46.105498173Z condor_collector[51]: Got QUERY_STARTD_PVT_ADS 2021-06-17T10:06:46.106769713Z condor_collector[51]: QueryWorker: forked new high priority worker with id 285 ( max 4 active 2 pending 0 ) 2021-06-17T10:06:46.106790805Z condor_collector[285]: (Sending 2 ads in response to query) 2021-06-17T10:06:46.106795758Z condor_collector[284]: Query info: matched=2; skipped=0; query_time=0.000295; send_time=0.000672; type=Machine; requirements={((DedicatedScheduler == "DedicatedScheduler@parallel_schedd@submit.pseven-htcondor"))}; locate=0; limit=0; from=SCHEDD; peer=<10.42.0.213:39655>; projection={}; filter_private_ads=0 2021-06-17T10:06:46.106966014Z condor_collector[51]: QueryWorker: Child 284 done 2021-06-17T10:06:46.106978726Z condor_collector[285]: Query info: matched=2; skipped=0; query_time=0.000266; send_time=0.000902; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:44533>; projection={}; filter_private_ads=0 2021-06-17T10:06:46.109534210Z condor_schedd[129]: Found 2 potential dedicated resources in 0 seconds 2021-06-17T10:06:46.109568975Z condor_schedd[129]: Duplicate x100 (100/0) partitionable res slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:46.109574007Z condor_schedd[129]: Duplicate x100 (100/0) partitionable res slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:46.109577325Z condor_schedd[129]: idle resource list 2021-06-17T10:06:46.109580269Z condor_schedd[129]: ************ empty ************ 2021-06-17T10:06:46.109583192Z condor_schedd[129]: limbo resource list 2021-06-17T10:06:46.109586066Z condor_schedd[129]: ************ empty ************ 2021-06-17T10:06:46.109589040Z condor_schedd[129]: serial c/i resource list 2021-06-17T10:06:46.109592005Z condor_schedd[129]: ************ empty ************ 2021-06-17T10:06:46.109595250Z condor_schedd[129]: unclaimed resource list 2021-06-17T10:06:46.109599131Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ... 2021-06-17T10:06:46.111034259Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:46.111298577Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:46.111320318Z condor_negotiator[52]: Getting Scheduler, Submitter and Machine ads ... 2021-06-17T10:06:46.111332707Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:06:46.111336875Z condor_collector[51]: Got QUERY_ANY_ADS 2021-06-17T10:06:46.111406141Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ... 2021-06-17T10:06:46.111920594Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:46.111940060Z condor_schedd[129]: busy resource list 2021-06-17T10:06:46.111945166Z condor_schedd[129]: ************ empty ************ 2021-06-17T10:06:46.112974409Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.0 2021-06-17T10:06:46.113019626Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.1 2021-06-17T10:06:46.113028772Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.2 2021-06-17T10:06:46.113032934Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.3 2021-06-17T10:06:46.113036611Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.4 2021-06-17T10:06:46.113050248Z condor_schedd[129]: satisfyJobs: testing 5 jobs against 200 slots 2021-06-17T10:06:46.113054261Z condor_schedd[129]: satisfyJobs: finding resources for 2.0 2021-06-17T10:06:46.113057362Z condor_schedd[129]: satisfyJobs: 2.0 satisfied with slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:46.113060643Z condor_schedd[129]: satisfyJobs: finding resources for 2.1 2021-06-17T10:06:46.113063913Z condor_schedd[129]: satisfyJobs: 2.1 satisfied with slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:46.113072256Z condor_schedd[129]: satisfyJobs: finding resources for 2.2 2021-06-17T10:06:46.113075961Z condor_schedd[129]: satisfyJobs: 2.2 satisfied with slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:46.113079392Z condor_schedd[129]: satisfyJobs: finding resources for 2.3 2021-06-17T10:06:46.113082764Z condor_schedd[129]: satisfyJobs: 2.3 satisfied with slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:46.113086226Z condor_schedd[129]: satisfyJobs: finding resources for 2.4 2021-06-17T10:06:46.113134208Z condor_schedd[129]: satisfyJobs: 2.4 satisfied with slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:46.113144393Z condor_schedd[129]: satisfyJobs: jobs were satisfied 2021-06-17T10:06:46.113147946Z condor_schedd[129]: Satisfied job 2 with 5 unclaimed resources 2021-06-17T10:06:46.113150684Z condor_schedd[129]: Generating 5 resource requests for job 2 2021-06-17T10:06:46.113153345Z condor_schedd[129]: Waiting to negotiate for 5 dedicated resource request(s) 2021-06-17T10:06:46.113155923Z condor_schedd[129]: In DedicatedScheduler::publishRequestAd() 2021-06-17T10:06:46.113158609Z condor_schedd[129]: Adding submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor to the submitter map for default pool. 2021-06-17T10:06:46.113161494Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:46.113177482Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:46.113180751Z condor_schedd[129]: Entering DedicatedScheduler::checkSanity() 2021-06-17T10:06:46.113183377Z condor_schedd[129]: Timer (51) will call checkSanity() again in 159 seconds 2021-06-17T10:06:46.113186047Z condor_schedd[129]: Finished DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:06:46.113196331Z condor_schedd[129]: Setting delay until next queue scan to 1 seconds 2021-06-17T10:06:46.113779424Z condor_collector[286]: (Sending 4 ads in response to query) 2021-06-17T10:06:46.113807873Z condor_collector[51]: QueryWorker: forked new high priority worker with id 286 ( max 4 active 2 pending 0 ) 2021-06-17T10:06:46.113813410Z condor_collector[51]: QueryWorker: Child 285 done 2021-06-17T10:06:46.113817467Z condor_collector[286]: Query info: matched=4; skipped=9; query_time=0.000923; send_time=0.000701; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:39859>; projection={}; filter_private_ads=0 2021-06-17T10:06:46.113837383Z condor_collector[51]: QueryWorker: Child 286 done 2021-06-17T10:06:46.113841985Z condor_negotiator[52]: Sorting 4 ads ... 2021-06-17T10:06:46.113845557Z condor_negotiator[52]: Ignoring submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor from schedd at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> with no requested jobs 2021-06-17T10:06:46.113849934Z condor_negotiator[52]: Ignoring submitter user20001@pseven-htcondor from schedd at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> with no requested jobs 2021-06-17T10:06:46.113854180Z condor_negotiator[52]: Got ads: 4 public and 2 private 2021-06-17T10:06:46.113857327Z condor_negotiator[52]: Public ads include 0 submitter, 2 startd 2021-06-17T10:06:46.113860272Z condor_negotiator[52]: Phase 2: Performing accounting ... 2021-06-17T10:06:46.113863441Z condor_negotiator[52]: Entering compute_significant_attrs() 2021-06-17T10:06:46.113866797Z condor_negotiator[52]: Leaving compute_significant_attrs() - result=_cp_orig_RequestCpus,_cp_orig_RequestDisk,_cp_orig_RequestMemory,KFlops,MachineLastMatchTime,Offline,RemoteOwner,RequestCpus,RequestDisk,RequestMemory,Scheduler,TotalJobRuntime 2021-06-17T10:06:46.113870160Z condor_collector[51]: SubmittorAd : Updating ... "< DedicatedScheduler@parallel_schedd@submit.pseven-htcondorparallel_schedd@xxxxxxxxxxxxxxxxxxxxxx , 10.42.0.213 >" 2021-06-17T10:06:46.113873659Z condor_negotiator[52]: Phase 3: Sorting submitter ads by priority ... 2021-06-17T10:06:46.113876538Z condor_negotiator[52]: Starting prefetch round; 0 potential prefetches to do. 2021-06-17T10:06:46.113879437Z condor_negotiator[52]: Assigned 0 units of work for prefetching. 2021-06-17T10:06:46.113882315Z condor_negotiator[52]: Prefetch summary: 0 attempted, 0 successful. 2021-06-17T10:06:46.113885067Z condor_negotiator[52]: Phase 4.1: Negotiating with schedds ... 2021-06-17T10:06:46.113888764Z condor_negotiator[52]: numSlots = 2 (after trimming=2) 2021-06-17T10:06:46.113891728Z condor_negotiator[52]: slotWeightTotal = 200.000000 2021-06-17T10:06:46.113894553Z condor_negotiator[52]: minSlotWeight = 100.000000 2021-06-17T10:06:46.113897240Z condor_negotiator[52]: pieLeft = 0.000 2021-06-17T10:06:46.113900345Z condor_negotiator[52]: NormalFactor = 0.000000 2021-06-17T10:06:46.113914720Z condor_negotiator[52]: MaxPrioValue = 0.000000 2021-06-17T10:06:46.113918594Z condor_negotiator[52]: NumSubmitterAds = 0 2021-06-17T10:06:46.113922271Z condor_negotiator[52]: resources used scheddUsed= 0.000000 2021-06-17T10:06:46.113936134Z condor_negotiator[52]: negotiateWithGroup resources used submitterAds length 0 2021-06-17T10:06:46.113940340Z condor_negotiator[52]: ---------- Finished Negotiation Cycle ---------- 2021-06-17T10:06:46.113943754Z condor_negotiator[52]: Updating collector with accounting information 2021-06-17T10:06:46.113951316Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:06:46.113962765Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:06:46.114035951Z condor_negotiator[52]: Done Updating collector with accounting information 2021-06-17T10:06:46.114440133Z condor_collector[51]: AccountingAd : Updating ... "< <none>negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:06:46.993855315Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:46.994101414Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:46.994114701Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:46.994147277Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:46.994200123Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000024; send_time=0.000060; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:45427>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:46.995201480Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:46.995655318Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 447 2021-06-17T10:06:46.995669238Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:47.005017158Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:47.005413784Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:47.006231168Z condor_collector[51]: QueryWorker: forked new worker with id 287 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:47.006245612Z condor_collector[287]: (Sending 2 ads in response to query) 2021-06-17T10:06:47.006249112Z condor_collector[287]: Query info: matched=2; skipped=0; query_time=0.000295; send_time=0.000142; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:33517>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:47.006478867Z condor_collector[51]: QueryWorker: Child 287 done 2021-06-17T10:06:47.998224276Z condor_schedd[129]: Found idle MPI cluster 2 2021-06-17T10:06:47.998244458Z condor_schedd[129]: Started timer (56) to call handleDedicatedJobs() in 0 secs 2021-06-17T10:06:47.998247846Z condor_schedd[129]: JobsRunning = 0 2021-06-17T10:06:47.998250072Z condor_schedd[129]: JobsIdle = 5 2021-06-17T10:06:47.998252268Z condor_schedd[129]: JobsHeld = 0 2021-06-17T10:06:47.998254361Z condor_schedd[129]: JobsRemoved = 0 2021-06-17T10:06:47.998256457Z condor_schedd[129]: LocalUniverseJobsRunning = 0 2021-06-17T10:06:47.998259031Z condor_schedd[129]: LocalUniverseJobsIdle = 0 2021-06-17T10:06:47.998262296Z condor_schedd[129]: SchedUniverseJobsRunning = 0 2021-06-17T10:06:47.998265411Z condor_schedd[129]: SchedUniverseJobsIdle = 0 2021-06-17T10:06:47.998268407Z condor_schedd[129]: NumSubmitters = 1 2021-06-17T10:06:47.998271575Z condor_schedd[129]: MaxJobsRunning = 10000 2021-06-17T10:06:47.998275291Z condor_schedd[129]: MaxRunningSchedulerJobsPerOwner = 200 2021-06-17T10:06:47.999122562Z condor_schedd[129]: TransferQueueManager stats: active up=0/100 down=0/100; waiting up=0 down=0; wait time up=0s down=0s 2021-06-17T10:06:47.999136536Z condor_schedd[129]: TransferQueueManager upload 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load 2021-06-17T10:06:47.999141860Z condor_schedd[129]: TransferQueueManager download 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load 2021-06-17T10:06:47.999235471Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:47.999250039Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:47.999677539Z condor_collector[51]: ScheddAd : Updating ... "< parallel_schedd@xxxxxxxxxxxxxxxxxxxxxx , 10.42.0.213 >" 2021-06-17T10:06:48.000100806Z condor_collector[51]: SubmittorAd : Updating ... "< user20001@pseven-htcondorparallel_schedd@submit.pseven-htcondor , 10.42.0.213 >" 2021-06-17T10:06:48.000393252Z condor_schedd[129]: Sent HEART BEAT ad to 1 collectors. Number of active submittors=1 2021-06-17T10:06:48.000408982Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:48.000414176Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:48.000418035Z condor_schedd[129]: Sent ad to 1 collectors for user20001 Hit=5 Tot=5 Idle=0 Run=0 2021-06-17T10:06:48.000421292Z condor_schedd[129]: ============ Begin clean_shadow_recs ============= 2021-06-17T10:06:48.000424522Z condor_schedd[129]: ============ End clean_shadow_recs ============= 2021-06-17T10:06:48.000427764Z condor_schedd[129]: Sending RESCHEDULE command to negotiator(s) 2021-06-17T10:06:48.000584135Z condor_schedd[129]: Will use TCP to update collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:48.000592970Z condor_schedd[129]: Trying to query collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:48.001132274Z condor_collector[51]: Got QUERY_NEGOTIATOR_ADS 2021-06-17T10:06:48.001143915Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:48.001147290Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:48.003122065Z condor_schedd[129]: SharedPortClient: sent connection request to negotiator negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx for shared port id negotiator_13_cf39 2021-06-17T10:06:48.003129449Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000022; send_time=0.000149; type=Negotiator; requirements={true}; locate=0; limit=0; from=SCHEDD; peer=<10.42.0.213:37441>; projection={}; filter_private_ads=0 2021-06-17T10:06:48.003150288Z condor_negotiator[52]: New cycle requested but just finished one -- delaying 3 secs 2021-06-17T10:06:48.003151240Z condor_schedd[129]: Completed RESCHEDULE to negotiator negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:48.003157055Z condor_schedd[129]: Starting DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:06:48.003161000Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:06:48.003164583Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:06:48.003168310Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:06:48.003171709Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:06:48.003174858Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:06:48.003178589Z condor_schedd[129]: Found 1 idle dedicated job(s) 2021-06-17T10:06:48.003181953Z condor_schedd[129]: DedicatedScheduler: Listing all dedicated jobs - 2021-06-17T10:06:48.003185271Z condor_schedd[129]: Dedicated job: 2.0 user20001 2021-06-17T10:06:48.003188492Z condor_schedd[129]: Trying to query collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:48.003455959Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:48.004459614Z condor_collector[51]: QueryWorker: forked new worker with id 288 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:48.004494667Z condor_collector[288]: (Sending 2 ads in response to query) 2021-06-17T10:06:48.004724784Z condor_collector[288]: Query info: matched=2; skipped=0; query_time=0.000290; send_time=0.000637; type=Machine; requirements={((DedicatedScheduler == "DedicatedScheduler@parallel_schedd@submit.pseven-htcondor"))}; locate=0; limit=0; from=SCHEDD; peer=<10.42.0.213:44239>; projection={}; filter_private_ads=0 2021-06-17T10:06:48.005168332Z condor_collector[51]: QueryWorker: Child 288 done 2021-06-17T10:06:48.005201696Z condor_schedd[129]: Found 2 potential dedicated resources in 1 seconds 2021-06-17T10:06:48.005231129Z condor_schedd[129]: Duplicate x100 (100/0) partitionable res slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:48.005235485Z condor_schedd[129]: Duplicate x100 (100/0) partitionable res slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:48.005240076Z condor_schedd[129]: idle resource list 2021-06-17T10:06:48.005276432Z condor_schedd[129]: ************ empty ************ 2021-06-17T10:06:48.005322788Z condor_schedd[129]: limbo resource list 2021-06-17T10:06:48.005330147Z condor_schedd[129]: ************ empty ************ 2021-06-17T10:06:48.005334434Z condor_schedd[129]: serial c/i resource list 2021-06-17T10:06:48.005341568Z condor_schedd[129]: ************ empty ************ 2021-06-17T10:06:48.005344011Z condor_schedd[129]: unclaimed resource list 2021-06-17T10:06:48.005352109Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:48.005356467Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ... 2021-06-17T10:06:48.008851060Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:48.008854701Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:48.008858231Z condor_schedd[129]: busy resource list 2021-06-17T10:06:48.008861558Z condor_schedd[129]: ************ empty ************ 2021-06-17T10:06:48.008865217Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.0 2021-06-17T10:06:48.008868272Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.1 2021-06-17T10:06:48.008872030Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.2 2021-06-17T10:06:48.008875422Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.3 2021-06-17T10:06:48.008878614Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.4 2021-06-17T10:06:48.008881825Z condor_schedd[129]: satisfyJobs: testing 5 jobs against 200 slots 2021-06-17T10:06:48.008885100Z condor_schedd[129]: satisfyJobs: finding resources for 2.0 2021-06-17T10:06:48.008889575Z condor_schedd[129]: satisfyJobs: 2.0 satisfied with slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:48.008897032Z condor_schedd[129]: satisfyJobs: finding resources for 2.1 2021-06-17T10:06:48.008900802Z condor_schedd[129]: satisfyJobs: 2.1 satisfied with slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:48.008904817Z condor_schedd[129]: satisfyJobs: finding resources for 2.2 2021-06-17T10:06:48.008908795Z condor_schedd[129]: satisfyJobs: 2.2 satisfied with slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:48.008912864Z condor_schedd[129]: satisfyJobs: finding resources for 2.3 2021-06-17T10:06:48.008916686Z condor_schedd[129]: satisfyJobs: 2.3 satisfied with slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:48.008920773Z condor_schedd[129]: satisfyJobs: finding resources for 2.4 2021-06-17T10:06:48.008924632Z condor_schedd[129]: satisfyJobs: 2.4 satisfied with slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:48.008928723Z condor_schedd[129]: satisfyJobs: jobs were satisfied 2021-06-17T10:06:48.008932414Z condor_schedd[129]: Satisfied job 2 with 5 unclaimed resources 2021-06-17T10:06:48.008935894Z condor_schedd[129]: Generating 5 resource requests for job 2 2021-06-17T10:06:48.008939835Z condor_schedd[129]: Waiting to negotiate for 5 dedicated resource request(s) 2021-06-17T10:06:48.008943252Z condor_schedd[129]: In DedicatedScheduler::publishRequestAd() 2021-06-17T10:06:48.008946592Z condor_schedd[129]: Adding submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor to the submitter map for default pool. 2021-06-17T10:06:48.008950057Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:48.008954275Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:48.008958078Z condor_schedd[129]: Entering DedicatedScheduler::checkSanity() 2021-06-17T10:06:48.008961551Z condor_schedd[129]: Timer (51) will call checkSanity() again in 157 seconds 2021-06-17T10:06:48.009008723Z condor_collector[51]: SubmittorAd : Updating ... "< DedicatedScheduler@parallel_schedd@submit.pseven-htcondorparallel_schedd@xxxxxxxxxxxxxxxxxxxxxx , 10.42.0.213 >" 2021-06-17T10:06:48.009068077Z condor_schedd[129]: Finished DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:06:48.009075214Z condor_schedd[129]: Setting delay until next queue scan to 1 seconds 2021-06-17T10:06:48.016087500Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:48.016655089Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:48.016670817Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:48.016697591Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:48.016809530Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000059; send_time=0.000054; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:43551>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:48.017843059Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:48.018238177Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 452 2021-06-17T10:06:48.018249891Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:48.025427732Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:48.025728629Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:48.026269913Z condor_collector[51]: QueryWorker: forked new worker with id 289 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:48.026281876Z condor_collector[289]: (Sending 2 ads in response to query) 2021-06-17T10:06:48.026825174Z condor_collector[289]: Query info: matched=2; skipped=0; query_time=0.000275; send_time=0.000149; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:36747>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:48.026954042Z condor_collector[51]: QueryWorker: Child 289 done 2021-06-17T10:06:49.020512082Z condor_schedd[129]: Found idle MPI cluster 2 2021-06-17T10:06:49.020542193Z condor_schedd[129]: Started timer (58) to call handleDedicatedJobs() in 0 secs 2021-06-17T10:06:49.020547988Z condor_schedd[129]: JobsRunning = 0 2021-06-17T10:06:49.020561023Z condor_schedd[129]: JobsIdle = 5 2021-06-17T10:06:49.020563696Z condor_schedd[129]: JobsHeld = 0 2021-06-17T10:06:49.020566186Z condor_schedd[129]: JobsRemoved = 0 2021-06-17T10:06:49.020568327Z condor_schedd[129]: LocalUniverseJobsRunning = 0 2021-06-17T10:06:49.020570426Z condor_schedd[129]: LocalUniverseJobsIdle = 0 2021-06-17T10:06:49.020572659Z condor_schedd[129]: SchedUniverseJobsRunning = 0 2021-06-17T10:06:49.020574973Z condor_schedd[129]: SchedUniverseJobsIdle = 0 2021-06-17T10:06:49.020577149Z condor_schedd[129]: NumSubmitters = 1 2021-06-17T10:06:49.020580792Z condor_schedd[129]: MaxJobsRunning = 10000 2021-06-17T10:06:49.020583038Z condor_schedd[129]: MaxRunningSchedulerJobsPerOwner = 200 2021-06-17T10:06:49.020770652Z condor_schedd[129]: TransferQueueManager stats: active up=0/100 down=0/100; waiting up=0 down=0; wait time up=0s down=0s 2021-06-17T10:06:49.020809824Z condor_schedd[129]: TransferQueueManager upload 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load 2021-06-17T10:06:49.020814357Z condor_schedd[129]: TransferQueueManager download 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load 2021-06-17T10:06:49.021473777Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:49.021485349Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:49.021619483Z condor_schedd[129]: Sent HEART BEAT ad to 1 collectors. Number of active submittors=1 2021-06-17T10:06:49.021973090Z condor_collector[51]: ScheddAd : Updating ... "< parallel_schedd@xxxxxxxxxxxxxxxxxxxxxx , 10.42.0.213 >" 2021-06-17T10:06:49.022328204Z condor_collector[51]: SubmittorAd : Updating ... "< user20001@pseven-htcondorparallel_schedd@submit.pseven-htcondor , 10.42.0.213 >" 2021-06-17T10:06:49.022357650Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:49.022371911Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:49.022377010Z condor_schedd[129]: Sent ad to 1 collectors for user20001 Hit=5 Tot=5 Idle=0 Run=0 2021-06-17T10:06:49.022380747Z condor_schedd[129]: ============ Begin clean_shadow_recs ============= 2021-06-17T10:06:49.022384493Z condor_schedd[129]: ============ End clean_shadow_recs ============= 2021-06-17T10:06:49.022388172Z condor_schedd[129]: Sending RESCHEDULE command to negotiator(s) 2021-06-17T10:06:49.023228958Z condor_schedd[129]: Will use TCP to update collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:49.023243928Z condor_schedd[129]: Trying to query collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:49.023351477Z condor_collector[51]: Got QUERY_NEGOTIATOR_ADS 2021-06-17T10:06:49.023363738Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:49.023367540Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:49.023532002Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000024; send_time=0.000137; type=Negotiator; requirements={true}; locate=0; limit=0; from=SCHEDD; peer=<10.42.0.213:38683>; projection={}; filter_private_ads=0 2021-06-17T10:06:49.024245402Z condor_schedd[129]: SharedPortClient: sent connection request to negotiator negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx for shared port id negotiator_13_cf39 2021-06-17T10:06:49.024655461Z condor_schedd[129]: Completed RESCHEDULE to negotiator negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:49.024669236Z condor_schedd[129]: Starting DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:06:49.024673589Z condor_schedd[129]: Delaying scheduling of parallel jobs because startd query time is long (1) seconds 2021-06-17T10:06:49.034129062Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:49.034419137Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:49.034481084Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:49.034489165Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:49.034495830Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000019; send_time=0.000059; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:37993>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:49.035520239Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:49.036223607Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 457 2021-06-17T10:06:49.036240590Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:49.043488188Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:49.043698067Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:49.044245948Z condor_collector[51]: QueryWorker: forked new worker with id 290 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:49.044371557Z condor_collector[290]: (Sending 2 ads in response to query) 2021-06-17T10:06:49.044482479Z condor_collector[290]: Query info: matched=2; skipped=0; query_time=0.000291; send_time=0.000172; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:34371>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:49.044963510Z condor_collector[51]: QueryWorker: Child 290 done 2021-06-17T10:06:50.026256026Z condor_negotiator[52]: ---------- Started Negotiation Cycle ---------- 2021-06-17T10:06:50.026347002Z condor_negotiator[52]: Phase 1: Obtaining ads from collector ... 2021-06-17T10:06:50.026358174Z condor_negotiator[52]: Getting startd private ads ... 2021-06-17T10:06:50.026362837Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:06:50.027352362Z condor_collector[51]: Got QUERY_STARTD_PVT_ADS 2021-06-17T10:06:50.030044192Z condor_collector[51]: QueryWorker: forked new high priority worker with id 291 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:50.030071668Z condor_collector[291]: (Sending 2 ads in response to query) 2021-06-17T10:06:50.030076190Z condor_collector[291]: Query info: matched=2; skipped=0; query_time=0.000442; send_time=0.001464; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:41569>; projection={}; filter_private_ads=0 2021-06-17T10:06:50.030417301Z condor_negotiator[52]: Getting Scheduler, Submitter and Machine ads ... 2021-06-17T10:06:50.030449275Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:06:50.033330712Z condor_collector[51]: Got QUERY_ANY_ADS 2021-06-17T10:06:50.034163854Z condor_collector[51]: QueryWorker: forked new high priority worker with id 292 ( max 4 active 2 pending 0 ) 2021-06-17T10:06:50.034180509Z condor_collector[292]: (Sending 4 ads in response to query) 2021-06-17T10:06:50.034185153Z condor_collector[51]: QueryWorker: Child 291 done 2021-06-17T10:06:50.034961266Z condor_collector[292]: Query info: matched=4; skipped=9; query_time=0.000325; send_time=0.000699; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:34905>; projection={}; filter_private_ads=0 2021-06-17T10:06:50.035209079Z condor_collector[51]: QueryWorker: Child 292 done 2021-06-17T10:06:50.035676398Z condor_negotiator[52]: Sorting 4 ads ... 2021-06-17T10:06:50.035772262Z condor_negotiator[52]: Ignoring submitter user20001@pseven-htcondor from schedd at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> with no requested jobs 2021-06-17T10:06:50.036342957Z condor_negotiator[52]: Got ads: 4 public and 2 private 2021-06-17T10:06:50.036349067Z condor_negotiator[52]: Public ads include 1 submitter, 2 startd 2021-06-17T10:06:50.036433228Z condor_negotiator[52]: Phase 2: Performing accounting ... 2021-06-17T10:06:50.036450401Z condor_negotiator[52]: Entering compute_significant_attrs() 2021-06-17T10:06:50.037241382Z condor_negotiator[52]: Leaving compute_significant_attrs() - result=_cp_orig_RequestCpus,_cp_orig_RequestDisk,_cp_orig_RequestMemory,KFlops,MachineLastMatchTime,Offline,RemoteOwner,RequestCpus,RequestDisk,RequestMemory,Scheduler,TotalJobRuntime 2021-06-17T10:06:50.043594848Z condor_negotiator[52]: Phase 3: Sorting submitter ads by priority ... 2021-06-17T10:06:50.043614852Z condor_negotiator[52]: Starting prefetch round; 1 potential prefetches to do. 2021-06-17T10:06:50.043619569Z condor_negotiator[52]: Assigned 1 units of work for prefetching. 2021-06-17T10:06:50.043622816Z condor_negotiator[52]: Starting prefetch loop. 2021-06-17T10:06:50.043625552Z condor_negotiator[52]: Starting prefetch negotiation for DedicatedScheduler@parallel_schedd@submit.pseven-htcondor. 2021-06-17T10:06:50.043628823Z condor_negotiator[52]: Socket to DedicatedScheduler@parallel_schedd@submit.pseven-htcondor (<10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c>) already in cache, reusing 2021-06-17T10:06:50.045266253Z condor_schedd[129]: Activity on stashed negotiator socket: <10.42.0.205:36513> 2021-06-17T10:06:50.045294140Z condor_schedd[129]: 2021-06-17T10:06:50.045299266Z condor_schedd[129]: Entered negotiate 2021-06-17T10:06:50.045302512Z condor_schedd[129]: Using negotiation protocol: NEGOTIATE 2021-06-17T10:06:50.045348383Z condor_negotiator[52]: Started NEGOTIATE with remote schedd; protocol version 1. 2021-06-17T10:06:50.045360145Z condor_negotiator[52]: Sending SEND_RESOURCE_REQUEST_LIST/200/eom 2021-06-17T10:06:50.045364609Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:50.045368008Z condor_negotiator[52]: Prefetch negotiation would block. 2021-06-17T10:06:50.045371574Z condor_negotiator[52]: Waiting on the results of 1 negotiation sessions. 2021-06-17T10:06:50.045375444Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:50.045378945Z condor_negotiator[52]: Got JOB_INFO command; getting classad/eom 2021-06-17T10:06:50.045690968Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:50.045696343Z condor_negotiator[52]: Assigned 0 units of work for prefetching. 2021-06-17T10:06:50.045699576Z condor_negotiator[52]: Starting prefetch loop. 2021-06-17T10:06:50.045726886Z condor_negotiator[52]: Waiting on the results of 1 negotiation sessions. 2021-06-17T10:06:50.045731059Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:50.045734498Z condor_negotiator[52]: Got JOB_INFO command; getting classad/eom 2021-06-17T10:06:50.045737912Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:50.045741270Z condor_negotiator[52]: Assigned 0 units of work for prefetching. 2021-06-17T10:06:50.045744717Z condor_negotiator[52]: Starting prefetch loop. 2021-06-17T10:06:50.045747904Z condor_negotiator[52]: Waiting on the results of 1 negotiation sessions. 2021-06-17T10:06:50.045754780Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:50.045763650Z condor_negotiator[52]: Got JOB_INFO command; getting classad/eom 2021-06-17T10:06:50.045768332Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:50.045772891Z condor_negotiator[52]: Assigned 0 units of work for prefetching. 2021-06-17T10:06:50.045776363Z condor_negotiator[52]: Starting prefetch loop. 2021-06-17T10:06:50.045779598Z condor_negotiator[52]: Waiting on the results of 1 negotiation sessions. 2021-06-17T10:06:50.045782805Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:50.045786197Z condor_negotiator[52]: Got JOB_INFO command; getting classad/eom 2021-06-17T10:06:50.045789389Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:50.045792802Z condor_negotiator[52]: Got JOB_INFO command; getting classad/eom 2021-06-17T10:06:50.045797974Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:50.045801337Z condor_negotiator[52]: Got NO_MORE_JOBS; schedd has no more requests 2021-06-17T10:06:50.045825466Z condor_schedd[129]: *** SwapSpace = 2147483647 2021-06-17T10:06:50.045829012Z condor_schedd[129]: *** ReservedSwap = 0 2021-06-17T10:06:50.045831884Z condor_schedd[129]: *** Shadow Size Estimate = 125 2021-06-17T10:06:50.045834638Z condor_schedd[129]: *** Start Limit For Swap = 17179869 2021-06-17T10:06:50.045838276Z condor_schedd[129]: Negotiating for owner: DedicatedScheduler@parallel_schedd@submit.pseven-htcondor 2021-06-17T10:06:50.045841911Z condor_schedd[129]: Sent job 2.0 (autocluster=1 resources_requested=1) to the negotiator 2021-06-17T10:06:50.045845163Z condor_schedd[129]: Sent job 2.1 (autocluster=2 resources_requested=1) to the negotiator 2021-06-17T10:06:50.045848575Z condor_schedd[129]: Sent job 2.2 (autocluster=3 resources_requested=1) to the negotiator 2021-06-17T10:06:50.045852121Z condor_schedd[129]: Sent job 2.3 (autocluster=4 resources_requested=1) to the negotiator 2021-06-17T10:06:50.045855513Z condor_schedd[129]: Sent job 2.4 (autocluster=5 resources_requested=1) to the negotiator 2021-06-17T10:06:50.045856178Z condor_negotiator[52]: Send END_NEGOTIATE to remote schedd 2021-06-17T10:06:50.045860057Z condor_schedd[129]: Finished sending rrls to negotiator 2021-06-17T10:06:50.045863826Z condor_schedd[129]: Finished negotiating for DedicatedScheduler@parallel_schedd in local pool: 0 matched, 0 rejected 2021-06-17T10:06:50.046006528Z condor_negotiator[52]: Assigned 0 units of work for prefetching. 2021-06-17T10:06:50.046015906Z condor_negotiator[52]: Prefetch summary: 1 attempted, 1 successful. 2021-06-17T10:06:50.046019688Z condor_negotiator[52]: Phase 4.1: Negotiating with schedds ... 2021-06-17T10:06:50.046033673Z condor_negotiator[52]: numSlots = 2 (after trimming=2) 2021-06-17T10:06:50.046168696Z condor_negotiator[52]: slotWeightTotal = 200.000000 2021-06-17T10:06:50.046388587Z condor_negotiator[52]: minSlotWeight = 100.000000 2021-06-17T10:06:50.046675053Z condor_negotiator[52]: pieLeft = 200.000 2021-06-17T10:06:50.046680176Z condor_negotiator[52]: NormalFactor = 1.000000 2021-06-17T10:06:50.046692649Z condor_negotiator[52]: MaxPrioValue = 503.848022 2021-06-17T10:06:50.046697184Z condor_negotiator[52]: NumSubmitterAds = 1 2021-06-17T10:06:50.046700933Z condor_negotiator[52]: Negotiating with DedicatedScheduler@parallel_schedd@submit.pseven-htcondor at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> 2021-06-17T10:06:50.046755344Z condor_negotiator[52]: 0 seconds so far for this submitter 2021-06-17T10:06:50.046762243Z condor_negotiator[52]: 0 seconds so far for this schedd 2021-06-17T10:06:50.046766483Z condor_negotiator[52]: Calculating submitter limit with the following parameters 2021-06-17T10:06:50.046770005Z condor_negotiator[52]: SubmitterPrio = 503.848022 2021-06-17T10:06:50.046773545Z condor_negotiator[52]: SubmitterPrioFactor = 1000.000000 2021-06-17T10:06:50.046777268Z condor_negotiator[52]: submitterShare = 1.000000 2021-06-17T10:06:50.046780715Z condor_negotiator[52]: submitterAbsShare = 1.000000 2021-06-17T10:06:50.046845522Z condor_negotiator[52]: submitterLimit = 200.000000 2021-06-17T10:06:50.046849701Z condor_negotiator[52]: submitterCeiling remaining = 2147483647 2021-06-17T10:06:50.046853415Z condor_negotiator[52]: submitterUsage = 0.000000 2021-06-17T10:06:50.046856501Z condor_negotiator[52]: Using resource request list from cache. 2021-06-17T10:06:50.046859851Z condor_negotiator[52]: Socket to DedicatedScheduler@parallel_schedd@submit.pseven-htcondor (<10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c>) already in cache, reusing 2021-06-17T10:06:50.046864648Z condor_negotiator[52]: Started NEGOTIATE with remote schedd; protocol version 1. 2021-06-17T10:06:50.046868246Z condor_negotiator[52]: Request 00002.00000: autocluster -1 (request count 1 of 0) 2021-06-17T10:06:50.046871871Z condor_negotiator[52]: matchmakingAlgorithm: limit 200.000000 used 0.000000 pieLeft 200.000000 2021-06-17T10:06:50.046876689Z condor_negotiator[52]: Sending PERMISSION, claim id, startdAd to schedd 2021-06-17T10:06:50.046959722Z condor_negotiator[52]: Matched 2.0 DedicatedScheduler@parallel_schedd@submit.pseven-htcondor <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> preempting none <10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.046990820Z condor_negotiator[52]: Notifying the accountant 2021-06-17T10:06:50.047272169Z condor_schedd[129]: Activity on stashed negotiator socket: <10.42.0.205:36513> 2021-06-17T10:06:50.047285045Z condor_schedd[129]: 2021-06-17T10:06:50.047288425Z condor_schedd[129]: Entered negotiate 2021-06-17T10:06:50.047340998Z condor_schedd[129]: Using negotiation protocol: NEGOTIATE 2021-06-17T10:06:50.047347541Z condor_schedd[129]: *** SwapSpace = 2147483647 2021-06-17T10:06:50.047350959Z condor_schedd[129]: *** ReservedSwap = 0 2021-06-17T10:06:50.047354285Z condor_schedd[129]: *** Shadow Size Estimate = 125 2021-06-17T10:06:50.047393415Z condor_schedd[129]: *** Start Limit For Swap = 17179869 2021-06-17T10:06:50.047448967Z condor_schedd[129]: Negotiating for owner: DedicatedScheduler@parallel_schedd@submit.pseven-htcondor 2021-06-17T10:06:50.049627808Z condor_schedd[129]: DedicatedScheduler: Received match for job 2.0: slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.049647785Z condor_schedd[129]: Enqueued contactStartd startd=<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> 2021-06-17T10:06:50.049716037Z condor_negotiator[52]: Successfully matched with slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.049727055Z condor_negotiator[52]: Match completed, match cost= 25 2021-06-17T10:06:50.049731587Z condor_negotiator[52]: Request 00002.00001: autocluster -1 (request count 1 of 0) 2021-06-17T10:06:50.049735302Z condor_negotiator[52]: matchmakingAlgorithm: limit 200.000000 used 25.000000 pieLeft 175.000000 2021-06-17T10:06:50.049739145Z condor_negotiator[52]: Sending PERMISSION, claim id, startdAd to schedd 2021-06-17T10:06:50.049743235Z condor_negotiator[52]: Matched 2.1 DedicatedScheduler@parallel_schedd@submit.pseven-htcondor <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> preempting none <10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.049756431Z condor_negotiator[52]: Notifying the accountant 2021-06-17T10:06:50.050633944Z condor_negotiator[52]: Successfully matched with slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.050660161Z condor_negotiator[52]: Match completed, match cost= 25 2021-06-17T10:06:50.050670372Z condor_negotiator[52]: Request 00002.00002: autocluster -1 (request count 1 of 0) 2021-06-17T10:06:50.050674753Z condor_negotiator[52]: matchmakingAlgorithm: limit 200.000000 used 50.000000 pieLeft 150.000000 2021-06-17T10:06:50.050678392Z condor_negotiator[52]: Sending PERMISSION, claim id, startdAd to schedd 2021-06-17T10:06:50.050682076Z condor_negotiator[52]: Matched 2.2 DedicatedScheduler@parallel_schedd@submit.pseven-htcondor <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> preempting none <10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.050688655Z condor_negotiator[52]: Notifying the accountant 2021-06-17T10:06:50.052156764Z condor_negotiator[52]: Successfully matched with slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.052179892Z condor_negotiator[52]: Match completed, match cost= 25 2021-06-17T10:06:50.052185354Z condor_negotiator[52]: Request 00002.00003: autocluster -1 (request count 1 of 0) 2021-06-17T10:06:50.052189459Z condor_negotiator[52]: matchmakingAlgorithm: limit 200.000000 used 75.000000 pieLeft 125.000000 2021-06-17T10:06:50.052193293Z condor_negotiator[52]: Sending PERMISSION, claim id, startdAd to schedd 2021-06-17T10:06:50.052196984Z condor_negotiator[52]: Matched 2.3 DedicatedScheduler@parallel_schedd@submit.pseven-htcondor <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> preempting none <10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.052203249Z condor_negotiator[52]: Notifying the accountant 2021-06-17T10:06:50.052311880Z condor_schedd[129]: In checkContactQueue(), args = 0x5628078b2650, host=<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> 2021-06-17T10:06:50.052324486Z condor_schedd[129]: In Scheduler::contactStartd() 2021-06-17T10:06:50.053765946Z condor_negotiator[52]: Successfully matched with slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.053784130Z condor_negotiator[52]: Match completed, match cost= 25 2021-06-17T10:06:50.053788615Z condor_negotiator[52]: Request 00002.00004: autocluster -1 (request count 1 of 0) 2021-06-17T10:06:50.053792009Z condor_negotiator[52]: matchmakingAlgorithm: limit 200.000000 used 100.000000 pieLeft 100.000000 2021-06-17T10:06:50.053805583Z condor_negotiator[52]: Sending PERMISSION, claim id, startdAd to schedd 2021-06-17T10:06:50.053809558Z condor_negotiator[52]: Matched 2.4 DedicatedScheduler@parallel_schedd@submit.pseven-htcondor <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> preempting none <10.42.0.201:32819?addrs=10.42.0.201-32819&alias=pseven-htcondorexecute-deploy-68b66d9c55-nvrwz.pseven-htcondor&noUDP&sock=startd_89_f06c> slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.053814958Z condor_negotiator[52]: Notifying the accountant 2021-06-17T10:06:50.057243782Z condor_negotiator[52]: Successfully matched with slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.057270282Z condor_negotiator[52]: Match completed, match cost= 25 2021-06-17T10:06:50.057275097Z condor_negotiator[52]: Send END_NEGOTIATE to remote schedd 2021-06-17T10:06:50.057278523Z condor_negotiator[52]: Submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor got all it wants; removing it. 2021-06-17T10:06:50.057475028Z condor_negotiator[52]: resources used by DedicatedScheduler@parallel_schedd@submit.pseven-htcondor are 125.000000 2021-06-17T10:06:50.057483736Z condor_negotiator[52]: resources used scheddUsed= 125.000000 2021-06-17T10:06:50.057487565Z condor_negotiator[52]: negotiateWithGroup resources used submitterAds length 0 2021-06-17T10:06:50.057490969Z condor_negotiator[52]: ---------- Finished Negotiation Cycle ---------- 2021-06-17T10:06:50.057494859Z condor_negotiator[52]: Updating collector with accounting information 2021-06-17T10:06:50.057498390Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:06:50.057502410Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:06:50.057506118Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:06:50.057769304Z condor_collector[51]: AccountingAd : Updating ... "< DedicatedScheduler@parallel_schedd@submit.pseven-htcondornegotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:06:50.057779337Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:06:50.057783554Z condor_negotiator[52]: Done Updating collector with accounting information 2021-06-17T10:06:50.057787142Z condor_collector[51]: AccountingAd : Updating ... "< <none>negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:06:50.057819248Z condor_schedd[129]: Partitionable slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx adjusted for job 2.0: cpus = 25, memory = 256, disk = 538687 2021-06-17T10:06:50.057830235Z condor_schedd[129]: SharedPortClient: sent connection request to startd slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3>#1623924088#102#... for DedicatedScheduler@parallel_schedd for shared port id startd_86_e1a3 2021-06-17T10:06:50.057835563Z condor_schedd[129]: DedicatedScheduler: Received match for job 2.1: slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.057839345Z condor_schedd[129]: Enqueued contactStartd startd=<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> 2021-06-17T10:06:50.057843223Z condor_schedd[129]: DedicatedScheduler: Received match for job 2.2: slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.057846918Z condor_schedd[129]: Enqueued contactStartd startd=<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> 2021-06-17T10:06:50.057851173Z condor_schedd[129]: DedicatedScheduler: Received match for job 2.3: slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.058796321Z condor_schedd[129]: Enqueued contactStartd startd=<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> 2021-06-17T10:06:50.058814951Z condor_schedd[129]: DedicatedScheduler: Received match for job 2.4: slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:06:50.060115791Z condor_schedd[129]: Enqueued contactStartd startd=<10.42.0.201:32819?addrs=10.42.0.201-32819&alias=pseven-htcondorexecute-deploy-68b66d9c55-nvrwz.pseven-htcondor&noUDP&sock=startd_89_f06c> 2021-06-17T10:06:50.062573296Z condor_schedd[129]: Negotiation ended - 5 jobs matched 2021-06-17T10:06:50.062597043Z condor_schedd[129]: Finished negotiating for DedicatedScheduler@parallel_schedd in local pool: 5 matched, 0 rejected 2021-06-17T10:06:50.063114240Z condor_startd[122]: slot1: Schedd addr = <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> 2021-06-17T10:06:50.063175702Z condor_startd[122]: slot1: Alive interval = 30 2021-06-17T10:06:50.064799781Z condor_startd[122]: slot1: Received ClaimId from schedd (<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3>#1623924088#102#...) 2021-06-17T10:06:50.064828654Z condor_startd[122]: slot1: Match requesting resources: Cpus=25 disk=0.1% Memory=256 2021-06-17T10:06:50.064834366Z condor_startd[122]: Got execute_dir = /var/lib/condor/execute 2021-06-17T10:06:50.064838412Z condor_startd[122]: slot1_1: bind_DevIds for slot1.1 before : 2021-06-17T10:06:50.064842118Z condor_startd[122]: slot1_1: bind_DevIds for slot1.1 after : 2021-06-17T10:06:50.064845953Z condor_startd[122]: slot1_1: New machine resource of type -1 allocated 2021-06-17T10:06:50.064849373Z condor_startd[122]: slot1_1: Using CP variant of WithinResourceLimits 2021-06-17T10:06:50.064853419Z condor_startd[122]: slot1_1: WithinResourceLimits = (ifThenElse(TARGET._cp_orig_RequestCpus isnt UNDEFINED, TARGET.RequestCpus <= MY.Cpus, MY.ConsumptionCpus <= MY.Cpus) && ifThenElse(TARGET._cp_orig_RequestDisk isnt UNDEFINED, TARGET.RequestDisk <= MY.Disk, MY.ConsumptionDisk <= MY.Disk) && ifThenElse(TARGET._cp_orig_RequestMemory isnt UNDEFINED, TARGET.RequestMemory <= MY.Memory, MY.ConsumptionMemory <= MY.Memory)) 2021-06-17T10:06:50.064875932Z condor_startd[122]: slot1_1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.066576150Z condor_startd[122]: Setting up slot pairings 2021-06-17T10:06:50.066593153Z condor_startd[122]: Swap space: 48075796 2021-06-17T10:06:50.066597547Z condor_startd[122]: slot1_1: Rank of this claim is: 1.000000 2021-06-17T10:06:50.066600988Z condor_startd[122]: slot1_1: Request accepted. 2021-06-17T10:06:50.066668209Z condor_schedd[129]: In checkContactQueue(), args = 0x562807939fb0, host=<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> 2021-06-17T10:06:50.066693508Z condor_schedd[129]: In Scheduler::contactStartd() 2021-06-17T10:06:50.066698028Z condor_schedd[129]: Partitionable slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx adjusted for job 2.1: cpus = 25, memory = 128, disk = 538687 2021-06-17T10:06:50.066701776Z condor_schedd[129]: In checkContactQueue(), args = 0x56280793fe60, host=<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> 2021-06-17T10:06:50.066705890Z condor_schedd[129]: In Scheduler::contactStartd() 2021-06-17T10:06:50.066709005Z condor_schedd[129]: Partitionable slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx adjusted for job 2.2: cpus = 25, memory = 256, disk = 538687 2021-06-17T10:06:50.066712780Z condor_schedd[129]: In checkContactQueue(), args = 0x56280794bb30, host=<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> 2021-06-17T10:06:50.066716965Z condor_schedd[129]: In Scheduler::contactStartd() 2021-06-17T10:06:50.066720067Z condor_schedd[129]: Partitionable slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx adjusted for job 2.3: cpus = 25, memory = 512, disk = 538687 2021-06-17T10:06:50.066723374Z condor_schedd[129]: In checkContactQueue(), args = 0x56280795a390, host=<10.42.0.201:32819?addrs=10.42.0.201-32819&alias=pseven-htcondorexecute-deploy-68b66d9c55-nvrwz.pseven-htcondor&noUDP&sock=startd_89_f06c> 2021-06-17T10:06:50.066727427Z condor_schedd[129]: In Scheduler::contactStartd() 2021-06-17T10:06:50.066730705Z condor_schedd[129]: Partitionable slot slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx adjusted for job 2.4: cpus = 25, memory = 768, disk = 538687 2021-06-17T10:06:50.066734980Z condor_schedd[129]: SharedPortClient: sent connection request to startd slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3>#1623924088#106#... for DedicatedScheduler@parallel_schedd for shared port id startd_86_e1a3 2021-06-17T10:06:50.066739891Z condor_schedd[129]: SharedPortClient: sent connection request to startd slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3>#1623924088#110#... for DedicatedScheduler@parallel_schedd for shared port id startd_86_e1a3 2021-06-17T10:06:50.066744823Z condor_schedd[129]: SharedPortClient: sent connection request to startd slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3>#1623924088#114#... for DedicatedScheduler@parallel_schedd for shared port id startd_86_e1a3 2021-06-17T10:06:50.066749540Z condor_schedd[129]: SharedPortClient: sent connection request to startd slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <10.42.0.201:32819?addrs=10.42.0.201-32819&alias=pseven-htcondorexecute-deploy-68b66d9c55-nvrwz.pseven-htcondor&noUDP&sock=startd_89_f06c>#1623924088#10#... for DedicatedScheduler@parallel_schedd for shared port id startd_89_f06c 2021-06-17T10:06:50.066758638Z condor_schedd[129]: Started timer (61) to call handleDedicatedJobs() in 0 secs 2021-06-17T10:06:50.068503070Z condor_startd[125]: slot1: Schedd addr = <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> 2021-06-17T10:06:50.068503188Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:50.068527487Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:50.068532693Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:50.068532958Z condor_startd[125]: slot1: Alive interval = 30 2021-06-17T10:06:50.068537689Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:50.068538448Z condor_startd[125]: slot1: Received ClaimId from schedd (<10.42.0.201:32819?addrs=10.42.0.201-32819&alias=pseven-htcondorexecute-deploy-68b66d9c55-nvrwz.pseven-htcondor&noUDP&sock=startd_89_f06c>#1623924088#10#...) 2021-06-17T10:06:50.068541533Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000053; send_time=0.000041; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:35653>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:50.068550965Z condor_startd[125]: slot1: Match requesting resources: Cpus=25 disk=0.1% Memory=768 2021-06-17T10:06:50.068554829Z condor_startd[125]: Got execute_dir = /var/lib/condor/execute 2021-06-17T10:06:50.068558501Z condor_startd[125]: slot1_1: bind_DevIds for slot1.1 before : 2021-06-17T10:06:50.068561824Z condor_startd[125]: slot1_1: bind_DevIds for slot1.1 after : 2021-06-17T10:06:50.068571306Z condor_schedd[129]: Starting DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:06:50.068581579Z condor_schedd[129]: Delaying scheduling of parallel jobs because startd query time is long (1) seconds 2021-06-17T10:06:50.068585405Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:50.070956029Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 462 2021-06-17T10:06:50.070962891Z condor_startd[125]: slot1_1: New machine resource of type -1 allocated 2021-06-17T10:06:50.070988593Z condor_startd[125]: slot1_1: Using CP variant of WithinResourceLimits 2021-06-17T10:06:50.070993893Z condor_startd[125]: slot1_1: WithinResourceLimits = (ifThenElse(TARGET._cp_orig_RequestCpus isnt UNDEFINED, TARGET.RequestCpus <= MY.Cpus, MY.ConsumptionCpus <= MY.Cpus) && ifThenElse(TARGET._cp_orig_RequestDisk isnt UNDEFINED, TARGET.RequestDisk <= MY.Disk, MY.ConsumptionDisk <= MY.Disk) && ifThenElse(TARGET._cp_orig_RequestMemory isnt UNDEFINED, TARGET.RequestMemory <= MY.Memory, MY.ConsumptionMemory <= MY.Memory)) 2021-06-17T10:06:50.070995656Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:50.070999602Z condor_startd[125]: slot1_1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.071010853Z condor_startd[125]: Setting up slot pairings 2021-06-17T10:06:50.071014449Z condor_startd[125]: Swap space: 48075796 2021-06-17T10:06:50.071017573Z condor_startd[125]: slot1_1: Rank of this claim is: 1.000000 2021-06-17T10:06:50.071020898Z condor_startd[125]: slot1_1: Request accepted. 2021-06-17T10:06:50.071993620Z condor_schedd[129]: Started timer (62) to call handleDedicatedJobs() in 0 secs 2021-06-17T10:06:50.072002733Z condor_schedd[129]: Starting DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:06:50.072006106Z condor_schedd[129]: Delaying scheduling of parallel jobs because startd query time is long (1) seconds 2021-06-17T10:06:50.082197262Z condor_startd[122]: slot1_1: Can't find hostname of client machine 10.42.0.213 2021-06-17T10:06:50.082197869Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:50.082219583Z condor_startd[122]: slot1_1: Remote owner is user20001@pseven-htcondor 2021-06-17T10:06:50.082224223Z condor_startd[122]: slot1_1: State change: claiming protocol successful 2021-06-17T10:06:50.082250611Z condor_startd[122]: slot1_1: Changing state: Owner -> Claimed 2021-06-17T10:06:50.082254329Z condor_startd[122]: slot1_1: Started ClaimLease timer (63) w/ 180 second lease duration 2021-06-17T10:06:50.082256661Z condor_startd[122]: slot1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.082259616Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.082263385Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.082267118Z condor_startd[122]: slot1: Sent update to 1 collector(s) 2021-06-17T10:06:50.082282305Z condor_collector[51]: StartdAd : Updating ... "< slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:50.082364615Z condor_startd[122]: slot1_1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.082491702Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.082499360Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.082533338Z condor_collector[51]: StartdPvtAd : Updating ... "< slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:50.082585658Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:50.082671901Z condor_startd[122]: slot1_1: Sent update to 1 collector(s) 2021-06-17T10:06:50.084192790Z condor_startd[122]: slot1: Schedd addr = <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> 2021-06-17T10:06:50.084210877Z condor_startd[122]: slot1: Alive interval = 30 2021-06-17T10:06:50.084214680Z condor_startd[122]: slot1: Received ClaimId from schedd (<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3>#1623924088#106#...) 2021-06-17T10:06:50.084219166Z condor_startd[122]: slot1: Match requesting resources: Cpus=25 disk=0.1% Memory=128 2021-06-17T10:06:50.084222642Z condor_startd[122]: Got execute_dir = /var/lib/condor/execute 2021-06-17T10:06:50.084225980Z condor_startd[122]: slot1_2: bind_DevIds for slot1.2 before : 2021-06-17T10:06:50.084237444Z condor_startd[122]: slot1_2: bind_DevIds for slot1.2 after : 2021-06-17T10:06:50.084240952Z condor_startd[122]: slot1_2: New machine resource of type -1 allocated 2021-06-17T10:06:50.084244332Z condor_startd[122]: slot1_2: Using CP variant of WithinResourceLimits 2021-06-17T10:06:50.084247776Z condor_startd[122]: slot1_2: WithinResourceLimits = (ifThenElse(TARGET._cp_orig_RequestCpus isnt UNDEFINED, TARGET.RequestCpus <= MY.Cpus, MY.ConsumptionCpus <= MY.Cpus) && ifThenElse(TARGET._cp_orig_RequestDisk isnt UNDEFINED, TARGET.RequestDisk <= MY.Disk, MY.ConsumptionDisk <= MY.Disk) && ifThenElse(TARGET._cp_orig_RequestMemory isnt UNDEFINED, TARGET.RequestMemory <= MY.Memory, MY.ConsumptionMemory <= MY.Memory)) 2021-06-17T10:06:50.084252787Z condor_startd[122]: slot1_2: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.085463594Z condor_collector[51]: QueryWorker: forked new worker with id 293 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:50.085487952Z condor_collector[293]: (Sending 2 ads in response to query) 2021-06-17T10:06:50.085492351Z condor_collector[293]: Query info: matched=2; skipped=0; query_time=0.000367; send_time=0.000192; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:40809>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:50.085496968Z condor_collector[51]: StartdAd : Inserting ** "< slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:50.085501003Z condor_collector[51]: StartdPvtAd : Inserting ** "< slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:50.085504855Z condor_collector[51]: QueryWorker: Child 293 done 2021-06-17T10:06:50.085519462Z condor_startd[122]: Setting up slot pairings 2021-06-17T10:06:50.085526913Z condor_startd[122]: Swap space: 48075796 2021-06-17T10:06:50.085576003Z condor_startd[122]: slot1_2: Rank of this claim is: 1.000000 2021-06-17T10:06:50.085582400Z condor_startd[122]: slot1_2: Request accepted. 2021-06-17T10:06:50.085774007Z condor_schedd[129]: Started timer (63) to call handleDedicatedJobs() in 0 secs 2021-06-17T10:06:50.086380003Z condor_schedd[129]: Starting DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:06:50.086713809Z condor_schedd[129]: Delaying scheduling of parallel jobs because startd query time is long (1) seconds 2021-06-17T10:06:50.086738896Z condor_startd[125]: slot1_1: Can't find hostname of client machine 10.42.0.213 2021-06-17T10:06:50.086756860Z condor_startd[125]: slot1_1: Remote owner is user20001@pseven-htcondor 2021-06-17T10:06:50.086761169Z condor_startd[125]: slot1_1: State change: claiming protocol successful 2021-06-17T10:06:50.086764733Z condor_startd[125]: slot1_1: Changing state: Owner -> Claimed 2021-06-17T10:06:50.086768323Z condor_startd[125]: slot1_1: Started ClaimLease timer (41) w/ 180 second lease duration 2021-06-17T10:06:50.086771781Z condor_startd[125]: slot1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.089250695Z condor_collector[51]: StartdAd : Updating ... "< slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.201 >" 2021-06-17T10:06:50.089283674Z condor_collector[51]: StartdPvtAd : Updating ... "< slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.201 >" 2021-06-17T10:06:50.089460838Z condor_startd[125]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.089474478Z condor_startd[125]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.089479442Z condor_startd[125]: slot1: Sent update to 1 collector(s) 2021-06-17T10:06:50.089483599Z condor_startd[125]: slot1_1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.089486484Z condor_startd[125]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.089490239Z condor_startd[125]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.089505549Z condor_startd[125]: slot1_1: Sent update to 1 collector(s) 2021-06-17T10:06:50.089520712Z condor_collector[51]: StartdAd : Inserting ** "< slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.201 >" 2021-06-17T10:06:50.089538295Z condor_collector[51]: StartdPvtAd : Inserting ** "< slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.201 >" 2021-06-17T10:06:50.100771813Z condor_startd[122]: slot1_2: Can't find hostname of client machine 10.42.0.213 2021-06-17T10:06:50.100796488Z condor_startd[122]: slot1_2: Remote owner is user20001@pseven-htcondor 2021-06-17T10:06:50.100801006Z condor_startd[122]: slot1_2: State change: claiming protocol successful 2021-06-17T10:06:50.100804325Z condor_startd[122]: slot1_2: Changing state: Owner -> Claimed 2021-06-17T10:06:50.100807682Z condor_startd[122]: slot1_2: Started ClaimLease timer (67) w/ 180 second lease duration 2021-06-17T10:06:50.101422622Z condor_startd[122]: slot1: Schedd addr = <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> 2021-06-17T10:06:50.101434904Z condor_startd[122]: slot1: Alive interval = 30 2021-06-17T10:06:50.101439063Z condor_startd[122]: slot1: Received ClaimId from schedd (<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3>#1623924088#110#...) 2021-06-17T10:06:50.101547754Z condor_startd[122]: slot1: Match requesting resources: Cpus=25 disk=0.1% Memory=256 2021-06-17T10:06:50.101571163Z condor_startd[122]: Got execute_dir = /var/lib/condor/execute 2021-06-17T10:06:50.101584248Z condor_startd[122]: slot1_3: bind_DevIds for slot1.3 before : 2021-06-17T10:06:50.101588543Z condor_startd[122]: slot1_3: bind_DevIds for slot1.3 after : 2021-06-17T10:06:50.101791784Z condor_startd[122]: slot1_3: New machine resource of type -1 allocated 2021-06-17T10:06:50.101803535Z condor_startd[122]: slot1_3: Using CP variant of WithinResourceLimits 2021-06-17T10:06:50.101817176Z condor_startd[122]: slot1_3: WithinResourceLimits = (ifThenElse(TARGET._cp_orig_RequestCpus isnt UNDEFINED, TARGET.RequestCpus <= MY.Cpus, MY.ConsumptionCpus <= MY.Cpus) && ifThenElse(TARGET._cp_orig_RequestDisk isnt UNDEFINED, TARGET.RequestDisk <= MY.Disk, MY.ConsumptionDisk <= MY.Disk) && ifThenElse(TARGET._cp_orig_RequestMemory isnt UNDEFINED, TARGET.RequestMemory <= MY.Memory, MY.ConsumptionMemory <= MY.Memory)) 2021-06-17T10:06:50.101955203Z condor_startd[122]: slot1_3: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.102713185Z condor_startd[122]: Setting up slot pairings 2021-06-17T10:06:50.102729417Z condor_startd[122]: Swap space: 48075796 2021-06-17T10:06:50.103014367Z condor_startd[122]: slot1_3: Rank of this claim is: 1.000000 2021-06-17T10:06:50.103025782Z condor_startd[122]: slot1_3: Request accepted. 2021-06-17T10:06:50.103336573Z condor_schedd[129]: Started timer (64) to call handleDedicatedJobs() in 0 secs 2021-06-17T10:06:50.103351051Z condor_schedd[129]: Starting DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:06:50.103355238Z condor_schedd[129]: Delaying scheduling of parallel jobs because startd query time is long (1) seconds 2021-06-17T10:06:50.118424468Z condor_startd[122]: slot1_3: Can't find hostname of client machine 10.42.0.213 2021-06-17T10:06:50.118452604Z condor_startd[122]: slot1_3: Remote owner is user20001@pseven-htcondor 2021-06-17T10:06:50.118467901Z condor_startd[122]: slot1_3: State change: claiming protocol successful 2021-06-17T10:06:50.118472562Z condor_startd[122]: slot1_3: Changing state: Owner -> Claimed 2021-06-17T10:06:50.118476515Z condor_startd[122]: slot1_3: Started ClaimLease timer (70) w/ 180 second lease duration 2021-06-17T10:06:50.119164305Z condor_startd[122]: slot1: Schedd addr = <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> 2021-06-17T10:06:50.119179711Z condor_startd[122]: slot1: Alive interval = 30 2021-06-17T10:06:50.119184569Z condor_startd[122]: slot1: Received ClaimId from schedd (<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3>#1623924088#114#...) 2021-06-17T10:06:50.119298394Z condor_startd[122]: slot1: Match requesting resources: Cpus=25 disk=0.1% Memory=512 2021-06-17T10:06:50.119323459Z condor_startd[122]: Got execute_dir = /var/lib/condor/execute 2021-06-17T10:06:50.119328806Z condor_startd[122]: slot1_4: bind_DevIds for slot1.4 before : 2021-06-17T10:06:50.119332484Z condor_startd[122]: slot1_4: bind_DevIds for slot1.4 after : 2021-06-17T10:06:50.119535512Z condor_startd[122]: slot1_4: New machine resource of type -1 allocated 2021-06-17T10:06:50.119544635Z condor_startd[122]: slot1_4: Using CP variant of WithinResourceLimits 2021-06-17T10:06:50.119549049Z condor_startd[122]: slot1_4: WithinResourceLimits = (ifThenElse(TARGET._cp_orig_RequestCpus isnt UNDEFINED, TARGET.RequestCpus <= MY.Cpus, MY.ConsumptionCpus <= MY.Cpus) && ifThenElse(TARGET._cp_orig_RequestDisk isnt UNDEFINED, TARGET.RequestDisk <= MY.Disk, MY.ConsumptionDisk <= MY.Disk) && ifThenElse(TARGET._cp_orig_RequestMemory isnt UNDEFINED, TARGET.RequestMemory <= MY.Memory, MY.ConsumptionMemory <= MY.Memory)) 2021-06-17T10:06:50.119690954Z condor_startd[122]: slot1_4: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.121012065Z condor_schedd[129]: Started timer (65) to call handleDedicatedJobs() in 0 secs 2021-06-17T10:06:50.121034180Z condor_startd[122]: Setting up slot pairings 2021-06-17T10:06:50.121039935Z condor_schedd[129]: Starting DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:06:50.121045474Z condor_schedd[129]: Delaying scheduling of parallel jobs because startd query time is long (1) seconds 2021-06-17T10:06:50.121048087Z condor_startd[122]: Swap space: 48075796 2021-06-17T10:06:50.121053216Z condor_startd[122]: slot1_4: Rank of this claim is: 1.000000 2021-06-17T10:06:50.121487715Z condor_startd[122]: slot1_4: Request accepted. 2021-06-17T10:06:50.135554019Z condor_startd[122]: slot1_4: Can't find hostname of client machine 10.42.0.213 2021-06-17T10:06:50.135675580Z condor_startd[122]: slot1_4: Remote owner is user20001@pseven-htcondor 2021-06-17T10:06:50.135683281Z condor_startd[122]: slot1_4: State change: claiming protocol successful 2021-06-17T10:06:50.135687200Z condor_startd[122]: slot1_4: Changing state: Owner -> Claimed 2021-06-17T10:06:50.135690673Z condor_startd[122]: slot1_4: Started ClaimLease timer (73) w/ 180 second lease duration 2021-06-17T10:06:50.137047577Z condor_startd[122]: slot1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.137074136Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.137079181Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.137131928Z condor_startd[122]: slot1: Sent update to 1 collector(s) 2021-06-17T10:06:50.137364889Z condor_startd[122]: slot1_2: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.137684882Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.137696526Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.137790528Z condor_startd[122]: slot1_2: Sent update to 1 collector(s) 2021-06-17T10:06:50.137985022Z condor_startd[122]: slot1_3: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.138237242Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.138252677Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.138564871Z condor_startd[122]: slot1_3: Sent update to 1 collector(s) 2021-06-17T10:06:50.138791434Z condor_startd[122]: slot1_4: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:06:50.138959710Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.138966509Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:06:50.139151692Z condor_startd[122]: slot1_4: Sent update to 1 collector(s) 2021-06-17T10:06:50.141342423Z condor_collector[51]: StartdAd : Updating ... "< slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:50.141417959Z condor_collector[51]: StartdPvtAd : Updating ... "< slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:50.141424200Z condor_collector[51]: StartdAd : Inserting ** "< slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:50.141428176Z condor_collector[51]: StartdPvtAd : Inserting ** "< slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:50.141432670Z condor_collector[51]: StartdAd : Inserting ** "< slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:50.141465142Z condor_collector[51]: StartdPvtAd : Inserting ** "< slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:50.141469416Z condor_collector[51]: StartdAd : Inserting ** "< slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:50.141473835Z condor_collector[51]: StartdPvtAd : Inserting ** "< slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:06:51.096796069Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:51.097136216Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:51.097160423Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:51.097239696Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:51.097251362Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000040; send_time=0.000077; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:40303>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:51.098402720Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:51.099055092Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 467 2021-06-17T10:06:51.099069761Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:51.107214058Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:51.108070390Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:51.108087944Z condor_collector[51]: QueryWorker: forked new worker with id 294 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:51.108165197Z condor_collector[294]: (Sending 7 ads in response to query) 2021-06-17T10:06:51.108369549Z condor_collector[294]: Query info: matched=7; skipped=0; query_time=0.000315; send_time=0.000190; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:37541>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:51.108956838Z condor_collector[51]: QueryWorker: Child 294 done 2021-06-17T10:06:52.121173924Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:52.121860581Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:52.121905298Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:52.121910896Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:52.121915151Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000064; send_time=0.000051; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:33677>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:52.124747931Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:52.124777072Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 472 2021-06-17T10:06:52.124783606Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:52.145596356Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:52.145644660Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:52.147488117Z condor_collector[51]: QueryWorker: forked new worker with id 295 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:52.147563877Z condor_collector[295]: (Sending 7 ads in response to query) 2021-06-17T10:06:52.147569694Z condor_collector[295]: Query info: matched=7; skipped=0; query_time=0.000310; send_time=0.000197; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:34735>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:52.147574039Z condor_collector[51]: QueryWorker: Child 295 done 2021-06-17T10:06:53.158731444Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:53.159049786Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:53.159079946Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:53.159116323Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:53.159594886Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000030; send_time=0.000044; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:39095>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:53.160266167Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:53.160885169Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 477 2021-06-17T10:06:53.161074834Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:53.169018287Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:53.169243337Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:53.170117645Z condor_collector[51]: QueryWorker: forked new worker with id 296 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:53.170193133Z condor_collector[296]: (Sending 7 ads in response to query) 2021-06-17T10:06:53.170392246Z condor_collector[296]: Query info: matched=7; skipped=0; query_time=0.000260; send_time=0.000168; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:36117>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:53.170833305Z condor_collector[51]: QueryWorker: Child 296 done 2021-06-17T10:06:54.178660797Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:54.178973794Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:54.178986454Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:54.178991752Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:54.179071184Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000039; send_time=0.000042; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:40965>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:54.180165747Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:54.180962250Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 482 2021-06-17T10:06:54.180978677Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:54.190434853Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:54.190762456Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:54.191306552Z condor_collector[51]: QueryWorker: forked new worker with id 297 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:54.191386478Z condor_collector[297]: (Sending 7 ads in response to query) 2021-06-17T10:06:54.191569044Z condor_collector[297]: Query info: matched=7; skipped=0; query_time=0.000300; send_time=0.000158; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:45303>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:54.191935963Z condor_collector[51]: QueryWorker: Child 297 done 2021-06-17T10:06:55.199950449Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:55.200181366Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:55.200192823Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:55.200205024Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:55.200298746Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000028; send_time=0.000042; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:35337>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:55.201685637Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:55.202492792Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 487 2021-06-17T10:06:55.202516561Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:55.209706497Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:55.209933062Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:55.210408145Z condor_collector[51]: QueryWorker: forked new worker with id 298 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:55.210528037Z condor_collector[298]: (Sending 7 ads in response to query) 2021-06-17T10:06:55.210745951Z condor_collector[298]: Query info: matched=7; skipped=0; query_time=0.000264; send_time=0.000185; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:35229>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:55.211223041Z condor_collector[51]: QueryWorker: Child 298 done 2021-06-17T10:06:56.060740098Z condor_negotiator[52]: ---------- Started Negotiation Cycle ---------- 2021-06-17T10:06:56.060818134Z condor_negotiator[52]: Phase 1: Obtaining ads from collector ... 2021-06-17T10:06:56.060842226Z condor_negotiator[52]: Getting startd private ads ... 2021-06-17T10:06:56.060859401Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:06:56.062632098Z condor_collector[51]: Got QUERY_STARTD_PVT_ADS 2021-06-17T10:06:56.064073877Z condor_collector[51]: QueryWorker: forked new high priority worker with id 299 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:56.064307197Z condor_collector[299]: (Sending 7 ads in response to query) 2021-06-17T10:06:56.068068412Z condor_collector[299]: Query info: matched=7; skipped=0; query_time=0.000879; send_time=0.003586; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:44449>; projection={}; filter_private_ads=0 2021-06-17T10:06:56.069709555Z condor_collector[51]: QueryWorker: Child 299 done 2021-06-17T10:06:56.070622340Z condor_negotiator[52]: Getting Scheduler, Submitter and Machine ads ... 2021-06-17T10:06:56.070696099Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:06:56.072618758Z condor_collector[51]: Got QUERY_ANY_ADS 2021-06-17T10:06:56.073774822Z condor_collector[51]: QueryWorker: forked new high priority worker with id 300 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:56.074358994Z condor_collector[300]: (Sending 9 ads in response to query) 2021-06-17T10:06:56.078547374Z condor_collector[300]: Query info: matched=9; skipped=9; query_time=0.000946; send_time=0.003406; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:36237>; projection={}; filter_private_ads=0 2021-06-17T10:06:56.079198630Z condor_collector[51]: QueryWorker: Child 300 done 2021-06-17T10:06:56.081839970Z condor_negotiator[52]: Sorting 9 ads ... 2021-06-17T10:06:56.082304064Z condor_negotiator[52]: Ignoring submitter user20001@pseven-htcondor from schedd at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> with no requested jobs 2021-06-17T10:06:56.084257213Z condor_negotiator[52]: Got ads: 9 public and 7 private 2021-06-17T10:06:56.084305782Z condor_negotiator[52]: Public ads include 1 submitter, 7 startd 2021-06-17T10:06:56.084327170Z condor_negotiator[52]: Phase 2: Performing accounting ... 2021-06-17T10:06:56.084345227Z condor_negotiator[52]: Entering compute_significant_attrs() 2021-06-17T10:06:56.085571338Z condor_negotiator[52]: Leaving compute_significant_attrs() - result=_cp_orig_RequestCpus,_cp_orig_RequestDisk,_cp_orig_RequestMemory,KFlops,MachineLastMatchTime,Offline,RequestCpus,RequestDisk,RequestMemory,Scheduler,TotalJobRuntime 2021-06-17T10:06:56.093284744Z condor_negotiator[52]: Phase 3: Sorting submitter ads by priority ... 2021-06-17T10:06:56.093323690Z condor_negotiator[52]: Starting prefetch round; 1 potential prefetches to do. 2021-06-17T10:06:56.093339353Z condor_negotiator[52]: Assigned 1 units of work for prefetching. 2021-06-17T10:06:56.093351474Z condor_negotiator[52]: Starting prefetch loop. 2021-06-17T10:06:56.093365160Z condor_negotiator[52]: Starting prefetch negotiation for DedicatedScheduler@parallel_schedd@submit.pseven-htcondor. 2021-06-17T10:06:56.093373606Z condor_negotiator[52]: Socket to DedicatedScheduler@parallel_schedd@submit.pseven-htcondor (<10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c>) already in cache, reusing 2021-06-17T10:06:56.093618692Z condor_negotiator[52]: Started NEGOTIATE with remote schedd; protocol version 1. 2021-06-17T10:06:56.093638530Z condor_negotiator[52]: Sending SEND_RESOURCE_REQUEST_LIST/200/eom 2021-06-17T10:06:56.093718167Z condor_schedd[129]: Activity on stashed negotiator socket: <10.42.0.205:36513> 2021-06-17T10:06:56.093760300Z condor_schedd[129]: 2021-06-17T10:06:56.093830472Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:56.093849027Z condor_negotiator[52]: Prefetch negotiation would block. 2021-06-17T10:06:56.093860955Z condor_negotiator[52]: Waiting on the results of 1 negotiation sessions. 2021-06-17T10:06:56.093881316Z condor_schedd[129]: Entered negotiate 2021-06-17T10:06:56.093895153Z condor_schedd[129]: Using negotiation protocol: NEGOTIATE 2021-06-17T10:06:56.093905927Z condor_schedd[129]: *** SwapSpace = 2147483647 2021-06-17T10:06:56.093914759Z condor_schedd[129]: *** ReservedSwap = 0 2021-06-17T10:06:56.093922866Z condor_schedd[129]: *** Shadow Size Estimate = 125 2021-06-17T10:06:56.093929439Z condor_schedd[129]: *** Start Limit For Swap = 17179869 2021-06-17T10:06:56.094085020Z condor_schedd[129]: Negotiating for owner: DedicatedScheduler@parallel_schedd@submit.pseven-htcondor 2021-06-17T10:06:56.094313905Z condor_schedd[129]: Finished negotiating for DedicatedScheduler@parallel_schedd in local pool: 0 matched, 0 rejected 2021-06-17T10:06:56.095214674Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:06:56.095310917Z condor_negotiator[52]: Got NO_MORE_JOBS; schedd has no more requests 2021-06-17T10:06:56.095325988Z condor_negotiator[52]: Assigned 0 units of work for prefetching. 2021-06-17T10:06:56.095337783Z condor_negotiator[52]: Prefetch summary: 1 attempted, 1 successful. 2021-06-17T10:06:56.095349974Z condor_negotiator[52]: Phase 4.1: Negotiating with schedds ... 2021-06-17T10:06:56.095360817Z condor_negotiator[52]: numSlots = 7 (after trimming=7) 2021-06-17T10:06:56.095370901Z condor_negotiator[52]: slotWeightTotal = 200.000000 2021-06-17T10:06:56.095377913Z condor_negotiator[52]: minSlotWeight = 0.000000 2021-06-17T10:06:56.095384444Z condor_negotiator[52]: pieLeft = 200.000 2021-06-17T10:06:56.095390897Z condor_negotiator[52]: NormalFactor = 1.000000 2021-06-17T10:06:56.095397410Z condor_negotiator[52]: MaxPrioValue = 509.843994 2021-06-17T10:06:56.095403987Z condor_negotiator[52]: NumSubmitterAds = 1 2021-06-17T10:06:56.095410622Z condor_negotiator[52]: Negotiating with DedicatedScheduler@parallel_schedd@submit.pseven-htcondor at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> 2021-06-17T10:06:56.095423142Z condor_negotiator[52]: 0 seconds so far for this submitter 2021-06-17T10:06:56.095434701Z condor_negotiator[52]: 0 seconds so far for this schedd 2021-06-17T10:06:56.095447673Z condor_negotiator[52]: Calculating submitter limit with the following parameters 2021-06-17T10:06:56.095458367Z condor_negotiator[52]: SubmitterPrio = 509.843994 2021-06-17T10:06:56.095465197Z condor_negotiator[52]: SubmitterPrioFactor = 1000.000000 2021-06-17T10:06:56.095471756Z condor_negotiator[52]: submitterShare = 1.000000 2021-06-17T10:06:56.095478313Z condor_negotiator[52]: submitterAbsShare = 1.000000 2021-06-17T10:06:56.095484928Z condor_negotiator[52]: submitterLimit = 200.000000 2021-06-17T10:06:56.095491613Z condor_negotiator[52]: submitterCeiling remaining = 2147483647 2021-06-17T10:06:56.095523242Z condor_negotiator[52]: submitterUsage = 0.000000 2021-06-17T10:06:56.095535747Z condor_negotiator[52]: Using resource request list from cache. 2021-06-17T10:06:56.095554907Z condor_negotiator[52]: Socket to DedicatedScheduler@parallel_schedd@submit.pseven-htcondor (<10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c>) already in cache, reusing 2021-06-17T10:06:56.096417866Z condor_negotiator[52]: Started NEGOTIATE with remote schedd; protocol version 1. 2021-06-17T10:06:56.096445233Z condor_negotiator[52]: Send END_NEGOTIATE to remote schedd 2021-06-17T10:06:56.096502921Z condor_schedd[129]: Activity on stashed negotiator socket: <10.42.0.205:36513> 2021-06-17T10:06:56.096825500Z condor_negotiator[52]: Submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor got all it wants; removing it. 2021-06-17T10:06:56.096852692Z condor_negotiator[52]: resources used by DedicatedScheduler@parallel_schedd@submit.pseven-htcondor are 0.000000 2021-06-17T10:06:56.096895567Z condor_negotiator[52]: resources used scheddUsed= 0.000000 2021-06-17T10:06:56.096904661Z condor_negotiator[52]: negotiateWithGroup resources used submitterAds length 0 2021-06-17T10:06:56.096913400Z condor_negotiator[52]: ---------- Finished Negotiation Cycle ---------- 2021-06-17T10:06:56.096920333Z condor_negotiator[52]: Updating collector with accounting information 2021-06-17T10:06:56.096952088Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:06:56.096965242Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:06:56.096973753Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:06:56.096979204Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:06:56.096984630Z condor_collector[51]: AccountingAd : Updating ... "< DedicatedScheduler@parallel_schedd@submit.pseven-htcondornegotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:06:56.096990888Z condor_negotiator[52]: Done Updating collector with accounting information 2021-06-17T10:06:56.097219712Z condor_schedd[129]: 2021-06-17T10:06:56.097235853Z condor_schedd[129]: Entered negotiate 2021-06-17T10:06:56.097244899Z condor_schedd[129]: Using negotiation protocol: NEGOTIATE 2021-06-17T10:06:56.097253036Z condor_schedd[129]: *** SwapSpace = 2147483647 2021-06-17T10:06:56.097260465Z condor_schedd[129]: *** ReservedSwap = 0 2021-06-17T10:06:56.097268119Z condor_schedd[129]: *** Shadow Size Estimate = 125 2021-06-17T10:06:56.097275845Z condor_schedd[129]: *** Start Limit For Swap = 17179869 2021-06-17T10:06:56.097284004Z condor_schedd[129]: Negotiating for owner: DedicatedScheduler@parallel_schedd@submit.pseven-htcondor 2021-06-17T10:06:56.097291646Z condor_schedd[129]: Finished sending rrls to negotiator 2021-06-17T10:06:56.097298004Z condor_schedd[129]: Finished negotiating for DedicatedScheduler@parallel_schedd in local pool: 0 matched, 0 rejected 2021-06-17T10:06:56.097361564Z condor_collector[51]: AccountingAd : Updating ... "< <none>negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:06:56.235601020Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:56.236886676Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:56.236946540Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:56.236973305Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:56.237393183Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000221; send_time=0.000187; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:36635>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:56.241405288Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:56.243485868Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 492 2021-06-17T10:06:56.243632803Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:56.262978211Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:56.263375363Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:56.264951262Z condor_collector[51]: QueryWorker: forked new worker with id 301 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:56.264976081Z condor_collector[301]: (Sending 7 ads in response to query) 2021-06-17T10:06:56.264984261Z condor_collector[301]: Query info: matched=7; skipped=0; query_time=0.000423; send_time=0.000288; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:43515>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:56.265250234Z condor_collector[51]: QueryWorker: Child 301 done 2021-06-17T10:06:57.272494891Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:57.272790539Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:57.272827198Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:57.272831101Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:57.272849826Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000024; send_time=0.000037; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:45567>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:57.273869897Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:57.274388911Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 497 2021-06-17T10:06:57.274418705Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:57.281361512Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:57.281690493Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:57.282111945Z condor_collector[51]: QueryWorker: forked new worker with id 302 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:57.282168518Z condor_collector[302]: (Sending 7 ads in response to query) 2021-06-17T10:06:57.282388593Z condor_collector[302]: Query info: matched=7; skipped=0; query_time=0.000267; send_time=0.000162; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:38439>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:57.282778254Z condor_collector[51]: QueryWorker: Child 302 done 2021-06-17T10:06:58.307019260Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:58.308275496Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:58.308339726Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:58.308598679Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:58.308645297Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000119; send_time=0.000205; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:43479>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:58.312520325Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:58.314218867Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 502 2021-06-17T10:06:58.314472486Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:58.333125448Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:58.333517097Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:58.334135361Z condor_collector[51]: QueryWorker: forked new worker with id 303 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:58.334315182Z condor_collector[303]: (Sending 7 ads in response to query) 2021-06-17T10:06:58.334616705Z condor_collector[303]: Query info: matched=7; skipped=0; query_time=0.000471; send_time=0.000263; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:33491>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:58.335189851Z condor_collector[51]: QueryWorker: Child 303 done 2021-06-17T10:06:59.369675653Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:59.370659526Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:06:59.370750180Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:06:59.372063994Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:06:59.372086488Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000123; send_time=0.000183; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:41151>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:06:59.374130739Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:59.375415378Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 507 2021-06-17T10:06:59.375459555Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:06:59.389079982Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:06:59.389603233Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:06:59.390161957Z condor_collector[51]: QueryWorker: forked new worker with id 304 ( max 4 active 1 pending 0 ) 2021-06-17T10:06:59.390321398Z condor_collector[304]: (Sending 7 ads in response to query) 2021-06-17T10:06:59.391198790Z condor_collector[304]: Query info: matched=7; skipped=0; query_time=0.000446; send_time=0.000237; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:42505>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:06:59.391215602Z condor_collector[51]: QueryWorker: Child 304 done 2021-06-17T10:07:00.100495741Z condor_negotiator[52]: enter Matchmaker::updateCollector 2021-06-17T10:07:00.101724195Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:07:00.101744897Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:07:00.101900018Z condor_negotiator[52]: exit Matchmaker::UpdateCollector 2021-06-17T10:07:00.102142462Z condor_collector[51]: NegotiatorAd : Updating ... "< negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:07:00.102541460Z condor_collector[51]: CollectorAd : Updating ... "< collector@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:07:00.401219531Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:00.401512575Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:00.401527452Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:00.401540817Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:00.401641694Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000052; send_time=0.000040; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:36795>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:00.402842677Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:00.404286076Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 512 2021-06-17T10:07:00.404330510Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:00.413454757Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:00.413784629Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:00.414340993Z condor_collector[51]: QueryWorker: forked new worker with id 312 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:00.414378507Z condor_collector[312]: (Sending 7 ads in response to query) 2021-06-17T10:07:00.414776602Z condor_collector[312]: Query info: matched=7; skipped=0; query_time=0.000325; send_time=0.000170; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:44293>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:00.415123944Z condor_collector[51]: QueryWorker: Child 312 done 2021-06-17T10:07:01.422627891Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:01.423135326Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:01.423145252Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:01.423149574Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:01.423152928Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000029; send_time=0.000038; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:40181>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:01.424302421Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:01.425556856Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 517 2021-06-17T10:07:01.425577639Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:01.433918142Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:01.434173106Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:01.434735334Z condor_collector[51]: QueryWorker: forked new worker with id 313 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:01.434747698Z condor_collector[313]: (Sending 7 ads in response to query) 2021-06-17T10:07:01.434956047Z condor_collector[313]: Query info: matched=7; skipped=0; query_time=0.000272; send_time=0.000176; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:42351>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:01.435443383Z condor_collector[51]: QueryWorker: Child 313 done 2021-06-17T10:07:02.103442717Z condor_negotiator[52]: ---------- Started Negotiation Cycle ---------- 2021-06-17T10:07:02.103502640Z condor_negotiator[52]: Phase 1: Obtaining ads from collector ... 2021-06-17T10:07:02.103522481Z condor_negotiator[52]: Getting startd private ads ... 2021-06-17T10:07:02.103526136Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:07:02.104092885Z condor_collector[51]: Got QUERY_STARTD_PVT_ADS 2021-06-17T10:07:02.104887862Z condor_collector[51]: QueryWorker: forked new high priority worker with id 314 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:02.104900322Z condor_collector[314]: (Sending 7 ads in response to query) 2021-06-17T10:07:02.106136858Z condor_collector[314]: Query info: matched=7; skipped=0; query_time=0.000322; send_time=0.001183; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:40743>; projection={}; filter_private_ads=0 2021-06-17T10:07:02.106787776Z condor_collector[51]: QueryWorker: Child 314 done 2021-06-17T10:07:02.106801952Z condor_negotiator[52]: Getting Scheduler, Submitter and Machine ads ... 2021-06-17T10:07:02.106816262Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:07:02.107374414Z condor_collector[51]: Got QUERY_ANY_ADS 2021-06-17T10:07:02.107853969Z condor_collector[51]: QueryWorker: forked new high priority worker with id 315 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:02.107866900Z condor_collector[315]: (Sending 9 ads in response to query) 2021-06-17T10:07:02.109401306Z condor_collector[315]: Query info: matched=9; skipped=9; query_time=0.000250; send_time=0.001054; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:39117>; projection={}; filter_private_ads=0 2021-06-17T10:07:02.110646249Z condor_collector[51]: QueryWorker: Child 315 done 2021-06-17T10:07:02.110660562Z condor_negotiator[52]: Sorting 9 ads ... 2021-06-17T10:07:02.110664839Z condor_negotiator[52]: Ignoring submitter user20001@pseven-htcondor from schedd at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> with no requested jobs 2021-06-17T10:07:02.111054527Z condor_negotiator[52]: Got ads: 9 public and 7 private 2021-06-17T10:07:02.111064055Z condor_negotiator[52]: Public ads include 1 submitter, 7 startd 2021-06-17T10:07:02.111205366Z condor_negotiator[52]: Phase 2: Performing accounting ... 2021-06-17T10:07:02.111785888Z condor_negotiator[52]: Entering compute_significant_attrs() 2021-06-17T10:07:02.111796930Z condor_negotiator[52]: Leaving compute_significant_attrs() - result=_cp_orig_RequestCpus,_cp_orig_RequestDisk,_cp_orig_RequestMemory,KFlops,MachineLastMatchTime,Offline,RequestCpus,RequestDisk,RequestMemory,Scheduler,TotalJobRuntime 2021-06-17T10:07:02.116225611Z condor_negotiator[52]: Phase 3: Sorting submitter ads by priority ... 2021-06-17T10:07:02.116282401Z condor_negotiator[52]: Starting prefetch round; 1 potential prefetches to do. 2021-06-17T10:07:02.116290625Z condor_negotiator[52]: Assigned 1 units of work for prefetching. 2021-06-17T10:07:02.116294467Z condor_negotiator[52]: Starting prefetch loop. 2021-06-17T10:07:02.116296825Z condor_negotiator[52]: Starting prefetch negotiation for DedicatedScheduler@parallel_schedd@submit.pseven-htcondor. 2021-06-17T10:07:02.116299206Z condor_negotiator[52]: Socket to DedicatedScheduler@parallel_schedd@submit.pseven-htcondor (<10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c>) already in cache, reusing 2021-06-17T10:07:02.116334006Z condor_negotiator[52]: Started NEGOTIATE with remote schedd; protocol version 1. 2021-06-17T10:07:02.116342819Z condor_negotiator[52]: Sending SEND_RESOURCE_REQUEST_LIST/200/eom 2021-06-17T10:07:02.116356612Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:07:02.116371282Z condor_negotiator[52]: Prefetch negotiation would block. 2021-06-17T10:07:02.116466420Z condor_negotiator[52]: Waiting on the results of 1 negotiation sessions. 2021-06-17T10:07:02.116512328Z condor_schedd[129]: Activity on stashed negotiator socket: <10.42.0.205:36513> 2021-06-17T10:07:02.116533104Z condor_schedd[129]: 2021-06-17T10:07:02.116538470Z condor_schedd[129]: Entered negotiate 2021-06-17T10:07:02.116624478Z condor_schedd[129]: Using negotiation protocol: NEGOTIATE 2021-06-17T10:07:02.116637775Z condor_schedd[129]: *** SwapSpace = 2147483647 2021-06-17T10:07:02.116642577Z condor_schedd[129]: *** ReservedSwap = 0 2021-06-17T10:07:02.116646166Z condor_schedd[129]: *** Shadow Size Estimate = 125 2021-06-17T10:07:02.116649862Z condor_schedd[129]: *** Start Limit For Swap = 17179869 2021-06-17T10:07:02.116734743Z condor_schedd[129]: Negotiating for owner: DedicatedScheduler@parallel_schedd@submit.pseven-htcondor 2021-06-17T10:07:02.117307754Z condor_schedd[129]: Finished negotiating for DedicatedScheduler@parallel_schedd in local pool: 0 matched, 0 rejected 2021-06-17T10:07:02.117351755Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:07:02.117372820Z condor_negotiator[52]: Got NO_MORE_JOBS; schedd has no more requests 2021-06-17T10:07:02.117377882Z condor_negotiator[52]: Assigned 0 units of work for prefetching. 2021-06-17T10:07:02.117381917Z condor_negotiator[52]: Prefetch summary: 1 attempted, 1 successful. 2021-06-17T10:07:02.117385729Z condor_negotiator[52]: Phase 4.1: Negotiating with schedds ... 2021-06-17T10:07:02.117389751Z condor_negotiator[52]: numSlots = 7 (after trimming=7) 2021-06-17T10:07:02.117393424Z condor_negotiator[52]: slotWeightTotal = 200.000000 2021-06-17T10:07:02.117408782Z condor_negotiator[52]: minSlotWeight = 0.000000 2021-06-17T10:07:02.117413926Z condor_negotiator[52]: pieLeft = 200.000 2021-06-17T10:07:02.117417552Z condor_negotiator[52]: NormalFactor = 1.000000 2021-06-17T10:07:02.117421176Z condor_negotiator[52]: MaxPrioValue = 509.818970 2021-06-17T10:07:02.117424807Z condor_negotiator[52]: NumSubmitterAds = 1 2021-06-17T10:07:02.117428378Z condor_negotiator[52]: Negotiating with DedicatedScheduler@parallel_schedd@submit.pseven-htcondor at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> 2021-06-17T10:07:02.117433238Z condor_negotiator[52]: 0 seconds so far for this submitter 2021-06-17T10:07:02.117437213Z condor_negotiator[52]: 0 seconds so far for this schedd 2021-06-17T10:07:02.117440886Z condor_negotiator[52]: Calculating submitter limit with the following parameters 2021-06-17T10:07:02.117444141Z condor_negotiator[52]: SubmitterPrio = 509.818970 2021-06-17T10:07:02.117447635Z condor_negotiator[52]: SubmitterPrioFactor = 1000.000000 2021-06-17T10:07:02.117450619Z condor_negotiator[52]: submitterShare = 1.000000 2021-06-17T10:07:02.117453890Z condor_negotiator[52]: submitterAbsShare = 1.000000 2021-06-17T10:07:02.117457190Z condor_negotiator[52]: submitterLimit = 200.000000 2021-06-17T10:07:02.117468230Z condor_negotiator[52]: submitterCeiling remaining = 2147483647 2021-06-17T10:07:02.117472120Z condor_negotiator[52]: submitterUsage = 0.000000 2021-06-17T10:07:02.117485451Z condor_negotiator[52]: Using resource request list from cache. 2021-06-17T10:07:02.117489816Z condor_negotiator[52]: Socket to DedicatedScheduler@parallel_schedd@submit.pseven-htcondor (<10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c>) already in cache, reusing 2021-06-17T10:07:02.117494607Z condor_negotiator[52]: Started NEGOTIATE with remote schedd; protocol version 1. 2021-06-17T10:07:02.117500802Z condor_negotiator[52]: Send END_NEGOTIATE to remote schedd 2021-06-17T10:07:02.117504411Z condor_negotiator[52]: Submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor got all it wants; removing it. 2021-06-17T10:07:02.117508265Z condor_negotiator[52]: resources used by DedicatedScheduler@parallel_schedd@submit.pseven-htcondor are 0.000000 2021-06-17T10:07:02.117517314Z condor_negotiator[52]: resources used scheddUsed= 0.000000 2021-06-17T10:07:02.117519908Z condor_negotiator[52]: negotiateWithGroup resources used submitterAds length 0 2021-06-17T10:07:02.117522078Z condor_negotiator[52]: ---------- Finished Negotiation Cycle ---------- 2021-06-17T10:07:02.117524241Z condor_negotiator[52]: Updating collector with accounting information 2021-06-17T10:07:02.117526844Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:07:02.117529088Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:07:02.117643467Z condor_schedd[129]: Activity on stashed negotiator socket: <10.42.0.205:36513> 2021-06-17T10:07:02.117652523Z condor_schedd[129]: 2021-06-17T10:07:02.117656238Z condor_schedd[129]: Entered negotiate 2021-06-17T10:07:02.117659713Z condor_schedd[129]: Using negotiation protocol: NEGOTIATE 2021-06-17T10:07:02.117662902Z condor_schedd[129]: *** SwapSpace = 2147483647 2021-06-17T10:07:02.117665912Z condor_schedd[129]: *** ReservedSwap = 0 2021-06-17T10:07:02.117669025Z condor_schedd[129]: *** Shadow Size Estimate = 125 2021-06-17T10:07:02.117672498Z condor_schedd[129]: *** Start Limit For Swap = 17179869 2021-06-17T10:07:02.117675781Z condor_schedd[129]: Negotiating for owner: DedicatedScheduler@parallel_schedd@submit.pseven-htcondor 2021-06-17T10:07:02.117711111Z condor_schedd[129]: Finished sending rrls to negotiator 2021-06-17T10:07:02.117715885Z condor_schedd[129]: Finished negotiating for DedicatedScheduler@parallel_schedd in local pool: 0 matched, 0 rejected 2021-06-17T10:07:02.117901103Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:07:02.117924748Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:07:02.117929422Z condor_negotiator[52]: Done Updating collector with accounting information 2021-06-17T10:07:02.117932885Z condor_collector[51]: AccountingAd : Updating ... "< DedicatedScheduler@parallel_schedd@submit.pseven-htcondornegotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:07:02.117936936Z condor_collector[51]: AccountingAd : Updating ... "< <none>negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:07:02.127666593Z condor_collector[51]: MasterAd : Updating ... "< pseven-htcondorexecute-deploy-68b66d9c55-nvrwz.pseven-htcondor >" 2021-06-17T10:07:02.443329968Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:02.443574409Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:02.443587573Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:02.443592903Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:02.443638489Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000034; send_time=0.000056; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:42653>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:02.444788553Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:02.445288281Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 522 2021-06-17T10:07:02.445304204Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:02.453493380Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:02.453661429Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:02.454271630Z condor_collector[51]: QueryWorker: forked new worker with id 316 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:02.454293223Z condor_collector[316]: (Sending 7 ads in response to query) 2021-06-17T10:07:02.454677437Z condor_collector[316]: Query info: matched=7; skipped=0; query_time=0.000284; send_time=0.000178; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:43503>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:02.455078433Z condor_collector[51]: QueryWorker: Child 316 done 2021-06-17T10:07:03.408723154Z condor_collector[51]: MasterAd : Updating ... "< pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor >" 2021-06-17T10:07:03.462749374Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:03.462991827Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:03.463009347Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:03.463016845Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:03.463148264Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000044; send_time=0.000051; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:43007>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:03.464212108Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:03.464718253Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 534 2021-06-17T10:07:03.464737352Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:03.472800190Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:03.473103051Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:03.473768188Z condor_collector[51]: QueryWorker: forked new worker with id 317 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:03.473778568Z condor_collector[317]: (Sending 7 ads in response to query) 2021-06-17T10:07:03.473970000Z condor_collector[317]: Query info: matched=7; skipped=0; query_time=0.000305; send_time=0.000170; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:46193>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:03.474484067Z condor_collector[51]: QueryWorker: Child 317 done 2021-06-17T10:07:04.132696699Z condor_collector[51]: MasterAd : Updating ... "< master@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:07:04.498443197Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:04.499700976Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:04.499845403Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:04.500135036Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:04.500178547Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000122; send_time=0.000212; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:37981>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:04.504061859Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:04.505947103Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 539 2021-06-17T10:07:04.506014637Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:04.522348478Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:04.522727050Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:04.523492742Z condor_collector[51]: QueryWorker: forked new worker with id 318 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:04.523645264Z condor_collector[318]: (Sending 7 ads in response to query) 2021-06-17T10:07:04.523885971Z condor_collector[318]: Query info: matched=7; skipped=0; query_time=0.000501; send_time=0.000266; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:37011>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:04.524472213Z condor_collector[51]: QueryWorker: Child 318 done 2021-06-17T10:07:05.549743988Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:05.551048990Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:05.551226173Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:05.551258915Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:05.551963257Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000133; send_time=0.000206; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:37245>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:05.555934329Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:05.558076648Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 544 2021-06-17T10:07:05.560071896Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:05.576721797Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:05.577296285Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:05.577905743Z condor_collector[319]: (Sending 7 ads in response to query) 2021-06-17T10:07:05.577920363Z condor_collector[51]: QueryWorker: forked new worker with id 319 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:05.578260313Z condor_collector[319]: Query info: matched=7; skipped=0; query_time=0.000417; send_time=0.000304; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:45175>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:05.578949744Z condor_collector[51]: QueryWorker: Child 319 done 2021-06-17T10:07:06.103786893Z condor_startd[125]: Swap space: 48075796 2021-06-17T10:07:06.105814696Z condor_startd[125]: Started polling timer. 2021-06-17T10:07:06.106542918Z condor_startd[125]: slot1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:07:06.107397677Z condor_startd[125]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.107437800Z condor_startd[125]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.108900158Z condor_startd[125]: slot1: Sent update to 1 collector(s) 2021-06-17T10:07:06.109511481Z condor_startd[125]: slot1_1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:07:06.109602244Z condor_collector[51]: StartdAd : Updating ... "< slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.201 >" 2021-06-17T10:07:06.110033969Z condor_collector[51]: StartdPvtAd : Updating ... "< slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.201 >" 2021-06-17T10:07:06.110408090Z condor_startd[125]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.110444273Z condor_startd[125]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.111440965Z condor_startd[125]: slot1_1: Sent update to 1 collector(s) 2021-06-17T10:07:06.112029913Z condor_collector[51]: StartdAd : Updating ... "< slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.201 >" 2021-06-17T10:07:06.112210402Z condor_collector[51]: StartdPvtAd : Updating ... "< slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.201 >" 2021-06-17T10:07:06.155452820Z condor_startd[122]: Swap space: 48075796 2021-06-17T10:07:06.156651091Z condor_startd[122]: Started polling timer. 2021-06-17T10:07:06.157003887Z condor_startd[122]: slot1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:07:06.157513393Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.157560356Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.158357094Z condor_startd[122]: slot1: Sent update to 1 collector(s) 2021-06-17T10:07:06.158721077Z condor_startd[122]: slot1_1: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:07:06.158753699Z condor_collector[51]: StartdAd : Updating ... "< slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:07:06.158916853Z condor_collector[51]: StartdPvtAd : Updating ... "< slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:07:06.159067123Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.159125197Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.159612502Z condor_startd[122]: slot1_1: Sent update to 1 collector(s) 2021-06-17T10:07:06.159616757Z condor_collector[51]: StartdAd : Updating ... "< slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:07:06.159797161Z condor_startd[122]: slot1_2: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:07:06.159824799Z condor_collector[51]: StartdPvtAd : Updating ... "< slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:07:06.160105705Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.160122924Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.160428982Z condor_startd[122]: slot1_2: Sent update to 1 collector(s) 2021-06-17T10:07:06.160612380Z condor_collector[51]: StartdAd : Updating ... "< slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:07:06.160871624Z condor_collector[51]: StartdPvtAd : Updating ... "< slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:07:06.162438133Z condor_collector[51]: StartdAd : Updating ... "< slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:07:06.162482271Z condor_collector[51]: StartdPvtAd : Updating ... "< slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:07:06.162600132Z condor_startd[122]: slot1_3: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:07:06.162627555Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.162635459Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.162643269Z condor_startd[122]: slot1_3: Sent update to 1 collector(s) 2021-06-17T10:07:06.162649536Z condor_startd[122]: slot1_4: Acquiring consumption policy configuration for slot type 1 2021-06-17T10:07:06.162659787Z condor_startd[122]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.162667303Z condor_startd[122]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:06.163778881Z condor_collector[51]: StartdAd : Updating ... "< slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:07:06.163829229Z condor_collector[51]: StartdPvtAd : Updating ... "< slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx , 10.42.0.202 >" 2021-06-17T10:07:06.163859823Z condor_startd[122]: slot1_4: Sent update to 1 collector(s) 2021-06-17T10:07:06.602750115Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:06.604210126Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:06.604349388Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:06.604378165Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:06.604408503Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000286; send_time=0.000173; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:39191>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:06.608280837Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:06.610217664Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 549 2021-06-17T10:07:06.610277070Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:06.626275506Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:06.626681315Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:06.627542772Z condor_collector[51]: QueryWorker: forked new worker with id 320 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:06.627575967Z condor_collector[320]: (Sending 7 ads in response to query) 2021-06-17T10:07:06.628016463Z condor_collector[320]: Query info: matched=7; skipped=0; query_time=0.000487; send_time=0.000331; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:39015>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:06.628863653Z condor_collector[51]: QueryWorker: Child 320 done 2021-06-17T10:07:07.655010806Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:07.656269486Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:07.656367434Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:07.656394059Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:07.656717405Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000152; send_time=0.000212; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:46199>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:07.660191430Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:07.661774008Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 554 2021-06-17T10:07:07.661954406Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:07.676574864Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:07.676917239Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:07.677545496Z condor_collector[51]: QueryWorker: forked new worker with id 321 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:07.677742205Z condor_collector[321]: (Sending 7 ads in response to query) 2021-06-17T10:07:07.678051961Z condor_collector[321]: Query info: matched=7; skipped=0; query_time=0.000414; send_time=0.000243; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:34201>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:07.678669821Z condor_collector[51]: QueryWorker: Child 321 done 2021-06-17T10:07:07.731698953Z condor_collector[51]: MasterAd : Updating ... "< submit.pseven-htcondor >" 2021-06-17T10:07:08.124240195Z condor_negotiator[52]: ---------- Started Negotiation Cycle ---------- 2021-06-17T10:07:08.124308292Z condor_negotiator[52]: Phase 1: Obtaining ads from collector ... 2021-06-17T10:07:08.124324125Z condor_negotiator[52]: Getting startd private ads ... 2021-06-17T10:07:08.124337058Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:07:08.126207331Z condor_collector[51]: Got QUERY_STARTD_PVT_ADS 2021-06-17T10:07:08.128057904Z condor_collector[51]: QueryWorker: forked new high priority worker with id 322 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:08.128112982Z condor_collector[322]: (Sending 7 ads in response to query) 2021-06-17T10:07:08.135262382Z condor_collector[322]: Query info: matched=7; skipped=0; query_time=0.000989; send_time=0.005944; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:42997>; projection={}; filter_private_ads=0 2021-06-17T10:07:08.137738224Z condor_negotiator[52]: Getting Scheduler, Submitter and Machine ads ... 2021-06-17T10:07:08.137783441Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:07:08.138048445Z condor_collector[51]: QueryWorker: Child 322 done 2021-06-17T10:07:08.139983064Z condor_collector[51]: Got QUERY_ANY_ADS 2021-06-17T10:07:08.141856544Z condor_collector[51]: QueryWorker: forked new high priority worker with id 323 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:08.141897750Z condor_collector[323]: (Sending 9 ads in response to query) 2021-06-17T10:07:08.145968128Z condor_collector[323]: Query info: matched=9; skipped=9; query_time=0.000942; send_time=0.003911; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:42999>; projection={}; filter_private_ads=0 2021-06-17T10:07:08.147386130Z condor_collector[51]: QueryWorker: Child 323 done 2021-06-17T10:07:08.153001673Z condor_negotiator[52]: Sorting 9 ads ... 2021-06-17T10:07:08.153127831Z condor_negotiator[52]: Ignoring submitter user20001@pseven-htcondor from schedd at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> with no requested jobs 2021-06-17T10:07:08.156684446Z condor_negotiator[52]: Got ads: 9 public and 7 private 2021-06-17T10:07:08.156756861Z condor_negotiator[52]: Public ads include 1 submitter, 7 startd 2021-06-17T10:07:08.156769257Z condor_negotiator[52]: Phase 2: Performing accounting ... 2021-06-17T10:07:08.156778900Z condor_negotiator[52]: Entering compute_significant_attrs() 2021-06-17T10:07:08.156932712Z condor_negotiator[52]: Leaving compute_significant_attrs() - result=_cp_orig_RequestCpus,_cp_orig_RequestDisk,_cp_orig_RequestMemory,KFlops,MachineLastMatchTime,Offline,RequestCpus,RequestDisk,RequestMemory,Scheduler,TotalJobRuntime 2021-06-17T10:07:08.161670802Z condor_negotiator[52]: Phase 3: Sorting submitter ads by priority ... 2021-06-17T10:07:08.161731544Z condor_negotiator[52]: Starting prefetch round; 1 potential prefetches to do. 2021-06-17T10:07:08.161740285Z condor_negotiator[52]: Assigned 1 units of work for prefetching. 2021-06-17T10:07:08.161745743Z condor_negotiator[52]: Starting prefetch loop. 2021-06-17T10:07:08.161750809Z condor_negotiator[52]: Starting prefetch negotiation for DedicatedScheduler@parallel_schedd@submit.pseven-htcondor. 2021-06-17T10:07:08.161756459Z condor_negotiator[52]: Socket to DedicatedScheduler@parallel_schedd@submit.pseven-htcondor (<10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c>) already in cache, reusing 2021-06-17T10:07:08.161841379Z condor_negotiator[52]: Started NEGOTIATE with remote schedd; protocol version 1. 2021-06-17T10:07:08.161950468Z condor_negotiator[52]: Sending SEND_RESOURCE_REQUEST_LIST/200/eom 2021-06-17T10:07:08.162177590Z condor_schedd[129]: Activity on stashed negotiator socket: <10.42.0.205:36513> 2021-06-17T10:07:08.162210058Z condor_schedd[129]: 2021-06-17T10:07:08.162219930Z condor_schedd[129]: Entered negotiate 2021-06-17T10:07:08.162228516Z condor_schedd[129]: Using negotiation protocol: NEGOTIATE 2021-06-17T10:07:08.162233782Z condor_schedd[129]: *** SwapSpace = 2147483647 2021-06-17T10:07:08.162238666Z condor_schedd[129]: *** ReservedSwap = 0 2021-06-17T10:07:08.162243439Z condor_schedd[129]: *** Shadow Size Estimate = 125 2021-06-17T10:07:08.162248383Z condor_schedd[129]: *** Start Limit For Swap = 17179869 2021-06-17T10:07:08.162385283Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:07:08.162393022Z condor_schedd[129]: Negotiating for owner: DedicatedScheduler@parallel_schedd@submit.pseven-htcondor 2021-06-17T10:07:08.162402867Z condor_negotiator[52]: Prefetch negotiation would block. 2021-06-17T10:07:08.162409141Z condor_negotiator[52]: Waiting on the results of 1 negotiation sessions. 2021-06-17T10:07:08.162620201Z condor_schedd[129]: Finished negotiating for DedicatedScheduler@parallel_schedd in local pool: 0 matched, 0 rejected 2021-06-17T10:07:08.162621042Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:07:08.162637787Z condor_negotiator[52]: Got NO_MORE_JOBS; schedd has no more requests 2021-06-17T10:07:08.162750902Z condor_negotiator[52]: Assigned 0 units of work for prefetching. 2021-06-17T10:07:08.162764958Z condor_negotiator[52]: Prefetch summary: 1 attempted, 1 successful. 2021-06-17T10:07:08.162773540Z condor_negotiator[52]: Phase 4.1: Negotiating with schedds ... 2021-06-17T10:07:08.162793947Z condor_negotiator[52]: numSlots = 7 (after trimming=7) 2021-06-17T10:07:08.162828505Z condor_negotiator[52]: slotWeightTotal = 200.000000 2021-06-17T10:07:08.162887316Z condor_negotiator[52]: minSlotWeight = 0.000000 2021-06-17T10:07:08.162904011Z condor_negotiator[52]: pieLeft = 200.000 2021-06-17T10:07:08.162912442Z condor_negotiator[52]: NormalFactor = 1.000000 2021-06-17T10:07:08.162973902Z condor_negotiator[52]: MaxPrioValue = 509.794006 2021-06-17T10:07:08.163003012Z condor_negotiator[52]: NumSubmitterAds = 1 2021-06-17T10:07:08.163057139Z condor_negotiator[52]: Negotiating with DedicatedScheduler@parallel_schedd@submit.pseven-htcondor at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> 2021-06-17T10:07:08.163071131Z condor_negotiator[52]: 0 seconds so far for this submitter 2021-06-17T10:07:08.163076552Z condor_negotiator[52]: 0 seconds so far for this schedd 2021-06-17T10:07:08.163187949Z condor_negotiator[52]: Calculating submitter limit with the following parameters 2021-06-17T10:07:08.163206853Z condor_negotiator[52]: SubmitterPrio = 509.794006 2021-06-17T10:07:08.163245669Z condor_negotiator[52]: SubmitterPrioFactor = 1000.000000 2021-06-17T10:07:08.163260872Z condor_negotiator[52]: submitterShare = 1.000000 2021-06-17T10:07:08.163268964Z condor_negotiator[52]: submitterAbsShare = 1.000000 2021-06-17T10:07:08.163304088Z condor_negotiator[52]: submitterLimit = 200.000000 2021-06-17T10:07:08.163313838Z condor_negotiator[52]: submitterCeiling remaining = 2147483647 2021-06-17T10:07:08.163354627Z condor_negotiator[52]: submitterUsage = 0.000000 2021-06-17T10:07:08.163398825Z condor_negotiator[52]: Using resource request list from cache. 2021-06-17T10:07:08.163451959Z condor_negotiator[52]: Socket to DedicatedScheduler@parallel_schedd@submit.pseven-htcondor (<10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c>) already in cache, reusing 2021-06-17T10:07:08.163581202Z condor_negotiator[52]: Started NEGOTIATE with remote schedd; protocol version 1. 2021-06-17T10:07:08.163744455Z condor_negotiator[52]: Send END_NEGOTIATE to remote schedd 2021-06-17T10:07:08.163762862Z condor_negotiator[52]: Submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor got all it wants; removing it. 2021-06-17T10:07:08.163782237Z condor_schedd[129]: Activity on stashed negotiator socket: <10.42.0.205:36513> 2021-06-17T10:07:08.163787341Z condor_negotiator[52]: resources used by DedicatedScheduler@parallel_schedd@submit.pseven-htcondor are 0.000000 2021-06-17T10:07:08.163820870Z condor_schedd[129]: 2021-06-17T10:07:08.163828710Z condor_schedd[129]: Entered negotiate 2021-06-17T10:07:08.163897426Z condor_negotiator[52]: resources used scheddUsed= 0.000000 2021-06-17T10:07:08.163900232Z condor_schedd[129]: Using negotiation protocol: NEGOTIATE 2021-06-17T10:07:08.163928767Z condor_schedd[129]: *** SwapSpace = 2147483647 2021-06-17T10:07:08.163936447Z condor_schedd[129]: *** ReservedSwap = 0 2021-06-17T10:07:08.163942537Z condor_schedd[129]: *** Shadow Size Estimate = 125 2021-06-17T10:07:08.163944150Z condor_negotiator[52]: negotiateWithGroup resources used submitterAds length 0 2021-06-17T10:07:08.163949659Z condor_schedd[129]: *** Start Limit For Swap = 17179869 2021-06-17T10:07:08.163974753Z condor_negotiator[52]: ---------- Finished Negotiation Cycle ---------- 2021-06-17T10:07:08.164045238Z condor_negotiator[52]: Updating collector with accounting information 2021-06-17T10:07:08.164051514Z condor_schedd[129]: Negotiating for owner: DedicatedScheduler@parallel_schedd@submit.pseven-htcondor 2021-06-17T10:07:08.164140241Z condor_schedd[129]: Finished sending rrls to negotiator 2021-06-17T10:07:08.164159969Z condor_schedd[129]: Finished negotiating for DedicatedScheduler@parallel_schedd in local pool: 0 matched, 0 rejected 2021-06-17T10:07:08.164163748Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:07:08.164173910Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:07:08.164290726Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:07:08.164299727Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:07:08.164403629Z condor_negotiator[52]: Done Updating collector with accounting information 2021-06-17T10:07:08.164412728Z condor_collector[51]: AccountingAd : Updating ... "< DedicatedScheduler@parallel_schedd@submit.pseven-htcondornegotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:07:08.164636145Z condor_collector[51]: AccountingAd : Updating ... "< <none>negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:07:08.702943088Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:08.704133442Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:08.704197460Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:08.704540329Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:08.704588887Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000128; send_time=0.000177; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:46721>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:08.708515573Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:08.710076615Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 559 2021-06-17T10:07:08.710288175Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:08.726607528Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:08.727131135Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:08.728029908Z condor_collector[51]: QueryWorker: forked new worker with id 324 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:08.728052108Z condor_collector[324]: (Sending 7 ads in response to query) 2021-06-17T10:07:08.728400936Z condor_collector[324]: Query info: matched=7; skipped=0; query_time=0.000495; send_time=0.000304; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:41143>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:08.729121883Z condor_collector[51]: QueryWorker: Child 324 done 2021-06-17T10:07:09.751677378Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:09.752198486Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:09.752226925Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:09.752288396Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:09.753319728Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000100; send_time=0.000089; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:37179>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:09.754345233Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:09.755504736Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 564 2021-06-17T10:07:09.755534610Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:09.766151923Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:09.766621797Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:09.766981053Z condor_collector[51]: QueryWorker: forked new worker with id 325 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:09.767400748Z condor_collector[325]: (Sending 7 ads in response to query) 2021-06-17T10:07:09.767425326Z condor_collector[325]: Query info: matched=7; skipped=0; query_time=0.000361; send_time=0.000213; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:37759>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:09.767868064Z condor_collector[51]: QueryWorker: Child 325 done 2021-06-17T10:07:10.775832883Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:10.776142896Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:10.776153559Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:10.776188603Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:10.776261986Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000029; send_time=0.000056; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:35723>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:10.777405815Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:10.778035774Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 569 2021-06-17T10:07:10.778056434Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:10.787114930Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:10.787450487Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:10.788136268Z condor_collector[51]: QueryWorker: forked new worker with id 326 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:10.788183882Z condor_collector[326]: (Sending 7 ads in response to query) 2021-06-17T10:07:10.788464176Z condor_collector[326]: Query info: matched=7; skipped=0; query_time=0.000363; send_time=0.000235; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:45615>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:10.789064519Z condor_collector[51]: QueryWorker: Child 326 done 2021-06-17T10:07:11.796961958Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:11.797190877Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:11.797198527Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:11.797216339Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:11.797288072Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000021; send_time=0.000039; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:33245>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:11.798618117Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:11.799332355Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 574 2021-06-17T10:07:11.799356747Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:11.806911481Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:11.807492068Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:11.807877013Z condor_collector[51]: QueryWorker: forked new worker with id 327 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:11.807889026Z condor_collector[327]: (Sending 7 ads in response to query) 2021-06-17T10:07:11.808077793Z condor_collector[327]: Query info: matched=7; skipped=0; query_time=0.000307; send_time=0.000172; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:40347>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:11.809208285Z condor_collector[51]: QueryWorker: Child 327 done 2021-06-17T10:07:12.816292303Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:12.816552366Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:12.816561534Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:12.816592466Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:12.817029507Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000027; send_time=0.000042; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:35377>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:12.817703648Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:12.819130437Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 579 2021-06-17T10:07:12.819155251Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:12.826280814Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:12.826529968Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:12.827104288Z condor_collector[51]: QueryWorker: forked new worker with id 328 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:12.827118457Z condor_collector[328]: (Sending 7 ads in response to query) 2021-06-17T10:07:12.827350732Z condor_collector[328]: Query info: matched=7; skipped=0; query_time=0.000269; send_time=0.000210; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:42963>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:12.827804901Z condor_collector[51]: QueryWorker: Child 328 done 2021-06-17T10:07:13.821230261Z condor_schedd[129]: -------- Begin starting jobs -------- 2021-06-17T10:07:13.821303858Z condor_schedd[129]: -------- Done starting jobs -------- 2021-06-17T10:07:13.852003876Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:13.853674445Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:13.855327302Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:13.855372114Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:13.855386489Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000131; send_time=0.000198; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:42687>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:13.858218689Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:13.860060126Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 584 2021-06-17T10:07:13.860131609Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:13.876886829Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:13.877288725Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:13.877984338Z condor_collector[51]: QueryWorker: forked new worker with id 329 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:13.878097524Z condor_collector[329]: (Sending 7 ads in response to query) 2021-06-17T10:07:13.878401234Z condor_collector[329]: Query info: matched=7; skipped=0; query_time=0.000416; send_time=0.000273; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:35265>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:13.879028540Z condor_collector[51]: QueryWorker: Child 329 done 2021-06-17T10:07:14.168412366Z condor_negotiator[52]: ---------- Started Negotiation Cycle ---------- 2021-06-17T10:07:14.168495606Z condor_negotiator[52]: Phase 1: Obtaining ads from collector ... 2021-06-17T10:07:14.168517565Z condor_negotiator[52]: Getting startd private ads ... 2021-06-17T10:07:14.168540217Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:07:14.171181966Z condor_collector[51]: Got QUERY_STARTD_PVT_ADS 2021-06-17T10:07:14.173246294Z condor_collector[51]: QueryWorker: forked new high priority worker with id 330 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:14.173298656Z condor_collector[330]: (Sending 7 ads in response to query) 2021-06-17T10:07:14.177755319Z condor_collector[330]: Query info: matched=7; skipped=0; query_time=0.000778; send_time=0.004285; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:36557>; projection={}; filter_private_ads=0 2021-06-17T10:07:14.179162893Z condor_collector[51]: QueryWorker: Child 330 done 2021-06-17T10:07:14.179951470Z condor_negotiator[52]: Getting Scheduler, Submitter and Machine ads ... 2021-06-17T10:07:14.179976168Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:07:14.181137665Z condor_collector[51]: Got QUERY_ANY_ADS 2021-06-17T10:07:14.182804447Z condor_collector[51]: QueryWorker: forked new high priority worker with id 331 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:14.182836905Z condor_collector[331]: (Sending 9 ads in response to query) 2021-06-17T10:07:14.185643286Z condor_collector[331]: Query info: matched=9; skipped=9; query_time=0.000822; send_time=0.002963; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:40687>; projection={}; filter_private_ads=0 2021-06-17T10:07:14.186573863Z condor_collector[51]: QueryWorker: Child 331 done 2021-06-17T10:07:14.187617383Z condor_negotiator[52]: Sorting 9 ads ... 2021-06-17T10:07:14.188161520Z condor_negotiator[52]: Ignoring submitter user20001@pseven-htcondor from schedd at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> with no requested jobs 2021-06-17T10:07:14.188877971Z condor_negotiator[52]: Got ads: 9 public and 7 private 2021-06-17T10:07:14.188896389Z condor_negotiator[52]: Public ads include 1 submitter, 7 startd 2021-06-17T10:07:14.188975220Z condor_negotiator[52]: Phase 2: Performing accounting ... 2021-06-17T10:07:14.188988756Z condor_negotiator[52]: Entering compute_significant_attrs() 2021-06-17T10:07:14.189707450Z condor_negotiator[52]: Leaving compute_significant_attrs() - result=_cp_orig_RequestCpus,_cp_orig_RequestDisk,_cp_orig_RequestMemory,KFlops,MachineLastMatchTime,Offline,RequestCpus,RequestDisk,RequestMemory,Scheduler,TotalJobRuntime 2021-06-17T10:07:14.196313047Z condor_negotiator[52]: Phase 3: Sorting submitter ads by priority ... 2021-06-17T10:07:14.196336571Z condor_negotiator[52]: Starting prefetch round; 1 potential prefetches to do. 2021-06-17T10:07:14.196344233Z condor_negotiator[52]: Assigned 1 units of work for prefetching. 2021-06-17T10:07:14.196350041Z condor_negotiator[52]: Starting prefetch loop. 2021-06-17T10:07:14.196355950Z condor_negotiator[52]: Starting prefetch negotiation for DedicatedScheduler@parallel_schedd@submit.pseven-htcondor. 2021-06-17T10:07:14.196361688Z condor_negotiator[52]: Socket to DedicatedScheduler@parallel_schedd@submit.pseven-htcondor (<10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c>) already in cache, reusing 2021-06-17T10:07:14.196420853Z condor_negotiator[52]: Started NEGOTIATE with remote schedd; protocol version 1. 2021-06-17T10:07:14.196429940Z condor_negotiator[52]: Sending SEND_RESOURCE_REQUEST_LIST/200/eom 2021-06-17T10:07:14.196434626Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:07:14.196451241Z condor_negotiator[52]: Prefetch negotiation would block. 2021-06-17T10:07:14.196455664Z condor_negotiator[52]: Waiting on the results of 1 negotiation sessions. 2021-06-17T10:07:14.196472006Z condor_schedd[129]: Activity on stashed negotiator socket: <10.42.0.205:36513> 2021-06-17T10:07:14.196488432Z condor_schedd[129]: 2021-06-17T10:07:14.196494344Z condor_schedd[129]: Entered negotiate 2021-06-17T10:07:14.196523699Z condor_schedd[129]: Using negotiation protocol: NEGOTIATE 2021-06-17T10:07:14.196573845Z condor_schedd[129]: *** SwapSpace = 2147483647 2021-06-17T10:07:14.196584322Z condor_schedd[129]: *** ReservedSwap = 0 2021-06-17T10:07:14.196589881Z condor_schedd[129]: *** Shadow Size Estimate = 125 2021-06-17T10:07:14.196617832Z condor_schedd[129]: *** Start Limit For Swap = 17179869 2021-06-17T10:07:14.196743806Z condor_schedd[129]: Negotiating for owner: DedicatedScheduler@parallel_schedd@submit.pseven-htcondor 2021-06-17T10:07:14.196915006Z condor_schedd[129]: Finished negotiating for DedicatedScheduler@parallel_schedd in local pool: 0 matched, 0 rejected 2021-06-17T10:07:14.196930845Z condor_negotiator[52]: Getting reply from schedd ... 2021-06-17T10:07:14.196945010Z condor_negotiator[52]: Got NO_MORE_JOBS; schedd has no more requests 2021-06-17T10:07:14.197048181Z condor_negotiator[52]: Assigned 0 units of work for prefetching. 2021-06-17T10:07:14.197064735Z condor_negotiator[52]: Prefetch summary: 1 attempted, 1 successful. 2021-06-17T10:07:14.197117374Z condor_negotiator[52]: Phase 4.1: Negotiating with schedds ... 2021-06-17T10:07:14.197129297Z condor_negotiator[52]: numSlots = 7 (after trimming=7) 2021-06-17T10:07:14.197245400Z condor_negotiator[52]: slotWeightTotal = 200.000000 2021-06-17T10:07:14.197254866Z condor_negotiator[52]: minSlotWeight = 0.000000 2021-06-17T10:07:14.197260391Z condor_negotiator[52]: pieLeft = 200.000 2021-06-17T10:07:14.197265968Z condor_negotiator[52]: NormalFactor = 1.000000 2021-06-17T10:07:14.197271037Z condor_negotiator[52]: MaxPrioValue = 509.769012 2021-06-17T10:07:14.197276536Z condor_negotiator[52]: NumSubmitterAds = 1 2021-06-17T10:07:14.197281927Z condor_negotiator[52]: Negotiating with DedicatedScheduler@parallel_schedd@submit.pseven-htcondor at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> 2021-06-17T10:07:14.197288952Z condor_negotiator[52]: 0 seconds so far for this submitter 2021-06-17T10:07:14.197294187Z condor_negotiator[52]: 0 seconds so far for this schedd 2021-06-17T10:07:14.197372953Z condor_negotiator[52]: Calculating submitter limit with the following parameters 2021-06-17T10:07:14.197386449Z condor_negotiator[52]: SubmitterPrio = 509.769012 2021-06-17T10:07:14.197402457Z condor_negotiator[52]: SubmitterPrioFactor = 1000.000000 2021-06-17T10:07:14.197408855Z condor_negotiator[52]: submitterShare = 1.000000 2021-06-17T10:07:14.197414108Z condor_negotiator[52]: submitterAbsShare = 1.000000 2021-06-17T10:07:14.197419692Z condor_negotiator[52]: submitterLimit = 200.000000 2021-06-17T10:07:14.197424728Z condor_negotiator[52]: submitterCeiling remaining = 2147483647 2021-06-17T10:07:14.197440691Z condor_negotiator[52]: submitterUsage = 0.000000 2021-06-17T10:07:14.197447062Z condor_negotiator[52]: Using resource request list from cache. 2021-06-17T10:07:14.197452067Z condor_negotiator[52]: Socket to DedicatedScheduler@parallel_schedd@submit.pseven-htcondor (<10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c>) already in cache, reusing 2021-06-17T10:07:14.197535668Z condor_negotiator[52]: Started NEGOTIATE with remote schedd; protocol version 1. 2021-06-17T10:07:14.197582514Z condor_negotiator[52]: Send END_NEGOTIATE to remote schedd 2021-06-17T10:07:14.197585423Z condor_schedd[129]: Activity on stashed negotiator socket: <10.42.0.205:36513> 2021-06-17T10:07:14.197612273Z condor_schedd[129]: 2021-06-17T10:07:14.197663923Z condor_schedd[129]: Entered negotiate 2021-06-17T10:07:14.197689656Z condor_negotiator[52]: Submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor got all it wants; removing it. 2021-06-17T10:07:14.197700039Z condor_negotiator[52]: resources used by DedicatedScheduler@parallel_schedd@submit.pseven-htcondor are 0.000000 2021-06-17T10:07:14.197771507Z condor_schedd[129]: Using negotiation protocol: NEGOTIATE 2021-06-17T10:07:14.197780916Z condor_schedd[129]: *** SwapSpace = 2147483647 2021-06-17T10:07:14.197784698Z condor_schedd[129]: *** ReservedSwap = 0 2021-06-17T10:07:14.197787915Z condor_schedd[129]: *** Shadow Size Estimate = 125 2021-06-17T10:07:14.197831985Z condor_schedd[129]: *** Start Limit For Swap = 17179869 2021-06-17T10:07:14.197844756Z condor_schedd[129]: Negotiating for owner: DedicatedScheduler@parallel_schedd@submit.pseven-htcondor 2021-06-17T10:07:14.197933260Z condor_negotiator[52]: resources used scheddUsed= 0.000000 2021-06-17T10:07:14.197941741Z condor_negotiator[52]: negotiateWithGroup resources used submitterAds length 0 2021-06-17T10:07:14.197946676Z condor_negotiator[52]: ---------- Finished Negotiation Cycle ---------- 2021-06-17T10:07:14.197951847Z condor_negotiator[52]: Updating collector with accounting information 2021-06-17T10:07:14.197957148Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:07:14.197962867Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:07:14.197980742Z condor_schedd[129]: Finished sending rrls to negotiator 2021-06-17T10:07:14.197988440Z condor_schedd[129]: Finished negotiating for DedicatedScheduler@parallel_schedd in local pool: 0 matched, 0 rejected 2021-06-17T10:07:14.198001575Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:07:14.198005704Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:07:14.198080764Z condor_negotiator[52]: Done Updating collector with accounting information 2021-06-17T10:07:14.198086535Z condor_collector[51]: AccountingAd : Updating ... "< DedicatedScheduler@parallel_schedd@submit.pseven-htcondornegotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:07:14.198512166Z condor_collector[51]: AccountingAd : Updating ... "< <none>negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:07:14.903224077Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:14.904702630Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:14.904768933Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:14.904829471Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:14.905121957Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000101; send_time=0.000205; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:38649>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:14.909480123Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:14.911261093Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 589 2021-06-17T10:07:14.911366150Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:14.930638548Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:14.931058329Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:14.931962541Z condor_collector[51]: QueryWorker: forked new worker with id 332 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:14.931994826Z condor_collector[332]: (Sending 7 ads in response to query) 2021-06-17T10:07:14.932350689Z condor_collector[332]: Query info: matched=7; skipped=0; query_time=0.000490; send_time=0.000275; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:41031>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:14.933045718Z condor_collector[51]: QueryWorker: Child 332 done 2021-06-17T10:07:15.958339285Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:15.959504175Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:15.959565810Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:15.959663706Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:15.960279898Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000163; send_time=0.000171; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:33741>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:15.963969268Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:15.965617431Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 594 2021-06-17T10:07:15.965666861Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:15.984999629Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:15.985444061Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:15.986108440Z condor_collector[51]: QueryWorker: forked new worker with id 333 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:15.986174127Z condor_collector[333]: (Sending 7 ads in response to query) 2021-06-17T10:07:15.986534092Z condor_collector[333]: Query info: matched=7; skipped=0; query_time=0.000456; send_time=0.000285; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:46735>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:15.987222882Z condor_collector[51]: QueryWorker: Child 333 done 2021-06-17T10:07:17.011419815Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:17.012547791Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:17.012615455Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:17.012739539Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:17.013622071Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000216; send_time=0.000182; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:36115>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:17.016961720Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:17.018789143Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 599 2021-06-17T10:07:17.019274690Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:17.040126553Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:17.040814318Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:17.041771654Z condor_collector[51]: QueryWorker: forked new worker with id 334 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:17.041918724Z condor_collector[334]: (Sending 7 ads in response to query) 2021-06-17T10:07:17.042467642Z condor_collector[334]: Query info: matched=7; skipped=0; query_time=0.000631; send_time=0.000460; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:35379>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:17.043644701Z condor_collector[51]: QueryWorker: Child 334 done 2021-06-17T10:07:18.069732888Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:18.070971253Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:18.071038086Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:18.071060794Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:18.071369729Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000120; send_time=0.000196; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:33823>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:18.075039006Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:18.077012658Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 604 2021-06-17T10:07:18.077074726Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:18.091196620Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:18.091749464Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:18.092457725Z condor_collector[51]: QueryWorker: forked new worker with id 335 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:18.092531732Z condor_collector[335]: (Sending 7 ads in response to query) 2021-06-17T10:07:18.092881008Z condor_collector[335]: Query info: matched=7; skipped=0; query_time=0.000430; send_time=0.000235; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:40047>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:18.093564992Z condor_collector[51]: QueryWorker: Child 335 done 2021-06-17T10:07:19.081232424Z condor_schedd[129]: Found idle MPI cluster 2 2021-06-17T10:07:19.081309947Z condor_schedd[129]: Started timer (67) to call handleDedicatedJobs() in 0 secs 2021-06-17T10:07:19.081364514Z condor_schedd[129]: JobsRunning = 0 2021-06-17T10:07:19.081394540Z condor_schedd[129]: JobsIdle = 5 2021-06-17T10:07:19.081411129Z condor_schedd[129]: JobsHeld = 0 2021-06-17T10:07:19.081426916Z condor_schedd[129]: JobsRemoved = 0 2021-06-17T10:07:19.081442374Z condor_schedd[129]: LocalUniverseJobsRunning = 0 2021-06-17T10:07:19.081622067Z condor_schedd[129]: LocalUniverseJobsIdle = 0 2021-06-17T10:07:19.081715627Z condor_schedd[129]: SchedUniverseJobsRunning = 0 2021-06-17T10:07:19.081742681Z condor_schedd[129]: SchedUniverseJobsIdle = 0 2021-06-17T10:07:19.081761472Z condor_schedd[129]: NumSubmitters = 1 2021-06-17T10:07:19.081772347Z condor_schedd[129]: MaxJobsRunning = 10000 2021-06-17T10:07:19.081782172Z condor_schedd[129]: MaxRunningSchedulerJobsPerOwner = 200 2021-06-17T10:07:19.082166735Z condor_schedd[129]: TransferQueueManager stats: active up=0/100 down=0/100; waiting up=0 down=0; wait time up=0s down=0s 2021-06-17T10:07:19.082198535Z condor_schedd[129]: TransferQueueManager upload 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load 2021-06-17T10:07:19.082214034Z condor_schedd[129]: TransferQueueManager download 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load 2021-06-17T10:07:19.084865151Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:19.084919778Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:19.085336018Z condor_schedd[129]: Sent HEART BEAT ad to 1 collectors. Number of active submittors=1 2021-06-17T10:07:19.086210637Z condor_collector[51]: ScheddAd : Updating ... "< parallel_schedd@xxxxxxxxxxxxxxxxxxxxxx , 10.42.0.213 >" 2021-06-17T10:07:19.088679488Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:19.088726622Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:19.088750729Z condor_schedd[129]: Sent ad to 1 collectors for user20001 Hit=5 Tot=5 Idle=0 Run=0 2021-06-17T10:07:19.088768306Z condor_schedd[129]: ============ Begin clean_shadow_recs ============= 2021-06-17T10:07:19.088785580Z condor_schedd[129]: ============ End clean_shadow_recs ============= 2021-06-17T10:07:19.088802097Z condor_schedd[129]: Starting DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:07:19.088818074Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:07:19.088906083Z condor_collector[51]: SubmittorAd : Updating ... "< user20001@pseven-htcondorparallel_schedd@submit.pseven-htcondor , 10.42.0.213 >" 2021-06-17T10:07:19.089004351Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:07:19.089165938Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:07:19.089216282Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:07:19.089458805Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:07:19.089495843Z condor_schedd[129]: Found 1 idle dedicated job(s) 2021-06-17T10:07:19.089509135Z condor_schedd[129]: DedicatedScheduler: Listing all dedicated jobs - 2021-06-17T10:07:19.089558693Z condor_schedd[129]: Dedicated job: 2.0 user20001 2021-06-17T10:07:19.089755888Z condor_schedd[129]: Trying to query collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:19.091799050Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:19.093646486Z condor_collector[51]: QueryWorker: forked new worker with id 336 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:19.093705200Z condor_collector[336]: (Sending 7 ads in response to query) 2021-06-17T10:07:19.100144837Z condor_collector[336]: Query info: matched=7; skipped=0; query_time=0.001020; send_time=0.004596; type=Machine; requirements={((DedicatedScheduler == "DedicatedScheduler@parallel_schedd@submit.pseven-htcondor"))}; locate=0; limit=0; from=SCHEDD; peer=<10.42.0.213:46591>; projection={}; filter_private_ads=0 2021-06-17T10:07:19.100320653Z condor_collector[51]: QueryWorker: Child 336 done 2021-06-17T10:07:19.103762615Z condor_schedd[129]: Found 7 potential dedicated resources in 0 seconds 2021-06-17T10:07:19.104185346Z condor_schedd[129]: New dynamic slot slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.104472232Z condor_schedd[129]: New dynamic slot slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.104704148Z condor_schedd[129]: New dynamic slot slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.105081889Z condor_schedd[129]: New dynamic slot slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.105360000Z condor_schedd[129]: Duplicate x1 (0/1) partitionable res slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.105451533Z condor_schedd[129]: Duplicate x75 (75/0) partitionable res slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.105533525Z condor_schedd[129]: idle resource list 2021-06-17T10:07:19.105627426Z condor_schedd[129]: LINUX#011X86_64#011slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.105800576Z condor_schedd[129]: LINUX#011X86_64#011slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.105820017Z condor_schedd[129]: LINUX#011X86_64#011slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.105843073Z condor_schedd[129]: LINUX#011X86_64#011slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.105858586Z condor_schedd[129]: LINUX#011X86_64#011slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.105966930Z condor_schedd[129]: limbo resource list 2021-06-17T10:07:19.106046038Z condor_schedd[129]: ************ empty ************ 2021-06-17T10:07:19.106146147Z condor_schedd[129]: serial c/i resource list 2021-06-17T10:07:19.106170063Z condor_schedd[129]: ************ empty ************ 2021-06-17T10:07:19.106197843Z condor_schedd[129]: unclaimed resource list 2021-06-17T10:07:19.106221555Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.106607686Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ... 2021-06-17T10:07:19.109744848Z condor_schedd[129]: LINUX#011X86_64#011slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.109750938Z condor_schedd[129]: busy resource list 2021-06-17T10:07:19.109755967Z condor_schedd[129]: ************ empty ************ 2021-06-17T10:07:19.109760967Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.0 2021-06-17T10:07:19.109765935Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.1 2021-06-17T10:07:19.109770851Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.2 2021-06-17T10:07:19.109775729Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.3 2021-06-17T10:07:19.109780672Z condor_schedd[129]: Trying to find 1 resource(s) for dedicated job 2.4 2021-06-17T10:07:19.109786163Z condor_schedd[129]: satisfyJobs: testing 5 jobs against 5 slots 2021-06-17T10:07:19.109793567Z condor_schedd[129]: satisfyJobs: finding resources for 2.0 2021-06-17T10:07:19.109801354Z condor_schedd[129]: Entering ded-schedd concurrency limit check... 2021-06-17T10:07:19.109808022Z condor_schedd[129]: job limit: "" 2021-06-17T10:07:19.109813149Z condor_schedd[129]: candidate limit: "" 2021-06-17T10:07:19.109818151Z condor_schedd[129]: ConcurrencyLimits match, can reuse claim 2021-06-17T10:07:19.109823233Z condor_schedd[129]: Leaving ded-schedd concurrency limit check... 2021-06-17T10:07:19.109892845Z condor_schedd[129]: satisfyJobs: 2.0 satisfied with slot slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.109912193Z condor_schedd[129]: satisfyJobs: finding resources for 2.1 2021-06-17T10:07:19.109965507Z condor_schedd[129]: Entering ded-schedd concurrency limit check... 2021-06-17T10:07:19.109978547Z condor_schedd[129]: job limit: "" 2021-06-17T10:07:19.109998830Z condor_schedd[129]: candidate limit: "" 2021-06-17T10:07:19.110019473Z condor_schedd[129]: ConcurrencyLimits match, can reuse claim 2021-06-17T10:07:19.110029151Z condor_schedd[129]: Leaving ded-schedd concurrency limit check... 2021-06-17T10:07:19.110072889Z condor_schedd[129]: satisfyJobs: 2.1 satisfied with slot slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.110096637Z condor_schedd[129]: satisfyJobs: finding resources for 2.2 2021-06-17T10:07:19.110275214Z condor_schedd[129]: Entering ded-schedd concurrency limit check... 2021-06-17T10:07:19.110293908Z condor_schedd[129]: job limit: "" 2021-06-17T10:07:19.110303047Z condor_schedd[129]: candidate limit: "" 2021-06-17T10:07:19.110310484Z condor_schedd[129]: ConcurrencyLimits match, can reuse claim 2021-06-17T10:07:19.110318758Z condor_schedd[129]: Leaving ded-schedd concurrency limit check... 2021-06-17T10:07:19.110388995Z condor_schedd[129]: satisfyJobs: 2.2 satisfied with slot slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.110405509Z condor_schedd[129]: satisfyJobs: finding resources for 2.3 2021-06-17T10:07:19.110508692Z condor_schedd[129]: Entering ded-schedd concurrency limit check... 2021-06-17T10:07:19.110558092Z condor_schedd[129]: job limit: "" 2021-06-17T10:07:19.110568273Z condor_schedd[129]: candidate limit: "" 2021-06-17T10:07:19.110578329Z condor_schedd[129]: ConcurrencyLimits match, can reuse claim 2021-06-17T10:07:19.110586759Z condor_schedd[129]: Leaving ded-schedd concurrency limit check... 2021-06-17T10:07:19.110602614Z condor_schedd[129]: satisfyJobs: 2.3 satisfied with slot slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.110610608Z condor_schedd[129]: satisfyJobs: finding resources for 2.4 2021-06-17T10:07:19.110749056Z condor_schedd[129]: Entering ded-schedd concurrency limit check... 2021-06-17T10:07:19.110765305Z condor_schedd[129]: job limit: "" 2021-06-17T10:07:19.110773835Z condor_schedd[129]: candidate limit: "" 2021-06-17T10:07:19.110781614Z condor_schedd[129]: ConcurrencyLimits match, can reuse claim 2021-06-17T10:07:19.110858617Z condor_schedd[129]: Leaving ded-schedd concurrency limit check... 2021-06-17T10:07:19.110872746Z condor_schedd[129]: satisfyJobs: 2.4 satisfied with slot slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.110882591Z condor_schedd[129]: satisfyJobs: jobs were satisfied 2021-06-17T10:07:19.110891305Z condor_schedd[129]: Satisfied job 2 with 5 idle resources 2021-06-17T10:07:19.111009310Z condor_schedd[129]: Allocation for job 2.0, nprocs: 5 2021-06-17T10:07:19.111026931Z condor_schedd[129]: 2.0.0: LINUX#011X86_64#011slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.111045132Z condor_schedd[129]: 2.1.0: LINUX#011X86_64#011slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.111053149Z condor_schedd[129]: 2.2.0: LINUX#011X86_64#011slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.111119062Z condor_schedd[129]: 2.3.0: LINUX#011X86_64#011slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.111133655Z condor_schedd[129]: 2.4.0: LINUX#011X86_64#011slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:19.111150116Z condor_schedd[129]: DedicateScheduler::spawnJobs() - job=2.0 on <10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> 2021-06-17T10:07:19.116544202Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:19.117833932Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:19.117852411Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:19.117857806Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:19.117862164Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000092; send_time=0.000086; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:38995>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:19.122380993Z condor_schedd[129]: Queueing job 2.0 in runnable job queue 2021-06-17T10:07:19.122402699Z condor_schedd[129]: start next job after 0 sec, JobsThisBurst 0 2021-06-17T10:07:19.122410661Z condor_schedd[129]: In DedicatedScheduler::publishRequestAd() 2021-06-17T10:07:19.122477741Z condor_schedd[129]: Adding submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor to the submitter map for default pool. 2021-06-17T10:07:19.122489718Z condor_schedd[129]: Trying to update collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:19.122494524Z condor_schedd[129]: Attempting to send update via TCP to collector pseven-htcondormanager-svc.default.svc.cluster.local <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:19.122666051Z condor_schedd[129]: Entering DedicatedScheduler::checkSanity() 2021-06-17T10:07:19.122784994Z condor_collector[51]: SubmittorAd : Updating ... "< DedicatedScheduler@parallel_schedd@submit.pseven-htcondorparallel_schedd@xxxxxxxxxxxxxxxxxxxxxx , 10.42.0.213 >" 2021-06-17T10:07:19.122893135Z condor_schedd[129]: Finished DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:07:19.123150637Z condor_schedd[129]: Job prep for 2.0 will not block, calling aboutToSpawnJobHandler() directly 2021-06-17T10:07:19.123418440Z condor_schedd[129]: aboutToSpawnJobHandler() completed for job 2.0, attempting to spawn job handler 2021-06-17T10:07:19.123781199Z condor_schedd[129]: Starting add_shadow_birthdate(2.0) 2021-06-17T10:07:19.124495265Z condor_schedd[129]: $$ substitution: Cmd="/Users/1/occupy-cluster-4x500.p7wf/@Runs/#00000057.p7run/.p7run.tmp/.start-f0d98f20345b4b4aa0c60dfdb7cac7df.LINUX.sh.bat" 2021-06-17T10:07:19.125247593Z condor_schedd[129]: Create_Process: using fast clone() to create child process. 2021-06-17T10:07:19.128710599Z condor_schedd[129]: Added shadow record for PID 609, job (2.0) 2021-06-17T10:07:19.128935944Z condor_schedd[129]: Started shadow for job 2.0 on slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3>#1623924088#1#... for DedicatedScheduler@parallel_schedd, (shadow pid = 609) 2021-06-17T10:07:19.130055488Z condor_schedd[129]: Cleared dirty attributes for job 2.0 2021-06-17T10:07:19.131290243Z condor_schedd[129]: Cleared dirty attributes for job 2.1 2021-06-17T10:07:19.132220794Z condor_schedd[129]: Cleared dirty attributes for job 2.2 2021-06-17T10:07:19.132547378Z condor_shadow[609]: ****************************************************** 2021-06-17T10:07:19.132560558Z condor_shadow[609]: ** condor_shadow (CONDOR_SHADOW) STARTING UP 2021-06-17T10:07:19.132566313Z condor_shadow[609]: ** /usr/sbin/condor_shadow 2021-06-17T10:07:19.132569459Z condor_shadow[609]: ** SubsystemInfo: name=SHADOW type=SHADOW(6) class=DAEMON(1) 2021-06-17T10:07:19.132583558Z condor_shadow[609]: ** Configuration: subsystem:SHADOW local:<NONE> class:DAEMON 2021-06-17T10:07:19.132596669Z condor_shadow[609]: ** $CondorVersion: 8.9.11 Dec 29 2020 BuildID: Debian-8.9.11-1.2 PackageID: 8.9.11-1.2 Debian-8.9.11-1.2 $ 2021-06-17T10:07:19.132600965Z condor_shadow[609]: ** $CondorPlatform: X86_64-Ubuntu_20.04 $ 2021-06-17T10:07:19.132603999Z condor_shadow[609]: ** PID = 609 2021-06-17T10:07:19.132631483Z condor_shadow[609]: ** Log last touched time unavailable (Success) 2021-06-17T10:07:19.132636506Z condor_shadow[609]: ****************************************************** 2021-06-17T10:07:19.132639220Z condor_shadow[609]: Using config source: /etc/condor/condor_config 2021-06-17T10:07:19.132641760Z condor_shadow[609]: Using local config sources: 2021-06-17T10:07:19.132644218Z condor_shadow[609]: /etc/condor/condor_config.local 2021-06-17T10:07:19.132646843Z condor_shadow[609]: config Macros = 119, Sorted = 119, StringBytes = 2891, TablesBytes = 1952 2021-06-17T10:07:19.132673695Z condor_shadow[609]: CLASSAD_CACHING is OFF 2021-06-17T10:07:19.132725270Z condor_shadow[609]: Daemon Log is logging: D_ALWAYS D_ERROR D_STATS 2021-06-17T10:07:19.132812507Z condor_shadow[609]: SharedPortEndpoint: waiting for connections to named socket shadow_129_a45e_2 2021-06-17T10:07:19.133354773Z condor_schedd[129]: Cleared dirty attributes for job 2.3 2021-06-17T10:07:19.133489831Z condor_shadow[609]: DaemonCore: command socket at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=shadow_129_a45e_2> 2021-06-17T10:07:19.133626311Z condor_shadow[609]: DaemonCore: private command socket at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=shadow_129_a45e_2> 2021-06-17T10:07:19.134602719Z condor_schedd[129]: Cleared dirty attributes for job 2.4 2021-06-17T10:07:19.134623257Z condor_schedd[129]: Job prep for 2.1 will not block, calling aboutToSpawnJobHandler() directly 2021-06-17T10:07:19.135130318Z condor_schedd[129]: aboutToSpawnJobHandler() completed for job 2.1, attempting to spawn job handler 2021-06-17T10:07:19.135143783Z condor_schedd[129]: Job prep for 2.2 will not block, calling aboutToSpawnJobHandler() directly 2021-06-17T10:07:19.135148191Z condor_schedd[129]: aboutToSpawnJobHandler() completed for job 2.2, attempting to spawn job handler 2021-06-17T10:07:19.135151659Z condor_schedd[129]: Job prep for 2.3 will not block, calling aboutToSpawnJobHandler() directly 2021-06-17T10:07:19.135238547Z condor_schedd[129]: aboutToSpawnJobHandler() completed for job 2.3, attempting to spawn job handler 2021-06-17T10:07:19.135247932Z condor_schedd[129]: Job prep for 2.4 will not block, calling aboutToSpawnJobHandler() directly 2021-06-17T10:07:19.135422664Z condor_schedd[129]: aboutToSpawnJobHandler() completed for job 2.4, attempting to spawn job handler 2021-06-17T10:07:19.135433014Z condor_schedd[129]: Expedited call to StartJobs() 2021-06-17T10:07:19.136322431Z condor_schedd[129]: -------- Begin starting jobs -------- 2021-06-17T10:07:19.136339736Z condor_schedd[129]: -------- Done starting jobs -------- 2021-06-17T10:07:19.136355428Z condor_shadow[609]: Initializing a PARALLEL shadow for job 2.0 2021-06-17T10:07:19.136359401Z condor_shadow[609]: LIMIT_DIRECTORY_ACCESS = <unset> 2021-06-17T10:07:19.136367644Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:19.136404849Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 610 2021-06-17T10:07:19.136420126Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:19.146459538Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:19.146670184Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:19.147252350Z condor_collector[51]: QueryWorker: forked new worker with id 337 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:19.147310878Z condor_collector[337]: (Sending 7 ads in response to query) 2021-06-17T10:07:19.147522252Z condor_collector[337]: Query info: matched=7; skipped=0; query_time=0.000355; send_time=0.000193; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:45169>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:19.148022521Z condor_collector[51]: QueryWorker: Child 337 done 2021-06-17T10:07:20.139231267Z condor_schedd[129]: Entering DedicatedScheduler::giveMatches() 2021-06-17T10:07:20.139270121Z condor_schedd[129]: Pushing 5 proc(s) for cluster 2 2021-06-17T10:07:20.139274946Z condor_schedd[129]: In proc 0, num_matches: 1 2021-06-17T10:07:20.139528730Z condor_schedd[129]: $$ substitution: Cmd="/Users/1/occupy-cluster-4x500.p7wf/@Runs/#00000057.p7run/.p7run.tmp/.start-f0d98f20345b4b4aa0c60dfdb7cac7df.LINUX.sh.bat" 2021-06-17T10:07:20.143286759Z condor_schedd[129]: In proc 1, num_matches: 1 2021-06-17T10:07:20.143359240Z condor_schedd[129]: $$ substitution: Cmd="/Users/1/occupy-cluster-4x500.p7wf/@Runs/#00000057.p7run/.p7run.tmp/.start-f0d98f20345b4b4aa0c60dfdb7cac7df.LINUX.sh.bat" 2021-06-17T10:07:20.143365862Z condor_schedd[129]: $$ substitution: TransferInput=".start-block.LINUX.sh.bat,.sidecar.yaml,.filesgate.key,sidecar,.start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.$$(OPSYS).sh.bat" 2021-06-17T10:07:20.143369770Z condor_schedd[129]: $$ substitution: TransferInput=".start-block.LINUX.sh.bat,.sidecar.yaml,.filesgate.key,sidecar,.start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.LINUX.sh.bat" 2021-06-17T10:07:20.143373578Z condor_schedd[129]: $$ substitution: Arguments="./.start-block.LINUX.sh.bat .start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.$$(OPSYS).sh.bat" 2021-06-17T10:07:20.143377170Z condor_schedd[129]: $$ substitution: Arguments="./.start-block.LINUX.sh.bat .start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.LINUX.sh.bat" 2021-06-17T10:07:20.143381064Z condor_schedd[129]: In proc 2, num_matches: 1 2021-06-17T10:07:20.143383809Z condor_schedd[129]: $$ substitution: Cmd="/Users/1/occupy-cluster-4x500.p7wf/@Runs/#00000057.p7run/.p7run.tmp/.start-f0d98f20345b4b4aa0c60dfdb7cac7df.LINUX.sh.bat" 2021-06-17T10:07:20.143387157Z condor_schedd[129]: $$ substitution: Arguments="./.start-block.LINUX.sh.bat .start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.$$(OPSYS).sh.bat" 2021-06-17T10:07:20.143390976Z condor_schedd[129]: $$ substitution: Arguments="./.start-block.LINUX.sh.bat .start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.LINUX.sh.bat" 2021-06-17T10:07:20.143394104Z condor_schedd[129]: $$ substitution: TransferInput=".start-block.LINUX.sh.bat,.sidecar.yaml,.filesgate.key,sidecar,.start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.$$(OPSYS).sh.bat" 2021-06-17T10:07:20.143397755Z condor_schedd[129]: $$ substitution: TransferInput=".start-block.LINUX.sh.bat,.sidecar.yaml,.filesgate.key,sidecar,.start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.LINUX.sh.bat" 2021-06-17T10:07:20.143401042Z condor_schedd[129]: In proc 3, num_matches: 1 2021-06-17T10:07:20.143403924Z condor_schedd[129]: $$ substitution: Cmd="/Users/1/occupy-cluster-4x500.p7wf/@Runs/#00000057.p7run/.p7run.tmp/.start-f0d98f20345b4b4aa0c60dfdb7cac7df.LINUX.sh.bat" 2021-06-17T10:07:20.143422498Z condor_schedd[129]: $$ substitution: TransferInput=".start-block.LINUX.sh.bat,.sidecar.yaml,.filesgate.key,sidecar,.start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.$$(OPSYS).sh.bat" 2021-06-17T10:07:20.143426123Z condor_schedd[129]: $$ substitution: TransferInput=".start-block.LINUX.sh.bat,.sidecar.yaml,.filesgate.key,sidecar,.start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.LINUX.sh.bat" 2021-06-17T10:07:20.143429345Z condor_schedd[129]: $$ substitution: Arguments="./.start-block.LINUX.sh.bat .start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.$$(OPSYS).sh.bat" 2021-06-17T10:07:20.143432345Z condor_schedd[129]: $$ substitution: Arguments="./.start-block.LINUX.sh.bat .start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.LINUX.sh.bat" 2021-06-17T10:07:20.143435291Z condor_schedd[129]: In proc 4, num_matches: 1 2021-06-17T10:07:20.143437961Z condor_schedd[129]: $$ substitution: Cmd="/Users/1/occupy-cluster-4x500.p7wf/@Runs/#00000057.p7run/.p7run.tmp/.start-f0d98f20345b4b4aa0c60dfdb7cac7df.LINUX.sh.bat" 2021-06-17T10:07:20.143441342Z condor_schedd[129]: $$ substitution: TransferInput=".start-block.LINUX.sh.bat,.sidecar.yaml,.filesgate.key,sidecar,.start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.$$(OPSYS).sh.bat" 2021-06-17T10:07:20.143753550Z condor_schedd[129]: $$ substitution: TransferInput=".start-block.LINUX.sh.bat,.sidecar.yaml,.filesgate.key,sidecar,.start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.LINUX.sh.bat" 2021-06-17T10:07:20.143770387Z condor_schedd[129]: $$ substitution: Arguments="./.start-block.LINUX.sh.bat .start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.$$(OPSYS).sh.bat" 2021-06-17T10:07:20.143775972Z condor_schedd[129]: $$ substitution: Arguments="./.start-block.LINUX.sh.bat .start-protoblock-9993a284a9e7411987858ed56ae49f6e-7.LINUX.sh.bat" 2021-06-17T10:07:20.148145235Z condor_startd[122]: Error: can't find resource with ClaimId (<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3>#1623924088#1#...) for 444 (ACTIVATE_CLAIM) 2021-06-17T10:07:20.148148117Z condor_shadow[609]: Request to run on slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx <10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3> was REFUSED 2021-06-17T10:07:20.148167143Z condor_shadow[609]: Job 2.0 is being evicted 2021-06-17T10:07:20.148182466Z condor_startd[122]: Failed to read end of message from <10.42.0.213:42237>; 1700 untouched bytes. 2021-06-17T10:07:20.149400078Z condor_startd[122]: Error: can't find resource with ClaimId (<10.42.0.202:38495?addrs=10.42.0.202-38495&alias=pseven-htcondorexecute-deploy-68b66d9c55-4cb26.pseven-htcondor&noUDP&sock=startd_86_e1a3>#1623924088#1#...) -- perhaps this claim was already removed? 2021-06-17T10:07:20.149420832Z condor_startd[122]: Error: problem finding resource for 403 (DEACTIVATE_CLAIM) 2021-06-17T10:07:20.149604067Z condor_shadow[609]: RemoteResource::killStarter(): Could not send command to startd 2021-06-17T10:07:20.150230079Z condor_startd[122]: Failed to write ToE tag to .job.ad file (2): No such file or directory 2021-06-17T10:07:20.150243992Z condor_startd[122]: slot1_2: Called deactivate_claim() 2021-06-17T10:07:20.150898139Z condor_startd[122]: Failed to write ToE tag to .job.ad file (2): No such file or directory 2021-06-17T10:07:20.150910910Z condor_startd[122]: slot1_3: Called deactivate_claim() 2021-06-17T10:07:20.151142335Z condor_startd[122]: Failed to write ToE tag to .job.ad file (2): No such file or directory 2021-06-17T10:07:20.151177453Z condor_startd[122]: slot1_4: Called deactivate_claim() 2021-06-17T10:07:20.152253873Z condor_startd[125]: Failed to write ToE tag to .job.ad file (2): No such file or directory 2021-06-17T10:07:20.152360379Z condor_startd[125]: slot1_1: Called deactivate_claim() 2021-06-17T10:07:20.153171876Z condor_shadow[609]: logEvictEvent with unknown reason (108), not logging. 2021-06-17T10:07:20.156761779Z condor_schedd[129]: QMGR Connection closed 2021-06-17T10:07:20.159612621Z condor_shadow[609]: **** condor_shadow (condor_SHADOW) pid 609 EXITING WITH STATUS 108 2021-06-17T10:07:20.159634031Z condor_schedd[129]: In DedicatedScheduler::reaper pid 609 has status 27648 2021-06-17T10:07:20.159638719Z condor_schedd[129]: Started timer (70) to call handleDedicatedJobs() in 2 secs 2021-06-17T10:07:20.159642013Z condor_schedd[129]: Shadow pid 609 exited with status 108 2021-06-17T10:07:20.159645488Z condor_schedd[129]: DedicatedScheduler::shutdownMpiJob, cluster 2 2021-06-17T10:07:20.159648762Z condor_schedd[129]: Dedicated job abnormally ended, releasing claim 2021-06-17T10:07:20.159981697Z condor_schedd[129]: SharedPortClient: sent connection request to <10.42.0.202:38495> for shared port id startd_86_e1a3 2021-06-17T10:07:20.159997121Z condor_schedd[129]: DedicatedScheduler: releasing claim on slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:20.160001320Z condor_schedd[129]: Deleted match rec for slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:20.160004717Z condor_schedd[129]: Dedicated job abnormally ended, releasing claim 2021-06-17T10:07:20.160184340Z condor_startd[122]: Can't read ClaimId 2021-06-17T10:07:20.160202394Z condor_startd[122]: Failed to read end of message from <10.42.0.213:42447>; 244 untouched bytes. 2021-06-17T10:07:20.160207296Z condor_startd[122]: condor_write(): Socket closed when trying to write 29 bytes to <10.42.0.213:42447>, fd is 10 2021-06-17T10:07:20.160214621Z condor_startd[122]: Buf::write(): condor_write() failed 2021-06-17T10:07:20.162700332Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:20.162728095Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:20.162732866Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:20.162736421Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:20.162739964Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000031; send_time=0.000044; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:40807>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:20.162766323Z condor_schedd[129]: SharedPortClient: sent connection request to <10.42.0.202:38495> for shared port id startd_86_e1a3 2021-06-17T10:07:20.162777054Z condor_schedd[129]: DedicatedScheduler: releasing claim on slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:20.162780761Z condor_schedd[129]: Deleted match rec for slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:20.162784024Z condor_schedd[129]: Dedicated job abnormally ended, releasing claim 2021-06-17T10:07:20.162787327Z condor_schedd[129]: SharedPortClient: sent connection request to <10.42.0.202:38495> for shared port id startd_86_e1a3 2021-06-17T10:07:20.162790826Z condor_schedd[129]: DedicatedScheduler: releasing claim on slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:20.162794435Z condor_schedd[129]: Deleted match rec for slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:20.162798026Z condor_schedd[129]: Dedicated job abnormally ended, releasing claim 2021-06-17T10:07:20.162801170Z condor_schedd[129]: SharedPortClient: sent connection request to <10.42.0.202:38495> for shared port id startd_86_e1a3 2021-06-17T10:07:20.162804980Z condor_schedd[129]: DedicatedScheduler: releasing claim on slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:20.162808767Z condor_schedd[129]: Deleted match rec for slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:20.162812342Z condor_schedd[129]: Dedicated job abnormally ended, releasing claim 2021-06-17T10:07:20.162815621Z condor_schedd[129]: SharedPortClient: sent connection request to <10.42.0.201:32819> for shared port id startd_89_f06c 2021-06-17T10:07:20.162867450Z condor_startd[122]: Can't read ClaimId 2021-06-17T10:07:20.162897903Z condor_startd[122]: Failed to read end of message from <10.42.0.213:42339>; 246 untouched bytes. 2021-06-17T10:07:20.162901720Z condor_startd[122]: Can't read ClaimId 2021-06-17T10:07:20.162905156Z condor_startd[122]: Failed to read end of message from <10.42.0.213:43349>; 246 untouched bytes. 2021-06-17T10:07:20.162908743Z condor_startd[122]: condor_write(): Socket closed when trying to write 29 bytes to <10.42.0.213:43349>, fd is 10 2021-06-17T10:07:20.162912400Z condor_startd[122]: Buf::write(): condor_write() failed 2021-06-17T10:07:20.162915891Z condor_startd[122]: Can't read ClaimId 2021-06-17T10:07:20.162919061Z condor_startd[122]: Failed to read end of message from <10.42.0.213:45603>; 246 untouched bytes. 2021-06-17T10:07:20.162922053Z condor_startd[122]: condor_write(): Socket closed when trying to write 29 bytes to <10.42.0.213:45603>, fd is 10 2021-06-17T10:07:20.162925687Z condor_startd[122]: Buf::write(): condor_write() failed 2021-06-17T10:07:20.164062293Z condor_startd[125]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:20.164902201Z condor_startd[125]: Can't read ClaimId 2021-06-17T10:07:20.164920105Z condor_startd[125]: Failed to read end of message from <10.42.0.213:36075>; 245 untouched bytes. 2021-06-17T10:07:20.165723001Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::unwrap. 2021-06-17T10:07:20.165739549Z condor_schedd[129]: DedicatedScheduler: releasing claim on slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:20.165744984Z condor_schedd[129]: Deleted match rec for slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:20.165748866Z condor_schedd[129]: DedicatedScheduler::removeAllocation, cluster 2 2021-06-17T10:07:20.165752363Z condor_schedd[129]: Deleting shadow rec for PID 609, job (2.0) 2021-06-17T10:07:20.165755556Z condor_schedd[129]: Entered check_zombie( 609, 0x0x56280791b044, st=2 ) 2021-06-17T10:07:20.165758880Z condor_schedd[129]: Marked job 2.0 as IDLE 2021-06-17T10:07:20.165773782Z condor_schedd[129]: Marked job 2.1 as IDLE 2021-06-17T10:07:20.165777762Z condor_schedd[129]: Marked job 2.2 as IDLE 2021-06-17T10:07:20.165781041Z condor_schedd[129]: Marked job 2.3 as IDLE 2021-06-17T10:07:20.165784021Z condor_schedd[129]: Marked job 2.4 as IDLE 2021-06-17T10:07:20.165786215Z condor_schedd[129]: Exited check_zombie( 609, 0x0x56280791b044 ) 2021-06-17T10:07:20.165788376Z condor_schedd[129]: Exited delete_shadow_rec( 609 ) 2021-06-17T10:07:20.166311528Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:20.167049142Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 615 2021-06-17T10:07:20.167063248Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:20.177975837Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:20.178206582Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:20.178881030Z condor_collector[51]: QueryWorker: forked new worker with id 338 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:20.179378327Z condor_collector[338]: (Sending 7 ads in response to query) 2021-06-17T10:07:20.179391642Z condor_collector[338]: Query info: matched=7; skipped=0; query_time=0.000300; send_time=0.000215; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:40265>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:20.179827985Z condor_collector[51]: QueryWorker: Child 338 done 2021-06-17T10:07:20.203554424Z condor_negotiator[52]: ---------- Started Negotiation Cycle ---------- 2021-06-17T10:07:20.203590118Z condor_negotiator[52]: Phase 1: Obtaining ads from collector ... 2021-06-17T10:07:20.203596982Z condor_negotiator[52]: Getting startd private ads ... 2021-06-17T10:07:20.203601074Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:07:20.204230803Z condor_collector[51]: Got QUERY_STARTD_PVT_ADS 2021-06-17T10:07:20.206107145Z condor_collector[51]: QueryWorker: forked new high priority worker with id 339 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:20.206129616Z condor_collector[339]: (Sending 7 ads in response to query) 2021-06-17T10:07:20.206134701Z condor_collector[339]: Query info: matched=7; skipped=0; query_time=0.000257; send_time=0.000875; type=MachinePrivate; requirements={true}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:38365>; projection={}; filter_private_ads=0 2021-06-17T10:07:20.206207331Z condor_collector[51]: QueryWorker: Child 339 done 2021-06-17T10:07:20.206279633Z condor_negotiator[52]: Getting Scheduler, Submitter and Machine ads ... 2021-06-17T10:07:20.206289954Z condor_negotiator[52]: Trying to query collector <10.42.0.205:19618> 2021-06-17T10:07:20.206719954Z condor_collector[51]: Got QUERY_ANY_ADS 2021-06-17T10:07:20.207202413Z condor_collector[51]: QueryWorker: forked new high priority worker with id 340 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:20.207213630Z condor_collector[340]: (Sending 9 ads in response to query) 2021-06-17T10:07:20.208374711Z condor_collector[340]: Query info: matched=9; skipped=9; query_time=0.000250; send_time=0.001065; type=Any; requirements={(((MyType == "Submitter")) || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR; peer=<10.42.0.205:43337>; projection={}; filter_private_ads=0 2021-06-17T10:07:20.208756597Z condor_collector[51]: QueryWorker: Child 340 done 2021-06-17T10:07:20.209691400Z condor_negotiator[52]: Sorting 9 ads ... 2021-06-17T10:07:20.209884978Z condor_negotiator[52]: Ignoring submitter DedicatedScheduler@parallel_schedd@submit.pseven-htcondor from schedd at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> with no requested jobs 2021-06-17T10:07:20.209915583Z condor_negotiator[52]: Ignoring submitter user20001@pseven-htcondor from schedd at <10.42.0.213:42327?addrs=10.42.0.213-42327&alias=submit.pseven-htcondor&noUDP&sock=schedd_87_913c> with no requested jobs 2021-06-17T10:07:20.210542008Z condor_negotiator[52]: Got ads: 9 public and 7 private 2021-06-17T10:07:20.210554502Z condor_negotiator[52]: Public ads include 0 submitter, 7 startd 2021-06-17T10:07:20.210644904Z condor_negotiator[52]: Phase 2: Performing accounting ... 2021-06-17T10:07:20.210655356Z condor_negotiator[52]: Entering compute_significant_attrs() 2021-06-17T10:07:20.211224597Z condor_negotiator[52]: Leaving compute_significant_attrs() - result=_cp_orig_RequestCpus,_cp_orig_RequestDisk,_cp_orig_RequestMemory,KFlops,MachineLastMatchTime,Offline,RequestCpus,RequestDisk,RequestMemory,Scheduler,TotalJobRuntime 2021-06-17T10:07:20.212010406Z condor_negotiator[52]: Phase 3: Sorting submitter ads by priority ... 2021-06-17T10:07:20.212020387Z condor_negotiator[52]: Starting prefetch round; 0 potential prefetches to do. 2021-06-17T10:07:20.212154037Z condor_negotiator[52]: Assigned 0 units of work for prefetching. 2021-06-17T10:07:20.212164539Z condor_negotiator[52]: Prefetch summary: 0 attempted, 0 successful. 2021-06-17T10:07:20.212169336Z condor_negotiator[52]: Phase 4.1: Negotiating with schedds ... 2021-06-17T10:07:20.212172918Z condor_negotiator[52]: numSlots = 7 (after trimming=7) 2021-06-17T10:07:20.212176347Z condor_negotiator[52]: slotWeightTotal = 200.000000 2021-06-17T10:07:20.212180006Z condor_negotiator[52]: minSlotWeight = 0.000000 2021-06-17T10:07:20.212183557Z condor_negotiator[52]: pieLeft = 0.000 2021-06-17T10:07:20.212197648Z condor_negotiator[52]: NormalFactor = 0.000000 2021-06-17T10:07:20.212202515Z condor_negotiator[52]: MaxPrioValue = 0.000000 2021-06-17T10:07:20.212205648Z condor_negotiator[52]: NumSubmitterAds = 0 2021-06-17T10:07:20.212209118Z condor_negotiator[52]: resources used scheddUsed= 0.000000 2021-06-17T10:07:20.212212438Z condor_negotiator[52]: negotiateWithGroup resources used submitterAds length 0 2021-06-17T10:07:20.212216323Z condor_negotiator[52]: ---------- Finished Negotiation Cycle ---------- 2021-06-17T10:07:20.212219592Z condor_negotiator[52]: Updating collector with accounting information 2021-06-17T10:07:20.212230847Z condor_negotiator[52]: Trying to update collector <10.42.0.205:19618> 2021-06-17T10:07:20.212239993Z condor_negotiator[52]: Attempting to send update via TCP to collector <10.42.0.205:19618> 2021-06-17T10:07:20.212321458Z condor_negotiator[52]: Done Updating collector with accounting information 2021-06-17T10:07:20.212422130Z condor_collector[51]: AccountingAd : Updating ... "< <none>negotiator@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >" 2021-06-17T10:07:21.190355026Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:21.190621341Z condor_collector[51]: Got QUERY_SCHEDD_ADS 2021-06-17T10:07:21.190637490Z condor_collector[51]: QueryWorker: about to handle query in-process 2021-06-17T10:07:21.190642182Z condor_collector[51]: (Sending 1 ads in response to query) 2021-06-17T10:07:21.190709159Z condor_collector[51]: Query info: matched=1; skipped=0; query_time=0.000029; send_time=0.000037; type=Scheduler; requirements={((TotalRunningJobs > 0 || TotalIdleJobs > 0 || TotalHeldJobs > 0 || TotalRemovedJobs > 0 || TotalJobAds > 0))}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:33267>; projection={ScheddIpAddr CondorVersion Name Machine}; filter_private_ads=0 2021-06-17T10:07:21.191876299Z condor_schedd[129]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:21.192581752Z condor_schedd[129]: ForkWorker::Fork: New child of 129 = 620 2021-06-17T10:07:21.192601586Z condor_schedd[129]: Number of Active Workers 0 2021-06-17T10:07:21.201757373Z condor_collector[51]: ZKM: In Condor_Auth_Passwd::wrap. 2021-06-17T10:07:21.201999338Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:21.203239697Z condor_collector[51]: QueryWorker: forked new worker with id 341 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:21.203259506Z condor_collector[341]: (Sending 7 ads in response to query) 2021-06-17T10:07:21.203265071Z condor_collector[341]: Query info: matched=7; skipped=0; query_time=0.000310; send_time=0.000145; type=Machine; requirements={true}; locate=0; limit=0; from=TOOL; peer=<10.42.0.213:37511>; projection={Activity Cpus Machine Memory Name OpSys State}; filter_private_ads=0 2021-06-17T10:07:21.203306477Z condor_collector[51]: QueryWorker: Child 341 done 2021-06-17T10:07:22.194918254Z condor_schedd[129]: Starting DedicatedScheduler::handleDedicatedJobs 2021-06-17T10:07:22.194956651Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:07:22.194961621Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:07:22.195042664Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:07:22.195056439Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:07:22.195116107Z condor_schedd[129]: SetAttribute modifying attribute Scheduler in nonexistent job 2.5 2021-06-17T10:07:22.195123194Z condor_schedd[129]: Found 1 idle dedicated job(s) 2021-06-17T10:07:22.195126344Z condor_schedd[129]: DedicatedScheduler: Listing all dedicated jobs - 2021-06-17T10:07:22.195129248Z condor_schedd[129]: Dedicated job: 2.0 user20001 2021-06-17T10:07:22.195160693Z condor_schedd[129]: Trying to query collector <10.43.129.19:19618?alias=pseven-htcondormanager-svc.default.svc.cluster.local> 2021-06-17T10:07:22.195964400Z condor_collector[51]: Got QUERY_STARTD_ADS 2021-06-17T10:07:22.196625270Z condor_collector[342]: (Sending 7 ads in response to query) 2021-06-17T10:07:22.196643180Z condor_collector[51]: QueryWorker: forked new worker with id 342 ( max 4 active 1 pending 0 ) 2021-06-17T10:07:22.197698391Z condor_collector[342]: Query info: matched=7; skipped=0; query_time=0.000342; send_time=0.001066; type=Machine; requirements={((DedicatedScheduler == "DedicatedScheduler@parallel_schedd@submit.pseven-htcondor"))}; locate=0; limit=0; from=SCHEDD; peer=<10.42.0.213:43259>; projection={}; filter_private_ads=0 2021-06-17T10:07:22.198254900Z condor_collector[51]: QueryWorker: Child 342 done 2021-06-17T10:07:22.198657831Z condor_schedd[129]: Found 7 potential dedicated resources in 0 seconds 2021-06-17T10:07:22.198670562Z condor_schedd[129]: New dynamic slot slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:22.198779874Z condor_schedd[129]: New dynamic slot slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:22.198791027Z condor_schedd[129]: WARNING: claim id not found for new dynamic slot slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -- ignoring this resource 2021-06-17T10:07:22.198793817Z condor_schedd[129]: New dynamic slot slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:22.198796582Z condor_schedd[129]: WARNING: claim id not found for new dynamic slot slot1_4@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -- ignoring this resource 2021-06-17T10:07:22.198815535Z condor_schedd[129]: New dynamic slot slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:22.199115207Z condor_schedd[129]: WARNING: claim id not found for new dynamic slot slot1_3@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -- ignoring this resource 2021-06-17T10:07:22.199124642Z condor_schedd[129]: New dynamic slot slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:22.199129214Z condor_schedd[129]: WARNING: claim id not found for new dynamic slot slot1_2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -- ignoring this resource 2021-06-17T10:07:22.199133232Z condor_schedd[129]: Duplicate x0 (0/0) partitionable res slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 2021-06-17T10:07:22.199136946Z condor_schedd[129]: Duplicate x75 (75/0) partitionable res slot1@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ...

Mailing List Archives

Authenticated access

[HTCondor-users] HTCondor can't execute the job with error: Error: can't find resource with ClaimId