Re: [HTCondor-users] materialized jobs and transforms

[Ben Jones <ben.dylan.jones@xxxxxxx>]

Hi Joe,

Would love to hear the recommended method to detect whether itâs the factory stage, especially if itâs less clunky than my current workaround of adding an extra isUndefined() to the Requirements of each of these transforms.

cheers,

Ben

On 23 May 2024, at 23:12, Joe Reuss via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:

Hi Ben!

The answer is yes, transforms did change from v9.0.17-v23.0.3. Transforms happen more than once now, they happen for the factory as well as for the job.

There is a way to detect in the transform whether we are transforming a factory or a job and if you need more assistance on configuration and such, feel free to let us know!

Best,

Joe Reuss

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Ben Jones <ben.dylan.jones@xxxxxxx>
Sent: Tuesday, May 21, 2024 9:43 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] materialized jobs and transforms

 

Hi,

Noticed something on our 23.0.3 schedds that doesnât seem to happen on our 9.0.17 schedds. Do transforms get handled differently for materialized jobs now?

Among other things, we use transforms to manipulate job requirements, and we tend to compound them, ie we might have a chain of âDefaultToEl9â, âCvmfsRequirementâ and âAFSJobâ that might end up with a set of requirements that look like:

Requirements = OpSysAndVer =?= âAlmaLinux9â && PreVerRequirements 
PreVerRequirements = CVMFS_HEALTHY =?= True && PreCvmfsRequirements
PreCvmfsRequirements = NODE_MOUNTS_AFS && PreAfsReqjuirements 
PreAfsRequirements = <standard list of vanilla requirements>

Itâs probably evil, but thatâs what we do. 

Now on 23.0.3, weâre seeing some of these become circular, _only_ for materialized jobs, which is obviously not good. Is there some way that transforms are handled differently? I note in the logs, thereâs a -1 step that I donât think I noticed, ie:

05/21/24 15:24:15 (pid:24247) job_transforms for 6867177.-1: 20 considered, 12 applied (DefaultCERNTransform,DefaultToEl9,DockerNetwork,RebalanceMemCpu,AccountingGroupCheck,ChargeGroupCheck,ChargeRoleCheck,CvmfsRequirements,AFSJob,DefaultToLocal,AutoWrapSingularity,AddHome)

Then for the jobs:

05/21/24 15:24:15 (pid:24247) job_transforms for 6867177.0: 20 considered, 9 applied (DefaultCERNTransform,RebalanceMemCpu,AccountingGroupCheck,ChargeGroupCheck,ChargeRoleCheck,CvmfsRequirements,AFSJob,DefaultToLocal,AddHome)
05/21/24 15:24:15 (pid:24247) job_transforms for 6867177.1: 20 considered, 9 applied (DefaultCERNTransform,RebalanceMemCpu,AccountingGroupCheck,ChargeGroupCheck,ChargeRoleCheck,CvmfsRequirements,AFSJob,DefaultToLocal,AddHome)
[âetcâ]

Do transforms happen more than once, or is there something special for requirement for these jobs now?

cheers,
Ben

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/