Re: [HTCondor-users] materialized jobs and transforms

[Joe Reuss <jrreuss@xxxxxxxx>]

Hi Ben!

The answer is yes, transforms did change from v9.0.17-v23.0.3. Transforms happen more than once now, they happen for the factory as well as for the job.

There is a way to detect in the transform whether we are transforming a factory or a job and if you need more assistance on configuration and such, feel free to let us know!

Best,

Joe Reuss

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Ben Jones <ben.dylan.jones@xxxxxxx>
Sent: Tuesday, May 21, 2024 9:43 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] materialized jobs and transforms

 

Hi,

Noticed something on our 23.0.3 schedds that doesn’t seem to happen on our 9.0.17 schedds. Do transforms get handled differently for materialized jobs now?

Among other things, we use transforms to manipulate job requirements, and we tend to compound them, ie we might have a chain of “DefaultToEl9”, “CvmfsRequirement” and “AFSJob” that might end up with a set of requirements that look like:

Requirements = OpSysAndVer =?= “AlmaLinux9” && PreVerRequirements
PreVerRequirements = CVMFS_HEALTHY =?= True && PreCvmfsRequirements
PreCvmfsRequirements = NODE_MOUNTS_AFS && PreAfsReqjuirements
PreAfsRequirements = <standard list of vanilla requirements>

It’s probably evil, but that’s what we do.

Now on 23.0.3, we’re seeing some of these become circular, _only_ for materialized jobs, which is obviously not good. Is there some way that transforms are handled differently? I note in the logs, there’s a -1 step that I don’t think I noticed, ie:

05/21/24 15:24:15 (pid:24247) job_transforms for 6867177.-1: 20 considered, 12 applied (DefaultCERNTransform,DefaultToEl9,DockerNetwork,RebalanceMemCpu,AccountingGroupCheck,ChargeGroupCheck,ChargeRoleCheck,CvmfsRequirements,AFSJob,DefaultToLocal,AutoWrapSingularity,AddHome)

Then for the jobs:

05/21/24 15:24:15 (pid:24247) job_transforms for 6867177.0: 20 considered, 9 applied (DefaultCERNTransform,RebalanceMemCpu,AccountingGroupCheck,ChargeGroupCheck,ChargeRoleCheck,CvmfsRequirements,AFSJob,DefaultToLocal,AddHome)
05/21/24 15:24:15 (pid:24247) job_transforms for 6867177.1: 20 considered, 9 applied (DefaultCERNTransform,RebalanceMemCpu,AccountingGroupCheck,ChargeGroupCheck,ChargeRoleCheck,CvmfsRequirements,AFSJob,DefaultToLocal,AddHome)
[…etc…]

Do transforms happen more than once, or is there something special for requirement for these jobs now?

cheers,
Ben

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/