Re: [HTCondor-users] materialized jobs and transforms

[Ben Jones <ben.dylan.jones@xxxxxxx>]

Thanks TJ, I didnât figure that out from the job ad - though in my case I think since Iâm setting a job attr having an isUndefined() on that attr will do.

Iâm curious though - whatâs the use case for a transform on the factory ? 

On 24 May 2024, at 17:02, John M Knoeller via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:

Hi Ben. 

The factory stage will have a ProcId with a value of 0,  and a JobFactoryId that is non-zero - the JobFactoryId is currently the same value as the ClusterId, but you should not depend on that. 

When it is a materialized job, the JobFactoryId attribute will not be present.  

You can use 

JOB_TRANSFORM_FOO @=end

  If $(MY.JobFactoryId)

       <factory-only-transform-statements>

  else

        <regular-submit-and-job-materialize-statements>

  endif

@end

You can also do this to write a transform that only transforms factories.

JOB_TRANSFORM_FOO @=end

   REQUIREMENTS  JobFactoryId > 0

   ....

-tj

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Ben Jones <ben.dylan.jones@xxxxxxx>
Sent: Thursday, May 23, 2024 4:40 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>

Subject: Re: [HTCondor-users] materialized jobs and transforms

 

Hi Joe,

Would love to hear the recommended method to detect whether itâs the factory stage, especially if itâs less clunky than my current workaround of adding an extra isUndefined() to the Requirements of each of these transforms.

cheers,

Ben

On 23 May 2024, at 23:12, Joe Reuss via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:

Hi Ben!

The answer is yes, transforms did change from v9.0.17-v23.0.3. Transforms happen more than once now, they happen for the factory as well as for the job.

There is a way to detect in the transform whether we are transforming a factory or a job and if you need more assistance on configuration and such, feel free to let us know!

Best,

Joe Reuss

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Ben Jones <ben.dylan.jones@xxxxxxx>
Sent: Tuesday, May 21, 2024 9:43 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] materialized jobs and transforms

 

Hi,

Noticed something on our 23.0.3 schedds that doesnât seem to happen on our 9.0.17 schedds. Do transforms get handled differently for materialized jobs now?

Among other things, we use transforms to manipulate job requirements, and we tend to compound them, ie we might have a chain of âDefaultToEl9â, âCvmfsRequirementâ and âAFSJobâ that might end up with a set of requirements that look like:

Requirements = OpSysAndVer =?= âAlmaLinux9â && PreVerRequirements 
PreVerRequirements = CVMFS_HEALTHY =?= True && PreCvmfsRequirements
PreCvmfsRequirements = NODE_MOUNTS_AFS && PreAfsReqjuirements 
PreAfsRequirements = <standard list of vanilla requirements>

Itâs probably evil, but thatâs what we do. 

Now on 23.0.3, weâre seeing some of these become circular, _only_ for materialized jobs, which is obviously not good. Is there some way that transforms are handled differently? I note in the logs, thereâs a -1 step that I donât think I noticed, ie:

05/21/24 15:24:15 (pid:24247) job_transforms for 6867177.-1: 20 considered, 12 applied (DefaultCERNTransform,DefaultToEl9,DockerNetwork,RebalanceMemCpu,AccountingGroupCheck,ChargeGroupCheck,ChargeRoleCheck,CvmfsRequirements,AFSJob,DefaultToLocal,AutoWrapSingularity,AddHome)

Then for the jobs:

05/21/24 15:24:15 (pid:24247) job_transforms for 6867177.0: 20 considered, 9 applied (DefaultCERNTransform,RebalanceMemCpu,AccountingGroupCheck,ChargeGroupCheck,ChargeRoleCheck,CvmfsRequirements,AFSJob,DefaultToLocal,AddHome)
05/21/24 15:24:15 (pid:24247) job_transforms for 6867177.1: 20 considered, 9 applied (DefaultCERNTransform,RebalanceMemCpu,AccountingGroupCheck,ChargeGroupCheck,ChargeRoleCheck,CvmfsRequirements,AFSJob,DefaultToLocal,AddHome)
[âetcâ]

Do transforms happen more than once, or is there something special for requirement for these jobs now?

cheers,
Ben

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/