Mailing List ArchivesAuthenticated access |
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}
|
Mailing List ArchivesAuthenticated access |
|
|
Yes $STRING(tmp) function will lookup tmp and if tmp has never been set in the variable table, it will use tmp as a literal.
-tj
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sever, Krunoslav <krunoslav.sever@xxxxxxx>
Sent: Saturday, October 16, 2021 7:08 AM To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible? Just noticed this: you actually gave the definition of $STRING(AA) in the mail before, that had totally slipped my mind.
$STRING(AA) will lookup AA, and evaluate the contents, then print the result. This has the side effect of removing quotes from things that evaluate to string values So I might actually be spot-on with my idea on the $STRING during transform, if print = convert to _expression_. But why does the condor_config_val then show raw: SET Z2 = $STRING(tmp) -> expanded: SET Z2 = tmp Lookup of tmp should result in undefined at that stage unless it defaults to "tmp" in that case? Best Kruno ----- Original Message ----- > From: "Krunoslav Sever" <krunoslav.sever@xxxxxxx> > To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx> > Sent: Saturday, 16 October, 2021 11:13:15 > Subject: Re: [HTCondor-users] transform syntax to build complex ads possible? > Hi, > > still struggling with the $STRING but I feel like making progress. > > Maybe you have some helpful comments for my testing below... > > So I was using the test transform > > JOB_TRANSFORM_MYTEST @=END > Y = 123 > tmp = [a="Project"]["a"] > SET Z1 = tmp > SET Z2 = $STRING(tmp) > SET Z3 = "tmp" > @END > > Then > > # condor_config_val -verbose JOB_TRANSFORM_MYTEST > JOB_TRANSFORM_MYTEST = Y = 123 > tmp = [a="Project"]["a"] > SET Z1 = tmp > SET Z2 = tmp > SET Z3 = "tmp" > # at: /etc/condor/config.d/99debug.conf, line 15 > # raw: JOB_TRANSFORM_MYTEST = Y = 123 > tmp = [a="Project"]["a"] > SET Z1 = tmp > SET Z2 = $STRING(tmp) > SET Z3 = "tmp" > > As far as I understand, the variable JOB_TRANSFORM_MYTEST is, due to its > heredoc-like definition, at this stage just a multi-line string, where only the > $*() functions are expanded? > > But while Z1 and Z2 look the same in evaluated (non-raw), there is still a > difference because when I submit a test job: > > # condor_q -l > ... > Z1 = tmp > Z2 = Project > Z3 = "tmp" > > Here the scheduler should have used the multiline string as transform rules for > the job ad. > > Testing with condor_transform_ads shows me that maybe the scheduler works with > the raw value? > > # cat test.ad > Project = "test" > X = 1 > # cat rules > Y = 123 > tmp = [a="Project"]["a"] > SET Z1 = tmp > SET Z2 = $STRING(tmp) > SET Z3 = "tmp" > # condor_transform_ads -in test.ad -rules rules -out:json out.ad > # cat out.ad > [ > { > "Project": "test", > "X": 1, > "Z1": "\/Expr(tmp)\/", > "Z2": "\/Expr(Project)\/", > "Z3": "tmp" > } > ] > > Not really sure why the tmp in Z1 stays unchanged while the one in Z2 is. > > Is it a feature of $STRING? Something like... > > expand $STRING(tmp) = eval tmp as _expression_ + treat resulting value as string + > convert string into an _expression_? > > Still feels like I am missing some crucial insight into the parsing process... > > Best > Kruno > > ----- Original Message ----- >> From: "John M Knoeller" <johnkn@xxxxxxxxxxx> >> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx> >> Sent: Monday, 11 October, 2021 18:40:55 >> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible? > >> EVALMACRO and $STRING() are both ways to evaluate-and-print >> >> $STRING(AA) will lookup AA, and evaluate the contents, then print the result. >> This has the side effect of removing quotes from things that evaluate to string >> values >> >> EVALMACRO does the same, except there is no lookup, the thing that it evaluates >> and prints >> is the right hand side of the statement after it. >> >> The big difference here is that $STRING() requires you to create a temp variable >> to hold what you want to evaluate, and it lets you specify a print format >> specifier for the printing part of the operation,. EVALMACRO is not limited to >> string evaluation, but it does not let you control the formatting of the print >> operation. >> >> So yes, this is a bit confusing. The thing to keep in mind is that there is not >> so much contexts as phases. >> $() substitution happens first, it is basically text manipulation, except for a >> few $FUNC() substitution functions which will evaluate and print. the rest of >> the $() substition is just a print operation. It is not necessary that each >> little bit that is $() substituted be a valid _expression_, it can be a part of >> an _expression_. only the final result after all of the $() >> subtitution/printing is done must be a valid classad _expression_. >> >> -tj >> ________________________________ >> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sever, >> Krunoslav <krunoslav.sever@xxxxxxx> >> Sent: Saturday, October 9, 2021 3:11 PM >> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx> >> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible? >> >> Hmm, >> >> it appears this remains very fiddly business that requires extreme care... >> >> So I can confirm that your workarounds (EVALMACRO+SET and tmp+SET $STRING(tmp)) >> work. >> >> With the EVALMACRO+SET I can sort of see why, the EVALMACRO probably giving >> slightly broader access to the scope. >> >> In that regard, I forgot that Project is not actually in the submit file itself, >> it is created by a previous transform, so it may be treated differently in this >> context. >> >> With the second one, I am still struggling, specifically what the $STRING >> function brings to the table... >> >> Also, I tried the EVALMACRO with my if chain and failed to build actual string >> values, i.e. >> >> EVALMACRO tmp = [ x01 = ifThenElse("Project" =?= undefined, "BIRD_noop", >> undefined) ]["x01"] >> >> sort of works but how I do get the evalmacro to treat the "BIRD_noop" as actual >> string? Probably easy, but didn't find it... >> >> I guess I could put the string value into another tmp variable and then access >> that, but this defeats the purpose of simplification. >> >> While testing around, I realized I can use the userMap function to drastically >> reduce our if-chain from 27 conditions to about 5 which may be sufficiently >> simple for our purposes. >> >> But I am still interested in understanding the new(er) classad features as of >> 9.0.x - it seems they still have a lot of quirks. All of the following contexts >> appear to throw in a another wrinkle: >> >> * old style -> implicitly transformed to new style >> * new style >> * config context >> -> $() evaluated differently? >> * job transform context >> -> from docs "superset of an earlier transform language based on New ClassAds" >> * router transform context >> >> It's hard to keep all those apart - there is a lot in the docs but I guess it is >> a little too much and currently is still more confusing than helping... >> >> Do you have some sort of cheat sheet summarizing the differences per context? >> That would be very helpful to me and probably others, too. >> >> Best >> Kruno >> >> ----- Original Message ----- >>> From: "John M Knoeller" <johnkn@xxxxxxxxxxx> >>> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx> >>> Sent: Friday, 8 October, 2021 22:16:16 >>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible? >> >>> I would need to see your actual transform. this EVALSET statement >>> >>> EVALSET AccountingGroup = [ >>> x1 = ifThenElse(cond1, value1,undefined); >>> x2 = x1 ?: ifThenElse(cond2, value2, undefined); >>> ... >>> x<n> = x<n-1>?: ifThenElse(cond<n>, value<n>, default); >>> ]["x<n>"] >>> >>> is not allowed to span multiple lines. so the net effect will be >>> >>> EVALSET AccountingGroup = [ >>> >>> followed by some statements that do nothing >>> >>> For your simpler example. >>> >>> EVALSET TEST = [ a=Project ]["a"] >>> >>> you say Project is set in submit. Does that mean the job has an attribute called >>> Project? >>> Because this is an EVALSET, it would set TEST to the whatever Project evaluates >>> to. >>> >>> If that is ending up being undefined, I think the problem here he one of scope. >>> since Project is not defined in the temporary ad [a=Project] it ends up being >>> undefined when evaluated. >>> >>> you can work around this limitation by generating a textual representation of >>> the _expression_ you want,and then then using SET to change the job. This works >>> because SET will parse the text as an _expression_. There are two ways to >>> accomplish this >>> >>> EVALMACRO tmp = [a="Project"]["a"] >>> SET test = $(tmp) >>> >>> or >>> >>> tmp = [a="Project"]["a"] >>> SET test = $STRING(tmp) >>> >>> note the use of a="Project" rather than a=Project here >>> >>> -tj >>> >>> ________________________________ >>> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sever, >>> Krunoslav <krunoslav.sever@xxxxxxx> >>> Sent: Friday, October 8, 2021 7:01 AM >>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx> >>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible? >>> >>> Hi, >>> >>> this looks like a useful feature and I tried to use it to break up our >>> IfThenElse() construct in our JobTransform: >>> >>> eval_set_AccountingGroup = ifThenElse( cond1, \ >>> value1 \ >>> ifThenElse(cond2, \ >>> value2, \ >>> ... >>> ) \ >>> ) >>> >>> As condor_status -limit 1 -af '[a=Name]["a"]' resolves to the Name from the >>> machine ad, I tried the replacement >>> >>> EVALSET AccountingGroup = [ >>> x1 = ifThenElse(cond1, value1,undefined); >>> x2 = x1 ?: ifThenElse(cond2, value2, undefined); >>> ... >>> x<n> = x<n-1>?: ifThenElse(cond<n>, value<n>, default); >>> ]["x<n>"] >>> >>> A reconfig accepted the transform (judging from the D_FULLDEBUG output in >>> SchedLog) but the conditions failed to access job ad to be transformed. >>> >>> Testing this with a simpler transform like >>> >>> EVALSET TEST = [ a=Project ]["a"] >>> >>> set undefined in the transformed job, even though Project is set in the original >>> submit. >>> >>> Aa an aside, I initially tried with eval_set, where it also reordered the x<i> >>> lines which does not happen with EVALSET. >>> >>> I assume this is a bug? If so, I'd appreciate a fix somewhere down the line, >>> modifying the current construct is very error-prone... >>> >>> Maybe it is not even too hard to add a function like >>> >>> ifThenElseChain(cond1, val1, ..., condn, valn, default) >>> >>> This would take care of short-circuiting which my replacement lacks - not too >>> important, but could be nice... >>> >>> Best >>> Kruno >>> >>> ----- Original Message ----- >>>> From: "John M Knoeller" <johnkn@xxxxxxxxxxx> >>>> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx> >>>> Sent: Thursday, 7 October, 2021 18:03:28 >>>> Subject: Re: [HTCondor-users] transform syntax to build complex ads possible? >>> >>>> the GROUP_SORT_EXPR cannot be a classad, because it needs to evaluate to an >>>> integer >>>> but the _expression_ can contain one or more classads as part of the _expression_ >>>> >>>> for example >>>> >>>>> condor_status -limit 1 -af '[a=1;b=2]["a"]' >>>> 1 >>>>> condor_status -limit 1 -af '[a=1;b=2]["b"]' >>>> 2 >>>>> condor_status -limit 1 -af '[a=1;b=2]["c"]' >>>> error >>>> >>>> In the above [a=1;b=2] declares a classad, and ["a"] does a lookup within that >>>> classad for attribute a. >>>> the result of evaluation will be 1 >>>> >>>> -tj >>>> ________________________________ >>>> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Thomas >>>> Hartmann <thomas.hartmann@xxxxxxx> >>>> Sent: Thursday, October 7, 2021 8:32 AM >>>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx> >>>> Subject: [HTCondor-users] transform syntax to build complex ads possible? >>>> >>>> Hi all, >>>> >>>> is it actually possible, to write a a class ad in the new transform >>>> syntax (i.e., w/o square brackets)? >>>> >>>> AFAIS it is no macro/function/... where I can return a value, or? >>>> >>>> Background is, that I would like to refactor a `GROUP_SORT_EXPR`, that >>>> has grown over time into a forest of IfThenElse()'s. >>>> >>>> If I get it correctly, a 'macro' in the []-syntax is effectively a >>>> complex class ad. But I am not sure, if I can evaluate/set a new >>>> transform-like _expression_ directly into an ad? >>>> >>>> Cheers, >>>> Thomas >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> HTCondor-users mailing list >>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a >>>> subject: Unsubscribe >>>> You can also unsubscribe by visiting >>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users >>>> >>>> The archives can be found at: >>>> https://lists.cs.wisc.edu/archive/htcondor-users/ >>> >>> -- >>> ------------------------------------------------------------------------ >>> Krunoslav Sever Deutsches Elektronen-Synchrotron (IT-Systems) >>> Ein Forschungszentrum der Helmholtz-Gemeinschaft >>> Notkestr. 85 >>> phone: +49-40-8998-1648 22607 Hamburg >>> e-mail: krunoslav.sever@xxxxxxx Germany >>> ------------------------------------------------------------------------ >>> _______________________________________________ >>> HTCondor-users mailing list >>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a >>> subject: Unsubscribe >>> You can also unsubscribe by visiting >>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users >>> >>> The archives can be found at: >>> https://lists.cs.wisc.edu/archive/htcondor-users/ >>> >>> _______________________________________________ >>> HTCondor-users mailing list >>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a >>> subject: Unsubscribe >>> You can also unsubscribe by visiting >>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users >>> >>> The archives can be found at: >>> https://lists.cs.wisc.edu/archive/htcondor-users/ >> >> -- >> ------------------------------------------------------------------------ >> Krunoslav Sever Deutsches Elektronen-Synchrotron (IT-Systems) >> Ein Forschungszentrum der Helmholtz-Gemeinschaft >> Notkestr. 85 >> phone: +49-40-8998-1648 22607 Hamburg >> e-mail: krunoslav.sever@xxxxxxx Germany >> ------------------------------------------------------------------------ >> _______________________________________________ >> HTCondor-users mailing list >> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a >> subject: Unsubscribe >> You can also unsubscribe by visiting >> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users >> >> The archives can be found at: >> https://lists.cs.wisc.edu/archive/htcondor-users/ >> >> _______________________________________________ >> HTCondor-users mailing list >> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a >> subject: Unsubscribe >> You can also unsubscribe by visiting >> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users >> >> The archives can be found at: >> https://lists.cs.wisc.edu/archive/htcondor-users/ > > -- > ------------------------------------------------------------------------ > Krunoslav Sever Deutsches Elektronen-Synchrotron (IT-Systems) > Ein Forschungszentrum der Helmholtz-Gemeinschaft > Notkestr. 85 > phone: +49-40-8998-1648 22607 Hamburg > e-mail: krunoslav.sever@xxxxxxx Germany > ------------------------------------------------------------------------ > _______________________________________________ > HTCondor-users mailing list > To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a > subject: Unsubscribe > You can also unsubscribe by visiting > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users > > The archives can be found at: > https://lists.cs.wisc.edu/archive/htcondor-users/ -- ------------------------------------------------------------------------ Krunoslav Sever Deutsches Elektronen-Synchrotron (IT-Systems) Ein Forschungszentrum der Helmholtz-Gemeinschaft Notkestr. 85 phone: +49-40-8998-1648 22607 Hamburg e-mail: krunoslav.sever@xxxxxxx Germany ------------------------------------------------------------------------ _______________________________________________ HTCondor-users mailing list To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a subject: Unsubscribe You can also unsubscribe by visiting https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users The archives can be found at: https://lists.cs.wisc.edu/archive/htcondor-users/ |