Re: [HTCondor-users] token jobs not being routed by HTCondor-CE

[Stefano Dal Pra <stefano.dalpra@xxxxxxxxxxxx>]

Hello Thomas,
This is the mapping "iss","sub" --> localuser:
SCITOKENS /^https:\/\/wlcg\.cloud\.cnaf\.infn\.it\/,9662c0b5-31a1-4478-963e-bdf3783232ed/ dteam007

When using (only) x509 and no token, the job is mapped (by argus) to dteam026.
StringListMember should work the same with dteam007 or dteam026
however it only matches with dteam026 (i.e. GSI). and not with dteam007.
I normally check for issuer and subject in the jobrouter; i tried with StringListMember to
restrict the check to Owner only.

Adding a detail on the submit file used for GSI and SCITOKENS
#submit file for GSI
[sdalpra@ui-htc CE5]$ cat ce_gsi308.sub
universe = vanilla
use_x509userproxy = true
+Owner = undefined
[...]

[sdalpra@ui-htc CE5]$ cat ce_scitok308.sub Â
universe = vanilla
use_scitokens = true
+Owner = undefined


Stefano



On 28/03/23 11:56, Thomas Hartmann wrote:

> Hi
>       Stefano,
>       
>       
>       how does your token mapping look like? ð
>       
>       
>       Just a suspicion, but maybe the token subject is mapped to another
>       user than the X509 mapped user and the requirement
>       
>       Â REQUIREMENTS StringListMember(Owner,
>       "dteam007|dteam026|cmssgm017","|")
>       
>       does not get triggered?
>       
>       
>       Cheers,
>       
>       Â Thomas
>       
>       
>       On 27/03/2023 22.50, Stefano Dal Pra wrote:
>       
> > Hello to all,
> >         
> >         
> >         htcondor-ce-5.1.6 + condor-9.0.17 Here.
> >         
> >         
> >         I'm having problems with HTCondor-CE not routing jobs submitted
> >         with iam token [1]. The same routing rule [2] or [3] working
> >         with GSI does not work with tokens.
> >         
> >         More notes in [4].
> >         
> >         
> >         USING GSI
> >         
> >         #This works
> >         
> >         [sdalpra@ui-htc CE5]$ export
> >         _condor_SEC_CLIENT_AUTHENTICATION_METHODS=GSI ; condor_submit
> >         -pool ce07-htc.cr.cnaf.infn.it:9619 -remote
> >         ce07-htc.cr.cnaf.infn.it ce_gsi308.sub
> >         
> >         Submitting job(s).
> >         
> >         1 job(s) submitted to cluster 3250129.
> >         
> >         
> >         #the job is routed and submitted to condor; note the local user
> >         (dteam026), that is mapped by argus
> >         
> >         [root@ce07-htc ~]# condor_ce_q 3250129. -af:j owner
> >         routedtojobid
> >         
> >         3250129.0 dteam026 4991835.0
> >         
> >         
> >         USING SCITOKENS
> >         
> >         #This does not work
> >         
> >         [sdalpra@ui-htc CE5]$ export
> >         _condor_SEC_CLIENT_AUTHENTICATION_METHODS=SCITOKENS ;
> >         condor_submit -pool ce07-htc.cr.cnaf.infn.it:9619 -remote
> >         ce07-htc.cr.cnaf.infn.it ce_scitok308.sub
> >         
> >         Submitting job(s).
> >         
> >         1 job(s) submitted to cluster 3250138.
> >         
> >         
> >         #the job is never routed. Note that the REQUIREMENTS _expression_
> >         evaluates to true.
> >         
> >         [root@ce07-htc ~]# condor_ce_q 3250138. -af:j owner
> >         routedtojobid 'StringListMember(Owner,
> >         "dteam007|dteam026|cmssgm017","|")'
> >         
> >         3250138.0 dteam007 undefined true
> >         
> >         
> >         
> >         [1] The token being used
> >         
> >         [sdalpra@ui-htc CE5]$ cat Â$BEARER_TOKEN_FILE|jwt.py -v
> >         
> >         {
> >         
> >         ÂÂ"alg": "RS256",
> >         
> >         ÂÂ"kid": "rsa1"
> >         
> >         }
> >         
> >         {
> >         
> >         ÂÂ"sub": "9662c0b5-31a1-4478-963e-bdf3783232ed",
> >         
> >         ÂÂ"iss": "https://wlcg.cloud.cnaf.infn.it/",
> >         
> >         ÂÂ"wlcg.groups": [
> >         
> >         ÂÂÂÂ"/wlcg",
> >         
> >         ÂÂÂÂ"/wlcg/pilots",
> >         
> >         ÂÂÂÂ"/wlcg/xfers"
> >         
> >         ÂÂ],
> >         
> >         ÂÂ"wlcg.ver": "1.0",
> >         
> >         ÂÂ"jti": "4270f069-81d9-48fb-88ef-817a83b98c6a",
> >         
> >         ÂÂ"exp": 1679943559,
> >         
> >         ÂÂ"iat": 1679939959,
> >         
> >         ÂÂ"client_id": "ad852b22-e517-44a4-99e8-7c0660f878a1",
> >         
> >         ÂÂ"scope": "openid compute.create profile compute.read
> >         storage.read:/ compute.modify eduperson_entitlement wlcg
> >         storage.create:/ offline_access compute.cancel eduperson
> >         
> >         _scoped_affiliation storage.modify:/ email wlcg.groups",
> >         
> >         ÂÂ"nbf": 1679939959,
> >         
> >         ÂÂ"aud": "https://wlcg.cern.ch/jwt/v1/any"
> >         
> >         }
> >         
> >         exp: Mon Mar 27 20:59:19 2023
> >         
> >         
> >         [2],[3] Jobrouter rules
> >         
> >         
> >         JOB_ROUTER_ROUTE_routestsci @=jrt
> >         
> >         REQUIREMENTS StringListMember(Owner,
> >         "dteam007|dteam026|cmssgm017","|")
> >         
> >         ÂÂ UNIVERSE VANILLA
> >         
> >         SET Requirements (TARGET.t1_group=?= "myfancygroup")
> >         
> >         ÂÂÂSET RequestMemory 400
> >         
> >         ÂÂÂSET MaxJobs 5
> >         
> >         ÂÂÂSET MaxIdleJobs 10
> >         
> >         @jrt
> >         
> >         
> >         JOB_ROUTER_ROUTE_routestgsi @=jrt
> >         
> >         REQUIREMENTS (x509UserProxyVOName== "dteam") || (AuthTokenIssuer
> >         =?= "https://wlcg.cloud.cnaf.infn.it/"&&
> >         AuthTokenSubject =?= "9662c0b5-31a1-4478-963e-bdf3783232ed")
> >         
> >         ÂÂUNIVERSE VANILLA
> >         
> >         SET Requirements (TARGET.t1_group=?= "testgroup")
> >         
> >         @jrt
> >         
> >         
> >         JOB_ROUTER_ROUTE_NAMES= routestsci routestgsi
> >         $(JOB_ROUTER_ROUTE_NAMES)
> >         
> >         
> >         [4] Notes
> >         
> >         
> >         - scitoken is "partially" valid as the mapping to the local user
> >         succeeds.
> >         
> >         - the REQUIREMENTS _expression_ matches with the condor-ce job,
> >         i.e.
> >         
> >         ÂÂÂÂ condor_ce_q <jobid> -af StringListMember(Owner,
> >         "dteam007|dteam026|cmssgm017","|")
> >         
> >         ÂÂ returns True.
> >         
> >         - These rules used to work as far as i know. More complex
> >         REQUIREMENTS expressions where successfully used with tokens.
> >         
> >         - I checked rule [2] against a condor-ce at another site where a
> >         colleague accepted to test it; the result is the same: using GSI
> >         the job is routed, using SCITOKENS it is not.
> >         
> >         - I find nothing useful in the condor-ce logs:
> >         
> >         
> >         [root@ce07-htc ~]# grep 3250492. /var/log/condor-ce/*Log
> >         
> >         /var/log/condor-ce/AuditLog:03/27/23 21:54:54 (cid:18395186)
> >         (D_AUDIT) Submitting new job 3250492.0
> >         
> >         /var/log/condor-ce/AuditLog:03/27/23 21:54:54 (cid:18395188)
> >         (D_AUDIT) Transferring files for jobs 3250492.0
> >         
> >         /var/log/condor-ce/SchedLog:03/27/23 21:54:55 (D_ALWAYS) Job
> >         3250492.0 released from hold: Data files spooled
> >         
> >         
> >         Also at maximum verbosity nothing is found in the JobRouterLog.
> >         
> >         I'm out of ideas now. Any hint to find out what's wrong?
> >         
> >         Thanks
> >         
> >         Stefano
> >         
> >         
> >         
> >         
> >         _______________________________________________
> >         
> >         HTCondor-users mailing list
> >         
> >         To unsubscribe, send a message to
> >         htcondor-users-request@xxxxxxxxxxx with a
> >         
> >         subject: Unsubscribe
> >         
> >         You can also unsubscribe by visiting
> >         
> >         https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> >         
> >         
> >         The archives can be found at:
> >         
> >         https://lists.cs.wisc.edu/archive/htcondor-users/
> >         
> >       
>