Re: [HTCondor-users] token jobs not being routed by HTCondor-CE

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 3/28/2023 5:42 AM, Stefano Dal Pra wrote:

When using (only) x509 and no token, the job is mapped (by argus) to dteam026.
StringListMember should work the same with dteam007 or dteam026
however it only matches with dteam026 (i.e. GSI). and not with dteam007.
I normally check for issuer and subject in the jobrouter; i tried with StringListMember to
restrict the check to Owner only.

    Hi Stefano -
    
    After changing the route to try StringListMember, did you remember
    to issue a "condor_ce_reconfig"
    command?  
    
    For job 3250138.0 below, it sure looks like the owner mapping from
    the token worked fine...  perhaps this command will give a clue:
root@host # condor_ce_history -l 3250138.0 | condor_ce_job_router_info -match-jobs -ignore-prior-routing -jobads -
    Also see the CE Manual for troubleshooting tips when a job does not
    route at URL:
     
https://htcondor.com/htcondor-ce/v4/troubleshooting/troubleshooting/#jobs-stay-idle-on-the-ce
    
    Hope the above helps, let us know how it goes, feel free to ask for
    more help if you continue to be stuck.
    
    regards,
    Todd

Adding a detail on the submit file used for GSI and SCITOKENS
#submit file for GSI
[sdalpra@ui-htc CE5]$ cat ce_gsi308.sub
universe = vanilla
use_x509userproxy = true
+Owner = undefined
[...]

[sdalpra@ui-htc CE5]$ cat ce_scitok308.sub  
universe = vanilla
use_scitokens = true
+Owner = undefined


Stefano



On 28/03/23 11:56, Thomas Hartmann wrote:

Hi Stefano,

how does your token mapping look like? ð

Just a suspicion, but maybe the token subject is mapped to another user than the X509 mapped user and the requirement
  REQUIREMENTS StringListMember(Owner, "dteam007|dteam026|cmssgm017","|")
does not get triggered?

Cheers,
  Thomas

On 27/03/2023 22.50, Stefano Dal Pra wrote:

Hello to all,

htcondor-ce-5.1.6 + condor-9.0.17 Here.

I'm having problems with HTCondor-CE not routing jobs submitted with iam token [1]. The same routing rule [2] or [3] working with GSI does not work with tokens.
More notes in [4].

USING GSI
#This works
[sdalpra@ui-htc CE5]$ export _condor_SEC_CLIENT_AUTHENTICATION_METHODS=GSI ; condor_submit -pool ce07-htc.cr.cnaf.infn.it:9619 -remote ce07-htc.cr.cnaf.infn.it ce_gsi308.sub
Submitting job(s).
1 job(s) submitted to cluster 3250129.

#the job is routed and submitted to condor; note the local user (dteam026), that is mapped by argus
[root@ce07-htc ~]# condor_ce_q 3250129. -af:j owner routedtojobid
3250129.0 dteam026 4991835.0

USING SCITOKENS
#This does not work
[sdalpra@ui-htc CE5]$ export _condor_SEC_CLIENT_AUTHENTICATION_METHODS=SCITOKENS ; condor_submit -pool ce07-htc.cr.cnaf.infn.it:9619 -remote ce07-htc.cr.cnaf.infn.it ce_scitok308.sub
Submitting job(s).
1 job(s) submitted to cluster 3250138.

#the job is never routed. Note that the REQUIREMENTS _expression_ evaluates to true.
[root@ce07-htc ~]# condor_ce_q 3250138. -af:j owner routedtojobid 'StringListMember(Owner, "dteam007|dteam026|cmssgm017","|")'
3250138.0 dteam007 undefined true


[1] The token being used
[sdalpra@ui-htc CE5]$ cat  $BEARER_TOKEN_FILE|jwt.py -v
{
  "alg": "RS256",
  "kid": "rsa1"
}
{
  "sub": "9662c0b5-31a1-4478-963e-bdf3783232ed",
  "iss": "https://wlcg.cloud.cnaf.infn.it/",
  "wlcg.groups": [
    "/wlcg",
    "/wlcg/pilots",
    "/wlcg/xfers"
  ],
  "wlcg.ver": "1.0",
  "jti": "4270f069-81d9-48fb-88ef-817a83b98c6a",
  "exp": 1679943559,
  "iat": 1679939959,
  "client_id": "ad852b22-e517-44a4-99e8-7c0660f878a1",
  "scope": "openid compute.create profile compute.read storage.read:/ compute.modify eduperson_entitlement wlcg storage.create:/ offline_access compute.cancel eduperson
_scoped_affiliation storage.modify:/ email wlcg.groups",
  "nbf": 1679939959,
  "aud": "https://wlcg.cern.ch/jwt/v1/any"
}
exp: Mon Mar 27 20:59:19 2023

[2],[3] Jobrouter rules

JOB_ROUTER_ROUTE_routestsci @=jrt
REQUIREMENTS StringListMember(Owner, "dteam007|dteam026|cmssgm017","|")
   UNIVERSE VANILLA
SET Requirements (TARGET.t1_group=?= "myfancygroup")
   SET RequestMemory 400
   SET MaxJobs 5
   SET MaxIdleJobs 10
@jrt

JOB_ROUTER_ROUTE_routestgsi @=jrt
REQUIREMENTS (x509UserProxyVOName== "dteam") || (AuthTokenIssuer =?= "https://wlcg.cloud.cnaf.infn.it/"&& AuthTokenSubject =?= "9662c0b5-31a1-4478-963e-bdf3783232ed")
  UNIVERSE VANILLA
SET Requirements (TARGET.t1_group=?= "testgroup")
@jrt

JOB_ROUTER_ROUTE_NAMES= routestsci routestgsi $(JOB_ROUTER_ROUTE_NAMES)

[4] Notes

- scitoken is "partially" valid as the mapping to the local user succeeds.
- the REQUIREMENTS _expression_ matches with the condor-ce job, i.e.
     condor_ce_q <jobid> -af StringListMember(Owner, "dteam007|dteam026|cmssgm017","|")
   returns True.
- These rules used to work as far as i know. More complex REQUIREMENTS expressions where successfully used with tokens.
- I checked rule [2] against a condor-ce at another site where a colleague accepted to test it; the result is the same: using GSI the job is routed, using SCITOKENS it is not.
- I find nothing useful in the condor-ce logs:

[root@ce07-htc ~]# grep 3250492. /var/log/condor-ce/*Log
/var/log/condor-ce/AuditLog:03/27/23 21:54:54 (cid:18395186) (D_AUDIT) Submitting new job 3250492.0
/var/log/condor-ce/AuditLog:03/27/23 21:54:54 (cid:18395188) (D_AUDIT) Transferring files for jobs 3250492.0
/var/log/condor-ce/SchedLog:03/27/23 21:54:55 (D_ALWAYS) Job 3250492.0 released from hold: Data files spooled

Also at maximum verbosity nothing is found in the JobRouterLog.
I'm out of ideas now. Any hint to find out what's wrong?
Thanks
Stefano



_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

-- 
Todd Tannenbaum <tannenba@xxxxxxxxxxx>  University of Wisconsin-Madison
Center for High Throughput Computing    Department of Computer Sciences
Calendar: https://tinyurl.com/yd55mtgd  1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                   Madison, WI 53706-1685