[HTCondor-users] token jobs not being routed by HTCondor-CE

[Stefano Dal Pra <stefano.dalpra@xxxxxxxxxxxx>]

    Hello to all,
      
    htcondor-ce-5.1.6
            + condor-9.0.17 Here.
          
          I'm having problems with HTCondor-CE not routing
        jobs submitted with iam token [1]. The same routing rule [2] or
        [3] working with GSI does not work with tokens.
        More notes in [4]. 
        
        USING GSI
        #This works
        [sdalpra@ui-htc CE5]$ export
        _condor_SEC_CLIENT_AUTHENTICATION_METHODS=GSI ; condor_submit
        -pool ce07-htc.cr.cnaf.infn.it:9619 -remote
        ce07-htc.cr.cnaf.infn.it ce_gsi308.sub Â
      Submitting job(s).
      
      1 job(s) submitted to cluster 3250129.
    
      #the job is routed and submitted to condor; note the local user
      (dteam026), that is mapped by argus
    [root@ce07-htc
        ~]# condor_ce_q 3250129. -af:j owner routedtojobid
      
      3250129.0 dteam026 4991835.0
      
      USING SCITOKENS
    #This does not work
    [sdalpra@ui-htc
          CE5]$ export
          _condor_SEC_CLIENT_AUTHENTICATION_METHODS=SCITOKENS ;
          condor_submit -pool ce07-htc.cr.cnaf.infn.it:9619 -remote
          ce07-htc.cr.cnaf.infn.it ce_scitok308.sub Â
        Submitting job(s).
        
        1 job(s) submitted to cluster 3250138.
      
    #the job is never routed. Note that the
      REQUIREMENTS _expression_ evaluates to true.
    [root@ce07-htc
                ~]# condor_ce_q 3250138. -af:j owner routedtojobid
                'StringListMember(Owner,
                "dteam007|dteam026|cmssgm017","|")'
              
              3250138.0 dteam007 undefined true
              
            
      [1] The token being used
    [sdalpra@ui-htc
        CE5]$ cat Â$BEARER_TOKEN_FILE|jwt.py -v ÂÂÂÂ
      {
      
      Â"alg": "RS256", Â
      Â"kid": "rsa1"
      
      }
      
      {
      
      Â"sub": "9662c0b5-31a1-4478-963e-bdf3783232ed", Â
      Â"iss": "https://wlcg.cloud.cnaf.infn.it/", Â
      Â"wlcg.groups": [
      
      ÂÂÂ"/wlcg", Â
      ÂÂÂ"/wlcg/pilots", Â
      ÂÂÂ"/wlcg/xfers"
      
      Â], Â
      Â"wlcg.ver": "1.0", Â
      Â"jti": "4270f069-81d9-48fb-88ef-817a83b98c6a", Â
      Â"exp": 1679943559, Â
      Â"iat": 1679939959, Â
      Â"client_id": "ad852b22-e517-44a4-99e8-7c0660f878a1", Â
      Â"scope": "openid compute.create profile compute.read
      storage.read:/ compute.modify eduperson_entitlement wlcg
      storage.create:/ offline_access compute.cancel eduperson
      _scoped_affiliation storage.modify:/ email wlcg.groups", Â
      Â"nbf": 1679939959, Â
      Â"aud": "https://wlcg.cern.ch/jwt/v1/any"
      
      }
      
      exp: Mon Mar 27 20:59:19 2023
      
      [2],[3] Jobrouter rules
    
    JOB_ROUTER_ROUTE_routestsci @=jrt
        
        Â REQUIREMENTS
          StringListMember(Owner, "dteam007|dteam026|cmssgm017","|")
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ
        Â UNIVERSE VANILLA
        
        ÂÂSET
          Requirements (TARGET.t1_group =?= "myfancygroup")
        
        ÂÂSET RequestMemory 400
        
        ÂÂSET MaxJobs 5
        
        ÂÂSET MaxIdleJobs 10
        
        @jrt
        
      JOB_ROUTER_ROUTE_routestgsi
        @=jrt
      
      ÂREQUIREMENTS
        (x509UserProxyVOName == "dteam") ||
        (AuthTokenIssuer =?= "https://wlcg.cloud.cnaf.infn.it/" &&
        AuthTokenSubject =?= "9662c0b5-31a1-4478-963e-bdf3783232ed")
      
      ÂUNIVERSE VANILLA
      
      ÂSET
        Requirements (TARGET.t1_group =?= "testgroup")
      
      @jrt
      
    JOB_ROUTER_ROUTE_NAMES = routestsci routestgsi $(JOB_ROUTER_ROUTE_NAMES)
      
      [4] Notes
      
      - scitoken is "partially" valid as the mapping to the local user
      succeeds.
      - the REQUIREMENTS _expression_ matches with the condor-ce job, i.e.
      
      ÂÂÂ condor_ce_q <jobid> -af StringListMember(Owner,
            "dteam007|dteam026|cmssgm017","|")Â 
      Â returns True.
      - These rules used to work as far as i know. More complex
      REQUIREMENTS expressions where successfully used with tokens.
      - I checked rule [2] against a condor-ce at another site where a
      colleague accepted to test it; the result is the same: using GSI
      the job is routed, using SCITOKENS it is not.
      - I find nothing useful in the condor-ce logs: 
      
    [root@ce07-htc
        ~]# grep 3250492. /var/log/condor-ce/*Log
      
      /var/log/condor-ce/AuditLog:03/27/23
        21:54:54 (cid:18395186) (D_AUDIT) Submitting new job 3250492.0
      
      /var/log/condor-ce/AuditLog:03/27/23
        21:54:54 (cid:18395188) (D_AUDIT) Transferring files for jobs 3250492.0
      
      /var/log/condor-ce/SchedLog:03/27/23
        21:54:55 (D_ALWAYS) Job 3250492.0 released from
        hold: Data files spooled
      
      Also at maximum verbosity nothing is found in the JobRouterLog.
      I'm out of ideas now. Any hint to find out what's wrong?
      Thanks
      Stefano