Re: [HTCondor-users] Remote Submit Scheds with grid submission to CondorCE

[Brian Lin <blin@xxxxxxxxxxx>]

    Hi all,
    
    If there's no local Schedd/Gridmanager but your users have access to
    condor_submit, then you can run something like:
    
    condor_submit -remote <CE FQDN> -pool <CE FQDN>:<CE
    PORT> <vanilla universe submit file>
    
    This is effectively what condor_ce_trace [1] and condor_ce_run [2]
    do. However, there are a few caveats to this method of submission:
    
    - HTCondor-CE is designed with pilot jobs in mind, not user jobs.
    This doesn't mean that users can't submit their jobs to a CE, it's
    just that the CE may be very aggressive in job cleanup in ways that
    are unfriendly to end users
    - X.509 proxies will be sent initially but renewals will not be
    forwarded on (assuming x509userproxy/use_x509userproxy are set
    appropriately)
    - Users will not to run commands to fetch their output, making this
    method particularly ineffective for long workflows
    
    - Brian
    
    [1]
    https://github.com/htcondor/htcondor-ce/blob/master/src/condor_ce_trace
    [2]
    https://github.com/htcondor/htcondor-ce/blob/master/src/condor_ce_run

On 2/11/21 11:13 AM, Thomas Hartmann wrote:

Hi Maarten,

maybe I am overthinking it, but AFAIS the problem is the remote submit needing to forward the proxy

The user servers are as remote submitters not running a local schedd but only forward the jobs to the actual schedulers. I am not sure, if the X509s are forwarded, or?

Cheers,
  Thomas

On 11/02/2021 18.09, Maarten Litmaath wrote:

Hi Thomas,
could it be as simple as using a JDL like this:

cmd = ...
output = ...
error = ...
log = ...
+TransferOutput = ""
periodic_hold = ...
periodic_remove = ...
universe = grid
grid_resource = condor htc-ce.some.domain htc-ce.some.domain:9619
use_x509userproxy = true
environment = ...
queue 1

That is how ALICE let a local HTCondor service deal with all
the intricacies of handling jobs for remote HTCondor CEs.

Maybe I am missing something?

________________________________________
From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of Thomas Hartmann [thomas.hartmann@xxxxxxx]
Sent: 11 February 2021 17:46
To: HTCondor-Users Mail List
Subject: [HTCondor-users] Remote Submit Scheds with grid submission to  CondorCE

Hi all,

does somebody has maybe already experiences with combining remote
submission with gridward submission to CondorCEs?

Thing is, that we have a number of nodes for our users, that act as
remote submitters to the actual schedulers.

Now it would come handy, not only to submit to the local Condor but also
allow our users to do submissions into the Grid targeting CondorCEs.
Since the Schedd would have to do the authentication through the users'
proxy X509, it would need to be passed forward like kerberos etc.
tickets/tokens.
And are there other caveats, that might need to be considered in such a
setup?

Maybe somebody has already a working similar setup?

Cheers,
    Thomas

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/