Re: [HTCondor-users] Have submit nodes inside a Kubernetes cluster

[Gaetan Geffroy <gage@xxxxxxxxx>]

Yes, I was using the wrong docker network.

I managed to have it work now, thanks you very much 😊

 

Gaetan Geffroy
Junior Software Engineer
Terma GmbH
T +49 6151 86005 43 (direct)
 


 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Miguel Garrido
Sent: Tuesday, May 2, 2023 16:06
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Have submit nodes inside a Kubernetes cluster

 

CAUTION: This email originated from outside of Terma. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi -

 

172.18.0.1 looks like a Docker bridge network address which is likely not reachable from another host on the network e.g. the CM. This shouldnât be a problem within the Kubernetes cluster since each pod should be able to route to another one within the cluster (with proper configs).

 

You could try in your CentOS test case to have your container use a bridge that assigns the container an externally reachable address from the host systemâs network. 

 

Regards 

 

On Tue, May 2, 2023 at 09:09 Gaetan Geffroy <gage@xxxxxxxxx> wrote:

Hi,

 

I know it is possible to run condor entirely inside Kubernetes.

But what if I have an existing pool outside of the Kubernetes cluster, with the central manager and the executor nodes, could I spawn one or several pods in the Kubernetes cluster and have them act as submit nodes ?

 

As a quick test I tried the following:

• Have a pool running on different machines (one CM and two nodes)
• On another machine, not a member of the pool but part of the same network, I start a Docker in Docker container
• Inside this dind container, I start a CentOS container , on which I run âcurl -fsSL https://get.htcondor.org | GET_HTCONDOR_PASSWORD="some_secret_password" /bin/bash -s -- --no-dry-run --submit my.cm.adressâ

After doing this, from inside the CentOS container I can use condor_status fine, but when I submit jobs they stay IDLE forever and the negotiator log says something like:

 

Failed to connect to condor@xxxxxxxxxxxxxxxxxx (<172.18.0.1:9618?addrs=172.18.0.1-9618&alias=condor.test.submit&noUDP&sock=schedd_412_7d83>)

Error: Ignoring submitter for this cycle

 

So it looks like the CM has trouble connecting back to the submit container.

I tried the same setup but outside of the dind container (spawning the CentOS container directly on the docker host machine) and there it works well.

 

So, seeing that, I am starting to have my doubts about how having Kubernetes pods connecting to the pool and submitting jobs.

Was this ever done before ? Did it work ? Could it work ?

 

Thanks,

GaÃtan

 

Gaetan Geffroy
Junior Software Engineer, Space

Terma GmbH
Europaarkaden II, BratustraÃe 7, 64293 Darmstadt, Germany
T +49 6151 86005 43 (direct)  â  T +49 6151 86005-0
Terma GmbH - Sitz Darmstadt  â  Handelsregister Nr.: HRB 7411, Darmstadt
GeschÃftsfÃhrer: Poul Vigh / Steen Vejby SÃrensen
www.terma.com â Linkedin â Twitter â Instagram â Youtube

---

Attention:
This e-mail (and attachment(s), if any) - intended for the addressee(s) only - may contain confidential, copyright, or legally privileged information or material, and no one else is authorized to read, print, store, copy, forward, or otherwise use or disclose any part of its contents or attachment(s) in any form. If you have received this e-mail in error, please notify me by telephone or return e-mail, and delete this e-mail and attachment(s). Thank you.

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

--

MG