Re: [HTCondor-users] shared processing folder???

[Alejandro AcuÃa <alejandro.acunia@xxxxxxxxxxxxxxxx>]

In that case, then a database sound better. Concurrent access to grant input data to each jobâit would be independent of the machine, permits, data size, etcâ.

 

De: Beyer, Christoph [mailto:christoph.beyer@xxxxxxx]
Enviado el: viernes, 6 de marzo de 2020 15:08
Para: Alejandro AcuÃa
CC: htcondor-users
Asunto: Re: [HTCondor-users] shared processing folder???

 

 

that sounds mor like you need a small nfs server, maybe on your collector or another central machine, access rights on nfs are still not great though unless you decide to give nfs v4 a try ... 

 

I am not aware that htcondor comes with a fileserver-like part, the file staging is great though and scales very good, maybe less of a hassle than a nfs server ...

 

best

christoph

 

--
Christoph Beyer
DESY Hamburg
IT-Department

Notkestr. 85
Building 02b, Room 009
22607 Hamburg

phone:+49-(0)40-8998-2317
mail: christoph.beyer@xxxxxxx

 

---

Von: "Alejandro AcuÃa" <alejandro.acunia@xxxxxxxxxxxxxxxx>
An: "htcondor-users" <htcondor-users@xxxxxxxxxxx>, "Christoph Beyer" <christoph.beyer@xxxxxxx>
Gesendet: Freitag, 6. MÃrz 2020 17:27:50
Betreff: RE: [HTCondor-users] shared processing folder???

 

Thanks for your Reply Christoph,

Our HTCondor cluster is just a private LAN with one master node as gateway and a few processing nodes. Under this paradigm, different hosts are no needed.

I vaguely imagined to share a folder in master node (with appropriates permits) and the jobs can consume files from this ubication.

We are attemping completely minimize transfer (through input_transfer_files) on each node. 

This would be possible?

 

Regards

Ale

 

De: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] En nombre de Beyer, Christoph
Enviado el: viernes, 6 de marzo de 2020 11:31
Para: htcondor-users
Asunto: Re: [HTCondor-users] shared processing folder???

 

Hi Ale,

 

do you think of a static directory with a bunch of files, that can be created just as you would on a regular host maybe through a ssh-loop.

 

If it is more about a common base of programs etc. cvmfs comes to mind.

 

If the stuff you host in that dir is more dynamic you could host it on a nfs file server etc.

 

If you want your jos t make sure this 'dir' is present on the workernode you could define an according filesystem domain and request that domain in your job....

 

Best

Christoph

 

--
Christoph Beyer
DESY Hamburg
IT-Department

Notkestr. 85
Building 02b, Room 009
22607 Hamburg

phone:+49-(0)40-8998-2317
mail: christoph.beyer@xxxxxxx

 

---

Von: "Alejandro AcuÃa" <alejandro.acunia@xxxxxxxxxxxxxxxx>
An: "htcondor-users" <htcondor-users@xxxxxxxxxxx>
Gesendet: Freitag, 6. MÃrz 2020 13:38:22
Betreff: [HTCondor-users] shared processing folder???

 

Hi friends.

I need to specify a folder with specific shared files for all jobs processing (like a input_transfer_files). I wonder if exists a kind of make available this folder to all processing nodes and put in there all of necesary files (probably files must be readed or writed too).

Finally, if itâs posible, how is the submit?

 

Thanks!!!

Ale

 

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/