Re: [HTCondor-users] How do I write plugins for public_input_files?

["Beyer, Christoph" <christoph.beyer@xxxxxxx>]

Argh,

sorry for the confusion here - it's to hot outside :(

Of course the  files only must be accessible at submit time - hence from the access point. They will be then transfered to the scheduler that is usually running on the same node.

From then on they do travel with the job and the sched transfers them to the worker and informs the startd once the transfer was successful. 

Best

christoph

---

Von: "Christoph Beyer" <christoph.beyer@xxxxxxx>
An: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
Gesendet: Mittwoch, 4. September 2024 13:30:24
Betreff: Re: [HTCondor-users] How do I write plugins for public_input_files?

Hi,

you do not need a plugin to do that (given your files are accessible network wise from the execute node and access pattern wise by the uid that runs the job).

Simply add:

should_transfer_files = YES
transfer_input_files = file1,file2

To your submit file - this will enable the file transfer and automatically transfer the executable and the files 'file1' and 'file2' ...

Best

christoph

--
Christoph Beyer
DESY Hamburg
IT-Department

Notkestr. 85
Building 02b, Room 009
22607 Hamburg

phone:+49-(0)40-8998-2317
mail: christoph.beyer@xxxxxxx

---

Von: "Leon Thielen" <L.Thielen@xxxxxxxxxxxx>
An: "htcondor-users" <htcondor-users@xxxxxxxxxxx>
Gesendet: Mittwoch, 4. September 2024 12:49:25
Betreff: [HTCondor-users] How do I write plugins for public_input_files?

Hi

 

Is it possible to write pulgins for public_input_files?

Or is there already a solution for this problem?

 

We have a data volume of approx. 3GB with 14000 files and 410 folders.

The transfer should take place as a compressed file.

The data is read only and valid for a cluster with 1400 jobs on Linux and also 1400 jobs on Windows.

Is there a process that transfers the data once per EP to a location of my choice e.g. tmp or creates links in the sandboxes?

 

Thank you

Leon

 

 

	Dipl.-Ing. Leon Thielen Software Development MAGMA GieÃereitechnologie GmbH P: +49 241 88901 244  Kackertstrasse 16-18, 52072 Aachen, Germany  www.magmasoft.de L.Thielen@xxxxxxxxxxxx
GERMANY â USA â BRAZIL â SINGAPORE â SOUTH KOREA â CHINA â INDIA â TURKEY â CZECH REPUBLIC

International MAGMA User Meeting 2024 - October 9-11 | RADISSON BLU - Frankfurt

MAGMA GieÃereitechnologie GmbH | KackertstraÃe 16-18, 52072 Aachen, Germany | Legal form: GmbH, Register court: Aachen HRB 3912, Value added tax identification number: DE121745780 | Management: Dr. Marc C. Schneider (CEO and President), Dipl.-Ing. Mathieu Weber (Managing Director)

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/