Re: [HTCondor-users] [EXTERNAL] Re: Does the filetransfer_example_plugin.py work on WINDOWS?

[John M Knoeller <johnkn@xxxxxxxxxxx>]

I found a very clever prefix you can add to a python script that will do the same thing. 

0<0# : ^

'''

@echo off

python "%~f0" %*

goto :EOF

'''

If you make a filetransfer_net_plugin.bat file that consists of the above lines followed by the contents of  filetransfer_net_plugin.py,  it should have the same effect as sending your py_starter.cmd along with the python transfer plugin.   The Windows bat file processor treats 0<0# : as a label, and the ^ at the end of that line causes it to ignore the ''' on the next line. 

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Leon Thielen <L.Thielen@xxxxxxxxxxxx>
Sent: Tuesday, November 19, 2024 9:29 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] [EXTERNAL] Re: Does the filetransfer_example_plugin.py work on WINDOWS?

 

I was afraid of that.

 

But I have found a simple workaround:

 

I create a python starter script

py_starter.cmd

python filetransfer_net_plugin.py %1  %2  %3  %4

exit %ErrorLevel%

 

In the submit file

transfer_plugins = example = bin/py_starter.cmd

 

transfer_input_files = bin/filetransfer_net_plugin.py, example://datastore/automaticTests/f9a561a6-WINDOWS64ms.zip

 

 

It works for us.

 

Leon

 

 

 

	Dipl.-Ing. Leon Thielen Software Development MAGMA Gießereitechnologie GmbH P: +49 241 88901 244  Kackertstrasse 16-18, 52072 Aachen, Germany  www.magmasoft.de L.Thielen@xxxxxxxxxxxx
GERMANY ● USA ● BRAZIL ● SINGAPORE ● SOUTH KOREA ● CHINA ● INDIA ● TURKEY ● CZECH REPUBLIC

Bewertung von Simulationsergebnissen mit MAGMASOFT® / Evaluation of Simulation Results with MAGMASOFT®

Sandguss / Sand Casting: 27.11.2024 + 05.03.2025
Druckguss / High Pressure Die Casting: 06.03.2025

MAGMA Gießereitechnologie GmbH | Kackertstraße 16-18, 52072 Aachen, Germany | Legal form: GmbH, Register court: Aachen HRB 3912, Value added tax identification number: DE121745780 | Management: Dr. Marc C. Schneider (CEO and President), Dipl.-Ing. Mathieu Weber (Managing Director)

Von: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> Im Auftrag von John M Knoeller via HTCondor-users
Gesendet: Montag, 18. November 2024 16:55
An: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Cc: John M Knoeller <johnkn@xxxxxxxxxxx>
Betreff: [EXTERNAL] Re: [HTCondor-users] Does the filetransfer_example_plugin.py work on WINDOWS?

 

Unfortunately, no.   

 

HTCondor will not invoke python.exe as the real executable for a .py file when it is trying to invoke a file transfer plugin, and Windows does not have a system python nor does it have a system level file association for .py and will not do so neither.   So transfer plugins that are python scripts do not currently work on Windows. 

 

-tj

 

 

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Leon Thielen <L.Thielen@xxxxxxxxxxxx>
Sent: Wednesday, November 13, 2024 5:54 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] Does the filetransfer_example_plugin.py work on WINDOWS?

 

Hello,

I work with the condor_version
$CondorVersion: 23.0.12 2024-06-13 BuildID: 739441 $
$CondorPlatform: x86_64_Windows10 $

The manual execution of the command works:
$ python filetransfer_example_plugin.py -infile input -outfile output

Input is:
[ LocalFileName = “D:\\condor\\tmp\\dir_4712\\f9a561a6-WINDOWS64ms.zip”;Url = "">“example://\\\\store\\repo\\services\\magma\\condor\\diagnose+test\\ATestNewHost\\bin\\\f9a561a6-WINDOWS64ms.zip”]


It does not work in a HTCondor JOB. The AP is LINUX.


transfer_plugins =  example = bin/filetransfer_example_plugin.py
example://\\\\store\\repo\\services\\magma\\condor\\diagnose+test\\ATestNewHost\\bin\\f9a561a6-WINDOWS64ms.zip

$ condor_q -better


-- Schedd: avdi0de037.corpdir.zz : <10.20.49.38:9618?...
The Requirements _expression_ for job 267.000 is

    ((OpSys == "WINDOWS") && (Machine == "aws0de227.corpdir.zz")) && (TARGET.Arch == "X86_64") && (TARGET.Disk >= RequestDisk) &&
    (TARGET.Memory >= RequestMemory) && (TARGET.HasJobTransferPlugins)

Job 267.000 defines the following attributes:

    DiskUsage = 1
    ImageSize = 1
    RequestDisk = DiskUsage
    RequestMemory = ifthenelse(MemoryUsage =!= undefined,MemoryUsage,(ImageSize + 1023) / 1024)

The Requirements _expression_ for job 267.000 reduces to these conditions:

         Slots
Step    Matched  Condition
-----  --------  ---------
[0]           2  OpSys == "WINDOWS"
[1]           2  Machine == "aws0de227.corpdir.zz"


267.000:  Job is held.

Hold reason: Transfer input files failure at execution point slot1_1@xxxxxxxxxxxxxxxxxxxx while receiving files from access point avdi0de037. Details: Error from slot1_1@xxxxxxxxxxxxxxxxxxxx: FILETRANSFER:1:|Error: file transfer plugin filetransfer_example_plugin.py exited with code 0, unable to open output file D:\condor\execute\dir_10364/.filetransfer_example_plugin.py.out

Last successful match: Wed Nov 13 11:05:37 2024




Even box_plugin.py does not work.

$ condor_config_val FILETRANSFER_PLUGINS
D:\condor\bin\curl_plugin.exe, D:\condor\bin\data_plugin.exe, D:\condor\bin\box_plugin.py, D:\condor\bin/gdrive_plugin.py, D:\condor\bin/onedrive_plugin.py

$python D:\condor\bin\box_plugin.py -classad
MultipleFileSupport = true
PluginType = "FileTransfer"
SupportedMethods = "box"
Version = "1.1.0"


$ condor_q -better


-- Schedd: avdi0de037.corpdir.zz : <10.20.49.38:9618?...
The Requirements _expression_ for job 264.000 is

    ((OpSys == "WINDOWS") && (Machine == "aws0de227.corpdir.zz")) && (TARGET.Arch == "X86_64") && (TARGET.Disk >= RequestDisk) &&
    (TARGET.Memory >= RequestMemory) && (TARGET.HasFileTransfer && stringListIMember("box",TARGET.HasFileTransferPluginMethods))

Job 264.000 defines the following attributes:

    DiskUsage = 2000000
    ImageSize = 1
    RequestDisk = DiskUsage
    RequestMemory = ifthenelse(MemoryUsage =!= undefined,MemoryUsage,(ImageSize + 1023) / 1024)

The Requirements _expression_ for job 264.000 reduces to these conditions:

         Slots
Step    Matched  Condition
-----  --------  ---------
[0]           2  OpSys == "WINDOWS"
[1]           2  Machine == "aws0de227.corpdir.zz"
[10]          7  stringListIMember("box",TARGET.HasFileTransferPluginMethods)
[12]          0  [0] && [10]

No successful match recorded.
Last failed match: Wed Nov 13 10:42:33 2024


Thank you,
Leon



Dipl.-Ing. Leon Thielen
Software Development
MAGMA Gießereitechnologie GmbH
T: +49 241 88901 244
A: Kackertstrasse 16-18, 52072 Aachen, Germany
W: https://urldefense.com/v3/__http://www.magmasoft.de__;!!Mak6IKo!LxKuJmCcRFBpmE_GlG3Zzd2o6m_8A_LdjuQn1fTZI63OaICAnx2dgQL7FTrzhfLt621Mdj6S_DNgnJTdU1ImCs4y$
E: L.Thielen@xxxxxxxxxxxx
GERMANY ● USA ● BRAZIL ● SINGAPORE ● SOUTH KOREA ● CHINA ● INDIA ● TURKEY ● CZECH REPUBLIC

ßereitechnologie GmbH | Kackertstraße 16-18, 52072 Aachen, Germany | Legal form: GmbH, Register court: Aachen HRB 3912, Value added tax identification number: DE121745780 | Management: Dr. Marc C. Schneider (CEO and President), Dipl.-Ing. Mathieu Weber (Managing Director)

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/