[Condor-users] Problem transferring files to initialdir?

[Jason Brown <brown27@xxxxxxxxx>]

Dear Condor Community,

 

This is my first post, so please bear with me!

I have a question regarding a behavior of condor that is puzzling me. I have Condor on a single standalone Windows machine as both server and client.  I am trying to run a job that goes great and seems to produce all the outputs (in C:\execute\dirWXYZ). When I look at my "initialdir" directory (C:\study\output) I have about 80% of the output files in there.

From watching this thing run 6 times in a row, here is what I think is happening:

* My job runs successfully, producing outputs that are 3450 CSV files

* I can tell my job ends because the last part of the script deletes XML files; they are gone from C:\execute\dirWXYZ

* Condor, I think, starts to move these CSVs over to C:\study

* It gets through about 80% of them and apparently has an error

* Condor re-runs the job from the start due to the error

 So I am trying to debug this an hoping someone might have seen a similar behavior or know a fix. There are lots of files, but the size comes to only 5-7MB. I found a post online about output files in the 2GB range causing problems, but nothing else obviously relevant.

Thanks in advance,

Jason