Mailing List Archives
Authenticated access
|
|
|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] transfer_in/output_files only if they exist
- Date: Thu, 7 Feb 2019 18:21:24 +0000
- From: John M Knoeller <johnkn@xxxxxxxxxxx>
- Subject: Re: [HTCondor-users] transfer_in/output_files only if they exist
the trailing / does let HTCondor know it's a directory, but it ALSO says that the contents of the directory should be transferred to the root of the destination, so that doesn't really work here because we need the checkpoint file to stay in a directory.
-tj
-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Tom Downes
Sent: Thursday, February 7, 2019 12:16 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] transfer_in/output_files only if they exist
Good to know! When I encountered that, I saw a user education nightmare.
I recall trying some trailing-slash tricks to see whether that made condor think about the arguments as if they are a directory. Is anything like that supported?
Tom
ïOn 2/7/19, 12:10 PM, "HTCondor-users on behalf of John M Knoeller" <htcondor-users-bounces@xxxxxxxxxxx on behalf of johnkn@xxxxxxxxxxx> wrote:
I think that only happens when you enable filechecks, which is now off by default.
condor_submit has no way of knowing that
transfer_output_files = checkpoint
refers to a directory if the directory does not yet exist. So with filechecks enabled, condor_submit would create a zero length file in order
to see if you have permission to transfer the output. This would then prevent creating a directory later.
But filechecks is now off by default, in part because it caused problems like this.
-tj
-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Tom Downes
Sent: Thursday, February 7, 2019 11:56 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] transfer_in/output_files only if they exist
TJ:
In my experience, when the directory doesn't exist at submit time, condor_submit creates a file that clobbers the reverse transfer of the directory upon job completion. So there's some badput involved.
I haven't tested that experience in a while, though, so can't claim that it affects today's stable/dev releases.
Tom
On 2/7/19, 11:49 AM, "HTCondor-users on behalf of John M Knoeller" <htcondor-users-bounces@xxxxxxxxxxx on behalf of johnkn@xxxxxxxxxxx> wrote:
You can do this by using a directory transfer. If you specify a directory in the transfer in/out list. then all of the files in the directory will be transferred - but it is not an error for the directory to be empty. so if checkpoint is a directory on the submit side, then
transfer_input_files = checkpoint
transfer_output_files = checkpoint
Will transfer everything in the checkpoint directory on the submit side, to a checkpoint directory on the execute node, and then
transfer everything from that directory back to the submit node.
Note - It is an error if the checkpoint directory does not exist on the submit side before you submit the job.
-tj
-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Duncan Brown
Sent: Thursday, February 7, 2019 8:01 AM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] transfer_in/output_files only if they exist
Hi Todd,
Is there a way to tell condor that it's OK if a specific file listed in transfer_input_files does not exist (and the same question with an output file)? The use case is using condor file i/o to manage a checkpoint file. The first time the job is run, the checkpoint file does not exist so the job gets stuck in hold state. I want to be able to tell condor that it's OK that this file is not there.
Cheers,
Duncan.
--
Duncan Brown Room 263-1, Physics Department
Charles Brightman Professor of Physics Syracuse University, NY 13244
http://dabrown.expressions.syr.edu Phone: 315 443 5993
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/