Re: [HTCondor-users] transfer_in/output_files only if they exist

[John M Knoeller <johnkn@xxxxxxxxxxx>]

I think that only happens when you enable filechecks, which is now off by default.

condor_submit has no way of knowing that 

     transfer_output_files = checkpoint
  
refers to a directory if the directory does not yet exist.  So with filechecks enabled, condor_submit would create a zero length file in order
to see if you have permission to transfer the output. This would then prevent creating a directory later. 

But filechecks is now off by default, in part because it caused problems like this.

-tj


-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Tom Downes
Sent: Thursday, February 7, 2019 11:56 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] transfer_in/output_files only if they exist

TJ:

In my experience, when the directory doesn't exist at submit time, condor_submit creates a file that clobbers the reverse transfer of the directory upon job completion. So there's some badput involved.

I haven't tested that experience in a while, though, so can't claim that it affects today's stable/dev releases.

Tom

ïOn 2/7/19, 11:49 AM, "HTCondor-users on behalf of John M Knoeller" <htcondor-users-bounces@xxxxxxxxxxx on behalf of johnkn@xxxxxxxxxxx> wrote:

    You can do this by using a directory transfer.  If you specify a directory in the transfer in/out list.  then all of the files in the directory will be transferred - but it is not an error for the directory to be empty.  so if checkpoint is a directory on the submit side, then 
    
    transfer_input_files = checkpoint
    transfer_output_files = checkpoint
    
    Will transfer everything in the checkpoint directory on the submit side, to a checkpoint directory on the execute node, and then
    transfer everything from that directory back to the submit node. 
    
    Note - It is an error if the checkpoint directory does not exist on the submit side before you submit the job.
    
    -tj
    
    -----Original Message-----
    From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Duncan Brown
    Sent: Thursday, February 7, 2019 8:01 AM
    To: htcondor-users@xxxxxxxxxxx
    Subject: [HTCondor-users] transfer_in/output_files only if they exist
    
    Hi Todd,
    
    Is there a way to tell condor that it's OK if a specific file listed in transfer_input_files does not exist (and the same question with an output file)? The use case is using condor file i/o to manage a checkpoint file. The first time the job is run, the checkpoint file does not exist so the job gets stuck in hold state. I want to be able to tell condor that it's OK that this file is not there.
    
    Cheers,
    Duncan.
    
    -- 
    
    Duncan Brown                              Room 263-1, Physics Department
    Charles Brightman Professor of Physics     Syracuse University, NY 13244
    http://dabrown.expressions.syr.edu                   Phone: 315 443 5993
    
    
    _______________________________________________
    HTCondor-users mailing list
    To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
    subject: Unsubscribe
    You can also unsubscribe by visiting
    https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
    
    The archives can be found at:
    https://lists.cs.wisc.edu/archive/htcondor-users/
    
    _______________________________________________
    HTCondor-users mailing list
    To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
    subject: Unsubscribe
    You can also unsubscribe by visiting
    https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
    
    The archives can be found at:
    https://lists.cs.wisc.edu/archive/htcondor-users/