Re: [HTCondor-users] transfer_in/output_files only if they exist

[Tom Downes <downes@xxxxxxx>]

Good to know! When I encountered that, I saw a user education nightmare.

I recall trying some trailing-slash tricks to see whether that made condor think about the arguments as if they are a directory. Is anything like that supported?

Tom

ïOn 2/7/19, 12:10 PM, "HTCondor-users on behalf of John M Knoeller" <htcondor-users-bounces@xxxxxxxxxxx on behalf of johnkn@xxxxxxxxxxx> wrote:

    I think that only happens when you enable filechecks, which is now off by default.
    
    condor_submit has no way of knowing that 
    
         transfer_output_files = checkpoint
      
    refers to a directory if the directory does not yet exist.  So with filechecks enabled, condor_submit would create a zero length file in order
    to see if you have permission to transfer the output. This would then prevent creating a directory later. 
    
    But filechecks is now off by default, in part because it caused problems like this.
    
    -tj
    
    
    -----Original Message-----
    From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Tom Downes
    Sent: Thursday, February 7, 2019 11:56 AM
    To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
    Subject: Re: [HTCondor-users] transfer_in/output_files only if they exist
    
    TJ:
    
    In my experience, when the directory doesn't exist at submit time, condor_submit creates a file that clobbers the reverse transfer of the directory upon job completion. So there's some badput involved.
    
    I haven't tested that experience in a while, though, so can't claim that it affects today's stable/dev releases.
    
    Tom
    
    On 2/7/19, 11:49 AM, "HTCondor-users on behalf of John M Knoeller" <htcondor-users-bounces@xxxxxxxxxxx on behalf of johnkn@xxxxxxxxxxx> wrote:
    
        You can do this by using a directory transfer.  If you specify a directory in the transfer in/out list.  then all of the files in the directory will be transferred - but it is not an error for the directory to be empty.  so if checkpoint is a directory on the submit side, then 
        
        transfer_input_files = checkpoint
        transfer_output_files = checkpoint
        
        Will transfer everything in the checkpoint directory on the submit side, to a checkpoint directory on the execute node, and then
        transfer everything from that directory back to the submit node. 
        
        Note - It is an error if the checkpoint directory does not exist on the submit side before you submit the job.
        
        -tj
        
        -----Original Message-----
        From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Duncan Brown
        Sent: Thursday, February 7, 2019 8:01 AM
        To: htcondor-users@xxxxxxxxxxx
        Subject: [HTCondor-users] transfer_in/output_files only if they exist
        
        Hi Todd,
        
        Is there a way to tell condor that it's OK if a specific file listed in transfer_input_files does not exist (and the same question with an output file)? The use case is using condor file i/o to manage a checkpoint file. The first time the job is run, the checkpoint file does not exist so the job gets stuck in hold state. I want to be able to tell condor that it's OK that this file is not there.
        
        Cheers,
        Duncan.
        
        -- 
        
        Duncan Brown                              Room 263-1, Physics Department
        Charles Brightman Professor of Physics     Syracuse University, NY 13244
        http://dabrown.expressions.syr.edu                   Phone: 315 443 5993
        
        
        _______________________________________________
        HTCondor-users mailing list
        To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
        subject: Unsubscribe
        You can also unsubscribe by visiting
        https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
        
        The archives can be found at:
        https://lists.cs.wisc.edu/archive/htcondor-users/
        
        _______________________________________________
        HTCondor-users mailing list
        To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
        subject: Unsubscribe
        You can also unsubscribe by visiting
        https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
        
        The archives can be found at:
        https://lists.cs.wisc.edu/archive/htcondor-users/
        
    
    _______________________________________________
    HTCondor-users mailing list
    To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
    subject: Unsubscribe
    You can also unsubscribe by visiting
    https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
    
    The archives can be found at:
    https://lists.cs.wisc.edu/archive/htcondor-users/

Attachment: smime.p7s
Description: S/MIME cryptographic signature