Re: [HTCondor-users] transfer_in/output_files only if they exist

[Tom Downes <downes@xxxxxxx>]

Now I remember - thanks.

At least HTCondor and rsync behave in the same well-defined-but-highly-detailed way.
 
Tom

ïOn 2/7/19, 12:22 PM, "HTCondor-users on behalf of John M Knoeller" <htcondor-users-bounces@xxxxxxxxxxx on behalf of johnkn@xxxxxxxxxxx> wrote:

    the trailing / does let HTCondor know it's a directory, but it ALSO says that the contents of the directory should be transferred to the root of the destination, so that doesn't really work here because we need the checkpoint file to stay in a directory.
    
    -tj
    
    -----Original Message-----
    From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Tom Downes
    Sent: Thursday, February 7, 2019 12:16 PM
    To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
    Subject: Re: [HTCondor-users] transfer_in/output_files only if they exist
    
    Good to know! When I encountered that, I saw a user education nightmare.
    
    I recall trying some trailing-slash tricks to see whether that made condor think about the arguments as if they are a directory. Is anything like that supported?
    
    Tom
    
    On 2/7/19, 12:10 PM, "HTCondor-users on behalf of John M Knoeller" <htcondor-users-bounces@xxxxxxxxxxx on behalf of johnkn@xxxxxxxxxxx> wrote:
    
        I think that only happens when you enable filechecks, which is now off by default.
        
        condor_submit has no way of knowing that 
        
             transfer_output_files = checkpoint
          
        refers to a directory if the directory does not yet exist.  So with filechecks enabled, condor_submit would create a zero length file in order
        to see if you have permission to transfer the output. This would then prevent creating a directory later. 
        
        But filechecks is now off by default, in part because it caused problems like this.
        
        -tj
        
        
        -----Original Message-----
        From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Tom Downes
        Sent: Thursday, February 7, 2019 11:56 AM
        To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
        Subject: Re: [HTCondor-users] transfer_in/output_files only if they exist
        
        TJ:
        
        In my experience, when the directory doesn't exist at submit time, condor_submit creates a file that clobbers the reverse transfer of the directory upon job completion. So there's some badput involved.
        
        I haven't tested that experience in a while, though, so can't claim that it affects today's stable/dev releases.
        
        Tom
        
        On 2/7/19, 11:49 AM, "HTCondor-users on behalf of John M Knoeller" <htcondor-users-bounces@xxxxxxxxxxx on behalf of johnkn@xxxxxxxxxxx> wrote:
        
            You can do this by using a directory transfer.  If you specify a directory in the transfer in/out list.  then all of the files in the directory will be transferred - but it is not an error for the directory to be empty.  so if checkpoint is a directory on the submit side, then 
            
            transfer_input_files = checkpoint
            transfer_output_files = checkpoint
            
            Will transfer everything in the checkpoint directory on the submit side, to a checkpoint directory on the execute node, and then
            transfer everything from that directory back to the submit node. 
            
            Note - It is an error if the checkpoint directory does not exist on the submit side before you submit the job.
            
            -tj
            
            -----Original Message-----
            From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Duncan Brown
            Sent: Thursday, February 7, 2019 8:01 AM
            To: htcondor-users@xxxxxxxxxxx
            Subject: [HTCondor-users] transfer_in/output_files only if they exist
            
            Hi Todd,
            
            Is there a way to tell condor that it's OK if a specific file listed in transfer_input_files does not exist (and the same question with an output file)? The use case is using condor file i/o to manage a checkpoint file. The first time the job is run, the checkpoint file does not exist so the job gets stuck in hold state. I want to be able to tell condor that it's OK that this file is not there.
            
            Cheers,
            Duncan.
            
            -- 
            
            Duncan Brown                              Room 263-1, Physics Department
            Charles Brightman Professor of Physics     Syracuse University, NY 13244
            http://dabrown.expressions.syr.edu                   Phone: 315 443 5993
            
            
            _______________________________________________
            HTCondor-users mailing list
            To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
            subject: Unsubscribe
            You can also unsubscribe by visiting
            https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
            
            The archives can be found at:
            https://lists.cs.wisc.edu/archive/htcondor-users/
            
            _______________________________________________
            HTCondor-users mailing list
            To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
            subject: Unsubscribe
            You can also unsubscribe by visiting
            https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
            
            The archives can be found at:
            https://lists.cs.wisc.edu/archive/htcondor-users/
            
        
        _______________________________________________
        HTCondor-users mailing list
        To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
        subject: Unsubscribe
        You can also unsubscribe by visiting
        https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
        
        The archives can be found at:
        https://lists.cs.wisc.edu/archive/htcondor-users/
    
    _______________________________________________
    HTCondor-users mailing list
    To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
    subject: Unsubscribe
    You can also unsubscribe by visiting
    https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
    
    The archives can be found at:
    https://lists.cs.wisc.edu/archive/htcondor-users/

Attachment: smime.p7s
Description: S/MIME cryptographic signature