Re: [HTCondor-users] transfer_in/output_files only if they exist

[Duncan Brown <dabrown@xxxxxxx>]

Hi TJ,

Thanks. I'm using Pegasus to make the DAG and it really wants to deal with files, not directories. However, since the checkpoint files aren't passed around in the workflow, I think I can use this as a workaround.

Cheers,
Duncan.

> On Feb 7, 2019, at 12:45 PM, John M Knoeller <johnkn@xxxxxxxxxxx> wrote:
> 
> You can do this by using a directory transfer.  If you specify a directory in the transfer in/out list.  then all of the files in the directory will be transferred - but it is not an error for the directory to be empty.  so if checkpoint is a directory on the submit side, then 
> 
> transfer_input_files = checkpoint
> transfer_output_files = checkpoint
> 
> Will transfer everything in the checkpoint directory on the submit side, to a checkpoint directory on the execute node, and then
> transfer everything from that directory back to the submit node. 
> 
> Note - It is an error if the checkpoint directory does not exist on the submit side before you submit the job.
> 
> -tj
> 
> -----Original Message-----
> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Duncan Brown
> Sent: Thursday, February 7, 2019 8:01 AM
> To: htcondor-users@xxxxxxxxxxx
> Subject: [HTCondor-users] transfer_in/output_files only if they exist
> 
> Hi Todd,
> 
> Is there a way to tell condor that it's OK if a specific file listed in transfer_input_files does not exist (and the same question with an output file)? The use case is using condor file i/o to manage a checkpoint file. The first time the job is run, the checkpoint file does not exist so the job gets stuck in hold state. I want to be able to tell condor that it's OK that this file is not there.
> 
> Cheers,
> Duncan.
> 
> -- 
> 
> Duncan Brown                              Room 263-1, Physics Department
> Charles Brightman Professor of Physics     Syracuse University, NY 13244
> http://dabrown.expressions.syr.edu                   Phone: 315 443 5993
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

-- 

Duncan Brown                              Room 263-1, Physics Department
Charles Brightman Professor of Physics     Syracuse University, NY 13244
http://dabrown.expressions.syr.edu                   Phone: 315 443 5993