Re: [Condor-users] Simultaneous writes to post-script in DAGMan

[Jeff Weber <weber@xxxxxxxxxxx>]

Kshitij:

DAGMan does not provide any additional tools for helping with historic
NFS synchronization issues.  So a POST script must be able to handle all
NFS file synchronization issues on its own.  If you pursue this path,
bear in mind:

* Never use NFS for your DAGMan job log files.  This has been the source
of many problems in the past.

* The DAG POST script runs on the host running DAGMan itself.  If you
have a pool with more than one node, there is a significant chance that
that DAGMan and the individual node jobs will run on different hosts. 
This provides another NFS file synchronization challenge.

Note that DAGMan provides a "-maxpost" option to throttle the number of
active POST scripts.

	Jeff

On Wed, 2005-01-05 at 01:36, Kshitij Sanghi wrote:
> Hi,
> 
> I am using DAGMan to submit 300,000 jobs. The CONDOR
> uses an NFS. I was planning to run a PERL script to
> delete the unnecessary files after each node has
> completed execution as I have limited space on my NFS.
> I can do so for the log, err, etc files but there is
> this one result file from which I want to extract
> relevant output data.
> 
> What I wanted to know is that whether it is possible
> to include in the post script a piece of code to
> extract information from these files and write it to a
> single file. In such a situation I would not have to
> maintain 300,000 of these result files but just a
> single file with relevant information. Basically I
> wanted to know whether such a thing would be possible
> without any chances of conflicts due to simultaneous
> writes to the same file on the NFS.
> 
> In case such a thing is not possible would it be
> possible to extract the relevant information from
> these files and populate a MySQL database in the post
> script itself?
> 
> Thanks and Regards,
> Kshitij
> 
> =====
> I've never met a man so ignorant that I couldn't learn something from him.
>                                                                                          -Galileo Galilie
> 
> 
> 	
> 	
> 		
> ___________________________________________________________ 
> ALL-NEW Yahoo! Messenger - all new features - even more fun! http://uk.messenger.yahoo.com
> _______________________________________________
> Condor-users mailing list
> Condor-users@xxxxxxxxxxx
> http://lists.cs.wisc.edu/mailman/listinfo/condor-users
-- 
Jeff Weber
University of Wisconsin, Madison