Re: [Condor-users] Simultaneous writes to post-script in DAGMan

[Kshitij Sanghi <kshitij_sanghi@xxxxxxxxx>]

Thanks.

 --- Jeff Weber <weber@xxxxxxxxxxx> wrote: 
> Kshitij:
> 
> DAGMan does not provide any additional tools for
> helping with historic
> NFS synchronization issues.  So a POST script must
> be able to handle all
> NFS file synchronization issues on its own.  If you
> pursue this path,
> bear in mind:
> 
> * Never use NFS for your DAGMan job log files.  This
> has been the source
> of many problems in the past.
> 
> * The DAG POST script runs on the host running
> DAGMan itself.  If you
> have a pool with more than one node, there is a
> significant chance that
> that DAGMan and the individual node jobs will run on
> different hosts. 
> This provides another NFS file synchronization
> challenge.
> 
> Note that DAGMan provides a "-maxpost" option to
> throttle the number of
> active POST scripts.
> 
> 	Jeff
> 
> On Wed, 2005-01-05 at 01:36, Kshitij Sanghi wrote:
> > Hi,
> > 
> > I am using DAGMan to submit 300,000 jobs. The
> CONDOR
> > uses an NFS. I was planning to run a PERL script
> to
> > delete the unnecessary files after each node has
> > completed execution as I have limited space on my
> NFS.
> > I can do so for the log, err, etc files but there
> is
> > this one result file from which I want to extract
> > relevant output data.
> > 
> > What I wanted to know is that whether it is
> possible
> > to include in the post script a piece of code to
> > extract information from these files and write it
> to a
> > single file. In such a situation I would not have
> to
> > maintain 300,000 of these result files but just a
> > single file with relevant information. Basically I
> > wanted to know whether such a thing would be
> possible
> > without any chances of conflicts due to
> simultaneous
> > writes to the same file on the NFS.
> > 
> > In case such a thing is not possible would it be
> > possible to extract the relevant information from
> > these files and populate a MySQL database in the
> post
> > script itself?
> > 
> > Thanks and Regards,
> > Kshitij



	
	
		
___________________________________________________________ 
ALL-NEW Yahoo! Messenger - all new features - even more fun! http://uk.messenger.yahoo.com