Re: [Condor-users] Managing millions of log files from large DAGs

[Armen Babikyan <armenb@xxxxxxxxxx>]

Stuart Anderson wrote:
> How are people managing the O(1M) files generated by large DAGs (all 
> the .out
> and .err files) that show up in the same directory. There are a few 
> filesystems
> that scale well to having that many files in a single directory, but 
> not many.

I have an application which generates millions of log files as well.  I 
thought about this problem early on - to cope, I split up the single DAG 
into several levels of nested DAGs, and wrote a program to organize 
these DAGs into corresponding levels of nested directories.  This 
strategy has worked quite well for me so far.

Armen

--
Armen Babikyan
MIT Lincoln Laboratory
armenb@xxxxxxxxxx . 781-981-1796