Re: [Condor-devel] mmap issue

[Erik Paulson <epaulson@xxxxxxxxxxx>]

On Fri, Mar 16, 2007 at 02:06:48PM -0500, Greg Thain wrote:
> Jiansheng Huang wrote:
> > Hi team,
> > 
> > I am periodically seeing the following crash when the new quill tries to
> > compute checksum of an output file used by a job. I used the mmap function
> > to map in the data from the file before passing it to the checksum
> > function. The return status is checked to make sure that it is not
> > MAP_FAILED before doing checksum.
> 
> Are you sure mmap is the right way to go?  What if the file is bigger 
> than your address space?
> 

Instead of mmap, it should read the file in.

Our code now (greatly simplified)

Condor_MD_MAC *checker = new Condor_MD_MAC();
fd = safe_open_wrapper(filePathName, O_RDONLY, 0);
data = (char *)mmap(0, fileSize, PROT_READ, MAP_PRIVATE, fd, 0);
checker->addMD((unsigned char *) data, fileSize);
checksum = checker->computeMD();

It seems like computing the checksum of a file could be a common operation,
so it might be better to augment the Condor_MD_MAC object with an

Condor_MD_MAC::addfile(const char *pathname)

which would either mmap the file in, or read it in in chunks, etc. 

If we were really clever, we'd arrange it so that we could do the checksum
calculations in another thread, once DaemonCore can be threaded. 

-Erik