[Condor-users] Memory issues when running condor jobs:

[Yeye He <heyeye@xxxxxxxxxxx>]

Hi all,

I am running a very memory-intensive job via Condor and every time when 
the virtual memory size goes beyond 1.6GB or so, the job was evicted and 
never picked up by anyone (condor_q -analyze shows that all machines 
that qualifies to run the program reject to do so). I understand it may 
have something to do with machine's local job policy, where when the 
image size of my job exceeds certain threshold it gets evicted.

One obvious solution on my end is to limit the memory footprint of this 
program. But this is a "naive" approach that someone proposed in a paper 
and we are proposing something else to beat it. We are trying to get 
some data points on some non-trivial dataset to show that we indeed beat 
it in terms of both efficiency and quality. The problem is that in the 
"naive" approach its memory footprint can easily go beyond 2-3 GB, which 
my 32bit workstation cannot handle due to limits of address space 
(allocation error). I don't have access to a 64 bit machine so that is 
why I am using Condor to start with.

I am just wondering, without rewriting current implementation 
(potentially by moving some data to disk and removing them from in-mem 
structure), is there any workaround to this problem?

Any suggestions are highly appreciated!
-Yeye