Re: [Condor-users] java machines with > 2gb

[Amitava <raychaudhuri_amitava@xxxxxxxxx>]

Ideally in 32-bit OS no process should have more than 4 GB space.

Thanks & regards
Amitava

--- On Tue, 9/16/08, Joe Meehean <jmeehean@xxxxxxxxxxx> wrote:

From: Joe Meehean <jmeehean@xxxxxxxxxxx>
Subject: Re: [Condor-users] java machines with > 2gb
To: "Condor-Users Mail List" <condor-users@xxxxxxxxxxx>
Date: Tuesday, September 16, 2008, 2:57 AM

This issue has everything to do with 32 vs. 64-bit 
operating systems.  With 32-bit OS's no process 
can have more than a 3GB address space.  If you 
need more than 2GB of Java heap space ensure you 
are submitting your job to a 64-bit machine with a 
64-bit OS.  If you don't need more than 2GB's, 
follow the
 workaround.

_joe



Matthew Farrellee wrote:
> A quick read of the work around[1], a few runs of java -Xmx... NOTHING, 
> and it seems this isn't entirely Condor's fault.
> 
> $ java -Xmx4g NOTHING
> Invalid maximum heap size: -Xmx4g
> The specified size exceeds the maximum representable size.
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> 
> vs
> 
> $ java -Xmx1g NOTHING
> Exception in thread "main" java.lang.NoClassDefFoundError:
NOTHING
> ...
> 
> On my 32 bit system, the JVM is not able to handle the 4GB max heap size 
> I've requested, but it can handle 1GB.
> 
> Condor is trying to be helpful and set the max heap size for you by 
> default, specifically to sysapi_phys_memory()/ sysapi_ncpus(). The 
> problem is sometimes that number is
 too big for the JVM. In those cases 
> you need to stop Condor from trying to help you out. This is what 
> setting JAVA_MAXHEAP_ARGUMENT to nothing is all about.
> 
>  From src/condor_c++_util/java_config.C:
> 
>     tmp = param("JAVA_MAXHEAP_ARGUMENT");
>     if(tmp) {
>        arg_buf.sprintf("%s%dm",tmp,sysapi_phys_memory()/
sysapi_ncpus());
>        args->AppendArg(arg_buf.Value());
>        free(tmp);
>     }
> 
> If it isn't set, i.e. "JAVA_MAXHEAP_ARGUMENT=" in your
config file, then 
> no max heap is passed to the JVM.
> 
> The work around mentions setting your own max heap with:
> JAVA_EXTRA_ARGUMENTS = -Xmx1906m
> 
> That is well and good, as long as you are setting the max heap to a size 
> that the JVM can handle! If you were to JAVA_EXTRA_ARGUMENTS = -Xmx4g on 
> my machine you'd still have a
 problem.
> 
> So, you can either not specify a max heap, and deal with whatever fall 
> out there is there if any, or make sure you are setting it to something 
> reasonable with the little java -Xmx... NOTHING test above. Be careful 
> though, this test doesn't seem to be fool-proof. From my testing:
> 
> $ java -Xmx2670m test
> Error occurred during initialization of VM
> ...
> $ java -Xmx2669m test
> Exception in thread "main" java.lang.NoClassDefFoundError: test
> ...
> $ java -Xmx2670m test
> Exception in thread "main" java.lang.NoClassDefFoundError: test
> ...
> $ java -Xmx2671m test
> Exception in thread "main" java.lang.NoClassDefFoundError: test
> ...
> $ java -Xmx2672m test
> Error occurred during initialization of VM
> ...
> $ java -Xmx2671m test
> Error occurred during initialization of VM
>
 ...
> 
> Best,
> 
> 
> matt
> 
> [1] http://www.cs.wisc.edu/condor/manual/v7.1/3_13Java_Support.html#29892
> 
> Michael Rusch wrote:
>> I share your surprise that it hasn't been fixed after 5+ years,
especially
>> now that machines with 2+gb of RAM are not really extraordinary. 
Condor
>> team: are there any plans to fix this?
>>
>> I'll probably wind up following suit and reverting to the vanilla
universe.
>> We have a mixed linux/windows pool, so I'm assuming I'll need
to write two
>> bootstrap scripts named with the appropriate magic so that I can write
the
>> submit file to use the correct one based on target machine's
architecture.
>> Is there any way that I can get around having to make sure Java is
installed
>> in the same place on all of the machines of the same architecture
 (and
then
>> hardcoding that path into the bootstrapping scripts)? 
>>
>> Michael
>>
>>> -----Original Message-----
>>> From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-
>>> bounces@xxxxxxxxxxx] On Behalf Of Filip Defoort
>>> Sent: Monday, September 08, 2008 1:34 PM
>>> To: Condor-Users Mail List
>>> Subject: Re: [Condor-users] java machines with > 2gb
>>>
>>> The only solution I found was bypassing the java universe
altogether
>>> and starting your own JVM via a regular java invocation.
>>> Works, but kinda defeats the purpose of the java universe.
>>>
>>> Quite surprised this hasn't been fixed yet - this bug has been
there for
>>> years.
>>> - Filip
>>>
>>>
>>> On Mon, Sep 8, 2008 at 11:25 AM,
 Michael Rusch
<mcrusch@xxxxxxxx> wrote:
>>>> Has anybody found a way with recent versions of condor to get
Java nodes
>>>> with >2GB of RAM to show up?  The workaround from the
manual is not
>>> working
>>>> for me.
>>>>
>>>>
>>>>
>>>> Michael
>>>>
>>>> _______________________________________________
>>>> Condor-users mailing list
>>>> To unsubscribe, send a message to
condor-users-request@xxxxxxxxxxx with
>>> a
>>>> subject: Unsubscribe
>>>> You can also unsubscribe by visiting
>>>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>>
>>>> The archives can be found at:
>>>>
 https://lists.cs.wisc.edu/archive/condor-users/
>>>>
>>>>
>>> _______________________________________________
>>> Condor-users mailing list
>>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/condor-users/
>> _______________________________________________
>> Condor-users mailing list
>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>
>> The archives can be found at:
 
>> https://lists.cs.wisc.edu/archive/condor-users/
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> 
> The archives can be found at: 
> https://lists.cs.wisc.edu/archive/condor-users/
_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at: 
https://lists.cs.wisc.edu/archive/condor-users/