Re: [HTCondor-users] Normal termination: return value 12

[Jeffrey Tsai <jtsai8@xxxxxxxx>]

Thank you for the suggestion! I tried increasing the request_memory value to 32GB, but I am still receiving the same output in my error log.

Is there a way I can see what the hard limit on memory usage is? Or alternatively, check how much memory my software is trying to allocate?

Again, thank you very much for the help!

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Michael Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>
Sent: Wednesday, September 19, 2018 3:42:23 PM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Normal termination: return value 12

 

Hey Jeff,

Exit code 12 on Linux (presumed via the "test.sh" executable) corresponds to a "out of memory" error:

http://www-numi.fnal.gov/offline_software/srt_public_context/WebDocs/Errors/unix_system_errors.html

#define ENOMEM          12      /* Out of memory */

It could be that your administrator has configured your HTCondor starter to impose a hard limit on memory usage based on your request_memory value, which would cause an attempt to allocate more than what you requested to fail, and then lead your executable to throw that ENOMEM error and exit. Or, your software is trying to allocate way, way more memory than it should and hitting a system-wide OOM.

The HTCondor default is to impose only a "soft" limit for memory, where you're guaranteed at least the requested amount of *physical* memory, and allocations over that amount are more likely to be swapped out if the system needs the physical memory, but it would appear that may not be the configuration here. I'm sure one of your pool administrators from wisc.edu is on the list and could comment on that.

Considering that you input-transfer nearly 32GB of files, 16GB of memory could be quite a bit less than what you need, particularly if there's a hard memory limit. Try increasing your request_memory value to 32g instead, and see how it goes. Good luck!

Michael V. Pelletier
Information Technology
Digital Transformation & Innovation
Integrated Defense Systems
Raytheon Company

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/