Re: [HTCondor-users] Jobs using large memory but profiler says that jobs are fine?

[Ben Jones <ben.dylan.jones@xxxxxxx>]

Do you know if youâre running on an el9 machine or similar? I think that on cgroup v2 machines the memory reporting currently reports file cache. Are you writing a large file? 

On 5 Mar 2024, at 17:08, Angel Campoverde <angelfcampoverde@xxxxxxxxx> wrote:

ï

Dear Condor experts,

I am running a job that seems to be going beyond 8Gb in memory usage. However when I run it with memray (a memory profiler for python projects) I see that my memory usage is only 107Mb, as you can see in the screenshot below:

<image.png>

I have contacted the python mailing list:

https://discuss.python.org/t/concurrent-features-vs-multiprocessing-vs-suprocess/47653/5

and despite I try to turn off multithreading, I still get this high memory usage. Why is this happening and how should I fix it? I cannot use 16Gb of memory in my jobs, there are very few machines that have that much memory.

Cheers.

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/