Re: [DynInst_API:] mutateLibcuda segfaults

[NISARG SHAH <nisargs@xxxxxxxxxxx>]

Thanks Ilya!

It looks like the instrumentation that figures out synchronization function in CUDA did not run completely to the end (it takes around 20-30 minutes to finish).

Do you know if the segfault occurs immediately (within 4-5s) after the last line is printed to screen ("Inserting signal start instra in main")? If this is so, the cause of error might be CUDA's kernel runtime limit. You might need to increase or disable it altogether.

Regards

Nisarg

---

From: Ilya Zhukov
Sent: Sunday, May 10, 2020 4:52 AM
To: NISARG SHAH; dyninst-api@xxxxxxxxxxx
Subject: Re: [DynInst_API:] mutateLibcuda segfaults

Hi Nisarg,

I do not have "MS_outputids.bin" directory but I have 5 *.dot files in
the directory I ran the program.

Cheers,
Ilya

On 09.05.20 00:15, NISARG SHAH wrote:
> Hi Ilya,
>
> From the backtrace, it looks like the error is due to the program not
> being able to read from a temporary file "MS_outputids.bin" that is
> creates initially. Can you check if it exists in the directory from
> where you ran the program? Also, can you check if 5 *.dot files are
> present in the same directory?
>
> Thanks
> Nisarg
>
> ------------------------------------------------------------------------
> *From:* Dyninst-api <dyninst-api-bounces@xxxxxxxxxxx> on behalf of Ilya
> Zhukov <i.zhukov@xxxxxxxxxxxxx>
> *Sent:* Wednesday, May 6, 2020 7:16 AM
> *To:* dyninst-api@xxxxxxxxxxx <dyninst-api@xxxxxxxxxxx>
> *Subject:* [DynInst_API:] mutateLibcuda segfaults
>  
> Dear dyinst developers,
>
> I'm testing your cuda_sync_analyze tool on our cluster for CUDA/10.1.105.
>
> I installed dyinst and cuda_sync_analyze (cmake and make logs in attach)
> successfully. But I get segmentation fault when I create fake CUDA library.
>
> Here is a backtrace
>> #0  0x00002b0a9658c4bc in fseek () from /usr/lib64/libc.so.6
>> #1  0x00002b0a93b7eb29 in LaunchIdentifySync::PostProcessing (this=this@entry=0x7fff1af88af0, allFound=...) at /p/project/cslts/zhukov1/work/tools/dyninst/tools/cuda_sync_analyzer/src/LaunchIdentifySync.cpp:90
>> #2  0x00002b0a93b7c00f in CSA_FindSyncAddress(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) () at /p/project/cslts/zhukov1/work/tools/dyninst/tools/cuda_sync_analyzer/src/FindCudaSync.cpp:34
>> #3  0x00000000004021fb in main () at /p/project/cslts/zhukov1/work/tools/dyninst/tools/cuda_sync_analyzer/src/main.cpp:15
>> #4  0x00002b0a96537505 in __libc_start_main () from /usr/lib64/libc.so.6
>> #5  0x000000000040253e in _start () at /p/project/cslts/zhukov1/work/tools/dyninst/tools/cuda_sync_analyzer/src/main.cpp:38
>
> Any help will be appreciated. If you need anything else let me know.
>
> Best wishes,
> Ilya
> --
> Ilya Zhukov
> Juelich Supercomputing Centre
> Institute for Advanced Simulation
> Forschungszentrum Juelich GmbH
> 52425 Juelich, Germany
>
> Phone: +49-2461-61-2054
> Fax: +49-2461-61-2810
> E-mail: i.zhukov@xxxxxxxxxxxxx
> WWW: http://www.fz-juelich.de/jsc