single thread cause core dump

[qiankunhuanying]

I make the source code to generate the libopenblas*.a, and use the APIs in my.a. Then I make a new.so  including both libopenblas*.a and my.a,  and make an executable file that reference this new.so to test my code. When I use the multi-thread libopenblas*.a(just use "make ONLY_CBLAS=1"), the main() function works normally. but when I use the single-thread libopenblas*.a(use "make USE_THREAD=0"), the main() causes core dump--"segmentation fault(core dump)".
I tried version 0.2.18 and 0.2.20, but both has the core dump, even I just called functions that donot call openblas APIs.
My platform is allwinner-R16, CPU is ARMV7, and I made the libopenblas*.a with cross compiler.

[martin-frbg]

1. Did you do a "make clean" before the second "make" with different options (the "make USE_THREAD=0") ?
2. Can you use a debugger (such as gdb) on that platform to find where the core dump occurs ?

[qiankunhuanying]

@martin-frbg thank you very much.

1. yes, I make clean first.
2. as i used a *.so, the gdb on my device noticed that "warning: unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code."
   so it didnot tell the function having core dump. I will change *.so to just call *.c file, and use GDB to have a try. then I will update the results.

[ashwinyes]

@qiankunhuanying From your first post, looks like you have used the following steps to compile.
Compiling for Multithread - "make ONLY_CBLAS=1"
Compiling for single thread - "make USE_THREAD=0"

If the above are the steps that you followed, then they are not consistent with each other.
Could you try compiling single thread with "make ONLY_CBLAS=1 USE_THREAD=0" just to confirm that it is not related to using the make param. "ONLY_CBLAS=1"

[brada4]

Such crash is usually sign of calling convention mismatch, i.e ld.so calls library initialiser function before main()
probably best start is to run 'file' command against libc and init on the target system and your built .so files

[qiankunhuanying]

@ashwinyes
I didnot describe concretely in my first post,

1. compiling for Multithread, I used
   "make clean
   make TARGET=ARMV7 HOSTCC=gcc CC=$TOOLCHAIN_BIN/arm-openwrt-linux-gcc AR=$TOOLCHAIN_BIN/arm-openwrt-linux-ar USE_THREAD=1 ONLY_CBLAS=1"
   or
   "make clean
   make TARGET=ARMV7 HOSTCC=gcc CC=$TOOLCHAIN_BIN/arm-openwrt-linux-gcc AR=$TOOLCHAIN_BIN/arm-openwrt-linux-ar ONLY_CBLAS=1"

2. compiling for single thread, I used
   "make clean
   make TARGET=ARMV7 HOSTCC=gcc CC=$TOOLCHAIN_BIN/arm-openwrt-linux-gcc AR=$TOOLCHAIN_BIN/arm-openwrt-linux-ar USE_THREAD=0 ONLY_CBLAS=1"
   So ,the difference is just the USE_THREAD

[qiankunhuanying]

@martin-frbg @ashwinyes
I tried to call the functions ,which call openblas APIs, directely( not wrapper to a lib**.so) in main() by pthread_create() . Result is the same (core dump with single thread libopenblas.a, and the functions in pthread_create() didnot enter any more; normal with multithread libopenblas.a).
but if I call the functions() in main(), just as "function1(); function2() ;", it can work normally.
So ,I tried just linking libopenblas.a without calling my other functions. the same result comes.
As ,I used -lpthread and ./libopenblas.a in makefile , I guess maybe libopenblas.a of single thread would change the -lpthread, so it caused the confusion of pthread_create()? or there is a libpthread.so in libopenblas.a?

[poryfly]

i also have a coredump when single thread.
my step is:
make clean
make USE_THREAD=0
make install
my test pragram is muti-thread runing in a docker with one cpu,the error info is:
BLAS : Program is Terminated. Because you tried to allocate too many memory regions.
Segmentation fault
that is why?
if i use single thread type,whether should i need set NUM_THREADS

[qiankunhuanying]

@poryfly I also got your error info once, unfortunately, I saw the file USAGE.md, and get the information below:

"#### Program is Terminated. Because you tried to allocate too many memory regions

In OpenBLAS, we mange a pool of memory buffers and allocate the number of
buffers as the following.

#define NUM_BUFFERS (MAX_CPU_NUMBER * 2)

This error indicates that the program exceeded the number of buffers.

Please build OpenBLAS with larger NUM_THREADS. For example, make NUM_THREADS=32 or make NUM_THREADS=64. In Makefile.system, we will set
MAX_CPU_NUMBER=NUM_THREADS."

And I set NUM_THREADS=64, but nothing changed... I have been very confused.

[ashwinyes]

@qiankunhuanying @poryfly Could you let us know how many threads your program has running concurrently? And which OpenBLAS APIs are your threads calling?

[poryfly]

@ashwinyes i just want to test single thread.so my docker(ubuntu) just have one cpu.my pragram has 5 threads.I call cblas_sgemm

[qiankunhuanying]

@ashwinyes

1. For this issue, finally I reduced to just 2 threads, the main thread and the new thread created by pthread_create(). In this situation, if I use single thread openblas, core dump occured. But if I just call the function in main thread ,not using pthread_create(), it work normally.
2. As the interfaces that call Openblas APIs are provided by other people, I donot know the concrete APIs.

[ashwinyes]

@poryfly I was able to reproduce your problem. And setting NUM_THREADS to a higher value solved the issue.

Here is how NUM_BUFFERS is defined.
(common.h) #define NUM_BUFFERS (MAX_CPU_NUMBER * 2)
(Makefile.system) -DMAX_CPU_NUMBER=$(NUM_THREADS)
(Makefile.system) - NUM_THREADS = $(NUM_CORES)

NUM_CORES is the number of cores on the machine where you are building OpenBLAS.

So if you are building from a docker with one cpu, NUM_CORES=1 and hence NUM_BUFFERS=2. This will not be enough for 5 threads. So if I hardcode NUM_THREADS=3, then NUM_BUFFERS=6 and it solves the problem.

[poryfly]

@ashwinyes i have a question.now when i use single thread--USE_THREAD=0,why i should set NUM_THREADS.i think single thread donot need set NUM_THREADS

[ashwinyes]

You are setting NUM_THREADS because eventually it is used to calculate NUM_BUFFERS, regardless of OpenBLAS being single threaded or multithreaded.

When you call an API like SGEMM, it needs its own buffers to work on. So if you have a multithreaded program and you call SGEMM from multiple threads at the same, each invocation of SGEMM will require its own buffer.

Hope you understood. Agree that usage of word THREADS is a little misleading.

[jakirkham]

Did that fix your issue, @poryfly? Am asking as I'm running into a similar problem and want to know if this might help before really digging into it.