Saturday, 4 January 2020

Weird bug in Pandas and Numpy regarding multithreading

Most of the Numpy's function will enable multithreading by default.

for example, I work on a 8-cores intel cpu workstation, if I run a script

import numpy as np    
x=np.random.random(1000000)
for i in range(100000):
    np.sqrt(x)

the linux top will show 800% cpu usage during running like enter image description here Which means numpy automatically detects that my workstation has 8 cores, and np.sqrt automatically use all 8 cores to accelerate computation.

However, I found a weird bug. If I run a script

import numpy as np
import pandas as pd
df=pd.DataFrame(np.random.random((10,10)))
df+df
x=np.random.random(1000000)
for i in range(100000):
    np.sqrt(x)

the cpu usage is 100%!!. enter image description here It means that if you plus two pandas DataFrame before running any numpy function, the auto multithreading feature of numpy is gone without any warning! This is absolutely not reasonable, why would Pandas dataFrame calculation affect Numpy threading setting? Is it a bug? How to work around this?


PS:

I dig further using Linux perf tool.

running first script shows

enter image description here

While running second script shows

enter image description here

So both script involves libmkl_vml_avx2.so, while the first script involves additional libiomp5.so which seems to be related to openMP.

And since vml means intel vector math library, so according to vml doc I guess at least below functions are all automatically multithreaded

enter image description here



from Weird bug in Pandas and Numpy regarding multithreading

No comments:

Post a Comment