To get a better understanding about parallel, I am comparing a set of different pieces of code.
I am trying to compute the distances between 2 matrices.
Here is the dataset
np.random.seed(0)
X = np.random.rand(1000,2)
Y = np.random.rand(2000,2)
Following test took on a machine that has 6 physical cores.
scikit-learn euclidean_distances
code_piece_1 is using sklearn.metrics.pairwise.euclidean_distances to compute the distances.
def edis():
return euclidean_distances(X, Y, squared=True)
my own implementation
code_piece_2 is adapted from this post and the source code of scikit-learn.
def matrix_mul():
return np.sum(X*X,axis=1)[:, np.newaxis] - 2*np.dot(X,Y.T) + np.sum(Y*Y,axis=1)[np.newaxis, :]
this function is to return a wall time
def wall_time(func, verbose=False):
start = time.time()
res = func()
lead_time = time.time() - start
if verbose:
print('time elapsed {}s'.format(lead_time))
return res, lead_time
based on above, I ran and record the wall time of 2 approaches.
lead_time_e = []
lead_time_m = []
for i in range(9):
dis_e, lead_time = wall_time(edis)
lead_time_e.append(lead_time)
dis_m, lead_time = wall_time(matrix_mul)
lead_time_m.append(lead_time)
here is the time of sklearn.metrics.pairwise.euclidean_distances
lead_time_e
[0.03670144081115723,
0.02938079833984375,
0.03159976005554199,
0.026919126510620117,
0.029164552688598633,
0.03278350830078125,
0.02711176872253418,
0.029696941375732422,
0.033094167709350586]
and time of my own implementation
lead_time_m
[0.0424044132232666,
0.03006124496459961,
0.03377890586853027,
0.03877425193786621,
0.030216455459594727,
0.03377699851989746,
0.03643012046813965,
0.030277013778686523,
0.03365325927734375]
here is the average time
np.average(lead_time_e)
0.029820866054958768
np.average(lead_time_m)
0.033370574315389
Question
Why does my own version perform slower than the scikit-learn version.
What am I missing?
Note: Please discuss with this particular case or other specific and concrete cases. Please do NOT talk generally.
from Using almost the same source code of euclidean_distances, why does my own version perform slower than the scikit-learn version?
No comments:
Post a Comment