Tuesday, 24 September 2019

Using almost the same source code of euclidean_distances, why does my own version perform slower than the scikit-learn version?

To get a better understanding about parallel, I am comparing a set of different pieces of code.

I am trying to compute the distances between 2 matrices.

Here is the dataset

np.random.seed(0)
X = np.random.rand(1000,2)
Y = np.random.rand(2000,2)

Following test took on a machine that has 6 physical cores.

scikit-learn euclidean_distances

code_piece_1 is using sklearn.metrics.pairwise.euclidean_distances to compute the distances.

def edis():
    return euclidean_distances(X, Y, squared=True)

my own implementation

code_piece_2 is adapted from this post and the source code of scikit-learn.

def matrix_mul():
    return np.sum(X*X,axis=1)[:, np.newaxis] - 2*np.dot(X,Y.T) + np.sum(Y*Y,axis=1)[np.newaxis, :]

this function is to return a wall time

def wall_time(func, verbose=False):
    start = time.time()
    res = func()
    lead_time = time.time() - start
    if verbose:
        print('time elapsed {}s'.format(lead_time))
    return res, lead_time

based on above, I ran and record the wall time of 2 approaches.

lead_time_e = []
lead_time_m = []
for i in range(9):
    dis_e, lead_time = wall_time(edis)
    lead_time_e.append(lead_time)
    dis_m, lead_time = wall_time(matrix_mul)
    lead_time_m.append(lead_time)

here is the time of sklearn.metrics.pairwise.euclidean_distances

lead_time_e
[0.03670144081115723,
 0.02938079833984375,
 0.03159976005554199,
 0.026919126510620117,
 0.029164552688598633,
 0.03278350830078125,
 0.02711176872253418,
 0.029696941375732422,
 0.033094167709350586]

and time of my own implementation

lead_time_m
[0.0424044132232666,
 0.03006124496459961,
 0.03377890586853027,
 0.03877425193786621,
 0.030216455459594727,
 0.03377699851989746,
 0.03643012046813965,
 0.030277013778686523,
 0.03365325927734375]

here is the average time

np.average(lead_time_e)
0.029820866054958768
np.average(lead_time_m)
0.033370574315389

Question

Why does my own version perform slower than the scikit-learn version.

What am I missing?

Note: Please discuss with this particular case or other specific and concrete cases. Please do NOT talk generally.



from Using almost the same source code of euclidean_distances, why does my own version perform slower than the scikit-learn version?

No comments:

Post a Comment