After some applying some procedure I am getting millions of numpy arrays (in the below case procedure converts e
to a numpy array):
for e in l:
procedure(e)
How can I save correctly each iteration into a single numpy file for later read it and load it?
So far I tried two options, with np.savez:
for i, e in enumerate(l):
np.savez(f'/Users/user/array.npz',i=e)
And with pandas:
(1) For saving into a single file:
for e in l:
arr = pd.DataFrame(procedure(i)).T
arr.to_csv('/Users/user/Downloads/arr.csv', mode='a', index=False, header=False)
(2) For reading:
arr = np.genfromtxt("/Users/user/Downloads/arr.csv", delimiter=',', dtype='float32', float_format='%.16f')
So far the solution that works is with pandas. However, I guess I am losing presicion in the numpy matrices. Because instead of having values like this (with the e):
-6.82821393e-01 -2.65419781e-01
I am getting values like this:
-0.6828214 , -0.26541978
However, the numpy matrices are not been saved correctly.
What is the most efficient and correct way to dump into a single file each numpy matrix after the for loop iteration?
from How to append successive calls to a single numpy file without losing precision?
No comments:
Post a Comment