I have 600 Dataframes saved and stored as .pickle and I'd like to merge (or rather append) them into one DataFrame. The total size of them is 10GB.
When I read each of them and append them into one big DataFrame and then save the full version to dist the entire process takes 2 hours on 16GB machine.
I think it takes a lot of time because each time I append a new DataFrame system allocates new memory space for the entire new DataFrame?
How can I do this faster?
from Merging many pickle Dataframes into one
No comments:
Post a Comment