I know Memory Error related questions have been asked before, for example here, here, here, here, or here. And the suggested solutions always are to switch to Python 3 and/or to Window 64bit, or in case of faulty code, to fix the code. However, I am already on Python 3 and Win 64. I also can see from windows task manager that I have at least 2 of my 8 available Gig of RAM unused when Python throw the Memory Error.
I have about 15 date-indexed, pandas dataframes each with 14000 rows and on average 5000 columns of float data, and about 40-50% NaN values, that I read in from hard drive. I can not simply drop NaNs because different columns have NaNs at different dates. Memory Error happens when I try to concatenate them with pd.concat(). So it's not a matter of some faulty code or while loop. If I leave some of the dataframes out of concatenation, Memory Error does not happen in concatenation, but then when I try to do a Scikit learn decision tree analysis on the concatenated data, it happens.
My question is, I have 8 Gig of RAM (and am increasing to 16 soon). How can I get windows to let Python have, say, up to 90% of my available RAM at any given time? Edit: screenshots added IPython interpreter screenshot (I don't have Python 2 even intalled): 
System information screenshot: 
from Memory Error in Python 3 and Windows 64
No comments:
Post a Comment