I have large data files of several Gb in size that I have to run the python script against. Possibly search for text and generate lists/sets. How can I handle large csv/txt files, read through them, know search text positions, without having to load them in the memory?
data = []
with open('largelogfile.csv', 'r', newline='') as datafile:
reader = csv.reader(datafile)
header = next(reader)
for row in reader:
if "keyword" in row:
#something
data.append(row)
from Searching through large data files by python
No comments:
Post a Comment