Tuesday, 19 November 2019

Searching through large data files by python

I have large data files of several Gb in size that I have to run the python script against. Possibly search for text and generate lists/sets. How can I handle large csv/txt files, read through them, know search text positions, without having to load them in the memory?

data = []

with open('largelogfile.csv', 'r', newline='') as datafile:
        reader = csv.reader(datafile)
        header = next(reader)
        for row in reader:
                if "keyword" in row:
    #something
                data.append(row)


from Searching through large data files by python

No comments:

Post a Comment