Sunday, 10 January 2021

Iterating over storage bucket contents in GCP whilst data is streaming into it

I want to iterate over the contents of a storage bucket at a specific point in time. I have the following code to do this:

from google.cloud import storage
client = storage.Client()
bucket = client.get_bucket("my_bucket")
blobs = bucket.list_blobs()
for blob in blobs:
    if "old_name" in blob.name:
        # process file

However, items are streaming into my bucket when I do this. The if statement will in theory mean that I will not process anything new that is streaming in, however I am nervous that the iterator will not contain all the blobs as the new files streaming in will mess it up somehow. Do you know if the above code will work or could there be problems and I miss out some of the files.



from Iterating over storage bucket contents in GCP whilst data is streaming into it

No comments:

Post a Comment