Sunday, 18 April 2021

Why are my tweet search requests only retrieving 200 tweets every time?

I have a script which returns tweets based on a keyword query and then appends this to a CSV. I can't see why in my script, this is only returning 200 tweets every time I run it. It is not the count parameter, because as far as I am aware this returns the number of tweets to return per page, up to a maximum of 100.

Can anyone see what is occurring?

def twitter_search(twitter_api, q, max_results = 3000, **kw):
    search_results = twitter_api.search.tweets(q=q, count = 100, **kw, lang = 'en', tweet_mode='extended', )
    
    statuses = search_results['statuses']
    
    #Iterate through batches of results until we get the number we want
    #Enforce a reasonable limit
    
    max_results = min(5000, max_results)
    
    for _ in range(100):
        try:
            next_results = search_results['search_metadata']['next_results']
        except KeyError as e: #no more results when next_results doesn't exist
            break
    
    #create a dictionary from next_results
        kwargs = dict([kv.split('=') for kv in next_results[1:].split("&")])
    
        search_results = twitter_api.search.tweets(**kwargs)
        statuses += search_results['statuses']
    
        if len(statuses) > max_results:
            break
    
    return statuses

I think it is to do with the cursor iterating over the next batch of results but I do not know why this is happening...



from Why are my tweet search requests only retrieving 200 tweets every time?

No comments:

Post a Comment