I have a script which returns tweets based on a keyword query and then appends this to a CSV. I can't see why in my script, this is only returning 200 tweets every time I run it. It is not the count parameter, because as far as I am aware this returns the number of tweets to return per page, up to a maximum of 100.
Can anyone see what is occurring?
def twitter_search(twitter_api, q, max_results = 3000, **kw):
search_results = twitter_api.search.tweets(q=q, count = 100, **kw, lang = 'en', tweet_mode='extended', )
statuses = search_results['statuses']
#Iterate through batches of results until we get the number we want
#Enforce a reasonable limit
max_results = min(5000, max_results)
for _ in range(100):
try:
next_results = search_results['search_metadata']['next_results']
except KeyError as e: #no more results when next_results doesn't exist
break
#create a dictionary from next_results
kwargs = dict([kv.split('=') for kv in next_results[1:].split("&")])
search_results = twitter_api.search.tweets(**kwargs)
statuses += search_results['statuses']
if len(statuses) > max_results:
break
return statuses
I think it is to do with the cursor iterating over the next batch of results but I do not know why this is happening...
from Why are my tweet search requests only retrieving 200 tweets every time?
No comments:
Post a Comment