Tuesday, 29 January 2019

Tweepy: Ignore previous tweets to improve optimization

Problem: Trying to pull tweets via tweepy using Cursor. I want to make sure I don't pull tweets I previously pulled.

Here is working code:

import tweepy
import pandas as pd
import numpy as np

ACCESS_TOKEN = ""
ACCESS_TOKEN_SECRET = ""
CONSUMER_KEY = ""
CONSUMER_SECRET = ""

# OAuth process, using the keys and tokens
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

# Creation of the actual interface, using authentication
api = tweepy.API(auth, wait_on_rate_limit=True)

csvFile = open(r'filename', 'a')

#Use csv writer
headers = ['UserName', 'Tweet', 'TweetId', 'tweet_date', 'source', 'fav_count', 'retweet_count', 'coordinates', 'geo']

# definitions for writing to CSV
csvWriter = csv.writer(csvFile, lineterminator='\n')
# write the headers once
csvWriter.writerow(headers)


handles = ['pycon', 'gvanrossum']
previousTweets = 
 ['222288832031240000',
 '222287080586362000',
 '222277240178741000',
 '221414283844653000',
 '221188011906445000',
 '205274818877210000']


for handle in handles:   
    for status in tweepy.Cursor(api.user_timeline, screen_name= handle, tweet_mode="extended").items():
        if status.id not in previousTweets:
            csvWriter.writerow([status.user.name.encode('utf-8'), status.full_text.encode('utf-8'), status.id, status.created_at, status.source, 
                    status.favorite_count, status.retweet_count, status.coordinates, status.geo])
print(handle)

This takes a long time and becomes unusable if you want to have a PreviousTweet list of over 75 tweets. Does anyone know a better way to filter out old tweets when using Tweepy and the Cursor function?



from Tweepy: Ignore previous tweets to improve optimization

No comments:

Post a Comment