Exploring your old tweets with the Twitter API, part 2

I was worried about this … in the few days since I posted part 1 of this guide, Twitter announced they’re ending free access to their API with a week’s notice. I may adapt this guide to be for another API in future – suggestions welcome!

I explained what I’m up to in part 1 of this guide:

• I’m interested in finding and re-using any useful notes I might have posted in Twitter, inspired by this “Your words are wasted” warning
• I’m interested in trying out tech skills in general, so this is a nice excuse

Last time, I talked about what an API is, how to get access to Twitter’s one, and how to try it out with Postman. I don’t think Postman lets me do the next step I want: fetching lots of tweets and filtering out just my quote tweets. So in this post we’ll move onto using Python to call the API instead.

Getting started with Python

You can call APIs from most programming languages. I’ve chosen to use Python, because I think it has a nice mix of being:

• beginner friendly (programs in it are more readable than lots of languages)
• professionally useful (it’s one of the most popular languages in the world)

To follow this guide and use Python to explore the Twitter API you’ll need to get Python set up on a computer and write a 10 line Python program.

If you’ve never tried programming at all before, or if you know some other languages but not Python, you might like to try out Codeacademy’s free interactive Learn Python 3 course. You can start trying out Python programming right away, no need to install any software or do any other setup. For this guide, there’s no need to complete the whole course: the first couple of lessons should give you enough learning to come back and carry on here.

To get Python set up on your computer, I recommend using Visual Studio Code – it’s a nice, free application that can be used for all kinds of programming language. Yes, I am encouraging you to start a lifetime of tinkering with technology concepts 🙂

Here’s a step-by-step guide to installing VS Code and setting it up for use with Python: https://code.visualstudio.com/docs/python/python-tutorial

When you’re happy you know how to run a few simple programs in Python, we can move on to using it with the Twitter API.

Using the Twitter API

To get what I wanted – a list of links to everything I’ve quote tweeted – here’s the Python code I put together:

import tweepy

client = tweepy.Client(bearer_token='<your token here>')
me = 299090531
pages = tweepy.Paginator(client.get_users_tweets, me, tweet_fields=["referenced_tweets,author_id"])

for tweet in pages.flatten():
  if tweet.author_id == me and tweet.referenced_tweets:
      if any(True for r in tweet.referenced_tweets if r.type == 'quoted'):
        print(f'https://twitter.com/neil_vass/status/{tweet.id}')Code language: Python (python)

If you followed the VS Code setup instructions, you’ll have a Python “hello world” file open – you can start by editing that.

We could use Python’s built-in features to send a message to the right web address, and parse the text that comes back – but helpfully, there’s an open-source library called tweepy that has a lot of the calls and responses wrapped in easier-to-use methods. Install this by typing into the terminal:

pip install tweepyCode language: Python (python)

And you should get a message saying it’s ready to use.

Then, start with just a few lines to check things are working:

import tweepy

client = tweepy.Client(bearer_token='<your token here>')
print(client.get_user(username='neil_vass'))Code language: JavaScript (javascript)

Replace the <your token here> part with the value of your bearer token. If you’re sharing this file anywhere, you need to be sure to not share that token! It’s fine to have it in a file on your own machine for a test like this, but in future you could use a library like python-dotenv to read secrets like from a separate, private file.

That get_user line is just to check we can call the API OK – if you run this 3-line program now (swapping my name for your own username if you like), you should see a response come back:

Response(data=<User id=299090531 
name=Neil Vass, also at @neil_vass@mastodon.me.uk 
username=neil_vass>, includes={}, errors=[], meta={})Code language: Python (python)

This means it’s working!

Next, delete that get_user line, and carry on adding the lines I showed in the complete example. Here’s what they do:

me = 299090531Code language: Python (python)

Just giving my user ID an easy-to-read name when we use it in the rest of this program.

Paginator(client.get_users_tweets, me, tweet_fields=["referenced_tweets,author_id"])Code language: Python (python)

When we get a lot of tweets, they come back in multiple chunks. Tweepy’s Paginator helpfully handles calling again and again until we have them all. We’re telling it to use the get_users_tweets API call, for the user ID we called me . By default, the list of tweets will just include the tweet’s ID number and its text. We can ask for extra tweet fields to be sent back too (called expansions in the API docs). Here we ask for:

• referenced_tweets : a list of any tweets that this one’s a retweet of, a quote tweet of, or a reply to
• author_id : user who wrote this tweet – if I retweet someone’s tweet, it’ll appear in my list of tweets but I won’t be the author.

Next, we’ll make the call and look through the results as they arrive:

for tweet in pages.flatten():Code language: Python (python)

The for line starts us looping through every tweet. flatten() helpfully lets us ignore the fact they come back in pages – we’ll just be fed one tweet after another without having to worry about finsihing and starting pages of them.

  if tweet.author_id == me and tweet.referenced_tweets:Code language: Python (python)

Note the indent at the start of this line is important – we’re doing this check for each tweet that comes into the for loop.

• If the author is me makes sure we ignore tweets written by others that I just retweeted – for this exercise, I’m looking for tweets I quoted and added my own thoughts.
• We’re only interested in tweets that have some referenced_tweets – we can check those and see if we’re referencing one by quote tweeting it.

      if any(True for r in tweet.referenced_tweets if r.type == 'quoted'):Code language: Python (python)

If it’s true that any of the tweets this one references is becasue we’ve quoted it…

        print(f'https://twitter.com/neil_vass/status/{tweet.id}')Code language: Python (python)

Then we’ve found something I’ve quote tweeted! Print it out its URL (web address) to the terminal. We can then ctrl-click (or cmd-click) each of those to go to that address in a browser and have a read.

Run this by calling it from the VS Code terminal – and if you like, you can send the results to a text file (make any filename you like) rather than having them print straight to the terminal. If your python file is called twit.py and you’d like to write to results.txt:

python twit.py > results.txtCode language: Bash (bash)

It’s done!

What did I find?

I got a list of just over 300 quote tweets – that’s quite a lot to look through, but much easier than browsing the thousands of things I’ve tweeted over the years.

And looking through them has been fun! There’s various things topics I’ve shared my thoughts on, that I’d like to write up as blog posts rather than leave scattered:

• Using OKRs (Objectives and Key Results) effectively, and whether you should even try
• Minimum Viable Estimation, and how you can get more value with less pain
• The first in a series of “Neil’s rules of business”, that I never came back to

There’s also quote tweets to lots of great ideas from other people that I’m going to re-read and see how I can put into practice, like:

• “Be tediously clear”, from Amy Hupe (I’ve quote tweeted this a lot)
• Building an organisational memory, from Cedric Chin
• Better post-talk Q&A sessions, from Eve Tuck
• Coaching advice (and not being too impressed with your own ideas), from Ron Jeffries and others

I’ll enjoy digging into these further.