Exploring your old tweets with the Twitter API, part 1

I’ve recently tried using Twitter’s built-in tools to help take a browse through my old tweets and see if there’s anything interesting there I’d like to use elsewhere. I’m sharing my notes in case you’re at all interested in trying something similar (which you can, even if you’re not sure what “API” stands for, let alone how to use one).

But first…

Why would you do this?

Writing in early 2023: Twitter’s been in the news a lot recently, and various people have wondered if it might disappear altogether before too long. I’ve been meaning to try out this tweet-exploring idea for a while, so this made me think “if I’m ever going to do this, now seems a good time.”

I’d like to take a look through my old tweets and see if anything’s worth saving. Lots of them will just be throwaway thoughts or comments, but I know there’s a few ideas in there I’ve found useful to come back to several times. I can think of some threads I’ve quote-tweeted or shared elsewhere, treating it like my public “here’s my thoughts on x” reference.

If these notes do matter to me, I’d like to move them somewhere more in my control. Scott Hanselman blogged about this: “Your words are wasted”. That post’s from 2012, saying these problems were well known, and solved, back in 1999. Maybe it’s time for me to catch up…

Apart from these content thoughts, I’m interested in trying out APIs and other tech tools in general: If you work with tech teams, you’ll hear lots of terms about the software they’re building or using. I think it’s helpful for everyone involved to have some kind of idea of what these things mean – and for me, picking a little thing I’d like to use them for and actually trying them out really helps.

So, rather than manually browse through my thousands of tweets, I wondered:

• is it true that me quote-tweeting a post or thread – especially if I do it more than once – is a good pointer to content I want to come back to?
• can I use this tool to quickly fetch me the list of things I’ve quote-tweeted?

There’s probably lots of other ways to discover interesting tweets in a huge pile – but this feels like a nice small challenge to get started with.

What’s an API?

API stands for Application Programming Interface, and is a way for computer systems (websites or other things) to let other systems get information from them or send them instructions. Lots of guides explain this idea well, like this “Intro to APIs”.

APIs are useful because they let people use your system’s data and functionality in lots of creative ways, without you needing to change your system:

• you can publish a menu of options that your API offers
• you can put lots of safety rules around them (you can make it so the calling system needs to be authorized, you can limit how much usage callers get, all sorts)
• this can let other teams in your company make use of your team’s data and functionality, without you needing to change your system
• you can also make it public (with whatever safety rules you like), so your users can try useful creative ideas too.

Using Twitter’s public API will help explain a bit more about what this means.

Getting access

Instructions on getting started are all here: https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api

• You’ll need an existing Twitter login, and use that to request a developer account.
• There’s a free “Essential” access level, which lets you get plenty of usage; if you decide in future you want to fetch millions of tweets or build your own public app that uses this API, you could upgrage to a paid level, but Essential is perfect for the kind of exploring I want to do.
• The docs talk you through creating a “Project” and “app” – all the default settings are fine for these.
• This will generate some important keys (like passwords) – called API key, bearer token, and other names. Paste all of these somewhere safe (like a password manager!). You’ll use these when calling the API to prove you, the person with access, is giving this code permission to use your account.

And that’s it! One developer account, one project with one app, and a small collection of keys and secrets you can use.

Trying out the API

We’ll use the latest version (Twitter API v2). The Twitter API documentation has lots of help on what API you can make, and how to use them. The individual “endpoints” (the address you can call, with information, to ask the API to do things) all have helpful buttons to try them out in Twitter’s easy to use API Explorer and API Playground sites.

As an example, here’s the “get users by username” call: you can send this a Twitter username (neil_vass in my case), and it’ll send you back the user ID number Twitter uses internally. You can use that ID number to go and ask for that user’s past tweets or other activity.

But there’s a complication (as of January 2023): The API Explorer and API Playground have both disappeared recently, with no news on when they might be back! We can work around this.

One method you can still use is the Postman site – and that’s a great site to get familar with, as you can use it to try out all kinds of APIs, not just the Twitter one. It’s a general-purpose tool to let you call any web API. Getting started is quick:

• The homepage lets you create a free account
• Then you’ll get a welcome page, where you can choose “Create New”
• On the popup, choose Create New → HTTP Request
• You’ll get a form to fill in the request details. Let’s try that “get users by username” call.
• In the bar near the top, leave the type as GET and paste in the enpoint URL from the docs page (https://api.twitter.com/2/users/by/username/:username)
• That end part with the colon should be changed to the actual username we want to look up, so change it (for my username it’s https://api.twitter.com/2/users/by/username/neil_vass)
• Under that bar, choose “Auth” – we need to say what “Authorization” we’re using to prove whose developer account this API call is coming from
• Set the Auth type to “Bearer Token”
• In the “Token” field, paste the value of your Bearer Token (you saved this earlier, with the other keys and secrets you got when setting up your developer account)
• And … press Send.

If it all works, you’ll see the “Body” that Twitter has replied with:

• The user ID you can use to go look up more details
• The current display name of this user
• The username (which you sent over in the request).

You can use this ID in a “get recent tweets” call, by replacing the “users by username” URL with https://api.twitter.com/2/users/:id/tweets and changing that :id to be the user ID number you just got.

{
    "data": [
        {
            "edit_history_tweet_ids": [
                "1620539785281687552"
            ],
            "id": "1620539785281687552",
            "text": "First blog post of the year! I started this _ages_ ago but had serious procrastination troubles ... thanks @IanAmes, @twicezer0, @markdalgarno for patiently encouraging me to stop messing around and get it written.\n\nTwitter API may disappear soon, act now\n\nhttps://t.co/9LO14kSzVv"
        },
        {
            "edit_history_tweet_ids": [
                "1620528306285330432"
            ],
            "id": "1620528306285330432",
            "text": "RT @IanAmes: I'm enjoying being back doing a delivery role and I'm thinking a lot about how to slice up delivery depending on context. I th…"
        },
        {
            "edit_history_tweet_ids": [
                "1620343156670857219"
            ],
            "id": "1620343156670857219",
            "text": "@robynpgolding And look there's still space for people to sign up 😃\n\nhttps://t.co/VHez1lfgeM"
        },
... Code language: JSON / JSON with Comments (json)

There’s lots more you can try out – and in the Twitter API docs, they link to a Postman collection they’ve set up with all the API calls already pasted over for easy use.

And while visiting Postman: If this post has piqued your interest on APIs in general, not just the Twitter API, the Postman learning centre has nice interactive training from the absolute basics up to expert-level ideas. Have a look around!

I found Postman very helpful for testing out an API and understanding what individual calls do, but I can’t see an easy way to do the next step I want: Fetch lots of tweets from this user, check which are quote tweets, and give me just those. If you’re interested, read part 2 to try calling the API from some Python code and doing those extra steps.