Twitter scraper API: how to scrape tweets with Python

To scrape tweets with Python, send a POST request to a Twitter scraper API such as Scrape Everything and read the JSON it returns. POST /v1/x/posts gives you a public profile plus up to 60 recent tweets (pinned post, retweets and the account's own replies included) for 2 credits. POST /v1/x/post returns one tweet with views, quotes, bookmarks and media for 1 credit.

You don't need a logged-in X account or a guest token. If you're weighing this against the official X API on cost, read our post on X API alternatives and pricing. This one is mostly code, and every X endpoint is listed on the X scraper API overview.

What is a Twitter scraper API?#

A Twitter scraper API is a hosted service that fetches public X pages and returns tweets and profiles as JSON. You send a username or a tweet URL, and you get back fields like follower_count, view_count and media, with no browser, proxies or reverse-engineering of how x.com loads its data on your side.

The alternative is running Playwright or Selenium yourself against x.com. That works until a markup change or a login wall breaks it, and then you're reading minified JavaScript instead of shipping. For read-only collection of public data, an API call is the boring option, and boring is what you want in a cron job.

Which way should you scrape X in 2026?#

Most Python developers land here after trying one of the usual libraries.

Tweepy is a Python client for the official API, so it costs whatever your X API plan costs. twscrape calls the internal GraphQL endpoints the x.com web app uses, which is why it needs real accounts. snscrape stopped working for Twitter once X put most content behind a login in 2023.

General-purpose providers sell X data too. As of October 22, 2026, Bright Data lists $1.50 per 1,000 records pay as you go for its Web Scraper API, and Apify's free plan includes $5 of usage per month, with each Actor priced separately. Prices may change. See our Bright Data comparison and Apify comparison.

Which X endpoints are there and what do they cost?#

Scrape Everything is a REST API for public social media data, returned as clean JSON. For X it has three endpoints:

Credits come in one-time packs that never expire. At limit: 60 a timeline call costs 2 credits for up to 60 tweets, which works out to about 33 credits per 1,000 tweets collected: roughly $0.05 on Builder and $0.03 on Scale. That's the best case: daily runs return many tweets you already have.

What does it cost to track 100 accounts?

Say you pull 100 timelines daily and re-check views on 500 tweets a week with x.post. That's about 6,000 plus 2,000 credits, roughly 8,000 a month.

If three of your jobs read the same timeline on the same day, a one-day cache makes two of those three calls free (more on that below). The pricing calculator runs these numbers for your own volume.

How do I set up the API key?#

1. Sign in with Google or GitHub. New accounts get 100 free credits, no payment details needed.
2. Create a key on the API keys page. Keys look like ssk_live_ followed by 40 characters.
3. Put it in an environment variable called SCRAPEEVERYTHING_API_KEY.
4. Send it as Authorization: Bearer <key> to https://api.scrapeeverything.com.

Keys passed in the URL are rejected with 400 API_KEY_IN_URL, and the API refuses browser requests, so this runs server-side. The authentication docs cover IP allowlists and key rotation.

export SCRAPEEVERYTHING_API_KEY=ssk_live_...

How do I scrape an X profile?#

The profile endpoint costs 1 credit. Start with cURL so you can see the raw shape:

curl -X POST https://api.scrapeeverything.com/v1/x/profile \
  -H "Authorization: Bearer $SCRAPEEVERYTHING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"username": "example_user"}'

Swap example_user for any public handle. @example_user and https://x.com/example_user work too. The response, trimmed (values are placeholders):

{
  "success": true,
  "data": {
    "user_id": "1234567890",
    "username": "example_user",
    "full_name": "Example User",
    "biography": "Shipping things. Occasionally tweeting about them.",
    "location": "Berlin",
    "external_url": "https://example.org",
    "is_private": false,
    "is_verified": true,
    "is_blue_verified": true,
    "follower_count": 18240,
    "following_count": 312,
    "tweet_count": 4120,
    "listed_count": 96,
    "created_at": "Wed Jun 02 20:12:29 +0000 2010",
    "profile_image_url": "https://pbs.twimg.com/profile_images/.../photo.jpg"
  },
  "meta": {
    "request_id": "req_...",
    "scraper": "x.profile",
    "cached": false,
    "credits_charged": 1,
    "credits_remaining": 99,
    "duration_ms": 1650
  }
}

Every response uses the same envelope: success, data, and a meta object with what the call cost. created_at is X's legacy date format, not ISO 8601; we parse it below. Field details live on the X profile scraper page, and if follower counts across several platforms are what you're after, see our follower count API guide.

How do I scrape tweets with Python from a timeline?#

This is the endpoint most people want: 2 credits for up to 60 tweets. Here's a small Python client using requests:

import os
import requests

API = "https://api.scrapeeverything.com"
HEADERS = {
    "Authorization": f"Bearer {os.environ['SCRAPEEVERYTHING_API_KEY']}",
    "Content-Type": "application/json",
}

def check(r: requests.Response) -> dict:
    try:
        payload = r.json()
    except ValueError:
        # A proxy or load balancer in between can answer with HTML.
        raise RuntimeError(f"{r.status_code} non-JSON response") from None
    if not payload.get("success"):
        raise RuntimeError(f"{r.status_code} {payload.get('code')}: {payload.get('message')}")
    return payload["data"]

def scrape(path: str, body: dict) -> dict:
    # timeout=90 leaves headroom for proxy retries on the API side.
    r = requests.post(API + path, json=body, headers=HEADERS, timeout=90)
    return check(r)

timeline = scrape("/v1/x/posts", {"username": "example_user", "limit": 60})

print(timeline["username"], timeline["follower_count"])
for t in timeline["posts"]:
    kind = "RT" if t["is_retweet"] else "reply" if t["is_reply"] else "post"
    pin = " [pinned]" if t["is_pinned"] else ""
    print(f"{t['id']} {kind}{pin} views={t['view_count']} likes={t['like_count']}")

Each item in posts has these fields:

A trimmed item:

{
  "id": "1850000000000000001",
  "link": "https://x.com/example_user/status/1850000000000000001",
  "text": "v2.3 is out. Changelog in the thread.",
  "lang": "en",
  "created_at": "Mon Oct 19 14:02:11 +0000 2026",
  "author_username": "example_user",
  "is_pinned": true,
  "is_retweet": false,
  "is_reply": false,
  "is_quote": false,
  "quoted": null,
  "reply_count": 14,
  "retweet_count": 31,
  "quote_count": 5,
  "like_count": 402,
  "bookmark_count": 27,
  "view_count": 38110,
  "media": []
}

Before you filter or store anything, check these:

• The timeline mixes the pinned post, retweets and the account's own replies, in timeline order. Want original posts only? Filter on is_retweet and is_reply yourself.
• For a retweet, the tweet fields describe the original tweet and retweeted_from holds the original author. So id is the original tweet's ID. There is no separate retweet ID.
• Protected accounts return an empty posts array. Look at is_private before you conclude the account went quiet.

How do I get tweet views for a single tweet?#

When you already have a URL (from a spreadsheet or a Slack alert) use x.post: 1 credit for view_count plus quotes, bookmarks and media. Here it is in Node 18+, no dependencies. Save it as views.mjs, because top-level await only works in ES modules. In a CommonJS .js file you'll get SyntaxError: await is only valid in async functions and the top level bodies of modules.

const res = await fetch("https://api.scrapeeverything.com/v1/x/post", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.SCRAPEEVERYTHING_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://x.com/example_user/status/1850000000000000001" }),
  signal: AbortSignal.timeout(90_000),
});

const { success, data, code } = await res.json();
if (!success) throw new Error(`${res.status} ${code}`);

console.log(data.view_count, data.quote_count, data.bookmark_count);
for (const m of data.media) {
  console.log(m.type, m.video_url ?? m.image_url, m.duration_ms);
}

media[].type is photo, video or animated_gif. For videos, video_url is already the highest-bitrate MP4, so you don't have to parse variant lists.

Send tweet IDs as strings. If you pass id instead of url, send "1850000000000000001", not 1850000000000000001. JSON itself has no precision limit, but JavaScript parses numbers as IEEE 754 doubles, which are exact only up to Number.MAX_SAFE_INTEGER (2^53 - 1). A numeric tweet ID above that gets rounded during parsing, and you get a different tweet back, or a 404, with no error telling you why. Store IDs the same way: BIGINT in Postgres is fine, a JavaScript Number is not.

How do I store tweets and dedupe them across daily runs?#

Run the timeline job daily and most of what comes back is stuff you already have. The pinned post shows up every single time.

Our rule: the tweet ID is the key, and the counts get overwritten.

Parse the date first (the snippets below use Python 3.10+ type hints):

from datetime import datetime

def parse_x_date(s: str | None):
    # "Wed Jun 02 20:12:29 +0000 2010". %a and %b assume a C/English locale.
    return datetime.strptime(s, "%a %b %d %H:%M:%S %z %Y") if s else None

Writing to CSV without duplicates

For a quick export, keep a set of seen IDs and append only new rows. Python's csv module is all you need. Booleans land in the file as True and False, which matters if you load it into a database later.

import csv
from pathlib import Path

FIELDS = ["id", "link", "created_at", "author_username", "is_retweet",
          "is_reply", "view_count", "like_count", "retweet_count", "text"]

def append_new(path: str, posts: list[dict]) -> int:
    p = Path(path)
    seen = set()
    write_header = not p.exists() or p.stat().st_size == 0
    if not write_header:
        with p.open(newline="", encoding="utf-8") as f:
            seen = {row["id"] for row in csv.DictReader(f)}
    new = [t for t in posts if t["id"] not in seen]
    with p.open("a", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=FIELDS, extrasaction="ignore")
        if write_header:
            w.writeheader()
        w.writerows(new)
    return len(new)

CSV is fine until you want view counts that change over time. Then you want a table.

Upserting into Postgres

If two accounts you track retweet the same tweet, both rows carry the original tweet's id, so a key on id alone lets the second account overwrite the first. That's why the table keys on (id, account). Decide this before you have a month of data.

CREATE TABLE IF NOT EXISTS tweets (
  id              BIGINT NOT NULL,
  account         TEXT NOT NULL,
  author_username TEXT,
  created_at      TIMESTAMPTZ,
  text            TEXT,
  is_retweet      BOOLEAN,
  is_reply        BOOLEAN,
  view_count      BIGINT,
  like_count      INTEGER,
  retweet_count   INTEGER,
  quote_count     INTEGER,
  first_seen      TIMESTAMPTZ DEFAULT now(),
  last_seen       TIMESTAMPTZ DEFAULT now(),
  PRIMARY KEY (id, account)
);

import psycopg  # psycopg 3

UPSERT = """
INSERT INTO tweets (id, account, author_username, created_at, text, is_retweet,
                    is_reply, view_count, like_count, retweet_count, quote_count)
VALUES (%(id)s, %(account)s, %(author_username)s, %(created_at)s, %(text)s,
        %(is_retweet)s, %(is_reply)s, %(view_count)s, %(like_count)s,
        %(retweet_count)s, %(quote_count)s)
ON CONFLICT (id, account) DO UPDATE SET
  view_count = EXCLUDED.view_count,
  like_count = EXCLUDED.like_count,
  retweet_count = EXCLUDED.retweet_count,
  quote_count = EXCLUDED.quote_count,
  last_seen = now();
"""

def save(conn, account: str, posts: list[dict]) -> None:
    rows = [{**t, "id": int(t["id"]), "account": account,
             "created_at": parse_x_date(t["created_at"])} for t in posts]
    with conn.cursor() as cur:
        # One batched call instead of a round trip per row.
        cur.executemany(UPSERT, rows)
    conn.commit()

ON CONFLICT ... DO UPDATE is standard Postgres (see the INSERT reference). For a view curve instead of the latest value, add a tweet_snapshots table keyed on (id, fetched_at) and insert every run.

If several jobs might ask for the same timeline on the same day, add "cache_days": 1 to the request body. A stored result from your own account within that window comes back for 0 credits with meta.cached: true. A different limit counts as a different request. Details are in the caching docs.

Can I scrape tweets from the terminal instead?#

Yes. The Scrape Everything CLI wraps every endpoint as a command and needs only Node.js 18. With the env var set:

npx scrapeeverything x posts --username example_user --limit 20 --output jsonl \
  --fields id,created_at,view_count,like_count,text > tweets.jsonl

twitter is an alias for x, and a global install adds the short binary se. Piped output is raw JSON by default, so se x post --url https://x.com/example_user/status/1850000000000000001 | jq .view_count does what you'd expect. Exit code 4 means out of credits and 5 means rate limited, so shell scripts can branch on them.

What happens when a request fails, and do I pay for it?#

Errors come back in the same envelope: success: false, a stable code, a human message and a request_id. The errors reference has the full list. The ones you'll see from an X scraper:

Failed requests on our side are refunded automatically. A 404 is a real answer about the target, so it's billed, and retrying bills you again. Cache those.

Here's a retry wrapper that reuses check() from the client above, so error envelopes raise instead of ending up in your database. RateLimit-Reset is the number of seconds until the window resets, not a timestamp.

import random
import time

RETRYABLE = {429, 500, 502, 503, 504}

def scrape_with_retry(path: str, body: dict, attempts: int = 4) -> dict:
    for i in range(attempts):
        r = requests.post(API + path, json=body, headers=HEADERS, timeout=90)
        if r.status_code in RETRYABLE and i < attempts - 1:
            reset = r.headers.get("RateLimit-Reset", "")
            if r.status_code == 429 and reset.isdigit():
                time.sleep(int(reset))
            else:
                time.sleep(2 ** i + random.random())
            continue
        return check(r)

What can't the X scraper API do?#

• No keyword or hashtag search. There's no X search endpoint. The practical workaround is to track the accounts that post about your topic.
• No follower or following lists. x.profile gives you the counts, not the lists.
• No reply threads. x.post tells you whether a tweet is a reply and to whom (in_reply_to), but it doesn't return the conversation.
• No full history. x.posts returns at most 60 recent tweets and has no pagination cursor. For history, start collecting now and let the upsert above build the archive.
• Request bodies over 100 KB are rejected with 413.

Rate limits are per account, per minute, with a separate cap on parallel requests. The defaults at the time of writing:

Check the rate limits page for current values and the RateLimit-* headers.

Public data only: no logins, no protected accounts, no DMs. You're still responsible for what you do with it. Public tweets can contain personal data, so collect only what you need and read X's Terms of Service. Our post on whether scraping public social media data is legal lists the questions to ask. Scrape Everything isn't affiliated with X Corp. To post, read DMs or act for a user, use the X API documentation.

FAQ#

Can I scrape tweets with Python without the official X API?

Yes. A Twitter scraper API returns public tweets as JSON over HTTPS, so any HTTP client works. With Scrape Everything, POST /v1/x/posts returns up to 60 recent tweets for 2 credits and POST /v1/x/post returns one tweet for 1 credit. You don't need an X developer account.

How do I get the view count of a tweet through an API?

Send the tweet URL to POST /v1/x/post. The response includes view_count next to like_count, retweet_count, quote_count, reply_count and bookmark_count. Timeline items from x.posts carry the same counts.

Does snscrape still work for Twitter?

Not for X content behind the login wall, which since 2023 is most of it. The usual replacements are twscrape (needs X accounts you log in with), Tweepy on the paid official API, or a hosted scraper API that returns public data as JSON.

Can I search tweets by keyword or hashtag?

Not with Scrape Everything, which has no X search endpoint. The workaround is to track the timelines of accounts that post about your topic.

Can I scrape a private or protected X account?

No. Scrape Everything reads public data only. A protected account returns its public profile fields and an empty posts array, and a protected tweet requested through x.post returns 404 NOT_FOUND.

How do I avoid storing the same tweet twice?

Key on the tweet id plus the tracked account and upsert. In Postgres, INSERT ... ON CONFLICT (id, account) DO UPDATE refreshes the counts each run. Keep IDs as strings or BIGINT, never JavaScript numbers.

Try it on your own accounts#

Get 100 free credits and run the timeline script above. That's 50 timeline pulls or 100 single-tweet lookups. If you'd rather query X from Claude or Cursor than from a script, the MCP server exposes the same three endpoints as tools.

Related: the n8n node for no-code workflows, the OpenAPI spec at https://scrapeeverything.com/openapi.json, and the X post scraper page.