Social media scraper CLI: scrape from the terminal with npx

TL;DR: The Scrape Everything social media scraper CLI runs with no install: set SCRAPEEVERYTHING_API_KEY and run npx scrapeeverything instagram profile --username nasa. It returns public Instagram, TikTok, X, YouTube and other platform data as JSON you can pipe into jq, and a global install adds the short alias se. Most calls cost 1 credit, and new accounts get 100 free credits.

Last updated: 21 October 2026

The scrapeeverything CLI is a command-line client for the Scrape Everything API that turns every endpoint into a terminal command and prints readable output, or raw JSON when piped. It covers nine platforms: Instagram, TikTok, X, YouTube, Facebook, Telegram, Spotify, Pinterest and Twitch. Public data only. You never log in to the platform.

I reach for it before I write any code. When someone asks "can we get follower counts for these 40 creators?", a one-line loop answers that faster than a new project does.

Which social media scraper CLI should I use?#

It depends on what you want back. For the video file itself, use a downloader such as yt-dlp. For sites far beyond social media, Apify fits better. For structured profile, post, comment and transcript data from the nine platforms above, with one key and one output format, use this CLI.

Hosted options compared, from each vendor's pricing page on 22 October 2026 (prices may change):

Product names are trademarks of their owners; we are not affiliated with them. Figures from each vendor's public pricing page, verified October 22, 2026. Spot an error? Email [email protected] and we correct it.

To be fair to them: if you need LinkedIn, Reddit or Threads today, ScrapeCreators covers platforms we don't. The full breakdowns are on our ScrapeCreators comparison and Apify comparison pages.

How do I install the social media scraper CLI?#

You need Node.js 18 or later. The CLI has no runtime dependencies and uses Node's built-in fetch.

1. Create an account at scrapeeverything.com/login with Google or GitHub. You get 100 free credits, no payment details needed.
2. Create an API key on the dashboard's API keys page. Keys look like ssk_live_....
3. Run it without installing:

export SCRAPEEVERYTHING_API_KEY=ssk_live_...
npx scrapeeverything instagram profile --username nasa

That call used 1 of your 100 free credits. Create a key and try a handle you know.

1. Install it globally if you'll use it often (and for cron jobs you should):

npm install -g scrapeeverything
se login        # prompts for the key with hidden input and checks it

The binary is scrapeeverything with the short alias se. The alias only comes with the global install, so with npx you type the full name. I'll use se from here on.

se login checks the key against GET /v1/account and saves it in your OS config directory (~/.config/scrapeeverything/ on Linux, ~/Library/Application Support/scrapeeverything/ on macOS, %APPDATA%\scrapeeverything\ on Windows), or wherever SCRAPEEVERYTHING_CONFIG_DIR points. The file is locked to your user, but the key sits there in plain text. On shared machines or CI, pass the environment variable from a secret manager instead. It takes precedence over the stored key.

How do I find which endpoints exist?#

Two commands cover it, and you'll rarely need the docs tab:

se list                          # every scraper with credit cost and inputs
se list tiktok                   # one platform
se describe instagram profile    # options, response fields, example

se list prints a table with COMMAND, CREDITS, INPUT and DESCRIPTION columns. se describe shows every option and response field for one endpoint.

The commands come from the live API catalog (GET /v1/scrapers), so new scrapers show up without upgrading the CLI. Offline, it falls back to a bundled snapshot and says so with an "offline catalog" note.

Flags use the catalog's parameter names in kebab-case, and platform aliases work too: ig, tt, yt, fb, and twitter for X.

How do I scrape Instagram, TikTok, X and YouTube from the terminal?#

These commands each cost 1 credit per successful call:

Some feed and search endpoints cost more: se x posts and se instagram photos are 2 credits, se instagram reels_search is 3. Every endpoint and its parameters are in the scraper catalog.

Scrape an Instagram profile

se instagram profile --username nasa

In a terminal you get colored output, then a dim line such as 1 credit charged, 99 remaining. A bare value fills the main input, so se ig profile nasa works too. Fields a private account hides come back as null. For a bigger build, see the Instagram profile scraping guide.

Scrape TikTok from the terminal

se tiktok profile --username khaby.lame

Piped, the same command prints JSON. Below is the shape, trimmed, with placeholder values in place of a real creator's numbers:

{
  "user_id": "6800000000000000000",
  "username": "example_user",
  "full_name": "Example User",
  "biography": "Short bio here",
  "is_private": false,
  "is_verified": false,
  "follower_count": 48213,
  "following_count": 312,
  "like_count": 1204551,
  "video_count": 286,
  "region": "US",
  "created_at": "2019-04-02T10:15:00Z"
}

How do I get X (Twitter) data from the shell?

Same pattern:

se x profile --username tesla --fields username,follower_count,tweet_count

For Python against X, see our tweet scraping guide.

Get a YouTube transcript

se youtube transcript --url "https://www.youtube.com/watch?v=jNQXAC9IVRw"

Without --lang, the API prefers a hand-written track in the video's spoken language, then YouTube's default track, then automatic captions. No captions, or none in your language, gets you a 404 with code NO_TRANSCRIPT, and the message lists the language codes that do exist. Machine-translated tracks aren't offered. Background is in our YouTube transcript API post.

How do I select fields and pipe the JSON to jq?#

The CLI picks its output mode from where stdout goes. A terminal gets pretty. A pipe or file gets json, the raw data object from the API response. --output forces either.

Trim at the source with --fields, then hand the rest to jq. These are the jq one-liners I actually use:

# Titles of the latest 5 uploads
se youtube videos --username MrBeast --limit 5 | jq '.videos[].title'

# Transcript as plain text, for a notes file or an LLM prompt
se yt transcript --url "https://www.youtube.com/watch?v=jNQXAC9IVRw" | jq -r '.text' > transcript.txt

# Timestamped lines (start seconds, tab, text)
se yt transcript --id jNQXAC9IVRw | jq -r '.segments[] | "\(.start)\t\(.text)"'

# Which caption languages exist?
se yt transcript --id jNQXAC9IVRw | jq -r '.available_languages[] | "\(.code)  \(.name)"'

On PowerShell you don't need jq. ConvertFrom-Json does the job:

se tiktok profile khaby.lame | ConvertFrom-Json | Select-Object username, follower_count

One caveat: YouTube view counts are parsed from rounded text ("74M views"), so view_count is approximate and published_text is relative ("12 days ago"). Good enough to rank a channel's uploads. Not good enough for a finance report.

How do I loop over handles, export CSV and run it on cron?#

Loop over a list of handles

Put one handle per line in handles.txt. This script writes one JSON line per profile, stops when credits run out, and retries rate-limited or unavailable handles once at the end:

#!/usr/bin/env bash
set -u
out="tiktok-$(date +%F).jsonl"
retry=()

fetch() {
  se tiktok profile --username "$1" --output jsonl --quiet \
    --fields username,follower_count,like_count,video_count >> "$out"
}

while read -r handle; do
  [ -z "$handle" ] && continue
  fetch "$handle"; code=$?
  case $code in
    0) ;;
    4) echo "out of credits, stopping at $handle" >&2; exit 4 ;;
    5|6) echo "will retry $handle (exit $code)" >&2; retry+=("$handle") ;;
    *) echo "skipped $handle (exit $code)" >&2 ;;
  esac
done < handles.txt

if [ ${#retry[@]} -gt 0 ]; then
  sleep 60
  for handle in "${retry[@]}"; do
    fetch "$handle" || echo "failed twice: $handle" >&2
  done
fi

The CLI never retries by itself, and I like it that way. Retry policy belongs in the script, where you can read it.

Errors go to stderr with the API code and request_id, so keep stderr in your logs. Failed calls are refunded automatically. The full code list is on the errors page.

How often will you see exit 5? Rarely, with a sequential loop. Rate limits apply per account: 60 requests per minute and 5 concurrent for free accounts, 300 per minute and 20 concurrent once your purchases total $39 or more. Details are in the rate limits docs.

What does tracking 40 creators cost?

Back to the 40 creators, using the 1-credit price of tiktok profile and the published packs:

Tracking 40 creators daily costs 1,200 credits a month, about $1.87 on the Builder pack. If follower counts are the whole job, read the follower count API guide.

Export to CSV

jq's @csv handles the quoting:

f="tiktok-$(date +%F).jsonl"
{
  echo "username,followers,likes,videos"
  jq -r '[.username, .follower_count, .like_count, .video_count] | @csv' "$f"
} > tiktok.csv

For a YouTube channel, --output jsonl emits one object per video from the videos list. Add set -o pipefail so a CLI failure isn't hidden behind a successful jq:

set -o pipefail
se youtube videos --username MrBeast --limit 30 --output jsonl \
  --fields title,view_count,duration_seconds,link \
  | jq -r '[.title, .view_count, .duration_seconds, .link] | @csv' > uploads.csv

Schedule it with cron

Cron runs with a minimal PATH and doesn't read your shell profile, so the export in .bashrc isn't there. Install globally, use absolute paths, and run se login once as the same user:

# 07:00 every day
0 7 * * * /home/me/bin/tiktok-snapshot.sh >> /home/me/logs/tiktok.log 2>&1

I'd keep npx out of cron. It resolves the package on every run, which is slow and one more thing to fail at 7 a.m. The crontab(5) man page covers the environment rules if your distro differs.

Run it in GitHub Actions

On CI, keep the key in a repository secret and let the environment variable do the work:

name: tiktok-snapshot
on:
  schedule:
    - cron: "0 7 * * *"
jobs:
  snapshot:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm install -g scrapeeverything
      - run: bash tiktok-snapshot.sh
        env:
          SCRAPEEVERYTHING_API_KEY: ${{ secrets.SCRAPEEVERYTHING_API_KEY }}

How do I check credits with se account?#

se account

It returns your email, plan, credit balance and rate limits. The underlying GET /v1/account call costs nothing and doesn't count against your rate limit, so put it at the top of scripts. jq -e fails loudly if the field is missing instead of letting the check pass silently:

left=$(se account | jq -e '.credits') || exit 1
[ "$left" -lt 50 ] && { echo "only $left credits left" >&2; exit 1; }

Credits come in one-time packs that never expire, from Builder at $39 for 25,000 credits up to Scale at $449 for 500,000. The pricing page lists all of them and the credits docs show the cost per endpoint.

One limitation I hit: the CLI validates options against the catalog, and cache_days isn't a catalog parameter. --param cache_days=3 fails with Unknown option --cache-days for tiktok profile. For free cache hits (up to 14 days), call the HTTP API and put cache_days in the body. See the caching docs.

Can I use the same thing over HTTP?#

Yes. The CLI is a thin client over the REST API. Here's the TikTok call with cURL, cache included:

curl -X POST https://api.scrapeeverything.com/v1/tiktok/profile \
  -H "Authorization: Bearer $SCRAPEEVERYTHING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"username": "khaby.lame", "cache_days": 3}'

And a YouTube transcript in Python with only the standard library:

import json, os, urllib.request

req = urllib.request.Request(
    "https://api.scrapeeverything.com/v1/youtube/transcript",
    data=json.dumps({"url": "https://www.youtube.com/watch?v=jNQXAC9IVRw"}).encode(),
    headers={
        "Authorization": f"Bearer {os.environ['SCRAPEEVERYTHING_API_KEY']}",
        "Content-Type": "application/json",
    },
    method="POST",
)
with urllib.request.urlopen(req, timeout=90) as resp:
    body = json.load(resp)

print(body["data"]["language"], body["data"]["segment_count"])
print(body["meta"]["credits_charged"], "credit(s) charged")

Over HTTP you get the full envelope: success, data and meta. Trimmed, with placeholder values:

{
  "success": true,
  "data": {
    "video_id": "jNQXAC9IVRw",
    "language": "en",
    "language_name": "English",
    "is_auto_generated": false,
    "available_languages": [{ "code": "en", "name": "English", "is_auto_generated": false }],
    "segment_count": 6,
    "text": "Example transcript text ...",
    "segments": [{ "start": 1.2, "duration": 2.16, "text": "Example segment" }]
  },
  "meta": {
    "request_id": "req_example123",
    "scraper": "youtube.transcript",
    "cached": false,
    "credits_charged": 1,
    "credits_remaining": 99
  }
}

The CLI prints only data, which is why jq '.text' works straight on its output. For production code, the API reference and the OpenAPI spec at https://scrapeeverything.com/openapi.json have every endpoint.

Can Claude Code or Codex use this CLI instead of an MCP server?#

Yes. Any coding agent that can run shell commands (Claude Code, Codex CLI, Gemini CLI, Cursor's terminal) can use it with nothing but the API key, which helps in sandboxes where you can't register an MCP server.

• se describe <platform> <action> gives the agent exact options and fields, so it doesn't guess flags.
• Exit codes say what went wrong. Exit 4 means stop and tell the human. Exit 2 means the agent built the command wrong.
• --fields keeps output, and the agent's context, small.

Where MCP is available, the MCP server (@scrapeeverything/mcp, listed in the official MCP Registry) exposes one tool per endpoint, so the assistant calls them directly. The Model Context Protocol docs explain how servers plug into clients like Claude, Cursor and VS Code.

What are the legal limits of scraping public data?#

The CLI only fetches what a logged-out visitor can see. That doesn't make every use fine. Platforms have their own terms of service, and if you store profiles of people in the EU, the General Data Protection Regulation (GDPR) applies to that personal data whether it was public or not. Keep only the fields you need, delete what you stop using, and don't use the data to harass or track individuals. Our post on whether scraping public social media data is legal goes through the details. It isn't legal advice, and neither is this.

What should I try next?#

Run se list, pick one endpoint you'd otherwise write a scraper for, and point it at a handle you know. If the data fits, move the call into code or into an n8n workflow. The CLI integration page keeps the current flag reference.

Get 100 free credits and run your first command.

FAQ#

Is there a command line Instagram scraper that doesn't need my Instagram login?

Yes. se instagram profile --username <handle> fetches public profile data through the Scrape Everything API using an API key, not an Instagram login. It returns what a logged-out visitor sees, and fields a private account hides come back as null.

Can I get a YouTube transcript from the CLI?

Yes. Run se youtube transcript --url "<video URL>", optionally with --lang en or another caption code. It costs 1 credit and returns timed segments plus the joined text. Videos without captions return a 404 with code NO_TRANSCRIPT.

Does the CLI work on Windows?

Yes. It runs anywhere Node.js 18 or later runs. On Windows the stored key lives in %APPDATA%\scrapeeverything\, and SCRAPEEVERYTHING_CONFIG_DIR moves it. In PowerShell, pipe the output to ConvertFrom-Json instead of jq.

How much does each CLI call cost?

The same as the API, because the CLI calls the API. Most endpoints cost 1 credit per successful call. A few feed and search endpoints cost 2 or 3, such as x posts (2) and instagram reels_search (3). se list shows every cost, failed calls are refunded, and se account is free.

Is there a free social media scraper CLI?

The CLI is free and MIT-licensed, and new accounts get 100 free credits with no payment details. After that, one-time packs start at $39 for 25,000 credits. Downloaders like yt-dlp are free too, but they fetch media files, not structured profile and comment data.

What happens when I run out of credits mid-script?

The CLI exits with code 4 (INSUFFICIENT_CREDITS) and prints the error to stderr. Check for exit 4 in your loop and stop, as the script above does. Credits never expire, so whatever is left in a pack stays on your account.