Scrape Everything
Researchers

Public social data for research

Pull profiles, comments and transcripts as typed JSON, and cache re-reads free for up to 14 days.

Get 100 free creditsRead the docs

Why it fits

Studying what people publish on social platforms usually starts with a fragile scraper, a spreadsheet of copied numbers or an official API that only covers accounts you own. Scrape Everything gives research teams a simpler starting point: send a username, a link or a video ID and receive typed JSON with stable field names, from public pages on nine platforms. You keep your effort for the questions, not for keeping a browser automation script alive.

  • YouTube and TikTok transcripts share one response shape: timed segments, full text, language and an auto-generated flag.
  • Comments come with like and reply counts: up to 100 per call on YouTube, the 15 newest on Instagram.
  • Every successful result is stored for 14 days. Re-reads with cache_days cost 0 credits and keep your dataset stable.
  • Limits are documented per endpoint, so you can describe them in your methods section.
  • One-time credit packs that never expire fit a grant budget better than a subscription.

In more detail

Text you can analyze

The transcript endpoints for YouTube and TikTok return timed segments plus the full text joined into one string, the language and whether the captions were generated automatically. Both share the same response shape, so one pipeline handles both platforms. The comments endpoints return top-level comments with like and reply counts: up to 100 per call on YouTube, and the 15 newest on Instagram, which is the public maximum for logged-out visitors. Search on YouTube returns results for a keyword, which helps build a sample before you collect the details.

Re-read without paying twice

Research means re-running code. Every successful live result is stored for 14 days. Add cache_days to a request and a repeat call for the same target within that window returns the stored copy for 0 credits, with cached: true and the original fetch time in the response. That keeps a dataset stable while you iterate on your analysis, and it means a bug in your parsing step does not cost a second round of collection. Cached results belong to your account only.

Know the limits of the data

Every endpoint returns what a logged-out visitor can see. Private accounts, follower lists and content behind a login are out of scope, and some counts are the platform's rounded figures. The docs and each endpoint page list these limits so you can describe them in your methods section. Request history is kept for 180 days, which gives you a log of when each item was collected.

Budget for a grant, not a subscription

Credits are bought in one-time packs and never expire, so a project can buy what its budget allows and spend it over months without a monthly fee. Most endpoints cost 1 credit per successful request, feed and search endpoints cost 2 or 3, and every response reports credits_charged and credits_remaining. A pilot collection on the 100 free credits every new account receives is usually enough to check that the fields you need exist before you commit funds.

Your responsibility

Public does not mean free of obligations. You remain responsible for how you use the data under the platforms' terms and data protection laws such as GDPR, including any ethics approval your institution requires. Our blog has a plain-language overview of the legal questions; it is not legal advice.

What it costs

A pilot study: 20 searches, then comments and a transcript for 500 videos

  • YouTube search20 calls × 1 credit = 20
  • YouTube comments500 calls × 1 credit = 500
  • YouTube transcript500 calls × 1 credit = 500
  • Total1,020 credits

≈ $1.59 at the Builder rate ($1.56 per 1,000 credits; the pack is 25,000 credits for $39.00)

≈ $1.21 at the Growth rate ($1.19 per 1,000 credits; the pack is 100,000 credits for $119.00)

≈ $0.92 at the Scale rate ($0.90 per 1,000 credits; the pack is 500,000 credits for $449.00)

Packs are bought whole. Credits you do not use stay on your balance and never expire. Failed calls are refunded and not counted here. Re-running the same collection within 14 days with cache_days costs 0 credits. Credit calculator

Endpoints to use

Each endpoint has its own page with fields, limits and an example response.

Example: collect comments for a set of videos

  1. Build a list of video URLs, for example from a YouTube search.
  2. Call the comments endpoint for each video with cache_days set to 14.
  3. Store the raw JSON with its request_id and fetch time.
  4. Re-run your pipeline freely: repeat calls within 14 days come from your cache at 0 credits.
cURL
curl -X POST https://api.scrapeeverything.com/v1/youtube/comments \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ","limit":20,"cache_days":14}'

Integrations that fit

Further reading

Researchers: questions

How long are results cached?

Every successful live result is stored for 14 days. Send cache_days from 1 to 14 to reuse it for free. Without cache_days, every request is live.

Can I get replies to comments?

Not today. The comments endpoints return top-level comments with a reply count.

Are transcripts speech-to-text?

No. They come from the video's existing captions, either added by the creator or generated by the platform. Videos without captions have no transcript.

Is this legal for research?

We return only data visible without logging in. You stay responsible for your use under platform terms and laws like GDPR. This is not legal advice.

Start with 100 free credits.

Continue with Google, create an API key and send your first request in under a minute.

Get 100 free credits