Data quality and limitsMenu
Data quality and limits
What to know before you analyze the data or describe it in a methods section. Everything here comes from the live endpoint catalog and its notes.
Versioning today
- All endpoints live under
/v1. The OpenAPI spec describes the current version. - Changes we ship are listed in the changelog, with the date.
- There is no published deprecation policy or schedule yet. Platforms change their public pages, and a field can become null when a platform stops showing it.
- For reproducible work, store the raw JSON with its
meta.request_id, the endpoint and the time you collected it.
Rounded and approximate fields
These fields are parsed from the platform's rounded or relative text (for example “521M” or “2 days ago”), or estimated. Exact values are not available for them.
| Endpoint | Field | Note |
|---|---|---|
| instagram.reels | reels[].taken_at | Approximate publish time, Unix seconds (decoded from the media ID; usually a few minutes early) |
| youtube.profile | subscriber_count | Approximate subscribers parsed from rounded text |
| youtube.profile | video_count | Approximate video count from rounded text |
| youtube.videos | subscriber_count | Approximate subscribers from rounded text |
| youtube.videos | video_count | Approximate video count from rounded text |
| youtube.videos | videos[].view_count | Approximate views from rounded text |
| youtube.videos | videos[].published_text | Relative time, e.g. '12 days ago' |
| youtube.search | results[].view_count | Video items: views (exact for regular results, rounded for Shorts shelf items; current viewers for live streams) |
| youtube.search | results[].published_text | Video items: relative time, e.g. '1 month ago' |
| youtube.search | results[].subscriber_count | Channel items: approximate subscribers from rounded text |
| youtube.comments | comments[].like_count | Likes, approximate from rounded text (0 when none) |
| youtube.comments | comments[].published_text | Relative time, e.g. '2 months ago' or '6 years ago (edited)' |
| youtube.playlist | videos[].view_count | Approximate views from rounded text |
| youtube.playlist | videos[].published_text | Relative time, e.g. '4 years ago' |
| telegram.posts | subscriber_count | Approximate subscribers from compact text |
| telegram.posts | photo_count | Approximate number of photos |
| telegram.posts | video_count | Approximate number of videos |
| telegram.posts | file_count | Approximate number of files |
| telegram.posts | link_count | Approximate number of links |
| telegram.posts | posts[].view_count | Approximate views from compact text |
| telegram.post | view_count | Approximate views from compact text |
How lists are ordered
List endpoints return items in the platform's own order, cut at the endpoint's maximum. Plan your sampling around this order.
| Endpoint | List | Order |
|---|---|---|
| instagram.photos | posts | Most recent posts, newest first, max limit |
| instagram.comments | comments | Most recent top-level comments, newest first, max limit (at most 15) |
| instagram.reels | reels | Reels in the order of the profile's Reels tab (newest first), max limit |
| instagram.reels_search | reels | Reels in search-engine order, at most limit |
| x.posts | posts | Timeline tweets in timeline order, max limit |
| tiktok.posts | posts | Newest videos from the embed, max limit |
| tiktok.comments | comments | Top-level comments in TikTok's relevance order, max limit |
| tiktok.transcript | segments | Caption lines in order |
| youtube.search | results | Results in YouTube's order, max limit. Fields depend on type |
| youtube.playlist | videos | Videos in playlist order, max limit |
| telegram.posts | posts | Messages, newest first, max limit |
| spotify.playlist | tracks | First ~50 playlist tracks |
| pinterest.search | pins | Matching pins in Pinterest's order, max limit |
| pinterest.board | pins | Board pins in board order, max limit |
| twitch.videos | videos | Videos, newest first, max limit |
| twitch.clips | clips | Clips, most viewed first, max limit |
Known gaps
| Area | What is missing |
|---|---|
| All platforms | Only data a logged-out visitor can see. No private accounts, follower or following lists, or analytics such as reach, saves or impressions. |
| Comments | Top-level comments only, with a reply count. Instagram returns the 15 newest per post; YouTube and TikTok up to 100 per call. |
| Feeds | Each call returns the latest items up to a fixed maximum (for example 60 Instagram posts, 60 X posts, 12 TikTok videos, 30 YouTube uploads, 100 Telegram messages). There is no paging into older history. |
| Search | YouTube search returns up to 50 results in YouTube's own ranking. Instagram reel search goes through a public search engine's index, so coverage and freshness follow that index. There is no X search. |
| Transcripts | Only videos with captions (creator-added or platform-generated). No speech-to-text; no machine-translated tracks. |
| Counts | Some counts are the platform's rounded figures (see the table below). Where a platform hides a count from logged-out visitors, the field is null. |
| Media links | Video and image URLs are platform CDN links. Some, such as Instagram reel MP4 links, expire within days. |
Each endpoint's “Good to know” notes in the endpoint reference list the details.
Freshness and caching
Without cache_days, every request fetches the public page live. With it, you may get a stored result up to that many days old (1 to 14), marked meta.cached: true with meta.cached_at. Record both if freshness matters to your analysis. See Caching.
Writing this up for a paper? See Citing Scrape Everything for a suggested citation and methods statement.