Web Page Reader
Turn a public web page into clean Markdown, metadata, links or an SEO report. Honours robots.txt.
base https://web-page-reader.p.rapidapi.com/web/v1- endpoints
- 6
- free plan, paid from
- $13/mo
- data
- Live source
Fetch a public web page and get its main content as Markdown ready for an LLM, its metadata, its links, an on-page SEO report, or the site's robots.txt and sitemap.
// What will a link preview of this page show?
curl "https://web-page-reader.p.rapidapi.com/web/v1/metadata?url=https://example.com" \
-H "X-RapidAPI-Key: $RAPIDAPI_KEY" \
-H "X-RapidAPI-Host: web-page-reader.p.rapidapi.com"{
"url": "https://example.com",
"title": "Example Domain",
"description": null,
"image": null,
"site_name": null,
"type": null,
"author": null,
"published": null,
"language": "en",
"canonical": null,
"favicon": "data:,",
"theme_color": null,
"twitter_card": null,
"open_graph": {},
"twitter": {},
"structured_data_types": [],
"status": 200,
"redirected_from": []
}Endpoints
Convert a web page to clean Markdown
Fetches the page, removes navigation, ads and boilerplate, and returns the main content as Markdown, ready to pass to a language model. JavaScript is not executed, so pages that render entirely in the browser return little content.
urlqueryrequiredFull page URL, starting with http:// or https://.
formatquerymarkdown or text.
include_linksqueryKeep hyperlinks in the Markdown.
include_imagesqueryKeep image references in the Markdown.
include_tablesquerymain_content_onlyqueryDrop navigation, footers and sidebars. Set false to keep more of the page.
Link preview: title, description, image and Open Graph data
Everything needed to render a link preview card, plus the page's raw Open Graph, Twitter card and structured-data types.
urlqueryrequiredFull page URL, starting with http:// or https://.
List the links on a page
urlqueryrequiredFull page URL, starting with http:// or https://.
scopequerylimitquery
On-page SEO audit
Checks the title, meta description, headings, images, canonical link, indexability, mobile viewport, structured data and social tags of one page, and returns a pass / warn / fail list with a score.
urlqueryrequiredFull page URL, starting with http:// or https://.
Read a site's robots.txt and test a URL against it
urlqueryrequiredFull page URL, starting with http:// or https://.
user_agentqueryCrawler name to test, for example Googlebot or GPTBot.
Parse an XML sitemap
Returns the URLs in a sitemap with last-modified dates, or the child sitemaps when it is a sitemap index. Gzipped sitemaps are supported.
urlqueryrequiredSitemap URL, or a site URL to discover its sitemap from robots.txt.
limitqueryoffsetquery
Data and limits
source
Fetches the page you name. The fetcher identifies itself and honours robots.txt.
Queries the upstream source on each request.
limits
- JavaScript is not run, so pages built in the browser return little.
- Sites that block bots or disallow crawlers in robots.txt return an error instead of content.
- Pages over 3 MB, non-standard ports and non-public addresses are refused.
- You are responsible for your right to use the content you fetch.
Plans
Monthly, in US dollars. Each plan stops at its quota, with no overage charges. Subscribe and pay on RapidAPI.
| plan | price | calls / month | per 1k calls | |
|---|---|---|---|---|
| Basic | Free | 100 | — | start free on RapidAPI ↗ |
| Pro | $13 | 5,000 | $2.60 | subscribe on RapidAPI ↗ |
| Ultra | $56 | 25,000 | $2.24 | subscribe on RapidAPI ↗ |
| Mega | $201 | 100,000 | $2.01 | subscribe on RapidAPI ↗ |