LLMs read Markdown well and HTML badly. Raw HTML is mostly scripts, styles, attributes and navigation, which wastes context and confuses the model. Fetching a page without a browser misses everything rendered by JavaScript. Render the page in a real browser, strip it to its content, and get Markdown back.
Any URL to Markdown
curl -X POST "https://shotkit.net/api/take" \
-H "X-Access-Key: YOUR_ACCESS_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com/blog/launch","format":"markdown","block_cookie_banners":true,"selector":"main"}' \
--fail-with-body -o shot.mdThe page is loaded in Chrome first, so single-page apps, client-side rendered content and pages assembled by JavaScript come through. Scripts, styles and <noscript> are removed; headings, lists, links, emphasis, code blocks and images stay.
Only the part that matters
Navigation, footers and sidebars repeat on every page and add nothing to a RAG index. selector returns only the matching element (try main, article or #content), and hide_selectors removes pieces inside it:
curl -X POST "https://shotkit.net/api/take" \
-H "X-Access-Key: YOUR_ACCESS_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://docs.example.com/guide/install","format":"markdown","selector":"article","hide_selectors":[".feedback-widget",".edit-on-github"]}' \
--fail-with-body -o shot.mdPages that need a click or a login
clickopens a "Read more", accepts an age gate or expands an FAQ before the content is read.scriptscan open every<details>element or switch tabs.cookiesorauthorizationload pages behind a login, for an internal knowledge base.include_shadow_domincludes content inside web components.wait_for_selectorwaits for content that loads after the page.
A screenshot and the text in one request
Agents that look and read can get both from one render: the screenshot in the response, the Markdown at a URL. See HTML and Markdown.
curl -X POST "https://shotkit.net/api/take" \
-H "X-Access-Key: YOUR_ACCESS_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","response_type":"json","metadata_content":true,"metadata_content_format":"markdown","image_width":1024}' \
--fail-with-body -o shot.jsonFor agents: use the MCP server
Claude, Cursor, VS Code and other MCP clients can connect to https://shotkit.net/mcp and get a page_to_markdown tool (and a screenshot tool) without any code. See MCP server.
Building a RAG index
- Get the URLs from the site's
sitemap.xml. shotkit renders one URL per request; it doesn't crawl. - Send them with
async=trueand awebhook_url, withexternal_identifierset to the URL or document ID. - Split each Markdown document on its headings; they make natural chunks with titles.
- Re-run on a schedule and re-embed only documents whose Markdown changed.
Frequently asked questions
Does it crawl a whole site?
No. Each request converts one URL. Get the list of URLs from the sitemap or your own crawler and send one request per page.
Are tables kept?
Table contents come through as text, but not as Markdown tables. If table structure matters, request format=html with selector on the table and convert it yourself.
How is this different from a plain HTTP fetch?
A fetch gets the HTML the server sends; for many modern sites that's an empty shell filled in by JavaScript. Rendering first gets what a person sees, with cookie banners and overlays removed.
Does it count as a screenshot?
Yes. A Markdown or HTML render counts as one request toward your monthly quota, like an image.