Turn a documentation site into one Markdown file for ChatGPT, Claude or Cursor

AI assistants answer better about a library or product when they have its documentation in front of them. Pasting 80 pages one at a time is not realistic, and saving pages as HTML wastes most of the context on navigation, footers and scripts. What you want is one clean Markdown file: every page's main content, with headings, code blocks and tables kept, and a source URL on each section.

Some sites now publish exactly that as llms-full.txt. For the many that do not, this guide builds it.

Disclosure: I'm Aurelix, and I built the Website to Markdown Crawler used below. It is a paid tool, priced per page converted; the current price is on its tool page.

The input

{
  "startUrls": ["https://docs.apify.com/academy"],
  "maxPages": 100,
  "scope": "path",
  "excludeGlobs": ["**/changelog/**"]
}

scope: "path" follows only links under the start URL's folder, so starting at /academy stays in the Academy; "domain" crawls the whole site.
includeGlobs and excludeGlobs fine-tune it (** matches anything).
maxPages caps the pages converted and charged; skipped pages (robots.txt, errors, duplicates) do not count.

The crawler obeys robots.txt and goes at most two requests per second per site.

Real output

One row per page. From a run of this input:

url	title	words
docs.apify.com/academy	Apify Academy	231
docs.apify.com/academy/scraping-with-ai	Web scraping with AI and Apify	419
docs.apify.com/academy/api-scraping	API scraping	889

The markdown field starts like this, with the site's menus and footer gone:

# Web scraping with AI and Apify

**Learn how to use AI to extract information from websites, starting from the absolute basics.**

---

In this practical course, we'll use AI assistants to create an ...

Each row also has description, language, last_modified (when the server sends it), the page's headings and links, crawl_depth and http_status.

Merge into one file in Python

pip install apify-client

import os

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("aurelix/website-to-markdown").call(run_input={
    "startUrls": ["https://docs.apify.com/academy"],
    "maxPages": 100,
    "scope": "path",
})
pages = sorted(client.dataset(run["defaultDatasetId"]).iterate_items(), key=lambda p: p["url"])

parts = [f"# Apify Academy documentation\n\n{len(pages)} pages, one section per page.\n"]
for p in pages:
    parts.append(f"\n---\n\nSource: {p['url']}\n\n{p['markdown'].strip()}\n")
text = "".join(parts)

with open("academy-docs.md", "w", encoding="utf-8") as f:
    f.write(text)
words = sum(p.get("word_count") or 0 for p in pages)
print(f"{len(pages)} pages, about {words:,} words, {len(text):,} characters -> academy-docs.md")

Sorting by URL keeps related pages together. The Source: line before each page lets the assistant cite where an answer came from.

Using the file

ChatGPT or Claude: attach academy-docs.md to a chat, or add it to a Claude Project or a custom GPT's knowledge so every chat can use it.
Cursor and other editors: put it in the repository (for example docs/vendor/academy-docs.md) and reference it in your prompt.
Too big for one context window? Split it by top-level path, or use the word count printed above to keep each file under your model's limit. For retrieval (RAG) rather than one big file, see Turn any website into clean Markdown chunks for RAG.

To keep the file current, schedule the run weekly and rebuild it.

Limits

No JavaScript rendering: pages that are only an empty app shell until JavaScript runs are skipped and listed in the log, not converted.
Only HTML pages are converted; PDFs, images and other files are skipped.
One site per start URL: www. and the bare domain count as one site, other subdomains are not followed.
Pages behind a login, paywall or bot challenge are not accessible.