> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.ada.cx/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.ada.cx/_mcp/server.

# Static user agent for web import

Ada's web import scraper now identifies itself with a static `User-Agent: AdaWebCrawler` header on every request. Websites that restrict crawling with an allowlist can add `AdaWebCrawler` to keep Ada's scraper authorized — previously the user agent was dynamic and could not be reliably allowlisted.

Learn more in our [Web import documentation](/docs/knowledge/content-ingestion/web-import#user-agent).