Web import
Overview
Web import lets you add website content to your AI Agent’s Knowledge base when that content doesn’t live in a connected knowledge base integration. The scraper goes through your public-facing website content and saves the text as new articles on your Knowledge page.
Limitations
Web import has the following constraints:
- You can only scrape content on public-facing websites (i.e., you can’t require users to log in to see it).
- The website scraper imports articles up to 100KB in size. Articles larger than this limit will be truncated.
- The scraper follows links from your starting URL up to five levels deep (e.g.,
www.website.com/level_1/level_2/level_3/level_4/level_5). It won’t import any articles that are deeper into your page hierarchy. - Your AI Agent can have a maximum of 50,000 articles in it by default, and a web source can have up to 1,000 articles in it. If your AI Agent reaches either number, the import will stop. Higher article limits are available for eligible plans — contact your Ada team.
- It may not be possible to import certain types of websites. Web imports work best with websites that are written in static semantic HTML. Some websites may not import properly:
- Websites with web crawlers blocked - if your website has a blocker for web crawlers, Ada’s scraper won’t be able to access its content.
- Websites that are not written with semantic HTML - articles scraped from these websites may contain content from things like navigation menus, headers, footers, or other page elements that don’t belong in your AI Agent’s knowledge.
- Websites with complex redirection - sites that rely on multiple or dynamic redirects (e.g., geo-based routing, authentication gates, or JavaScript-based redirects) may cause the scraper to retrieve unexpected, or incomplete content.
- URLs cannot exceed 1024 characters.
- Multi-language content is not supported – all scraped articles are currently set to English, regardless of the original page language.
Use cases
Web import is useful when your content is not in a supported knowledge base integration.
- Marketing and product pages: Import public-facing website content such as product descriptions, feature pages, or pricing information.
- Blog and news content: Add blog posts, announcements, or news articles to your AI Agent’s knowledge.
- Support content outside your help center: Import FAQ pages, troubleshooting guides, or documentation that lives on your main website rather than a dedicated knowledge base.
- Supplementary information: Add content from partner sites, landing pages, or other sources that your AI Agent should reference.
Capabilities & configuration
Web import can scrape static semantic HTML websites and convert the content into Knowledge articles.
Ignored HTML elements
Your website likely contains page elements you don’t want to have scraped and saved in your AI Agent (e.g., headers and footers). By default, the scraper is programmed to skip the elements that are least likely to contain relevant page information.
The website scraper is programmed to ignore HTML elements that match the following tags:
buttoniframeimgmetanavnoscriptpicturescriptstylesvgaudiovideo
Exclude page elements
Some websites repeat content on every page that does not belong in your AI Agent’s Knowledge, such as a large site footer. You can give a website import a list of CSS selectors. Before each page becomes an article, the scraper removes every element that matches one of these selectors, in addition to the ignored HTML elements above.
The selectors apply from the next sync of the source. Existing articles keep their content until that sync.
To exclude page elements in the dashboard:
-
On the Ada dashboard, go to Config > AI AGENT > Knowledge. Then do one of these steps:
- To add a new source, click Add source and select Website.
- To change an existing source, open the Sources tab, find the website, and click Settings.
-
In the Add website window or the Settings window, open Advanced settings.
-
In Exclude selectors, enter one CSS selector on each line.
To see the selectors that you can use, point to the info icon next to Exclude selectors.
-
For a new source, fill in the other fields as in Quick start, then click Add. For an existing source, click Update or Sync now.
You can also set the selectors in these places:
- Knowledge API:
exclude_selectorsonPATCH /v2/knowledge/sources/{id}. The source responses also return it. For more information, see Sources in the API reference. - MCP server:
exclude_selectorsinedit_agent_configwhen you create or update a website import. Only an owner or an admin can set it. - Ada CLI:
--exclude-selectorsonknowledge scrapeandknowledge sources update, and--clear-exclude-selectorsonknowledge sources update. For more information, see Ada CLI.
The selectors have these rules:
- A source can have up to 20 selectors. Each selector can be up to 75 characters long.
- You can use these selectors:
- Element names (
footer), classes (.site-footer), IDs (#promo) and attribute selectors ([role="contentinfo"]). An attribute selector can use the=,~=,|=,^=,$=and*=operators, and theiorsflag. - The descendant (
main aside), child (main > aside) and~(h2 ~ ul) combinators. :not()with one simple selector,:empty,:linkand:any-link.
- Element names (
- You cannot use these selectors:
- Position-based pseudo-classes:
:first-child,:last-child,:nth-*,:only-*and*-of-type. - The
+combinator. - A leading
*, the!=attribute operator, pseudo-elements such as::before, CSS comments and namespaces. - Other pseudo-classes, such as
:hover.
- Position-based pseudo-classes:
- The scraper applies a selector again after each removal. Position-based pseudo-classes and the
+combinator would then remove more elements than they match on the original page. - You cannot change the selectors while a sync of the source runs. Wait for the sync to finish, then try again.
If the selectors would remove all of the content of a page, the page keeps its last imported version, and the sync does not update it. Change the selectors and sync the source again to update the page.
User agent
When importing website content, Ada’s scraper identifies itself with a static User-Agent header:
If your website restricts crawlers with an allowlist (or you operate a site whose content Ada is importing), allowlist the AdaWebCrawler user agent to ensure Ada can continue to access your content.
Quick start
Import website content to your AI Agent’s Knowledge base in a few steps.
To import a website:
For more options, see Import a website.
Implementation & usage
Import, update, and remove website content to control what information your AI Agent uses.
Import a website
Import your website content to make it available as Knowledge for your AI Agent.
To import your website’s content:
-
On the Ada dashboard, go to Config > AI AGENT > Knowledge, then click Add source and select Website.
The Add website window opens.
-
Under Source name, give your source a name. Each source name in your AI Agent must be unique, so you can identify and filter by the source on your Knowledge page.
-
Under Content to add, choose the pages you want to import.
-
To import your entire website, where you provide a single URL and your AI Agent follows the links on that website and scrapes those pages too, select Every webpage starting from one URL. Then, add the URL you want your AI Agent to start scraping from.
- For best results, use a root domain, like
https://mywebsite.com, instead of a section of your website, likehttps://mywebsite.com/pages. - Be aware of any redirects in your website. The scraper will import redirected sites, as long as they start with the URL you enter.
To leave pages out of the import, add their paths under Exclude pages, separating the paths with commas. A path excludes that page and every page under it. For example,
/blogexcludeshttps://mywebsite.com/blogandhttps://mywebsite.com/blog/my-post. Use*to match any text:/*/archiveexcludes/news/archiveand/events/archive. You can also paste a full URL, and your AI Agent uses its path. You can add up to 50 paths. - For best results, use a root domain, like
-
To import specific pages on your website, where your AI Agent only scrapes the pages you provide, select A specific list of webpages. Then, add the list of URLs you want your AI to import, separating the URLs with commas. Your list can be up to 5000 characters long.
-
-
Under Automatically sync, choose when your AI Agent re-syncs this source:
- Scheduled (default): pick the days of the week, one or two times a day at least 4 hours apart, and the timezone. The default is once a week, on the current day of the week, at 23:00 in your timezone. A sync starts at each chosen time, and a large website can take a few hours to finish. The dashboard does not show the next sync time.
- Manual: your AI Agent re-syncs the source only when you click Sync now.
-
Under Availability, choose which end users the imported articles are available to:
- Everyone (default): your AI Agent can use the imported articles in any conversation.
- Based on the following rules…: your AI Agent can use the imported articles only in conversations where the variable conditions you set are met. Every page that the import adds gets these availability rules. For more information, see Restrict article availability.
-
Click Add.
The Add website window closes, and your AI Agent saves your page source on the Sources tab and starts importing its content.
Large websites can take up to 24 hours to finish importing. Feel free to leave the webpage while this is happening, as your AI Agent will email you when your import has finished.
By default, all of your imported articles are set to Active, but you can change availability settings as needed for any of your articles. For more information, see Manage your knowledge content.
Re-import a website
Keep your Knowledge current by re-importing website content after updates.
Scheduled syncs: your AI Agent re-syncs each Scheduled source on the days and at the times you set, in the source’s timezone. You set the schedule when you add a source and can change it later from the source’s Settings.
On-demand sync: You can trigger an immediate re-sync at any time, without waiting for the next scheduled sync. Use Sync now on the Sources tab, or call the Knowledge API sync endpoint. Neither changes the schedule. The sync endpoint allows one sync per source every 4 hours. The Knowledge API also reads and sets the schedule as sync_schedule on the source.
Article updates preserve existing references in your coaching and rules, ensuring your automations continue to work seamlessly.
Availability rules that you set on a source apply only to pages that an import adds after you set them. Pages that already exist as articles keep their own availability rules, even if you change the source’s rules later. To change the rules on existing articles, use Set availability on the Knowledge page. A page that was removed from your website and is later imported again counts as a new page.
To re-import your website’s content:
-
On the Ada dashboard, go to Config > AI AGENT > Knowledge, then open the Sources tab.
-
Find the website you want to re-import, and click Settings.
The Settings window opens, with the import settings pre-populated.
For a source that imports every webpage starting from one URL, you can change the Exclude pages list here. A change applies from the next sync. The Knowledge API also reads and sets the list as
exclude_pathson the source. Articles already imported from a page you exclude are not removed, and they no longer update. To remove them, delete them or set them to Inactive on the Knowledge page. For more information, see Manage your knowledge content. -
Click Update to save your changes without a sync, or Sync now to start a sync. Sync now saves a changed sync schedule, availability rules or Exclude pages list, but not a changed source name. To rename the source, click Update.
After Sync now, the Settings window closes and starts re-importing your website. Refresh your page to see updates on your import’s progress. Large imports may take a while to fully import.
Delete a website source
Remove a website source when you no longer need its content in your AI Agent.
Removing a website as a source also deletes all of the articles from that website from your AI Agent.
To delete a website source:
-
On the Ada dashboard, go to Config > AI AGENT > Knowledge, then open the Sources tab.
-
Find the website you want to delete, and click Settings.
The Settings window opens, with the import settings pre-populated.
-
Click Delete source.
A confirmation message appears, to remind you that deleting the source also deletes all of the articles from your AI Agent. To proceed, click Delete.
Related features
Expand your AI Agent’s Knowledge with these related capabilities:
- Knowledge integrations: Connect external knowledge bases like Zendesk, Salesforce, or Contentful.
- Article management: Create and manage Knowledge articles directly in Ada.