Web scraper improvements

We’ve made some improvements to our web scraper you can use to import content from your knowledge base into your AI Agent:

Your AI Agent now cleans up URL inputs for you, so you won’t see errors if you don’t start a URL with https:// or have trailing slashes / at the end.
You now have two ways of knowing when your import is finished:
- The dashboard automatically updates, so you don’t have to manually refresh the page anymore.
- Your AI Agent will now send you an email to let you know the import is done.
We’ve added the ability to scrape websites that have content dynamically populated with JavaScript.
If you’re importing a list of URLs, we’ve increased the limit for that list to 5000 characters.
We’ll still tell you if you’re importing duplicate URLs, but if you’re okay with that, you can decide to scrape duplicate pages anyway.

We’ve made some improvements to our web scraper you can use to import content from your knowledge base into your AI Agent:

Your AI Agent now cleans up URL inputs for you, so you won’t see errors if you don’t start a URL with https:// or have trailing slashes / at the end.
You now have two ways of knowing when your import is finished:
- The dashboard automatically updates, so you don’t have to manually refresh the page anymore.
- Your AI Agent will now send you an email to let you know the import is done.
We’ve added the ability to scrape websites that have content dynamically populated with JavaScript.
If you’re importing a list of URLs, we’ve increased the limit for that list to 5000 characters.
We’ll still tell you if you’re importing duplicate URLs, but if you’re okay with that, you can decide to scrape duplicate pages anyway.

Release notes