Domain Crawler for Imported URLs in Knowledge

What is it?

When adding a public URL as a knowledge source in Customer Agent, admins can now enable automatic crawling of up to 5,000 pages linked from that domain, with path-level controls to include or exclude specific sections of the site.

Why does it matter?

Adding website knowledge to Customer Agent used to mean importing URLs one at a time and manually keeping them current. For companies with large support sites or documentation hubs, that's a real barrier to good agent knowledge. This feature removes the maintenance burden; admins set it once, and the agent's knowledge stays fresh automatically.

How does it work?

• When configuring a public URL source, admins will see a toggle to enable "Import related URLs."
  
• From there, admins can choose which pages to import — all pages on the domain, or filtered by path. Path filters let admins include only specific sections (e.g. /support/) or skip others (e.g. /blog/).

• The crawler discovers pages linked from the root URL, up to 5,000 pages total. All crawled pages refresh automatically on a weekly cadence, no manual re-import required.

Who gets it?

HubSpot Credits, Content Hub Professional, Content Hub Enterprise, Marketing Hub Professional, Marketing Hub Enterprise, Data Hub Professional, Data Hub Enterprise, Sales Hub Professional, Sales Hub Enterprise, Service Hub Professional, Service Hub Enterprise, Smart CRM Professional, Smart CRM Enterprise