πŸ“–Crawl Operations

Chatting for the contents on a website is very simple. Just use our 3 APIs:

  • Create Crawl: start crawling a website.

  • Get Crawl: get the crawl results, including the crawl status, crawled pages, etc.

  • Index Crawl: after crawl completes and checks the crawl results, index the crawl results into the collection.

  • Delete Crawl: delete all crawled pages from the collection.

If you re-crawl the website, please make sure to call delete_crawl API before calling the index_crawl API. Or else, the same page may be indexed twice in the collection. Enhancements are planned to support incremental crawl.

Last updated