CF-Incidents AI Search - AI Search makes it easier to build a search engine for your data

[ENF News Bot]

AI Search gets you from a data source to a working search endpoint quickly. This release adds what you need to put that endpoint in front of real users: your own domain, authentication, and one endpoint across several instances. It also adds crawling for sites without a complete sitemap, so your index covers everything you want it to find.

Each of the following is a new option. The previous behavior is still the default, so nothing changes until you change it.

Serve search from your own domain​

https://developers.cloudflare.com/changelog/rss/#serve-search-from-your-own-domain

A public endpoint is a URL that a site or app can query directly, with no authentication in front of it. By default that URL is a generated hostname on search.ai.cloudflare.com. You can now serve the same endpoint from a custom domain, a hostname in a zone that you own:


https://search.example.com/search

Restrict who can query your content​

https://developers.cloudflare.com/changelog/rss/#restrict-who-can-query-your-content

Once your endpoint is on your own domain, you can put Cloudflare Access in front of it. For example, you usually want to give /mcp to specific agents rather than to anyone who finds the URL. Agents authenticate with an Access service token, and people who open the endpoint in a browser sign in through your identity provider.

Search several instances from one URL​

https://developers.cloudflare.com/changelog/rss/#search-several-instances-from-one-url

A namespace can expose its own public endpoint with /search, /chat/completions, and /mcp paths that fan out across the instances you choose:

curl https://ns-<NAMESPACE_ENDPOINT_ID>.search.ai.cloudflare.com/search \
  --header "Content-Type: application/json" \
  --data '{
    "messages": [{ "content": "How do I configure AI Search?", "role": "user" }],
    "ai_search_options": { "instance_ids": ["docs", "support"] }
  }'

Index your sites without a sitemap​

https://developers.cloudflare.com/changelog/rss/#index-your-sites-without-a-sitemap

Website data sources support a new discover parse type. It starts at the source URL and collects pages from both your sitemaps and the links it finds while crawling:

curl -X POST "https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/ai-search/instances" \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "my-ai-search",
    "type": "web-crawler",
    "source": "example.com",
    "source_params": {
      "web_crawler": {
        "parse_type": "discover",
        "discover_options": { "source": "links", "limit": 5000, "depth": 3 }
      }
    }
  }'

To learn more, refer to the AI Search documentation.

Continue reading...