CF-Incidents Browser Run - Crawl results include URL outcomes and robots.txt diagnostics

[ENF News Bot]

Browser Run /crawl results now include more detailed diagnostics. Each URL record can include an outcome that identifies the specific processing result, such as success, robots_disallowed, timeout, or page_error.

The job-level robots array reports the robots.txt state for up to 50 origins encountered by the crawl. Each entry includes the origin, whether usable rules were available, and the HTTP status when the crawler received a response.

{
	"robots": [
		{
			"origin": "https://example.com",
			"state": "available",
			"status": 200
		}
	],
	"records": [
		{
			"url": "https://example.com/docs/",
			"status": "completed",
			"outcome": "success"
		}
	]
}

For field definitions and all possible outcome values, refer to URL outcomes and robots.txt diagnostics.

Continue reading...