check if a website has a robots.txt file using PHP

In PHP, you can use the file_get_contents() function to fetch the contents of the robots.txt file, and then check if the file exists and is accessible using the http_response_code() function. Here's an example code snippet:

$robotsTxtUrl = '';
$robotsTxtContent = @file_get_contents($robotsTxtUrl);

if (http_response_code() === 200 && $robotsTxtContent !== false) {
   echo "The robots.txt file exists and is accessible";
} else {
   echo"The robots.txt file doesn't exist or is not accessible";

Note that you need to replace with the actual URL of the robots.txt file that you want to check. Also, keep in mind that some websites may block requests to their robots.txt file, so this method may not work in all cases.

Related Articles

- All From ChatGPT