check if a website has a robots.txt file using PHP
Mark E.
In PHP, you can use the file_get_contents()
function to fetch the contents of the robots.txt
file, and then check if the file exists and is accessible using the http_response_code()
function. Here's an example code snippet:
<?php
$robotsTxtUrl = 'http://buildyourowngokart.com/robots.txt';
$robotsTxtContent = @file_get_contents($robotsTxtUrl);
if (http_response_code() === 200 && $robotsTxtContent !== false) {
echo "The robots.txt file exists and is accessible";
} else {
echo"The robots.txt file doesn't exist or is not accessible";
}
?>
Note that you need to replace http://example.com/robots.txt
with the actual URL of the robots.txt
file that you want to check. Also, keep in mind that some websites may block requests to their robots.txt
file, so this method may not work in all cases.