X-Robots-Tag HTTP header parser class

PHP class to parse X-Robots-Tag HTTP headers according to Google X-Robots-Tag HTTP header specifications.

Installation

The library is available for install via Composer package. To install via Composer, please add the requirement to your composer.json file, like this:

{
    "require": {
        "vipnytt/x-robots-tag-parser": "0.*"
    }
}

and then use composer to load the lib:

<?php
    require 'vendor/autoload.php';
    $parser = new \vipnytt\XRobotsTagParser($url, $userAgent);
    ...

You can find out more about Composer here: https://getcomposer.org/

Usage

Coming soon...

Directives

Supported:

• all - There are no restrictions for indexing or serving.
• none - Equivalent to noindex, nofollow
• noindex - Do not show this page in search results and do not show a "Cached" link in search results.
• nofollow - Do not follow the links on this page
• noarchive - Do not show a "Cached" link in search results.
• nosnippet - Do not show a snippet in the search results for this page
• noodp - Do not use metadata from the Open Directory project for titles or snippets shown for this page.
• notranslate - Do not offer translation of this page in search results.
• noimageindex - Do not index images on this page.

Unsupported (work in progress):

• unavailable_after: [RFC-850 date/time] -Do not show this page in search results after the specified date/time.