Robots Meta Directives

Control indexing and snippets page by page, or per file.

Robots rules tell search engines whether to index a page and how it may appear in results. They go in a robots meta tag in the page's head, in a googlebot meta tag for Google only, or in an X-Robots-Tag HTTP header, which is the only way to apply them to PDFs, images and other files that aren't HTML. Rules can be combined with commas, and when two rules conflict Google applies the more restrictive one.

The most important thing to understand is how robots rules interact with robots.txt. A crawler has to fetch a page to read its noindex rule, so if robots.txt blocks the page, the noindex is never seen and the URL can still be indexed from links pointing at it, usually without a snippet. To remove a page from Google, allow it to be crawled and return noindex, or return 404 or 410. The table below lists every rule from Google's specification; noarchive and nocache are no longer used by Google Search.

Rules

Rule	What it does	Example
all	No restrictions. The default, so it never needs to be written.	<meta name="robots" content="all">
noindex	Don't show this page, media or resource in search results.	<meta name="robots" content="noindex">
nofollow	Don't follow the links on this page.	<meta name="robots" content="nofollow">
none	Same as noindex, nofollow.	<meta name="robots" content="none">
nosnippet	Don't show a text snippet or video preview for this page in search results.	<meta name="robots" content="nosnippet">
max-snippet:[number]	Use at most this many characters of text in a snippet. 0 is like nosnippet; -1 means no limit.	<meta name="robots" content="max-snippet:160">
max-image-preview:[setting]	Largest image preview: none, standard or large.	<meta name="robots" content="max-image-preview:large">
max-video-preview:[number]	Longest video preview in seconds. 0 allows only a static image; -1 means no limit.	<meta name="robots" content="max-video-preview:30">
noimageindex	Don't index images on this page. Images also linked or embedded elsewhere can still be indexed from there.	<meta name="robots" content="noimageindex">
notranslate	Don't offer a translation of this page in search results.	<meta name="googlebot" content="notranslate">
indexifembedded	Allow indexing when embedded in another page through an iframe, despite noindex. Only works together with noindex.	<meta name="googlebot" content="noindex, indexifembedded">
unavailable_after:[date]	Stop showing the page in results after this date and time. Use a widely accepted format such as ISO 8601.	<meta name="robots" content="unavailable_after: 2027-06-30">

Where to put them

Syntax	What it does	Works on
<meta name="robots">	In the <head> of an HTML page. Read by Google and other crawlers that support it.	HTML pages
<meta name="googlebot">	The same rules, for Google's crawlers only. Other names such as googlebot-news target one crawler.	HTML pages
X-Robots-Tag: noindex	An HTTP response header with the same rules. The only option for PDFs, images and other non-HTML files.	Any file type, set in the server config
X-Robots-Tag: googlebot: nofollow	A header rule for one crawler, named before the colon.	Any file type
<span data-nosnippet>…</span>	Keeps part of a page out of snippets. Allowed on span, div and section elements.	Part of an HTML page

Good to know

noindex and robots.txt don't mix

Don't block a URL in robots.txt and add noindex to it. Google can't crawl the page, so it never sees the noindex. Remove the robots.txt block until the page has dropped out of the index.

Rules are case-insensitive

Rule names and the meta name attribute are not case-sensitive, so NOINDEX and noindex do the same thing. Separate several rules with commas in one tag, or use several tags.

JavaScript-added rules

Google can read a robots meta tag added by JavaScript, but if the original HTML says noindex, Google may skip rendering and never see a script that removes it. Put the rule you mean in the HTML from the server.