Does it matter if the Disallow path is different from Drupal’s directory? -
i'm looking noindex tag pages i.e.
http://example.com/tags/tabs http://example.com/tags/people
etc.
if add following robots.txt page (see: http://jsfiddle.net/psac2uzy/)
disallow: /tags/ disallow: /tags/*
will stop google indexing tag pages?
even though paths aren't same drupal structure (since drupal keeps content in database)?
note: can’t disallow indexing robots.txt, can disallow crawling (related answer).
what matters actual urls users, among them search engines, see. don’t have access backend, don’t know how site works interally.
the line disallow: /tags/
(no need other 1 *
) means urls paths start /tags/
should not crawled. so, assuming robots.txt @ http://example.com/robots.txt
, block example:
http://example.com/tags/
http://example.com/tags/foo
http://example.com/tags/foo/bar
if tags available under different url (for example, drupal’s default /taxonomy/term/…
), , bot finds these alternative urls, may of course crawl them. it’s idea redirect 1 canonical url want use.
Comments
Post a Comment