Comments on: Ultimate WordPress Robots.txt for Silo SEO

By: corsolinguaromena

corsolinguaromena — Fri, 03 Feb 2012 10:13:36 +0000

very useful article indeed. I have a question : you sad that the access of the search engine at uploads folder is not possible and we should add a new folder and Allow: /images, this means I have to move all my uploads content to images folder and the modify all my content of the site for the new hyperlinks to the images? or is a easer way to avoid this?

By: Sal

Sal — Fri, 20 Jan 2012 10:43:28 +0000

What a great article, thank you!

When my blog gets indexed by Google, it indexes the post itself (which I want) but ti also indexes the blog home page, which I probably don't want. I've got the XML-sitemap generator plugin installed and I see there's an option for "sitemap content" and home page is checked. Is this something I should be un-check to have the home page delisted in Google? Many thanks!

By: awan

awan — Fri, 13 Jan 2012 10:56:33 +0000

weheheh nice tips my bro, ..
but when we are USE robots meta , are we could not to USE robots.txt?

By: Floris

Floris — Thu, 05 Jan 2012 09:19:09 +0000

Very useful info. I used quite a bit of it. I think by default wordpress also creates feeds with urls like
/rss
/rss2
/atom
/rdf

I therefore recommend adding rules like:
Disallow: /rss
Disallow: /*/rss
Disallow: /rss2
Disallow: /*/rss2
Disallow: /atom
Disallow: /*/atom
Disallow: /rdf
Disallow: /*/rdf

By: Magazine

Magazine — Tue, 03 Jan 2012 10:31:57 +0000

I agree with Bogan , but one thing in this article im wonder about.
1.Why to create new storage dir for images like /images when you didnt block whole wp-content dir (images are stored by default in wp-content/uploads )?
2. Some of SER does not reach .gz sitemap so what if you want to serve both xml.gz and .xml sitemap ?

By: Donovan

Donovan — Sun, 01 Jan 2012 16:10:00 +0000

I was looking around for robot txt information and discovered this site. Thanks for the info.

By: un dijeta

un dijeta — Sat, 24 Dec 2011 10:25:04 +0000

Wonderful website, really informative and engaging.

By: Vezbe

Vezbe — Wed, 21 Dec 2011 11:33:55 +0000

I always allow indexing of your cgi-bin because i am so lazy to learn something more about wordpress

By: La Chatroulette

La Chatroulette — Sun, 11 Dec 2011 15:47:49 +0000

Thanks for your fantastic guide. I just created robots.txt for my new blog

User-agent: *

Disallow: /cgi-bin

Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/

Allow: /wp-content/uploads/

Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /author
Disallow: /archives
Disallow: /iframes
Disallow: /tag/

Disallow: /*?*
Disallow: /*?
http://www.mydomain.com/sitemap.xml

What do you think

By: Banja Srbija

Banja Srbija — Sun, 11 Dec 2011 12:54:37 +0000

Excellent post. I want to thank you for this informative read. I will bookmark this site and visit again.. Banje u Srbiji Banje Srbije