Crawling & No Index

Hi There,
Why am i getting “blocked by noindex” on a site crawl for I don’t have meta info that is NoFollow or NoIndex

check your robots.txt file. looks like it has an error.
Screen Shot 2022-02-11 at 3.56.27 PM

Looks like in your robots.txt file it should be something like this
User-agent: RavenCrawler
Allow: /

Also your pages do look like they are blocked with a meta tag. If this is a custom module check your module Item properties setting and make sure “Show New Items for Search Engines” is checked. Also check Settings / SEO and make sure this module isn’t in the “Exclude Module List”.

You will probable have to edit all the items in this module to make sure each one is checked “Show this page for search engines” under the SEO tab. You can quickly fix this with an export/import.

1 Like