I’m experiencing a Google / robots.txt issue of some kind. According to my Google Search Console, my site (shellsharks.com) is crawlable and indexable, but it is blocked because of my robots.txt file. When inspecting the robots.txt file WITH the Google search console it says everything is fine and I should not be blocked. Up until recently, I had a “noindex” meta tag in my HTML markup but have since deleted that line so that shouldnt be an issue any more. Anyone have any ideas why Google still believes my site is blocked by robots.txt? Is this just a wait until Google can organically crawl my site again after having deleted the noindex meta tag?
Appreciate any help! Right now nothing on my site is findable it seems via Google which is kinda a bummer =/.
@dwsmart Hey! So the new robots.txt file seems to have made my site crawlable+indexable but for some reason I can’t get Google to just crawl / index my entire site. Even after seemingly successfully submitting my site map via the search console tool. Any ideas on why things might be failing? Thanks in advance!
@shellsharks there can be many reasons, the main one perhaps being Google just doesn't tend to index everything on a site, & is definitely less eager to do so than in the past
Couple of things:
use the URL inspection tool in search console (1st step, not live test), what status do they have?
Do the pages that aren't indexed have good internal links for other pages on your site?
Do the pages that aren't getting index offer content that might be useful to searchers?
Which kinda makes sense I think, though my site map does include both these post types (e.g. “notes” and “logs”). Looks like in the site map they are “discovered” but not crawled.
Both of these post types have good linkage and I think would be considered “useful” fwiw. Thanks!
Add comment