Magento URL's Rejected by Google Merchants for "Too Many Redirects"

Google product feeds denied after installing SUPEE-8788?
Google product feeds denied after installing SUPEE-8788?

Symptom: Your Magento site's nightly product feed to Google starts getting denied products by the boatload. The reason, Google says, is "Too Many Redirects" but ths URL's they are using seem to load just fine. Perhaps you are running Varnish? Maybe you recently installed security patch SUPEE-8788 too?

If you go to http://www.redirect-checker.org/ and set the User-Agent to GoogleBot the URL's result in a 302 "Temporarily Moved" error, but other common settings such as Yahoo, Bing! and Yandex work just fine?

If you are running the Varnish extension by Nexcess, try going into its configuration and check the "Crawler User Agents" to see if it includes any RegEx pattern that matches "googlebot" and if it does, remove it. This setting tries to serve Google the fastest page it can, which is a normally-good thing, but I've seen that after installing the 8788 patch (version two) a new conflict was introduced that is causing irreconcilable redirects.

I haven't traced down what's happening under the hood yet, but by removing GoogleBot from this field the Google Merchants feed recovered instantly! I suppose it would be good to get the fastest pages possible serving to Google again soon, but faced with a sharp decline in sales I figured this trade-off was worth it to get back in good graces in the Google product feeds as fast as possible!

By the way, this doesn't just impact Google product feeds - I suspect these pages are getting trashed in the Google index itself! As long as the user agent is Googlebot, the page is effectively not rendering at all. Protect your SEO, and test your product URL's with the redirect checker!

Good luck, I hope this tip works for you too, and check back here later in case I trace this problem out any further...

Tags: