A few months ago I received a manual action penalty from Google as they detected spam pages on our domain. The problem was that when people were searching on our site they are directed to a page with the following:
https://$domain/search?query=$QUERY
Some users (most likely bots) are generating huge spam searches on our search page and somehow Google is indexing these and there are no inbound links to these pages (at least I cannot find any).
To resolve this I did the following:
* On our search page I set the following header: X-Robots-Tag: noindex (based off of the documentation here https://developers.google.com/search/reference/robots_meta_tag).
* Submitted URLs to be dropped from Google Index via Webmaster console
* Submitted 3 reconsideration requests to Google to avoid the penalties
In theory this should stop all search pages being indexed (as they all contain the noindex header) and it has helped drop the number of indexed pages marked as spam by 99% however we still have a significant number of urls marked as spam and so our site has a penalty from Google.
Has anyone had this issue before? How can I stop these pages becoming indexed when I have the noindex header set _and_ if you search the spam urls there are no inbound links to them?
Any help appreciated folks!