For web crawlers, is there a way to exclude certain mimetypes from being crawled/indexed? For instance, mimetype:"application/rss+xml"
I can try to exclude it through the use of certain URL patterns under Excluded URLs (like RSS or MCRSS) but it is not comprehensive.
For web crawlers, is there a way to exclude certain mimetypes from being crawled/indexed? For instance, mimetype:"application/rss+xml"
I can try to exclude it through the use of certain URL patterns under Excluded URLs (like RSS or MCRSS) but it is not comprehensive.