(from github.com/PakanAngel)
I cannot find a clear regexp to exclude indexing files from a whole domain
e.g.:
URL: http://example.com
includde URls: http://example.com/.*
Exludeed URls for crawling: http://example.com/./.jpg
(from github.com/marevol)
(?i).*(css|js|jpeg|jpg|gif|png|bmp|wmv|exe|mp4)
For php, it contains request parameters.
.*php\?.*
(from github.com/PakanAngel)
Thnx a lot @marevol . Another question is how can I exclude an entire URL which contains a specific word? Words like category or tag
e.g.:
http://www.example.org/category/65354/discussion
What would be the correct regexp to exclude category from the above URL?