(from github.com/clauded)
Is there a way to have a detailed log of crawling : I’m trying to index a site and tried http://www.mysite.org/.* as well as http://www.mysite.org.* as indexing url but nothing gets indexed (it’s a sharepoint site with no robots.txt).
Note. the actual log file does not help.