Crawling basic authentication sites

English:Fess
1

(from github.com/kotesh)
I am trying to crawl a basic authentication site. I configured “Web Authentication” option for this site but still the site is not getting crawled. As per crawl logs, the site is redirecting to login page and only indexing the login page. I tried configuring Request Header with Basic Authentication credentials, but still the behavior is same.
Seems crawler is not able to bypass login page using Request Header or Web Authentication. I tested this using CURL command and able to bypass the login page.

Can you please help debugging this issue further.

2

(from github.com/marevol)
Basic authentication does not have a login page.
If a login page is displayed, it’s Form authentication.
Therefore I think the target site might support both Basic and Form authentication.

To change a log level for crawler, modify Default Crawler job at Scheduler page:

return container.getComponent("crawlJob").logLevel("debug").execute(executor);

and then check logs/fess-crawler.log.

3

(from github.com/kotesh)
Thanks. I am able to find the issue and now it is crawling fine.