(from github.com/ECJansen)
I noticed that the web crawlers (regardless of the number of threads) process the URLs in the queue quickly and then wait 10 seconds before processing the next set of URLs. Looking through the code I noticed in IntervalControlHelper.checkCrawlerStatus that when the crawler is not running, it waits for crawlerWaitMillis, which is set to 10 seconds. Is this possibly causing the delay for some reason ?
We’re not having this delay issue with smb, only https. Any idea what could be causing this ?
Thanks,
Eric
(from github.com/marevol)
If there are a lot of documents in the queue, Fess stops a crawler and processes them in the queue.
The setting is in fess_config.properties:
indexer.unprocessed.document.size=1000