(from github.com/charles-pinkston)
I’m running Fess with the default configuration. It crawls my site fine, but after 3 days, when I attempt to do a search, there are no results. When I check the Elasticsearch index curl http://localhost:9200/fess.search/_count it’s reporting 0 documents. To work around this, have to disable the Doc Purger Scheduled job.
I’m trying to figure out why this is happening. I suspect it has something to do with the Crawler config for incremental searches (Admin > System > General > Check Last Modifed). But when I disable that check, it fails to crawl my site at all.
(from github.com/charles-pinkston)
The issue I’m having is that it doesn’t re-index any documents - so it removes everything, and doesn’t keep valid documents. I’ve tried disabling the Check Last Modified, but then it doesn’t index any documents.
I’m trying to figure out how to remove deleted pages from my index, but it seems like it wipes out everything.
(from github.com/marevol)
Incremental crawling(enabling check last modified) updates expires field. So, if you run crawling with the incremental crawling everyday, valid documents will not removed.
(from github.com/charles-pinkston)
It appears as though the problem is that the crawler hasn’t completed crawling my site within that 3 day period, so records are not being updated. I’ll close this question and open a different one regarding crawl speed.