File Crawler error : No response from Job.

Nicolas_H · October 16, 2024, 4:42pm

Hello,

since i’m running version 14.16 i’m facing below error message :

seems like the error is always at 00:00 :

The strange thing is that the crawler is still working, so it’s clearly a false positive.
I checked the log and nothing.

Do you have any idea what the problem might be?

Thank you.
Best regards.

shinsuke · October 17, 2024, 12:00am

Please check the fess-crawler.log and fess.log files.

Nicolas_H · October 17, 2024, 10:26am

Hello,

here is what i have in the fess-crawler.log at the time of the error :

2024-10-15 23:59:53,738 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/AK0W2348/
2024-10-15 23:59:53,797 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AX0W6040/
2024-10-15 23:59:53,896 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AX0V8841/
2024-10-15 23:59:53,926 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/BXEX1784/
2024-10-15 23:59:54,035 [Crawler-20241014182154-1-1] INFO  Cpu Load 81% is greater than 80%. 1 waiting thread(s).
2024-10-15 23:59:54,172 [Crawler-20241014182154-1-3] INFO  Cpu Load 81% is greater than 80%. 2 waiting thread(s).
2024-10-15 23:59:54,188 [Crawler-20241014182154-1-6] INFO  Cpu Load 81% is greater than 80%. 3 waiting thread(s).
2024-10-15 23:59:54,277 [Crawler-20241014182154-1-5] INFO  Cpu Load 81% is greater than 80%. 4 waiting thread(s).
2024-10-15 23:59:54,394 [Crawler-20241014182154-1-4] INFO  Cpu Load 81% is greater than 80%. 5 waiting thread(s).
2024-10-15 23:59:54,489 [Crawler-20241014182154-1-2] INFO  Cpu Load 81% is greater than 80%. 6 waiting thread(s).
2024-10-15 23:59:55,290 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AX0V5244/
2024-10-15 23:59:55,426 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/AX0W2347/
2024-10-15 23:59:55,440 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AX0W2214/
2024-10-15 23:59:55,530 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AX0V6575/
2024-10-15 23:59:55,647 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/AX0V6442/
2024-10-15 23:59:55,727 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/BX0V9009/
2024-10-15 23:59:55,741 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AX0V9070/
2024-10-15 23:59:55,793 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AX0W4743/
2024-10-15 23:59:56,148 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AX0W1016/
2024-10-15 23:59:56,346 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/AX0V5379/
2024-10-15 23:59:56,389 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/AX0V6443/
2024-10-15 23:59:56,389 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AX0W0916/
2024-10-15 23:59:56,389 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AX0V3817/
2024-10-15 23:59:56,389 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/GX0Y0788/
2024-10-15 23:59:56,596 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/BX0W2400/
2024-10-15 23:59:56,633 [IndexUpdater] INFO  Processing no docs in indexing queue (Doc:{access 3ms, cleanup 82ms}, Mem:{used 11.012GB, heap 16.001GB, max 20.48GB})
2024-10-15 23:59:56,820 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/BX0W5038/
2024-10-15 23:59:56,883 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AX0W6176/
2024-10-15 23:59:56,997 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AX0W8576/
2024-10-15 23:59:57,021 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/AX0W5938/
2024-10-15 23:59:57,045 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AX0W9637/
2024-10-15 23:59:57,063 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/BX0X3578/
2024-10-15 23:59:57,280 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AX0X4323/
2024-10-15 23:59:57,325 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AX0X3258/
2024-10-15 23:59:57,476 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AX0W8568/
2024-10-15 23:59:57,591 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AX0X3127/
2024-10-15 23:59:57,943 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/AK0X4325/
2024-10-15 23:59:57,989 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/AX0W5809/
2024-10-15 23:59:58,041 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AXEW7966/
2024-10-15 23:59:58,141 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AX0V5120/
2024-10-15 23:59:58,185 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AK0V8854/
2024-10-15 23:59:58,274 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/AX0V4054/
2024-10-15 23:59:58,286 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AX0X4329/
2024-10-15 23:59:58,401 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AX0V6450/
2024-10-15 23:59:58,686 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AX0X0880/
2024-10-15 23:59:58,760 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AKEV4520/
2024-10-15 23:59:58,760 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/AX0W9507/
2024-10-15 23:59:58,766 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AX0V3952/
2024-10-15 23:59:58,808 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/AX0V4183/
2024-10-15 23:59:58,878 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/BX0W2790/
2024-10-15 23:59:59,252 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AK0V9920/
2024-10-15 23:59:59,310 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/YXEW6010/
2024-10-15 23:59:59,353 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AK0V6328/
2024-10-15 23:59:59,447 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AX0V6448/
2024-10-15 23:59:59,462 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/BKEV7822/
2024-10-15 23:59:59,631 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AX0V8978/
2024-10-15 23:59:59,693 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/AX0W0077/
2024-10-15 23:59:59,747 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AX0V9075/
2024-10-15 23:59:59,760 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AXEW1603/
2024-10-15 23:59:59,781 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AX0V5112/
2024-10-15 23:59:59,942 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/AX0W4875/
2024-10-16 00:00:00,034 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/AX0V3947/
2024-10-16 00:00:00,073 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AR0V7816/
2024-10-16 00:00:00,177 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AX0W2211/
2024-10-16 00:00:00,275 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AX0W0917/
2024-10-16 00:00:00,373 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AK0W9632/
2024-10-16 00:00:00,844 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/AR0W2774/
2024-10-16 00:00:00,883 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AX0W5945/
2024-10-16 00:00:01,025 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AX0W8574/
2024-10-16 00:00:01,111 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AX0W6044/
2024-10-16 00:00:01,115 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/AKEV9546/
2024-10-16 00:00:01,461 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AX0W7110/
2024-10-16 00:00:01,543 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/BX0V9392/
2024-10-16 00:00:01,630 [IndexUpdater] INFO  Cpu Load 96% is greater than 80%. 1 waiting thread(s).
2024-10-16 00:00:01,641 [Crawler-20241014182154-1-5] INFO  Cpu Load 96% is greater than 80%. 2 waiting thread(s).
2024-10-16 00:00:01,679 [Crawler-20241014182154-1-1] INFO  Cpu Load 96% is greater than 80%. 3 waiting thread(s).
2024-10-16 00:00:01,696 [Crawler-20241014182154-1-2] INFO  Cpu Load 96% is greater than 80%. 4 waiting thread(s).
2024-10-16 00:00:01,722 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/BX0X1173/
2024-10-16 00:00:01,863 [Crawler-20241014182154-1-6] INFO  Cpu Load 96% is greater than 80%. 5 waiting thread(s).
2024-10-16 00:00:01,944 [Crawler-20241014182154-1-4] INFO  Cpu Load 96% is greater than 80%. 6 waiting thread(s).
2024-10-16 00:00:02,121 [Crawler-20241014182154-1-3] INFO  Cpu Load 96% is greater than 80%. 7 waiting thread(s).
2024-10-16 00:00:02,631 [IndexUpdater] INFO  Cpu Load 81% is greater than 80%. 7 waiting thread(s).
2024-10-16 00:00:02,642 [Crawler-20241014182154-1-5] INFO  Cpu Load 81% is greater than 80%. 7 waiting thread(s).
2024-10-16 00:00:02,680 [Crawler-20241014182154-1-1] INFO  Cpu Load 81% is greater than 80%. 7 waiting thread(s).
2024-10-16 00:00:02,697 [Crawler-20241014182154-1-2] INFO  Cpu Load 81% is greater than 80%. 7 waiting thread(s).
2024-10-16 00:00:02,864 [Crawler-20241014182154-1-6] INFO  Cpu Load 81% is greater than 80%. 7 waiting thread(s).
2024-10-16 00:00:02,945 [Crawler-20241014182154-1-4] INFO  Cpu Load 81% is greater than 80%. 7 waiting thread(s).
2024-10-16 00:00:03,124 [Crawler-20241014182154-1-3] INFO  Cpu Load 81% is greater than 80%. 7 waiting thread(s).
2024-10-16 00:00:03,633 [IndexUpdater] INFO  6 waiting thread(s).
2024-10-16 00:00:03,892 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AX0W1017/
2024-10-16 00:00:03,931 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AX0W8567/
2024-10-16 00:00:03,947 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AX0X1952/
2024-10-16 00:00:04,114 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AX0X3125/
2024-10-16 00:00:04,196 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/AX0X0887/
2024-10-16 00:00:04,376 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/AK010003/
2024-10-16 00:00:04,400 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AK0V7651/
2024-10-16 00:00:04,644 [IndexUpdater] INFO  Processing no docs in indexing queue (Doc:{access 8ms, cleanup 82ms}, Mem:{used 11.102GB, heap 16.001GB, max 20.48GB})
2024-10-16 00:00:04,713 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AX0W7237/
2024-10-16 00:00:04,862 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/AK0V4059/
2024-10-16 00:00:05,106 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AX0X4458/
2024-10-16 00:00:05,142 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AK0V7658/
2024-10-16 00:00:05,159 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/AK0V7789/
2024-10-16 00:00:05,224 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/BX0W2660/
2024-10-16 00:00:05,287 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/AK0V7655/
2024-10-16 00:00:05,288 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/BK0W4932/
2024-10-16 00:00:05,494 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/BX0V5443/
2024-10-16 00:00:05,527 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/AK0X4587/
2024-10-16 00:00:05,605 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AK0V8852/
2024-10-16 00:00:05,804 [Crawler-20241014182154-1-4] INFO  Crawling URL: file:/E:/private/BX0V5571/
2024-10-16 00:00:05,870 [Crawler-20241014182154-1-2] INFO  Crawling URL: file:/E:/private/AK0V7525/
2024-10-16 00:00:05,958 [Crawler-20241014182154-1-6] INFO  Cpu Load 86% is greater than 80%. 1 waiting thread(s).
2024-10-16 00:00:05,966 [Crawler-20241014182154-1-3] INFO  Cpu Load 86% is greater than 80%. 2 waiting thread(s).
2024-10-16 00:00:05,974 [Crawler-20241014182154-1-1] INFO  Cpu Load 86% is greater than 80%. 3 waiting thread(s).
2024-10-16 00:00:06,062 [Crawler-20241014182154-1-5] INFO  Crawling URL: file:/E:/private/AK0W4603/
2024-10-16 00:00:06,083 [Crawler-20241014182154-1-4] INFO  Cpu Load 86% is greater than 80%. 4 waiting thread(s).
2024-10-16 00:00:06,141 [Crawler-20241014182154-1-2] INFO  Cpu Load 86% is greater than 80%. 5 waiting thread(s).
2024-10-16 00:00:06,337 [Crawler-20241014182154-1-5] INFO  Cpu Load 86% is greater than 80%. 6 waiting thread(s).
2024-10-16 00:00:07,210 [Crawler-20241014182154-1-6] INFO  Crawling URL: file:/E:/private/AKEW0677/
2024-10-16 00:00:07,217 [Crawler-20241014182154-1-3] INFO  Crawling URL: file:/E:/private/AX0W4872/
2024-10-16 00:00:07,227 [Crawler-20241014182154-1-1] INFO  Crawling URL: file:/E:/private/PX0W1712/

There is no error and it keeps going without any error (it is still running now)

And the Fess.log file is empty.

Thank you.
Best regards.

shinsuke · October 17, 2024, 11:28am

The directory file:/E:/private appears to contain too many folders, causing Fess to be unable to find the files, eventually leading to a crawler timeout. To avoid this issue, you can either adjust your crawling configuration or modify the timeout value in the fess_config.properties file.

indexer.webfs.max.empty.list.count=3600

Nicolas_H · October 24, 2024, 10:11am

Ok thank you for your feedback, i’m giving it a try right now and will give you some feedback soon.

Thank you.
Regards.

Nicolas_H · November 19, 2024, 12:37pm

Hello,

I’m sorry for my late reply, but I think I’ve finally found the problem and it’s not directly related to Fess.
The server I’m running Fess on reboots every Sunday night and it seems to cause a problem if the crawler is running during the reboot, causing it to continue after the reboot but Fess to see it as in ‘error’ with the message “No Response from Job”

I’ve changed the server configuration to make sure this doesn’t happen again.
I’ll let you know when I can confirm this.

Thank you.
Best regards.