Crawling a large number of SMB files will slow down when fess is running in docker container

English:Fess
1

(from github.com/sutra806)
I am trying to crawl many smb files in Fess 13.2 in docker container.
After crawling about 1000 or more files, crawl speed becomes very slow.

When “The number of thread” is large or “Interval time” is short in the crawl setting, it is often occurs.

I crawl with the log level set to debug, the following log is output to fess-crawler.log, and then the crawler thread blocks for about 1 to 3 minutes.

  • DEBUG Closing FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
2

(from github.com/marevol)
Did you check CPU&Memory resource on crawling?
You need to set proper values for “The number of thread” and “Interval time”.

3

(from github.com/sutra806)
Yes, I’ve not restricted cpu and memory resource on docker, and not changed JVM option of fess and crawler from the default settings.
I’ve tried to crawl in same setting on Windows zip distribution of fess, it’s not slows down.

4

(from github.com/sutra806)
I’ve tried the rpm version of fess 13.2(without docker), the problem does not resolved. Do you have any solutions?

5

(from github.com/marevol)
Could you provide the debug logs?

6

(from github.com/sutra806)
fess-crawler.log
Here is an extracted crawl log file (some information is masked.)

7

(from github.com/marevol)
Is it reproduced if replacing smb:// with smb1://?

8

(from github.com/sutra806)
It is not reproduced!

9

(from github.com/sutra806)
Could I get any workaround to crawl SMB2 files?

10

(from github.com/sutra806)
Did you have any plans to improve this issues?

11

(from github.com/marevol)
For smb2 feature, we merge it from jcifs-ng to our jcifs.
So, if it’s fixed in jcifs-ng, it will be fixed in Fess.

12

(from github.com/sutra806)
It makes me happy by upgrading jcifs to 2.1.13. Thanks!