(from github.com/ppq200)
I am trying to use fess-gitbucket-plugin to search gitbucket repositories.
But crawling takes 60 sec ~ per each page.
What would be a good way to speed up?
CentOS6
GitBucket : 4.17.0
fess-plugin : 1.2.0
FESS：11.3.1
Thanks.
$ tailf /var/log/fess/fess-crawler.log
2017-10-12 18:09:45,524 [main] INFO Starting Crawler..
2017-10-12 18:09:45,524 [main] DEBUG infoMap: CrawlerStartTime=2017-10-12T18:09:45.524+0900 => {}
2017-10-12 18:09:45,537 [DataStoreCrawler] DEBUG infoMap: DataCrawlStartTime=2017-10-12T18:09:45.537+0900 => {CrawlerStartTime=2017-10-12T18:09:45.524+0900}
2017-10-12 18:09:45,988 [AV6c0FRdj3Hc5iErPdUs-1] INFO There exist 4 repositories
2017-10-12 18:10:11,751 [AV6c0FRdj3Hc5iErPdUs-1] INFO Crawl root/private-root
2017-10-12 18:10:39,053 [AV6c0FRdj3Hc5iErPdUs-1] INFO Get a content from http://xxx.xxx.xxx:8100/gitbucket/api/v3/repos/root/private-root/contents/README.md
2017-10-12 18:10:39,067 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Initializing org.codelibs.fess.crawler.client.http.HcHttpClient
2017-10-12 18:10:39,094 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Accessing http://xxx.xxx.xxx:8100/gitbucket/api/v3/repos/root/private-root/contents/README.md?ref=ceff6b0344bdd5cea910aaa070b4fbb8fde8f914&large_file=true
2017-10-12 18:10:39,104 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG CookieSpec selected: default
2017-10-12 18:10:39,117 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Connection request: [route: {}->http://xxx.xxx.xxx:8100][total kept alive: 0; route allocated: 0 of 20; total allocated: 0 of 200]
2017-10-12 18:10:39,127 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Connection leased: [id: 0][route: {}->http://xxx.xxx.xxx:8100][total kept alive: 0; route allocated: 1 of 20; total allocated: 1 of 200]
2017-10-12 18:10:39,128 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Opening connection {}->http://xxx.xxx.xxx:8100
2017-10-12 18:10:39,129 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Connecting to /xxx.xxx.xxx:8100
2017-10-12 18:10:39,130 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Connection established xxx.xxx.xxx:54418<->xxx.xxx.xxx:8100
2017-10-12 18:10:39,130 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Executing request GET /gitbucket/api/v3/repos/root/private-root/contents/README.md?ref=ceff6b0344bdd5cea910aaa070b4fbb8fde8f914&large_file=true HTTP/1.1
2017-10-12 18:10:39,130 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Proxy auth state: UNCHALLENGED
2017-10-12 18:10:39,131 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> GET /gitbucket/api/v3/repos/root/private-root/contents/README.md?ref=ceff6b0344bdd5cea910aaa070b4fbb8fde8f914&large_file=true HTTP/1.1
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> Authorization: token 3050e2d55556aedb28601e80a0250ea86972da0c
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> Accept: application/vnd.github.v3.raw
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> Host: xxx.xxx.xxx:8100
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> Connection: Keep-Alive
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> User-Agent: Crawler
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> Accept-Encoding: gzip,deflate
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> "GET /gitbucket/api/v3/repos/root/private-root/contents/README.md?ref=ceff6b0344bdd5cea910aaa070b4fbb8fde8f914&large_file=true HTTP/1.1[\r][\n]"
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> "Authorization: token 3050e2d55556aedb28601e80a0250ea86972da0c[\r][\n]"
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> "Accept: application/vnd.github.v3.raw[\r][\n]"
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> "Host: xxx.xxx.xxx:8100[\r][\n]"
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> "User-Agent: Crawler[\r][\n]"
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2017-10-12 18:10:39,132 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 >> "[\r][\n]"
2017-10-12 18:10:44,078 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:10:44,079 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:10:49,080 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:10:49,080 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:10:54,080 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:10:54,080 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:10:59,081 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:10:59,081 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:04,081 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:04,081 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:09,082 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:09,082 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:14,082 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:14,083 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:19,083 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:19,083 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:24,084 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:24,084 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:29,084 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:29,084 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:34,085 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:34,085 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:39,086 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:39,086 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:44,086 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:44,086 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:49,087 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:49,087 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:54,087 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:54,087 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:11:59,088 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:11:59,088 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:04,088 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:04,088 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:09,089 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:09,089 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:14,089 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:14,090 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:19,090 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:19,090 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:24,090 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:24,091 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:29,091 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:29,091 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:34,092 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:34,092 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:39,092 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:39,092 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:44,093 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:44,093 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:49,093 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:49,093 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:54,094 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:54,094 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:12:59,094 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:12:59,095 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:13:04,095 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2017-10-12 18:13:04,095 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2017-10-12 18:13:08,066 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << "HTTP/1.1 200 [\r][\n]"
2017-10-12 18:13:08,066 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << "Set-Cookie: JSESSIONID=715A31552F38055DC6D4FA6CAD762663; Path=/gitbucket; HttpOnly[\r][\n]"
2017-10-12 18:13:08,066 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << "Content-Type: application/vnd.github.v3.raw;charset=UTF-8[\r][\n]"
2017-10-12 18:13:08,066 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << "Content-Length: 94[\r][\n]"
2017-10-12 18:13:08,066 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << "Date: Thu, 12 Oct 2017 09:13:08 GMT[\r][\n]"
2017-10-12 18:13:08,066 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << "[\r][\n]"
2017-10-12 18:13:08,066 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << "private-root[\n]"
2017-10-12 18:13:08,066 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << "===============[\n]"
2017-10-12 18:13:08,066 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << "[\n]"
2017-10-12 18:13:08,066 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << "private [0xe3][0x81][0xaa] repository[0xe3][0x81][0xae][0xe3][0x82][0xaf][0xe3][0x83][0xad][0xe3][0x83][0xbc][0xe3][0x83][0xab][0xe3][0x83][0x86][0xe3][0x82][0xb9][0xe3][0x83][0x88][0xe7][0x94][0xa8][0xe3][0x83][0xaa][0xe3][0x83][0x9d][0xe3][0x82][0xb8][0xe3][0x83][0x88][0xe3][0x83][0xaa]"
2017-10-12 18:13:08,068 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << HTTP/1.1 200
2017-10-12 18:13:08,068 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << Set-Cookie: JSESSIONID=715A31552F38055DC6D4FA6CAD762663; Path=/gitbucket; HttpOnly
2017-10-12 18:13:08,069 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << Content-Type: application/vnd.github.v3.raw;charset=UTF-8
2017-10-12 18:13:08,069 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << Content-Length: 94
2017-10-12 18:13:08,069 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG http-outgoing-0 << Date: Thu, 12 Oct 2017 09:13:08 GMT
2017-10-12 18:13:08,072 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Connection can be kept alive indefinitely
2017-10-12 18:13:08,076 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Cookie accepted [JSESSIONID="715A31552F38055DC6D4FA6CAD762663", version:0, domain:xxx.xxx.xxx, path:/gitbucket, expiry:null]
2017-10-12 18:13:08,079 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Connection [id: 0][route: {}->http://xxx.xxx.xxx:8100] can be kept alive indefinitely
2017-10-12 18:13:08,080 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Connection released: [id: 0][route: {}->http://xxx.xxx.xxx:8100][total kept alive: 1; route allocated: 1 of 20; total allocated: 1 of 200]
2017-10-12 18:13:08,108 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG metadata: Content-Encoding=UTF-8 resourceName=README.md?ref=ceff6b0344bdd5cea910aaa070b4fbb8fde8f914&large_file=true Content-Type=application/vnd.github.v3.raw
2017-10-12 18:13:08,152 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG type: text/x-web-markdown, metadata: Content-Encoding=UTF-8 resourceName=README.md?ref=ceff6b0344bdd5cea910aaa070b4fbb8fde8f914&large_file=true Content-Type=text/x-web-markdown , maxCompressionRatio: 2, maxUncompressionSize: 10000000
2017-10-12 18:13:08,204 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Result: metadata: X-Parsed-By=org.apache.tika.parser.DefaultParser X-Parsed-By=org.apache.tika.parser.txt.TXTParser Content-Encoding=UTF-8 resourceName=README.md?ref=ceff6b0344bdd5cea910aaa070b4fbb8fde8f914&large_file=true Content-Type=text/x-web-markdown; charset=UTF-8
2017-10-12 18:13:08,204 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG ExtractData: ExtractData [metadata={X-Parsed-By=[Ljava.lang.String;@13fa1bd7, Content-Encoding=[Ljava.lang.String;@7c1737e5, resourceName=[Ljava.lang.String;@7b86330, Content-Type=[Ljava.lang.String;@d45bcf7}, content=private-root ========== test]
2017-10-12 18:13:08,231 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Adding {filetype=others, expires=Sun Oct 15 18:09:45 JST 2017, thumbnail=http://xxx.xxx.xxx:8100/gitbucket/api/v3/repos/root/private-root/contents/README.md?ref=ceff6b0344bdd5cea910aaa070b4fbb8fde8f914&large_file=true, role=[1root], created=Thu Oct 12 18:13:08 JST 2017, indexingTarget=true, label=[gitbucket_source], title=README.md, content=private-root ========== test, url=http://xxx.xxx.xxx:8100/gitbucket/root/private-root/blob/ceff6b0344bdd5cea910aaa070b4fbb8fde8f914/README.md, site=xxx.xxx.xxx:8100/gitbucket/api/v3/repos/root/pri..., filename=README.md, config_id=DAV6c0FRdj3Hc5iErPdUs, segment=AV6c0FRdj3Hc5iErPdUs, anchor=, digest=private-root ========== test, host=xxx.xxx.xxx:8100, boost=1.0, mimetype=application/vnd.github.v3.raw, _id=http:%2F%2Fxxx.xxx.xxx:8100%2Fgitbucket%2Fapi%2Fv3%2Frepos%2Froot%2Fprivate-root%2Fcontents%2FREADME.md?ref=ceff6b0344bdd5cea910aaa070b4fbb8fde8f914&large_file=true;role=Rguest, content_length=94, timestamp=Thu Oct 12 18:13:08 JST 2017, virtual_host=[Ljava.lang.String;@2ad382f2}
2017-10-12 18:13:08,239 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Click Count: 0, url: http://xxx.xxx.xxx:8100/gitbucket/root/private-root/blob/ceff6b0344bdd5cea910aaa070b4fbb8fde8f914/README.md
2017-10-12 18:13:08,243 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Favorite Count: 0, url: http://xxx.xxx.xxx:8100/gitbucket/root/private-root/blob/ceff6b0344bdd5cea910aaa070b4fbb8fde8f914/README.md
2017-10-12 18:13:08,243 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG Added the document. The number of a document cache is 1.
2017-10-12 18:13:08,244 [AV6c0FRdj3Hc5iErPdUs-1] DEBUG The number of an added document is 1.
2017-10-12 18:13:08,244 [AV6c0FRdj3Hc5iErPdUs-1] INFO Crawl issues in root/private-root
2017-10-12 18:13:08,244 [AV6c0FRdj3Hc5iErPdUs-1] INFO Crawl Wiki in root/private-root
2017-10-12 18:13:08,293 [AV6c0FRdj3Hc5iErPdUs-1] INFO Get a content from http://xxx.xxx.xxx:8100/gitbucket/api/v3/fess/root/private-root/wiki/contents/Home.md