・環境
CentOS 7
Java11
ElasticSearch 7.7.1
FESS 13.7.1
fess-ds-gsuite 13.7.0
・現象
GoogleDriveに入っている1000件超の経歴書データ(pdf)をクローリングしたところ、高い頻度でElasticSearchへのエラーが発生する
※成功するときもあります
※ElasticSearchは問題なく動作しているので、FESS側の原因かと思われます
※関連するか分かりませんが、直前でSSLのWARNが発生しています
・影響
クロールが途中で止まってしまい、最後までクロールが完了しません。
また、FESSの管理画面上で、「システムエラー」が表示されてしまいます。
→ fessを再起動すると復旧します
・ElasticSearchに関するエラーログ(fess-crawler.log )
※fess.log にも同様のエラーが記録されています
org.codelibs.curl.CurlException: Failed to access to http://localhost:9200/fess_log.click_log/_search?typed_keys=true&batched_reduce_size=512&max_concurrent_shard_requests=5&ccs_minimize_roundtrips=true
at org.codelibs.curl.CurlRequest.lambda$connect$3(CurlRequest.java:201) ~[curl4j-1.2.4.jar:?]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) ~[?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[?:?]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?]
at java.net.Socket.connect(Socket.java:609) ~[?:?]
at java.net.Socket.connect(Socket.java:558) ~[?:?]
at sun.net.NetworkClient.doConnect(NetworkClient.java:182) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[?:?]
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:?]
at sun.net.www.http.HttpClient.New(HttpClient.java:341) ~[?:?]
at sun.net.www.http.HttpClient.New(HttpClient.java:362) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1248) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1362) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1337) ~[?:?]
at org.codelibs.curl.CurlRequest.lambda$connect$3(CurlRequest.java:193) ~[curl4j-1.2.4.jar:?]
... 6 more
・SSLに関するWARNログ(fess-crawler.log )
2020-06-17 15:36:10,639 [pool-6-thread-1] WARN Failed to get contents: 9120b47d-5999-4668-9fe7-7cb48a7a5be6
org.codelibs.fess.crawler.exception.CrawlingAccessException: Failed to create an input stream from 1e6HOJbkNV-KCrpMF1m61xhHpgEtmUI4V
at org.codelibs.fess.ds.gsuite.GSuiteClient.getFileInputStream(GSuiteClient.java:208) ~[fess-ds-gsuite-13.7.0.jar:?]
at org.codelibs.fess.ds.gsuite.GoogleDriveDataStore.getFileContents(GoogleDriveDataStore.java:505) [fess-ds-gsuite-13.7.0.jar:?]
at org.codelibs.fess.ds.gsuite.GoogleDriveDataStore.processFile(GoogleDriveDataStore.java:277) [fess-ds-gsuite-13.7.0.jar:?]
at org.codelibs.fess.ds.gsuite.GoogleDriveDataStore.lambda$storeFiles$2(GoogleDriveDataStore.java:220) [fess-ds-gsuite-13.7.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1313) ~[?:?]
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1055) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168) ~[?:?]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:148) ~[google-http-client-1.34.2.jar:1.34.2]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[google-http-client-1.34.2.jar:1.34.2]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.34.2.jar:1.34.2]
at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:244) ~[fess-ds-gsuite-13.7.0.jar:?]
at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:198) ~[fess-ds-gsuite-13.7.0.jar:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:562) ~[fess-ds-gsuite-13.7.0.jar:?]
at com.google.api.services.drive.Drive$Files$Get.executeMediaAndDownloadTo(Drive.java:3256) ~[fess-ds-gsuite-13.7.0.jar:?]
at org.codelibs.fess.ds.gsuite.GSuiteClient.getFileInputStream(GSuiteClient.java:199) ~[fess-ds-gsuite-13.7.0.jar:?]
... 6 more
Suppressed: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:?]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) ~[?:?]
at java.net.SocketOutputStream.write(SocketOutputStream.java:150) ~[?:?]
at sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:357) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:269) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:398) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168) ~[?:?]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:148) ~[google-http-client-1.34.2.jar:1.34.2]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[google-http-client-1.34.2.jar:1.34.2]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.34.2.jar:1.34.2]
at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:244) ~[fess-ds-gsuite-13.7.0.jar:?]
at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:198) ~[fess-ds-gsuite-13.7.0.jar:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:562) ~[fess-ds-gsuite-13.7.0.jar:?]
at com.google.api.services.drive.Drive$Files$Get.executeMediaAndDownloadTo(Drive.java:3256) ~[fess-ds-gsuite-13.7.0.jar:?]
at org.codelibs.fess.ds.gsuite.GSuiteClient.getFileInputStream(GSuiteClient.java:199) ~[fess-ds-gsuite-13.7.0.jar:?]
at org.codelibs.fess.ds.gsuite.GoogleDriveDataStore.getFileContents(GoogleDriveDataStore.java:505) [fess-ds-gsuite-13.7.0.jar:?]
at org.codelibs.fess.ds.gsuite.GoogleDriveDataStore.processFile(GoogleDriveDataStore.java:277) [fess-ds-gsuite-13.7.0.jar:?]
at org.codelibs.fess.ds.gsuite.GoogleDriveDataStore.lambda$storeFiles$2(GoogleDriveDataStore.java:220) [fess-ds-gsuite-13.7.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:108) ~[?:?]
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1144) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1055) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:168) ~[?:?]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:148) ~[google-http-client-1.34.2.jar:1.34.2]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[google-http-client-1.34.2.jar:1.34.2]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.34.2.jar:1.34.2]
at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:244) ~[fess-ds-gsuite-13.7.0.jar:?]
at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:198) ~[fess-ds-gsuite-13.7.0.jar:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:562) ~[fess-ds-gsuite-13.7.0.jar:?]
at com.google.api.services.drive.Drive$Files$Get.executeMediaAndDownloadTo(Drive.java:3256) ~[fess-ds-gsuite-13.7.0.jar:?]
at org.codelibs.fess.ds.gsuite.GSuiteClient.getFileInputStream(GSuiteClient.java:199) ~[fess-ds-gsuite-13.7.0.jar:?]
... 6 more