クロール後、fessが起動しない

HAHA · June 8, 2023, 9:31am

・windows
・elasticsearch-8.6.0
・fess-14.6.1

数万ドキュメントを夜間にクロールした後、
翌日fessを起動させるとPC挙動が重くなりブラウザからの接続が出来なくなりました。

経緯として、翌日にPCにログインしたところwindows自体にログインできず（elasticsearch,
fessはwindowsサービスとして登録）、PC再起動後にelasticsearchを起動し、fessを起動して上記事象に直面しました。

以下の対応を行いましたが改善無しでした。
■indexが破損していることを想定し、elasticsearch/dataのバックアップを取得後にフォルダ削除
→elasticsearchが問題なく起動してそう。fessを起動→事象改善無し
■fessを再インストール→事象改善無し

毎回ではありませんが、起動後以下のエラーメッセージ（fess.log）が出力されました。
elasticsearch.logも添付いたします。（字数制限のため返信の形で記載しております。）

2023-06-08 12:41:01,256 [main] ERROR Failed to initialize Lasta Di.
org.lastaflute.di.exception.ContainerInitFailureException: Look! Read the message below.
/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
Failed to initialize the container.

[Path]
esclient.xml

[Namespace]
null

[Included by]
esflute_config.xml
esflute_user.xml
esflute_log.xml

*/
at org.lastaflute.di.core.meta.impl.LaContainerImpl.throwContainerInitFailureException(LaContainerImpl.java:463) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:415) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:431) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:431) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:431) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.factory.SingletonLaContainerFactory.init(SingletonLaContainerFactory.java:65) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.web.container.WebLastaContainerInitializer.doInitContainer(WebLastaContainerInitializer.java:81) ~[lastaflute-1.2.4.jar:?]
at org.lastaflute.web.container.WebLastaContainerInitializer.initialize(WebLastaContainerInitializer.java:45) ~[lastaflute-1.2.4.jar:?]
at org.lastaflute.web.servlet.filter.LastaPrepareFilter.initializeContainer(LastaPrepareFilter.java:137) ~[lastaflute-1.2.4.jar:?]
at org.lastaflute.web.servlet.filter.LastaPrepareFilter.init(LastaPrepareFilter.java:94) ~[lastaflute-1.2.4.jar:?]
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:272) ~[classes/:?]
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:253) ~[classes/:?]
at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:102) ~[classes/:?]
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4609) ~[classes/:?]
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5248) ~[classes/:?]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[classes/:?]
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393) ~[classes/:?]
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383) ~[classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[classes/:?]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145) ~[?:?]
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916) ~[classes/:?]
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835) ~[classes/:?]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[classes/:?]
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393) ~[classes/:?]
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383) ~[classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[classes/:?]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145) ~[?:?]
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916) ~[classes/:?]
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265) ~[classes/:?]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[classes/:?]
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:430) ~[classes/:?]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[classes/:?]
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) ~[classes/:?]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[classes/:?]
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:486) ~[classes/:?]
at org.dbflute.tomcat.TomcatBoot.startServer(TomcatBoot.java:713) ~[classes/:?]
at org.dbflute.tomcat.TomcatBoot.go(TomcatBoot.java:402) ~[classes/:?]
at org.dbflute.tomcat.TomcatBoot.bootAwait(TomcatBoot.java:347) ~[classes/:?]
at org.codelibs.fess.FessBoot.main(FessBoot.java:111) ~[classes/:?]
Caused by: org.lastaflute.di.core.exception.IllegalMethodRuntimeException: [ESSR0060]Can not configure method(open) of org.codelibs.fess.es.client.SearchEngineClient, because [fess.20230608] IndexNotFoundException[no such index [fess.20230608]]
at org.lastaflute.di.core.assembler.AbstractMethodAssembler.invoke(AbstractMethodAssembler.java:61) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.assembler.DefaultInitMethodAssembler.assemble(DefaultInitMethodAssembler.java:40) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.assemble(SingletonComponentDeployer.java:57) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.deploy(SingletonComponentDeployer.java:41) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.init(SingletonComponentDeployer.java:35) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.meta.impl.ComponentDefImpl.init(ComponentDefImpl.java:85) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:434) ~[lasta-di-0.9.1.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.9.1.jar:?]
… 42 more
Caused by: org.opensearch.index.IndexNotFoundException: no such index [fess.20230608]
at org.codelibs.fesen.client.action.HttpGetMappingsAction.lambda$execute$0(HttpGetMappingsAction.java:48) ~[fesen-httpclient-2.0.0.jar:?]
at org.codelibs.curl.CurlRequest.lambda$execute$5(CurlRequest.java:260) ~[curl4j-1.2.7.jar:?]
at org.codelibs.curl.CurlRequest.lambda$connect$4(CurlRequest.java:231) ~[curl4j-1.2.7.jar:?]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395) ~[?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) ~[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) ~[?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) ~[?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) ~[?:?]

elasticsearchの設定が原因過去例を参考に確認しましたが、特に設定ミスはなさそうでした。

長文失礼いたしましたが、発生している事象としては以下の通りです。
・クロール後に起動できなくなる。
・再度起動させると、PC自体の挙動が重くなる。（PCスペックは推奨値以上）
・fessを再インストールしても事象解消しない。

対応方法が不明のため、お忙しいところ申し訳ございませんが、ご教示いただけますと幸いです。
ログ等情報に不足あればご指摘いただきたく存じます。

HAHA · June 8, 2023, 9:32am

elasticsearch.logは以下の通りです。

[2023-06-08T12:33:50,059][INFO ][o.e.n.Node ] [DEV18-136156] version[8.6.0], pid[3608], build[zip/f67ef2df40237445caa70e2fef79471cc608d70d/2023-01-04T09:35:21.782467981Z], OS[Windows Server 2019/10.0/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.1/19.0.1+10-21]
[2023-06-08T12:33:50,075][INFO ][o.e.n.Node ] [DEV18-136156] JVM home [C:\elasticsearch-8.6.0\jdk], using bundled JDK [true]
[2023-06-08T12:33:50,075][INFO ][o.e.n.Node ] [DEV18-136156] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=C:\Users\user\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.distribution.type=zip, --module-path=C:\elasticsearch-8.6.0\lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
[2023-06-08T12:33:53,589][INFO ][c.a.c.i.j.JacksonVersion ] [DEV18-136156] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: Troubleshoot dependency version conflicts when using the Azure SDK for Java | Microsoft Learn
[2023-06-08T12:33:55,886][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [aggregations]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [aggs-matrix-stats]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [analysis-common]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [apm]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [configsync]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [constant-keyword]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [data-streams]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [frozen-indices]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [ingest-attachment]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [ingest-common]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [ingest-geoip]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [ingest-user-agent]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [kibana]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [lang-expression]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [lang-mustache]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [lang-painless]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [legacy-geo]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [mapper-extras]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [mapper-version]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [old-lucene-versions]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [parent-join]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [percolator]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [rank-eval]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [reindex]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repositories-metering-api]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repository-azure]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repository-encrypted]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repository-gcs]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repository-s3]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repository-url]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [runtime-fields-common]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [search-business-rules]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [searchable-snapshots]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [snapshot-based-recoveries]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [snapshot-repo-test-kit]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [spatial]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [transform]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [transport-netty4]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [unsigned-long]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [vector-tile]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [wildcard]
[2023-06-08T12:33:55,902][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-aggregate-metric]
[2023-06-08T12:33:55,917][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-analytics]
[2023-06-08T12:33:55,917][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-async]
[2023-06-08T12:33:55,917][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-async-search]
[2023-06-08T12:33:55,917][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-autoscaling]
[2023-06-08T12:33:55,917][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-ccr]
[2023-06-08T12:33:55,917][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-core]
[2023-06-08T12:33:55,917][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-deprecation]
[2023-06-08T12:33:55,917][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-enrich]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-eql]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-fleet]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-graph]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-identity-provider]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-ilm]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-logstash]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-ml]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-monitoring]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-ql]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-rollup]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-security]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-shutdown]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-sql]
[2023-06-08T12:33:55,933][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-stack]
[2023-06-08T12:33:55,948][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-text-structure]
[2023-06-08T12:33:55,948][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-voting-only-node]
[2023-06-08T12:33:55,948][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-watcher]
[2023-06-08T12:33:55,948][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-write-load-forecaster]
[2023-06-08T12:33:55,948][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded plugin [analysis-extension]
[2023-06-08T12:33:55,948][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded plugin [analysis-fess]
[2023-06-08T12:33:55,948][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded plugin [minhash]
[2023-06-08T12:33:59,213][WARN ][stderr ] [DEV18-136156] 6 08, 2023 12:33:59 午後 org.apache.lucene.store.MMapDirectory lookupProvider
[2023-06-08T12:33:59,213][WARN ][stderr ] [DEV18-136156] 警告: You are running with Java 19. To make full use of MMapDirectory, please pass ‘–enable-preview’ to the Java command line.
[2023-06-08T12:33:59,229][INFO ][o.e.e.NodeEnvironment ] [DEV18-136156] using [1] data paths, mounts [[(C:)]], net usable_space [16.7tb], net total_space [16.7tb], types [NTFS]
[2023-06-08T12:33:59,229][INFO ][o.e.e.NodeEnvironment ] [DEV18-136156] heap size [31gb], compressed ordinary object pointers [true]
[2023-06-08T12:33:59,245][INFO ][o.e.n.Node ] [DEV18-136156] node name [DEV18-136156], node ID [D6ykXz7mQnCrpwcMK9ZbvQ], cluster name [elasticsearch], roles [ingest, data_cold, data, remote_cluster_client, master, data_warm, data_content, transform, data_hot, ml, data_frozen]
[2023-06-08T12:34:02,197][INFO ][o.e.x.s.Security ] [DEV18-136156] Security is disabled
[2023-06-08T12:34:02,385][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [DEV18-136156] [controller/13284] [Main.cc@123] controller (64 bit): Version 8.6.0 (Build 85eb78d83cdd94) Copyright (c) 2022 Elasticsearch BV
[2023-06-08T12:34:03,103][INFO ][o.e.t.n.NettyAllocator ] [DEV18-136156] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
[2023-06-08T12:34:03,134][INFO ][o.e.i.r.RecoverySettings ] [DEV18-136156] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-06-08T12:34:03,181][INFO ][o.e.d.DiscoveryModule ] [DEV18-136156] using discovery type [multi-node] and seed hosts providers [settings]
[2023-06-08T12:34:04,501][INFO ][o.e.n.Node ] [DEV18-136156] initialized
[2023-06-08T12:34:04,501][INFO ][o.e.n.Node ] [DEV18-136156] starting …
[2023-06-08T12:34:04,517][INFO ][o.e.x.s.c.f.PersistentCache] [DEV18-136156] persistent cache index loaded
[2023-06-08T12:34:04,517][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [DEV18-136156] deprecation component started
[2023-06-08T12:34:04,767][INFO ][o.e.t.TransportService ] [DEV18-136156] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2023-06-08T12:34:04,892][WARN ][o.e.b.BootstrapChecks ] [DEV18-136156] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2023-06-08T12:34:04,892][INFO ][o.e.c.c.ClusterBootstrapService] [DEV18-136156] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to []
[2023-06-08T12:34:04,907][INFO ][o.e.c.c.ClusterBootstrapService] [DEV18-136156] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2023-06-08T12:34:07,935][INFO ][o.e.c.c.Coordinator ] [DEV18-136156] setting initial configuration to VotingConfiguration{D6ykXz7mQnCrpwcMK9ZbvQ}
[2023-06-08T12:34:08,099][INFO ][o.e.c.s.MasterService ] [DEV18-136156] elected-as-master ([1] nodes joined)[FINISH_ELECTION, {DEV18-136156}{D6ykXz7mQnCrpwcMK9ZbvQ}{D2c3IkR-SwODmNnICOv0tA}{DEV18-136156}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw} completing election], term: 1, version: 1, delta: master node changed {previous [], current [{DEV18-136156}{D6ykXz7mQnCrpwcMK9ZbvQ}{D2c3IkR-SwODmNnICOv0tA}{DEV18-136156}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}
[2023-06-08T12:34:08,160][INFO ][o.e.c.c.CoordinationState] [DEV18-136156] cluster UUID set to [rnj_KHCRRruMEeFWrDr5qA]
[2023-06-08T12:34:08,223][INFO ][o.e.c.s.ClusterApplierService] [DEV18-136156] master node changed {previous [], current [{DEV18-136156}{D6ykXz7mQnCrpwcMK9ZbvQ}{D2c3IkR-SwODmNnICOv0tA}{DEV18-136156}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2023-06-08T12:34:08,254][INFO ][o.e.r.s.FileSettingsService] [DEV18-136156] starting file settings watcher …
[2023-06-08T12:34:08,254][INFO ][o.e.r.s.FileSettingsService] [DEV18-136156] file settings service up and running [tid=163]
[2023-06-08T12:34:08,301][INFO ][o.e.h.AbstractHttpServerTransport] [DEV18-136156] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2023-06-08T12:34:08,302][INFO ][o.e.n.Node ] [DEV18-136156] started {DEV18-136156}{D6ykXz7mQnCrpwcMK9ZbvQ}{D2c3IkR-SwODmNnICOv0tA}{DEV18-136156}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{ml.allocated_processors=56, ml.machine_memory=410842562560, xpack.installed=true, ml.allocated_processors_double=56.0, ml.max_jvm_size=33285996544}
[2023-06-08T12:34:08,317][INFO ][o.e.g.GatewayService ] [DEV18-136156] recovered [0] indices into cluster_state
[2023-06-08T12:34:08,582][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.watch-history-16] for index patterns [.watcher-history-16*]
[2023-06-08T12:34:08,645][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding component template [.deprecation-indexing-mappings]
[2023-06-08T12:34:08,707][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding component template [.deprecation-indexing-settings]
[2023-06-08T12:34:08,770][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.slm-history] for index patterns [.slm-history-5*]
[2023-06-08T12:34:08,817][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [ilm-history] for index patterns [ilm-history-5*]
[2023-06-08T12:34:08,879][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding component template [logs-mappings]
[2023-06-08T12:34:08,926][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding component template [synthetics-settings]
[2023-06-08T12:34:08,957][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding component template [metrics-settings]
[2023-06-08T12:34:09,004][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding component template [logs-settings]
[2023-06-08T12:34:09,051][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding component template [metrics-mappings]
[2023-06-08T12:34:09,098][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding component template [data-streams-mappings]
[2023-06-08T12:34:09,145][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding component template [synthetics-mappings]
[2023-06-08T12:34:09,192][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.ml-state] for index patterns [.ml-state*]
[2023-06-08T12:34:09,238][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.ml-notifications-000002] for index patterns [.ml-notifications-000002]
[2023-06-08T12:34:09,285][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.ml-stats] for index patterns [.ml-stats-]
[2023-06-08T12:34:09,348][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.monitoring-kibana-mb] for index patterns [.monitoring-kibana-8-]
[2023-06-08T12:34:09,395][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.monitoring-logstash-mb] for index patterns [.monitoring-logstash-8-]
[2023-06-08T12:34:09,457][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.monitoring-ent-search-mb] for index patterns [.monitoring-ent-search-8-]
[2023-06-08T12:34:09,520][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.ml-anomalies-] for index patterns [.ml-anomalies-]
[2023-06-08T12:34:09,551][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]
[2023-06-08T12:34:09,598][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-]
[2023-06-08T12:34:09,645][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-]
[2023-06-08T12:34:09,691][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-]
[2023-06-08T12:34:09,754][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding template [.monitoring-es] for index patterns [.monitoring-es-7-]
[2023-06-08T12:34:09,816][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.monitoring-beats-mb] for index patterns [.monitoring-beats-8-]
[2023-06-08T12:34:09,879][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.monitoring-es-mb] for index patterns [.monitoring-es-8-]
[2023-06-08T12:34:09,973][INFO ][o.e.c.m.MetadataCreateIndexService] [DEV18-136156] [configsync] creating index, cause [api], templates [], shards [1]/[0]
[2023-06-08T12:34:10,066][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [.deprecation-indexing-template] for index patterns [.logs-deprecation.]
[2023-06-08T12:34:10,113][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [synthetics] for index patterns [synthetics--]
[2023-06-08T12:34:10,160][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [logs] for index patterns [logs--]
[2023-06-08T12:34:10,207][INFO ][o.e.c.m.MetadataIndexTemplateService] [DEV18-136156] adding index template [metrics] for index patterns [metrics--]
[2023-06-08T12:34:10,458][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [180-days-default]
[2023-06-08T12:34:10,522][INFO ][o.e.c.r.a.AllocationService] [DEV18-136156] current.health=“GREEN” message=“Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[configsync][0]]]).” previous.health=“YELLOW” reason=“shards started [[configsync][0]]”
[2023-06-08T12:34:10,569][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [synthetics]
[2023-06-08T12:34:10,569][INFO ][o.c.e.c.s.ConfigSyncService] [DEV18-136156] ConfigFileUpdater is started at 1m intervals.
[2023-06-08T12:34:10,600][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [metrics]
[2023-06-08T12:34:10,631][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [365-days-default]
[2023-06-08T12:34:10,662][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [ilm-history-ilm-policy]
[2023-06-08T12:34:10,709][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [30-days-default]
[2023-06-08T12:34:10,741][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [.deprecation-indexing-ilm-policy]
[2023-06-08T12:34:10,772][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [90-days-default]
[2023-06-08T12:34:10,803][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [logs]
[2023-06-08T12:34:10,834][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [slm-history-ilm-policy]
[2023-06-08T12:34:10,865][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [7-days-default]
[2023-06-08T12:34:10,912][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [watch-history-ilm-policy-16]
[2023-06-08T12:34:10,959][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [.monitoring-8-ilm-policy]
[2023-06-08T12:34:11,006][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [ml-size-based-ilm-policy]
[2023-06-08T12:34:11,069][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [DEV18-136156] adding index lifecycle policy [.fleet-actions-results-ilm-policy]
[2023-06-08T12:34:11,178][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [DEV18-136156] Node [{DEV18-136156}{D6ykXz7mQnCrpwcMK9ZbvQ}] is selected as the current health node.
[2023-06-08T12:34:11,287][INFO ][o.e.l.LicenseService ] [DEV18-136156] license [5784bb49-823a-4b3f-8dba-1423c7cd5123] mode [basic] - valid
[2023-06-08T12:34:22,519][ERROR][o.e.i.g.GeoIpDownloader ] [DEV18-136156] exception during geoip databases update
java.net.UnknownHostException: geoip.elastic.co
at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:560) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
at java.net.Socket.connect(Socket.java:666) ~[?:?]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304) ~[?:?]
at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:531) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:636) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:378) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1241) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1127) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1661) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1585) ~[?:?]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:36) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:155) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:143) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:274) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:102) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:48) ~[?:?]
at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) ~[elasticsearch-8.6.0.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917) ~[elasticsearch-8.6.0.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.6.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
[2023-06-08T12:36:10,690][INFO ][o.c.e.c.s.ConfigSyncService] [DEV18-136156] Updated c:\elasticsearch-8.6.0\data\config\gl\protwords.txt …同様のログが続くため割愛

shinsuke · June 8, 2023, 10:55pm

Windowsで謎の破損等が起きる場合としては、ウィルスチェックなどにより、Elasticsearchのインデックスが壊されて起動しなくなるなどがあるので、ウィルスチェックの対象外にするなどが必要になります。

HAHA · June 9, 2023, 9:17am

ご回答いただきありがとうございます。

ウイルスチェックを見直し、以下を実施しました。
①elasticsearch/fessフォルダの削除
②elasticsearch/fessの再インストール・設定
→事象改善無し、FESS起動時に重くなってしまう。

タスクマネージャでリソースを見てみると、どうもelasticsearchで使用しているjava.exeでメモリ30GBを使用していました。（メモリは余剰に積んでおり30GBの使用でもelasticsearch起動じはPC挙動には問題なし。fess起動のみ重くブラウザに表示されない）

もしかしたら何かしらのデータが残り続けているのでしょうか。
この辺の知見が乏しく、お手数ですが、ご教示いただけますと幸いです。

shinsuke · June 10, 2023, 1:29am

-Xms31744m, -Xmx31744m

ログ上で、上記のようになっているので、30Gくらいのヒープを指定されているようです。
Elasticsearch/OpenSearchの利用で、メモリー割当を間違っている人がよくいるのですが、物理メモリーが多くあるからといって、Javaのヒープを割り当ててはダメで、Elasticsearch/OpenSearchはOSのファイルキャッシュとかも使うので、Javaのヒープ外のメモリー量を結構空けておく必要があります。なので、たぶん、メモリー割当がおかしいから問題が起きているようにも思います。

商用サポートでは要件に基づいて、最適なメモリー割当を考えて対応していますが、多くの方に参考になるように、要件を気にせず、ベースライン的なメモリー割当の話を書いておくと、たとえば、Fess&OpenSearchを1台に載せると考えると、

物理メモリーが4Gのとき
・Fess(Web)：デフォルト(512m)
・Fess(Crawler)：デフォルト(512m)
・OpenSearch：1〜2G

物理メモリーが8Gのとき
・Fess(Web)：デフォルト(512m)
・Fess(Crawler)：デフォルト(512m)
・OpenSearch：1〜2G(4G以上は割り当てない)

物理メモリーが16Gのとき
・Fess(Web)：デフォルト(512m)
・Fess(Crawler)：デフォルト(512m)
・OpenSearch：2〜4G(8G以上は割り当てない)
・物理メモリーは少なくても8Gくらいは空けておく

物理メモリーが32Gのとき
・Fess(Web)：デフォルト(512m)
・Fess(Crawler)：デフォルト(512m)
・OpenSearch：4G(16G以上は割り当てない)
・物理メモリーは少なくても16Gくらいは空けておく

みたいな感じで考えて、
・激しいクロールが必要な場合は、Fess(Crawler)の割当を増やす
・検索対象数が多い場合には、OpenSearchの割当を増やす
・Fess(Web)を変更する必要はほぼない
という感じです。他にも考慮点はいろいろとあるとは思いますが…。
要件によって変わるので、上記が常に正解になるわけではありませんが、メモリー起因のトラブルは回避できると思います。
あとは、要件次第で考える、みたいな感じです。

HAHA · June 12, 2023, 10:20am

何度も、ご回答いただきありがとうございます。
ご指摘内容を参考に修正しましたが、fess起動後PCが重くなり、ブラウザで表示されない事象の改善が見られませんでした。。。

メモリ自体は100GB以上あるため、不足はない想定です。
一応、ご指摘いただいた内容を基に以下の修正を行いました。
・Fess(Web)：デフォルト(512m)→デフォルトのまま設定変更しない
・Fess(Crawler)：デフォルト(512m)→デフォルトのまま設定変更しない
・elasticsearch：4G(16G以上は割り当てない)
→以下の設定を実施。
<elasticsearch/config/jvm.options>
JVM heap size
-Xms4g ←コメントアウト外す
-Xmx4g ←コメントアウト外す

お手数をお掛けし申し訳ございませんが、
何か他考えうる事象があれば、ご教示いただきたく存じます。

以下、ログを添付いたします。
fess.log→表示なし
elasticsearch.log

[2023-06-12T19:00:49,110][INFO ][o.e.n.Node ] [DEV18-136156] version[8.6.0], pid[11824], build[zip/f67ef2df40237445caa70e2fef79471cc608d70d/2023-01-04T09:35:21.782467981Z], OS[Windows Server 2019/10.0/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.1/19.0.1+10-21]
[2023-06-12T19:00:49,126][INFO ][o.e.n.Node ] [DEV18-136156] JVM home [C:\elasticsearch-8.6.0\jdk], using bundled JDK [true]
[2023-06-12T19:00:49,126][INFO ][o.e.n.Node ] [DEV18-136156] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Xms4g, -Xmx4g, -XX:+UseG1GC, -Djava.io.tmpdir=C:\Users\takaihrk\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=2147483648, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=zip, --module-path=C:\elasticsearch-8.6.0\lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
[2023-06-12T19:00:55,891][INFO ][c.a.c.i.j.JacksonVersion ] [DEV18-136156] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: Troubleshoot dependency version conflicts when you use the Azure SDK for Java - Java on Azure | Microsoft Learn
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [aggregations]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [aggs-matrix-stats]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [analysis-common]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [apm]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [configsync]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [constant-keyword]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [data-streams]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [frozen-indices]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [ingest-attachment]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [ingest-common]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [ingest-geoip]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [ingest-user-agent]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [kibana]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [lang-expression]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [lang-mustache]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [lang-painless]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [legacy-geo]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [mapper-extras]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [mapper-version]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [old-lucene-versions]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [parent-join]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [percolator]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [rank-eval]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [reindex]
[2023-06-12T19:01:01,970][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repositories-metering-api]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repository-azure]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repository-encrypted]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repository-gcs]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repository-s3]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [repository-url]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [runtime-fields-common]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [search-business-rules]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [searchable-snapshots]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [snapshot-based-recoveries]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [snapshot-repo-test-kit]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [spatial]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [transform]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [transport-netty4]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [unsigned-long]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [vector-tile]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [wildcard]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-aggregate-metric]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-analytics]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-async]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-async-search]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-autoscaling]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-ccr]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-core]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-deprecation]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-enrich]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-eql]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-fleet]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-graph]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-identity-provider]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-ilm]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-logstash]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-ml]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-monitoring]
[2023-06-12T19:01:01,986][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-ql]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-rollup]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-security]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-shutdown]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-sql]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-stack]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-text-structure]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-voting-only-node]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-watcher]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded module [x-pack-write-load-forecaster]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded plugin [analysis-extension]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded plugin [analysis-fess]
[2023-06-12T19:01:02,001][INFO ][o.e.p.PluginsService ] [DEV18-136156] loaded plugin [minhash]
[2023-06-12T19:01:05,061][WARN ][stderr ] [DEV18-136156] 6 12, 2023 7:01:05 午後 org.apache.lucene.store.MMapDirectory lookupProvider
[2023-06-12T19:01:05,061][WARN ][stderr ] [DEV18-136156] 警告: You are running with Java 19. To make full use of MMapDirectory, please pass ‘–enable-preview’ to the Java command line.
[2023-06-12T19:01:05,076][INFO ][o.e.e.NodeEnvironment ] [DEV18-136156] using [1] data paths, mounts [[(C:)]], net usable_space [16.7tb], net total_space [16.7tb], types [NTFS]
[2023-06-12T19:01:05,076][INFO ][o.e.e.NodeEnvironment ] [DEV18-136156] heap size [4gb], compressed ordinary object pointers [true]
[2023-06-12T19:01:05,732][INFO ][o.e.n.Node ] [DEV18-136156] node name [DEV18-136156], node ID [mVspMPZ4QFOaRGv_aq132w], cluster name [elasticsearch], roles [ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold]
[2023-06-12T19:01:09,084][INFO ][o.e.x.s.Security ] [DEV18-136156] Security is disabled
[2023-06-12T19:01:09,272][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [DEV18-136156] [controller/12056] [Main.cc@123] controller (64 bit): Version 8.6.0 (Build 85eb78d83cdd94) Copyright (c) 2022 Elasticsearch BV
[2023-06-12T19:01:10,053][INFO ][o.e.t.n.NettyAllocator ] [DEV18-136156] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2023-06-12T19:01:10,084][INFO ][o.e.i.r.RecoverySettings ] [DEV18-136156] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2023-06-12T19:01:10,131][INFO ][o.e.d.DiscoveryModule ] [DEV18-136156] using discovery type [multi-node] and seed hosts providers [settings]
[2023-06-12T19:01:11,423][INFO ][o.e.n.Node ] [DEV18-136156] initialized
[2023-06-12T19:01:11,423][INFO ][o.e.n.Node ] [DEV18-136156] starting …
[2023-06-12T19:01:11,485][INFO ][o.e.x.s.c.f.PersistentCache] [DEV18-136156] persistent cache index loaded
[2023-06-12T19:01:11,485][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [DEV18-136156] deprecation component started
[2023-06-12T19:01:11,735][INFO ][o.e.t.TransportService ] [DEV18-136156] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2023-06-12T19:01:12,397][WARN ][o.e.b.BootstrapChecks ] [DEV18-136156] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2023-06-12T19:01:12,397][INFO ][o.e.c.c.ClusterBootstrapService] [DEV18-136156] this node is locked into cluster UUID [v3t_v5IIThqjJXtQ4VExVg] and will not attempt further cluster bootstrapping
[2023-06-12T19:01:12,442][INFO ][o.e.c.c.ClusterBootstrapService] [DEV18-136156] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2023-06-12T19:01:12,565][INFO ][o.e.c.s.MasterService ] [DEV18-136156] elected-as-master ([1] nodes joined)[FINISH_ELECTION, {DEV18-136156}{mVspMPZ4QFOaRGv_aq132w}{olO3vsA3QuepVTRmwyqSUA}{DEV18-136156}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw} completing election], term: 5, version: 219, delta: master node changed {previous , current [{DEV18-136156}{mVspMPZ4QFOaRGv_aq132w}{olO3vsA3QuepVTRmwyqSUA}{DEV18-136156}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}
[2023-06-12T19:01:12,672][INFO ][o.e.c.s.ClusterApplierService] [DEV18-136156] master node changed {previous , current [{DEV18-136156}{mVspMPZ4QFOaRGv_aq132w}{olO3vsA3QuepVTRmwyqSUA}{DEV18-136156}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}, term: 5, version: 219, reason: Publication{term=5, version=219}
[2023-06-12T19:01:12,719][INFO ][o.e.r.s.FileSettingsService] [DEV18-136156] starting file settings watcher …
[2023-06-12T19:01:12,719][INFO ][o.e.r.s.FileSettingsService] [DEV18-136156] file settings service up and running [tid=148]
[2023-06-12T19:01:12,750][INFO ][o.e.h.AbstractHttpServerTransport] [DEV18-136156] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2023-06-12T19:01:12,750][INFO ][o.e.n.Node ] [DEV18-136156] started {DEV18-136156}{mVspMPZ4QFOaRGv_aq132w}{olO3vsA3QuepVTRmwyqSUA}{DEV18-136156}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{ml.max_jvm_size=4294967296, ml.allocated_processors_double=56.0, xpack.installed=true, ml.machine_memory=410842562560, ml.allocated_processors=56}
[2023-06-12T19:01:12,832][INFO ][o.e.l.LicenseService ] [DEV18-136156] license [25a250cd-0cb3-4ded-8f9b-032e06d147c5] mode [basic] - valid
[2023-06-12T19:01:12,847][INFO ][o.e.g.GatewayService ] [DEV18-136156] recovered [27] indices into cluster_state
[2023-06-12T19:01:13,048][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [DEV18-136156] Node [{DEV18-136156}{mVspMPZ4QFOaRGv_aq132w}] is selected as the current health node.
[2023-06-12T19:01:17,154][INFO ][o.e.c.r.a.AllocationService] [DEV18-136156] current.health=“GREEN” message=“Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[configsync][0]]]).” previous.health=“RED” reason=“shards started [[configsync][0]]”
[2023-06-12T19:01:17,201][INFO ][o.c.e.c.s.ConfigSyncService] [DEV18-136156] ConfigFileUpdater is started at 1m intervals.
[2023-06-12T19:01:25,066][ERROR][o.e.i.g.GeoIpDownloader ] [DEV18-136156] exception during geoip databases update
java.net.UnknownHostException: geoip.elastic.co
at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:560) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
at java.net.Socket.connect(Socket.java:666) ~[?:?]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304) ~[?:?]
at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:531) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:636) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:378) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1241) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1127) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1661) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1585) ~[?:?]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:36) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:155) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:143) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:274) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:102) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:48) ~[?:?]
at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) ~[elasticsearch-8.6.0.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917) ~[elasticsearch-8.6.0.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.6.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
[2023-06-12T19:06:41,310][INFO ][o.e.c.m.MetadataCreateIndexService] [DEV18-136156] [fess_user.role] creating index, cause [api], templates , shards [1]/[0]
[2023-06-12T19:06:41,653][INFO ][o.e.c.r.a.AllocationService] [DEV18-136156] current.health=“GREEN” message=“Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[fess_user.role][0]]]).” previous.health=“YELLOW” reason=“shards started [[fess_user.role][0]]”
[2023-06-12T19:07:05,840][INFO ][o.e.c.m.MetadataMappingService] [DEV18-136156] [fess_user.role/0tCYy88OQIaYBQXIT4tk8A] update_mapping [_doc]
[2023-06-12T19:07:22,618][INFO ][o.e.c.m.MetadataCreateIndexService] [DEV18-136156] [fess_user.group] creating index, cause [api], templates , shards [1]/[0]
[2023-06-12T19:07:22,977][INFO ][o.e.c.r.a.AllocationService] [DEV18-136156] current.health=“GREEN” message=“Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[fess_user.group][0]]]).” previous.health=“YELLOW” reason=“shards started [[fess_user.group][0]]”
[2023-06-12T19:07:54,311][INFO ][o.e.c.m.MetadataMappingService] [DEV18-136156] [fess_user.group/IuRyJmN3RuuyayxFnWMvuA] update_mapping [_doc]
[2023-06-12T19:08:20,415][INFO ][o.e.c.m.MetadataCreateIndexService] [DEV18-136156] [fess_log.click_log] creating index, cause [api], templates , shards [5]/[0]
[2023-06-12T19:08:21,214][INFO ][o.e.c.r.a.AllocationService] [DEV18-136156] current.health=“GREEN” message=“Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[fess_log.click_log][4]]]).” previous.health=“YELLOW” reason=“shards started [[fess_log.click_log][4]]”
[2023-06-12T19:08:52,008][INFO ][o.e.c.m.MetadataMappingService] [DEV18-136156] [fess_log.click_log/ukKDyASzQDarUe0Iv8w8sQ] update_mapping [_doc]
[2023-06-12T19:09:22,542][INFO ][o.e.c.m.MetadataCreateIndexService] [DEV18-136156] [fess_log.favorite_log] creating index, cause [api], templates , shards [5]/[0]
[2023-06-12T19:09:23,443][INFO ][o.e.c.r.a.AllocationService] [DEV18-136156] current.health=“GREEN” message=“Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[fess_log.favorite_log][4]]]).” previous.health=“YELLOW” reason=“shards started [[fess_log.favorite_log][4]]”
[2023-06-12T19:09:40,767][INFO ][o.e.c.m.MetadataMappingService] [DEV18-136156] [fess_log.favorite_log/YLdSS29cRIKYMLrg7_nZIA] update_mapping [_doc]
[2023-06-12T19:11:23,521][INFO ][o.e.b.Elasticsearch ] [DEV18-136156] running graceful exit on windows
[2023-06-12T19:11:23,521][INFO ][o.e.n.Node ] [DEV18-136156] stopping …
[2023-06-12T19:11:23,521][INFO ][o.e.r.s.FileSettingsService] [DEV18-136156] shutting down watcher thread
[2023-06-12T19:11:23,521][INFO ][o.e.r.s.FileSettingsService] [DEV18-136156] watcher service stopped
[2023-06-12T19:11:23,521][INFO ][o.e.x.w.WatcherService ] [DEV18-136156] stopping watch service, reason [shutdown initiated]
[2023-06-12T19:11:23,521][INFO ][o.e.x.w.WatcherLifeCycleService] [DEV18-136156] watcher has stopped and shutdown
[2023-06-12T19:11:24,021][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [DEV18-136156] [controller/12056] [Main.cc@176] ML controller exiting
[2023-06-12T19:11:24,021][INFO ][o.e.x.m.p.NativeController] [DEV18-136156] Native controller process has stopped - no new native processes can be started
[2023-06-12T19:11:24,177][INFO ][o.e.n.Node ] [DEV18-136156] stopped
[2023-06-12T19:11:24,177][INFO ][o.e.n.Node ] [DEV18-136156] closing …
[2023-06-12T19:11:24,197][INFO ][o.e.n.Node ] [DEV18-136156] closed

shinsuke · June 12, 2023, 1:33pm

Windowsでそのような問題に遭遇していないのでわかりませんが、そのログだけ見ると、

[2023-06-12T19:11:23,521][INFO ][o.e.b.Elasticsearch ] [DEV18-136156] running graceful exit on windows

なので、シグナルが飛んで終了しているようにも見えます。Windowsのサービスに登録して、起動していないようにも見えるので、サービス登録して、確認してみるとかでしょうか。

HAHA · June 13, 2023, 11:18am

ご回答ありがとうございます。
ご指摘を踏まえ、windowsサービスに登録したものの事象の改善は見られませんでした。。。

サービス登録後、elasticsearch-service.logを確認したところ、JVM destroyedといったログの出力が見られました。
JAVAの再インストールの実施をすればよいかと考えておりますが、
ご知見ございましたらご教示いただけますと幸いです。

elasticsearch-service.log

[2023-06-13 20:02:16] [debug] ( prunsrv.c:1994) [ 3676] Apache Commons Daemon procrun log initialized.
[2023-06-13 20:02:16] [info] ( prunsrv.c:2002) [ 3676] Apache Commons Daemon procrun (1.3.1.0 64-bit) started.
[2023-06-13 20:02:16] [info] ( prunsrv.c:1906) [ 3676] Running Service ‘elasticsearch-service-x64’…
[2023-06-13 20:02:16] [debug] ( prunsrv.c:1677) [12088] Inside serviceMain()…
[2023-06-13 20:02:16] [debug] ( prunsrv.c:1123) [12088] reportServiceStatusE: dwCurrentState = 2 (SERVICE_START_PENDING), dwWin32ExitCode = 0, dwWaitHint = 3000 milliseconds, dwServiceSpecificExitCode = 0.
[2023-06-13 20:02:16] [info] ( prunsrv.c:1422) [12088] Starting service…
[2023-06-13 20:02:16] [debug] ( prunsrv.c:497 ) [12088] Checking Java options for environment variable requirements
[2023-06-13 20:02:16] [debug] ( prunsrv.c:501 ) [12088] Checking environment variable requirements for ‘-XX:+UseSerialGC’
[2023-06-13 20:02:16] [debug] ( prunsrv.c:501 ) [12088] Checking environment variable requirements for ‘-Des.path.home=C:\elasticsearch-8.6.0’
[2023-06-13 20:02:16] [debug] ( prunsrv.c:501 ) [12088] Checking environment variable requirements for ‘-Des.path.conf=C:\elasticsearch-8.6.0\config’
[2023-06-13 20:02:16] [debug] ( prunsrv.c:501 ) [12088] Checking environment variable requirements for ‘-Des.distribution.type=zip’
[2023-06-13 20:02:16] [debug] ( prunsrv.c:501 ) [12088] Checking environment variable requirements for ‘-Dcli.name=windows-service-daemon’
[2023-06-13 20:02:16] [debug] ( prunsrv.c:501 ) [12088] Checking environment variable requirements for ‘-Dcli.libs=lib/tools/server-cli,lib/tools/windows-service-cli’
[2023-06-13 20:02:16] [debug] ( javajni.c:216 ) [12088] Explicit RuntimeLib specified ‘C:\elasticsearch-8.6.0\jdk\bin\server\jvm.dll’
[2023-06-13 20:02:16] [debug] ( javajni.c:280 ) [12088] Adding Java bin path to the PATH to fix loading of awt.dll: ‘C:\elasticsearch-8.6.0\jdk\bin’
[2023-06-13 20:02:16] [debug] ( javajni.c:286 ) [12088] Loading JVM DLL ‘C:\elasticsearch-8.6.0\jdk\bin\server\jvm.dll’
[2023-06-13 20:02:16] [debug] ( javajni.c:414 ) [12088] Loaded JVM DLL ‘C:\elasticsearch-8.6.0\jdk\bin\server\jvm.dll’, home ‘(null)’.
[2023-06-13 20:02:16] [debug] ( javajni.c:419 ) [12088] JNI_GetCreatedJavaVMs…
[2023-06-13 20:02:16] [debug] ( javajni.c:817 ) [12080] JVM Option[0] -XX:+UseSerialGC
[2023-06-13 20:02:16] [debug] ( javajni.c:817 ) [12080] JVM Option[1] -Des.path.home=C:\elasticsearch-8.6.0
[2023-06-13 20:02:16] [debug] ( javajni.c:817 ) [12080] JVM Option[2] -Des.path.conf=C:\elasticsearch-8.6.0\config
[2023-06-13 20:02:16] [debug] ( javajni.c:817 ) [12080] JVM Option[3] -Des.distribution.type=zip
[2023-06-13 20:02:16] [debug] ( javajni.c:817 ) [12080] JVM Option[4] -Dcli.name=windows-service-daemon
[2023-06-13 20:02:16] [debug] ( javajni.c:817 ) [12080] JVM Option[5] -Dcli.libs=lib/tools/server-cli,lib/tools/windows-service-cli
[2023-06-13 20:02:16] [debug] ( javajni.c:817 ) [12080] JVM Option[6] -Djava.class.path=C:\elasticsearch-8.6.0/lib/ecs-logging-core-1.2.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-8.6.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-cli-8.6.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-core-8.6.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-geo-8.6.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-log4j-8.6.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-logging-8.6.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-lz4-8.6.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-plugin-analysis-api-8.6.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-plugin-api-8.6.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-plugin-classloader-8.6.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-secure-sm-8.6.0.jar;C:\elasticsearch-8.6.0/lib/elasticsearch-x-content-8.6.0.jar;C:\elasticsearch-8.6.0/lib/HdrHistogram-2.1.9.jar;C:\elasticsearch-8.6.0/lib/hppc-0.8.1.jar;C:\elasticsearch-8.6.0/lib/jna-5.10.0.jar;C:\elasticsearch-8.6.0/lib/jopt-simple-5.0.2.jar;C:\elasticsearch-8.6.0/lib/log4j-api-2.19.0.jar;C:\elasticsearch-8.6.0/lib/log4j2-ecs-layout-1.2.0.jar;C:\elasticsearch-8.6.0/lib/lucene-analysis-common-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lucene-backward-codecs-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lucene-core-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lucene-grouping-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lucene-highlighter-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lucene-join-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lucene-memory-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lucene-misc-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lucene-queries-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lucene-queryparser-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lucene-sandbox-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lucene-suggest-9.4.2.jar;C:\elasticsearch-8.6.0/lib/lz4-java-1.8.0.jar;C:\elasticsearch-8.6.0/lib/t-digest-3.2.jar;C:\elasticsearch-8.6.0/lib/cli-launcher/cli-launcher-8.6.0.jar
[2023-06-13 20:02:16] [debug] ( javajni.c:817 ) [12080] JVM Option[7] exit
[2023-06-13 20:02:16] [debug] ( javajni.c:817 ) [12080] JVM Option[8] abort
[2023-06-13 20:02:18] [debug] ( javajni.c:1062) [12080] Java worker thread started for org/elasticsearch/launcher/CliToolLauncher:main
[2023-06-13 20:02:18] [debug] ( javajni.c:1068) [12080] JNI calling static void method org/elasticsearch/launcher/CliToolLauncher:main
[2023-06-13 20:02:19] [debug] ( prunsrv.c:1486) [12088] Java started ‘org/elasticsearch/launcher/CliToolLauncher’.
[2023-06-13 20:02:19] [info] ( prunsrv.c:1585) [12088] Service started in 2577 milliseconds.
[2023-06-13 20:02:19] [debug] ( prunsrv.c:1123) [12088] reportServiceStatusE: dwCurrentState = 4 (SERVICE_RUNNING), dwWin32ExitCode = 0, dwWaitHint = 0 milliseconds, dwServiceSpecificExitCode = 0.
[2023-06-13 20:02:19] [debug] ( prunsrv.c:1832) [12088] Waiting for worker to finish…
[2023-06-13 20:02:19] [debug] ( javajni.c:1157) [12088] apxJavaWait → WaitForSingleObject (0x000000000000027C, -1 milliseconds) (-1=INFINITE)…
[2023-06-13 20:02:53] [debug] ( javajni.c:1087) [12080] Java worker thread finished org/elasticsearch/launcher/CliToolLauncher:main with status = 0
[2023-06-13 20:02:53] [debug] ( javajni.c:1160) [12088] apxJavaWait ← WaitForSingleObject (0x000000000000027C, -1 milliseconds) = 0
[2023-06-13 20:02:53] [debug] ( prunsrv.c:1837) [12088] Worker finished.
[2023-06-13 20:02:53] [debug] ( prunsrv.c:1864) [12088] Waiting for all threads to exit.
[2023-06-13 20:07:54] [info] ( prunsrv.c:1606) [ 3676] Service SERVICE_CONTROL_STOP signalled.
[2023-06-13 20:07:54] [debug] ( prunsrv.c:1123) [ 3676] reportServiceStatusE: dwCurrentState = 3 (SERVICE_STOP_PENDING), dwWin32ExitCode = 0, dwWaitHint = 3000 milliseconds, dwServiceSpecificExitCode = 0.
[2023-06-13 20:07:54] [info] ( prunsrv.c:1223) [ 2272] Stopping service…
[2023-06-13 20:07:54] [debug] ( javajni.c:414 ) [ 2272] Loaded JVM DLL ‘C:\elasticsearch-8.6.0\jdk\bin\server\jvm.dll’, home ‘(null)’.
[2023-06-13 20:07:54] [debug] ( javajni.c:419 ) [ 2272] JNI_GetCreatedJavaVMs…
[2023-06-13 20:07:54] [debug] ( javajni.c:1062) [ 9396] Java worker thread started for org/elasticsearch/launcher/CliToolLauncher:close
[2023-06-13 20:07:54] [debug] ( javajni.c:1068) [ 9396] JNI calling static void method org/elasticsearch/launcher/CliToolLauncher:close
[2023-06-13 20:07:55] [debug] ( prunsrv.c:1273) [ 2272] Waiting for Java JNI stop worker to finish for org/elasticsearch/launcher/CliToolLauncher:close…
[2023-06-13 20:07:55] [debug] ( javajni.c:1157) [ 2272] apxJavaWait → WaitForSingleObject (0x00000000000004C0, -1 milliseconds) (-1=INFINITE)…
[2023-06-13 20:07:57] [debug] ( javajni.c:1087) [ 9396] Java worker thread finished org/elasticsearch/launcher/CliToolLauncher:close with status = 0
[2023-06-13 20:07:57] [debug] ( javajni.c:1160) [ 2272] apxJavaWait ← WaitForSingleObject (0x00000000000004C0, -1 milliseconds) = 0
[2023-06-13 20:07:57] [debug] ( prunsrv.c:1278) [ 2272] Java JNI stop worker finished.
[2023-06-13 20:07:57] [debug] ( prunsrv.c:1123) [12088] reportServiceStatusE: dwCurrentState = 3 (SERVICE_STOP_PENDING), dwWin32ExitCode = 0, dwWaitHint = 0 milliseconds, dwServiceSpecificExitCode = 0.
[2023-06-13 20:07:57] [debug] ( prunsrv.c:1868) [12088] JVM destroyed.
[2023-06-13 20:07:57] [debug] ( prunsrv.c:1123) [12088] reportServiceStatusE: dwCurrentState = 1 (SERVICE_STOPPED), dwWin32ExitCode = 0, dwWaitHint = 0 milliseconds, dwServiceSpecificExitCode = 0.
[2023-06-13 20:07:57] [info] ( prunsrv.c:1908) [ 3676] Run service finished.
[2023-06-13 20:07:57] [info] ( prunsrv.c:2086) [ 3676] Apache Commons Daemon procrun finished.

shinsuke · June 13, 2023, 12:09pm

Fessは関係なさそうな話になってきていますが、OSがJavaを終了させているのかもしれないので、OS側のログを見てみるとかでしょうか。

HAHA · June 21, 2023, 1:25am

ご回答いただきありがとうございます。
返信が遅くなり申し訳ございません。

OSのログを確認しましたが、直接的な原因が判明しなかったため、
OSの再インストールから実施したところ、fessが起動できることを確認いたしました。

構築にあたって追加で1点質問させてください。
公式サイトを参考に、JAVA17をインストールし、OSの環境変数にJAVA＿HOMEを設定を行っています。
ですが以下のように、JAVA_HOMEがignoreされ、バンドルされたJAVAが使用されているように見受けられました。
起動が出来ていること、現在クロールが動作しているため、問題として認識しておりませんが、もしご指摘等ございましたらご教示いただきたく存じます。

elasticsearch.log

[2023-06-12T19:00:49,110][INFO ][o.e.n.Node ] [DEV18-136156] version[8.6.0], pid[11824], build[zip/f67ef2df40237445caa70e2fef79471cc608d70d/2023-01-04T09:35:21.782467981Z], OS[Windows Server 2019/10.0/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.1/19.0.1+10-21]
[2023-06-12T19:00:49,126][INFO ][o.e.n.Node ] [DEV18-136156] JVM home [C:\elasticsearch-8.6.0\jdk], using bundled JDK [true]

zolgear · June 21, 2023, 2:51am

Elastic のインストールドキュメントに目を通して、以下の環境変数について理解、設定する事をお勧めします。

ES_JAVA_HOME
ES_TMPDIR

ES_TMPDIR が未設定だと、Windows をFUした際に temp ディレクトリがクリアされてしまい、Elasticsearch サービスが起動しないトラブルが発生します。

わたしは、クリアされない %ES_HOME%\temp ディレクトリを作成しておき、
elasticsearch-service.bat にて、環境変数をセットしています。

set ES_TMPDIR=%ES_HOME%\temp

HAHA · July 7, 2023, 8:31am

返信出来ておりませんでした。
ご回答いただきありがとうございます。

お忙しいところ恐縮ですが、改めて確認をさせてください。
クロール途中にfessがブラウザに表示できない事象が再発しました。
エラーの原因の特定と、復旧方法をご教示いただきたいです。

【経緯】
fess-crawler.logではfessが起動しなくなったタイミング付近から、なぜかopenserchに関するログが出力されていました。
頂いた設定を未設定だったため、
elasticserchのtempファイルであろう、以下ディレクトリも中身が空になっていました。
C:\Users\ユーザ名\AppData\Local\Temp
winsowsFUは発生していない認識でしたが、ご回答いただいていたelasticserchが接続できなくなる事象かと考えています。

一応、openserchに切り替わる、fess-crawler.logを共有いたします。情報に不足あればご指摘ください。(確認すべき観点など）
大変お手数ですが、上記の認識で誤っていないか、復旧方法（elasticsearchの再インストール？）を教えていただきたく存じます。

2023-07-06 12:55:14,705 [pool-5-thread-20] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362368946176,“total”:410842562560},“swap_space”:{“free”:403155447808,“total”:453792235520}},“cpu”:{“percent”:11},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:9,“total”:225421},“virtual_memory”:{“total”:1291390976}},“jvm”:{“memory”:{“heap”:{“used”:261865464,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:135626432,“committed”:139722752}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:426,“time”:1782},“old”:{“count”:0,“time”:0}},“threads”:{“count”:73,“peak”:73},“classes”:{“loaded”:13332,“total_loaded”:13365,“unloaded”:33},“uptime”:692302},“search_engine”:{“nodes”:{}},“timestamp”:“2023-07-06T03:55:14.705Z”}
2023-07-06 12:55:28,907 [Crawler-20230706124332-1-3] INFO Crawling URL: file:/D:/
2023-07-06 12:55:28,907 [Crawler-20230706124332-1-1] INFO Crawling URL: file:/D:/
2023-07-06 12:55:28,907 [Crawler-20230706124332-1-2] INFO Crawling URL: file:/D:/
2023-07-06 12:55:28,907 [Crawler-20230706124332-1-4] INFO Crawling URL: file:/D:/
2023-07-06 12:55:28,907 [Crawler-20230706124332-1-5] INFO Crawling URL: file:/D:/
2023-07-06 12:56:25,183 [pool-5-thread-22] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362382139392,“total”:410842562560},“swap_space”:{“free”:403149344768,“total”:453792235520}},“cpu”:{“percent”:2},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:227187},“virtual_memory”:{“total”:1297145856}},“jvm”:{“memory”:{“heap”:{“used”:274707512,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:136691560,“committed”:140705792}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:428,“time”:1790},“old”:{“count”:0,“time”:0}},“threads”:{“count”:72,“peak”:73},“classes”:{“loaded”:13618,“total_loaded”:13651,“unloaded”:33},“uptime”:762544},“exception”:“OpenSearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.];\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.appendFesenStats(SystemMonitorTarget.java:159)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.expired(SystemMonitorTarget.java:57)\r\n\tat org.codelibs.core.timer.TimeoutTask.expired(TimeoutTask.java:113)\r\n\tat org.codelibs.core.timer.TimeoutManager.lambda$processTask$0(TimeoutManager.java:198)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base\/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\r\n\tat org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257)\r\n\tat org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82)\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94)\r\n\t… 9 more\r\n”,“search_engine”:null,“timestamp”:“2023-07-06T03:56:25.183Z”}
2023-07-06 12:56:26,198 [pool-5-thread-23] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362382127104,“total”:410842562560},“swap_space”:{“free”:403148255232,“total”:453792235520}},“cpu”:{“percent”:1},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:227203},“virtual_memory”:{“total”:1297145856}},“jvm”:{“memory”:{“heap”:{“used”:274707512,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:136699712,“committed”:140705792}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:428,“time”:1790},“old”:{“count”:0,“time”:0}},“threads”:{“count”:72,“peak”:73},“classes”:{“loaded”:13618,“total_loaded”:13651,“unloaded”:33},“uptime”:763559},“exception”:“OpenSearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.];\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.appendFesenStats(SystemMonitorTarget.java:159)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.expired(SystemMonitorTarget.java:57)\r\n\tat org.codelibs.core.timer.TimeoutTask.expired(TimeoutTask.java:113)\r\n\tat org.codelibs.core.timer.TimeoutManager.lambda$processTask$0(TimeoutManager.java:198)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base\/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\r\n\tat org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257)\r\n\tat org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82)\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94)\r\n\t… 9 more\r\n”,“search_engine”:null,“timestamp”:“2023-07-06T03:56:26.198Z”}
2023-07-06 12:56:27,213 [pool-5-thread-24] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362382098432,“total”:410842562560},“swap_space”:{“free”:403151822848,“total”:453792235520}},“cpu”:{“percent”:1},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:227203},“virtual_memory”:{“total”:1297371136}},“jvm”:{“memory”:{“heap”:{“used”:274707512,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:136716736,“committed”:140771328}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:428,“time”:1790},“old”:{“count”:0,“time”:0}},“threads”:{“count”:72,“peak”:73},“classes”:{“loaded”:13618,“total_loaded”:13651,“unloaded”:33},“uptime”:764574},“exception”:“OpenSearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.];\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.appendFesenStats(SystemMonitorTarget.java:159)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.expired(SystemMonitorTarget.java:57)\r\n\tat org.codelibs.core.timer.TimeoutTask.expired(TimeoutTask.java:113)\r\n\tat org.codelibs.core.timer.TimeoutManager.lambda$processTask$0(TimeoutManager.java:198)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base\/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\r\n\tat org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257)\r\n\tat org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82)\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94)\r\n\t… 9 more\r\n”,“search_engine”:null,“timestamp”:“2023-07-06T03:56:27.213Z”}
2023-07-06 12:56:28,229 [pool-5-thread-25] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362382159872,“total”:410842562560},“swap_space”:{“free”:403151921152,“total”:453792235520}},“cpu”:{“percent”:1},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:227203},“virtual_memory”:{“total”:1297530880}},“jvm”:{“memory”:{“heap”:{“used”:274707512,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:136716736,“committed”:140771328}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:428,“time”:1790},“old”:{“count”:0,“time”:0}},“threads”:{“count”:72,“peak”:73},“classes”:{“loaded”:13618,“total_loaded”:13651,“unloaded”:33},“uptime”:765589},“exception”:“OpenSearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.];\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.appendFesenStats(SystemMonitorTarget.java:159)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.expired(SystemMonitorTarget.java:57)\r\n\tat org.codelibs.core.timer.TimeoutTask.expired(TimeoutTask.java:113)\r\n\tat org.codelibs.core.timer.TimeoutManager.lambda$processTask$0(TimeoutManager.java:198)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base\/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\r\n\tat org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257)\r\n\tat org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82)\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94)\r\n\t… 9 more\r\n”,“search_engine”:null,“timestamp”:“2023-07-06T03:56:28.229Z”}
2023-07-06 12:56:29,244 [pool-5-thread-26] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362382434304,“total”:410842562560},“swap_space”:{“free”:403152166912,“total”:453792235520}},“cpu”:{“percent”:1},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:227203},“virtual_memory”:{“total”:1297686528}},“jvm”:{“memory”:{“heap”:{“used”:274707512,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:136719864,“committed”:140771328}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:428,“time”:1790},“old”:{“count”:0,“time”:0}},“threads”:{“count”:72,“peak”:73},“classes”:{“loaded”:13618,“total_loaded”:13651,“unloaded”:33},“uptime”:766609},“exception”:“OpenSearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.];\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.appendFesenStats(SystemMonitorTarget.java:159)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.expired(SystemMonitorTarget.java:57)\r\n\tat org.codelibs.core.timer.TimeoutTask.expired(TimeoutTask.java:113)\r\n\tat org.codelibs.core.timer.TimeoutManager.lambda$processTask$0(TimeoutManager.java:198)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base\/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\r\n\tat org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257)\r\n\tat org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82)\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94)\r\n\t… 9 more\r\n”,“search_engine”:null,“timestamp”:“2023-07-06T03:56:29.244Z”}
2023-07-06 12:56:30,260 [pool-5-thread-27] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362382315520,“total”:410842562560},“swap_space”:{“free”:403152281600,“total”:453792235520}},“cpu”:{“percent”:1},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:227218},“virtual_memory”:{“total”:1297842176}},“jvm”:{“memory”:{“heap”:{“used”:274707512,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:136719864,“committed”:140771328}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:428,“time”:1790},“old”:{“count”:0,“time”:0}},“threads”:{“count”:72,“peak”:73},“classes”:{“loaded”:13618,“total_loaded”:13651,“unloaded”:33},“uptime”:767620},“exception”:“OpenSearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.];\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.appendFesenStats(SystemMonitorTarget.java:159)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.expired(SystemMonitorTarget.java:57)\r\n\tat org.codelibs.core.timer.TimeoutTask.expired(TimeoutTask.java:113)\r\n\tat org.codelibs.core.timer.TimeoutManager.lambda$processTask$0(TimeoutManager.java:198)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base\/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\r\n\tat org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257)\r\n\tat org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82)\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94)\r\n\t… 9 more\r\n”,“search_engine”:null,“timestamp”:“2023-07-06T03:56:30.260Z”}
2023-07-06 12:56:31,275 [pool-5-thread-28] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362382856192,“total”:410842562560},“swap_space”:{“free”:403152617472,“total”:453792235520}},“cpu”:{“percent”:1},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:227218},“virtual_memory”:{“total”:1297842176}},“jvm”:{“memory”:{“heap”:{“used”:274707512,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:136721400,“committed”:140771328}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:428,“time”:1790},“old”:{“count”:0,“time”:0}},“threads”:{“count”:72,“peak”:73},“classes”:{“loaded”:13618,“total_loaded”:13651,“unloaded”:33},“uptime”:768636},“exception”:“OpenSearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.];\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.appendFesenStats(SystemMonitorTarget.java:159)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.expired(SystemMonitorTarget.java:57)\r\n\tat org.codelibs.core.timer.TimeoutTask.expired(TimeoutTask.java:113)\r\n\tat org.codelibs.core.timer.TimeoutManager.lambda$processTask$0(TimeoutManager.java:198)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base\/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\r\n\tat org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257)\r\n\tat org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82)\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94)\r\n\t… 9 more\r\n”,“search_engine”:null,“timestamp”:“2023-07-06T03:56:31.275Z”}
2023-07-06 12:56:32,290 [pool-5-thread-1] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362382188544,“total”:410842562560},“swap_space”:{“free”:403152674816,“total”:453792235520}},“cpu”:{“percent”:1},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:227218},“virtual_memory”:{“total”:1297842176}},“jvm”:{“memory”:{“heap”:{“used”:274707512,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:136721400,“committed”:140771328}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:428,“time”:1790},“old”:{“count”:0,“time”:0}},“threads”:{“count”:72,“peak”:73},“classes”:{“loaded”:13618,“total_loaded”:13651,“unloaded”:33},“uptime”:769651},“exception”:“OpenSearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.];\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.appendFesenStats(SystemMonitorTarget.java:159)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.expired(SystemMonitorTarget.java:57)\r\n\tat org.codelibs.core.timer.TimeoutTask.expired(TimeoutTask.java:113)\r\n\tat org.codelibs.core.timer.TimeoutManager.lambda$processTask$0(TimeoutManager.java:198)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base\/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\r\n\tat org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257)\r\n\tat org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82)\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94)\r\n\t… 9 more\r\n”,“search_engine”:null,“timestamp”:“2023-07-06T03:56:32.290Z”}
2023-07-06 12:56:33,306 [pool-5-thread-2] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362382135296,“total”:410842562560},“swap_space”:{“free”:403152674816,“total”:453792235520}},“cpu”:{“percent”:1},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:227218},“virtual_memory”:{“total”:1297842176}},“jvm”:{“memory”:{“heap”:{“used”:274707512,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:136734480,“committed”:140771328}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:428,“time”:1790},“old”:{“count”:0,“time”:0}},“threads”:{“count”:72,“peak”:73},“classes”:{“loaded”:13618,“total_loaded”:13651,“unloaded”:33},“uptime”:770671},“exception”:“OpenSearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.];\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.appendFesenStats(SystemMonitorTarget.java:159)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.expired(SystemMonitorTarget.java:57)\r\n\tat org.codelibs.core.timer.TimeoutTask.expired(TimeoutTask.java:113)\r\n\tat org.codelibs.core.timer.TimeoutManager.lambda$processTask$0(TimeoutManager.java:198)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base\/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\r\n\tat org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257)\r\n\tat org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82)\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94)\r\n\t… 9 more\r\n”,“search_engine”:null,“timestamp”:“2023-07-06T03:56:33.306Z”}
2023-07-06 12:56:34,321 [pool-5-thread-3] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362382311424,“total”:410842562560},“swap_space”:{“free”:403152805888,“total”:453792235520}},“cpu”:{“percent”:1},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:227234},“virtual_memory”:{“total”:1297842176}},“jvm”:{“memory”:{“heap”:{“used”:274707512,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:136747192,“committed”:140771328}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:428,“time”:1790},“old”:{“count”:0,“time”:0}},“threads”:{“count”:72,“peak”:73},“classes”:{“loaded”:13618,“total_loaded”:13651,“unloaded”:33},“uptime”:771682},“exception”:“OpenSearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.];\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.appendFesenStats(SystemMonitorTarget.java:159)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.expired(SystemMonitorTarget.java:57)\r\n\tat org.codelibs.core.timer.TimeoutTask.expired(TimeoutTask.java:113)\r\n\tat org.codelibs.core.timer.TimeoutManager.lambda$processTask$0(TimeoutManager.java:198)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base\/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\r\n\tat org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257)\r\n\tat org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82)\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94)\r\n\t… 9 more\r\n”,“search_engine”:null,“timestamp”:“2023-07-06T03:56:34.321Z”}
2023-07-06 12:57:41,901 [IndexUpdater] WARN Could not add a doc: file:/D:/
org.opensearch.OpenSearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.
at org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:73) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:63) ~[opensearch-2.4.1.jar:2.4.1]
at org.codelibs.fess.es.log.allcommon.EsAbstractBehavior.delegateSelectCountUniquely(EsAbstractBehavior.java:109) ~[classes/:?]
at org.dbflute.bhv.AbstractBehaviorReadable.doSelectCountUniquely(AbstractBehaviorReadable.java:159) ~[dbflute-runtime-1.2.6.jar:?]
at org.dbflute.bhv.AbstractBehaviorReadable.facadeSelectCount(AbstractBehaviorReadable.java:154) ~[dbflute-runtime-1.2.6.jar:?]
at org.codelibs.fess.es.log.bsbhv.BsFavoriteLogBhv.selectCount(BsFavoriteLogBhv.java:96) ~[classes/:?]
at org.codelibs.fess.helper.SearchLogHelper.getFavoriteCount(SearchLogHelper.java:228) ~[classes/:?]
at org.codelibs.fess.indexer.IndexUpdater.addFavoriteCountField(IndexUpdater.java:477) ~[classes/:?]
at org.codelibs.fess.indexer.IndexUpdater.updateDocument(IndexUpdater.java:429) ~[classes/:?]
at org.codelibs.fess.indexer.IndexUpdater.processAccessResults(IndexUpdater.java:368) ~[classes/:?]
at org.codelibs.fess.indexer.IndexUpdater.run(IndexUpdater.java:233) ~[classes/:?]
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
at org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82) ~[opensearch-2.4.1.jar:2.4.1]
at org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94) ~[opensearch-2.4.1.jar:2.4.1]
… 12 more
2023-07-06 12:57:44,818 [pool-5-thread-4] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:362675417088,“total”:410842562560},“swap_space”:{“free”:403611058176,“total”:453792235520}},“cpu”:{“percent”:2},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:227265},“virtual_memory”:{“total”:1298071552}},“jvm”:{“memory”:{“heap”:{“used”:276804664,“committed”:859832320,“max”:3221225472,“percent”:8},“non_heap”:{“used”:136761960,“committed”:140836864}},“pools”:{“mapped”:{“count”:0,“used”:0,“capacity”:0},“direct”:{“count”:25,“used”:3552219,“capacity”:3552219},“mapped - ‘non-volatile memory’”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:428,“time”:1790},“old”:{“count”:0,“time”:0}},“threads”:{“count”:72,“peak”:73},“classes”:{“loaded”:13618,“total_loaded”:13651,“unloaded”:33},“uptime”:842184},“exception”:“OpenSearchTimeoutException[java.util.concurrent.TimeoutException: Timeout waiting for task.]; nested: TimeoutException[Timeout waiting for task.];\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79)\r\n\tat org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.appendFesenStats(SystemMonitorTarget.java:159)\r\n\tat org.codelibs.fess.timer.SystemMonitorTarget.expired(SystemMonitorTarget.java:57)\r\n\tat org.codelibs.core.timer.TimeoutTask.expired(TimeoutTask.java:113)\r\n\tat org.codelibs.core.timer.TimeoutManager.lambda$processTask$0(TimeoutManager.java:198)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base\/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.util.concurrent.TimeoutException: Timeout waiting for task.\r\n\tat org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257)\r\n\tat org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82)\r\n\tat org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94)\r\n\t… 9 more\r\n”,“search_engine”:null,“timestamp”:“2023-07-06T03:57:44.818Z”}

shinsuke · July 7, 2023, 11:56am

openserchに切り替わる

OpenSearchnに切り替わるわけではなく、FessからはElasticsearchもOpenSearchもOpenSearchとして、アクセスしているので、Elasticsearchにアクセスできなくなった、というだけだと思います。
Elasticsearchの問題だと思うので、ElasticsearchやOSのログを確認するしかないと思います。
よくわからない問題で、ハードウェアのリソース不足とかでなければ、Dockerで構築したほうが楽だと思います。

HAHA · July 10, 2023, 9:42am

ご回答ありがとうございます。

elasticsearch.logを見る限りだと、接続できなくなったタイミング前後で特に異常メッセージは見られませんでした。（以下添付。7/7 13時ごろに接続不可）
windowsログも精査中ですが、目立ったエラーもありませんでした。

原因を判断したく、elaticserch.log以外の確認すべきログなどありましたらご教授いただきたく存じます。
加えて、一応教えていただきたいのですが、
elasticserchが原因だと仮定した場合、
dockerを導入せず再度復旧する方法としては、elasticserchの再インストールだと考えていますが、認識相違ありますでしょうか。

[2023-07-06T12:00:07,970][INFO ][o.e.c.r.a.AllocationService] [DEV18-136156] current.health=“GREEN” message=“Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[fess.suggest.20230706120007][0]]]).” previous.health=“YELLOW” reason=“shards started [[fess.suggest.20230706120007][0]]”
[2023-07-06T12:00:10,265][INFO ][o.e.c.m.MetadataMappingService] [DEV18-136156] [fess.suggest.20230706120007/RJpdYMgJTEyL9AXAKI7zzQ] update_mapping [_doc]
[2023-07-06T17:34:55,862][ERROR][o.e.i.g.GeoIpDownloader ] [DEV18-136156] exception during geoip databases update
java.net.UnknownHostException: geoip.elastic.co
at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:560) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
at java.net.Socket.connect(Socket.java:666) ~[?:?]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304) ~[?:?]
at sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:531) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:636) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:378) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1241) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1127) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1661) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1585) ~[?:?]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.lambda$get$0(HttpClient.java:46) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.doPrivileged(HttpClient.java:88) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.get(HttpClient.java:40) ~[?:?]
at org.elasticsearch.ingest.geoip.HttpClient.getBytes(HttpClient.java:36) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloader.fetchDatabasesOverview(GeoIpDownloader.java:155) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:143) ~[?:?]
at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:274) ~[?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:850) ~[elasticsearch-8.6.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
[2023-07-06T18:40:12,353][INFO ][o.e.n.Node ] [DEV18-136156] stopping …

zolgear · July 10, 2023, 9:50am

エラーの通り、geoip databases update で、接続エラーがおきています。
社内プロキシ、ファイアウォールの問題でしょう。

FESS 用途だけであれば gis 関連は不要なので、
config/elasticsearch.yml の末尾に、無効化する設定を追記します。

ingest.geoip.downloader.enabled: false

HAHA · July 19, 2023, 8:51am

ご回答ありがとうございます。
度々申し訳ございませんが、質問させてください。

ご指摘の設定をし、elasticsearchの再インストールを実施しましたが、
fessとelasticsearchとの接続が上手くいっていないように見えます。

復旧方法をご教示いただけますと幸いです。（以下にfess.logの出力を添付します）

2023-07-18 14:11:46,269 [main] WARN [0] Unauthorized access: http://localhost:9200
org.opensearch.OpenSearchStatusException: OpenSearch exception [type=security_exception, reason=missing authentication credentials for REST request [/_cluster/health/?wait_for_no_relocating_shards=false&wait_for_no_initializing_shards=false&wait_for_nodes=&wait_for_status=yellow&wait_for_active_shards=0&wait_for_active_shards=0&timeout=30s&master_timeout=30s]]
at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:207) ~[opensearch-2.4.1.jar:2.4.1]
at org.codelibs.fesen.client.action.HttpAction.toOpenSearchException(HttpAction.java:139) ~[fesen-httpclient-2.0.0.jar:?]
at org.codelibs.fesen.client.action.HttpClusterHealthAction.lambda$execute$0(HttpClusterHealthAction.java:62) ~[fesen-httpclient-2.0.0.jar:?]
at org.codelibs.curl.CurlRequest.lambda$execute$5(CurlRequest.java:260) ~[curl4j-1.2.7.jar:?]
at org.codelibs.curl.CurlRequest.lambda$connect$4(CurlRequest.java:231) ~[curl4j-1.2.7.jar:?]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395) ~[?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) ~[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) ~[?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) ~[?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) ~[?:?]
Suppressed: org.opensearch.common.xcontent.XContentParseException: [1:820] [cluster_health_response] status doesn’t support values of type: VALUE_NUMBER
at org.opensearch.common.xcontent.ObjectParser$FieldParser.assertSupports(ObjectParser.java:660) ~[opensearch-x-content-2.4.1.jar:2.4.1]
at org.opensearch.common.xcontent.ObjectParser.parse(ObjectParser.java:327) ~[opensearch-x-content-2.4.1.jar:2.4.1]
at org.opensearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:186) ~[opensearch-x-content-2.4.1.jar:2.4.1]
at org.opensearch.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:178) ~[opensearch-x-content-2.4.1.jar:2.4.1]
at org.codelibs.fesen.client.action.HttpClusterHealthAction.lambda$execute$0(HttpClusterHealthAction.java:59) ~[fesen-httpclient-2.0.0.jar:?]
at org.codelibs.curl.CurlRequest.lambda$execute$5(CurlRequest.java:260) ~[curl4j-1.2.7.jar:?]
at org.codelibs.curl.CurlRequest.lambda$connect$4(CurlRequest.java:231) ~[curl4j-1.2.7.jar:?]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395) ~[?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) ~[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) ~[?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) ~[?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) ~[?:?]
Suppressed: org.codelibs.fesen.client.action.HttpAction$CurlResponseException: {“error”:{“root_cause”:[{“type”:“security_exception”,“reason”:“missing authentication credentials for REST request [/_cluster/health/?wait_for_no_relocating_shards=false&wait_for_no_initializing_shards=false&wait_for_nodes=&wait_for_status=yellow&wait_for_active_shards=0&wait_for_active_shards=0&timeout=30s&master_timeout=30s]”,“header”:{“WWW-Authenticate”:[“Basic realm="security" charset="UTF-8"”,“ApiKey”]}}],“type”:“security_exception”,“reason”:“missing authentication credentials for REST request [/_cluster/health/?wait_for_no_relocating_shards=false&wait_for_no_initializing_shards=false&wait_for_nodes=&wait_for_status=yellow&wait_for_active_shards=0&wait_for_active_shards=0&timeout=30s&master_timeout=30s]”,“header”:{“WWW-Authenticate”:[“Basic realm="security" charset="UTF-8"”,“ApiKey”]}},“status”:401}
2023-07-18 14:12:05,610 [main] WARN [1] Unauthorized access: http://localhost:9200
org.opensearch.OpenSearchStatusException: OpenSearch exception [type=security_exception, reason=missing authentication credentials for REST request [/_cluster/health/?wait_for_no_relocating_shards=false&wait_for_no_initializing_shards=false&wait_for_nodes=&wait_for_status=yellow&wait_for_active_shards=0&wait_for_active_shards=0&timeout=30s&master_timeout=30s]]
at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:207) ~[opensearch-2.4.1.jar:2.4.1]
at org.codelibs.fesen.client.action.HttpAction.toOpenSearchException(HttpAction.java:139) ~[fesen-httpclient-2.0.0.jar:?]
at org.codelibs.fesen.client.action.HttpClusterHealthAction.lambda$execute$0(HttpClusterHealthAction.java:62) ~[fesen-httpclient-2.0.0.jar:?]
at org.codelibs.curl.CurlRequest.lambda$execute$5(CurlRequest.java:260) ~[curl4j-1.2.7.jar:?]
at org.codelibs.curl.CurlRequest.lambda$connect$4(CurlRequest.java:231) ~[curl4j-1.2.7.jar:?]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395) ~[?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) ~[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) ~[?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) ~[?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) ~[?:?]
Suppressed: org.opensearch.common.xcontent.XContentParseException: [1:820] [cluster_health_response] status doesn’t support values of type: VALUE_NUMBER
at org.opensearch.common.xcontent.ObjectParser$FieldParser.assertSupports(ObjectParser.java:660) ~[opensearch-x-content-2.4.1.jar:2.4.1]
at org.opensearch.common.xcontent.ObjectParser.parse(ObjectParser.java:327) ~[opensearch-x-content-2.4.1.jar:2.4.1]
at org.opensearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:186) ~[opensearch-x-content-2.4.1.jar:2.4.1]
at org.opensearch.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:178) ~[opensearch-x-content-2.4.1.jar:2.4.1]
at org.codelibs.fesen.client.action.HttpClusterHealthAction.lambda$execute$0(HttpClusterHealthAction.java:59) ~[fesen-httpclient-2.0.0.jar:?]
at org.codelibs.curl.CurlRequest.lambda$execute$5(CurlRequest.java:260) ~[curl4j-1.2.7.jar:?]
at org.codelibs.curl.CurlRequest.lambda$connect$4(CurlRequest.java:231) ~[curl4j-1.2.7.jar:?]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395) ~[?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) ~[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) ~[?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) ~[?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) ~[?:?]
Suppressed: org.codelibs.fesen.client.action.HttpAction$CurlResponseException: {“error”:{“root_cause”:[{“type”:“security_exception”,“reason”:“missing authentication credentials for REST request [/_cluster/health/?wait_for_no_relocating_shards=false&wait_for_no_initializing_shards=false&wait_for_nodes=&wait_for_status=yellow&wait_for_active_shards=0&wait_for_active_shards=0&timeout=30s&master_timeout=30s]”,“header”:{“WWW-Authenticate”:[“Basic realm="security" charset="UTF-8"”,“ApiKey”]}}],“type”:“security_exception”,“reason”:“missing authentication credentials for REST request [/_cluster/health/?wait_for_no_relocating_shards=false&wait_for_no_initializing_shards=false&wait_for_nodes=&wait_for_status=yellow&wait_for_active_shards=0&wait_for_active_shards=0&timeout=30s&master_timeout=30s]”,“header”:{“WWW-Authenticate”:[“Basic realm="security" charset="UTF-8"”,“ApiKey”]}},“status”:401}
2023-07-18 14:12:13,749 [main] WARN [2] Unauthorized access: http://localhost:9200

zolgear · July 19, 2023, 10:10am

Elasticsearch 8.x は、xpack.security.enabled がデフォルトで true になっています（たぶん）

7.x はデフォルト無効だった為、REST API でエラーは出ません。

xpack.security.enabled を false にすれば、Unauthorized access は解決できます。

false にする前に、念のため、9200ポートが外部からアクセスできない事を確認すると良いです。