度々すみません。
CsvListDataStoreを使用して約12万件のファイルのインデックスを作成すると
elasticsearchのログに以下の警告が大量に出力されます。
■elasticsearch.log
[2020-05-15T12:58:49,945][WARN ][r.suppressed ] [ホスト名] path: /.suggest_analyzer/_analyze, params: {index=.suggest_analyzer}
org.elasticsearch.transport.RemoteTransportException: [ホスト名][127.0.0.1:9300][indices:admin/analyze[s]]
Caused by: java.lang.IllegalStateException: The number of tokens produced by calling _analyze has exceeded the allowed maximum of [10000]. This limit can be set by changing the [index.analyze.max_token_count] index level setting.
at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction$TokenCounter.increment(TransportAnalyzeAction.java:406) ~[elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction$TokenCounter.access$100(TransportAnalyzeAction.java:396) ~[elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.simpleAnalyze(TransportAnalyzeAction.java:241) ~[elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.analyze(TransportAnalyzeAction.java:214) ~[elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.analyze(TransportAnalyzeAction.java:138) ~[elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.shardOperation(TransportAnalyzeAction.java:121) ~[elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.action.admin.indices.analyze.TransportAnalyzeAction.shardOperation(TransportAnalyzeAction.java:73) ~[elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.lambda$asyncShardOperation$0(TransportSingleShardAction.java:110) ~[elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
「index.analyze.max_token_count」を変更すれば防げるようですが、ヒープとの兼ね合いで変更は避けております。
この警告はFessから短時間に大量のリクエストがある場合に出力されると推測しておりまして
この警告が出力されない程度にリクエストの感覚を広げたいのですが、目安等ありますでしょうか。
例えば、CsvListDataStoreの件数を1万件ずつにして、30分毎に動かす等
よろしくお願いします。