AWS OpenSearch

Hi, I followed article on news.mynavi to configure fess and AWS OpenSearch service.

I used ElasticSearch 7.9 and Fess 13.12.

With command: docker run -d -p 8080:8080 -e ES_HTTP_URL=“link” -e ES_TYPE=aws -e ES_USERNAME=xxx -e ES_PASSWORD=xxx fess:13.12

As ES_HTTP_URL I use Domain endpoint. And I get error:

2023-01-29 14:18:26 ==> /var/log/fess/audit.log <==
2023-01-29 14:18:26
2023-01-29 14:18:26 ==> /var/log/fess/fess-crawler.log <==
2023-01-29 14:18:26
2023-01-29 14:18:26 ==> /var/log/fess/fess.log <==
2023-01-29 14:18:26
2023-01-29 14:18:26 ==> /var/log/fess/fess-suggest.log <==
2023-01-29 14:18:26
2023-01-29 14:18:26 ==> /var/log/fess/fess-thumbnail.log <==
2023-01-29 14:18:26 * Starting Fess Server
2023-01-29 14:18:27 …done.
2023-01-29 14:18:26 sysctl: setting key “vm.max_map_count”: Read-only file system
2023-01-29 14:19:35
2023-01-29 14:19:35 ==> /var/log/fess/fess.log <==
2023-01-29 14:19:35 2023-01-29 14:19:35,134 [main] ERROR Failed to initialize Lasta Di.
2023-01-29 14:19:35 org.lastaflute.di.exception.ContainerInitFailureException: Look! Read the message below.
2023-01-29 14:19:35 /* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
2023-01-29 14:19:35 Failed to initialize the container.
2023-01-29 14:19:35
2023-01-29 14:19:35 [Path]
2023-01-29 14:19:35 esclient.xml
2023-01-29 14:19:35
2023-01-29 14:19:35 [Namespace]
2023-01-29 14:19:35 null
2023-01-29 14:19:35
2023-01-29 14:19:35 [Included by]
2023-01-29 14:19:35 esflute_config.xml
2023-01-29 14:19:35 esflute_user.xml
2023-01-29 14:19:35 esflute_log.xml
2023-01-29 14:19:35 * * * * * * * * * */
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.throwContainerInitFailureException(LaContainerImpl.java:463) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:415) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:431) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:431) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:431) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.factory.SingletonLaContainerFactory.init(SingletonLaContainerFactory.java:67) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.web.container.WebLastaContainerInitializer.doInitContainer(WebLastaContainerInitializer.java:81) ~[lastaflute-1.2.0.jar:?]
2023-01-29 14:19:35 at org.lastaflute.web.container.WebLastaContainerInitializer.initialize(WebLastaContainerInitializer.java:45) ~[lastaflute-1.2.0.jar:?]
2023-01-29 14:19:35 at org.lastaflute.web.servlet.filter.LastaPrepareFilter.initializeContainer(LastaPrepareFilter.java:137) [lastaflute-1.2.0.jar:?]
2023-01-29 14:19:35 at org.lastaflute.web.servlet.filter.LastaPrepareFilter.init(LastaPrepareFilter.java:94) [lastaflute-1.2.0.jar:?]
2023-01-29 14:19:35 at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:270) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:251) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:102) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4566) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5203) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374) [classes/:?]
2023-01-29 14:19:35 at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
2023-01-29 14:19:35 at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [classes/:?]
2023-01-29 14:19:35 at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) [?:?]
2023-01-29 14:19:35 at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:843) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374) [classes/:?]
2023-01-29 14:19:35 at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
2023-01-29 14:19:35 at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [classes/:?]
2023-01-29 14:19:35 at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) [?:?]
2023-01-29 14:19:35 at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.StandardService.startInternal(StandardService.java:434) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [classes/:?]
2023-01-29 14:19:35 at org.apache.catalina.startup.Tomcat.start(Tomcat.java:486) [classes/:?]
2023-01-29 14:19:35 at org.dbflute.tomcat.TomcatBoot.startServer(TomcatBoot.java:713) [classes/:?]
2023-01-29 14:19:35 at org.dbflute.tomcat.TomcatBoot.go(TomcatBoot.java:402) [classes/:?]
2023-01-29 14:19:35 at org.dbflute.tomcat.TomcatBoot.bootAwait(TomcatBoot.java:347) [classes/:?]
2023-01-29 14:19:35 at org.codelibs.fess.FessBoot.main(FessBoot.java:96) [classes/:?]
2023-01-29 14:19:35 Caused by: org.lastaflute.di.core.exception.IllegalMethodRuntimeException: [ESSR0060]Can not configure method(open) of org.codelibs.fess.es.client.SearchEngineClient, because org.lastaflute.di.exception.ContainerInitFailureException: Fesen (https://search-fess-public-qnkgmdcjm5shywhgy2uj2ztmtm.eu-west-2.es.amazonaws.com/) is not available. Check the state of your Fesen cluster (fesen) in 62558ms.
2023-01-29 14:19:35 at org.lastaflute.di.core.assembler.AbstractMethodAssembler.invoke(AbstractMethodAssembler.java:61) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.assembler.DefaultInitMethodAssembler.assemble(DefaultInitMethodAssembler.java:40) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.deployer.SingletonComponentDeployer.assemble(SingletonComponentDeployer.java:57) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.deployer.SingletonComponentDeployer.deploy(SingletonComponentDeployer.java:41) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.deployer.SingletonComponentDeployer.init(SingletonComponentDeployer.java:35) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.ComponentDefImpl.init(ComponentDefImpl.java:85) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:434) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 … 42 more
2023-01-29 14:19:35 Caused by: org.lastaflute.di.exception.ContainerInitFailureException: Fesen (https://search-fess-public-qnkgmdcjm5shywhgy2uj2ztmtm.eu-west-2.es.amazonaws.com/) is not available. Check the state of your Fesen cluster (fesen) in 62558ms.
2023-01-29 14:19:35 at org.codelibs.fess.es.client.SearchEngineClient.waitForYellowStatus(SearchEngineClient.java:655) ~[classes/:?]
2023-01-29 14:19:35 at org.codelibs.fess.es.client.SearchEngineClient.open(SearchEngineClient.java:286) ~[classes/:?]
2023-01-29 14:19:35 at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
2023-01-29 14:19:35 at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at org.lastaflute.di.util.LdiMethodUtil.invoke(LdiMethodUtil.java:49) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.assembler.AbstractMethodAssembler.invoke(AbstractMethodAssembler.java:59) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.assembler.DefaultInitMethodAssembler.assemble(DefaultInitMethodAssembler.java:40) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.deployer.SingletonComponentDeployer.assemble(SingletonComponentDeployer.java:57) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.deployer.SingletonComponentDeployer.deploy(SingletonComponentDeployer.java:41) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.deployer.SingletonComponentDeployer.init(SingletonComponentDeployer.java:35) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.ComponentDefImpl.init(ComponentDefImpl.java:85) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:434) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.8.3.jar:?]
2023-01-29 14:19:35 … 42 more
2023-01-29 14:19:35 Caused by: org.codelibs.fesen.FesenStatusException: Fesen exception [type=illegal_argument_exception, reason=request [//_cluster/health/] contains unrecognized parameters: [master_timeout], [timeout], [wait_for_active_shards], [wait_for_no_initializing_shards], [wait_for_no_relocating_shards], [wait_for_nodes], [wait_for_status]]
2023-01-29 14:19:35 at org.codelibs.fesen.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[fesen-engine-server-0.12.0.jar:0.12.0]
2023-01-29 14:19:35 at org.codelibs.fesen.client.action.HttpAction.toFesenException(HttpAction.java:138) ~[fesen-httpclient-0.12.0.jar:?]
2023-01-29 14:19:35 at org.codelibs.fesen.client.action.HttpClusterHealthAction.lambda$execute$0(HttpClusterHealthAction.java:47) ~[fesen-httpclient-0.12.0.jar:?]
2023-01-29 14:19:35 at org.codelibs.curl.CurlRequest.lambda$execute$4(CurlRequest.java:220) ~[curl4j-1.2.4.jar:?]
2023-01-29 14:19:35 at org.codelibs.curl.CurlRequest.lambda$connect$3(CurlRequest.java:199) ~[curl4j-1.2.4.jar:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinPool.scan(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) ~[?:?]
2023-01-29 14:19:35 Suppressed: org.codelibs.fesen.common.xcontent.XContentParseException: [1:576] [cluster_health_response] status doesn’t support values of type: VALUE_NUMBER
2023-01-29 14:19:35 at org.codelibs.fesen.common.xcontent.ObjectParser$FieldParser.assertSupports(ObjectParser.java:601) ~[fesen-x-content-0.12.0.jar:0.12.0]
2023-01-29 14:19:35 at org.codelibs.fesen.common.xcontent.ObjectParser.parse(ObjectParser.java:299) ~[fesen-x-content-0.12.0.jar:0.12.0]
2023-01-29 14:19:35 at org.codelibs.fesen.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:171) ~[fesen-x-content-0.12.0.jar:0.12.0]
2023-01-29 14:19:35 at org.codelibs.fesen.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:163) ~[fesen-x-content-0.12.0.jar:0.12.0]
2023-01-29 14:19:35 at org.codelibs.fesen.action.admin.cluster.health.ClusterHealthResponse.fromXContent(ClusterHealthResponse.java:350) ~[fesen-engine-server-0.12.0.jar:0.12.0]
2023-01-29 14:19:35 at org.codelibs.fesen.client.action.HttpClusterHealthAction.lambda$execute$0(HttpClusterHealthAction.java:44) ~[fesen-httpclient-0.12.0.jar:?]
2023-01-29 14:19:35 at org.codelibs.curl.CurlRequest.lambda$execute$4(CurlRequest.java:220) ~[curl4j-1.2.4.jar:?]
2023-01-29 14:19:35 at org.codelibs.curl.CurlRequest.lambda$connect$3(CurlRequest.java:199) ~[curl4j-1.2.4.jar:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinPool.scan(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) ~[?:?]
2023-01-29 14:19:35 at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) ~[?:?]
2023-01-29 14:19:35 Suppressed: org.codelibs.fesen.client.action.HttpAction$CurlResponseException: {“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“request [//_cluster/health/] contains unrecognized parameters: [master_timeout], [timeout], [wait_for_active_shards], [wait_for_no_initializing_shards], [wait_for_no_relocating_shards], [wait_for_nodes], [wait_for_status]”}],“type”:“illegal_argument_exception”,“reason”:“request [//_cluster/health/] contains unrecognized parameters: [master_timeout], [timeout], [wait_for_active_shards], [wait_for_no_initializing_shards], [wait_for_no_relocating_shards], [wait_for_nodes], [wait_for_status]”},“status”:400}
2023-01-29 14:21:35 Fess is not available.

I also used AWS OpenSearch inside VPC, then I can get IP of running elasticsearch instance inside “Network Interfaces”. This IP enables fess connection to elasticsearch.

My goal is to make connection between Fess and AWS OpenSearch independent of IP address. Can you confirm if this is possible?

What version is OpenSearch? Although Fess 13.12 is EOL-ed, it supports Elasticsearch 7.12.

I followed https://news.mynavi.jp/techplus/article/techp5557/ article, so used ElasticSearch 7.9 and Fess 13.12.

I also tried OpenSearch 2.3(latest), but gsuite plugin had problem with files scrape.
I think I also checked Fess 14.5.0, but the problem was the same.

I can check Elasticsearch 7.12, if you think it might help?

I wonder if issue is with OpenSearch AWS service implementation. Inside EC2 → Network interfaces tab, I can see that it creates 3 instances, where one is active, and its IP I use to connect.

Fess 13.12 is EOL-ed, so you need to use the latest version or ask for commercial support.

gsuite plugin had problem with files scrape.

What is the problem?

I used Fess 13.12, just to check if it will help. With 14.5 there is the same problem.

Using AWS OpenSearch service with OpenSearch 2.3 engine, gsuite had problem with files content for google sheet, docx, xlsx formats, for csv format there was no problem. I do not have logs but it was error returned by OpenSearch engine.

I solved my problem.
I used ES_HTTP_URL with https://search-fess-public-qnkgmdcjm5shywhgy2uj2ztmtm.eu-west-2.es.amazonaws.com/ link without https:// and it worked.