unable to restart fess

discuss · May 28, 2019, 8:43am

(from github.com/wumpz)
I need some kind of starter. I am using the latest docker image with elasticsearch included. At the moment there is no configuration change except a web crawler.

Fess indexed successfully. Something about 60000 Docs, Index size 600MB.

Now I am not able to restart fess due to

ElasticSearch does not start anymore. Fess is restarting itself again and again to find a running elasticsearch.

I can’t see any exception warning or stuff. Where should I look for hints?

discuss · May 28, 2019, 1:00pm

(from github.com/marevol)
Did you check the console log printed by docker?
I think elasticsearch failed to start.

discuss · May 28, 2019, 1:38pm

(from github.com/wumpz)
Do you mean this:

OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
fessserver_1 | Starting Elasticsearch Server:.
fessserver_1 | Checking Elasticsearch status #1
fessserver_1 | [2019-05-28T13:36:34,704][INFO ][o.e.e.NodeEnvironment ] [3642823c5c4e] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/sdb1)]], net usable_space [48.4gb], net total_space [98.3gb], types [ext4]
fessserver_1 | [2019-05-28T13:36:34,712][INFO ][o.e.e.NodeEnvironment ] [3642823c5c4e] heap size [1gb], compressed ordinary object pointers [true]
fessserver_1 | [2019-05-28T13:36:34,934][INFO ][o.e.n.Node ] [3642823c5c4e] node name [3642823c5c4e], node ID [sSFniH4pQa639J7Fr-Qyjw]
fessserver_1 | [2019-05-28T13:36:34,935][INFO ][o.e.n.Node ] [3642823c5c4e] version[7.0.1], pid[84], build[oss/deb/e4efcb5/2019-04-29T12:56:03.145736Z], OS[Linux/4.4.0-145-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.3/11.0.3+1-Debian-1bpo91]
fessserver_1 | [2019-05-28T13:36:34,936][INFO ][o.e.n.Node ] [3642823c5c4e] JVM home [/usr/lib/jvm/java-11-openjdk-amd64]
fessserver_1 | [2019-05-28T13:36:34,937][INFO ][o.e.n.Node ] [3642823c5c4e] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:InitiatingHeapOccupancyPercent=75, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Dio.netty.allocator.type=unpooled, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb, -Des.bundled_jdk=true]
fessserver_1 | Checking Elasticsearch status #2
fessserver_1 | Checking Elasticsearch status #3
fessserver_1 | [2019-05-28T13:36:36,744][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [aggs-matrix-stats]
fessserver_1 | [2019-05-28T13:36:36,746][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [analysis-common]
fessserver_1 | [2019-05-28T13:36:36,747][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [ingest-common]
fessserver_1 | [2019-05-28T13:36:36,748][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [ingest-geoip]
fessserver_1 | [2019-05-28T13:36:36,750][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [ingest-user-agent]
fessserver_1 | [2019-05-28T13:36:36,751][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [lang-expression]
fessserver_1 | [2019-05-28T13:36:36,752][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [lang-mustache]
fessserver_1 | [2019-05-28T13:36:36,753][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [lang-painless]
fessserver_1 | [2019-05-28T13:36:36,754][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [mapper-extras]
fessserver_1 | [2019-05-28T13:36:36,755][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [parent-join]
fessserver_1 | [2019-05-28T13:36:36,756][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [percolator]
fessserver_1 | [2019-05-28T13:36:36,757][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [rank-eval]
fessserver_1 | [2019-05-28T13:36:36,757][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [reindex]
fessserver_1 | [2019-05-28T13:36:36,758][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [repository-url]
fessserver_1 | [2019-05-28T13:36:36,758][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded module [transport-netty4]
fessserver_1 | [2019-05-28T13:36:36,760][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded plugin [analysis-extension]
fessserver_1 | [2019-05-28T13:36:36,760][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded plugin [analysis-fess]
fessserver_1 | [2019-05-28T13:36:36,761][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded plugin [configsync]
fessserver_1 | [2019-05-28T13:36:36,761][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded plugin [dataformat]
fessserver_1 | [2019-05-28T13:36:36,762][INFO ][o.e.p.PluginsService ] [3642823c5c4e] loaded plugin [minhash]
fessserver_1 | Checking Elasticsearch status #4
fessserver_1 | Checking Elasticsearch status #5
fessserver_1 | Checking Elasticsearch status #6
fessserver_1 | Checking Elasticsearch status #7
fessserver_1 | Checking Elasticsearch status #8
fessserver_1 | Checking Elasticsearch status #9
fessserver_1 | Checking Elasticsearch status #10
fessserver_1 | Checking Elasticsearch status #11
fessserver_1 | Checking Elasticsearch status #12
fessserver_1 | Checking Elasticsearch status #13
fessserver_1 | Checking Elasticsearch status #14
fessserver_1 | Checking Elasticsearch status #15
fessserver_1 | Checking Elasticsearch status #16
fessserver_1 | Checking Elasticsearch status #17
fessserver_1 | Checking Elasticsearch status #18

discuss · May 28, 2019, 2:09pm

(from github.com/wumpz)
And after checking for elasticsearch til #60, it restarts fess again.

discuss · May 28, 2019, 9:01pm

(from github.com/marevol)
Does your server have enough memory?
OOM Killer might kill the process.

discuss · May 29, 2019, 5:25am

(from github.com/wumpz)
I started the docker container without any memory limitations. Therefore the standard configuration should be in place. By the way, this server runs with 20GB ram.

However, shouldn’t there be some kind of error message if the process is killed?

discuss · May 29, 2019, 9:15pm

(from github.com/marevol)
What command did you run?

discuss · May 30, 2019, 8:47am

(from github.com/wumpz)
It is part of a docker compose file. I did not apply a command here. Therefore the one in your image applies. Right?

discuss · May 31, 2019, 6:05am

(from github.com/wumpz)
This is the part of my docker-compose file, that starts fess.

fessserver:
image: codelibs/fess:latest
restart: always
volumes:
- fess_config:/opt/fess
- elasticsearch_config:/etc/elasticsearch
- elasticsearch_data:/var/lib/elasticsearch
ports:
- “8083:8080”

discuss · June 2, 2019, 1:32pm

(from marevol (Shinsuke Sugaya) · GitHub)

elasticsearch_config:/etc/elasticsearch

It’s better to check configuration files in the above.

discuss · June 3, 2019, 7:54am

(from github.com/wumpz)
I attached the three files within this directory. I skipped the keystore file.

fess_es_config.zip

Since I did not change anything, except over FESS web gui, I assume this is the standard configuration.

discuss · June 6, 2019, 12:56pm

(from github.com/wumpz)
So where can I look next for some problems?

discuss · June 6, 2019, 1:09pm

(from github.com/marevol)
Did you check /var/log/messages or syslog in the host?

discuss · June 7, 2019, 12:14pm

(from github.com/wumpz)
Found a hs_err - file.

hs_err_pid84.zip

Could this something Java 11 problematic using elasticsearch be:

Event: 17.265 Thread 0x00007f2364013800 Exception <a 'java/lang/IncompatibleClassChangeError'{0x00000000ff7134b0}: Found class java.lang.Object, but interface was expected> (0x00000000ff7134b0) thrown at [/build/openjdk-11-1uPVZa/openjdk-11-11.0.3+1/src/hotspot/share/interpreter/linkResolver.cp
Event: 17.346 Thread 0x00007f2364013800 Exception <a 'java/lang/IncompatibleClassChangeError'{0x00000000ff443d60}: Found class java.lang.Object, but interface was expected> (0x00000000ff443d60) thrown at [/build/openjdk-11-1uPVZa/openjdk-11-11.0.3+1/src/hotspot/share/interpreter/linkResolver.cp
Event: 17.366 Thread 0x00007f2364013800 Exception <a 'java/lang/IncompatibleClassChangeError'{0x00000000ff4f8fb8}: Found class java.lang.Object, but interface was expected> (0x00000000ff4f8fb8) thrown at [/build/openjdk-11-1uPVZa/openjdk-11-11.0.3+1/src/hotspot/share/interpreter/linkResolver.cp
Event: 17.578 Thread 0x00007f2364013800 Exception <a 'java/lang/NoSuchMethodError'{0x00000000fee8bff0}: java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V> (0x00000000fee8bff0) thrown at [/build/openjdk-11-1uPVZa/op

discuss · June 12, 2019, 5:18am

(from github.com/wumpz)
Reverted back to version 12.6.1, rebuilt the complete index.

Now fess restarts without any problems.

Curiously at the same directory fess finds nearly 5000 documents less.

discuss · June 20, 2019, 9:02am

(from github.com/wumpz)
So the older version runs without any problems. I stick with it. Thx for your help.