(from github.com/guenther-orth)
I crawl a directory with over 4TB PDFs. After a long time the crawler got an error:
2019-03-03 19:25:39,735 [IndexUpdater] WARN Failed to access data. Retry to access.. 1
org.codelibs.fess.crawler.exception.EsAccessException: failure in bulk execution:
[0]: index [.crawler.data], type [data], id [20190303153026-1.ZmlsZTovbW50L0FyY2hpdi9PQkgvTmV1YmF1XzIwMDcyNi0yMDA4ODkvMzAwMDIyMjNfQ29sbGZhXzIwMDczOF8xLnBkZg], message [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
I stoped fess and try to restart fess, but fess no longer starts:
INFO: Boot successful: url -> http://localhost:8080
Mar 04, 2019 6:03:27 PM org.apache.catalina.loader.WebappClassLoaderBase checkStateForResourceLoading
INFO: Illegal access: this web application instance has been stopped already. Could not load [org.elasticsearch.transport.TransportKeepAlive$1]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.elasticsearch.transport.TransportKeepAlive$1]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1363)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1351)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1211)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1173)
at org.elasticsearch.transport.TransportKeepAlive.sendPing(TransportKeepAlive.java:119)
at org.elasticsearch.transport.TransportKeepAlive.access$500(TransportKeepAlive.java:48)
at org.elasticsearch.transport.TransportKeepAlive$ScheduledPing.doRunInLifecycle(TransportKeepAlive.java:180)
at org.elasticsearch.common.util.concurrent.AbstractLifecycleRunnable.doRun(AbstractLifecycleRunnable.java:67)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Elasticsearch is running and there are no errors in the logfile /var/log/elasticsearch/elasticsearch.log
Can anybody help me to find the issue?
Thanks a lot
Guenther
(from github.com/marevol)
Elasticsearch checks disk space.
See elasticsearch doc.
(from github.com/guenther-orth)
Thank you very much for this information. I’m checking the hard drive space. It’s 75% free. In elasticsearch.yml I update the threshold to 98%.
But fess won’t start. I find some information about an error in /var/log/fess/fess.log. I think there is a file lock, but I can’t find it to remove it.
How can I reset fess without a new installation? Where are the files with the status information of the crawler?
Thanks a lot
Guenther
/var/log/fess/fess.log:
2019-03-05 09:48:39,814 [main] ERROR Failed to initialize Lasta Di.
org.lastaflute.di.exception.ContainerInitFailureException: Look! Read the message below.
/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
Failed to initialize the container.
[Path]
app.xml
[Namespace]
null
* * * * * * * * * */
at org.lastaflute.di.core.meta.impl.LaContainerImpl.throwContainerInitFailureException(LaContainerImpl.java:463) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:415) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.factory.SingletonLaContainerFactory.init(SingletonLaContainerFactory.java:66) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.web.container.WebLastaContainerInitializer.doInitContainer(WebLastaContainerInitializer.java:81) ~[lastaflute-1.0.9.jar:?]
at org.lastaflute.web.container.WebLastaContainerInitializer.initialize(WebLastaContainerInitializer.java:45) ~[lastaflute-1.0.9.jar:?]
at org.lastaflute.web.servlet.filter.LastaPrepareFilter.initializeContainer(LastaPrepareFilter.java:136) [lastaflute-1.0.9.jar:?]
at org.lastaflute.web.servlet.filter.LastaPrepareFilter.init(LastaPrepareFilter.java:94) [lastaflute-1.0.9.jar:?]
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:270) [classes/:?]
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:251) [classes/:?]
at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:102) [classes/:?]
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4511) [classes/:?]
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5157) [classes/:?]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [classes/:?]
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1382) [classes/:?]
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1372) [classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [classes/:?]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) [?:1.8.0_181]
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:907) [classes/:?]
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831) [classes/:?]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [classes/:?]
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1382) [classes/:?]
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1372) [classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [classes/:?]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) [?:1.8.0_181]
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:907) [classes/:?]
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) [classes/:?]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [classes/:?]
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423) [classes/:?]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [classes/:?]
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:933) [classes/:?]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [classes/:?]
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:398) [classes/:?]
at org.dbflute.tomcat.TomcatBoot.startServer(TomcatBoot.java:681) [classes/:?]
at org.dbflute.tomcat.TomcatBoot.go(TomcatBoot.java:385) [classes/:?]
at org.dbflute.tomcat.TomcatBoot.bootAwait(TomcatBoot.java:330) [classes/:?]
at org.codelibs.fess.FessBoot.main(FessBoot.java:90) [classes/:?]
Caused by: org.lastaflute.di.core.exception.IllegalMethodRuntimeException: [ESSR0060]Can not configure method(init) of org.codelibs.fess.helper.SuggestHelper, because org.codelibs.fess.suggest.exception.SuggestSettingsException: Failed to delete all from array.
at org.lastaflute.di.core.assembler.AbstractMethodAssembler.invoke(AbstractMethodAssembler.java:61) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.assembler.DefaultInitMethodAssembler.assemble(DefaultInitMethodAssembler.java:40) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.assemble(SingletonComponentDeployer.java:57) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.deploy(SingletonComponentDeployer.java:41) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.init(SingletonComponentDeployer.java:35) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.ComponentDefImpl.init(ComponentDefImpl.java:85) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:434) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.7.7.jar:?]
... 36 more
Caused by: org.codelibs.fess.suggest.exception.SuggestSettingsException: Failed to delete all from array.
at org.codelibs.fess.suggest.settings.ArraySettings.deleteKeyFromArray(ArraySettings.java:166) ~[fess-suggest-6.6.0.jar:?]
at org.codelibs.fess.suggest.settings.ArraySettings.delete(ArraySettings.java:85) ~[fess-suggest-6.6.0.jar:?]
at org.codelibs.fess.helper.SuggestHelper.init(SuggestHelper.java:105) ~[classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at org.lastaflute.di.util.LdiMethodUtil.invoke(LdiMethodUtil.java:49) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.assembler.AbstractMethodAssembler.invoke(AbstractMethodAssembler.java:59) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.assembler.DefaultInitMethodAssembler.assemble(DefaultInitMethodAssembler.java:40) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.assemble(SingletonComponentDeployer.java:57) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.deploy(SingletonComponentDeployer.java:41) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.init(SingletonComponentDeployer.java:35) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.ComponentDefImpl.init(ComponentDefImpl.java:85) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:434) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.7.7.jar:?]
... 36 more
Caused by: org.codelibs.fess.suggest.exception.SuggesterException: Failed to exec delete by query.
at org.codelibs.fess.suggest.util.SuggestUtil.deleteByQuery(SuggestUtil.java:257) ~[fess-suggest-6.6.0.jar:?]
at org.codelibs.fess.suggest.settings.ArraySettings.deleteKeyFromArray(ArraySettings.java:164) ~[fess-suggest-6.6.0.jar:?]
at org.codelibs.fess.suggest.settings.ArraySettings.delete(ArraySettings.java:85) ~[fess-suggest-6.6.0.jar:?]
at org.codelibs.fess.helper.SuggestHelper.init(SuggestHelper.java:105) ~[classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at org.lastaflute.di.util.LdiMethodUtil.invoke(LdiMethodUtil.java:49) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.assembler.AbstractMethodAssembler.invoke(AbstractMethodAssembler.java:59) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.assembler.DefaultInitMethodAssembler.assemble(DefaultInitMethodAssembler.java:40) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.assemble(SingletonComponentDeployer.java:57) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.deploy(SingletonComponentDeployer.java:41) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.init(SingletonComponentDeployer.java:35) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.ComponentDefImpl.init(ComponentDefImpl.java:85) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:434) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.7.7.jar:?]
... 36 more
Caused by: org.codelibs.fess.suggest.exception.SuggesterException: failure in bulk execution:
[0]: index [.suggest_array.fess], type [fess], id [a2V5OnN1cHBvcnRlZEZpZWxkc3ZhbHVlOmNvbnRlbnQ=], message [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
[1]: index [.suggest_array.fess], type [fess], id [a2V5OnN1cHBvcnRlZEZpZWxkc3ZhbHVlOnRpdGxl], message [ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]]
at org.codelibs.fess.suggest.util.SuggestUtil.deleteByQuery(SuggestUtil.java:250) ~[fess-suggest-6.6.0.jar:?]
at org.codelibs.fess.suggest.settings.ArraySettings.deleteKeyFromArray(ArraySettings.java:164) ~[fess-suggest-6.6.0.jar:?]
at org.codelibs.fess.suggest.settings.ArraySettings.delete(ArraySettings.java:85) ~[fess-suggest-6.6.0.jar:?]
at org.codelibs.fess.helper.SuggestHelper.init(SuggestHelper.java:105) ~[classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at org.lastaflute.di.util.LdiMethodUtil.invoke(LdiMethodUtil.java:49) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.assembler.AbstractMethodAssembler.invoke(AbstractMethodAssembler.java:59) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.assembler.DefaultInitMethodAssembler.assemble(DefaultInitMethodAssembler.java:40) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.assemble(SingletonComponentDeployer.java:57) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.deploy(SingletonComponentDeployer.java:41) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.deployer.SingletonComponentDeployer.init(SingletonComponentDeployer.java:35) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.ComponentDefImpl.init(ComponentDefImpl.java:85) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.doInit(LaContainerImpl.java:434) ~[lasta-di-0.7.7.jar:?]
at org.lastaflute.di.core.meta.impl.LaContainerImpl.init(LaContainerImpl.java:413) ~[lasta-di-0.7.7.jar:?]
... 36 more
(from github.com/guenther-orth)
Yes, you are right: the file system is too full for elasticsearch. I deleted the entire index with
curl -XDELETE http://127.0.0.1:9200/_all
and then I could start fess again.
The problem is that the file system still has 25% free, which is only 6GB. But the limit for elasticsearch is 10 GB.
The configs in /etc/elasticsearch/elasticsearch.yml like cluster.routing.allocation.disk.* does not work.
Thanks a lot for your help and your patience
Guenther