office365クロール時のプロキシ設定

myk · February 3, 2022, 11:14am

以下の手順をもとにoffice365のクローリングをしたところエラーがでます。

【第31回】Office 365のクロール

プロキシ環境下で動かしていまして、
プロキシの設定箇所がわからず、以下の設定をしましたが、 ClassNotFoundが出力されます。
何か解決手段はございますでしょうか？

[プロキシ設定]
fess\app\WEB-INF\classes\fess_config.properties
http.proxy.host=〇〇〇
http.proxy.port=〇〇〇
http.proxy.username=
http.proxy.password=

[クローリングエラー]
ClassNotFound
io.netty.handler.proxy.proxyconnectexception

よろしくお願いいたします。

shinsuke · February 3, 2022, 11:55am

例外の情報は例外名だけではわからないので、スタックトレースもお願いします。

myk · February 3, 2022, 12:05pm

失礼いたしました。以下になります。
よろしくお願いいたします。

ファイル：fess-crawler.log
2022-02-03 16:19:15,070 [reactor-http-nio-2] WARN An exception was thrown by reactor.netty.transport.TransportConnector$$Lambda$1043/0x0000000100904440.operationComplete()
java.lang.NoClassDefFoundError: io/netty/handler/proxy/ProxyConnectException
at com.azure.core.http.netty.NettyAsyncHttpClient.lambda$send$1(NettyAsyncHttpClient.java:97) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.util.retry.RetrySpec.lambda$generateCompanion$5(RetrySpec.java:365) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:375) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:491) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:299) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.Operators.error(Operators.java:198) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.MonoError.subscribe(MonoError.java:53) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.Mono.subscribe(Mono.java:4338) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.MonoSingle$SingleSubscriber.onError(MonoSingle.java:150) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:225) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:274) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:414) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:491) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:299) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$ClientTransportSubscriber.onError(HttpClientConnect.java:305) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onError(DefaultPooledConnectionProvider.java:173) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:444) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.lambda$drainLoop$9(SimpleDequePool.java:399) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.onError(DefaultPooledConnectionProvider.java:555) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:225) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:274) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:414) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:251) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:491) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:299) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:190) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:517) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.netty.transport.TransportConnector$MonoChannelPromise.setFailure(TransportConnector.java:471) ~[fess-ds-office365-13.16.0.jar:?]
at reactor.netty.transport.TransportConnector.lambda$doConnect$7(TransportConnector.java:206) ~[fess-ds-office365-13.16.0.jar:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) [netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) [netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) [netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) [netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) [netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) [netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) [netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.65.Final.jar:4.1.65.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.ClassNotFoundException: io.netty.handler.proxy.ProxyConnectException
at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
… 70 more

shinsuke · February 3, 2022, 12:47pm

ありがとうございます。そのクラスを追加してみたものを fess-ds-office365-13.16.1-20220203.123855-2.jar に置きました。

myk · February 4, 2022, 5:35am

早急にご対応いただき、ありがとうございます。
無事、前述のエラーは解消したのですが、別のエラーが出て困っております。
何かご知見はございますでしょうか？

ログを張ろうにもcodelibsのエラーが出て張れないため、抜粋いたします。
よろしくお願いいたします。

2022-02-04 12:10:44,446 [ForkJoinPool.commonPool-worker-3] ERROR [Correlation ID: ●●●●] Execution of class com.microsoft.aad.msal4j.AcquireTokenSilentSupplier failed.
com.microsoft.aad.msal4j.MsalClientException: reactor.core.Exceptions$ReactiveException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: login.microsoftonline.com/XXX.XXX.XXX.XXX:443
at com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:53) ~[fess-ds-office365-13.16.1-20220203.123855-2.jar:?]

～中略～
Caused by: reactor.core.Exceptions$ReactiveException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: login.microsoftonline.com/XXX.XXX.XXX.XXX:443
at reactor.core.Exceptions.propagate(Exceptions.java:392) ~[fess-ds-office365-13.16.1-20220203.123855-2.jar:?]

shinsuke · February 4, 2022, 11:57am

断片的なスタックトレースでは何とも言えないです…。gistとかに置いてもらうとかでも良いと思います。

myk · February 28, 2022, 8:30am

ログは申し訳ございませんが、割愛させてください。

今回プロキシ経由の設定を施しましたが、
クローリングに際して、Fessから外部へプロキシを経由せず、通信している処理はございますでしょうか。

何卒、よろしくお願いいたします。

shinsuke · March 2, 2022, 11:16pm

判断に必要な情報がないのでよくわかりませんが、他にクロール設定とかなければないと思います。