Issue with SSL certificate validation

Hello,

I have been setting up fess crawler in private network ( intranet ) to crawl secured sites.

While I had downloaded the sites certificates and imported in local keystore using keytool. It worked fine for one site and facing issues with additional sites.

I have tested with curl that I can connect to the site with no error. Any suggestion how this can be sorted out. Pl see the following error. Would greatly appreciate your quick response. Thanks

org.codelibs.fess.crawler.exception.CrawlingAccessException: I/O exception(PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target): https://hidden.intranet.group/rbbhidden/
at org.codelibs.fess.crawler.client.http.HcHttpClient.processHttpMethod(HcHttpClient.java:829)
at org.codelibs.fess.crawler.client.http.HcHttpClient.doHttpMethod(HcHttpClient.java:654)
at org.codelibs.fess.crawler.client.http.HcHttpClient.doGet(HcHttpClient.java:617)
at org.codelibs.fess.crawler.client.AbstractCrawlerClient.execute(AbstractCrawlerClient.java:132)
at org.codelibs.fess.crawler.client.FaultTolerantClient.execute(FaultTolerantClient.java:67)

Try this post :slight_smile:

1 Like

Hello @shinsuke,
It works well for web crawling but how to do for datastore crawling…for example for ConfluenceDataStore i get the same issue related to ssl.

org.codelibs.fess.ds.atlassian.AtlassianDataStoreException: Failed to access org.codelibs.fess.ds.atlassian.api.confluence.content.GetContentsRequest@2206cc52
at org.codelibs.fess.ds.atlassian.api.confluence.content.GetContentsRequest.execute(GetContentsRequest.java:92)
at org.codelibs.fess.ds.atlassian.api.confluence.ConfluenceClient.getContents(ConfluenceClient.java:96)
at org.codelibs.fess.ds.atlassian.ConfluenceDataStore.storeData(ConfluenceDataStore.java:66)
at org.codelibs.fess.ds.AbstractDataStore.store(AbstractDataStore.java:121)
at org.codelibs.fess.helper.DataIndexHelper$DataCrawlingThread.process(DataIndexHelper.java:216)
at org.codelibs.fess.helper.DataIndexHelper$DataCrawlingThread.run(DataIndexHelper.java:202)
Caused by: org.codelibs.fess.ds.atlassian.AtlassianDataStoreException: Failed to access https://XXX.XXX.com/rest/api/latest/content
at org.codelibs.fess.ds.atlassian.api.AtlassianRequest.getCurlResponse(AtlassianRequest.java:118)
at org.codelibs.fess.ds.atlassian.api.AtlassianRequest.getCurlResponse(AtlassianRequest.java:73)
at org.codelibs.fess.ds.atlassian.api.confluence.content.GetContentsRequest.execute(GetContentsRequest.java:86)
… 5 more
Caused by: org.codelibs.curl.CurlException: Failed to process a request.
at org.codelibs.curl.CurlRequest.lambda$execute$5(CurlRequest.java:249)
at org.codelibs.curl.CurlRequest.lambda$connect$3(CurlRequest.java:211)
at org.codelibs.curl.CurlRequest.connect(CurlRequest.java:221)
at org.codelibs.curl.CurlRequest.execute(CurlRequest.java:248)
at org.codelibs.fess.ds.atlassian.api.AtlassianRequest.getCurlResponse(AtlassianRequest.java:116)
… 7 more
Caused by: org.codelibs.curl.CurlException: Failed to access to https://XXX.XXX.com/rest/api/latest/content?expand=space%2Cversion%2Cbody.view&start=0&limit=25
… 11 more
Caused by: org.codelibs.curl.CurlException: Failed to access the response.
at org.codelibs.curl.CurlRequest$RequestProcessor.accept(CurlRequest.java:290)
at org.codelibs.curl.CurlRequest$RequestProcessor.accept(CurlRequest.java:267)
at org.codelibs.curl.CurlRequest.lambda$connect$3(CurlRequest.java:209)
… 10 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1500)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1415)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:421)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURL…
Thanks :slight_smile:

where i can put this config in this CASE?

ignoreSslCertificate is not for DataStore.
SSL problem for confluence is Unable to connect to SSL services due to “PKIX Path Building Failed” error.
To set JVM option for a crawler, you can add jvmOptions to the crawler setting as below.

return container.getComponent("crawlJob").logLevel("info").gcLogging().jvmOptions("-D...").execute(executor);