Failed to crawl https://ir.crystalgroup.com/sc/global/home.php

(from github.com/riceming)
Fess failed to crawl https://ir.crystalgroup.com/sc/global/home.php, I have checked the root cert of the domain with the cacerts file for my jvm, it is actually there. What else could cause the problem?

org.codelibs.fess.crawler.exception.CrawlingAccessException: I/O exception(sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target): https://ir.crystalgroup.com/sc/global/home.php at org.codelibs.fess.crawler.client.http.HcHttpClient.processHttpMethod(HcHttpClient.java:821) at org.codelibs.fess.crawler.client.http.HcHttpClient.doHttpMethod(HcHttpClient.java:622) at org.codelibs.fess.crawler.client.http.HcHttpClient.doGet(HcHttpClient.java:581) at org.codelibs.fess.crawler.client.AbstractCrawlerClient.execute(AbstractCrawlerClient.java:135) at org.codelibs.fess.crawler.client.FaultTolerantClient.execute(FaultTolerantClient.java:67) at org.codelibs.fess.crawler.CrawlerThread.run(CrawlerThread.java:164) at java.lang.Thread.run(Thread.java:745) Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.codelibs.fess.crawler.client.http.HcHttpClient.executeHttpClient(HcHttpClient.java:845) at org.codelibs.fess.crawler.client.http.HcHttpClient.processHttpMethod(HcHttpClient.java:659) ... 6 more Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) at sun.security.validator.Validator.validate(Validator.java:260) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ... 26...

(from github.com/marevol)
I think it’s a problem on a certificate setting.
Checking it on ssltest, “This server’s certificate chain is incomplete.” is displayed.

(from github.com/riceming)
Thanks, since the certificate is provided by client, the issue may not able to address in short time.
At mean while, can I config Fess to bypass such error and crawl the page?

(from github.com/marevol)
No simple solution. I think you need to implement HttpClient’s class to do that.

(from github.com/riceming)
Hi there, there is another website which could not be crawled.
https://www.lifung.com

I have checked with SSLTEST and returning A grade, with no chain error
https://www.ssllabs.com/ssltest/analyze.html?d=www.lifung.com&s=54.192.142.207&latest

What could the problem be?

(from github.com/marevol)
It seems to work on my environment.
https://www.lifung.com” is crawled&indexed.

(from github.com/riceming)
It is my fault, i set -Djsse.enableSNIExtension=false\n\ before
It can be crawled when I change -Djsse.enableSNIExtension=true\n\