我在用webmagic抓取https网站时报错

发布于 2021-11-27 18:17:33 字数 6552 浏览 341 评论 2

@黄亿华你好，想跟你请教个问题：

我在使用你写的webmagic时如果抓去普通http网站是可以的，但是抓取https网站就出错。

16-03-02 21:45:25,487 INFO org.springframework.context.support.ClassPathXmlApplicationContext(AbstractApplicationContext.java:495) ## Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@fa3ac1: startup date [Wed Mar 02 21:45:25 CST 2016]; root of context hierarchy
16-03-02 21:45:25,582 INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader(XmlBeanDefinitionReader.java:315) ## Loading XML bean definitions from file [C:Documents and SettingsAdministratorworkspace_javacowpu-crawlerWebRootWEB-INFclassesspringapplicationContext-myBatis.xml]
16-03-02 21:45:25,701 INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader(XmlBeanDefinitionReader.java:315) ## Loading XML bean definitions from file [C:Documents and SettingsAdministratorworkspace_javacowpu-crawlerWebRootWEB-INFclassesspringapplicationContext.xml]
16-03-02 21:45:26,037 INFO org.springframework.beans.factory.support.DefaultListableBeanFactory(DefaultListableBeanFactory.java:557) ## Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@51052d: defining beans [sqlSessionFactory,org.mybatis.spring.mapper.MapperScannerConfigurer#0,dataSource,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,jobCrawler,HongLingCapitalConsolePipeline,hongLingCapitalDAO,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0]; root of factory hierarchy
16-03-02 21:45:27,683 INFO us.codecraft.webmagic.model.OOSpider(Spider.java:292) ## Spider https://www.my089.com started!
16-03-02 21:45:27,684 INFO us.codecraft.webmagic.downloader.HttpClientDownloader(HttpClientDownloader.java:77) ## downloading page https://www.my089.com/Loan/default.aspx
16-03-02 21:45:28,140 WARN us.codecraft.webmagic.downloader.HttpClientDownloader(HttpClientDownloader.java:109) ## download page https://www.my089.com/Loan/default.aspx error
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1591)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:187)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:181)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:975)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:123)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:516)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:454)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:884)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1096)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1123)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1107)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:275)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:254)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at us.codecraft.webmagic.downloader.HttpClientDownloader.download(HttpClientDownloader.java:95)
at us.codecraft.webmagic.Spider.processRequest(Spider.java:373)
at us.codecraft.webmagic.Spider$1.run(Spider.java:308)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:285)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:191)
at sun.security.validator.Validator.validate(Validator.java:218)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:126)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:209)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:249)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:954)
... 25 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:280)
... 31 more