【bug】当出现特殊换字符时 lxml报错: input conversion failed due to

Question

错误提示：encoding error : input conversion failed due to
采集一个页面，页面中有无法转码的特殊字符，一般情况下用

.decode('gbk','ignore') 或者.decode('gbk', 'replace') 即可忽略

可是用response.doc()报错且无法获取完整的页面（response.text和response.content 没有问题）

环境版本：win10 32位 pyspider0.3.7
采集的页面无法正确获取编码，所以自己指定了GBK

自己胡乱改了一下源码，问题解决:
打开文件pyspider/libs/response.py
找到：

def etree(self):
        """Returns a lxml object of the response's content that can be selected by xpath"""
        if not hasattr(self, '_elements'):

            try:
                parser = lxml.html.HTMLParser(encoding=self.encoding)
                self._elements = lxml.html.fromstring(self.content, parser=parser)

            except LookupError:
                # lxml would raise LookupError when encoding not supported
                # try fromstring without encoding instead.
                # on windows, unicode is not availabe as encoding for lxml
                self._elements = lxml.html.fromstring(self.content)
        if isinstance(self._elements, lxml.etree._ElementTree):
            self._elements = self._elements.getroot()
        return self._elements

改成：

def etree(self):
        """Returns a lxml object of the response's content that can be selected by xpath"""
        if not hasattr(self, '_elements'):
            try:
                content = self.content.decode(self.encoding, 'replace')
            except LookupError:
                content = self.content.decode('utf-8', 'replace')
                
            try:
                parser = lxml.html.HTMLParser(encoding=self.encoding)
                self._elements = lxml.html.fromstring(content, parser=parser)

            except LookupError:
                # lxml would raise LookupError when encoding not supported
                # try fromstring without encoding instead.
                # on windows, unicode is not availabe as encoding for lxml
                self._elements = lxml.html.fromstring(content)
        if isinstance(self._elements, lxml.etree._ElementTree):
            self._elements = self._elements.getroot()
        return self._elements

Answer 1

如果不想改源码，可以 response.content = response.content.decode('gbk', 'replace').encode('gbk')