Question

上一节完成了爬虫模块的编写，下面开始编写Pipeline，主要是完成Item到MongoDB的存储，分成两个集合进行存储：UserInfo和Relation。代码如下：

  class ZhihucrawlPipeline(object):
  
     def __init__(self, mongo_uri, mongo_db):
       self.mongo_uri = mongo_uri
       self.mongo_db = mongo_db
  
     @classmethod
     def from_crawler(cls, crawler):
       return cls(
            mongo_uri=crawler.settings.get('MONGO_URI'),
            mongo_db=crawler.settings.get('MONGO_DATABASE', 'zhihu')
       )
  
     def open_spider(self, spider):
       self.client = pymongo.MongoClient(self.mongo_uri)
       self.db = self.client[self.mongo_db]
  
     def close_spider(self, spider):
       self.client.close()
  
     def process_item(self, item, spider):
  
       if isinstance(item,UserInfoItem):
            self._process_user_item(item)
       else:
            self._process_relation_item(item)
       return item
  
     def _process_user_item(self,item):
       self.db.UserInfo.insert(dict(item))
  
  
     def _process_relation_item(self,item):
       self.db.Relation.insert(dict(item))

代码和13.3节的非常类似，首先从Settings中加载MONGO_URI和MONGO_DATABASE，初始化连接URL和数据库名称，在process_item方法中判断item的类型，然后使用_process_user_item和_process_user_item方法存储到不同的集合中。下面在Settings中设置MONGO_URI，MONGO_DATABASE和激活Pipeline：

  MONGO_URI = 'mongodb:// 127.0.0.1:27017/'
  MONGO_DATABASE='zhihu'
  ITEM_PIPELINES = {
     'zhihuCrawl.pipelines.ZhihucrawlPipeline': 300,
  }