webmagic抓取离散形式的很多网页

Question

@黄亿华 你好，想跟你请教个问题：

我用webmagic抓取股票数据，类似

http://finance.sina.com.cn/realstock/company/sz000622/nc.shtml

每一个地址中只有股票代码不一样，上面的是sz000622.

没有办法在PageProcessor中动态加入新的网址，我是从一个文本文件中读取股票代码然后组合成URL．

当我抓的时候每次从文本读一行(一个股票代码)，组合url，然后

Spider.create(processor).scheduler(new FileCacheQueueScheduler(spiderWorkingDir+System.currentTimeMillis())).pipeline(new FilePipeline("/tmp")).thread(1).run();

这句话的外层是个循环．当抓取进行中的时候出现了错误：

14-01-08 18:04:09,138 WARN  us.codecraft.webmagic.pipeline.FilePipeline(FilePipeline.java:56) ## write file error
java.io.FileNotFoundException: /tmp/finance.sina.com.cn/4d90238b23d42884d5e164edabb47f5a.html (打开的文件过多)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
at java.io.FileWriter.<init>(FileWriter.java:90)
at us.codecraft.webmagic.pipeline.FilePipeline.process(FilePipeline.java:41)
at us.codecraft.webmagic.Spider.processRequest(Spider.java:384)
at us.codecraft.webmagic.Spider$1.run(Spider.java:304)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at us.codecraft.webmagic.Spider.run(Spider.java:300)
at com.github.xuchaoo.basedata.Main.main(Main.java:56)
14-01-08 18:04:14,141 INFO  us.codecraft.webmagic.scheduler.FileCacheQueueScheduler(FileCacheQueueScheduler.java:100) ## init cache file /tmp/stock1389175454139/finance.sina.com.cn.urls.txt
Exception in thread "main" java.lang.RuntimeException: init cache scheduler error
at us.codecraft.webmagic.scheduler.FileCacheQueueScheduler.initWriter(FileCacheQueueScheduler.java:88)
at us.codecraft.webmagic.scheduler.FileCacheQueueScheduler.init(FileCacheQueueScheduler.java:68)
at us.codecraft.webmagic.scheduler.FileCacheQueueScheduler.push(FileCacheQueueScheduler.java:149)
at us.codecraft.webmagic.Spider.initComponent(Spider.java:277)
at us.codecraft.webmagic.Spider.run(Spider.java:287)
at com.github.xuchaoo.basedata.Main.main(Main.java:56)
Caused by: java.io.FileNotFoundException: /tmp/stock1389175454139/finance.sina.com.cn.urls.txt (打开的文件过多)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:142)
at java.io.FileWriter.<init>(FileWriter.java:78)
at us.codecraft.webmagic.scheduler.FileCacheQueueScheduler.initWriter(FileCacheQueueScheduler.java:85)
... 5 more

怀疑是这个循环太多了．进程创建太多文件．

请问这中需求怎么处理？

Answer 1

感谢patch!已合并。

至于这个异常真的没有遇到过，建议确认一下FilePipeline是否正常close了。

Answer 2

private void readCursorFile() throws IOException {
        BufferedReader fileCursorReader = null;
        try {
            new BufferedReader(new FileReader(getFileName(fileCursor)));
            String line;
            //read the last number
            while ((line = fileCursorReader.readLine()) != null) {
                cursor = new AtomicInteger(NumberUtils.toInt(line));
            }
        } finally {
            if (fileCursorReader != null) {
                IOUtils.closeQuietly(fileCursorReader);
            }
        }
    }

fileCursorReader　这个变量似乎没有初始化啊