仅抓取 HTML 页面，同时检查响应标头

Question

我试图获取所有标题为 Content-Type:text/html 的 url，因此我检查每个 url 的响应标头，如果它们具有 content-type: text/html，那么我想打印该 url内容类型：文本/html。但是在我的代码中，当我检查标头是否具有 Content-Type 时，它​​不会显示任何内容。如果我删除 if 循环，那么它会打印与我想要抓取的特定 url 相关的每个链接及其响应标头..

public class MyCrawler extends WebCrawler {

    Pattern filters = Pattern.compile(".*(\\.(css|js|bmp|gif|jpe?g"
            + "|png|tiff?|mid|mp2|mp3|mp4" + "|wav|avi|mov|mpeg|ram|m4v|pdf"
            + "|rm|smil|wmv|swf|wma|zip|rar|gz))$");


    /*
    Pattern filters = Pattern.compile("(\\.(html))");
*/
    public MyCrawler() {
    }

    public boolean shouldVisit(WebURL url) {
        String href = url.getURL().toLowerCase();
        //System.out.println("Href: " +href);
        /*
        if (filters.matcher(href).matches()) {
            return false;
        }*/
        if (href.startsWith("http://www.somehost.com/")) {
            return true;
        }
        return false;
    }

    public void visit(Page page) {

        int docid = page.getWebURL().getDocid();

        String url = page.getWebURL().getURL();         
        String text = page.getText();
        List<WebURL> links = page.getURLs();
        int parentDocid = page.getWebURL().getParentDocid();


        //HttpGet httpget = new HttpGet(url);


        try {
            URL url1 = new URL(url);
            URLConnection connection = url1.openConnection();

            Map responseMap = connection.getHeaderFields();
      for (Iterator iterator = responseMap.keySet().iterator(); iterator.hasNext();) 
     {
                String key = (String) iterator.next();
                if(key==("Content-Type")) //(Anything wrong with this if loop)
                {
                System.out.print(key + " = ");

                List values = (List) responseMap.get(key);
                for (int i = 0; i < values.size(); i++) {
                    Object o = values.get(i);
                    System.out.print(o + ", ");
                }
                System.out.println("");
System.out.println(url1);
                }

            }
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }


        //System.out.println("Docid: " + docid);
        //System.out.println("URL: " + url);
        //System.out.println("Text length: " + text.length());
        //System.out.println("Number of links: " + links.size());
        //System.out.println("Docid of parent page: " + parentDocid);
        System.out.println("=============");
    }   
}

Answer 1

key 变量包含：

Content-Type=[text/html; charset=ISO-8859-1]

因此不能用 == 或 .equals("Content-Type") 捕获

如果您尝试运行以下代码，看看它打印出什么

URLConnection connection = url1.openConnection();

Map responseMap = connection.getHeaderFields();
Iterator iterator = responseMap.entrySet().iterator();
while (iterator.hasNext())
{
    String key = iterator.next().toString();
    if (key.contains("Content-Type"))
    {
        System.out.println(key);
        // Content-Type=[text/html; charset=ISO-8859-1]
        if (filters.matcher(key) != null){
            System.out.println(url1);
            // http://google.com
        }
    }
}

这是输出：

Content-Type=[text/html; charset=ISO-8859-1]
http://google.com

看起来您也可以只使用一个 if 语句，如下所示：

while (iterator.hasNext())
{
    String key = iterator.next().toString();
    if (key.contains("text/html"))
    {
        System.out.println(url1);
        // http://google.com
    }
}

BTW string Comparison in Java 非常直观，一直让我着迷！

Answer 2

对于字符串比较，请使用 .equals()。

Answer 3

它应该与

if (key != null && key.equals("Content-Type"))