如何判断网页是否被修改
我有多个网页的快照，拍摄了两次。有什么可靠的方法来确定哪些网页已被修改？
我不能依赖像 RSS feed 这样的东西，而且我需要忽略像日期文本这样的小噪音。
理想情况下，我正在寻找 Python 解决方案，但直观的算法也很棒。
谢谢！
I have snapshots of multiple webpages taken at 2 times. What is a reliable method to determine which webpages have been modified?
I can't rely on something like an RSS feed, and I need to ignore minor noise like date text.
Ideally I am looking for a Python solution, but an intuitive algorithm would also be great.
Thanks!
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（4）
只需使用 MD5 或 SHA1 拍摄文件快照...如果下次检查时值不同，则它们会被修改。
just take snapshots of the files with MD5 or SHA1...if the values differ the next time you check, then they are modified.
好吧，首先您需要确定什么是噪音，什么不是。您可以使用 BeautifulSoup 之类的 HTML 解析器来消除噪音，漂亮地打印结果，然后将其作为字符串进行比较。
如果您正在寻找自动解决方案，可以使用
difflib.SequenceMatcher< /code>要计算页面之间的差异，请计算 相似度并将其与阈值进行比较。Well, first you need to decide what is noise and what isn't. You can use a HTML parser like BeautifulSoup to remove the noise, pretty-print the result, and compare it as a string.
If you are looking for an automatic solution, you can use
difflib.SequenceMatcherto calculate the differences between the pages, calculate the similarity and compare it to a threshold.解决方案实际上取决于您是否正在抓取特定网站，或者正在尝试创建一个适用于任何网站的程序。
您可以通过执行以下操作来查看哪些区域经常发生变化：
如果您只担心单个站点，则可以创建一些 sed 表达式来过滤掉时间戳等内容。您可以重复，直到小字段没有显示任何差异。
一般问题要困难得多，我建议初学者比较页面上的总字数。
The solution really depends if you are scraping a specific site, or are trying to create a program which will work for any site.
You can see which areas change frequently doing something like this:
If your only worried about a single site, you can create some sed expressions to filter out stuff like time stamps. You can repeat until no difference is shown for small fields.
The general problem is much harder, and I would suggest comparing the total word count on a page for starters.
如果您将更改的阈值设置为以下距离，类似 Levenshtein Distance 的东西可能会派上用场忽略了适合您的适量噪音。
Something like Levenshtein Distance could come in handy if you set the threshold of the changes to a distance that ignored the right amount of noise for you.