存储/访问大量数据的正确解决方案
我编写了一个程序，它抓取网站、处理 html 页面并将结果存储在 MySql 数据库中。我所说的“结果”是指 html 内容、所有链接及其属性以及爬虫无法获取时出现的各种错误。我使用该程序进行分析。
一切正常，但主要问题是数据占用太多磁盘空间。对于每爬行 100000 个网站（每个网站最多 20 个页面），我有大约 5 个 mysql 表，总计约 60 GB 空间，并且我需要处理 20-30 倍的网站。
当然，我无法一次在我的家用电脑上处理那么多数据，并且我被迫只处理其中的一小部分，这既耗时又低效。
所以我正在寻求建议或解决方案：
1) 提供与关系数据库相同的数据访问灵活性
2) 允许智能且高效地保存数据
I wrote a program which crawls website, processes html pages and stores results in MySql database. By 'results' i mean html contents, all the links with their attributes and various errors in case when crawler couldn't fetch. I use this program for analytical purposes.
Everything works fine but the main problem is that data takes way too much disk space. For each 100000 websites crawled (20 pages per site maximum) i have like 5 mysql tables totaling ~60 Gigabytes of space and i need to process 20-30 times more websites.
Of course i cannot process that much data on my home pc at once and i am forced to process only small chunks of it which is time consuming and not efficient.
So i am seeking for advice or solution that would:
1) give the same flexibility accessing data that relational DB does
2) allow smart and efficient saving of data
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
我怀疑另一种存储引擎会比这更高效 - 如果您将所有内容存储在一个表中，没有任何索引，并使用自然主键，那么几乎不会产生存储开销，即使您确实添加了一点结构，它应该仍然保持理智。
我的猜测是，您的问题在于您收集的数据量巨大，因此您可能希望在存储之前删除相当一部分示例数据：例如，您可能希望将页面源代码简化为一堆（标准化）关键字，您可能想跳过大量内容（图像等）和您不感兴趣的内容（例如 CSS 样式表、javascript 等）等。
I doubt a different storage engine will get much more efficient than that - if you store everything in one table, without any indexes, and using natural primary keys, then almost no storage overhead will be incurred, and even if you do add a bit of structure, it should still remain sane.
My guess would be that your problem is the sheer amount of data you collect, so you probably want to remove considerable portions of your sample data before storing: for example, you may want to boil the page source down to a bunch of (normalized) keywords, you may want to skip heavy content (images etc.) and stuff that doesn't interest you (e.g. CSS stylesheets, javascript, etc.), etc.
您可能需要研究 InnoDB 数据压缩选项。
还有一些 BI 产品，例如面向列的 Infobright 透明地使用压缩。
You may want to look into InnoDB Data Compression option.
There are also BI products like the column-oriented Infobright that transparently use compression.