理论上在 Hive 中可以实现并置连接 (a-la-netezza) 吗？
当您连接分布在同一键上的表并在连接条件中使用这些键列时，netezza 中的每个 SPU（机器）将 100% 独立于其他设备工作（请参阅 nz-interview）。
在 hive 中，有bucketed map join，但是代表表到datanode是HDFS的责任，它不是根据hive CLUSTERED BY key完成的！
所以假设我有 2 个表，由相同的键聚集，并且我通过该键加入 - hive 可以从 HDFS 获得匹配的存储桶将位于同一节点上的保证吗？或者总是必须将小表的匹配存储桶移动到包含大表存储桶的数据节点？
谢谢，ido
（注意：这是我之前问题的更好措辞：hive/hadoop 如何确保每个映射器处理其本地数据？）
When you join tables which are distributed on the same key and used these key columns in the join condition, then each SPU (machine) in netezza works 100% independent of the other (see nz-interview).
In hive, there's bucketed map join, but the distribution of the files representing the tables to datanode is the responsibility of HDFS, it's not done according to hive CLUSTERED BY key!
so suppose I have 2 tables, CLUSTERED BY the same key, and I join by that key - can hive get a guarantee from HDFS that matching buckets will sit on the same node? or will it always have to move the matching bucket of the small table to the datanode containing the big table bucket?
Thanks, ido
(note: this is a better phrasing of my previous question: How does hive/hadoop assures that each mapper works on data that is local for it?)
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我认为不可能告诉 HDFS 在哪里存储数据块。
我可以考虑以下技巧，这适用于小型集群 - 将其中一个表的复制因子增加到接近或等于集群中节点数的数量。
因此，在连接过程中，适当的数据几乎总是（或总是）出现在所需的节点上。
I think it is not possible to tell to HDFS where to store blocks of data.
I can consider the following trick, which will do for small clusters - to increase replication factor for one of the tables to the number close or equal to the number of nodes in the cluster.
As a result - during join process appropriate data will be almost always (or always) present on the required node.