向现有 HIVE 表添加列会产生什么后果?
假设在开始使用 HIVE 后,我想添加一个列。 来自各种文章和我看过的页面,我无法理解所需
- 存储空间(双?)
- 阻塞(我仍然可以在其他进程中读取该表)方面的后果吗?
- 时间(像 MysqL 更改一样快还是慢?)
- 底层存储(我需要更改所有底层文件吗?如何使用 RCFile 来完成?)
对于能够回答有关 HIVE 列中的结构的相同问题的人来说是奖励。
Suppose that a couple hundred Gigs after starting to use HIVE I want to add a column.
From the various articles & pages I have seen, I cannot understand the consequences in terms of
- storage space required (double ?)
- blocking (can I still read the table in other processes) ?
- time (is it quick or as slow as a MysqL change ?)
- underlying storage (do I need to change all the underlying files ? How can it be done using RCFile ?)
Bonus to whoever can answer the same question on structs in a HIVE column.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(3)
如果向 Hive 表添加列,则仅更新基础元存储。
我希望这会有所帮助。
If you add a column to a hive table, only the underlying metastore is updated.
I hope this helps.
ALTER TABLE 命令仅修改元数据。基础数据保持不变。但是,用户有责任确保任何更改不会破坏数据一致性。
此外,对元数据的任何更改都会应用于元存储(最典型的是 MySQL),在这种情况下,响应时间是相当的。
ALTER TABLE commands modifies the METADATA only. The underlying data remains untouched. However, it is user's responsibility to ensure that the any alteration does not break the data consistency.
Also any changes to METADATA is applied to the metastore - which is most typically MySQL - in which case the response time is comparable.
更改定义只会修改文件的读取方式,而不会修改底层文件的内容。
如果您的文件是具有 3 列的制表符分隔文本,您可以创建一个表,使用 new_table(line STRING) 之类的方案引用这些文件,该方案将读取整行,而无需根据制表符解析列。
当您添加列时,由于记录中不再有分隔符,因此它将默认为 NULL,正如 Helmut 提到的那样。
Altering the definition will only modify how the files are read, not the contents of the underlying files.
If your files were tab delimited text with 3 columns, you could create a table that references those files with a scheme like new_table(line STRING) that would read the entire line without parsing out columns based upon the tab characters.
When you add a column, since there are no more delimiters in the record, it will default to NULL, as Helmut mentioned.