Azure Databricks delta表会在从Spark DataFrame写作时修改时间戳格式
我是Azure Databricks的新手，我正在尝试将数据框架输出写入Delta表，该表构成了时间戳列。但奇怪的是，它在写入Delta表之后会改变时间戳图案。 我的DataFrame输出列以此格式保存该值2022-05-13 17：52：09.771， 但是将其写入表后，当
2022-05-13T17:52:09.771+0000
我使用以下函数来生成此数据帧输出
val pretsUTCText = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")
val tsUTCText: String = pretsUTCTextNew.format(ts)
val tsUTCCol : Column = lit(tsUTCText)
val df = df2.withColumn(to_timestamp(timestampConverter.tsUTCCol,"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"))
时，列值将被填充。数据框架输出返回2022-05-13 17：52：09.771作为时间戳模式。 但是，将其写入Delta表后，我看到相同的值被填充为2022-05-13T17：52：09.771+0000，
谢谢。我找不到任何解决方案。
I am new to Azure Databricks,I am trying to write a dataframe output to a delta table which consists TIMESTAMP column. But strangely it changes the TIMESTAMP pattern after writing to delta table.
My DataFrame Output column holds the value in this format 2022-05-13 17:52:09.771,
But After writing it to the Table, The column value is getting populated as
2022-05-13T17:52:09.771+0000
I am using below function to generate this Dataframe output
val pretsUTCText = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")
val tsUTCText: String = pretsUTCTextNew.format(ts)
val tsUTCCol : Column = lit(tsUTCText)
val df = df2.withColumn(to_timestamp(timestampConverter.tsUTCCol,"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"))
The Dataframe output is returning 2022-05-13 17:52:09.771 as TIMESTAMP pattern.
But After writing it to Delta Table I see the same value is getting populated as 2022-05-13T17:52:09.771+0000
Thanks in Advance. I could not find any solution.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我刚刚在数据括号上找到了与您相同的行为，其行为与databricks 文档。似乎有一些版本的数据映射显示为默认情况，因此您会看到其他+0000。我认为，如果您不想要数据，则可以使用date_format函数。另外，我认为您不需要格式文本中的“ z”，而是时间区域。请参阅下面的屏幕截图。
I have just found the same behaviour on Databricks as you, and it behaves differently than the Databricks document. It seems after some versions Databricks show timezone as a default so you see additional +0000. I think you can use date_format function when you populate data if you don't want it. Also, I think you don't need 'Z' in format text as it is for timezone. See the screenshot below.