使用SPARK或AWS步骤功能的延迟格式
我如何在AWS步骤中的JSON文件中获得格式化日期
示例JSON文件:这是通过AWS上的内部CICD创建步骤函数
datesuffix具有“ 04-05-2022”格式
Parameters:{
"loads.$":$.loads",
tasktimeOutseconds.$":"$.taskTimeOutSeconds",
"dateSuffix.$":"$.cluster.Payload.Datesuffix",
},
"Next"."Spark-submit-test"
"Type":"Pass"
},
--
--
--
--"Spark-submit-test":{
"Resultpath:$.FeaturecomputeforEMR",
"Parameters" :{
"taskName":"Spark-submit-test",
"Args":[
{
"Args1":"spark-submit"
},
{
"Args2":"--master"
}
--
--
--
{
args16.$":"States.Format('s3a://{{e_bucket_name}}/{}/test/',$.dateSuffix)"
},
{
args17.$":"States.Format('s3a://{{e_bucket_name}}/{}/test2/',$.**dateSuffix2**)"
},
datesuffix2我想要格式为“ 20220405”,因为在s3:// drdata/usic/20220405中 任何想法我如何在步骤函数文件中格式化日期,请建议
或如何直接从Spark读取它
s3 location cMerge="s3://drdata/usa/20220405/test.csv"
val formcdi=new simepleDateformat("YYYYMMdd")
val formatnew=formcdi.format(c.getTime()).toString;
val c=spark.read.option("delimeter",",").csv(config.CMerge).{$formatnew)
how i can get formatted date in json file in aws step functions
sample json file: this creates step function via internal CICD on aws
datesuffix has "04-05-2022" format
Parameters:{
"loads.quot;:$.loads",
tasktimeOutseconds.quot;:"$.taskTimeOutSeconds",
"dateSuffix.quot;:"$.cluster.Payload.Datesuffix",
},
"Next"."Spark-submit-test"
"Type":"Pass"
},
--
--
--
--"Spark-submit-test":{
"Resultpath:$.FeaturecomputeforEMR",
"Parameters" :{
"taskName":"Spark-submit-test",
"Args":[
{
"Args1":"spark-submit"
},
{
"Args2":"--master"
}
--
--
--
{
args16.quot;:"States.Format('s3a://{{e_bucket_name}}/{}/test/',$.dateSuffix)"
},
{
args17.quot;:"States.Format('s3a://{{e_bucket_name}}/{}/test2/',$.**dateSuffix2**)"
},
datesuffix2 i want format as "20220405" because in s3://drdata/usa/20220405
any idea how i can format date in step functions json file , please suggest
or how can i read it from spark directly
s3 location cMerge="s3://drdata/usa/20220405/test.csv"
val formcdi=new simepleDateformat("YYYYMMdd")
val formatnew=formcdi.format(c.getTime()).toString;
val c=spark.read.option("delimeter",",").csv(config.CMerge).{$formatnew)
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
Step Functions 本身无法转换字符串。其内在函数< /a> 非常短。
使用 Lambda 任务状态 来操作日期字符串。
Step Functions cannot natively transform strings. Its list of intrinsic functions is very short.
Use a Lambda Task State to manipulate the date string.