面对触发从AWS S3到雪管的SQS通知的问题

发布于 2025-01-23 23:28:25 字数 1878 浏览 2 评论 0 原文

我是Snowflake的新手,试图将文件放在S3存储桶中并通过SQL事件通知 Snowpipe 需要将数据加载到相应表中。

以下是我遵循的步骤。

步骤-1:在Snowflake 中创建一个表

Create Table Customer_notification(ID varchar(10),name varchar(25));

step -2:创建一个文件格式

创建或替换文件格式csv_notification type ='csv'compression ='auto'field_delimiter =','record_delimiter ='\ n skip_header = 1 field_optionally_enclosed_by ='\ 042'null_if =('\\ n'\ \ n'

'外部阶段

创建stage customer_notification_s3 url ='s3:// snow-flake-test/contaiuldataload/customernotification/'recortentials =(aws_key_id ='*********'aws_secret_keykey ='******************************************************************************************************************************************* ***')file_format = csv_notification注释='feed sustomernotification files';

step -4:创建管道

创建或替换pipe custome_notification_notification_pipe auto_ingest = true as as as as as as as as aS copy_notification in @cultines_notification_s3 file_s3 file_s3 file_s3 file_format = csv_notification = csv_notification = csv_notificative =' customNotification。

* -ARN:AWS:SQS:us-east-2:306559659303:sf-snowpipe-aidauoydbfut7y5dfhk4k-rdvaoumamjyd8diwas1eww

spep -6:打开 aws aws s3 bucce通知 。 事件名称 - customer_notification

前缀 - 可选-Snow -Flake -Test/连续Dataload / customernotification /

suffix/suffix-可选 - .csv

事件类型 - 所有对象创建事件

目标 - 选定的SQS队列

Enter SQS队列ARN SQS队列-ARN:AWS:SQS:us-east-2:306559659303:sf-snowpipe-aidauoydbfut7y5dfhk4k-rdvaoumamjyd8diywas1ewwwww

eventification notification>

abovesteps 连续dataload / customernotification /文件夹中,管道没有选择文件而不加载文件。 也检查了管道状态,它处于运行状态。 注意:我处于角色 AccountAadmin

任何人都可以帮助我弄清楚丢失的内容。

I am new to Snowflake, Trying to notify the snowpipe when a file is placed in a s3 bucket and thru SQL event Notification the snowpipe needs to load the data into the respective table.

Below are the steps I have followed.

Step -1: created a table in Snowflake

create table Customer_Notification (Id Varchar(10),Name Varchar(25) );

Step -2: Created a File Format

create or replace file format csv_Notification type = 'csv' compression = 'auto' field_delimiter = ',' record_delimiter = '\n' skip_header = 1 field_optionally_enclosed_by = '\042' null_if = ('\\N');

Step - 3: Created an external Stage

create stage Customer_Notification_S3 url = 's3://snow-flake-test/ContinuousDataLoad/CustomerNotification/' CREDENTIALS = (AWS_KEY_ID = '*******' AWS_SECRET_KEY = '**********') file_format = csv_Notification comment = 'feed CustomerNotification files';

Step - 4: Create Pipe

create or replace pipe Customer_Notification_pipe auto_ingest = true as copy into customer_Notification from @Customer_Notification_S3 file_format = csv_Notification pattern='.*customernotification.*[.]csv' ON_ERROR = 'CONTINUE';

Step - 5 : Get the ARN Number

show pipes; -- arn:aws:sqs:us-east-2:306559659303:sf-snowpipe-AIDAUOYDBFUT7Y5DFHK4K-RdvAOUMAMjYD8diWaS1eWw

Step -6 : Open aws S3 bucket in the parent folder, click on properties to create an even notification .
Event name - customer_notification

Prefix - optional - snow-flake-test/ContinuousDataLoad/CustomerNotification/

Suffix - optional - .csv

Event types - All object create events

Destination - Selected SQS queue

Enter SQS queue ARN
SQS queue - arn:aws:sqs:us-east-2:306559659303:sf-snowpipe-AIDAUOYDBFUT7Y5DFHK4K-RdvAOUMAMjYD8diWaS1eWw

Finally Saved the Event Notification

I have followed the abovesteps, even after this, when a file is placed in the respective snow-flake-test/ContinuousDataLoad/CustomerNotification/ folder the pipe is not picking the file and not loading it.
checked the pipe status too , it is in running state.
NOTE : I am in a role Accountaadmin

Could anyone please help me to figure out what is missing.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(1

一片旧的回忆 2025-01-30 23:28:25

我意识到这个问题已经旧了,所以您可能已经有了解决方案,但是我认为我会分享我的发现。我面临类似的问题,并遇到了可能会有所帮助的这篇文章:

我相信问题是您在前缀名称中具有前锋斜线。尝试删除前缀中的前锋斜线。

I realise this question is old so you may have a solution already, but thought I'd share my findings. I am facing similar issues and came across this article that may help: https://community.snowflake.com/s/article/Autoingest-not-working-for-Snowpipe.

I believe the issue is that you have a leading forward slash in the prefix name. Try removing the leading forward slash in the prefix.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文