加快使用SSIS ODBC使用SSIS写入Aurora Postgres
首先,我所有的数据\实例等都坐在AWS云上。我正在使用SSI来从各个数据库中提取数据,然后发送到其他数据库。 有2个不同的来源
- 源
- 我
: (也如上所述PG1)。
我正在使用ODBC(已安装驱动程序)连接到PG1实例,并集成安全性连接到SQL1。所有连接经理都使用ADO.NET连接类型。
现在,我知道这里还有许多其他因素,包括AWS中的VPN设置,但是数据库之间的数据传输似乎有点慢。我在数据巴上有略低的100万行。将其提取到DatabaSec大约需要30分钟。 数据库有800万行,需要5-6小时才能提取\负载。
直接击倒蝙蝠,我可以专门考虑尝试加快数据的质量\载荷数据吗?
我猜想问题是使用ODBC驱动程序(尽管我不确定在SSIS中还能使用什么)。例如,我已将AWS胶水用于其他作业,并将数据加载到Postgres中的时间很少。如果它可以帮助您了解我正在使用的设置,我可以根据需要提供任何额外的信息!
Firstly, all my data\instances etc are sat on AWS Cloud. I'm using SSIS to extract data from various databases and send to a different database. I have 2 different sources:
- SOURCE A - A table in DATABASE A on AWS Aurora PostgreSQL (Pg1)
- SOURCE B - A table in DATABASE B on MS SQL Server (Sql1)
The Destination - a couple of tables in DATABASE C on AWS Aurora PostgreSQL (also Pg1 as above).
I am using an ODBC (driver installed) to connect to the Pg1 instance and Integrated Security to connect to Sql1. All connection managers are using ADO.NET connection types.
Now I know there will be lots of other factors here including VPN setup in AWS but the data transfer between databases seems a bit slow. I have a little under 1 million rows in DatabaseA. It takes about 30 minutes to Extract\Load that to DatabaseC.
DatabaseB has 8 million rows and takes between 5-6 hours to Extract\Load.
Straight off the bat, is there anything I can specifically look at to try and speed up the Extract\Load of data?
I'm guessing the problem is with using ODBC drivers (although I am not sure what else I can use within SSIS). eg I have used AWS Glue for other jobs and loading data to Postgres takes a fraction of this time. I can provide any extra information as necessary if it would help understand the setup I am using!
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论