持续传输来自合作伙伴的数据
我的任务是实现一个从合作伙伴那里持续接收大量数据的系统。合作伙伴的数据模式与我们不同，因此在接收数据并将其导入我们的数据库时必须进行一些转换。
解决此任务的最佳方法是什么？
我对等式的两边都有一定的控制权，即如果我们选择使用 WCF，我们的合作伙伴将实施该服务。但重要的是要尽可能限制合作伙伴方面必须进行的编程量。理想情况下，他们会使用 SQL Server 的某些功能来导出数据，然后让我们处理其余的事情。
其他相关因素：
- 双方都使用 SQL Server（我们这边是 2008 年，合作伙伴未知）。
- .NET 4 和/或 Microsoft 开箱即用的任何产品，无第三方产品。
- 转让是一种方式，即仅从合作伙伴转让给我们。
- 每天传输的数据量约为数万条更新记录/对象。
- 数据模型相当复杂，有多个表和关系。
- 解决方案应该能够容忍合作伙伴方面的更改（我们不知道他们何时以及如何更改模型）
- 可维护性和可靠性比性能更重要，即我们不需要最新的数据，但服务应该易于修改并且不会失败/停止。
我问这个问题是因为我不知所措，但可以实现这一目标的方法有很多。作为一名 .NET 开发人员，我倾向于实现 WCF 甚至一些基于 XML 的自定义交换。但我知道 BizTalk 可能适合这里，而且 SQL Server 可能内置了一些我不知道的漂亮功能。
欢迎任何指点、想法和建议！
I’ve been tasked with implementing a system for continuously receiving large amounts of data from a partner. The data schema on the partners’ side is different from ours, so some transformation has to take place when the data is received and imported into our database.
What’s the best approach for solving this task?
I have some control over both sides of the equation, i.e. if we chose to use WCF, our partner will implement that service. But it’s important to limit the amount of programming that has to take place on the partners’ side as much as possible. Ideally they would use some feature of SQL Server to export their data and then let us deal with the rest.
Other relevant factors:
- SQL Server is used on both sides (2008 at our side, unknown for the partner).
- .NET 4 and/or anything that comes out of the box from Microsoft, no 3rd party products.
- Transfer is one way, i.e. from the partner to us only.
- Amount of data is around tens of thousands of updated records/objects transferred daily.
- The data model is fairly complicated, with several tables and relations.
- The solution should be tolerant to changes on the partner's side (we don’t know when & how they change their model)
- Maintainability and reliability are more important than performance, i.e. we don't need the latest data, but the service should be easy to modify and it can't fail/stop.
I’m asking this question because I overwhelmed but the number of ways this can be accomplished. As a .NET developer, I’m leaning towards implementing WCF or maybe even some custom XML-based exchange. But I know BizTalk would probably fit the bill here, and SQL Server probably has some nifty features built in that I don’t know about.
Any pointers, ideas, and suggestions are most welcome!
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
除了 Will 的建议之外，您还可以看看 SSIS（SQL Server Integration Services）。通过此功能，您可以导出到 FTP 站点，然后在另一端导入。
In addition to Will's suggestions, you can take a look at SSIS (SQL Server Integration Services). With this you can export to an FTP site and then import it on the other side.
如果您想要其数据库的完整副本，请日志传送 是一个不错的选择，否则请查看复制，它可以为您提供更好的效果- 对复制内容进行精细控制。
您可以按原样复制数据并在您这边处理转换逻辑。
If you'd like a complete copy of their database, then log shipping is a good option, otherwise check out replication which gives you finer-grained control over what's copied.
You can copy across the data as-is and deal with the transformation logic on your side of the wire.
也许，我会这样做：
据我所知，当架构更改时，数据比较效果很好（它忽略架构更改）。 CDC 也运行良好，您所需要做的就是在架构发生更改时创建第二个捕获实例并删除第一个捕获实例。
Probably, here is how I'd do it:
As far as I know, Data Compare works well when the schema is changed (it ignores the schema changes). CDC also works well, all you need to do is create a second capture instance if there is a schema change and remove the first one.