从 Windows Azure 平台上的第三方 Web 服务异步提取数据
我想经常从不同的第三方 API Web 服务中提取大量数据，并将其存储在暂存区域（这是我现在想要决定的），然后根据需要将其一一移动到我的应用程序中数据库。
我想知道我可以使用Azure平台来实现上述功能吗？使用Azure平台来完成这项任务有多好？
如果拉取的数据量很大，拉取的频率很高，可能是半小时或者每小时拉取2000个不同的用户怎么办？
我认为，如果这一切都是可能的，那么带宽、数据存储和服务器能力等对我来说就不会是一个值得担心的事情，而对©微软来说则更是如此。显然，我应该能够在需要时随时访问数据。
如果我必须在 Windows 服务器上实现它，那么我知道我会使用 Windows 服务来执行此操作。但我不知道如何为 Windows Azure 平台做到这一点，如果可能的话？
I want to pull large amount of data, frequently from different third party API web services and store it in a staging area (this is what I want to decide right now) from where it will be then moved one by one as required into my application's database.
I wanted to know that can I use Azure platform to achieve the above? How good is it to use Azure platform for this task?
What if the data to be pulled is of large amount and the frequency of the pull is high i.e. may be half-hourly or hourly for 2,000 different users?
I assume that if at all this is possible, then the bandwidth, data storage and server capability etc. will not be a thing to worry for me but for ©Microsoft. And obviously, I should be able to access the data back whenever I need it.
If I would have to implement it on Windows Servers, then I know that I would use a windows service to do this. But I don't know how it can be done for Windows Azure Platform if at all it is possible?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
在 Windows Azure 中，通常是用于托管云处理的辅助角色。为了完成您的任务，您需要自己实现此消息传递/调度基础设施，或者使用类似 Lokad.Cloud 或 Lokad.CQRS 开源项目天蓝色。
我们使用 Lokad.Cloud 对数十万个系列进行分布式 BI 处理，而 Lokad.CQRS 允许按计划可靠地检索和同步数百万个产品。
这两个项目都有示例、文档和社区来帮助您入门。
In Windows Azure that's usually Worker Role used to host the cloud processing. In order to accomplish your tasks you'll either need to implement this messaging/scheduling infrastructure yourself or use something like Lokad.Cloud or Lokad.CQRS open source projects for Azure.
We use Lokad.Cloud for distributed BI processing of hundreds of thousands of series and Lokad.CQRS allows to reliably retrieve and synchronize millions of products on schedule.
There are samples, docs and community in both projects to get you started.
正如 Rinat 所说，您可以使用 Lokad 的解决方案。如果您选择自己执行此操作，则可以在辅助角色中运行定时任务 - 可能会生成一个休眠线程，每 30 分钟唤醒一次以执行其任务。然后，它可以访问相关的 Web 服务（或者每个 Web 服务一个线程？）并获取数据。您可以将其临时存储在 Azure 表存储中，这只是 SQL Azure 成本的一小部分（每 GB 0.15），然后轻松地按需从表存储中读取并传输到 SQL Azure。
假设您将服务、存储和 SQL Azure 托管在同一个数据中心（通过适当设置关联性），那么您只需在从 Web 服务提取数据时支付带宽费用。从表存储检索或插入 SQL Azure 不会产生带宽费用。
As Rinat stated, you can use Lokad's solution. If you choose to do it yourself, you can run a timed task in your worker role - maybe spawn a thread that sleeps, waking every 30 minutes to perform its task. It can then reach out to the Web Services in question (or maybe one thread per Web Service?) and fetch data. You can store it temporarily in Azure Table Storage, which is a fraction of the cost of SQL Azure (0.15 per GB), and then easily read it out of Table Storage on-demand and transfer to SQL Azure.
Assuming you host your services, storage and SQL Azure are in the same data center (by setting the affinity appropriately), you'd only pay for bandwidth when pulling data from the web service. There'd be no bandwidth charges to retrieve from Table Storage or insert into SQL Azure.