ASP.NET 自动化与手动ETL过程
这是一个有趣的问题。我有一个用 C# 编写的 ETL 脚本,我一直定期手动运行该脚本。它用于更新我的网络应用程序的数据库。
我想要自动化 ETL 流程,并为 Web 应用程序管理员创建一个界面来手动启动 ETL 流程。
我可以让 sql server 按计划启动 ETL 过程,并在网页上实现一个按钮或其他东西来执行相同的操作,但我不想将我的代码放在两个不同的地方,因为我不想当它发生变化时,在 2 个地方更新它。但我不知道如何让我的网络应用程序告诉 SQL Server 手动启动计划的进程。这可以做到吗?
或者
我可以以某种方式在网络应用程序本身中实现调度,但现在大多数人都熟悉尝试时遇到的问题(应用程序可能无法在某些时间运行,必须等待请求才能启动进程(没有一些诡计))。另外,由于 ETL 过程需要一段时间,我不想让一些可怜的最终用户等待响应,所以它肯定必须使用新线程。
我还能做什么?你会怎么办?
Here's an interesting problem. I have an ETL script written in c# that I have been running manually on a somewhat regular basis. It is used to update my web app's database.
I want to automate the ETL process AND create an interface for the web app admins to manually start the ETL process.
I could have sql server kick off the ETL process on a schedule and implement a button or something on a web page that will do the same thing, but I don't want to put my code in 2 different places because I don't want to update it in 2 places when it changes. But I don't know how to make my web app tell SQL server to manually start a scheduled process. Can this be done?
OR
I could somehow implement the scheduling in the web app itself, but by now most people are familiar with the problems that are faced when trying that (app may not be running at certain times, must wait on request to start a process (without some trickery)). Also, since the ETL process takes a while, I don't want to make some poor end user wait on a response, so it would definitely have to use a new thread.
What else could I do? What would you do?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
您提到您正在使用 SQL Server。它是什么版本? SQL Server Integration Services(该平台的成熟 ETL 工具)适合您吗?如果您想要计划的提取、转换和加载作业,SSIS 是 Microsoft 平台上的一个很棒的工具,并且包含在大多数 SQL Server 许可证中。
或者,查看 Quartz。
You mentioned you are using SQL Server. What version is it? Is SQL Server Integration Services (which is the full-blown ETL tool for the platform) an option for you? If you want scheduled Extraction, Transformation and Load jobs, SSIS is a great tool on the Microsoft platform and is included in most SQL Server licenses.
Or, check out Quartz.