如何在解决方案中的每个软件包中找到每个数据流任务中的源和目标?

发布于 2025-02-03 18:36:32 字数 122 浏览 2 评论 0原文

我有数百个未记录的SSIS软件包。我想提出代码/程序,以在给定解决方案中的每个软件包中列出所有数据流任务。列表应包括: 软件包名称,数据流任务名称,源连接或源数据库,源表/文件,目标连接或DB,目标表。

可以可行吗?

I have hundreds of SSIS packages that are not documented. I'd like to come up with the code/program to list all data flow tasks in every package in the given solution. The list should include:
Package Name, Data Flow Task Name, Source Connection or Source DB, Source Table/File, Destination Connection or DB, Destination Table.

Is it doable?

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(1

二智少女 2025-02-10 18:36:32

您可以绝对使用SQL Server二进制文件的正确版本和您喜欢的.NET语言编写此内容。

我一生中的SSI做了很多事情,我绝对不想写这句话,因为可以起草包裹的方式有多种。您会发现的是,每个容器可能包含数据流,并且可以在不属于的地方(例如事件处理程序)中有数据流。并且可能有com要处理。

市场上有一项工具的报酬:BI记录员(我认为那是名字)是其中之一。自那以后,开源项目出价的助手在其中具有一些文档功能。

但是,到目前为止,请把绝对最好的方法处理方法,而我唯一提倡的方法是您将BIML用于这样的项目。 BIML是商业智能标记语言,它是一种特定领域的语言,可将SSIS(以及表和立方体)减少到最小的单元。然后允许您查询它。

步骤1。下载 bimlexpress 这是Visual Studio的免费附加组件。您将使用它来将所有SSIS软件包倒入相应的BIML

步骤

2

。查询(Postgres),然后是SQL Server中的目标表。源查询位假定数据流上有表达式,因此不包括。

<#
string dataflowName = "";
string sourceComponentName = "";
string targetComponentName = "";

// Loop through all the packages
foreach (AstPackageNode p in this.RootNode.Packages)
{

    // Find all the Data Flows on the canvas itself
    // Your TODO: inspect the containers via  p.Tasks.OfType<AstContainerTaskNode>() looking for dataflows
    foreach (AstDataflowTaskNode t in p.Tasks.OfType<AstDataflowTaskNode>())
    {
        // TODO: Lots of gotchas here, 
        // What if there are multiple sources
        // what if the source doesn't support "names" for the table/query
        // What if the package was built via Import/Export wizard which has 
        //  several independent source to destinations all within the same DFT
        // etc
        // Same applies to destinations

        dataflowName = t.Name;
    
            // This is an example of how to iterate through all the OLE DB Destination components identifying various bits I needed to know
            foreach (AstOleDbDestinationNode tn in t.Transformations.OfType<AstOleDbDestinationNode>())
            {
                // What connection is being used
                WriteLine(tn.Connection.Name);
                // Extract table best we can
                System.Xml.Linq.XElement xe = System.Xml.Linq.XElement.Parse(tn.GetBiml());
                foreach (System.Xml.Linq.XElement node in xe.Elements().Where(xnode => xnode.Name == "ExternalTableOutput"))
                {
                    tableName = node.Attributes("Table").FirstOrDefault().Value.ToString().Split('.')[1];
                    // Table will end up looking like either [srcdb].[foo] OR "srcdb"."foo"
                    tableName = tableName.Substring(1, tableName.Length-2);
                }
                // What if we need columns?
                // <Columns><Column SourceColumn="id" TargetColumn="Id" />
                foreach (System.Xml.Linq.XElement node in xe.Elements().Where(xnode => xnode.Name == "Columns"))
                {
                    foreach (System.Xml.Linq.XElement inode in node.Elements().Where(xnode => xnode.Name == "Column"))
                    {
                        WriteLine(inode.ToString());
                        WriteLine("");
                    }
                    WriteLine("");
                }
            }

#>

对于BimlexPress的工作方式,您需要打开所有软件包(SO_69215306.BIML,如我的图像所示),并且各种writeline(S)将在预览面板中显示。

当您想尝试通过整个包装生态系统进行此操作时,您将需要在Visual Studio的其他部分中进行多选,然后右键单击Inspector脚本(对于我的我屏幕截图,这将是bimlscript6.biml),然后检查BIML是否有错误或生成SSIS软件包。

两者都会产生相同的效果 - 假设您只是将BIML用于对象模型功能。我假设您会写信给 json/csv/txt/database/

step 4。爱上Biml的所有功能并购买bimlstudio

示例 ,而不是有助于调试有用的写入。将项目转换为BIML

  • 转换为BIML
  • 选择项目和[import](这标识您可以将带入BIML项目定义)
  • 选择您要[添加到项目]的资产(可能所有内容)
  • 右键单击项目并选择

You can absolutely write this using the correct version of the SQL Server binaries and your favorite .NET language to do so.

I did a lot with SSIS in my life and I would absolutely not want to write that as there's such a variety of ways a package could be drafted. Things you'll discover is that every container might contain a Data Flow and there can be data flows in places they don't belong, like an Event Handler. And there may be COM to deal with.

There were paid for tools on the market at one point: BI Documenter (I think that was the name) was one. The open source project BIDS Helper which has since been rebranded had some documentation functionality in it.

But, by far, hands down the absolute best way to handle it and the only approach I would advocate is that you use Biml for a project like this. Biml, is the Business Intelligence Markup Language, it's a domain specific language that reduces SSIS (and tables and cubes) to the smallest unit. And then allows you to query over it.

Step 1. Download BimlExpress It's a free add-in for Visual Studio. You will use this to reverse engineer all your SSIS packages into the corresponding Biml

Step 2. Right click on the project and select Add new Biml File

Step 3. Write the Biml

I used the following query in a recent project because we needed to extract source queries (Postgres) and then the target tables in SQL Server. The source query bit assumes there are Expressions on the data flow so not included.

<#
string dataflowName = "";
string sourceComponentName = "";
string targetComponentName = "";

// Loop through all the packages
foreach (AstPackageNode p in this.RootNode.Packages)
{

    // Find all the Data Flows on the canvas itself
    // Your TODO: inspect the containers via  p.Tasks.OfType<AstContainerTaskNode>() looking for dataflows
    foreach (AstDataflowTaskNode t in p.Tasks.OfType<AstDataflowTaskNode>())
    {
        // TODO: Lots of gotchas here, 
        // What if there are multiple sources
        // what if the source doesn't support "names" for the table/query
        // What if the package was built via Import/Export wizard which has 
        //  several independent source to destinations all within the same DFT
        // etc
        // Same applies to destinations

        dataflowName = t.Name;
    
            // This is an example of how to iterate through all the OLE DB Destination components identifying various bits I needed to know
            foreach (AstOleDbDestinationNode tn in t.Transformations.OfType<AstOleDbDestinationNode>())
            {
                // What connection is being used
                WriteLine(tn.Connection.Name);
                // Extract table best we can
                System.Xml.Linq.XElement xe = System.Xml.Linq.XElement.Parse(tn.GetBiml());
                foreach (System.Xml.Linq.XElement node in xe.Elements().Where(xnode => xnode.Name == "ExternalTableOutput"))
                {
                    tableName = node.Attributes("Table").FirstOrDefault().Value.ToString().Split('.')[1];
                    // Table will end up looking like either [srcdb].[foo] OR "srcdb"."foo"
                    tableName = tableName.Substring(1, tableName.Length-2);
                }
                // What if we need columns?
                // <Columns><Column SourceColumn="id" TargetColumn="Id" />
                foreach (System.Xml.Linq.XElement node in xe.Elements().Where(xnode => xnode.Name == "Columns"))
                {
                    foreach (System.Xml.Linq.XElement inode in node.Elements().Where(xnode => xnode.Name == "Column"))
                    {
                        WriteLine(inode.ToString());
                        WriteLine("");
                    }
                    WriteLine("");
                }
            }

#>

For the way the BimlExpress works, you need to either have all the packages open (so_69215306.biml as my image shows) and the various WriteLine(s) will show in the preview panel.

enter image description here

When you want to attempt to do this across your whole ecosystem of packages, then you'll want to multi-select in the Miscellaneous section of Visual Studio and then right click on your inspector script (for my screenshot, that would be BimlScript6.biml) and Check Biml for Error or Generate SSIS package.

Both will have the same effect - assuming you're just using Biml for the object model capability. Instead of WriteLines which are helpful for debugging, I assume you'll write to a something json/csv/txt/database/whatever

Step 4. Fall in love with all the power of Biml and buy BimlStudio

Example of converting a project to Biml

  • Convert to Biml
  • Select the project and [Import] (this identifies what you could bring into the Biml project definition)
  • Select the assets you want to [Add to Project] (likely everything)
  • Right click on project and choose
    enter image description here
~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文