如何将多个Drop表语句传递到DBT中的循环?

发布于 2025-02-05 18:12:12 字数 3044 浏览 3 评论 0原文

我想在雪花上放多个桌子。我没有将多个命令添加到我的DBT作业中,而只需传递表名称,而是在数据库的Information_Schema下使用了表视图,并找到了一种从那里构造我的Drop语句的方法。

这是我用来构建我的下降语句的语句:

SELECT
    CONCAT('"DROP TABLE IF EXISTS ', CONCAT(CONCAT_WS('.', TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME), '"'))
FROM "{{ var('db_name') }}"."INFORMATION_SCHEMA"."TABLES"
WHERE table_owner = '{{ var("table_owner") }}'
AND table_schema = '{{ var("schema_name") }}'
AND table_name LIKE '{{ var("table_name") }}'

我的第一种方法(我也发布了一个问题)是将上述语句的结果放到一个字符串中,然后将其传递到变量,然后将该变量传递给我的post_hook。但是显然,post_hook仅接受字符串值。请参阅此 post。

我的下一个方法是将结果列表传递给带有run_query命令的for loop而不是将结果列表传递给一个字符串。

这是我的整个代码:

{%- set generate_drop_table_statement -%}
SELECT
    CONCAT('"DROP TABLE IF EXISTS ', CONCAT(CONCAT_WS('.', TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME), '"'))
FROM "{{ var('db_name') }}"."INFORMATION_SCHEMA"."TABLES"
WHERE table_owner = '{{ var("table_owner") }}'
AND table_schema = '{{ var("schema_name") }}'
AND table_name LIKE '{{ var("table_name") }}'
{%- endset -%}

{%- set get_drop_statements = run_query(generate_drop_table_statement) -%}

{%- if execute -%}
{%- set drop_statements = get_drop_statements.columns[0].values() -%}

{%- set exec_query -%}
    {{ log(drop_statements, info=True) }}
    
    {% for drop_statement in drop_statements %}
        {{ log(drop_statement, info=True) }}
        {%- do run_query(drop_statement) -%}
    {% endfor %}
{%- endset -%}

{%- else -%}
{%- set drop_statements = [] -%}
{%- endif -%}

{{ config (
alias='DROP_STATEMENTS_TABLE',
materialized='table',
post_hook='{{ exec_query }}'
) }}

with statements as (
SELECT
    1 as id
)

SELECT *
FROM statements

从日志中我可以看到,由于以下行,查询已经传递:{log(drop_statement,info = true)}}},但是在显示第一个语句后,它失败了到run_query部分。

这是我遇到的错误:

Tried to run an empty query on model 'model.learn_dbt.drop_table'. If you are conditionally running
sql, eg. in a model hook, make sure your `else` clause contains valid sql!

Provided SQL:
/* {"app": "dbt", "dbt_version": "1.1.0", "profile_name": "eric-snowflake-dbt", "target_name": "dev", "node_id": "model.learn_dbt.drop_table"} */

"DROP TABLE IF EXISTS ANALYTICS.DBT.TEST_X_TABLE"

如何解决这个问题?

提前致谢。

Update

我刚刚更新了我的声明:

SELECT
    CONCAT('DROP TABLE IF EXISTS ', CONCAT_WS('.', TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME))
FROM "{{ var('db_name') }}"."INFORMATION_SCHEMA"."TABLES"
WHERE table_owner = '{{ var("table_owner") }}'
AND table_schema = '{{ var("schema_name") }}'
AND table_name LIKE '{{ var("table_name") }}'

我在示例中注意到,我发现传递给run_query的语句已包含在 single quotes 引号。那解决了我的问题。

希望这将来将来刚刚开始学习DBT的人。

I want to drop multiple tables in Snowflake. Instead of adding multiple commands to my dbt job and just pass the tables names, I used the TABLES view under the INFORMATION_SCHEMA of my database and found a way to construct my drop statements from there.

Here's the statement I used to construct my drop statements:

SELECT
    CONCAT('"DROP TABLE IF EXISTS ', CONCAT(CONCAT_WS('.', TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME), '"'))
FROM "{{ var('db_name') }}"."INFORMATION_SCHEMA"."TABLES"
WHERE table_owner = '{{ var("table_owner") }}'
AND table_schema = '{{ var("schema_name") }}'
AND table_name LIKE '{{ var("table_name") }}'

My first approach (Which I posted a question too) was to put the result of the above statement to a single string and pass it to a variable then pass that variable to my post_hook. But apparently, post_hook only accepts string values. See this post.

My next approach was instead of putting the result to a single string, I'll pass the result list to a for loop with run_query command inside.

Here's my whole code:

{%- set generate_drop_table_statement -%}
SELECT
    CONCAT('"DROP TABLE IF EXISTS ', CONCAT(CONCAT_WS('.', TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME), '"'))
FROM "{{ var('db_name') }}"."INFORMATION_SCHEMA"."TABLES"
WHERE table_owner = '{{ var("table_owner") }}'
AND table_schema = '{{ var("schema_name") }}'
AND table_name LIKE '{{ var("table_name") }}'
{%- endset -%}

{%- set get_drop_statements = run_query(generate_drop_table_statement) -%}

{%- if execute -%}
{%- set drop_statements = get_drop_statements.columns[0].values() -%}

{%- set exec_query -%}
    {{ log(drop_statements, info=True) }}
    
    {% for drop_statement in drop_statements %}
        {{ log(drop_statement, info=True) }}
        {%- do run_query(drop_statement) -%}
    {% endfor %}
{%- endset -%}

{%- else -%}
{%- set drop_statements = [] -%}
{%- endif -%}

{{ config (
alias='DROP_STATEMENTS_TABLE',
materialized='table',
post_hook='{{ exec_query }}'
) }}

with statements as (
SELECT
    1 as id
)

SELECT *
FROM statements

I can see from the log that the query has been passed because of this line: {{ log(drop_statement, info=True) }} but after displaying the first statement, it failed to the run_query part.

This is the error that I encountered:

Tried to run an empty query on model 'model.learn_dbt.drop_table'. If you are conditionally running
sql, eg. in a model hook, make sure your `else` clause contains valid sql!

Provided SQL:
/* {"app": "dbt", "dbt_version": "1.1.0", "profile_name": "eric-snowflake-dbt", "target_name": "dev", "node_id": "model.learn_dbt.drop_table"} */

"DROP TABLE IF EXISTS ANALYTICS.DBT.TEST_X_TABLE"

Any idea how to resolve this?

Thanks, in advance.

UPDATE

I just updated my statement to this:

SELECT
    CONCAT('DROP TABLE IF EXISTS ', CONCAT_WS('.', TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME))
FROM "{{ var('db_name') }}"."INFORMATION_SCHEMA"."TABLES"
WHERE table_owner = '{{ var("table_owner") }}'
AND table_schema = '{{ var("schema_name") }}'
AND table_name LIKE '{{ var("table_name") }}'

I noticed in the examples I found that statements passed to run_query was enclosed in single quotes, not double quotes. That fixed my problem.

Hope this will someone in future who are just starting to learn dbt.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文