使用UDF时，Pyspark应用程序无法找到Python

发布于 2025-02-11 22:45:15 字数 1078 浏览 1 评论 0原文

当我尝试在本地Pyspark应用程序中使用UDF时，出现错误，说找不到Python。

示例UDF我正在使用：

import pyspark.sql.functions as sql_func 

test_udf = sql_func.udf(lambda x: x.lower(), StringType())
df.withColumn("Sample", test_udf("SampleCol")).show(5)

错误消息，

Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
22/06/30 10:49:45 WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped
22/06/30 10:49:46 ERROR Executor: Exception in task 0.0 in stage 4.0 (TID 7)
org.apache.spark.SparkException: Python worker failed to connect back.

但是，仅当我评估Spark UDF时，此错误才会发生。我能够将Spark DataFrame转换为PANDAS DataFrame，并在没有问题的情况下应用相同的功能。但是，每当我使用df.show（）应用UDF后，每当我评估数据帧时，它都无法正常工作。

关于我的设置的注释：

在Windows 10上运行
使用python 3.9.6
Spark 3.3

我尝试过的事物

：从设置＆GT中禁用捷径；管理应用程序执行别名
将Pyspark_python添加到我的系统环境变量中，

谢谢您的帮助！

原文

When I try to use a UDF in my local pyspark application, an error is produced saying python cannot be found.

Sample UDF I am using:

import pyspark.sql.functions as sql_func 

test_udf = sql_func.udf(lambda x: x.lower(), StringType())
df.withColumn("Sample", test_udf("SampleCol")).show(5)

Error message

Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
22/06/30 10:49:45 WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped
22/06/30 10:49:46 ERROR Executor: Exception in task 0.0 in stage 4.0 (TID 7)
org.apache.spark.SparkException: Python worker failed to connect back.

However, this error only happens when I am evaluating a Spark UDF. I am able to convert the Spark DataFrame to a Pandas DataFrame and apply the same function without an issue. However, whenever I evaluate the DataFrame with a df.show() after the udf is applied it does not work.

Notes about my set up: