pyspark 查看hdfs文件是否存在

在PySpark中，你可以使用`HadoopFileSystem`或者`SparkContext`提供的`isFile`方法来检查HDFS（Hadoop Distributed File System）文件是否存在。这里是一个简单的例子：

from pyspark.sql import SparkSession

# 创建SparkSession
spark = SparkSession.builder.getOrCreate()

# 假设hdfs_path是你要检查的文件路径
hdfs_path = "hdfs://your_hdfs_address/path/to/your/file"

# 使用HadoopFileSystem
fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration())
exists = fs.exists(hdfs_path)
file_exists = exists if exists else False

if file_exists:
    print(f"The file {hdfs_path} exists in HDFS.")
else:
    print(f"The file {hdfs_path} does not exist in HDFS.")

# 或者通过SparkContext
import org.apache.hadoop.fs.Path
path = Path(hdfs_path)
exists_in_spark = spark.sparkContext._gateway.jvm.org.apache.hadoop.fs.FileUtil.confirmPathExists(path.toString())
file_exists_spark = exists_in_spark if exists_in_spark else False