pyspark二手车价格预测天池数据集的代码

以下是使用pyspark对天池二手车价格预测数据集进行模型训练和预测的代码：

首先，我们需要导入必要的库和读取数据集：

from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import LinearRegression

# 创建SparkSession
spark = SparkSession.builder.appName("UsedCarPricePrediction").getOrCreate()

# 读取数据集
data = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("data.csv")

接下来，我们需要进行数据预处理，包括特征工程和数据清洗：

# 特征工程
assembler = VectorAssembler(inputCols=["Brand", "Body", "Mileage", "EngineV", "Engine Type", "Registration", "Year"],
                            outputCol="features")
data = assembler.transform(data)

# 数据清洗
data = data.select("Price", "features").na.drop()

然后，我们将数据集分为训练集和测试集，用训练集训练线性回归模型：

# 将数据集分为训练集和测试集
(trainingData, testData) = data.randomSplit([0.7, 0.3], seed=100)

# 训练线性回归模型
lr = LinearRegression(featuresCol="features", labelCol="Price", maxIter=10, regParam=0.3, elasticNetParam=0.8)
model = lr.fit(trainingData)

最后，我们用测试集进行模型评估和预测：

# 模型评估
predictions = model.transform(testData)
predictions.select("prediction", "Price", "features").show()

# 预测单个数据
single_data = spark.createDataFrame([(2013, "Audi", "suv", 120000, 2.0, "Diesel", "yes")], ["Year", "Brand", "Body", "Mileage", "EngineV", "Engine Type", "Registration"])
single_data = assembler.transform(single_data)
prediction = model.transform(single_data)
prediction.select("prediction").show()

完整代码如下：

from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import LinearRegression

# 创建SparkSession
spark = SparkSession.builder.appName("UsedCarPricePrediction").getOrCreate()

# 读取数据集
data = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("data.csv")

# 特征工程
assembler = VectorAssembler(inputCols=["Brand", "Body", "Mileage", "EngineV", "Engine Type", "Registration", "Year"],
                            outputCol="features")
data = assembler.transform(data)

# 数据清洗
data = data.select("Price", "features").na.drop()

# 将数据集分为训练集和测试集
(trainingData, testData) = data.randomSplit([0.7, 0.3], seed=100)

# 训练线性回归模型
lr = LinearRegression(featuresCol="features", labelCol="Price", maxIter=10, regParam=0.3, elasticNetParam=0.8)
model = lr.fit(trainingData)

# 模型评估
predictions = model.transform(testData)
predictions.select("prediction", "Price", "features").show()

# 预测单个数据
single_data = spark.createDataFrame([(2013, "Audi", "suv", 120000, 2.0, "Diesel", "yes")], ["Year", "Brand", "Body", "Mileage", "EngineV", "Engine Type", "Registration"])
single_data = assembler.transform(single_data)
prediction = model.transform(single_data)
prediction.select("prediction").show()