pyspark二手车价格预测代码
时间: 2023-08-18 13:04:01 浏览: 56
这里给出一个简单的PySpark二手车价格预测代码示例:
```python
# 导入必要的库
from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import LinearRegression
from pyspark.ml.evaluation import RegressionEvaluator
# 创建SparkSession对象
spark = SparkSession.builder.appName("UsedCarPricePrediction").getOrCreate()
# 读取数据
df = spark.read.csv("used_car.csv", header=True, inferSchema=True)
# 数据清洗和处理
df = df.dropna() # 删除缺失值
df = df.dropDuplicates() # 删除重复值
df = df.withColumn("age", 2021-df["year"]) # 计算车龄
df = df.select("price", "age", "mileage", "brand", "model", "condition") # 选择相关特征
# 特征工程
assembler = VectorAssembler(inputCols=["age", "mileage"], outputCol="features")
df = assembler.transform(df)
# 划分训练集和测试集
train, test = df.randomSplit([0.7, 0.3], seed=123)
# 模型训练
lr = LinearRegression(featuresCol="features", labelCol="price")
model = lr.fit(train)
# 模型评估
evaluator = RegressionEvaluator(labelCol="price", predictionCol="prediction", metricName="rmse")
predictions = model.transform(test)
rmse = evaluator.evaluate(predictions)
print(f"Root Mean Squared Error (RMSE) on test data: {rmse:.2f}")
# 预测新数据
new_data = spark.createDataFrame([(2, 120000)], ["age", "mileage"])
new_data = assembler.transform(new_data)
prediction = model.transform(new_data)
print(f"Predicted price for a car with age 2 and mileage 120000: {prediction.collect()[0]['prediction']:.2f}")
# 关闭SparkSession对象
spark.stop()
```
需要注意的是,这只是一个简单的示例,实际情况下需要更多的特征工程和模型调优。