pyspark 计算余弦相似度

在pyspark中计算余弦相似度可以使用Spark ML库中的VectorAssembler和CosineSimilarity来实现。首先，使用VectorAssembler将需要计算相似度的特征向量组合成一个向量列。然后，使用CosineSimilarity计算这些向量之间的余弦相似度。

以下是一个示例代码：

from pyspark.ml.feature import VectorAssembler
from pyspark.ml.feature import Normalizer
from pyspark.ml.linalg import Vectors
from pyspark.ml.feature import StringIndexer
from pyspark.ml.feature import IndexToString
from pyspark.ml.recommendation import ALS
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml.recommendation import ALSModel
from pyspark.ml import Pipeline
from pyspark.ml.feature import HashingTF, IDF
from pyspark.ml.feature import CountVectorizer
from pyspark.ml.feature import StopWordsRemover
from pyspark.ml.feature import NGram
from pyspark.ml.feature import Tokenizer
from pyspark.ml.feature import PCA
from pyspark.ml.feature import OneHotEncoder
from pyspark.ml.feature import PCA


# 创建一个DataFrame
df = spark.createDataFrame([
    (0, Vectors.dense([1.0, 0.5])),
    (1, Vectors.dense([2.0, 1.0])),
    (2, Vectors.dense([4.0, 2.0])),
    (3, Vectors.dense([6.0, 3.0]))
], ["id", "features"])

# 使用VectorAssembler将特征向量组合成一个向量列
assembler = VectorAssembler(
    inputCols=["features"],
    outputCol="vector"
)

output = assembler.transform(df)

# 使用CosineSimilarity计算余弦相似度
cosine_similarity = output.select("vector").crossJoin(output.select("vector")).toDF("v1", "v2").selectExpr("v1", "v2", "float(CosineSimilarity(v1, v2)) as similarity")

cosine_similarity.show()

这段代码中，我们首先创建一个包含特征向量的DataFrame，然后使用VectorAssembler将特征向量组合成一个向量列。接着，我们使用CosineSimilarity计算向量之间的余弦相似度。最后，我们将计算结果打印出来。

请注意，这只是一个示例代码，你需要根据你的实际数据和需求进行相应的修改和调整。<span class="em">1</span><span class="em">2</span><span class="em">3</span>
#### 引用[.reference_title]
- *1* [Spark机器学习——余弦相似性算法](https://blog.csdn.net/a805814077/article/details/113267214)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_2"}}] [.reference_item style="max-width: 50%"]
- *2* *3* [推荐系统01--余弦相似度](https://blog.csdn.net/weixin_40008870/article/details/110766812)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_2"}}] [.reference_item style="max-width: 50%"]
[ .reference_list ]