spark json数据 查询年龄 age 的平均值;
时间: 2023-10-27 20:14:27 浏览: 53
假设有以下 JSON 数据:
```
{
"employees": [
{
"name": "John Doe",
"age": 30
},
{
"name": "Jane Smith",
"age": 25
},
{
"name": "Bob Johnson",
"age": 35
}
]
}
```
可以使用 Spark SQL 的方式查询年龄的平均值:
```scala
import org.apache.spark.sql.functions.avg
import org.apache.spark.sql.types.{IntegerType, StructField, StructType}
import org.apache.spark.sql.{Row, SparkSession}
val spark = SparkSession.builder()
.appName("JsonData")
.master("local[*]")
.getOrCreate()
val jsonData =
"""
|{
| "employees": [
| {
| "name": "John Doe",
| "age": 30
| },
| {
| "name": "Jane Smith",
| "age": 25
| },
| {
| "name": "Bob Johnson",
| "age": 35
| }
| ]
|}
""".stripMargin
val schema = StructType(Seq(
StructField("name", StringType, nullable = true),
StructField("age", IntegerType, nullable = true)
))
val rows = spark.read
.schema(schema)
.json(Seq(jsonData).toDS())
.select("age")
.collect()
val avgAge = rows.map(_.getInt(0)).sum.toDouble / rows.length
println(f"Avg age: $avgAge%.2f")
```
输出结果为:
```
Avg age: 30.00
```
相关推荐
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![xlsx](https://img-home.csdnimg.cn/images/20210720083732.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)