datax将postgree多张表写入impala一张表中
时间: 2023-06-12 10:07:47 浏览: 245
可以使用datax的多个reader和writer进行数据传输。具体步骤如下:
1. 配置PostgreSQL的reader,读取多张表的数据。
```json
{
"job": {
"content": [
{
"reader": {
"name": "postgresqlreader",
"parameter": {
"column": [
"id",
"name",
"age"
],
"connection": [
{
"jdbcUrl": "jdbc:postgresql://localhost:5432/db1",
"table": [
"table1",
"table2",
"table3"
],
"username": "user",
"password": "password"
}
],
"where": "1=1"
}
},
"writer": null
}
],
"setting": {
"errorLimit": {
"record": "0"
},
"speed": {
"channel": "3"
}
}
}
}
```
2. 配置Impala的writer,将数据写入一张表中。
```json
{
"job": {
"content": [
{
"reader": {
"name": "postgresqlreader",
"parameter": {
"column": [
"id",
"name",
"age"
],
"connection": [
{
"jdbcUrl": "jdbc:postgresql://localhost:5432/db1",
"table": [
"table1",
"table2",
"table3"
],
"username": "user",
"password": "password"
}
],
"where": "1=1"
}
},
"writer": {
"name": "impalawriter",
"parameter": {
"preSql": "TRUNCATE TABLE impalatable",
"connection": {
"jdbcUrl": "jdbc:impala://localhost:21050/default",
"username": "user",
"password": "password"
},
"table": "impalatable",
"column": [
"id",
"name",
"age"
],
"postSql": null
}
}
}
],
"setting": {
"errorLimit": {
"record": "0"
},
"speed": {
"channel": "3"
}
}
}
}
```
注意:需要在Impala中创建好表,并且Impala的JDBC驱动需要提前配置好。
阅读全文