if self.config.deduplicate_order_by_type == "asc": order_by_columns = [F.col(column_name) for column_name in self.config.deduplication_order_columns] ranking_column = "duplicates_rank" is_deleted_column = "IsDeleted" if (self.config.filter_deleted_flag) & ( is_deleted_column in df.columns): # if True removes records that marked deleted filter = (F.col(ranking_column) == 1) & (F.col(is_deleted_column) == "False") else: self.logger.warning(f"Records marked as deleted will be loaded to {self.config.table_full_name} table!") filter = F.col(ranking_column) == 1 return df.withColumn(ranking_column, F.row_number().over(Window.partitionBy(key_columns).orderBy(order_by_columns))).filter( filter).drop(ranking_column)将这个转为sql形式

将这段 Python 代码转换成 SQL 代码的大致形式如下：

IF self.config.deduplicate_order_by_type = "asc" THEN
    SET @order_by_columns = CONCAT_WS(',', column_name1, column_name2, ...);
ELSEIF self.config.deduplicate_order_by_type = "desc" THEN
    SET @order_by_columns = CONCAT_WS(',', CONCAT(column_name1, ' DESC'), CONCAT(column_name2, ' DESC'), ...);
END IF;

IF self.config.filter_deleted_flag AND FIND_IN_SET('IsDeleted', @column_names) THEN
    SET @filter = CONCAT('(', duplicates_rank, '= 1 AND IsDeleted = "False")');
ELSE
    SET @filter = CONCAT('(', duplicates_rank, '= 1)');
END IF;

SELECT column_name1, column_name2, ..., 
       ROW_NUMBER() OVER (PARTITION BY @key_columns ORDER BY @order_by_columns) AS duplicates_rank
FROM table_name
WHERE @filter;

其中，`self.config.deduplicate_order_by_type` 是用于判断排序方式的变量，`column_name1, column_name2, ...` 是需要去重的列名，`table_name` 是表名，`key_columns` 是用于分组的列名，`duplicates_rank` 是新生成的列名，用于标记数据是否重复，`IsDeleted` 是用于标记数据是否已经删除的列名。

该 SQL 代码使用了 ROW_NUMBER() 函数来生成新列 `duplicates_rank`，并且使用 IF 和 ELSEIF 语句来判断排序方式，使用 CONCAT_WS() 和 CONCAT() 函数来生成排序的列名和筛选条件。如果 `self.config.filter_deleted_flag` 为 True 并且表中包含 `IsDeleted` 列，则会在筛选时去除被标记为删除的记录；否则会提示将标记为删除的记录加载到表中。最后，生成的结果集中包含原来的列和新列 `duplicates_rank`，并且根据 `duplicates_rank` 列进行了筛选。