写一个用Ehcache读取2G的CSV文件并缓存到临时文件中,最后以List<dto>的形式传出的demo
时间: 2024-05-07 22:23:22 浏览: 168
以下是一个基于 Ehcache 的读取大型 CSV 文件并缓存到临时文件中的示例代码:
```java
import org.ehcache.Cache;
import org.ehcache.CacheManager;
import org.ehcache.config.builders.CacheConfigurationBuilder;
import org.ehcache.config.builders.ResourcePoolsBuilder;
import org.ehcache.impl.config.persistence.CacheManagerPersistenceConfiguration;
import org.ehcache.impl.persistence.DefaultLocalPersistenceService;
import org.ehcache.persistence.PersistentStorageFactory;
import org.supercsv.cellprocessor.ParseBool;
import org.supercsv.cellprocessor.ParseDate;
import org.supercsv.cellprocessor.ParseInt;
import org.supercsv.cellprocessor.ParseLong;
import org.supercsv.cellprocessor.ParseBigDecimal;
import org.supercsv.cellprocessor.Optional;
import org.supercsv.cellprocessor.constraint.NotNull;
import org.supercsv.cellprocessor.ift.CellProcessor;
import org.supercsv.io.CsvBeanReader;
import org.supercsv.io.ICsvBeanReader;
import org.supercsv.prefs.CsvPreference;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class CsvCacheDemo {
private static final String CACHE_NAME = "csv-cache";
private static final String CACHE_PATH = "temp/cache";
private static final String CSV_PATH = "data.csv";
public static void main(String[] args) throws IOException {
// 创建缓存管理器
CacheManager cacheManager = createCacheManager();
// 从缓存获取数据,如果缓存不存在则从 CSV 文件读取并缓存
List<DemoDto> demoList = readFromCache(cacheManager);
// 打印结果
demoList.forEach(System.out::println);
// 关闭缓存管理器
cacheManager.close();
}
private static CacheManager createCacheManager() {
// 创建本地持久化服务
DefaultLocalPersistenceService persistenceService = new DefaultLocalPersistenceService(
new CacheManagerPersistenceConfiguration(new File(CACHE_PATH)));
// 创建缓存管理器
return CacheManagerBuilder.newCacheManagerBuilder()
.using(persistenceService)
.withCache(CACHE_NAME, CacheConfigurationBuilder.newCacheConfigurationBuilder(
String.class, List.class, ResourcePoolsBuilder.heap(1))
.withExpiry(Expirations.timeToLiveExpiration(Duration.ofMinutes(10))))
.build(true);
}
private static List<DemoDto> readFromCache(CacheManager cacheManager) throws IOException {
// 获取或创建缓存
Cache<String, List<DemoDto>> cache = cacheManager.getCache(CACHE_NAME, String.class, List.class);
// 尝试从缓存读取数据
String key = CSV_PATH;
List<DemoDto> demoList = cache.get(key);
if (demoList == null) {
// 缓存不存在,从 CSV 文件读取并缓存
demoList = readFromCsv();
cache.put(key, demoList);
}
return demoList;
}
private static List<DemoDto> readFromCsv() throws IOException {
// 创建 CSV 读取器
FileReader fileReader = new FileReader(CSV_PATH);
ICsvBeanReader beanReader = new CsvBeanReader(fileReader, CsvPreference.STANDARD_PREFERENCE);
// 定义 CSV 列名和属性映射关系
final String[] header = beanReader.getHeader(true);
final CellProcessor[] processors = new CellProcessor[] {
new NotNull(), new ParseLong(), new Optional(new ParseInt()), new Optional(new ParseBigDecimal()),
new Optional(new ParseDate("yyyy-MM-dd")), new Optional(new ParseBool())};
// 读取 CSV 数据并转换为 DTO 对象
List<DemoDto> demoList = new ArrayList<>();
DemoDto demoDto;
while ((demoDto = beanReader.read(DemoDto.class, header, processors)) != null) {
demoList.add(demoDto);
}
// 关闭 CSV 读取器
beanReader.close();
// 将数据写入临时文件
File tempFile = File.createTempFile("csv-cache", ".tmp");
FileWriter fileWriter = new FileWriter(tempFile);
tempFile.deleteOnExit();
for (DemoDto dto : demoList) {
fileWriter.write(dto.toString());
fileWriter.write(System.lineSeparator());
}
fileWriter.close();
return demoList;
}
public static class DemoDto {
private String name;
private long id;
private Integer age;
private BigDecimal amount;
private Date date;
private Boolean flag;
// getter 和 setter 省略
}
}
```
此示例代码使用 Ehcache 缓存管理器和本地持久化服务,将读取的 CSV 数据缓存到 `temp/cache` 目录下的临时文件中,并以 `List<DemoDto>` 的形式返回给调用方。如果缓存中存在数据,则直接从缓存中读取;否则从 CSV 文件读取并缓存到缓存和临时文件中。
阅读全文