点击House：列式数据库 ClickHouse 全解析与操作指南

需积分: 50 42 浏览量更新于2024-07-18 收藏 3.14MB PDF 举报

ClickHouse是一种列式数据库管理系统（Column-Oriented DBMS），专为在线分析处理（OLAP）查询而设计。与传统的行式数据库管理系统（如MySQL、PostgreSQL和Microsoft SQL Server）不同，ClickHouse采用列存架构，数据存储方式更为高效。在行式数据库中，数据按照特定顺序排列，如上面提供的示例所示，每行包含所有相关的字段值。例如，对于用户事件记录，每一行会依次包含watch_id、JavaEnable、Title、GoodEvent和EventTime等字段，所有这些数据紧密地存储在一起。相反，ClickHouse的列式存储意味着数据以列的形式组织。每个列单独存储，比如watch_id列只包含所有watch_id的值，JavaEnable列则包含所有JavaEnable的状态，如此类推。这样做的优势在于： 1. **优化读取**：当需要查询某一列时，系统可以并行读取该列的所有值，因为它们都在同一位置，无需遍历整个行。这对于执行分析性查询，特别是聚合函数（如COUNT、SUM、AVG等）非常有利，因为列式存储减少了I/O操作次数。 2. **压缩存储**：由于列式存储减少了冗余，可以更好地利用空间，尤其对于非频繁更新的静态数据，列式存储能够节省存储空间。 3. **性能提升**：对于大数据量和复杂查询，ClickHouse的列式设计有助于减少计算时间和内存消耗，因为它允许快速定位所需的列数据，而不是全表扫描。 4. **扩展性**：ClickHouse的列式结构支持水平扩展，即通过添加更多的服务器来增加特定列的处理能力，这使得系统在处理大量数据时更具灵活性。 5. **实时处理**：ClickHouse设计用于实时分析，它可以快速响应实时或近实时的数据请求，特别适合实时监控、日志分析和流处理场景。举例来说，当你在ClickHouse中查询某个时间段内所有用户的JavaEnable状态时，系统可以快速定位到JavaEnable列，而无需遍历整个表。这使得ClickHouse在处理大数据集时具有很高的查询效率。 ClickHouse凭借其独特的列式存储架构，提供了高性能的分析查询功能，适用于需要处理大规模数据、实时分析以及复杂查询的场景。如果你打算深入学习或使用ClickHouse，理解其工作原理和优点至关重要。

Thenthedatamustbepre-processedinPostgreSQL.Thiswillcreateselectionsofpointsinthepolygons(tomatchpointson

themapwiththeboroughsofNewYorkCity)andcombineallthedataintoasingledenormalizedflattablebyusingaJOIN.

Todothis,youwillneedtoinstallPostgreSQLwithPostGISsupport.

Becarefulwhenrunning initialize_database.sh andmanuallyre-checkthatallthetableswerecreatedcorrectly.

Ittakesabout20-30minutestoprocesseachmonth'sworthofdatainPostgreSQL,foratotalofabout48hours.

Youcancheckthenumberofdownloadedrowsasfollows:

(Thisisslightlymorethan1.1billionrowsreportedbyMarkLitwintschikinaseriesofblogposts.)

ThedatainPostgreSQLuses370GBofspace.

ExportingthedatafromPostgreSQL:

sed -E '/(.*,){18,}/d' data/yellow_tripdata_2010-02.csv > data/yellow_tripdata_2010-02.csv_

sed -E '/(.*,){18,}/d' data/yellow_tripdata_2010-03.csv > data/yellow_tripdata_2010-03.csv_

mv data/yellow_tripdata_2010-02.csv_ data/yellow_tripdata_2010-02.csv

mv data/yellow_tripdata_2010-03.csv_ data/yellow_tripdata_2010-03.csv

time psql nyc-taxi-data -c "SELECT count(*) FROM trips;"

### Count

1298979494

(1 row)

real 7m9.164s

Thedatasnapshotiscreatedataspeedofabout50MBpersecond.Whilecreatingthesnapshot,PostgreSQLreadsfrom

thediskataspeedofabout28MBpersecond.Thistakesabout5hours.TheresultingTSVfileis590612904969bytes.

CreateatemporarytableinClickHouse:

COPY

(

SELECT trips.id,

trips.vendor_id,

trips.pickup_datetime,

trips.dropoff_datetime,

trips.store_and_fwd_flag,

trips.rate_code_id,

trips.pickup_longitude,

trips.pickup_latitude,

trips.dropoff_longitude,

trips.dropoff_latitude,

trips.passenger_count,

trips.trip_distance,

trips.fare_amount,

trips.extra,

trips.mta_tax,

trips.tip_amount,

trips.tolls_amount,

trips.ehail_fee,

trips.improvement_surcharge,

trips.total_amount,

trips.payment_type,

trips.trip_type,

trips.pickup,

trips.dropoff,

cab_types.type cab_type,

weather.precipitation_tenths_of_mm rain,

weather.snow_depth_mm,

weather.snowfall_mm,

weather.max_temperature_tenths_degrees_celsius max_temp,

weather.min_temperature_tenths_degrees_celsius min_temp,

weather.average_wind_speed_tenths_of_meters_per_second wind,

pick_up.gid pickup_nyct2010_gid,

pick_up.ctlabel pickup_ctlabel,

pick_up.borocode pickup_borocode,

pick_up.boroname pickup_boroname,

pick_up.ct2010 pickup_ct2010,

pick_up.boroct2010 pickup_boroct2010,

pick_up.cdeligibil pickup_cdeligibil,

pick_up.ntacode pickup_ntacode,

pick_up.ntaname pickup_ntaname,

pick_up.puma pickup_puma,

drop_off.gid dropoff_nyct2010_gid,

drop_off.ctlabel dropoff_ctlabel,

drop_off.borocode dropoff_borocode,

drop_off.boroname dropoff_boroname,

drop_off.ct2010 dropoff_ct2010,

drop_off.boroct2010 dropoff_boroct2010,

drop_off.cdeligibil dropoff_cdeligibil,

drop_off.ntacode dropoff_ntacode,

drop_off.ntaname dropoff_ntaname,

drop_off.puma dropoff_puma

FROM trips

LEFT JOIN cab_types

ON trips.cab_type_id = cab_types.id

LEFT JOIN central_park_weather_observations_raw weather

ON weather.date = trips.pickup_datetime::date

LEFT JOIN nyct2010 pick_up

ON pick_up.gid = trips.pickup_nyct2010_gid

LEFT JOIN nyct2010 drop_off

ON drop_off.gid = trips.dropoff_nyct2010_gid

) TO '/opt/milovidov/nyc-taxi-data/trips.tsv';

Itisneededforconvertingfieldstomorecorrectdatatypesand,ifpossible,toeliminateNULLs.

Dataisreadataspeedof112-140Mb/second.LoadingdataintoaLogtypetableinonestreamtook76minutes.Thedatain

thistableuses142GB.

(ImportingdatadirectlyfromPostgresisalsopossibleusing COPY...TOPROGRAM .)

Unfortunately,allthefieldsassociatedwiththeweather(precipitation...average_wind_speed)werefilledwithNULL.Because

ofthis,wewillremovethemfromthefinaldataset.

Tostart,we'llcreateatableonasingleserver.Laterwewillmakethetabledistributed.

CREATE TABLE trips

(

trip_id UInt32,

vendor_id String,

pickup_datetime DateTime,

dropoff_datetime Nullable(DateTime),

store_and_fwd_flag Nullable(FixedString(1)),

rate_code_id Nullable(UInt8),

pickup_longitude Nullable(Float64),

pickup_latitude Nullable(Float64),

dropoff_longitude Nullable(Float64),

dropoff_latitude Nullable(Float64),

passenger_count Nullable(UInt8),

trip_distance Nullable(Float64),

fare_amount Nullable(Float32),

extra Nullable(Float32),

mta_tax Nullable(Float32),

tip_amount Nullable(Float32),

tolls_amount Nullable(Float32),

ehail_fee Nullable(Float32),

improvement_surcharge Nullable(Float32),

total_amount Nullable(Float32),

payment_type Nullable(String),

trip_type Nullable(UInt8),

pickup Nullable(String),

dropoff Nullable(String),

cab_type Nullable(String),

precipitation Nullable(UInt8),

snow_depth Nullable(UInt8),

snowfall Nullable(UInt8),

max_temperature Nullable(UInt8),

min_temperature Nullable(UInt8),

average_wind_speed Nullable(UInt8),

pickup_nyct2010_gid Nullable(UInt8),

pickup_ctlabel Nullable(String),

pickup_borocode Nullable(UInt8),

pickup_boroname Nullable(String),

pickup_ct2010 Nullable(String),

pickup_boroct2010 Nullable(String),

pickup_cdeligibil Nullable(FixedString(1)),

pickup_ntacode Nullable(String),

pickup_ntaname Nullable(String),

pickup_puma Nullable(String),

dropoff_nyct2010_gid Nullable(UInt8),

dropoff_ctlabel Nullable(String),

dropoff_borocode Nullable(UInt8),

dropoff_boroname Nullable(String),

dropoff_ct2010 Nullable(String),

dropoff_boroct2010 Nullable(String),

dropoff_cdeligibil Nullable(String),

dropoff_ntacode Nullable(String),

dropoff_ntaname Nullable(String),

dropoff_puma Nullable(String)

) ENGINE = Log;

time clickhouse-client --query="INSERT INTO trips FORMAT TabSeparated" < trips.tsv

real 75m56.214s

Createandpopulateasummarytable:

CREATE TABLE trips_mergetree

ENGINE = MergeTree(pickup_date, pickup_datetime, 8192)

AS SELECT

trip_id,

CAST(vendor_id AS Enum8('1' = 1, '2' = 2, 'CMT' = 3, 'VTS' = 4, 'DDS' = 5, 'B02512' = 10, 'B02598' = 11,

'B02617' = 12, 'B02682' = 13, 'B02764' = 14)) AS vendor_id,

toDate(pickup_datetime) AS pickup_date,

ifNull(pickup_datetime, toDateTime(0)) AS pickup_datetime,

toDate(dropoff_datetime) AS dropoff_date,

ifNull(dropoff_datetime, toDateTime(0)) AS dropoff_datetime,

assumeNotNull(store_and_fwd_flag) IN ('Y', '1', '2') AS store_and_fwd_flag,

assumeNotNull(rate_code_id) AS rate_code_id,

assumeNotNull(pickup_longitude) AS pickup_longitude,

assumeNotNull(pickup_latitude) AS pickup_latitude,

assumeNotNull(dropoff_longitude) AS dropoff_longitude,

assumeNotNull(dropoff_latitude) AS dropoff_latitude,

assumeNotNull(passenger_count) AS passenger_count,

assumeNotNull(trip_distance) AS trip_distance,

assumeNotNull(fare_amount) AS fare_amount,

assumeNotNull(extra) AS extra,

assumeNotNull(mta_tax) AS mta_tax,

assumeNotNull(tip_amount) AS tip_amount,

assumeNotNull(tolls_amount) AS tolls_amount,

assumeNotNull(ehail_fee) AS ehail_fee,

assumeNotNull(improvement_surcharge) AS improvement_surcharge,

assumeNotNull(total_amount) AS total_amount,

CAST((assumeNotNull(payment_type) AS pt) IN ('CSH', 'CASH', 'Cash', 'CAS', 'Cas', '1') ? 'CSH' : (pt IN

('CRD', 'Credit', 'Cre', 'CRE', 'CREDIT', '2') ? 'CRE' : (pt IN ('NOC', 'No Charge', 'No', '3') ? 'NOC' : (pt

IN ('DIS', 'Dispute', 'Dis', '4') ? 'DIS' : 'UNK'))) AS Enum8('CSH' = 1, 'CRE' = 2, 'UNK' = 0, 'NOC' = 3,

'DIS' = 4)) AS payment_type_,

assumeNotNull(trip_type) AS trip_type,

ifNull(toFixedString(unhex(pickup), 25), toFixedString('', 25)) AS pickup,

ifNull(toFixedString(unhex(dropoff), 25), toFixedString('', 25)) AS dropoff,

CAST(assumeNotNull(cab_type) AS Enum8('yellow' = 1, 'green' = 2, 'uber' = 3)) AS cab_type,

assumeNotNull(pickup_nyct2010_gid) AS pickup_nyct2010_gid,

toFloat32(ifNull(pickup_ctlabel, '0')) AS pickup_ctlabel,

assumeNotNull(pickup_borocode) AS pickup_borocode,

CAST(assumeNotNull(pickup_boroname) AS Enum8('Manhattan' = 1, 'Queens' = 4, 'Brooklyn' = 3, '' = 0, 'Bronx' =

2, 'Staten Island' = 5)) AS pickup_boroname,

toFixedString(ifNull(pickup_ct2010, '000000'), 6) AS pickup_ct2010,

toFixedString(ifNull(pickup_boroct2010, '0000000'), 7) AS pickup_boroct2010,

CAST(assumeNotNull(ifNull(pickup_cdeligibil, ' ')) AS Enum8(' ' = 0, 'E' = 1, 'I' = 2)) AS pickup_cdeligibil,

toFixedString(ifNull(pickup_ntacode, '0000'), 4) AS pickup_ntacode,

CAST(assumeNotNull(pickup_ntaname) AS Enum16('' = 0, 'Airport' = 1, 'Allerton-Pelham Gardens' = 2, 'Annadale-

Huguenot-Prince\'s Bay-Eltingville' = 3, 'Arden Heights' = 4, 'Astoria' = 5, 'Auburndale' = 6, 'Baisley Park'

= 7, 'Bath Beach' = 8, 'Battery Park City-Lower Manhattan' = 9, 'Bay Ridge' = 10, 'Bayside-Bayside Hills' =

11, 'Bedford' = 12, 'Bedford Park-Fordham North' = 13, 'Bellerose' = 14, 'Belmont' = 15, 'Bensonhurst East' =

16, 'Bensonhurst West' = 17, 'Borough Park' = 18, 'Breezy Point-Belle Harbor-Rockaway Park-Broad Channel' =

19, 'Briarwood-Jamaica Hills' = 20, 'Brighton Beach' = 21, 'Bronxdale' = 22, 'Brooklyn Heights-Cobble Hill' =

23, 'Brownsville' = 24, 'Bushwick North' = 25, 'Bushwick South' = 26, 'Cambria Heights' = 27, 'Canarsie' =

28, 'Carroll Gardens-Columbia Street-Red Hook' = 29, 'Central Harlem North-Polo Grounds' = 30, 'Central Harlem

South' = 31, 'Charleston-Richmond Valley-Tottenville' = 32, 'Chinatown' = 33, 'Claremont-Bathgate' = 34,

'Clinton' = 35, 'Clinton Hill' = 36, 'Co-op City' = 37, 'College Point' = 38, 'Corona' = 39, 'Crotona Park

East' = 40, 'Crown Heights North' = 41, 'Crown Heights South' = 42, 'Cypress Hills-City Line' = 43, 'DUMBO-

Vinegar Hill-Downtown Brooklyn-Boerum Hill' = 44, 'Douglas Manor-Douglaston-Little Neck' = 45, 'Dyker

Heights' = 46, 'East Concourse-Concourse Village' = 47, 'East Elmhurst' = 48, 'East Flatbush-Farragut' = 49,

'East Flushing' = 50, 'East Harlem North' = 51, 'East Harlem South' = 52, 'East New York' = 53, 'East New

York (Pennsylvania Ave)' = 54, 'East Tremont' = 55, 'East Village' = 56, 'East Williamsburg' = 57,

'Eastchester-Edenwald-Baychester' = 58, 'Elmhurst' = 59, 'Elmhurst-Maspeth' = 60, 'Erasmus' = 61, 'Far

Rockaway-Bayswater' = 62, 'Flatbush' = 63, 'Flatlands' = 64, 'Flushing' = 65, 'Fordham South' = 66, 'Forest

Hills' = 67, 'Fort Greene' = 68, 'Fresh Meadows-Utopia' = 69, 'Ft. Totten-Bay Terrace-Clearview' = 70,

'Georgetown-Marine Park-Bergen Beach-Mill Basin' = 71, 'Glen Oaks-Floral Park-New Hyde Park' = 72, 'Glendale'

= 73, 'Gramercy' = 74, 'Grasmere-Arrochar-Ft. Wadsworth' = 75, 'Gravesend' = 76, 'Great Kills' = 77,

'Greenpoint' = 78, 'Grymes Hill-Clifton-Fox Hills' = 79, 'Hamilton Heights' = 80, 'Hammels-Arverne-Edgemere' =

81, 'Highbridge' = 82, 'Hollis' = 83, 'Homecrest' = 84, 'Hudson Yards-Chelsea-Flatiron-Union Square' = 85,

'Hunters Point-Sunnyside-West Maspeth' = 86, 'Hunts Point' = 87, 'Jackson Heights' = 88, 'Jamaica' = 89,

'Jamaica Estates-Holliswood' = 90, 'Kensington-Ocean Parkway' = 91, 'Kew Gardens' = 92, 'Kew Gardens Hills' =

93, 'Kingsbridge Heights' = 94, 'Laurelton' = 95, 'Lenox Hill-Roosevelt Island' = 96, 'Lincoln Square' = 97,

'Lindenwood-Howard Beach' = 98, 'Longwood' = 99, 'Lower East Side' = 100, 'Madison' = 101, 'Manhattanville' =

102, 'Marble Hill-Inwood' = 103, 'Mariner\'s Harbor-Arlington-Port Ivory-Graniteville' = 104, 'Maspeth' =

105, 'Melrose South-Mott Haven North' = 106, 'Middle Village' = 107, 'Midtown-Midtown South' = 108, 'Midwood'

= 109, 'Morningside Heights' = 110, 'Morrisania-Melrose' = 111, 'Mott Haven-Port Morris' = 112, 'Mount Hope'

= 113, 'Murray Hill' = 114, 'Murray Hill-Kips Bay' = 115, 'New Brighton-Silver Lake' = 116, 'New Dorp-Midland

Beach' = 117, 'New Springville-Bloomfield-Travis' = 118, 'North Corona' = 119, 'North Riverdale-Fieldston-

Riverdale' = 120, 'North Side-South Side' = 121, 'Norwood' = 122, 'Oakland Gardens' = 123, 'Oakwood-Oakwood

Beach' = 124, 'Ocean Hill' = 125, 'Ocean Parkway South' = 126, 'Old Astoria' = 127, 'Old Town-Dongan Hills-

South Beach' = 128, 'Ozone Park' = 129, 'Park Slope-Gowanus' = 130, 'Parkchester' = 131, 'Pelham Bay-Country

Club-City Island' = 132, 'Pelham Parkway' = 133, 'Pomonok-Flushing Heights-Hillcrest' = 134, 'Port Richmond' =

135, 'Prospect Heights' = 136, 'Prospect Lefferts Gardens-Wingate' = 137, 'Queens Village' = 138, 'Queensboro

Hill' = 139, 'Queensbridge-Ravenswood-Long Island City' = 140, 'Rego Park' = 141, 'Richmond Hill' = 142,

'Ridgewood' = 143, 'Rikers Island' = 144, 'Rosedale' = 145, 'Rossville-Woodrow' = 146, 'Rugby-Remsen Village'

= 147, 'Schuylerville-Throgs Neck-Edgewater Park' = 148, 'Seagate-Coney Island' = 149, 'Sheepshead Bay-

Gerritsen Beach-Manhattan Beach' = 150, 'SoHo-TriBeCa-Civic Center-Little Italy' = 151, 'Soundview-Bruckner'

= 152, 'Soundview-Castle Hill-Clason Point-Harding Park' = 153, 'South Jamaica' = 154, 'South Ozone Park' =

155, 'Springfield Gardens North' = 156, 'Springfield Gardens South-Brookville' = 157, 'Spuyten Duyvil-

Kingsbridge' = 158, 'St. Albans' = 159, 'Stapleton-Rosebank' = 160, 'Starrett City' = 161, 'Steinway' = 162,

'Stuyvesant Heights' = 163, 'Stuyvesant Town-Cooper Village' = 164, 'Sunset Park East' = 165, 'Sunset Park

West' = 166, 'Todt Hill-Emerson Hill-Heartland Village-Lighthouse Hill' = 167, 'Turtle Bay-East Midtown' =

168, 'University Heights-Morris Heights' = 169, 'Upper East Side-Carnegie Hill' = 170, 'Upper West Side' =

171, 'Van Cortlandt Village' = 172, 'Van Nest-Morris Park-Westchester Square' = 173, 'Washington Heights

North' = 174, 'Washington Heights South' = 175, 'West Brighton' = 176, 'West Concourse' = 177, 'West Farms-

Bronx River' = 178, 'West New Brighton-New Brighton-St. George' = 179, 'West Village' = 180, 'Westchester-

Unionport' = 181, 'Westerleigh' = 182, 'Whitestone' = 183, 'Williamsbridge-Olinville' = 184, 'Williamsburg' =

185, 'Windsor Terrace' = 186, 'Woodhaven' = 187, 'Woodlawn-Wakefield' = 188, 'Woodside' = 189, 'Yorkville' =

190, 'park-cemetery-etc-Bronx' = 191, 'park-cemetery-etc-Brooklyn' = 192, 'park-cemetery-etc-Manhattan' =

193, 'park-cemetery-etc-Queens' = 194, 'park-cemetery-etc-Staten Island' = 195)) AS pickup_ntaname,

toUInt16(ifNull(pickup_puma, '0')) AS pickup_puma,

assumeNotNull(dropoff_nyct2010_gid) AS dropoff_nyct2010_gid,

toFloat32(ifNull(dropoff_ctlabel, '0')) AS dropoff_ctlabel,

assumeNotNull(dropoff_borocode) AS dropoff_borocode,

CAST(assumeNotNull(dropoff_boroname) AS Enum8('Manhattan' = 1, 'Queens' = 4, 'Brooklyn' = 3, '' = 0, 'Bronx' =

2, 'Staten Island' = 5)) AS dropoff_boroname,

toFixedString(ifNull(dropoff_ct2010, '000000'), 6) AS dropoff_ct2010,

toFixedString(ifNull(dropoff_boroct2010, '0000000'), 7) AS dropoff_boroct2010,

CAST(assumeNotNull(ifNull(dropoff_cdeligibil, ' ')) AS Enum8(' ' = 0, 'E' = 1, 'I' = 2)) AS

dropoff_cdeligibil,

toFixedString(ifNull(dropoff_ntacode, '0000'), 4) AS dropoff_ntacode,

CAST(assumeNotNull(dropoff_ntaname) AS Enum16('' = 0, 'Airport' = 1, 'Allerton-Pelham Gardens' = 2, 'Annadale-