Post

Fun fact: DuckDB is faster at counting the lines of a CSV file than the UNIX word count command – and it also parses the file to identify its dialect (separator, quote character, etc.). Here are the timings for a 3 GB CSV file: 2.966 seconds for `wc -l` and 1.261 seconds for DuckDB.

2/12/24, 3:49 pm25 reposts132 likes

Uwe Maurer

@uwemaurer.bsky.social

impressive!

The results also hold up for warm runs, see the output of the Hyperfine benchmark tool.

Gabor Szarnyas

@szarnyasg.org

4 Dec 2024

My post on DuckDB vs. wc received a lot of feedback. Based on these, I ran a few more experiments to see how DuckDB stacks up against parallelized wc and grep/ripgrep on Linux. I wrote up my results in a blog post. TL;DR: it depends but DuckDB is still pretty fast! szarnyasg.org/posts/duckdb...

Longrod von HugenDong

@avitoholeus.bsky.social

📌

Right is it largely because it's multithreaded?

Travis Campbell

@hcoyote.bsky.social

2 Dec 2024

Am I reading this right? 2x faster, but using 10x the cpu?

DuckDB

@duckdb.org

2 Dec 2024

Correct. DuckDB's code path is a more generic one that also includes running the CSV sniffer to identify the CSV file's dialect, column names and types.