Crunchy Data

Crunchy Data

@crunchydata.com

Just Postgres www.crunchydata.com

2.1K followers269 following96 posts

Today we're announcing the availability of logical replication from Postgres to Iceberg with Crunchy Data Warehouse. Now you can seamlessly move data and stream changes from your operational database into an analytical system. www.crunchydata.com/blog/logical...

crunchydata.comLogical replication from Postgres to Iceberg | Crunchy Data BlogWe've launched native logical replication from Postgres tables in any Postgres server to Iceberg tables managed by Crunchy Data Warehouse.

How long has it been since you checked your cache hit ratio? Ideally most of your frequently queried data is in the buffer cache. We recommend 98-99% in the cache for transactional workloads - analytical workloads are lower.

Query the cache hit with: SELECT sum(heap_blks_read) as heap_read, sum(heap_blks_hit) as heap_hit, sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) as ratio FROM pg_statio_user_tables; For Bridge customers, cache hit data is in your clusters insights.

Parameter | Value ---------------------------------+---------------------------------------------- TimeZone | America/Chicago application_name | psql client_encoding | UTF8 lock_timeout | 10s

Postgres does a good job of keeping internal statistics of your which are used to plan how queries are executed. But Postgres doesn't always know how columns are related. In this deep dive we look at hacking the statistics for improved performance

crunchydata.comHacking the Postgres Statistics Tables for Faster Queries | Crunchy Data BlogAdding extended statistics can add information about how columns are related. Louise has some real life example queries and tips for working with extended table stats that can dramatically improve…
View full thread

4) After major maintenance operations Resetting pg_stat_statements after major operations (pg_repack, reindexes, altering table structures, etc) helps to measure how any changes affect performance.

View full thread

3) When benchmarking query performance If you are running benchmarks and tests to evaluate query improvements, resetting pg_stat_statements makes sure that your results reflect only the queries executed during the benchmark periods.

1) At the start of a new monitoring period If you analyze performance trends daily, weekly, or monthly, resetting pg_stat_statements at the beginning of such a period helps ensure that each period starts with a clean dataset.

2) After deploying significant query changes If you have made significant changes to an app, optimized queries, or modified indexes, resetting pg_stat_statements helps to measure the impact of those changes more accurately.

Excited to announce built-in maintenance for Iceberg via Postgres. Now within Crunchy Data Warehouse we will automatically vacuum and continuously optimize your Iceberg data by compacting and cleaning up files. Dig into the details of how this works www.crunchydata.com/blog/automat...

crunchydata.comAutomatic Iceberg Maintenance Within Postgres | Crunchy Data BlogIceberg can create orphan files during snapshot changes or transaction rollbacks. Crunchy Data Warehouse automatically cleans up the orphan files using a new autovacuum feature.

SQL output can be messy. Psql options for formatting output. A handy option is \pset border 2 This will add top and bottom borders with double lines (╔, ╚, ╤, ╧, etc.). A Header row bordered with a double-line separator . Each row is separated by single lines. Each column is clearly separated.

Crunchy Data reposted

Great to see this ability for "creating processing pipelines for append-only streams of data...We believe it is a foundational building block for building IoT applications on PostgreSQL that should be available to everyone, similar to pg_cron, pg_parquet, and pg_partman."

Crunchy Data@crunchydata.com

We’re excited to release pg_incremental today - a new extension for automated incremental updates. pg_incremental is like a supercharged pg_cron that runs data pipelines, data syncs, rollups, imports and exports. www.crunchydata.com/blog/pg_incr...

Crunchy Data reposted

There are many incremental processing solutions, but they seem to never quite do what I need. I decided to build an extension that just keeps running the same command in Postgres with different parameters to do fast, reliable incremental data processing. That's pg_incremental. 1/n

Crunchy Data@crunchydata.com

We’re excited to release pg_incremental today - a new extension for automated incremental updates. pg_incremental is like a supercharged pg_cron that runs data pipelines, data syncs, rollups, imports and exports. www.crunchydata.com/blog/pg_incr...

Exporting files: For folks archiving or exporting data, like maybe individual partitions, pg_incremental can batch data into files and send them to a long term archive.

Rollups and aggregate tables: Databases that have a lot of event or IoT stream data can create rollup, aggregate, or counts tables and incrementally add data to these tables. pg_incremental will even pick up old data that’s edited to changed.

Interval pipelines: Similar to the rollups and aggregates, lots of folks are creating interval range data summaries. Either by date, week, or month pg_incremental can build your summary tables.

Many big Postgres databases today use partitioning. Do you have a default partition? If not, you probably should. Default partitions are super important because they let you catch inconsistent data or bugs in your application code. www.crunchydata.com/blog/postgre...

crunchydata.comPostgres Partitioning with a Default Partition | Crunchy Data BlogKeith discusses the importance of having a default partition, how to monitor the default, and how to move rows to new child tables.
View full thread

Gold Layer: Business Intelligence and Reporting with PostgreSQL This layer focuses on performance and usability for downstream applications. By using PostgreSQL tables or materialized views (both with indexing) for querying insights, data is readily available and extremely performant.

>> This combination leverages the simplicity of flat files, the scalability and transactional capabilities of Iceberg, and the structured query efficiency of Postgres to create a robust, flexible medallion architecture.

View full thread

Silver layer: Cleaned and processed in Iceberg This layer has cleaned and standardized raw data work done on quality and consistency. A great file format for this is Iceberg. It serves as a central repository of organized data with a schema and metadata.

Raw data is easily organized into structured, partitioned, and optimized sets for query efficiency. Using schema evolution, versioning, and support for ACID transactions - data quality and consistency are not lost. The scalability of Iceberg also supports large datasets with efficient performance.

View full thread

Common layers of the medallion architecture: • Bronze layer: Raw unprocessed data • Silver layer: Cleaned and processed • Gold Layer: Business intelligence and reporting Crunchy Data Warehouse is a full data stack offering all three of these in one system.

Bronze layer: Raw unprocessed data in flat files This is the foundation of the modern data pipeline. Typical files our customers use here are logs, JSON files, CSV files, and raw extracts. Files are cheap to store and simple to write. Files are easily ingested by any simple ETL or noETL processes.

Having specific ‘tiers’ of data lets you organize data at different levels, each further refined than the other. Data flows through multiple stages, each serving a specific purpose in the pipeline.