Post

@crunchydata.com

We’re excited to release pg_incremental today - a new extension for automated incremental updates. pg_incremental is like a supercharged pg_cron that runs data pipelines, data syncs, rollups, imports and exports. www.crunchydata.com/blog/pg_incr...

crunchydata.compg_incremental: Incremental Data Processing in Postgres | Crunchy Data BlogWe are excited to release a new open source extension called pg_incremental. pg_incremental works with pg_cron to do incremental batch processing for data aggregations, data transformations, or import...

@crunchydata.com

We see a lot of uses for this with Postgres clusters:

@crunchydata.com

Rollups and aggregate tables: Databases that have a lot of event or IoT stream data can create rollup, aggregate, or counts tables and incrementally add data to these tables. pg_incremental will even pick up old data that’s edited to changed.

17/12/24, 4:40 pm0 reposts1 like

@crunchydata.com

Interval pipelines: Similar to the rollups and aggregates, lots of folks are creating interval range data summaries. Either by date, week, or month pg_incremental can build your summary tables.

@crunchydata.com

Exporting files: For folks archiving or exporting data, like maybe individual partitions, pg_incremental can batch data into files and send them to a long term archive.

@crunchydata.com

Importing files: For cases when data is loaded through remote repositories in s3, functions can look for new files and pg_incremental can load these.