Post

There are many incremental processing solutions, but they seem to never quite do what I need. I decided to build an extension that just keeps running the same command in Postgres with different parameters to do fast, reliable incremental data processing. That's pg_incremental. 1/n

A pg_cron job checks the current state of the world (e.g. last sequence value), waits for ongoing writes to finish, and then runs the command for unprocessed values if needed. The command runs as a transaction with pg_incremental's own bookkeeping to ensure exactly once processing. 3/n

Can you say how this is done exactly? Is it based on public API accessible from SQL itself or only an API accessible to extensions only? Asking because I've always needed something like that exactly!