Post

Jens von Bergmann
@jensvb.bsky.social

{safeconduct} solves the problem when programmatic data download has been disabled. It's a drop-in solution for `file.download` or `curl` to fetch remote data, although in some cases may require user interaction/captcha solves. This is a response to StatCan locking off some of their data.

mountainmath.github.ioFetch Files Through a Real Browser When a Plain Request Is RefusedSome servers refuse any request that does not come from a web browser, with a browser check or a block page instead of the file, Cloudflare being the common case. This package opens a real Chrome with...
Jens von Bergmann
@jensvb.bsky.social

Consider this example of trying to download data from StatCan. Traditional methods fail, but calling through `safeconduct::fetch()` works. (Although it requires waiting through a 2 minute Cloudflare curfue or solving a captcha to speed it up.)

Jens von Bergmann
@jensvb.bsky.social

{safeconduct} also provides a way to read webpages, for some StatCan data the only way to discover if data gets updated is scraping the web page. Which is blocked now using regular methods, safeconduct provides a drop-in method to still navigate this.

1 repost5 likes

Jens von Bergmann
@jensvb.bsky.social

r2parquet goes in a different direction, it facilitates the self-hosting of data. In that it is opinionated but not dogmatic. It strongly encourages publishing data in parquet, with rich metadata, hosted on R2. And it designs for selective reading of remote parquet data.

mountainmath.github.ioParquet and GeoParquet Files on Cloudflare R2 and S3Publish parquet and GeoParquet files on Cloudflare R2 or Amazon S3 and query them lazily. Creates R2 buckets with private or public access, including custom domains, uploads and removes (partitioned) ...
Jens von Bergmann
@jensvb.bsky.social

Why so opinionated, is parquet really better than sharing CSV? Yes! Let me count (some of) the ways. 1. parquet has a schema, no more reading character columns as integer 2. parquet compresses (don't zip a cloud hosted-parquet file for further compression though, see point 4)

Jens von Bergmann
@jensvb.bsky.social

3. parquet can carry rich metadata 4. cloud hosted parquet can act like a database server, you can remotely query parquet files for specific subsets of the data. (see attached example fetching just the 4k CoV dissemination blocks in sub-second time from remote parquet.)

Jens von Bergmann
@jensvb.bsky.social

But it's not just the ability to efficiently query subsets of the data, one can also perform joins on two remote parquet files, just like one would do if both datasets were in a local database. Again, a sub-second query, now involving two remote parquet files.