Post

2 years ago I inherited a team that was having each vendor (over 100!) run their own SFTP and we'd upload data dumps nightly. We shifted to using S3 with AWS transfer family gateway and migrated everyone over. The whole setup is insanely simple and scalability is awesome!

We're loving the preview image, btw! 😍🗂

What are folks' preferred ways to convert parquet to jsonl? I tried writing a python script and got hit by "Unsupported encoding: RLE_DICTIONARY" errors.

What's the point au data contracts with open table format which are strongly typed and support column constraints ?

You are right. The UX isn't there yet, but it's slowly getting easier to interact with. There are multiple CLIs that you can use with Datafusion: github.com/datafusion-c... github.com/JanKaul/fros...

Thank you so much for the shoutout and thanks to @mmullins.coginiti.co for the support and heads up! 🙏