AutoMQ is a cloud-first alternative to Kafka by decoupling durability to S3 and EBS. 10x cost-effective. Autoscale in seconds. Single-digit ms latency.

Medium

How do we run Kafka 100% on the object storage?

Let’s see how AutoMQ makes this dream come true.

https://github.com/duolingo/metasearch

GitHub

GitHub - duolingo/metasearch: Search aggregator for Slack, Google Docs, GitHub, and more :mag:

Search aggregator for Slack, Google Docs, GitHub, and more :mag: - duolingo/metasearch

Snowflake infrastructure as code.
https://github.com/Titan-Systems/titan

GitHub

GitHub - Titan-Systems/titan: Titan Core - Snowflake infrastructure-as-code. Provision environments, automate deploys, CI/CD. Manage…

Titan Core - Snowflake infrastructure-as-code. Provision environments, automate deploys, CI/CD. Manage RBAC, users, roles, and data access. Declarative Python Resource API. Change Management tool f...

It looks like Databricks is up to something. First it was Iceberg and now Hudi. It is not clear if they try to converge on one open table format or just want to make sure Delta Lake becomes the one. In my opinion, Iceberg has more potential to become the one.

TechCrunch

Databricks acquires Tabular to build a common data lakehouse standard

Databricks has acquired Tabular, a data management startup, in its quest to build a common standard for data lakehouses.

https://github.com/opendatadiscovery/awesome-data-catalogs

GitHub

GitHub - opendatadiscovery/awesome-data-catalogs: 📙 Awesome Data Catalogs and Observability Platforms.

***📙*** Awesome Data Catalogs and Observability Platforms. - GitHub - opendatadiscovery/awesome-data-catalogs: ***📙*** Awesome Data Catalogs and Observability Platforms.

DataFrames at Scale Comparison: TPC-H

https://docs.coiled.io/blog/tpch.html

The Past, Present and Future of Stream Processing - Kai Waehner
https://www.kai-waehner.de/blog/2024/03/20/the-past-present-and-future-of-stream-processing/

Kai Waehner

The Past, Present and Future of Stream Processing

Stream Processing Journey with IBM, Apama, TIBCO StreamBase, Kafka Streams, Apache Flink, Streaming Databases, GenAI and Apache Iceberg.

https://delta-io.github.io/delta-rs/integrations/delta-lake-datafusion/

TaskWeaver: A code-first agent framework for seamlessly planning and executing data analytics tasks.

GitHub

GitHub - microsoft/TaskWeaver: A code-first agent framework for seamlessly planning and executing data analytics tasks.

A code-first agent framework for seamlessly planning and executing data analytics tasks. - GitHub - microsoft/TaskWeaver: A code-first agent framework for seamlessly planning and executing data an...

https://www.datanami.com/2024/01/23/data-engineering-in-2024-predictions-for-data-lakes-and-the-serving-layer/

Datanami

Data Engineering in 2024: Predictions For Data Lakes and The Serving Layer

The data landscape experienced significant changes in 2023, presenting new opportunities (and potential challenges) for data engineering teams. I believe