Jim Apple

Worth reading: How the Apache Arrow Format Accelerates Query Result Transfer by Ian Cook One of the hidden problems of intersystem communication is transcription of data formats. Ian does a wonderful job of explaining the "why" of Apache Arrow as well as the "how" of its advantages. Read on...

8

Looking to get started with in-database machine learning with MySQL HeatWave? This guide will help get you on the right path. https://lnkd.in/gPjuJU9g

1

Rewrite bigdata in Rust!

8

1 Comment

See how simple it is to create chatbots with Microsoft Azure On Your Data and vector embeddings from an existing Elasticsearch index: https://lnkd.in/gBHZbkjq #ElasticSearchLabs #RAG #VectorDatabase

2

Data streaming based on #Kafka is sometimes viewed as a niche service best suited for the journeyman work of getting data from a source database or application to different target, where it's coalesced with data from other sources and then transformed and refined. Platforms like #Snowflake, #Databricks, Oracle, #Redshift, #BigQuery, and #MicrosoftFabric model and analyze this data for the business. It's true that Event Driven Architectures (EDA) and logging/observability pipelines are popular use cases for Kafka. But building a Data Streaming Platform with current, governed, shareable, enterprise-wide data available for both operations and analytics seems a bridge too far. In the future, will customers see #datastreaming and platforms based on Kafka as indispensable? In fact, they already do, because the enterprise requires shared, real-time data that blurs the lines between the operational and analytic. Over 100,000 organizations have already adopted Kafka. Supply creates its own demand. Yes there are technical hurdles and challenges that remain, but remember, before smartphones came along, you assumed you needed separate phone, GPS, email/messaging, camera, etc. Then you realized all these use cases could all be integrated onto one platform that enabled unique and powerful workflows and applications via shared data and infrastructure, and you never looked back. The unnatural data fragmentation across disparate applications, databases, and lakehouses results partly from vendor incentives to get data-at-rest onto their own platform, and partly from the limitations of those platforms to share data. Data-in-motion streaming platforms like Confluent have the exact opposite incentive, get the data shareable in real-time with as many other platforms as possible. This is unique. And indispensable.

38

Check out both the talks from #posetteconf about pgvector + Python! One from me about pgvector with everything but Django, then one from Paolo Melchiorre on Django: https://lnkd.in/g6ixPn_8 #python #postgresql #pgvector

11

The final deep dive on change query support in the table formats is ready. This one covers Apache Paimon, focusing on understanding the change query support through the lens of its internals. https://lnkd.in/dFfM_8nA

29

There’s a focus on Observability 2.0, with a greater emphasis on rich events over different data types. I like this, but I think it’s short sighted. Events are the most expensive method of storing your telemetry. They require indexing and high performance storage (except Coralogix hint hint). I wonder if this focus on single data types is going to limit a team’s ability to store and process their data as they need, and the commensurate benefits lost.

22

3 Comments

Data preparation is a crucial step for fine-tuning large language models with your own private data. I've published a new blog post providing a comprehensive guide on preparing data and resources for fine-tuning the Anthropic Claude 3 Haiku model using Amazon Bedrock. 👉 https://lnkd.in/g7mdrvHi The guide covers creating an IAM role, setting up an S3 bucket, processing a CNN news dataset into the required format, and uploading the datasets to S3 for the fine-tuning process. 🏌‍♂️

15

Materialized views in #PostgreSQL store pre-computed results of frequent, complex queries to boost performance. They are periodically refreshed for up-to-date data but require careful management of refresh frequency, disk space, and resources. #PostgreSQLDBA @MinervaDB https://zurl.co/7ixU

11

Having trouble scaling PostgreSQL with AWS RDS? Considering migrating to AWS Aurora to solve this problem? I wrote an article about the Pro's and Con's of choosing AWS Aurora. Its the first in a series where I will also be looking at newer Postgres based options such as Timescale and Neon as well as talking about things you should be considering when benchmarking or designing your application for scale. If you are having problems like this and need help, StepChange.work would be happy to hop on a call and discuss your problems and how we can be of assistance. We have decades of experience building products at scale at top companies such as Meta and Coinbase. https://lnkd.in/gMgAbA4D

19

2 Comments

Database Center preview for Spanner, Cloud SQL and AlloyDB now open to all customers! Get a unified view, performance and security recommendations, and AI-powered assistance for your entire database fleet. Learn more!

4

This is an interesting article from Amazon: https://lnkd.in/gB5a83w3 Although Ray is commonly associated with machine learning workloads, it looks like Amazon is using Ray to replace Spark for certain non-machine learning use-cases. In this case, they use giant Ray Clusters with over 24,000 vCPUs to do "copy-on-write" merges or "compaction" on tables at an exabyte scale. An exabyte is 1 million terabytes of data!

1

Confluent's Field CTO, Kai Waehner, shares insights on the future of streaming analytics and data management, highlighting the importance of reliable data flow, collaboration, and security. Discover how Confluent's cutting-edge platform revolutionizes streaming analytics. Read the full interview: https://lnkd.in/gSMVCmvy #StreamingAnalytics #DataManagement #FutureOfData #BusinessIntelligence

80

Attending #DataAISummit next week? Swing by the O'Reilly booth (#127) at the Data+AI Summit on June 12 from 3:15PM - 3:45PM PT for a book signing with me! We're giving away my earlier book "Modern Data Engineering with Apache Spark" but we can talk Delta, Spark, or anything really or all things The Delta Lake Definitive Guide. #opensource #oss #deltalake #freebooks

86

6 Comments

I presented about Apache XTable's architecture last week at LinkedIn's Big Data Meetup. For context, XTable is an omni-directional translation layer on top of open table formats such as Apache Hudi, #ApacheIceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure Read the paper (link in comments) for details. #dataengineering #softwareengineering

199

5 Comments

Customers are using the Apache Iceberg format because they can read and write data concurrently using different frameworks. Iceberg format also allows for time travel and rollback so you can run SQL queries on a point-in-time snapshot of your data, or rollback data to a previously known good version. Now, you can stream real-time data into Apache Iceberg tables in Amazon S3 using Amazon Data Firehose. https://lnkd.in/g8g3FUah #Data #StreamingData #DataStreaming #Firehose #S3 #AWS #ApacheIceberg #Iceberg

50

Recently Haytham Abuelfutuh presented about Batch Workloads and how Union and Flyte are used for Large scale batch and near realtime workloads. Check out this post to 1. Learn about the challenges in using Kubernetes for running Batch workloads 2. The problems in the programming model with Batch workloads. Often times its not a single job - but it may start with one 3. The simplification that can be introduced by using Union. Union runs them across multiple clouds, multiple kubernetes clusters, securely accessing large amounts of data, while maintaining efficiency through - caching, smart techniques in which container state is saved, how pods are used, how back pressure is maintained etc. Check out the blog.

9