Where It Fits Into IBM’s Architecture And Where It Integrates On Day One
IBM recently spent $11BN on acquiring Confluent, and many of you (me included) would be asking the 6 W’s (who, what and why, etc).
So, to understand some of the background, it’s useful to know some of the history, as is the case with many growth platforms, i.e., Databricks was spawned out of Apache Spark.
Well, you may not have known this but Confluent’s story began inside LinkedIn, one of the world’s largest professional networking platforms. As LinkedIn grew rapidly, its engineers faced a challenge: millions of users were generating data every second through profile updates, messages, clicks, and searches. Traditional databases and batch processing systems simply could not keep up with the speed and volume of this information. To solve this problem, some key engineers, Jay Kreps, Jun Rao, and Neha Narkhede, created Apache Kafka, an open-source platform designed to move data continuously and reliably between applications in real time. Kafka was open-sourced in 2011 and quickly became popular across the technology industry.
Roll on to 2014, the creators founded Confluent to make Kafka easier for enterprises to adopt and operate at scale ie productionise it
To understand Kafka, it helps to think of it as a central nervous system for data. Every event that happens within a business creates information. A customer places an order, a payment is made, a shipment leaves a warehouse, or a sensor reports a temperature reading. Kafka captures these events and streams them instantly to the systems that need them. Instead of waiting hours or days for data to be processed in batches, businesses can react as events happen. Kafka allows organisations to publish, store, process, and distribute streams of events reliably across thousands of applications and users.
This ability makes Kafka ideal for streaming environments. Today, many modern organisations operate in real time. Banks need to detect fraud while a transaction is taking place. Retailers want to update stock levels instantly after a purchase. Logistics companies need to track vehicles and deliveries continuously. AI systems require up-to-date information to make accurate decisions. Kafka provides the scalable and resilient platform that allows all these systems to share information as it is generated. Rather than moving data in slow batches, businesses can work with “data in motion”, giving them faster insights and better customer experiences.
As Kafka adoption grew, many organisations discovered that running large Kafka environments required specialist skills. As a result, Confluent built a commercial platform around Kafka, adding enterprise security, governance, cloud services, connectors, stream processing, and management tools. This allowed companies to focus on using the data rather than spending time managing infrastructure. Over time, Confluent became the leading data streaming platform, helping thousands of organisations build real-time applications and analytics.
This strategic importance explains why IBM acquired Confluent for approximately $11 billion. IBM recognised that artificial intelligence is only as good as the data it receives. Most organisations have data spread across multiple systems, clouds, and applications. AI agents and machine learning models need access to trusted, current information, not data that is hours or days old. By combining Confluent’s real-time data streaming capabilities with IBM’s watsonx, hybrid cloud, automation, and integration technologies, IBM aims to create a platform where AI can continuously access live business data.
In simple terms, IBM did not just buy a streaming company; it bought the technology that helps ensure AI is connected to what is happening right now across the enterprise, similar to the RED HAT acquisition (albeit that was $34BN)
So today you will find that Confluent and Kafka sit at the heart of many modern digital businesses, acting as the bridge between systems, applications, analytics, and AI, enabling organisations to become truly real-time enterprises.
YOU may well be asking what makes Confluent different from Apache Kafka?
The simplest way to think about it is:
Apache Kafka is the engine. Confluent is the complete car built around that engine (could use the Formula Car analogy).
Apache Kafka
Apache Kafka is an open-source event streaming platform originally created at LinkedIn. It allows applications to publish, store, and consume streams of events in real time. Kafka provides the core capabilities needed for high-throughput, fault-tolerant data streaming.
With Apache Kafka alone, you get:
- Real-time event streaming
- Durable data storage
- Scalability across many servers
- High availability and resilience
- Stream processing capabilitiesapache+1
However, operating Kafka at enterprise scale can be complex and often requires specialist skills, and these can be in short supply or limited.
Confluent
Whereas Confluent was founded by Kafka’s creators to make Kafka easier to deploy, manage, secure, and govern in enterprise environments. It includes Kafka at its core, but adds many commercial capabilities on top of that as a PLATFORM.
Confluent provides:
- Fully managed cloud service (Confluent Cloud)
- Enterprise security and access controls
- Stream governance and data lineage
- Hundreds of pre-built connectors
- Simplified administration and monitoring
- Stream processing services such as Flink
- Multi-cloud deployment options
- Enterprise support and SLAswikipedia+2
Here’s a very brief Side-by-Side Comparison
| Apache Kafka | Confluent |
| Open source platform | Commercial enterprise platform |
| Requires self-management | Fully managed options available |
| Core streaming capabilities | Streaming + governance + security + management |
| Limited built-in tooling | Rich enterprise tooling |
| Community support | Enterprise support |
| Requires specialised skills | Simplifies deployment and operations |
| Focus on event streaming | Focus on complete data-in-motion platform |
Why Enterprises Choose Confluent
Many organisations will start with Apache Kafka, but later they may well adopt Confluent because they need:
- Faster deployment
- Better governance
- Reduced operational overhead
- Cloud-native services
- Enterprise-grade security
- Support for AI and analytics initiatives.
SO WHY DOES this matter to IBM
IBM’s acquisition of Confluent wasn’t really about owning Kafka. Kafka is open source and available to everyone. BUT, IBM wanted the enterprise platform that sits on top of Kafka, including governance, real-time integration, cloud services, and the ability to deliver live data to AI models and agents. IBM sees real-time data streaming as a key foundation for watsonx and enterprises deploying AI solutions.
In one sentence: Kafka helps move data in real time; Confluent turns Kafka into an enterprise-ready platform for data, analytics, governance, and AI.
Additionally, there are other benefits in using Confluent as a platform, such as connectors. So what are the Confluent connectors?
Confluent Connectors are pre-built integrations that allow data to flow between Kafka and other systems without developers having to write custom code. Giving the business better control and auditability/governance of the same. Often folks think of them as plug adapters for data.
As a simple example, if Kafka is the motorway carrying data, connectors are the on-ramps and off-ramps that allow data to enter and leave the motorway.
Typically there are TWO Types of Connectors
Source Connectors: These bring data into Kafka from external systems.
Examples of these are:
- Oracle Database
- SQL Server
- Db2
- Salesforce
- SAP
- MongoDB
- PostgreSQL
So in one example, every time a customer record changes in Salesforce, a source connector can automatically stream that change into Kafka.
Sink Connectors: These move data out of Kafka to other destinations.
Examples of these are names you will be familiar with:
- Snowflake
- Amazon S3
- Databricks
- Elasticsearch
- Google BigQuery
- IBM watsonx.data
In most of these examples, sales transactions arriving in Kafka can be continuously loaded into a data warehouse for analytics.
Why Are They Important?
Without connectors, developers would need to build and maintain custom integration code for every application.
With connectors, you can:
- Connect systems quickly
- Reduce development effort
- Standardise integrations
- Stream data in real time
- Improve reliability and scalability
This is one of the reasons Confluent is more than just Kafka. Kafka provides the streaming engine, while Confluent provides hundreds of enterprise-ready connectors and tools that make enterprise integration much easier and production-ready.
A few Simple IBM examples: Imagine a retailer using:
- SAP for orders
- Salesforce for customers
- IBM watsonx.data for analytics
- AI agents in watsonx
Confluent connectors can:
- Capture customer updates from Salesforce.
- Capture order events from SAP.
- Stream both into Kafka in real time.
- Deliver the data to watsonx.data.
- Feed AI models with up-to-date information.
Instead of building dozens of point-to-point integrations, the organisation connects everything through Confluent’s streaming platform.
The Business Value
For customers, connectors mean:
- Faster projects
- Lower integration costs
- Less custom coding
- Real-time insights
- Better AI outcomes
As a result, IBM saw strategic value in Confluent. The connectors help break down data silos and allow data to move continuously between applications, data platforms, and AI systems.
We will offer another BLOG on the integration available on DAY ONE into IBM platforms and technology, but for now this was just to set the ‘scene’. IF you would like to understand MORE about the Confluent Platform, Kafka Use cases and if you are using Apache Kafka and need to productionise this for the business simply get in touch.
