Building a real-time data pipeline with Kafka, Spark and Hive

SECTION 1

Why real-time analytics demands a robust data pipeline

Real-time analytics has moved from competitive advantage to operational standard. Fraud detection, customer-journey personalisation, logistics tracking, feeding AI models with fresh features: in all these cases the value of a data point decays with age, and organisations able to decide within seconds capture opportunities that others only discover the next day in a batch report.

That still requires a chain able to ingest, process and store data as it arrives. The Apache Kafka, Apache Spark and Apache Hive trio remains a reference answer to that need, but each of its building blocks has been deeply transformed in recent years: Kafka eliminated ZooKeeper in favour of KRaft mode, Spark unified batch and streaming behind a single API, and Hive reinvented itself as a lakehouse building block backed by open table formats.

Understanding the exact role of each component, the architectures that assemble them and the guarantees a production pipeline must offer is therefore a prerequisite before any tooling decision. That is the purpose of this guide, updated to the state of the art of data streaming platforms in 2026.

SECTION 2

Kafka 4: event-stream ingestion without ZooKeeper

Apache Kafka handles the first stage of the pipeline: real-time stream ingestion. It is a distributed, scalable, low-latency event-streaming platform built on a publish-subscribe model. The durability of its replicated log and its fault tolerance make it effective where traditional message brokers such as RabbitMQ reach their limits under sustained high volumes.

### Topics, partitions and producer-consumer decoupling

An event represents a change in the system, an order created, a record updated, a user click, and events are organised into topics, themselves split into partitions to parallelise reads and writes. Producers publish without knowing anything about consumers, who subscribe and read at their own pace: this decoupling is the key to the scalability and resilience of ingestion, and each consumer can also replay the stream from any offset to rebuild state or fix a processing error.

### What the Kafka 4.x generation changes

Since Kafka 4.0, the cluster runs exclusively in KRaft mode: ZooKeeper has been removed, which simplifies operations, shrinks the failure surface and speeds up scaling of the broker itself. The 4.x generation, Kafka 4.3 is the stable release as of mid-2026,also brings the new consumer-group rebalance protocol, which sharply reduces disruption during topology changes, and share groups, a queue-style semantics for use cases where consumption throughput matters more than per-partition ordering.

The ecosystem around the broker completes ingestion: Kafka Connect industrialises capture from databases and SaaS applications through ready-made connectors, change data capture (CDC) turns every transactional write into an exploitable event, and the schema registry enforces a validated format from the moment of publication. These building blocks avoid reinventing integration source by source and secure the data contract right at the pipeline's entrance.

@cite:data-lake-et-ses-benefices

SECTION 3

Spark 4: distributed processing that unifies batch and streaming

Apache Spark consumes the streams coming from Kafka and transforms them at scale, distributing computation across a cluster and working in memory rather than on disk. Its unified DataFrame API lets you write the same business logic for a batch job and for continuous processing, avoiding the maintenance of two parallel codebases, one of the historical criticisms of early big data platforms.

In a typical pipeline, Spark plays three complementary roles: it cleans and enriches events on the fly, joining them with customer or product reference data, it computes aggregates consumed by dashboards and decision engines, and it materialises up-to-date features for machine learning models, whose relevance depends directly on the freshness of the data feeding them.

### Structured Streaming and the Kafka integration

Structured Streaming treats streams as unbounded tables on which incremental queries are applied: filtering, windowed aggregations, joins between a stream and a reference dataset. The native Kafka connector handles offset tracking, watermarking bounds late-arriving data, and stateful operators enable complex processing such as sequence detection or session-based aggregations, with exactly-once guarantees towards transactional sinks.

### What the Spark 4.x series brings

The 4.x series, of which Spark 4.2 released in July 2026 is the latest major version, has taken the platform up a level: ANSI SQL mode enabled by default for standards-compliant results and explicit errors instead of silently corrupted values, a VARIANT type to handle semi-structured JSON efficiently, and Spark Connect, a lightweight client-server architecture that decouples applications from the cluster and eases integration from any language or IDE.

SECTION 4

Hive 4 and the lakehouse: from HDFS directories to Iceberg tables

Apache Hive remains the historical SQL warehouse of the Hadoop ecosystem: it exposes data-lake content through SQL-like syntax and optimisations such as partitioning and bucketing, which pre-divide data into manageable chunks to speed up queries and sampling. Hive 4.2, the current release, modernises that foundation with Java 21 support, built-in automatic compaction and a considerably deepened Iceberg integration.

### Open table formats and the Iceberg v3 specification

Open table formats such as Apache Iceberg and Delta Lake have become the standard for transactional data management in the lakehouse: ACID transactions on object storage, schema evolution without rewriting files, time travel to query a past state of the table. The Iceberg v3 specification, generally available across major vendors in 2026, notably adds deletion vectors for more efficient deletes and the variant type for semi-structured data.

In this architecture, Hive's role shifts from storage engine to catalog: the Hive Metastore and its REST interface act as the shared table registry between Spark, Trino, Flink and the other engines, guaranteeing that a single Iceberg table is seen consistently across the whole platform, whatever tool reads or writes it.

The transition rarely happens in one go: most organisations run classic Hive tables and Iceberg tables side by side during the migration, converting the most heavily queried datasets first. Modern engines read both formats, allowing a progressive switch, table by table, without interrupting existing analytics uses or rewriting the entire accumulated history.

@cite:principes-d-architecture-de-donnees

SECTION 5

Lambda and Kappa architectures: which pipeline model in 2026

The Lambda architecture combines three layers: a batch layer that processes and stores the full history, a speed layer that computes real-time views over recent data, and a serving layer that merges both to expose queryable results. It offers robustness and full reprocessing capability, at the cost of a dual computation logic to develop, test and keep consistent.

The Kappa architecture drops the batch layer: all processing goes through the stream, and reprocessing is done by replaying the Kafka log from the beginning. It reduces operational complexity and eliminates divergence between the two computation chains, provided the streaming platform retains enough history and the jobs are designed to be replayable.

The choice is easiest to read through concrete cases: an e-commerce clickstream, where only the up-to-date view of customer behaviour matters, lends itself naturally to Kappa; a bank subject to massive regulatory recalculations over several years of history will keep a dedicated batch chain, even if it sits on the same Iceberg storage foundation as the real-time path.

### Streaming-batch convergence reshuffles the deck

By 2026 the boundary has largely blurred: Spark runs the same query in batch or continuously, and Apache Flink, whose 2.x series pushes stream-batch unification, a disaggregated state backend and even AI inference expressed in SQL, has established itself as the reference alternative for ultra-low-latency processing. Combined with transactional Iceberg storage, this convergence makes Kappa the default choice for most new projects, with Lambda remaining relevant when heavy, regulated historical workloads coexist with real time.

SECTION 6

The characteristics of a robust, governed data pipeline

The first expected guarantee is end-to-end exactly-once semantics: idempotent producers and transactions on the Kafka side, checkpointing and transactional sinks on the Spark side, atomic commits on the Iceberg side. Without it, every incident or retransmission risks duplicating or losing records, and business trust in the numbers produced erodes for good.

### Observability, data contracts and self-service

A production pipeline is monitored like a critical application: data freshness, consumer lag, per-partition throughput, schema error rates. Data contracts, backed by a schema registry, formalise the commitment between producing and consuming teams and block incompatible changes before they break downstream systems. Self-service access, catalog, lineage, query environments, finally prevents every new analytics need from becoming a full-blown integration project.

Security is built in from the design stage: encryption of streams in transit, access control per topic and per table, masking of personal data as early as possible in the chain to comply with GDPR. Enforcing these rules at platform level, rather than in each individual job, avoids discrepancies between teams and considerably simplifies audits.

Cost control completes the picture: sizing Kafka retention, compacting and clustering Iceberg tables, autoscaling processing clusters. A poorly calibrated real-time pipeline can cost several times its batch equivalent; a well-designed one adjusts its consumption to the value actually delivered to the business.

@cite:data-mesh-principes-et-benefices

SECTION 7

Industrialising your real-time pipeline with Adservio

At Adservio, we design data pipelines as systems to operate over time, not merely to assemble. Architecture choices, Kappa or Lambda, exactly-once guarantees, Kafka foundation sizing, Iceberg table strategy and governance model are weighed against your real business uses and your teams' maturity, never against passing trends.

We support your teams from design to industrialisation: standing up the streaming foundation, migrating existing workloads to a unified chain, tooling for observability and quality, then progressive skills transfer, so that the platform stays under internal control and durably turns raw streams into actionable decisions.

FAQ

Frequently asked questions