Release Notes - Kafka - Version 3.2.0

Below is a summary of the JIRA issues addressed in the 3.2.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

New Feature

[KAFKA-6718] - Rack Aware Stand-by Task Assignment for Kafka Streams
[KAFKA-13479] - Interactive Query v2

Improvement

[KAFKA-7077] - KIP-318: Make Kafka Connect Source idempotent
[KAFKA-7589] - Allow configuring num.network.threads per listener
[KAFKA-9648] - Add configuration to adjust listen backlog size for Acceptor
[KAFKA-9847] - Add config to set default store type
[KAFKA-12959] - Prioritize assigning standby tasks to threads without any active tasks
[KAFKA-12980] - Allow consumers to return from poll when position advances due to aborted transactions
[KAFKA-13200] - Fix version of MirrorMaker2 connectors
[KAFKA-13323] - The words are ambiguous
[KAFKA-13348] - Allow Source Tasks to Handle Producer Exceptions
[KAFKA-13426] - Add recordMetadata to StateStoreContext
[KAFKA-13441] - improve upgrade doc
[KAFKA-13445] - Add ECDSA test for JWT validation
[KAFKA-13449] - Comment optimization for parameter log.cleaner.delete.retention.ms
[KAFKA-13451] - Add reason to JoinGroupRequest and LeaveGroupRequest
[KAFKA-13455] - The Apache Kafka quickstart guide does not contain any steps for running Kafka Connect
[KAFKA-13497] - Debug Log RegexRouter transform
[KAFKA-13503] - Validate broker configs for KRaft
[KAFKA-13511] - Add support for different unix precisions in TimestampConverter SMT
[KAFKA-13549] - Add "delete interval" config
[KAFKA-13583] - Fix FetchRequestBetweenDifferentIbpTest flaky tests
[KAFKA-13585] - Fix `kafka.server.ReplicaManagerTest.testReplicaAlterLogDirsWithAndWithoutIds` flaky test
[KAFKA-13587] - Implement unclean leader election in KIP-704
[KAFKA-13590] - Rename InternalTopologyBuilder's #topicGroups method to be more descriptive
[KAFKA-13595] - Allow producing records with null values in Kafka Console Producer
[KAFKA-13603] - empty active segment can trigger recovery after clean shutdown and restart
[KAFKA-13618] - BatchAccumulator `Exptected` rename to `Expected`
[KAFKA-13629] - Client producer use faster ByteUtils sizeOfXxx algorithm
[KAFKA-13637] - User default.api.timeout.ms config as default timeout for KafkaConsumer.endOffsets
[KAFKA-13646] - Implement KIP-801: KRaft authorizer
[KAFKA-13671] - Power (ppc64le) support for kafka
[KAFKA-13673] - disable idempotent producer when acks/retries/max.in.flight config conflicts
[KAFKA-13676] - When processing in ALOS, when one task encounters a task-specific exception we could still commit progress made by other tasks
[KAFKA-13694] - Some InvalidRecordException messages are thrown away
[KAFKA-13752] - Using `equals` instead of `==` when Uuid compare in Java
[KAFKA-13767] - Fetch from consumers should return immediately when preferred read replica is defined by the leader
[KAFKA-13776] - Update rocksdb to support arm64 M1 Mac

Bug

[KAFKA-6502] - Kafka streams deserialization handler not committing offsets on error records
[KAFKA-7540] - Flaky Test ConsumerBounceTest#testClose
[KAFKA-7572] - Producer should not send requests with negative partition id
[KAFKA-8659] - SetSchemaMetadata SMT fails on records with null value and schema
[KAFKA-9279] - Silent data loss in Kafka producer
[KAFKA-12226] - High-throughput source tasks fail to commit offsets
[KAFKA-12256] - auto commit causes delays due to retriable UNKNOWN_TOPIC_OR_PARTITION
[KAFKA-12841] - NPE from the provided metadata in client callback in case of ApiException
[KAFKA-12879] - Compatibility break in Admin.listOffsets()
[KAFKA-13128] - Flaky Test StoreQueryIntegrationTest.shouldQueryStoresAfterAddingAndRemovingStreamThread
[KAFKA-13255] - Mirrormaker config property config.properties.exclude is not working as expected
[KAFKA-13306] - Null connector config value passes validation, but fails creation
[KAFKA-13310] - KafkaConsumer cannot jump out of the poll method, and the consumer is blocked in the ConsumerCoordinator method maybeAutoCommitOffsetsSync(Timer timer). Cpu and traffic of Broker‘s side increase sharply
[KAFKA-13374] - [Docs] - All reads from the leader of the partition even after KIP-392?
[KAFKA-13388] - Kafka Producer nodes stuck in CHECKING_API_VERSIONS
[KAFKA-13397] - MirrorMaker2 shouldn't mirror Connect's internal topics
[KAFKA-13412] - Retry of initTransactions after timeout may cause invalid transition
[KAFKA-13418] - Brokers disconnect intermittently with TLS1.3
[KAFKA-13423] - Error falsely reported on Kafka Streams app when GlobalKTables are used
[KAFKA-13435] - Static membership protocol should let the leader skip assignment (KIP-814)
[KAFKA-13448] - Kafka Producer Client Callback behaviour does not align with Javadoc
[KAFKA-13457] - SocketChannel in Acceptor#accept is not closed upon IOException
[KAFKA-13466] - delete unused config batch.size in kafka-console-producer.sh
[KAFKA-13469] - End-of-life offset commit for source task can take place before all records are flushed
[KAFKA-13476] - Streams crashes when non Base64 Offset Metadata is found
[KAFKA-13507] - GlobalProcessor ignores user specified names
[KAFKA-13510] - KIP-769: Connect APIs to list all connector plugins and retrieve their configuration definitions
[KAFKA-13527] - Add top-level error code field to DescribeLogDirsResponse
[KAFKA-13539] - Improve propagation and processing of SSL handshake failures
[KAFKA-13552] - Unable to dynamically change broker log levels on KRaft
[KAFKA-13558] - NioEchoServer fails to close resources
[KAFKA-13563] - FindCoordinatorFuture never get cleared in non-group mode( consumer#assign)
[KAFKA-13579] - Upgrade vulnerable dependencies
[KAFKA-13584] - Fix `kafka.network.SocketServerTest.testUnmuteChannelWithBufferedReceives` flaky test
[KAFKA-13591] - Fix flaky test ControllerIntegrationTest.testTopicIdCreatedOnUpgrade
[KAFKA-13598] - idempotence producer is not enabled by default if not set explicitly
[KAFKA-13600] - Rebalances while streams is in degraded state can cause stores to be reassigned and restore from scratch
[KAFKA-13614] - Leader replication quota is applied to consumer fetches
[KAFKA-13636] - Committed offsets could be deleted during a rebalance if a group did not commit for a while
[KAFKA-13658] - Upgrade vulnerable dependencies jan 2022
[KAFKA-13660] - Replace log4j with reload4j
[KAFKA-13661] - KRaft uses the wrong permission for adding topic partitions
[KAFKA-13669] - Log messages for source tasks with no offsets to commit are noisy and confusing
[KAFKA-13672] - Race condition in DynamicBrokerConfig
[KAFKA-13674] - Failure on ZOS due to IOException when attempting to fsync the parent directory
[KAFKA-13699] - ProcessorContext does not expose Stream Time
[KAFKA-13706] - org.apache.kafka.test.MockSelector doesn't remove closed connections from its 'ready' field
[KAFKA-13714] - Flaky test IQv2StoreIntegrationTest
[KAFKA-13717] - KafkaConsumer.close throws authorization exception even when commit offsets is empty
[KAFKA-13719] - connector restart cause duplicate tasks
[KAFKA-13721] - Left-join still emit spurious results in stream-stream joins in some cases
[KAFKA-13728] - PushHttpMetricsReporter no longer pushes metrics when network failure is recovered.
[KAFKA-13735] - Flaky kafka.network.SocketServerTest.remoteCloseWithoutBufferedReceives
[KAFKA-13739] - Sliding window without grace not working
[KAFKA-13741] - Cluster IDs should not have leading dash
[KAFKA-13748] - Do not include file stream connectors in Connect's CLASSPATH and plugin.path by default
[KAFKA-13750] - Client Compatability KafkaTest uses invalid idempotency configs
[KAFKA-13759] - Disable producer idempotence by default in producers instantiated by Connect
[KAFKA-13770] - Regression when Connect uses 0.10.x brokers due to recently added retry logic in KafkaBasedLog
[KAFKA-13772] - Partitions are not correctly re-partitioned when the fetcher thread pool is resized
[KAFKA-13775] - CVE-2020-36518 - Upgrade jackson-databind to 2.12.6.1
[KAFKA-13782] - Producer may fail to add the correct partition to transaction
[KAFKA-13783] - Remove reason prefixing in JoinGroupRequest and LeaveGroupRequest
[KAFKA-13794] - Producer batch lost silently in TransactionManager
[KAFKA-13828] - Ensure reasons sent by the consumer are small

Task

[KAFKA-13542] - Utilize the new Consumer#enforceRebalance(reason) API in Streams
[KAFKA-13599] - Upgrade RocksDB to 6.27.3
[KAFKA-13682] - Implement auto preferred leader election in KRaft Controller

Wish

[KAFKA-13351] - Add possibility to write kafka headers in Kafka Console Producer

Test

[KAFKA-10405] - Flaky Test org.apache.kafka.streams.integration.PurgeRepartitionTopicIntegrationTest.shouldRestoreState
[KAFKA-13159] - Enable system tests for transactions in KRaft mode
[KAFKA-13589] - fix flaky `PlaintextAdminIntegrationTest.testReplicaCanFetchFromLogStartOffsetAfterDeleteRecords` test
[KAFKA-13666] - Tests should not ignore exceptions for supported OS
[KAFKA-13710] - Fix failed testProduceWithInvalidTimestamp/testSendWithInvalidCreateTime tests

Sub-task

[KAFKA-8941] - Add RocksDB Metrics that Could not be Added due to RocksDB Version
[KAFKA-13221] - Add metric for `PartitionsWithLateTransactionsCount`
[KAFKA-13316] - Convert CreateTopicsRequestWithPolicyTest to use ClusterTest
[KAFKA-13438] - Replace EasyMock and PowerMock with Mockito for WorkerTest
[KAFKA-13492] - IQ Parity: queries for key/value store range and scan
[KAFKA-13494] - Session and Window Queries for IQv2
[KAFKA-13495] - Add reason to JoinGroupRequest
[KAFKA-13496] - Add reason to LeaveGroupRequest
[KAFKA-13509] - Support max timestamp in GetOffsetShell
[KAFKA-13577] - Replace EasyMock and PowerMock with Mockito in core module