KAFKA-18052: Decouple the dependency of feature stable version to the metadata version #17886

dongnuo123 · 2024-11-20T20:46:32Z

Currently the validation of feature upgrade relies on the supported version range generated during registration. For a given feature, its max supported feature version in production is set to be the default version value (the latest feature version with bootstrap metadata value smaller or equal to the latest production metadata value).

This patch introduces a LATEST_PRODUCTION value independent from the metadata version to each feature so that the highest supported feature version can be customized by the feature owner.

The change only applies to dynamic feature upgrade. During formatting, we still use the default value associated the metadata version.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

server-common/src/main/java/org/apache/kafka/server/common/TestFeatureVersion.java

junrao

@dongnuo123 : Thanks for the PR. Left a comment.

server-common/src/main/java/org/apache/kafka/server/common/Features.java

junrao

@dongnuo123 : Thanks for the updated PR. A few minor comments.

server-common/src/main/java/org/apache/kafka/server/common/Features.java

server-common/src/test/java/org/apache/kafka/server/common/FeaturesTest.java

server-common/src/main/java/org/apache/kafka/server/common/TestFeatureVersion.java

dongnuo123 · 2024-11-25T18:15:56Z

metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java

@@ -313,7 +313,7 @@ Map<String, Short> calculateEffectiveFeatureLevels() {
                    Optional.ofNullable(newFeatureLevels.get(KRaftVersion.FEATURE_NAME))));
            } else if (!newFeatureLevels.containsKey(supportedFeature.featureName())) {
                newFeatureLevels.put(supportedFeature.featureName(),
-                    supportedFeature.defaultValue(releaseVersion));
+                    supportedFeature.defaultLevel(releaseVersion));
            }
        });
        // Verify that the specified features support the given levels. This requires the full


FAILED ❌ FormatterTest > "testFeatureFlag(short).version=0" FAILED ❌ FormatterTest > "testFeatureFlag(short).version=1" FAILED ❌ FormatterTest > testInvalidFeatureFlag()

Got 3 failed tests. Will fix it today

junrao

@dongnuo123 : Thanks for the updated PR. A few more comments.

junrao · 2024-11-25T19:03:51Z

server-common/src/main/java/org/apache/kafka/server/common/Features.java

+     *     - The feature X has default version = XV_10 (dependency = {}), latest production = XV_5 (dependency = {})
+     *       (Violating rule 2. The latest production value XV_5 is smaller than the default value)
+     *     - The feature X has default version = XV_10 (dependency = {Y: YV_3}), latest production = XV_11 (dependency = {Y: YV_4})
+     *       The feature Y has default version = YV_3 (dependency = {}), latest production = YV_3 (dependency = {})


The default version part seems unrelated to this example.

junrao · 2024-11-25T19:05:18Z

server-common/src/main/java/org/apache/kafka/server/common/Features.java

+     *       The feature Y has default version = YV_3 (dependency = {}), latest production = YV_3 (dependency = {})
+     *       (Violating rule 3. For latest production XV_11, Y's latest production YV_3 is smaller than the dependency value YV_4)
+     *     - The feature X has default version = XV_10 (dependency = {Y: YV_4}), latest production = XV_11 (dependency = {Y: YV_4})
+     *       The feature Y has default version = YV_3 (dependency = {}), latest production = YV_4 (dependency = {})


The latest production part seems unrelated to this example.

junrao · 2024-11-25T19:10:39Z

server-common/src/main/java/org/apache/kafka/server/common/Features.java

+     *     - The feature X has default version = XV_10 (dependency = {Y: YV_3}), latest production = XV_11 (dependency = {Y: YV_4})
+     *       The feature Y has default version = YV_3 (dependency = {}), latest production = YV_4 (dependency = {})
+     *     - The feature X has default version = latest production = XV_10 (dependency = {MetadataVersion: IBP_4_0_IV0})
+     *       (Some features can depend on MetadataVersion, which is not checked in this method)


Hmm, how do we enforce the dependency on MV? For example, if feature level X_1 depends on MV_10 and is production ready, but MV_10 is not production ready, how do we detect that in tests?

junrao · 2024-11-25T19:20:07Z

server-common/src/main/java/org/apache/kafka/server/common/Features.java

-    KRAFT_VERSION("kraft.version", KRaftVersion.values()),
-    TRANSACTION_VERSION("transaction.version", TransactionVersion.values()),
-    GROUP_VERSION("group.version", GroupVersion.values());
+    TEST_VERSION("test.feature.version", TestFeatureVersion.values(), TestFeatureVersion.LATEST_PRODUCTION),


This is an existing issue. It seems that Features should be Feature since it represents a single feature?

This file contains many different features so I'm not sure I follow.

Well, Features.FEATURES contains many features, but the enum Features represents a single feature. For example, Features.name() provides the name of a single feature.

junrao · 2024-11-25T19:26:58Z

server-common/src/main/java/org/apache/kafka/server/common/Features.java

+    // value calculated with {@link #defaultValue(MetadataVersion)}.
+    public final FeatureVersion latestProduction;
+
+    private static final Logger log = LoggerFactory.getLogger(Features.class);


This seems unused?

junrao · 2024-11-25T19:59:10Z

server-common/src/main/java/org/apache/kafka/server/common/UnitTestFeatureVersion.java

+ */
+public enum UnitTestFeatureVersion implements FeatureVersion {
+    UT_FV0_0(0, MetadataVersion.MINIMUM_KRAFT_VERSION, Collections.emptyMap()),
+    UT_FV0_1(1, MetadataVersion.IBP_3_7_IV0, Collections.emptyMap()),


Could we add a comment on the purpose of each feature?

junrao · 2024-11-25T20:00:09Z

server-common/src/main/java/org/apache/kafka/server/common/UnitTestFeatureVersion.java

+/**
+ * The enum is only used for testing invalid features in FeaturesTest.java.
+ */
+public enum UnitTestFeatureVersion implements FeatureVersion {


Should we consolidate UnitTestFeatureVersion with TestFeatureVersion?

junrao · 2024-11-25T20:00:55Z

server-common/src/main/java/org/apache/kafka/server/common/UnitTestFeatureVersion.java

+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.server.common;


Should this be in the test directory?

junrao · 2024-11-25T20:01:50Z

server-common/src/main/java/org/apache/kafka/server/common/Features.java

+    UNIT_TEST_VERSION_2(UnitTestFeatureVersion.FEATURE_NAME + ".2", new FeatureVersion[]{UT_FV2_0, UT_FV2_1}, UT_FV2_0),
+    UNIT_TEST_VERSION_3(UnitTestFeatureVersion.FEATURE_NAME + ".3", new FeatureVersion[]{UT_FV3_0, UT_FV3_1}, UT_FV3_1),
+    UNIT_TEST_VERSION_4(UnitTestFeatureVersion.FEATURE_NAME + ".4", new FeatureVersion[]{UT_FV4_0, UT_FV4_1}, UT_FV4_1),
+    UNIT_TEST_VERSION_5(UnitTestFeatureVersion.FEATURE_NAME + ".5", new FeatureVersion[]{UT_FV5_0, UT_FV5_1}, UT_FV5_1);


It's kind of weird to include unit test code in production code.

junrao · 2024-11-25T20:07:14Z

server-common/src/test/java/org/apache/kafka/server/common/FeaturesTest.java

+        "UNIT_TEST_VERSION_2",
+        "UNIT_TEST_VERSION_3",
+        "UNIT_TEST_VERSION_4",
+        "UNIT_TEST_VERSION_5"}, mode = EnumSource.Mode.EXCLUDE)


Hmm, is it correct for EXCLUDE to be empty?

add latest production to features

0efae34

github-actions bot added core Kafka Broker small Small PRs labels Nov 20, 2024

dongnuo123 changed the title ~~[Draft] add latest production to features~~ KAFKA-18052: Decouple the dependency of feature stable version to the metadata version Nov 20, 2024

dongnuo123 changed the title ~~KAFKA-18052: Decouple the dependency of feature stable version to the metadata version~~ [Draft] KAFKA-18052: Decouple the dependency of feature stable version to the metadata version Nov 20, 2024

jolshan reviewed Nov 20, 2024

View reviewed changes

server-common/src/main/java/org/apache/kafka/server/common/TestFeatureVersion.java Outdated Show resolved Hide resolved

dongnuo123 added 2 commits November 20, 2024 17:28

check latestProduction == latestTesting in TestFeature

540f99e

check latestProduction == latestTesting in TestFeature

381ca63

dongnuo123 changed the title ~~[Draft] KAFKA-18052: Decouple the dependency of feature stable version to the metadata version~~ KAFKA-18052: Decouple the dependency of feature stable version to the metadata version Nov 21, 2024

junrao reviewed Nov 21, 2024

View reviewed changes

server-common/src/main/java/org/apache/kafka/server/common/Features.java Outdated Show resolved Hide resolved

validateDefaultValueAndLatestProductionValue

c8c290e

github-actions bot removed the small Small PRs label Nov 21, 2024

dajac added the ci-approved label Nov 22, 2024

junrao reviewed Nov 22, 2024

View reviewed changes

more unit tests + rename defaultValue

d2ea645

github-actions bot added tools kraft labels Nov 23, 2024

dongnuo123 added 2 commits November 23, 2024 15:48

validate test version

d49643d

spotlessapply

c7731f7

dongnuo123 commented Nov 25, 2024

View reviewed changes

junrao reviewed Nov 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-18052: Decouple the dependency of feature stable version to the metadata version #17886

KAFKA-18052: Decouple the dependency of feature stable version to the metadata version #17886

dongnuo123 commented Nov 20, 2024 •

edited

Loading

junrao left a comment

junrao left a comment

dongnuo123 Nov 25, 2024

junrao left a comment

junrao Nov 25, 2024

junrao Nov 25, 2024

junrao Nov 25, 2024

junrao Nov 25, 2024

jolshan Nov 25, 2024

junrao Nov 25, 2024

junrao Nov 25, 2024

junrao Nov 25, 2024

junrao Nov 25, 2024

junrao Nov 25, 2024

junrao Nov 25, 2024

junrao Nov 25, 2024

KAFKA-18052: Decouple the dependency of feature stable version to the metadata version #17886

Are you sure you want to change the base?

KAFKA-18052: Decouple the dependency of feature stable version to the metadata version #17886

Conversation

dongnuo123 commented Nov 20, 2024 • edited Loading

Committer Checklist (excluded from commit message)

junrao left a comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongnuo123 commented Nov 20, 2024 •

edited

Loading