Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SchemaName is not unique for schema generated in Kafka Connect Avro Converter #63

Open
kamleshbhadane87 opened this issue Jan 16, 2025 · 0 comments

Comments

@kamleshbhadane87
Copy link

I have noticed Azure Schema Registry uniquely identifies Avro type schema using "name" and "namespace" fields from Avro schema, Please refer below sample schema and screenshot from Azure Portal.

For 1 connector (respective topic lets say topic1 ) all seems good, when we create new connector(say connector2) for that same values of "name" and "namespace" will be used for respective topic( lets say topic2) schema and connector task will throw below exception due to conflicts. To cross verify my understanding I deleted schema for topic1 and started connector2 and same was running without any exception.

I wanted to highlight this issue and correct my understanding, please let me know if I am missing anything. Also is there any workaround to generate unique name for schemName and identify schema ?

Thanks in advance!

ERROR:

[2025-01-16 18:13:38,199] ERROR [SRC_Mysql_OrderDtls|task-0] WorkerSourceTask{id=SRC_Mysql_OrderDtls-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:221)
org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
        at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:244)
        at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:166)
Caused by: com.azure.data.schemaregistry.implementation.models.ErrorException: Status code 400, "{"error":{"code":"InvalidRequest","message":"Avro schema validation failed: Backward compatibility test failed when checking against schema com.microsoft.azure.ConnectSchema, version 1, id 50370a60-c425-4ec9-a26d-c7a74a5a7ff4. TrackingId:1f4c49d4-6789-4f4c-9c53-a52e3a2caaaa_G1, SystemTracker:kafka-schema-registry.servicebus.windows.net:$schemaGroups\/kafka\/schemas\/com.microsoft.azure.ConnectSchema, Timestamp:2025-01-16T12:43:38"}}"
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
        Suppressed: java.lang.Exception: #block terminated with an error
                at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:190)
                at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:224)
                at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:166)
json

{
    "type": "record",
    "name": "ConnectSchema",
    "namespace": "com.microsoft.azure",
    "fields": [
        {
            "name": "erporder",
            "type": {
                "type": "bytes",
                "scale": 0,
                "precision": 8,
                "logicalType": "decimal"
            }
        },
        {
            "name": "sfdcorder",
            "type": {
                "type": "bytes",
                "scale": 0,
                "precision": 8,
                "logicalType": "decimal"
            }
        },
        {
            "name": "ordamt",
            "type": {
                "type": "bytes",
                "scale": 2,
                "precision": 10,
                "logicalType": "decimal"
            }
        },
        {
            "name": "orddt",
            "type": {
                "type": "int",
                "logicalType": "date"
            }
        },
        {
            "name": "ordsts",
            "type": "string"
        },
        {
            "name": "custmr",
            "type": {
                "type": "bytes",
                "scale": 0,
                "precision": 8,
                "logicalType": "decimal"
            }
        },
        {
            "name": "CREATED_AT",
            "type": [
                "null",
                {
                    "type": "long",
                    "logicalType": "timestamp-millis"
                }
            ],
            "default": null
        }
    ]
}

Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant