Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AVRO Record types without fields can't be consumed #305

Open
PHameete opened this issue Mar 9, 2021 · 4 comments · May be fixed by #361
Open

AVRO Record types without fields can't be consumed #305

PHameete opened this issue Mar 9, 2021 · 4 comments · May be fixed by #361

Comments

@PHameete
Copy link

PHameete commented Mar 9, 2021

When consuming a topic that contains an AVRO type that has no fields, the following error is produced:

% ERROR: Failed to format message in event-rule-engine-eventTriggerStateStore-changelog [2] at offset 18734304: Avro/Schema-registry message deserialization: Record type must have at least one field : terminating

The AVRO specification does not indicate that such types are not allowed. You can create them using the official Java AVRO library, and the Confluent Java AVRO serdes can also serialize/deserialize them without issues.

Is it possible to remove this check: https://github.com/confluentinc/avro-c-packaging/blob/master/src/schema.c#L922 so kafkacat can also consume this type of record?

@OneCricketeer
Copy link

Can you clarify the purpose of a record without fields? Perhaps you'd be better off just sending a boolean primitive instead if you simply want some message?

does not indicate that such types are not allowed

How so? The docs say "fields: a JSON array, listing fields (required)"

@PHameete
Copy link
Author

Can you clarify the purpose of a record without fields? Perhaps you'd be better off just sending a boolean primitive instead if you simply want some message?

For forward compatibility I find it easier to have a record without fields and then add (optional) fields to it later than to replace a boolean with a record.

How so? The docs say "fields: a JSON array, listing fields (required)"

Yes the "fields" array must exist, but it is allowed to be empty. At least that is my interpretation, and the offical AVRO library will also accept and generate classes for such a schema.

@bashtanov
Copy link

I'd find it useful too, as we have empty record types as well. Other avro-aware software works well with it, so, in case one finds the spec ambiguous, it's allowed de-facto.

@r-glyde
Copy link

r-glyde commented Dec 29, 2021

It looks like a fix for this was merged into avro-c a while ago (apache/avro#159) and is in the release-1.11.0 tag. https://github.com/confluentinc/avro-c-packaging doesn't have the same fix but I'm not really sure what the purpose of that repo is as it just looks like a copy and paste of an old version of the apache/avro c implementation?

Would we be able to just update the version being used here?

github_download "apache/avro" "release-1.8.2" "avroc"

We generally build this using the bootstrap script but not sure if this would fix the problem for everyone (haven't tested for ourselves yet either) or if there is somewhere else that would need to be updated?

@r-glyde r-glyde linked a pull request Dec 31, 2021 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants