Tag: Pub/Sub

Re-publish Kafka record not setting partition number?

Well, I plan to re-publish Kafka records that my app is consuming to another Kafka topic on another host (internal only). This sort of lets other groups consume data from this topic without having to create connections outside of the private network, since the original data source is a 3rd party vendor. They can then also do their own transformations or other business logic on it as if they were consuming from the original source. Delay from source to re-publish should be minimal. Will still be close to real time.

What I normally do when producing records is not to set a partition by default. Would this be okay when re-publishing records? Will there be conflict with original record partition number to the target topic partition?

ANSWER

How many partitions does your target topic have? Do you have freedom to create as many partitions as the source?

(1) If you can create 1:1 partitions then my suggestion is to retain the original partition number of that record when you publish to the destination topic.

(2) You can also assign the partition in a round-robin way when you have lesser partitions on your target topic. Save metadata info of that record in the headers, such as original partition assignment and etc. Perhaps that will be useful down the line for your consumers.

(3) Lastly, you can NOT assign a partition number. Instead, let Kafka do the magic for you. When no partitions are set, Kafka will determine the next partition in a round robin method. You can do this with a null value instead of an int for partition number.

Bottom line is that it depends on the project requirements, I suppose, or more so on the infrastructure of your destination Kafka.

Kafka Consumer Error – AVRO with Field Type as a Union of null and boolean

Recently I ran into some issues in consuming Kafka AVRO messages. Everything went well up to the de-serialization of the record from the Kafka topic. It all just went crazy once it was being serialized into JSON. That’s pretty much where the proverbial buck stopped.

I was scratching my head on why this odd behavior was happening. Looking at the message published on the Kafka topic, it was following the defined AVRO schema as expected. There were no deviations.

Somehow when being converted to JSON it started throwing an exception. It was always at the same spot every time. It would not go any further.

Debugging the flow of the program, it showed me that what should have been this:

{
	"fieldOne": "hello world",
	"fieldTwo": true,
	"foobar": false
}

Became something like this:

{
	"fieldOne": "hello world",
	"fieldTwo": true,
	"foobar": {
		"boolean": false
	}
}

And that is why the program was throwing an error!

ANSWER

The solution I did was to remove null as a field type. Set the field to be strictly boolean. What I wanted was to initially have the state of this field as null. Then further down the line it can be set to either true or false.

After the AVRO schema was updated, the error went away.

This was reported to Apache AVRO Jira. More details about this problem here – https://www.joseyamut.xyz/2020/07/29/de-serializing-kafka-messages-with-union-defined-field/