Re-publish Kafka record not setting partition number?

Well, I plan to re-publish Kafka records that my app is consuming to another Kafka topic on another host (internal only). This sort of lets other groups consume data from this topic without having to create connections outside of the private network, since the original data source is a 3rd party vendor. They can then also do their own transformations or other business logic on it as if they were consuming from the original source. Delay from source to re-publish should be minimal. Will still be close to real time.

What I normally do when producing records is not to set a partition by default. Would this be okay when re-publishing records? Will there be conflict with original record partition number to the target topic partition?

ANSWER

How many partitions does your target topic have? Do you have freedom to create as many partitions as the source?

(1) If you can create 1:1 partitions then my suggestion is to retain the original partition number of that record when you publish to the destination topic.

(2) You can also assign the partition in a round-robin way when you have lesser partitions on your target topic. Save metadata info of that record in the headers, such as original partition assignment and etc. Perhaps that will be useful down the line for your consumers.

(3) Lastly, you can NOT assign a partition number. Instead, let Kafka do the magic for you. When no partitions are set, Kafka will determine the next partition in a round robin method. You can do this with a null value instead of an int for partition number.

Bottom line is that it depends on the project requirements, I suppose, or more so on the infrastructure of your destination Kafka.

How does Mixcloud deliver its streams?

I don’t know much about Web programming, but I am able to search a website for references of .m3u and such. I would like to find out how Mixcloud does deliver the sources of the streams but it seems not so easy.

I think it is done with React, at least that’s a reference I can find in the code. There seems to be something like an event when clicking on the button and then there is an object t and I think maybe this object has the stream URL as a property somewhere. But it’s also possible that I am totally wrong.

I am just trying to play some of my own mixes via a Windows system tray program I developed, just as I can already play some of my favorite radio stations with just one click using my application.

I would send the app to anybody that can help and wants it. 🙂

Go to Source
Author: Ben