Skip to content

New murmur3 based super stream exchange type #8319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kjnilsson opened this issue May 24, 2023 · 0 comments
Open

New murmur3 based super stream exchange type #8319

kjnilsson opened this issue May 24, 2023 · 0 comments

Comments

@kjnilsson
Copy link
Contributor

Is your feature request related to a problem? Please describe.

Currently Super Streams are bound to a direct exchange type with their partition id as the routing key (e.g. 1, 2, 3). Super stream publishing clients use murmur3 to hash a provided key which is used for selecting the target stream partition.

When messages come from other protocols which would not naturally have any awareness of the super stream topology there is no means to re-create this routing in the broker.

Describe the solution you'd like

Instead of using a direct exchange when creating a super stream we should provide an option to instead use a new exchange type, rabbit_super_stream_exchange. This exchange type will take the routing key and use murmur3 hashing to select the partition to route the message to. This will provide the same routing logic as is done client side for stream clients but in the broker.

The exchange type would be similar to

-module(rabbit_sharding_exchange_type_modulus_hash).
but using murmur3 based hashing instead.

Describe alternatives you've considered

Alternatives tend to include leaking knowledge of super stream topology onto client applications which isn't always practical.

Additional context

murmur3 based hashing isn't available in OTP until OTP 27 at the earliest so until then we will have to use a slower erlang based implementation. This implementation needs to be validated against the java and .NET implementations and ideally against the SMHasher test framework.

Here is small potential starting point:

https://github.com/bipthelin/murmerl3/tree/master

@kjnilsson kjnilsson changed the title New exchange type for super streams that is compatible New murmur3 based super stream exchange type May 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant