You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Currently Super Streams are bound to a direct exchange type with their partition id as the routing key (e.g. 1, 2, 3). Super stream publishing clients use murmur3 to hash a provided key which is used for selecting the target stream partition.
When messages come from other protocols which would not naturally have any awareness of the super stream topology there is no means to re-create this routing in the broker.
Describe the solution you'd like
Instead of using a direct exchange when creating a super stream we should provide an option to instead use a new exchange type, rabbit_super_stream_exchange. This exchange type will take the routing key and use murmur3 hashing to select the partition to route the message to. This will provide the same routing logic as is done client side for stream clients but in the broker.
Alternatives tend to include leaking knowledge of super stream topology onto client applications which isn't always practical.
Additional context
murmur3 based hashing isn't available in OTP until OTP 27 at the earliest so until then we will have to use a slower erlang based implementation. This implementation needs to be validated against the java and .NET implementations and ideally against the SMHasher test framework.
Is your feature request related to a problem? Please describe.
Currently Super Streams are bound to a direct exchange type with their partition id as the routing key (e.g. 1, 2, 3). Super stream publishing clients use murmur3 to hash a provided key which is used for selecting the target stream partition.
When messages come from other protocols which would not naturally have any awareness of the super stream topology there is no means to re-create this routing in the broker.
Describe the solution you'd like
Instead of using a direct exchange when creating a super stream we should provide an option to instead use a new exchange type,
rabbit_super_stream_exchange
. This exchange type will take the routing key and use murmur3 hashing to select the partition to route the message to. This will provide the same routing logic as is done client side for stream clients but in the broker.The exchange type would be similar to
rabbitmq-server/deps/rabbitmq_sharding/src/rabbit_sharding_exchange_type_modulus_hash.erl
Line 8 in 6746462
Describe alternatives you've considered
Alternatives tend to include leaking knowledge of super stream topology onto client applications which isn't always practical.
Additional context
murmur3 based hashing isn't available in OTP until OTP 27 at the earliest so until then we will have to use a slower erlang based implementation. This implementation needs to be validated against the java and .NET implementations and ideally against the SMHasher test framework.
Here is small potential starting point:
https://github.com/bipthelin/murmerl3/tree/master
The text was updated successfully, but these errors were encountered: