Question: Explicitly specify partition to send message to when said partition can change

[ghost]

The way I understand it there are 2 ways to produce messages in kafka-go. You either use Conn to produce messages to a particular partition of a particular topic, or you use Writer to produce messages to a particular topic letting balancer to choose which partition to send message to.

So I guess if one wants to explicitly specify partition to send message to while using only one underlying connection, he should use custom balancer which always returns the same partition? Something like

type CustomBalancer struct {
	// lock may be required depending on the usage, I guess...
	Partition int
}

func (cb *CustomBalancer) Balance(_ Message, _ ...int) int {
	return cb.Partition
}

func (cb *CustomBalancer) SetPartition(partition int) {
	cb.Partition = partition
}

[achille-roussel]

Hello @CheKhenKho.

Partitions are local to brokers in a kafka cluster, you can use one connection per broker but you cannot use a single connection per cluster.

The implementation of kafka.Writer isn't cheap in terms of connections, but we've been working on improving this for the 0.4.x releases.

[achille-roussel]

Closing since #461 has improved connection management.