Inconsistency in handling Bluetooth addresses

[serhiy-storchaka]

Bug report

1. The format accepted by BTPROTO_HCI protocol is incorrectly documented.
   • it accepts a bytes object bdaddr, not a tuple.
   • bdaddr is a bytes object, not a string.
2. Some protocols accept bdaddr as a string, others accept it as a bytes object. getsockname() returns it as a string or a bytes object, and this is not always consistent with the accepted type.
3. getsockname() not always return an address in the acceptable format. It returns device_id when the acceptable format is a tuple (device_id,), can return a string when the acceptable format is a bytes object.

First problem should be solved by updating the documentation.

For second problem, I propose to make both string and bytes be accepted as a Bluetooth address. Also, if a 1-tuple is accepted, then its element should be accepted, and if bdaddr is accepted, then a 1-tuple (bdaddr,) should be accepted.

The third problem cannot be resolved in backward compatible way. The solution for the second problem formally fixes incompatibility between input and output formats, but formats are still inconsistent between protocols. And after adding support for hci_channel (see #70145), the type of the getsockname() result will depend on the hci_channel value.

Linked PRs

• gh-132099: Fix documentation for the BTPROTO_HCI protocol #132118
• [3.13] gh-132099: Fix documentation for the BTPROTO_HCI protocol (GH-132118) #132482
• gh-132099: Harmonize Bluetooth address handling #132486
• [3.13] gh-132099: Harmonize Bluetooth address handling (GH-132486) #132497
• gh-132099: Accept an integer as the address for BTPROTO_HCI on Linux #132525

[serhiy-storchaka]

Fortunately, the code was completely broken on *BSD systems, so most of this inconsistencies was shielded by other bugs. We were able to change the accepted and returned types without breaking code even more.

The only inconsistency left is in the BTPROTO_HCI protocol on Linux: accepted address is a tuple, but getsockname() returns device_id as an integer. After adding support of channel in #70145, getsockname() can now return a tuple if channel is not zero. It is still weird that the type of the result depends on the value, but this is for backward compatibility. To make input/output consistent, we mast an integer device_id be accepted as an address.