[SchemaRegistry] update typehints/bug bash fixes (Azure#24112)

* update typehints

* update typehints + version

* update typevar name

* make group_name optional in constructor

* bug bash readme updates

* samples docstring/readme/MessageContent docstring

* update changelog

* libbas comments

Author: swathipil

sdk/schemaregistry/azure-schemaregistry-avroencoder/CHANGELOG.md

@@ -1,15 +1,38 @@
 # Release History
 
-## 1.0.0b4 (Unreleased)
+## 1.0.0 (Unreleased)
+
+**Note:** This is the first stable release of our efforts to create a user-friendly and Pythonic client library for Azure Schema Registry.
 
 ### Features Added
 
+- `AvroEncoder` sync and async classes provide the functionality to encode and decode content which follows a schema with the RecordSchema format, as defined by the Apache Avro specification. The Apache Avro library is used as the implementation for encoding and decoding.
+The encoder will automatically register and retrieve schemas from Azure Schema Registry Service. It provides the following methods:
+  - constructor: If `auto_register=True` keyword is passed in, will automatically register schemas passed in to the `encode` method. Otherwise, and by default, will require pre-registering of schemas passed to `encode`. Takes a `group_name` argument that is optional when decoding, but required for encoding.
+  - `encode`: Encodes dict content into bytes according to the given schema and registers schema if needed. Returns either a dict of encoded content and corresponding content type or a `MessageType` subtype object, depending on arguments provided.
+  - `decode`: Decodes bytes content into dict content by automatically retrieving schema from the service.
+- `MessageContent` TypedDict has been introduced with the following required keys:
+  - `content`: The bytes content.
+  - `content_type`: The string content type, which holds the schema ID and the record format indicator.
+- `MessageType` has been introduced with the following methods:
+  - `from_message_content`: Class method that creates an object with given bytes content and string content type.
+  - `__message_content__`: Returns a `MessageContent` object with content and content type values set to their respective properties on the object.
+- Schemas and Schema IDs are cached locally, so that multiple calls with the same schema/schema ID will not trigger multiple service calls.
+- The number of hits, misses, and total entries for the schema/schema ID caches will be logged at an info level when a new entry is added.
+- `InvalidContentError` has been introduced for errors related to invalid content and content types, where `__cause__` will contain the underlying exception raised by the Avro library.
+- `InvalidSchemaError` has been introduced for errors related to invalid schemas, where `__cause__` will contain the underlying exception raised by the Apache Avro library.
+- The `encode` and `decode` methods on `AvroEncoder` support the following message models:
+  - `azure.eventhub.EventData` in `azure-eventhub>=5.9.0`
+
 ### Breaking Changes
 
 ### Bugs Fixed
 
 ### Other Changes
 
+- This package is meant to replace the azure-schemaregistry-avroserializer package, which will no longer be supported.
+- `group_name` is now an optional parameter in the sync and async `AvroEncoder` constructors.
+
 ## 1.0.0b3 (2022-04-05)
 
 ### Breaking Changes

sdk/schemaregistry/azure-schemaregistry-avroencoder/README.md

@@ -14,10 +14,10 @@ _Azure SDK Python packages support for Python 2.7 has ended 01 January 2022. For
 
 ### Install the package
 
-Install the Azure Schema Registry Avro Encoder client library and Azure Identity client library for Python with [pip][pip]:
+Install the Azure Schema Registry Avro Encoder client library for Python with [pip][pip]:
 
 ```Bash
-pip install azure-schemaregistry-avroencoder azure-identity
+pip install azure-schemaregistry-avroencoder
 ```
 
 ### Prerequisites:
@@ -47,14 +47,15 @@ pip install aiohttp
 **Create AvroEncoder using the azure-schemaregistry library:**
 
 ```python
+import os
 from azure.schemaregistry import SchemaRegistryClient
 from azure.schemaregistry.encoder.avroencoder import AvroEncoder
 from azure.identity import DefaultAzureCredential
 
 credential = DefaultAzureCredential()
 # Namespace should be similar to: '<your-eventhub-namespace>.servicebus.windows.net'
-fully_qualified_namespace = '<< FULLY QUALIFIED NAMESPACE OF THE SCHEMA REGISTRY >>'
-group_name = '<< GROUP NAME OF THE SCHEMA >>'
+fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
+group_name = os.environ['SCHEMAREGISTRY_GROUP']
 schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, credential)
 encoder = AvroEncoder(client=schema_registry_client, group_name=group_name)
 ```
@@ -70,11 +71,11 @@ content type with schema ID. Uses [SchemaRegistryClient][schemaregistry_client]
 
 Support has been added to certain Azure Messaging SDK model classes for interoperability with the `AvroEncoder`. These models are subtypes of the `MessageType` protocol defined under the `azure.schemaregistry.encoder.avroencoder` namespace. Currently, the supported model classes are:
 
-- `azure.eventhub.EventData` for `azure-eventhub==5.9.0b3`
+- `azure.eventhub.EventData` for `azure-eventhub>=5.9.0`
 
 ### Message format
 
-If a message type that follows the MessageType protocol is provided to the encoder, it will encode the corresponding content and content type properties as follows:
+If a message type that follows the MessageType protocol is provided to the encoder for encoding, it will set the corresponding content and content type properties, where:
 
 - `content`: Avro payload (in general, format-specific payload)
   - Avro Binary Encoding
@@ -86,6 +87,10 @@ If a message type that follows the MessageType protocol is provided to the encod
   - `avro/binary` is the format indicator
   - `<schema ID>` is the hexadecimal representation of GUID, same format and byte order as the string from the Schema Registry service.
 
+If `EventData` is passed in as the message type, the following properties will be set on the `EventData` object:
+ - The `body` property will be set to the content value.
+ - The `content_type` property will be set to the content type value.
+
 If message type is not provided, and by default, the encoder will create the following dict:
 `{"content": <Avro encoded payload>, "content_type": 'avro/binary+<schema ID>' }`
 
@@ -100,8 +105,8 @@ The following sections provide several code snippets covering some of the most c
 
 ### Encoding
 
-Use `AvroEncoder.encode` method to encode dict content with the given Avro schema.
-The method will use a schema previously registered to the Schema Registry service and keep the schema cached for future encoding usage. It is also possible to avoid pre-registering the schema to the service and automatically register with the `encode` method by instantiating the `AvroEncoder` with the keyword argument `auto_register=True`.
+Use the `AvroEncoder.encode` method to encode content with the given Avro schema.
+The method will use a schema previously registered to the Schema Registry service and keep the schema cached for future encoding usage. In order to avoid pre-registering the schema to the service and automatically register it with the `encode` method, the keyword argument `auto_register=True` should be passed to the `AvroEncoder` constructor.
 
 ```python
 import os
@@ -112,7 +117,7 @@ from azure.eventhub import EventData
 
 token_credential = DefaultAzureCredential()
 fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
-group_name = "<your-group-name>"
+group_name = os.environ['SCHEMAREGISTRY_GROUP']
 name = "example.avro.User"
 format = "Avro"
 
@@ -128,7 +133,7 @@ definition = """
 }"""
 
 schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, token_credential)
-schema_register_client.register(group_name, name, definition, format)
+schema_registry_client.register_schema(group_name, name, definition, format)
 encoder = AvroEncoder(client=schema_registry_client, group_name=group_name)
 
 with encoder:
@@ -143,7 +148,7 @@ with encoder:
 
 ### Decoding
 
-Use `AvroEncoder.decode` method to decode the bytes value into dict content by either:
+Use the `AvroEncoder.decode` method to decode the Avro-encoded content by either:
  - Passing in a message object that is a subtype of the MessageType protocol.
  - Passing in a dict with keys `content`(type bytes) and `content_type` (type string).
 The method automatically retrieves the schema from the Schema Registry Service and keeps the schema cached for future decoding usage.
@@ -159,10 +164,12 @@ fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE
 group_name = "<your-group-name>"
 
 schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, token_credential)
-encoder = AvroEncoder(client=schema_registry_client, group_name=group_name)
+encoder = AvroEncoder(client=schema_registry_client)
 
 with encoder:
     # event_data is an EventData object with Avro encoded body
+    dict_content = {"name": "Ben", "favorite_number": 7, "favorite_color": "red"}
+    event_data = encoder.encode(dict_content, schema=definition, message_type=EventData)
     decoded_content = encoder.decode(event_data)
 
     # OR 
@@ -175,7 +182,7 @@ with encoder:
 
 ### Event Hubs Sending Integration
 
-Integration with [Event Hubs][eventhubs_repo] to send encoded Avro dict content as the body of EventData.
+Integration with [Event Hubs][eventhubs_repo] to send an `EventData` object with `body` set to Avro-encoded content and corresponding `content_type`.
 
 ```python
 import os
@@ -186,7 +193,7 @@ from azure.identity import DefaultAzureCredential
 
 token_credential = DefaultAzureCredential()
 fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
-group_name = "<your-group-name>"
+group_name = os.environ['SCHEMAREGISTRY_GROUP']
 eventhub_connection_str = os.environ['EVENT_HUB_CONN_STR']
 eventhub_name = os.environ['EVENT_HUB_NAME']
 
@@ -219,7 +226,7 @@ with eventhub_producer, avro_encoder:
 
 ### Event Hubs Receiving Integration
 
-Integration with [Event Hubs][eventhubs_repo] to receive `EventData` and decoded raw bytes into Avro dict content.
+Integration with [Event Hubs][eventhubs_repo] to receive an `EventData` object and decode the the Avro-encoded `body` value.
 
 ```python
 import os
@@ -230,7 +237,7 @@ from azure.identity import DefaultAzureCredential
 
 token_credential = DefaultAzureCredential()
 fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
-group_name = "<your-group-name>"
+group_name = os.environ['SCHEMAREGISTRY_GROUP']
 eventhub_connection_str = os.environ['EVENT_HUB_CONN_STR']
 eventhub_name = os.environ['EVENT_HUB_NAME']
 
@@ -254,7 +261,7 @@ with eventhub_consumer, avro_encoder:
 
 ### General
 
-Azure Schema Registry Avro Encoder raises exceptions defined in [Azure Core][azure_core].
+Azure Schema Registry Avro Encoder raises exceptions defined in [Azure Core][azure_core] if errors are encountered when communicating with the Schema Registry service. Errors related to invalid content/content types and invalid schemas will be raised as `azure.schemaregistry.encoder.avroencoder.InvalidContentError` and `azure.schemaregistry.encoder.avroencoder.InvalidSchemaError`, respectively, where `__cause__` will contain the underlying exception raised by the Apache Avro library.
 
 ### Logging
 This library uses the standard
@@ -266,6 +273,7 @@ Detailed DEBUG level logging, including request/response bodies and unredacted
 headers, can be enabled on a client with the `logging_enable` argument:
 ```python
 import sys
+import os
 import logging
 from azure.schemaregistry import SchemaRegistryClient
 from azure.schemaregistry.encoder.avroencoder import AvroEncoder
@@ -279,23 +287,25 @@ logger.setLevel(logging.DEBUG)
 handler = logging.StreamHandler(stream=sys.stdout)
 logger.addHandler(handler)
 
+fully_qualified_namespace = os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']
+group_name = os.environ['SCHEMAREGISTRY_GROUP']
 credential = DefaultAzureCredential()
-schema_registry_client = SchemaRegistryClient("<your-fully_qualified_namespace>", credential, logging_enable=True)
+schema_registry_client = SchemaRegistryClient(fully_qualified_namespace, credential, logging_enable=True)
 # This client will log detailed information about its HTTP sessions, at DEBUG level
-encoder = AvroEncoder(client=schema_registry_client, group_name="<your-group-name>")
+encoder = AvroEncoder(client=schema_registry_client, group_name=group_name)
 ```
 
 Similarly, `logging_enable` can enable detailed logging for a single operation,
 even when it isn't enabled for the client:
 ```py
-encoder.encode(dict_content, schema=schema_definition, logging_enable=True)
+encoder.encode(dict_content, schema=definition, logging_enable=True)
 ```
 
 ## Next steps
 
 ### More sample code
 
-Please find further examples in the [samples][sr_avro_samples] directory demonstrating common Azure Schema Registry Avro Encoder scenarios.
+Further examples demonstrating common Azure Schema Registry Avro Encoder scenarios are in the [samples][sr_avro_samples] directory.
 
 ## Contributing
 

sdk/schemaregistry/azure-schemaregistry-avroencoder/azure/schemaregistry/encoder/avroencoder/_message_protocol.py

@@ -25,8 +25,8 @@ def from_message_content(
         cls, content: bytes, content_type: str, **kwargs: Any
     ) -> "MessageType":
         """
-        Creates an object that is a subtype of MessageType given content type and
-         a content value to be set as body.
+        Creates an object that is a subtype of MessageType, given content type and
+         a content value to be set on the object.
 
         :param bytes content: The content value to be set as the body of the message.
         :param str content_type: The content type to be set on the message.
@@ -35,4 +35,10 @@ def from_message_content(
         ...
 
     def __message_content__(self) -> MessageContent:
+        """
+        A MessageContent object, with `content` and `content_type` set to
+         the values of their respective properties on the MessageType object.
+
+        :rtype: ~azure.schemaregistry.encoder.avroencoder.MessageContent
+        """
         ...

sdk/schemaregistry/azure-schemaregistry-avroencoder/azure/schemaregistry/encoder/avroencoder/_schema_registry_avro_encoder.py

@@ -33,41 +33,39 @@
     Optional,
     Type,
     overload,
-    TypeVar,
     Union,
 )
 from ._utils import (  # pylint: disable=import-error
     validate_schema,
     create_message_content,
     validate_message,
     decode_content,
+    MessageType
 )
 
 from ._apache_avro_encoder import (  # pylint: disable=import-error
     ApacheAvroObjectEncoder as AvroObjectEncoder,
 )
 from ._message_protocol import (  # pylint: disable=import-error
     MessageContent,
-    MessageType,
 )
 
 if TYPE_CHECKING:
     from azure.schemaregistry import SchemaRegistryClient
 
 _LOGGER = logging.getLogger(__name__)
 
-MessageTypeT = TypeVar("MessageTypeT", bound=MessageType)
-
 
 class AvroEncoder(object):
     """
     AvroEncoder provides the ability to encode and decode content according
     to the given avro schema. It would automatically register, get and cache the schema.
 
-    :keyword client: Required. The schema registry client
-     which is used to register schema and retrieve schema from the service.
+    :keyword client: Required. The schema registry client which is used to register schema
+     and retrieve schema from the service.
     :paramtype client: ~azure.schemaregistry.SchemaRegistryClient
-    :keyword str group_name: Required. Schema group under which schema should be registered.
+    :keyword Optional[str] group_name: Required for encoding. Not used for decoding.
+     Schema group under which schema should be registered.
     :keyword bool auto_register: When true, register new schemas passed to encode.
      Otherwise, and by default, encode will fail if the schema has not been pre-registered in the registry.
 
@@ -76,13 +74,13 @@ class AvroEncoder(object):
     def __init__(self, **kwargs):
         # type: (Any) -> None
         try:
-            self._schema_group = kwargs.pop("group_name")
             self._schema_registry_client = kwargs.pop(
                 "client"
             )  # type: "SchemaRegistryClient"
         except KeyError as exc:
-            raise TypeError("'{}' is a required keyword.".format(exc.args[0]))
+            raise TypeError(f"'{exc.args[0]}' is a required keyword.")
         self._avro_encoder = AvroObjectEncoder(codec=kwargs.get("codec"))
+        self._schema_group = kwargs.pop("group_name", None)
         self._auto_register = kwargs.get("auto_register", False)
         self._auto_register_schema_func = (
             self._schema_registry_client.register_schema
@@ -146,10 +144,10 @@ def encode(
         content: Mapping[str, Any],
         *,
         schema: str,
-        message_type: Type[MessageTypeT],
+        message_type: Type[MessageType],
         request_options: Optional[Dict[str, Any]] = None,
         **kwargs: Any,
-    ) -> MessageTypeT:
+    ) -> MessageType:
         ...
 
     @overload
@@ -166,13 +164,13 @@ def encode(
 
     def encode(
         self,
-        content,
+        content: Mapping[str, Any],
         *,
-        schema,
-        message_type=None,
-        request_options=None,
-        **kwargs,
-    ):
+        schema: str,
+        message_type: Optional[Type[MessageType]] = None,
+        request_options: Optional[Dict[str, Any]] = None,
+        **kwargs: Any,
+    ) -> Union[MessageType, MessageContent]:
         """
         Encode content with the given schema. Create content type value, which consists of the Avro Mime Type string
          and the schema ID corresponding to given schema. If provided with a MessageType subtype, encoded content
@@ -202,6 +200,8 @@ def encode(
         """
 
         raw_input_schema = schema
+        if not self._schema_group:
+            raise TypeError("'group_name' in constructor cannot be None, if encoding.")
         schema_fullname = validate_schema(self._avro_encoder, raw_input_schema)
 
         cache_misses = (

sdk/schemaregistry/azure-schemaregistry-avroencoder/azure/schemaregistry/encoder/avroencoder/_utils.py

@@ -4,7 +4,7 @@
 # --------------------------------------------------------------------------------------------
 
 from io import BytesIO
-from typing import TYPE_CHECKING, Any, Dict, Mapping, Optional, Type, Union, cast
+from typing import TYPE_CHECKING, Any, Dict, Mapping, Optional, Type, Union, cast, TypeVar
 from avro.errors import SchemaResolutionException  # type: ignore
 
 from ._exceptions import (  # pylint: disable=import-error
@@ -13,7 +13,7 @@
 )
 from ._message_protocol import (  # pylint: disable=import-error
     MessageContent,
-    MessageType,
+    MessageType as MessageTypeProtocol,
 )
 from ._constants import (  # pylint: disable=import-error
     AVRO_MIME_TYPE,
@@ -24,6 +24,8 @@
         ApacheAvroObjectEncoder as AvroObjectEncoder,
     )
 
+MessageType = TypeVar("MessageType", bound=MessageTypeProtocol)
+
 
 def validate_schema(avro_encoder: "AvroObjectEncoder", raw_input_schema: str):
     try:
@@ -61,7 +63,7 @@ def create_message_content(
 
     if message_type:
         try:
-            return message_type.from_message_content(payload, content_type, **kwargs)
+            return cast(MessageType, message_type.from_message_content(payload, content_type, **kwargs))
         except AttributeError as exc:
             raise TypeError(
                 f"""Cannot set content and content type on model object. The content model

sdk/schemaregistry/azure-schemaregistry-avroencoder/azure/schemaregistry/encoder/avroencoder/_version.py

@@ -24,4 +24,4 @@
 #
 # --------------------------------------------------------------------------
 
-VERSION = "1.0.0b4"
+VERSION = "1.0.0"