Merge branch 'main' of https://github.com/Azure/azure-sdk-for-python into add_back_error_message

* 'main' of https://github.com/Azure/azure-sdk-for-python: (127 commits)
  [tool]update llc-quick-start-tool for py3 only (Azure#22507)
  [mgmt] update setupy for python3.6 (Azure#22505)
  [pipeline] Upgrade python version to `python3.8` (Azure#22503)
  [AutoRelease] t2-synapse-2022-01-11-70733 (Azure#22439)
  Update tests.md (Azure#22500)
  disable per Azure#22498 (Azure#22499)
  [AnomalyDetector] Release 3.0.0b4 (Azure#22191)
  [translation] update docs to include managed identity option (Azure#22366)
  add back py3.6 (Azure#22478)
  Add Python 3.6 support back to Storage packages (Azure#22483)
  allow Python 3.6 with the sdk (Azure#22456)
  remove hardcoded subscription ID (Azure#22481)
  clean up tests/docs (Azure#22480)
  Increment version for eventhub releases (Azure#22485)
  [Cosmos] Remove support for Python27, set Python36 as minimum, and update cosmos emulator pipeline (Azure#22475)
  Increment version for servicebus releases (Azure#22484)
  add back py3.6 (Azure#22477)
  [AutoRelease] t2-datafactory-2022-01-06-31864 (Azure#22362)
  [AutoRelease] t2-apimanagement-2022-01-06-51326 (Azure#22361)
  [AutoRelease] t2-netapp-2022-01-12-75881 (Azure#22467)
  ...

Author: iscai-msft

.github/CODEOWNERS

@@ -23,7 +23,7 @@
 /sdk/eventhub/                                                       @annatisch @yunhaoling @swathipil @rakshith91
 
 # PRLabel: %Storage
-/sdk/storage/                                                        @amishra-dev @zezha-msft @annatisch @tasherif-msft
+/sdk/storage/                                                        @amishra-dev @zezha-msft @annatisch @tasherif-msft @jalauzon-msft
 
 # PRLabel: %App Configuration
 /sdk/appconfiguration/                                               @xiangyan99 @YalinLi0312
@@ -42,6 +42,7 @@
 /sdk/communication/azure-communication-phonenumbers/                 @RoyHerrod @danielav7 @whisper6284 @AlonsoMondal
 /sdk/communication/azure-communication-sms/                          @RoyHerrod @arifibrahim4
 /sdk/communication/azure-communication-identity/                     @Azure/acs-identity-sdk
+/sdk/communication/**/_shared/                                       @Azure/acs-identity-sdk
 
 # PRLabel: %KeyVault
 /sdk/keyvault/                                                       @schaabs @mccoyp @YalinLi0312
@@ -66,7 +67,7 @@
 /sdk/containerservice/                                               @samkreter @zqingqing1 @GaneshaThirumurthi
 
 # PRLabel: %Cosmos
-/sdk/cosmos/                                                         @kushagraThapar @simorenoh @simplynaveen20 @xinlian12 @moderakh 
+/sdk/cosmos/                                                         @kushagraThapar @simorenoh @simplynaveen20 @xinlian12 @moderakh
 
 # ServiceLabel: %Cosmos %Service Attention
 #/<NotInRepo>/                                                       @simorenoh @gahl-levy
@@ -139,6 +140,9 @@
 # PRLabel: %VideoAnalyzer
 /sdk/videoanalyzer/                                                  @hivyas
 
+# PRLabel: %WebPubSub
+/sdk/webpubsub/                                                      @msyyc @iscai-msft
+
 # Smoke Tests
 /common/smoketest/                                                   @lmazuel @chlowell @annatisch @rakshith91 @shurd @southpolesteve
 
@@ -729,7 +733,7 @@
 
 
 # Management Plane
-/**/*mgmt*/                                                          @RAY-316 @msyyc
+/**/*mgmt*/                                                          @BigCat20196	 @msyyc
 
 
 ###########

README.md

@@ -6,7 +6,7 @@ This repository is for active development of the Azure SDK for Python. For consu
 
 ## _Disclaimer_
 
-_Azure SDK Python packages support for Python 2.7 is ending 01 January 2022. For more information and questions, please refer to https://github.com/Azure/azure-sdk-for-python/issues/20691_
+_Azure SDK Python packages support for Python 2.7 has ended 01 January 2022. For more information and questions, please refer to https://github.com/Azure/azure-sdk-for-python/issues/20691_
 
 ## Getting started
 

doc/dev/private_package/python_multiapi_configuration.md

@@ -16,7 +16,7 @@ the target tag should only contain files from one folder:
 
 ![](one_folder.png)
 
-If your tag contains files from different folders, you need to define some split tags. For example: `ApplicationInsights` has [Tag: package-2021-11-01](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/applicationinsights/resource-manager#tag-package-2021-11-01), but it contains different folders. If you want to publish the package for the tag, you need to split it to different tags: [[applicationinsights\] python multiapi config by RAY-316 · Pull Request #16799 · Azure/azure-rest-api-specs (github.com)](https://github.com/Azure/azure-rest-api-specs/pull/16799/files)
+If your tag contains files from different folders, you need to define some split tags. For example: `ApplicationInsights` has [Tag: package-2021-11-01](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/applicationinsights/resource-manager#tag-package-2021-11-01), but it contains different folders. If you want to publish the package for the tag, you need to split it to different tags: [sample](https://github.com/Azure/azure-rest-api-specs/pull/16799/files)
 
 ![](different_folders.png)
 

doc/dev/tests.md

@@ -162,7 +162,7 @@ If your library has a management plane library, you can build a preparer specifi
 
 1. Create an Azure Resource Management Template for your specific service and the configuration you need. This can be done in the portal by creating the resources and at the very last step (Review + Create) clicking "Download a template for automation". Save this template to a `test-resources.json` file under the directory that contains your library (`sdk/<my-library>/test-resources.json`).
 2. Use the [`New-TestResources.ps1`](https://github.com/Azure/azure-sdk-for-python/tree/main/eng/common/TestResources#on-the-desktop) script to deploy those resources.
-3. Set the environment variables returned from step 2 in your current shell or add them to your `.env` file at the root of the repo to save these secrets. If you choose the latter method, you will have to make sure all the key-value pairs are in the format `<key_name>=<value>`, rather than the  `${env:<key_name>} = '<value>'` formatting used in PowerShell. The names of the environment variables should be in all capital letters, snake case, and be prefixed with the library name. Ie. `TABLES_PRIMARY_KEY`, `FORMRECOGNIZER_ACCOUNT_URL`, `EVENTHUBS_SECRET_KEY`.
+3. Set the environment variables returned from step 2 in your current shell or add them to your `.env` file at the root of the repo to save these secrets. If you choose the latter method, you will have to make sure all the key-value pairs are in the format `<key_name>=<value>`, rather than the  `${env:<key_name>} = '<value>'` formatting used in PowerShell. The names of the environment variables should be in all capital letters, snake case, and be prefixed with the library name. Ie. `TABLES_PRIMARY_KEY`, `FORMRECOGNIZER_ACCOUNT_URL`, `EVENTHUBS_SECRET_KEY`. If the name of the service is more than one word, like Form Recognizer, don't include an underscore between the words. Use `FORMRECOGNIZER_ACCOUNT_URL`, not `FORM_RECOGNIZER_ACCOUNT_URL`.
 4. Create a partial implementation of the PowerShellPreparer to pass in your specific environment variables. An example implementation is shown below for schemaregistry
 
 ```python

eng/ci_tools.txt

@@ -1,5 +1,5 @@
 # requirements leveraged by ci tools
-cryptography==3.1
+cryptography==3.3.2
 setuptools==44.1.0; python_version == '2.7'
 setuptools==46.4.0; python_version >= '3.5'
 virtualenv==20.0.23
@@ -27,7 +27,7 @@ cffi==1.15.0rc2; python_version >= '3.10'
 
 # locking packages defined as deps from azure-sdk-tools or azure-devtools
 pytoml==0.1.21
-pyOpenSSL==19.1.0
+pyOpenSSL==21.0.0
 json-delta==2.0
 ConfigArgParse==1.2.3
 six==1.14.0

eng/common/TestResources/New-TestResources.ps1

@@ -127,6 +127,52 @@ function Retry([scriptblock] $Action, [int] $Attempts = 5)
     }
 }
 
+# NewServicePrincipalWrapper creates an object from an AAD graph or Microsoft Graph service principal object type.
+# This is necessary to work around breaking changes introduced in Az version 7.0.0:
+# https://azure.microsoft.com/en-us/updates/update-your-apps-to-use-microsoft-graph-before-30-june-2022/
+function NewServicePrincipalWrapper([string]$subscription, [string]$resourceGroup, [string]$displayName)
+{
+    $servicePrincipal = Retry {
+        New-AzADServicePrincipal -Role "Owner" -Scope "/subscriptions/$SubscriptionId/resourceGroups/$ResourceGroupName" -DisplayName $displayName
+    }
+    $spPassword = ""
+    $appId = ""
+    if (Get-Member -Name "Secret" -InputObject $servicePrincipal -MemberType property) {
+        Write-Verbose "Using legacy PSADServicePrincipal object type from AAD graph API"
+        # Secret property exists on PSADServicePrincipal type from AAD graph in Az # module versions < 7.0.0
+        $spPassword = $servicePrincipal.Secret
+        $appId = $servicePrincipal.ApplicationId
+    } else {
+        if ((Get-Module Az.Resources).Version -eq "5.1.0") {
+            Write-Verbose "Creating password and credential for service principal via MS Graph API"
+            Write-Warning "Please update Az.Resources to >= 5.2.0 by running 'Update-Module Az'"
+            # Microsoft graph objects (Az.Resources version == 5.1.0) do not provision a secret on creation so it must be added separately.
+            # Submitting a password credential object without specifying a password will result in one being generated on the server side.
+            $password = New-Object -TypeName "Microsoft.Azure.PowerShell.Cmdlets.Resources.MSGraph.Models.ApiV10.MicrosoftGraphPasswordCredential"
+            $password.DisplayName = "Password for $displayName"
+            $credential = Retry { New-AzADSpCredential -PasswordCredentials $password -ServicePrincipalObject $servicePrincipal }
+            $spPassword = ConvertTo-SecureString $credential.SecretText -AsPlainText -Force
+            $appId = $servicePrincipal.AppId
+        } else {
+            Write-Verbose "Creating service principal credential via MS Graph API"
+            # In 7.1.0 the password credential issue was fixed (see https://github.com/Azure/azure-powershell/pull/16690) but the
+            # parameter set was changed making the above call fail due to a missing ServicePrincipalId parameter.
+            $credential = Retry { $servicePrincipal | New-AzADSpCredential }
+            $spPassword = ConvertTo-SecureString $credential.SecretText -AsPlainText -Force
+            $appId = $servicePrincipal.AppId
+        }
+    }
+
+    return @{
+        AppId = $appId
+        ApplicationId = $appId
+        # This is the ObjectId/OID but most return objects use .Id so keep it consistent to prevent confusion
+        Id = $servicePrincipal.Id
+        DisplayName = $servicePrincipal.DisplayName
+        Secret = $spPassword
+    }
+}
+
 function LoadCloudConfig([string] $env)
 {
     $configPath = "$PSScriptRoot/clouds/$env.json"
@@ -522,8 +568,8 @@ try {
     # If no test application ID was specified during an interactive session, create a new service principal.
     if (!$CI -and !$TestApplicationId) {
         # Cache the created service principal in this session for frequent reuse.
-        $servicePrincipal = if ($AzureTestPrincipal -and (Get-AzADServicePrincipal -ApplicationId $AzureTestPrincipal.ApplicationId) -and $AzureTestSubscription -eq $SubscriptionId) {
-            Log "TestApplicationId was not specified; loading cached service principal '$($AzureTestPrincipal.ApplicationId)'"
+        $servicePrincipal = if ($AzureTestPrincipal -and (Get-AzADServicePrincipal -ApplicationId $AzureTestPrincipal.AppId) -and $AzureTestSubscription -eq $SubscriptionId) {
+            Log "TestApplicationId was not specified; loading cached service principal '$($AzureTestPrincipal.AppId)'"
             $AzureTestPrincipal
         } else {
             Log "TestApplicationId was not specified; creating a new service principal in subscription '$SubscriptionId'"
@@ -537,19 +583,17 @@ try {
                 $displayName = "$($baseName)$suffix.test-resources.azure.sdk"
             }
 
-            $servicePrincipal = Retry {
-                New-AzADServicePrincipal -Role "Owner" -Scope "/subscriptions/$SubscriptionId/resourceGroups/$ResourceGroupName" -DisplayName $displayName
-            }
+            $servicePrincipalWrapper = NewServicePrincipalWrapper -subscription $SubscriptionId -resourceGroup $ResourceGroupName -displayName $DisplayName
 
-            $global:AzureTestPrincipal = $servicePrincipal
+            $global:AzureTestPrincipal = $servicePrincipalWrapper
             $global:AzureTestSubscription = $SubscriptionId
 
-            Log "Created service principal '$($AzureTestPrincipal.ApplicationId)'"
-            $AzureTestPrincipal
+            Log "Created service principal. AppId: '$($AzureTestPrincipal.AppId)' ObjectId: '$($AzureTestPrincipal.Id)'"
+            $servicePrincipalWrapper
             $resourceGroupRoleAssigned = $true
         }
 
-        $TestApplicationId = $servicePrincipal.ApplicationId
+        $TestApplicationId = $servicePrincipal.AppId
         $TestApplicationOid = $servicePrincipal.Id
         $TestApplicationSecret = (ConvertFrom-SecureString $servicePrincipal.Secret -AsPlainText)
     }
@@ -886,7 +930,7 @@ Bicep templates, test-resources.bicep.env.
 
 .PARAMETER SuppressVsoCommands
 By default, the -CI parameter will print out secrets to logs with Azure Pipelines log
-commands that cause them to be redacted. For CI environments that don't support this (like 
+commands that cause them to be redacted. For CI environments that don't support this (like
 stress test clusters), this flag can be set to $false to avoid printing out these secrets to the logs.
 
 .EXAMPLE

eng/common/pipelines/templates/steps/install-pipeline-generation.yml

@@ -9,8 +9,8 @@ steps:
   - script: >
       dotnet tool install
       Azure.Sdk.Tools.PipelineGenerator
-      --version 1.0.2-dev.20210621.4
-      --add-source https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk/nuget/v3/index.json
+      --version 1.0.2-dev.20220106.2
+      --add-source https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json
       --tool-path ${{parameters.ToolPath}}
     workingDirectory: $(Pipeline.Workspace)/pipeline-generator
     displayName: 'Install pipeline generator tool'

eng/common/pipelines/templates/steps/prepare-pipelines.yml

@@ -71,7 +71,26 @@ steps:
         --debug
         ${{parameters.TestsConventionOptions}}
       displayName: Create Live Test pipelines for public repository
-      condition: ne('${{parameters.TestsConventionOptions}}','')
+      condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}',''))
+      env:
+        PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
+    - script: >
+        $(Pipeline.Workspace)/pipeline-generator/pipeline-generator
+        --organization https://dev.azure.com/azure-sdk
+        --project internal
+        --prefix ${{parameters.Prefix}}
+        --devopspath "\${{parameters.Prefix}}"
+        --path $(System.DefaultWorkingDirectory)/sdk
+        --endpoint Azure
+        --repository ${{parameters.Repository}}
+        --convention weekly
+        --agentpool Hosted
+        --branch refs/heads/$(DefaultBranch)
+        --patvar PATVAR
+        --debug
+        ${{parameters.TestsConventionOptions}}
+      displayName: Create Weekly (Multi-Cloud) Live Test pipelines for public repository
+      condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}',''))
       env:
         PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
 
@@ -132,6 +151,6 @@ steps:
         --no-schedule
         ${{parameters.TestsConventionOptions}}
       displayName: Create Live Test pipelines for private repository
-      condition: ne('${{parameters.TestsConventionOptions}}','')
+      condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}',''))
       env:
         PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)

eng/common/scripts/stress-testing/stress-test-deployment-lib.ps1

@@ -156,9 +156,10 @@ function DeployStressPackage(
     }
     $imageTag += "/$($pkg.Namespace)/$($pkg.ReleaseName):${deployId}"
 
-    if ($pushImages) {
+    $dockerFilePath = "$($pkg.Directory)/Dockerfile"
+    if ($pushImages -and (Test-Path $dockerFilePath)) {
         Write-Host "Building and pushing stress test docker image '$imageTag'"
-        $dockerFile = Get-ChildItem "$($pkg.Directory)/Dockerfile"
+        $dockerFile = Get-ChildItem $dockerFilePath
         Run docker build -t $imageTag -f $dockerFile.FullName $dockerFile.DirectoryName
         if ($LASTEXITCODE) { return }
         Run docker push $imageTag

eng/pipelines/templates/stages/cosmos-sdk-client.yml

@@ -33,12 +33,9 @@ stages:
     - job: Emulator
       strategy:
         matrix:
-          Windows_Python35:
+          Windows_Python36:
             OSVmImage: 'windows-2019'
-            PythonVersion: '3.5'
-          Windows_Python27:
-            OSVmImage: 'windows-2019'
-            PythonVersion: '2.7'
+            PythonVersion: '3.6'
       pool:
         vmImage: $(OSVmImage)
 

eng/pipelines/templates/stages/platform-matrix-conda-support.json

@@ -1,9 +1,8 @@
 {
   "matrix": {
     "Agent": {
-      "ubuntu-20.04": { "OSVmImage": "MMSUbuntu20.04", "Pool": "azsdk-pool-mms-ubuntu-2004-general" },
-      "windows-2019": { "OSVmImage": "MMS2019", "Pool": "azsdk-pool-mms-win-2019-general" }
+      "ubuntu-20.04": { "OSVmImage": "MMSUbuntu20.04", "Pool": "azsdk-pool-mms-ubuntu-2004-general" }
     },
-    "PythonVersion": [ "3.6", "3.8", "3.9" ]
+    "PythonVersion": [ "3.6" ]
   }
 }

eng/pipelines/templates/stages/platform-matrix-cryptography-dependency.json

@@ -10,20 +10,6 @@
     "CoverageArg": "--disablecov"
   },
   "include": [
-    {
-      "Agent": {
-        "macOS-10.15": { "OSVmImage": "macOS-10.15", "Pool": "Azure Pipelines" }
-      },
-      "PythonVersion": "2.7",
-      "CoverageArg": "--disablecov"
-    },
-    {
-      "Agent": {
-        "windows-2019": { "OSVmImage": "MMS2019", "Pool": "azsdk-pool-mms-win-2019-general" }
-      },
-      "PythonVersion": "3.6",
-      "CoverageArg": "--disablecov"
-    },
     {
       "Config": {
         "Ubuntu2004_310": {

eng/pipelines/templates/stages/platform-matrix.json

@@ -10,7 +10,7 @@
       "windows-2019": { "OSVmImage": "MMS2019", "Pool": "azsdk-pool-mms-win-2019-general" },
       "macOS-10.15": { "OSVmImage": "macOS-10.15", "Pool": "Azure Pipelines" }
     },
-    "PythonVersion": [ "pypy3", "2.7", "3.6", "3.7", "3.8" ],
+    "PythonVersion": [ "pypy3", "3.6", "3.7", "3.8" ],
     "CoverageArg": "--disablecov",
     "TestSamples": "false"
   },

eng/pipelines/templates/steps/analyze.yml

@@ -126,9 +126,10 @@ steps:
     parameters:
       Artifacts: ${{ parameters.Artifacts }}
 
-  - template: /eng/common/pipelines/templates/steps/detect-api-changes.yml
-    parameters:
-      Artifacts: ${{ parameters.Artifacts }}
+  # Temporarily disabling due to #22416
+  # - template: /eng/common/pipelines/templates/steps/detect-api-changes.yml
+  #   parameters:
+  #     Artifacts: ${{ parameters.Artifacts }}
 
   - template: /eng/common/pipelines/templates/steps/eng-common-workflow-enforcer.yml
 

eng/pipelines/templates/steps/build-artifacts.yml

@@ -101,6 +101,17 @@ steps:
 
   - ${{ parameters.BeforePublishSteps }}
 
+  - task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0
+    displayName: 'Upload Package SBOM'
+    inputs:
+      BuildDropPath: $(Build.ArtifactStagingDirectory)
+
+  - task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0
+    displayName: 'Upload Documentation SBOM'
+    condition: and(succeeded(), ${{ parameters.BuildDocs }})
+    inputs:
+      BuildDropPath: $(Build.SourcesDirectory)/_docs
+
   - template: /eng/common/pipelines/templates/steps/publish-artifact.yml
     parameters:
       ArtifactPath: '$(Build.ArtifactStagingDirectory)'

eng/pipelines/templates/steps/build-conda-artifacts.yml

@@ -78,6 +78,11 @@ steps:
       displayName: 'Activate Conda Environment and Build ${{ artifact.name }}'
       workingDirectory: $(Build.SourcesDirectory)/sdk/${{ parameters.ServiceDirectory }}/conda-recipe
   
+  - task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0
+    displayName: 'Upload Conda Package SBOM'
+    inputs:
+      BuildDropPath: '$(Agent.BuildDirectory)/conda/output'
+
   - template: /eng/common/pipelines/templates/steps/publish-artifact.yml
     parameters:
       ArtifactPath: '$(Agent.BuildDirectory)/conda/output'

eng/pipelines/templates/variables/globals.yml

@@ -1,4 +1,5 @@
 variables:
   PythonVersion: '3.9'
   skipComponentGovernanceDetection: true
-  AzureSDKCondaChannel: https://azuresdkconda.blob.core.windows.net/channel1/
+  AzureSDKCondaChannel: https://azuresdkconda.blob.core.windows.net/channel1/
+  Package.EnableSBOMSigning: true

eng/regression_test_tools.txt

@@ -14,7 +14,7 @@ bandit==1.6.2
 # locking packages defined as deps from azure-sdk-tools or azure-devtools
 pytoml==0.1.21
 readme-renderer[md]==25.0
-pyOpenSSL==19.1.0
+pyOpenSSL==21.0.0
 json-delta==2.0
 ConfigArgParse==1.2.3
 six==1.14.0

eng/regression_tools.txt

@@ -1,5 +1,5 @@
 # requirements leveraged by ci tools
-cryptography==3.1
+cryptography==3.3.2
 setuptools==44.1.0; python_version == '2.7'
 setuptools==46.4.0; python_version >= '3.5'
 virtualenv==20.0.23
@@ -23,7 +23,7 @@ black==21.6b0; python_version >= '3.6'
 
 # locking packages defined as deps from azure-sdk-tools or azure-devtools
 pytoml==0.1.21
-pyOpenSSL==19.1.0
+pyOpenSSL==21.0.0
 json-delta==2.0
 ConfigArgParse==1.2.3
 six==1.14.0