re-recording

Author: seankane-msft

CONTRIBUTING.md

@@ -111,6 +111,10 @@ Mypy install and run.
 **Example: Invoke tox, breaking into the debugger on failure**
 `tox -e whl -c ../../../eng/tox/tox.ini -- --pdb`
 
+### More Reading
+
+We maintain an [additional document](doc/eng_sys_checks.md) that has a ton of detail as to what is actually _happening_ in these executions.
+
 ### Dev Feed
 Daily dev build version of Azure sdk packages for python are available and are uploaded to Azure devops feed daily. We have also created a tox environment to test a package against dev built version of dependent packages. Below is the link to Azure devops feed.
 [`https://dev.azure.com/azure-sdk/public/_packaging?_a=feed&feed=azure-sdk-for-python`](https://dev.azure.com/azure-sdk/public/_packaging?_a=feed&feed=azure-sdk-for-python)

doc/README.md

@@ -2,4 +2,6 @@ This folder contains some documentations for this repository:
 
 The folder structure is the following
 - [sphinx](./sphinx) : contains the documentation source code for https://azure.github.io/azure-sdk-for-python/
-- [dev](./dev) : contains advanced documentation for _developers_ of SDK (not _consumers_ of SDK)
+- [dev](./dev) : contains advanced documentation for _developers_ of SDK (not _consumers_ of SDK)
+
+The file [eng_sys_checks](eng_sys_checks.md) is a read up as to what a standard `ci.yml` will actually execute.

doc/eng_sys_checks.md

@@ -0,0 +1,196 @@
+# Azure SDK for Python - Engineering System
+
+There are various tests currently enabled in Azure pipeline for Python SDK and some of them are enabled only for nightly CI checks. We also run some static analysis tool to verify code completeness, security and lint check.  
+
+Check the [contributing guide](../CONTRIBUTING.md#building-and-testing) for an intro to `tox`.
+
+As a contributor, you will see the build jobs run in two modes: `Nightly Scheduled` and `Pull Request`.
+
+These utilize the _same build definition_, except that the `nightly` builds run additional, deeper checks that run for a bit longer.
+
+Example PR build:
+
+![res/job_snippet.png](res/job_snippet.png)
+
+ - `Analyze` tox envs run during the `Analyze job.
+ - `Test <platform>_<pyversion>` runs PR/Nightly tox envs, depending on context.
+
+## Analyze Checks
+Analyze job in both nightly CI and pull request validation pipeline runs a set of static analysis using external and internal tools. Following are the list of these static analysis.
+
+#### MyPy
+`Mypy` is a static analysis tool that runs type checking of python package. Following are the steps to run `MyPy` locally for a specific package
+1. Go to root of the package
+2. Execute following command
+   ```tox -e mypy -c ../../../eng/tox/tox.ini ``` 
+
+#### Pylint
+`Pylint` is a static analysis tool to run lint checking. Following are the steps to run `pylint` locally for a specific package.
+ 
+1. Go to root of the package.
+2. Execute following command
+   ```tox -e pylint -c ../../../eng/tox/tox.ini```
+ 
+
+#### Bandit
+`Bandit` is static security analysis tool. This check is triggered for all Azure SDK package as part of analyze job. Following are the steps to `Bandit` tool locally for a specific package.
+
+1. Got to package root directory.
+2. Execute following command
+   ```tox -e bandit -c ../../../eng/tox/tox.ini```
+
+
+#### ApiStubGen
+`ApiStubGen` is an internal tool used to create API stub to help reviewing public APIs in our SDK package using [`APIViewTool`.](https://apiview.dev/) This tool also has some built in lint checks available and purpose of having this step in analyze job is to ensure any  change in code is not impacting stubbing process and also to have more and more custom lint checks added in future.
+
+#### Change log verification
+
+Change log verification is added to ensure package has valid change log for current version. Guidelines to properly maintain the change log is documented [here]()
+
+## PR Validation Checks
+Each pull request runs various tests using `pytest` in addition to all the tests mentioned above in analyze check. Pull request validation performs 3 different types of test: `whl, sdist and depends`. Following section explains the purpose of each of these tests and how to execute them locally. All pull requests are validated on multiple python versions across different platforms and below is the test matrix for pull request.
+
+
+|`Python  Version`|`Platform`  |
+|--|--|
+|2.7|Linux|
+|3.5|Windows|
+|3.8|Linux|
+
+### PR validation tox test environments
+Tests are executed using tox environment and following are the tox test names that are part of pull request validation
+#### whl
+This test installs wheel of the package being tested and runs all tests cases in the package using `pytest`. Following is the command to run this test environment locally.
+
+1. Go to package root folder on a command line
+2. Run following command
+   ``tox -e whl -c ../../../eng/tox/tox.ini``
+
+#### sdist
+This test installs sdist of the package being tested and runs all tests cases in the package using `pytest`. Following is the command to run this test environment locally.
+
+1. Go to package root folder on a command line
+2. Run following command
+   ``tox -e sdist -c ../../../eng/tox/tox.ini``
+
+####depends
+This test is to ensure all modules in the package being tested can be successfully imported. This is to ensure all package requirement is properly set in setup.py as well as to ensure modules are imported using valid namespace. This test install the package and it's required packages and executes `from <package-root-namespace> import *`. For e.g. `from azure.core import *`.
+
+Following is the command to run this test environment locally.
+
+1. Go to package root folder on a command line
+2. Run following command
+   ``tox -e sdist -c ../../../eng/tox/tox.ini``
+
+
+## Nightly CI Checks
+
+Nightly continuous integration checks run all tests mentioned above in Analyze and Pull request checks in addition to multiple other tests. Nightly CI checks run on all python versions that are supported by Azure SDK packages across multiple platforms.
+
+![res/full_matrix.png](res/full_matrix.png)
+
+Regression also executes:
+![res/regression.png](res/regression.png)
+
+Nightly CI check runs following additional tests to ensure the dependency between a package being developed against released packages to ensure backward compatibility. Following is the explanation of why we need dependency tests to ensure backward compatibility.
+
+Imagine a situation where package `XYZ` requires another package `ABC` and as per the package requirement of `XYZ`, It should work with any version between 1.0 and 2.0 of package `ABC`. 
+ 
+Package `XYZ` requires package `ABC` 
+ 
+As a developer of package `XYZ`, we need to ensure that our package works fine with all versions of ABC as long as it is within package requirement specification. 
+ 
+Another scenario where regression test( reverse dependency) is required. Let's take same example above and assume we are developers of package `ABC` which is taken as required package by another  package `XYZ` 
+ 
+Package `ABC is required by package `XYZ`
+
+
+As a developer of `ABC`, we need to ensure that any new change in `ABC` is not breaking the use of `XYZ` and hence ensures backward compatibility. 
+ 
+Let's take few Azure SDK packages instead of dummy names to explain this in a context we are more familiar of. 
+ 
+Most of the Azure SDK packages require `azure-core` and this requirement is within a range for e.g. `azure-storage-blob` that requires `azure-core >1.0.0, <2.0.0`. So any new change in azure-storage-blob needs to make sure it works fine with all versions of azure-core between 1.0.0 and 2.0.0(Both included).  
+Similarly any new version of azure-core needs to ensure that it is still compatible with all released package versions which takes azure-core as required package. 
+ 
+It is lot of combinations if we need to run tests for all released versions within the range of requirement specification. In order to reduce the test matrix and at the same time ensures the quality, we currently run the test using oldest released and latest released packages and skips any version in between. 
+
+Following are the additional tests we run during nightly CI checks.
+
+####Latest Dependency Test 
+ 
+This test makes sure that a package being developed works absolutely fine using latest released version of required Azure SDK package as long as there is a released version which satisfies the requirement specification. Workflow of this test is as follows: 
+ 
+1.  Identify if any azure SDK package is marked as required package in setup.py of current package being tested. 
+Note: Any dependency mentioned only in dev_requirements are not considered to identify dependency. 
+2.  Identify latest released version of required azure sdk package on PyPI 
+3.  Install latest released version of required package instead of dev dependency to package in code repo 
+4.  Install current package that is being tested 
+5.  Run pytest of all test cases in current package 
+ 
+Tox name of this test is `latestdependency` and steps to manually run this test locally is as follows. 
+1.  Go to package root. For e.g azure-storage-blob or azure-identity 
+2.  Run following command 
+        `Tox –e latestdependency –c ../../../tox/tox.ini`
+ 
+ 
+####Oldest Dependency Test 
+ 
+This test makes sure that a package being developed works absolutely fine using oldest released version of required Azure SDK package as long as there is a released version which satisfies the requirement specification. Workflow of this test is as follows: 
+ 
+1.  Identify if any azure SDK package is marked as required package in setup.py of current package being tested. 
+Note: Any dependency mentioned only in dev_requirements are not considered to identify dependency. 
+2.  Identify oldest released version of required azure sdk package on PyPI 
+3.  Install oldest released version of required package instead of dev dependency to package in code repo 
+4.  Install current package that is being tested 
+5.  Run pytest of all test cases in current package 
+ 
+Tox name of this test is `mindependency` and steps to manually run this test locally is as follows. 
+1.  Go to package root. For e.g azure-storage-blob or azure-identity 
+2.  Run following command 
+`Tox –e mindependency –c ../../../tox/tox.ini`
+ 
+ 
+####Regression Test 
+ 
+As mentioned earlier, regression test or reverse dependency test is added to avoid a regression scenario for customers when any new change is made in a package that is required by other packages. Currently we have only very few Azure SDK packages that are added as required package by other Azure SDK package. As of now, list of these required packages are: 
+`azure-core`
+`azure-eventhub`
+`azure-storage-blob`
+ 
+Our regression framework automatically finds any such package that is added as required package so this list is not hardcoded. 
+ 
+We have two different set of regression tests to verify regression scenarios against oldest and latest released dependent packages. 
+•   Regression using latest released dependent package 
+•   Regression using oldest released dependent package 
+ 
+One main difference between regression tests and forward dependency test( latest and mindependency) is in terms of what test cases are executed as part of the tests. While forward dependency tests executes the test cases in current code repo, regression tests execute the tests that were part of repo at the time of dependent package release. To make it more clear, let's look at an example here. 
+ 
+Let's assume that we are testing regression for azure-core and this test is for regression against latest released dependent packages. Test will identify all packages that takes azure-core as required package and finds latest released version of those packages. Test framework install currently being developed azure-core and latest released dependent package and runs the test cases in dependent package, for e.g. azure-identity, that were part of repo at the time of releasing depending package. 
+ 
+Workflow of this test is as follows when running regression for an SDK package. 
+1.  Identify any packages that takes currently being tested package as required package 
+2.  Find latest and oldest released versions of dependent package from PyPI 
+3.  Install currently being developed version of package  we are testing regression for. E.g. azure-core 
+4.  Checkout the release tag of dependent package from github 
+5.  Install latest/oldest version of dependent package. For e.g. azure-identity 
+6.  Run test cases within dependent package from checked out branch.  
+ 
+ 
+Steps to manually run regression test locally: 
+1.  Run below command from your git code repo to generate the wheel of package being developed. Currently we have restricted to have prebuilt wheel. 
+`./scripts/devops_tasks/build_packages.py --service= <service-name> -d <out-folder>`
+2.  Run below command to start regression test locally 
+`./scripts/devops_tasks/test_regression.py azure-* --service=<service-name> --whl-dir=<out-folder given above in step 2>`
+ 
+ 
+How to run these additional tests on azure pipelines manually 
+ 
+Following variables can be set at queueing time in order to run these additional tests which are by default run only for scheduled runs. 
+ 
+•   Latest and oldest dependency test in addition to basic testing 
+Variable name: `Run.DependencyTest`
+Value: true 
+ 
+•   Regression test 
+Variable name: `Run.Regression`
+Value: true 

doc/res/full_matrix.png

@@ -11,7 +11,7 @@ parameters:
   PushArgs:
   WorkingDirectory: $(System.DefaultWorkingDirectory)
   PRTitle: not-specified
-  PRBody: not-specified 
+  PRBody: ''
   ScriptDirectory: eng/common/scripts
   GHReviewersVariable: '' 
   GHTeamReviewersVariable: ''
@@ -68,12 +68,13 @@ steps:
       -PRBranch "${{ parameters.PRBranchName }}"
       -AuthToken "$(azuresdk-github-pat)"
       -PRTitle "${{ parameters.PRTitle }}"
+      -PRBody "${{ coalesce(parameters.PRBody, parameters.CommitMsg, parameters.PRTitle) }}"
       -PRLabels "${{ parameters.PRLabels}}"
-      -PRBody "${{ parameters.PRBody }}"
 
 - task: PowerShell@2
   displayName: Tag a Reviewer on PR
   condition: and(succeeded(), eq(variables['HasChanges'], 'true'))
+  continueOnError: true
   inputs:
     pwsh: true
     workingDirectory: ${{ parameters.WorkingDirectory }}

doc/res/job_snippet.png

@@ -18,12 +18,20 @@ steps:
         --kusto-database-var KUSTO_DB `
         --kusto-table-var KUSTO_TABLE `
         --identity "$(Build.QueuedBy)"
-      $resolvedIdentity = $result[-1] | ConvertFrom-Json
 
-      Write-Host $resolvedIdentity
+      $resolvedIdentity = ""
+      try { $resolvedIdentity = $result[-1] | ConvertFrom-Json } catch {}
 
-      Write-Output "##vso[task.setvariable variable=${{ parameters.TargetVariable }}]$($resolvedIdentity.GithubUserName)"
+      if($resolvedIdentity) {
+        Write-Host $resolvedIdentity
+
+        Write-Host "##vso[task.setvariable variable=${{ parameters.TargetVariable }}]$($resolvedIdentity.GithubUserName)"
+      }
+      else {
+        Write-Host "Unable to locate a github user for identity $(Build.QueuedBy)"
+      }
     displayName: 'Resolving Queuing User'
+    continueOnError: true
     workingDirectory: $(Build.SourcesDirectory)/tools_repo/tools/notification-configuration/identity-resolver
     env:
       APP_ID: $(notification-aad-app-id)
@@ -41,6 +49,6 @@ steps:
       $originalValue = "$(${{ parameters.TargetVariable }})"
       $result = $(Build.SourcesDirectory)/eng/common/scripts/get-codeowners.ps1 -TargetDirectory /sdk/${{ parameters.ServiceDirectory }}/ -RootDirectory $(Build.SourcesDirectory)
       if ($result) {
-        Write-Output "##vso[task.setvariable variable=${{ parameters.TargetVariable }}]$originalValue,$result"
+        Write-Host "##vso[task.setvariable variable=${{ parameters.TargetVariable }}]$originalValue,$result"
       }
     displayName: Add CodeOwners if Present

doc/res/regression.png

@@ -17,6 +17,8 @@ The branch which we want to create a pull request for.
 A personal access token
 .PARAMETER PRTitle
 The title of the pull request.
+.PARAMETER PRBody
+The body message for the pull request. 
 .PARAMETER PRLabels
 The labels added to the PRs. Multple labels seperated by comma, e.g "bug, service"
 #>
@@ -42,7 +44,9 @@ param(
 
   [Parameter(Mandatory = $true)]
   [string]$PRTitle,
-  $PRBody = $PRTitle,
+
+  [Parameter(Mandatory = $false)]
+  [string]$PRBody = $PRTitle,
 
   [Parameter(Mandatory = $false)]
   [string]$PRLabels

eng/common/pipelines/templates/steps/create-pull-request.yml

@@ -18,6 +18,32 @@ param(
     $AuthToken
 )
 
+function AddMembers($memberName, $additionSet) {
+  $headers = @{
+    Authorization = "bearer $AuthToken"
+  }
+  $uri = "https://api.github.com/repos/$RepoOwner/$RepoName/pulls/$PRNumber/requested_reviewers"
+  $errorOccurred = $false
+
+  foreach ($id in $additionSet) {
+    try {
+      $postResp = @{}
+      $postResp[$memberName] = @($id)
+      $postResp = $postResp | ConvertTo-Json
+
+      Write-Host $postResp
+      $resp = Invoke-RestMethod -Method Post -Headers $headers -Body $postResp -Uri $uri -MaximumRetryCount 3
+      $resp | Write-Verbose
+    }
+    catch {
+      Write-Host "Error attempting to add $user `n$_"
+      $errorOccurred = $true
+    }
+  }
+
+  return $errorOccurred
+}
+
 # at least one of these needs to be populated
 if (-not $GitHubUsers -and -not $GitHubTeams) {
   Write-Host "No user provided for addition, exiting."
@@ -27,54 +53,9 @@ if (-not $GitHubUsers -and -not $GitHubTeams) {
 $userAdditions = @($GitHubUsers.Split(",") | % { $_.Trim() } | ? { return $_ })
 $teamAdditions = @($GitHubTeams.Split(",") | % { $_.Trim() } | ? { return $_ })
 
-$headers = @{
-  Authorization = "bearer $AuthToken"
-}
-$uri = "https://api.github.com/repos/$RepoOwner/$RepoName/pulls/$PRNumber/requested_reviewers"
+$errorsOccurredAddingUsers = AddMembers -memberName "reviewers" -additionSet $userAdditions
+$errorsOccurredAddingTeams = AddMembers -memberName "team_reviewers" -additionSet $teamAdditions
 
-try {
-  $resp = Invoke-RestMethod -Headers $headers $uri -MaximumRetryCount 3
-}
-catch {
-  Write-Error "Invoke-RestMethod [$uri] failed with exception:`n$_"
+if ($errorsOccurredAddingUsers -or $errorsOccurredAddingTeams) {
   exit 1
 }
-
-# the response object takes this form: https://developer.github.com/v3/pulls/review_requests/#response-1
-# before we can push a new reviewer, we need to pull the simple Ids out of the complex objects that came back in the response
-$userReviewers = @($resp.users | % { return $_.login })
-$teamReviewers = @($resp.teams | % { return $_.slug })
-
-if (!$userReviewers) { $modifiedUserReviewers = @() } else { $modifiedUserReviewers = $userReviewers.Clone() }
-$modifiedUserReviewers += ($userAdditions | ? { !$modifiedUserReviewers.Contains($_) })
-
-if ($teamReviewers) { $modifiedTeamReviewers = @() } else { $modifiedTeamReviewers = $teamReviewers.Clone() }
-$modifiedTeamReviewers += ($teamAdditions | ? { !$modifiedTeamReviewers.Contains($_) })
-
-$detectedUserDiffs = Compare-Object -ReferenceObject $userReviewers -DifferenceObject $modifiedUserReviewers
-$detectedTeamDiffs = Compare-Object -ReferenceObject $teamReviewers -DifferenceObject $modifiedTeamReviewers
-
-# Compare-Object returns values when there is a difference between the comparied objects.
-# we only want to run the update if there IS a difference.
-if ($detectedUserDiffs -or $detectedTeamDiffs) {
-  $postResp = @{}
-
-  if ($modifiedUserReviewers) { $postResp["reviewers"] = $modifiedUserReviewers }
-  if ($modifiedTeamReviewers) { $postResp["team_reviewers"] = $modifiedTeamReviewers }
-
-  $postResp = $postResp | ConvertTo-Json
-
-  try {
-    Write-Host $postResp
-    $resp = Invoke-RestMethod -Method Post -Headers $headers -Body $postResp -Uri $uri -MaximumRetryCount 3
-    $resp | Write-Verbose
-  }
-  catch {
-    Write-Error "Unable to update PR reviewers. `n$_"
-  }
-}
-else {
-  $results = $GitHubUsers + $GitHubTeams
-  Write-Host "Reviewers $results already added. Exiting."
-  exit(0)
-}