[Cosmos] Infra: use Bicep for resource provisioning (#24198)

* initial commit

* Client Constructor (#20310)

* Removed some stuff

* Looking at constructors

* Updated request

* Added client close

* working client creation

Co-authored-by: simorenoh <[email protected]>

* read database

database read works, but ignored exception is returned:
Fatal error on SSL transport
NoneType has no attribute 'send' (_loop._proactor.send)
RuntimeError: Event loop is closed
Unclosed connector/ connection

* Update simon_testfile.py

* with coroutine

Added methods needed to use async with when initializing client, but logs output "Exception ignored... Runtime Error: Event loop is closed"

* Update simon_testfile.py

* small changes

* async with returns no exceptions

* async read container

* async item read

* cleaning up

* create item/ database methods

* item delete working

* docs replace functionality

missing upsert and other resources

* upsert functionality

missing read_all_items and both query methods for container class

* missing query methods

* CRUD for udf, sproc, triggers

* initial query logic + container methods

* missing some execution logic and tests

* oops

* fully working queries

* small fix to query_items()

also fixed README and added examples_async

* Update _cosmos_client_connection_async.py

* Update _cosmos_client_connection.py

* documentation update

* updated MIT dates and get_user_client() description

* Update CHANGELOG.md

* Delete simon_testfile.py

* leftover retry utility

* Update README.md

* docs and removed six package

* changes based on comments

still missing discussion resolution on SSL verification and tests for async functionality under test module (apart from samples which are basically end to end tests)

* small change in type hints

* updated readme

* fixes based on conversations

* added missing type comments

* update changelog for ci pipeline

* added typehints, moved params into keywords, added decorators, made _connection_policy private

* changes based on sync with central sdk

* remove is_system_key from scripts (only used in execute_sproc)

is_system_key verifies that an empty partition key is properly dealt with if ['partitionKey']['systemKey'] exists in the container options - however, we do not allow containers to be created with empty partition key values in the python sdk, so the functionality is needless

* Revert "remove is_system_key from scripts (only used in execute_sproc)"

Reverting last commit, will find way to init is_system_key for now

* async script proxy using composition

* pylint

* capitalized constants

* Apply suggestions from code review

Clarifying comments for README

Co-authored-by: Gahl Levy <[email protected]>

* closing python code snippet

* last doc updates

* Update sdk/cosmos/azure-cosmos/CHANGELOG.md

Co-authored-by: Simon Moreno <[email protected]>

* version update

* cosmos updates for release

* fix connection string comma

* Update CHANGELOG.md

* fixing extra await keyword in sample

* Update CHANGELOG.md

* Update CHANGELOG.md

* using bicep file to provision test resources

* Update tests.yml

Co-authored-by: annatisch <[email protected]>
Co-authored-by: Gahl Levy <[email protected]>
Co-authored-by: Travis Prescott <[email protected]>

Author: 4 people

sdk/cosmos/test-resources.bicep

@@ -0,0 +1,66 @@
+param baseName string
+
+@description('Flag to enable or disable multiple write locations on CosmosDB Account')
+param enableMultipleWriteLocations bool
+
+@description('Default Cosmosdb Account level consistency')
+param defaultConsistencyLevel string = 'Session'
+
+@description('Enable multiple regions, default value is false')
+param enableMultipleRegions bool = false
+
+@description('Location for the Cosmos DB account.')
+param location string = resourceGroup().location
+
+@description('The api version to be used by Bicep to create resources')
+param apiVersion string = '2020-04-01'
+
+var accountName = toLower(baseName)
+var resourceId = cosmosAccount.id
+var singleRegionConfiguration = [
+  {
+    locationName: 'East US 2'
+    provisioningState: 'Succeeded'
+    failoverPriority: 0
+    isZoneRedundant: false
+  }
+]
+var multiRegionConfiguration = [
+  {
+    locationName: 'East US 2'
+    provisioningState: 'Succeeded'
+    failoverPriority: 0
+    isZoneRedundant: false
+  }
+  {
+    locationName: 'East US'
+    provisioningState: 'Succeeded'
+    failoverPriority: 1
+    isZoneRedundant: false
+  }
+]
+var locationsConfiguration = (enableMultipleRegions ? multiRegionConfiguration : singleRegionConfiguration)
+
+resource cosmosAccount 'Microsoft.DocumentDB/databaseAccounts@2020-04-01' = {
+  name: toLower(accountName)
+  location: location
+  kind: 'GlobalDocumentDB'
+  properties: {
+    publicNetworkAccess: 'Enabled'
+    enableAutomaticFailover: false
+    enableMultipleWriteLocations: enableMultipleWriteLocations
+    isVirtualNetworkFilterEnabled: false
+    disableKeyBasedMetadataWriteAccess: false
+    enableFreeTier: false
+    enableAnalyticalStorage: false
+    databaseAccountOfferType: 'Standard'
+    consistencyPolicy: {
+      defaultConsistencyLevel: defaultConsistencyLevel
+    }
+    locations: locationsConfiguration
+  }
+}
+
+
+output ACCOUNT_HOST string = reference(resourceId, apiVersion).documentEndpoint
+output ACCOUNT_KEY string = listKeys(resourceId, apiVersion).primaryMasterKey

sdk/cosmos/tests.yml

@@ -3,29 +3,12 @@ trigger: none
 stages:
   - template: ../../eng/pipelines/templates/stages/archetype-sdk-tests.yml
     parameters:
+      CloudConfig:
+        Cosmos_Public:
+          SubscriptionConfigurations:
+            - $(sub-config-azure-cloud-test-resources)
+            - $(sub-config-cosmos-azure-cloud-test-resources)
+      Clouds: 'Cosmos_Public'
       MaxParallel: 1
+      BuildTargetingString: azure-cosmos
       ServiceDirectory: cosmos
-      TestMarkArgument: not globaldb
-      EnvVars:
-        ACCOUNT_KEY: $(python-cosmos-live-account-key)
-        ACCOUNT_HOST: $(python-cosmos-live-account-host)
-
-      # Workaround to ensure deletion of databases from a statically defined
-      # CosmosDB account.
-      PreSteps:
-        - pwsh: |
-            $testDatabaseId = "Python Test Database " + (New-Guid).ToString()
-            Write-Host Test Database ID: $testDatabaseId
-            Write-Host "##vso[task.setvariable variable=COSMOS_TEST_DATABASE_ID]$testDatabaseId"
-          displayName: Set COSMOS_TEST_DATABASE_ID
-      PostSteps:
-        - pwsh: >
-            az cosmosdb sql database delete
-            --db-name $env:COSMOS_TEST_DATABASE_ID
-            --url-connection $(python-cosmos-live-account-host)
-            --key $(python-cosmos-live-account-key)
-          ignoreLASTEXITCODE: true
-          # Always clean up the database (works when a build is canceled)
-          condition: always()
-          displayName: Cleanup test database
-