gigantic re-organization of tests.

* /testing/resources have been moved to their respective samples.
* /testing/utils.py has been broken up into several files, many of which will be moved to gcp-python-repo-tools.
* Managed VMs tests have been re-written and enabled.
* /scripts/check_requirements.py and /scripts/update_requirements.txt have been removed in favor of gcp-python-repo tools.
* /lib has been removed, the tox environment for gae will just create it when needed.
* Switched to using manual encryption and a secrets.tar file so that we can store both the service account and the environment variables needed.
* Moved the testing section in CONTRIBUTING.md to its own file.
* Changed the way that environment variables are used in tests. See testing.cloud.config.

Author: Jon Wayne Parrott

.coveragerc

@@ -8,6 +8,7 @@ include =
     monitoring/*
     storage/*
     cloud_logging/*
+    managed_vms/*
 [report]
 exclude_lines =
     pragma: NO COVER

.gitignore

@@ -5,8 +5,14 @@ coverage-gae.json
 .tox
 coverage.xml
 nosetests.xml
+nosetests-*.xml
 python-docs-samples.json
+service-account.json
 __pycache__
 *db\.sqlite3
 managed_vms/django_tutorial/static/*
 **/migrations/*
+lib
+testing/resources/test-env.sh
+testing/resources/service-account.json
+secrets.tar

.travis.yml

@@ -1,35 +1,27 @@
 sudo: false
-#add language, etc. here
 language: python
-
+services:
+  - memcached
+  - mysql
 branches:
   only:
-    - master
-
+  - master
 cache:
   directories:
   - $HOME/.cache
-
 env:
   global:
   - PATH=${PATH}:${HOME}/gcloud/google-cloud-sdk/bin
-  - GOOGLE_APPLICATION_CREDENTIALS=${TRAVIS_BUILD_DIR}/python-docs-samples.json
+  - GOOGLE_APPLICATION_CREDENTIALS=${TRAVIS_BUILD_DIR}/testing/resources/service-account.json
   - GAE_PYTHONPATH=${HOME}/.cache/google_appengine
-  - TEST_BUCKET_NAME=bigquery-devrel-samples-bucket
-  - TEST_PROJECT_ID=bigquery-devrel-samples
-
+  - secure: YIowCOMJ97rTcehKVT6Gi3u0Etm8s9+TBRGsNPJLgSF2zZdsh9IHcIc+tMDUMR3lpOe8y2a060RuODQcRsW1W1LIHej+ZE/gv6vATT6qNA3eKfKmZ9AyrpBO0fTOHlHrGBuU9ktBPR+iqvnq8MLWjnUozPFMJbuNBFITU7JP8jc=
 before_install:
-- tests/scripts/travis-before-install.sh
-
+- openssl aes-256-cbc -k "$secrets_password" -in secrets.tar.enc -out secrets.tar -d
+- tar xvf secrets.tar
 install:
-# TODO: unpin tox when the following issue is fixed
-# https://bitbucket.org/hpk42/tox/issues/285/tox-220-breaks-some-toxini-config-files
 - pip install tox
-
+- pip install -e git+https://github.com/GoogleCloudPlatform/python-repo-tools#egg=python-repo-tools
+- gcp-python-repo-tools download-appengine-sdk `dirname "${GAE_PYTHONPATH}"`
 script:
+- source ${TRAVIS_BUILD_DIR}/testing/resources/test-env.sh
 - tox
-
-# Coveralls is disabled in travis because travis does not run all of our
-# tests. The full suite of tests is run on an internal Jenkins instance.
-# after_success:
-# - coveralls

CONTRIBUTING.md

@@ -36,102 +36,4 @@ be able to accept your pull requests.
 
 ## Testing
 
-The tests in this repository run against live services, therefore, it
-takes a bit of configuration to run all of the tests locally.
-
-### Local setup
-
-Before you can run tests locally you must have:
-
-* The latest [tox](https://tox.readthedocs.org/en/latest/) and
-  [pip](https://pypi.python.org/pypi/pip) installed.
-
-        $ sudo pip install --upgrade tox pip
-
-* The [Google Cloud SDK](https://cloud.google.com/sdk/) installed. You
-  can do so with the following command:
-
-        $ curl https://sdk.cloud.google.com | bash
-
-* Most tests require you to have an active, billing-enabled project on
-  the
-  [Google Developers Console](https://console.developers.google.com).
-
-* You will need a set of
-  [Service Account Credentials](https://console.developers.google.com/project/_/apiui/credential)
-  for your project in ``json`` form.
-
-* Set the environment variables appropriately for your project.
-
-        $ export GOOGLE_APPLICATION_CREDENTIALS=your-service-account-json-file
-        $ export TEST_PROJECT_ID=your-project-id
-        $ export TEST_BUCKET_NAME=your-bucket-name
-
-If you want to run the Google App Engine tests, you will need:
-
-* The App Engine Python SDK. You can install this by downloading it [here]
-(https://cloud.google.com/appengine/downloads?hl=en)
-
-* You can also download it programatically with the
-  tests/scripts/fetch_gae_sdk.py
-
-        $ test/scripts/fetch_gae_sdk.py <dest dir>
-
-* You will need to set an additional environment variable:
-
-        $ export GAE_PYTHONPATH=<path your AppeEngine sdk>
-
-To run the bigquery tests:
-
-* Create a dataset in your project named `test_dataset`.
-
-        gcloud alpha bigquery datasets create test_dataset
-
-* Load sample data into google cloud storage (for import tests):
-
-        gsutil cp tests/resources/data.csv gs://$TEST_BUCKET_NAME/data.csv
-
-* Load the sample data into a table named `test_table` (for export and streaming tests):
-
-        gcloud alpha bigquery import \
-          gs://$TEST_BUCKET_NAME/data.csv \
-          test_dataset/test_table \
-          --schema-file tests/resources/schema.json
-
-### Test environments
-
-We use [tox](https://tox.readthedocs.org/en/latest/) to configure
-multiple python environments:
-
-* ``py27`` contains tests for samples that run in a normal Python 2.7
-  environment. This is (mostly) everything outside of the
-  ``appengine`` directory.
-* ``gae`` contains tests for samples that run only in Google App
-  Engine. This is (mostly) everything in the ``appengine`` directory.
-* ``pep8`` just runs the linter.
-
-To run tests for a particular environment, invoke tox with the ``-e``
-flag:
-
-    tox -e py27
-
-To run one particular test suite or provide additional parameters to
-``nose``, invoke tox like this:
-
-    toxe -e py27 -- storage/tests/test_list_objects.py
-
-*Note*: The ``gae`` environment can't be told to run one particular
- test at this time.
-
-### Adding new tests
-
-There are a handful of common testing utilities are located under
-``tests``, see existing tests for example usage.
-
-When adding a new top-level directory, be sure to edit ``.coveragerc``
-to include it in coveralls.
-
-To add new tests that require Google App Engine, please place them in
-the ``appengine`` directory if possible. If you place them elsewhere,
-you will need to modify ``tox.ini`` to make the environments
-appropriately run or ignore your test.
+See [TESTING.md](TESTING.md).

TESTING.md

@@ -0,0 +1,79 @@
+# Testing
+
+The tests in this repository are system tests and run against live services, therefore, it takes a bit of configuration to run all of the tests locally.
+
+Before you can run tests locally you must have:
+
+* The latest [tox](https://tox.readthedocs.org/en/latest/),
+  [pip](https://pypi.python.org/pypi/pip), and [gcp-python-repo-tools](https://pypi.python.org/pypi/gcp-python-repo-tools) installed.
+
+        $ sudo pip install --upgrade tox pip gcp-python-repo-tools
+
+* The [Google Cloud SDK](https://cloud.google.com/sdk/) installed. You
+  can do so with the following command:
+
+        $ curl https://sdk.cloud.google.com | bash
+
+## Preparing a project for testing
+
+Most tests require you to have an active, billing-enabled project on the
+[Google Cloud Console](https://console.cloud.google.com).
+
+### Creating resources
+
+Some resources need to be created in a project ahead of time before testing. We have a script that can create everything needed:
+
+    gcloud config set project <your-project-id>
+    scripts/prepare-testing-project.sh
+
+The script will also instruct you to follow a URL to enable APIs. You will need to do that.
+
+### Getting a service account key
+
+From the Cloud Console, create a new Service Account and download its json key. Place this file in `testing/resources/service-account.json`.
+
+## Environment variables
+
+* Copy `testing/resources/test-env.tmpl.sh` to `testing/resources/test-env.sh`, and updated it with your configuration.
+* Run `source testing/resources/test-env.sh`.
+* Run `export GOOGLE_APPLICATION_CREDENTIALS=testing/resources/service-account.json`.
+
+If you want to run the Google App Engine tests, you will need:
+
+* The App Engine Python SDK. You can also download it programatically with `gcp-python-repo-tools`:
+
+        $ gcp-python-repo-tools download-appengine-sdk <dest>
+
+* Set the `GAE_PYTHONPATH` variable:
+
+        $ export GAE_PYTHONPATH=<path your AppeEngine sdk>
+
+### Test environments
+
+We use [tox](https://tox.readthedocs.org/en/latest/) to configure
+multiple python environments:
+
+* ``py27`` and ``py34`` contains tests for samples that run in a normal Python 2.7 pr 3.4 environment. This is everything outside of the ``appengine`` directory that isn't slow or flaky.
+* ``py27-all`` and ``py34-all`` runs all tests except for App Engine tests. This can time some time and some tests are flaky.
+* ``gae`` contains tests for samples that run only in Google App Engine. This is (mostly) everything in the ``appengine`` directory.
+* ``pep8`` just runs the linter.
+
+To run tests for a particular environment, invoke tox with the ``-e``
+flag:
+
+    tox -e py27
+
+To run one particular test suite or provide additional parameters to
+``nose``, invoke tox like this:
+
+    toxe -e py27 -- storage/tests/test_list_objects.py
+
+### Adding new tests
+
+When adding a new top-level directory, be sure to edit ``.coveragerc``
+to include it in coveralls.
+
+To add new tests that require Google App Engine, please place them in
+the ``appengine`` directory if possible. If you place them elsewhere,
+you will need to modify ``tox.ini`` to make the environments
+appropriately run or ignore your test.

appengine/app_identity/signing/main_test.py

@@ -12,13 +12,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from tests import AppEngineTestbedCase
+from testing import AppEngineTest
 import webtest
 
 from . import main
 
 
-class TestAppIdentityHandler(AppEngineTestbedCase):
+class TestAppIdentityHandler(AppEngineTest):
     def setUp(self):
         super(TestAppIdentityHandler, self).setUp()
 

appengine/bigquery/main_test.py

@@ -12,23 +12,22 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import os
 import re
 
 from apiclient.http import HttpMock
 import mock
-import tests
+import testing
 import webtest
 
 from . import main
 
 
-class TestAuthSample(tests.AppEngineTestbedCase):
+class TestAuthSample(testing.AppEngineTest):
 
     def setUp(self):
         super(TestAuthSample, self).setUp()
         self.app = webtest.TestApp(main.app)
-        main.PROJECTID = self.project_id
+        main.PROJECTID = self.config.GCLOUD_PROJECT
 
     def test_anonymous_get(self):
         response = self.app.get('/')
@@ -39,7 +38,7 @@ def test_anonymous_get(self):
                                  r'.*accounts.*Login.*')
 
     def test_loggedin_get(self):
-        self.loginUser()
+        self.login_user()
 
         response = self.app.get('/')
 
@@ -49,10 +48,10 @@ def test_loggedin_get(self):
 
     @mock.patch.object(main.decorator, 'has_credentials', return_value=True)
     def test_oauthed_get(self, *args):
-        self.loginUser()
+        self.login_user()
 
         mock_http = HttpMock(
-            os.path.join(self.resource_path, 'datasets-list.json'),
+            self.resource_path('datasets-list.json'),
             {'status': '200'})
 
         with mock.patch.object(main.decorator, 'http', return_value=mock_http):

tests/resources/datasets-list.json renamed to appengine/bigquery/resources/datasets-list.json

@@ -12,20 +12,20 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import tests
+import testing
 import webtest
 
 from . import main
 
 
-class TestBlobstoreSample(tests.AppEngineTestbedCase):
+class TestBlobstoreSample(testing.AppEngineTest):
 
     def setUp(self):
         super(TestBlobstoreSample, self).setUp()
         self.app = webtest.TestApp(main.app)
 
     def test_form(self):
-        self.loginUser()
+        self.login_user()
         response = self.app.get('/')
 
         self.assertTrue('/_ah/upload' in response)

appengine/blobstore/main_test.py

@@ -16,13 +16,13 @@
 import re
 from unittest.case import SkipTest
 
-import tests
+import testing
 import webtest
 
 from . import main
 
 
-class TestMySQLSample(tests.AppEngineTestbedCase):
+class TestMySQLSample(testing.AppEngineTest):
 
     def setUp(self):
         if not os.path.exists('/var/run/mysqld/mysqld.sock'):

appengine/cloudsql/main_test.py

@@ -13,13 +13,13 @@
 # limitations under the License.
 
 import mock
-from tests import AppEngineTestbedCase
+from testing import AppEngineTest
 import webtest
 
 from . import main
 
 
-class TestHandlers(AppEngineTestbedCase):
+class TestHandlers(AppEngineTest):
     def setUp(self):
         super(TestHandlers, self).setUp()
 

appengine/images/main_test.py

@@ -12,8 +12,9 @@
 #    See the License for the specific language governing permissions and
 #    limitations under the License.
 
-import operator
 # [START taskqueue]
+import operator
+import os
 import unittest
 
 from google.appengine.api import taskqueue
@@ -28,7 +29,8 @@ def setUp(self):
 
         # root_path must be set the the location of queue.yaml.
         # Otherwise, only the 'default' queue will be available.
-        self.testbed.init_taskqueue_stub(root_path='tests/resources')
+        self.testbed.init_taskqueue_stub(
+            root_path=os.path.join(os.path.dirname(__file__), 'resources'))
         self.taskqueue_stub = self.testbed.get_stub(
             testbed.TASKQUEUE_SERVICE_NAME)