Merge redis changes mlops (#914)

* Exclude command line functionality from test coverage (#855)

* FIX: resilient environment settings (#825)

if the application uses generalimport to manage optional depedencies,
it's possible that generalimport.MissingOptionalDependency is raised.
In this case, we should not report the module as it is not actually loaded and
is not a runtime dependency of the application.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <[email protected]>

* Replace drop_transaction logic by using transaction context manager (#832)

* Replace drop_transaction call

* [Mega-Linter] Apply linters fixes

* Empty commit to start tests

* Change logic in BG Wrappers

---------

Co-authored-by: lrafeei <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Upgrade to Pypy38 for TypedDict (#861)

* Fix base branch

* Revert tox dependencies

* Replace all pypy37 with pypy38

* Remove action.yml file

* Push Empty Commit

* Fix skip_missing_interpreters behavior

* Fix skip_missing_interpreters behavior

* Pin dev CI image sha

* Remove unsupported Tornado tests

* Add latest tests to Tornado

* Remove pypy38 (for now)

---------

Co-authored-by: Tim Pansino <[email protected]>

* Add profile_trace testing (#858)

* Include isort stdlibs for determining stdlib modules

* Use isort & sys to eliminate std & builtin modules

Previously, the logic would fail to identify third party modules installed within the
local user socpe. This fixes that issue by skipping builtin and stdlib modules by name,
instead of attempting to identify third party modules based on file paths.

* Handle importlib_metadata.version being a callable

* Add isort into third party notices

* [Mega-Linter] Apply linters fixes

* Remove Python 2.7 and pypy2 testing (#835)

* Change setup-python to @v2 for py2.7

* Remove py27 and pypy testing

* Fix syntax errors

* Fix comma related syntax errors

* Fix more issues in tox

* Remove gearman test

* Containerized CI Pipeline (#836)

* Revert "Remove Python 2.7 and pypy2 testing (#835)"

This reverts commit abb6405.

* Containerize CI process

* Publish new docker container for CI images

* Rename github actions job

* Copyright tag scripts

* Drop debug line

* Swap to new CI image

* Move pip install to just main python

* Remove libcurl special case from tox

* Install special case packages into main image

* Remove unused packages

* Remove all other triggers besides manual

* Add make run command

* Cleanup small bugs

* Fix CI Image Tagging (#838)

* Correct templated CI image name

* Pin pypy2.7 in image

* Fix up scripting

* Temporarily Restore Old CI Pipeline (#841)

* Restore old pipelines

* Remove python 2 from setup-python

* Rework CI Pipeline (#839)

Change pypy to pypy27 in tox.

Fix checkout logic

Pin tox requires

* Fix Tests on New CI (#843)

* Remove non-root user

* Test new CI image

* Change pypy to pypy27 in tox.

* Fix checkout logic

* Fetch git tags properly

* Pin tox requires

* Adjust default db settings for github actions

* Rename elasticsearch services

* Reset to new pipelines

* [Mega-Linter] Apply linters fixes

* Fix timezone

* Fix docker networking

* Pin dev image to new sha

* Standardize gearman DB settings

* Fix elasticsearch settings bug

* Fix gearman bug

* Add missing odbc headers

* Add more debug messages

* Swap out dev ci image

* Fix required virtualenv version

* Swap out dev ci image

* Swap out dev ci image

* Remove aioredis v1 for EOL

* Add coverage paths for docker container

* Unpin ci container

---------

Co-authored-by: TimPansino <[email protected]>

* Trigger tests

* Add testing for profile trace.

* [Mega-Linter] Apply linters fixes

* Ignore __call__ from coverage on profile_trace.

* [Mega-Linter] Apply linters fixes

---------

Co-authored-by: Hannah Stepanek <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: hmstepanek <[email protected]>
Co-authored-by: Lalleh Rafeei <[email protected]>
Co-authored-by: Timothy Pansino <[email protected]>
Co-authored-by: TimPansino <[email protected]>
Co-authored-by: umaannamalai <[email protected]>

* Add Transaction API Tests (#857)

* Test for suppress_apdex_metric

* Add custom_metrics tests

* Add distributed_trace_headers testing in existing tests

* [Mega-Linter] Apply linters fixes

* Remove redundant if-statement

* Ignore deprecated transaction function from coverage

* [Mega-Linter] Apply linters fixes

* Push empty commit

* Update newrelic/api/transaction.py

---------

Co-authored-by: lrafeei <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Timothy Pansino <[email protected]>
Co-authored-by: Uma Annamalai <[email protected]>

* Add tests for jinja2. (#842)

* Add tests for jinja2.

* [Mega-Linter] Apply linters fixes

* Update tox.ini

Co-authored-by: Timothy Pansino <[email protected]>

---------

Co-authored-by: umaannamalai <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Timothy Pansino <[email protected]>

* Add tests for newrelic/config.py (#860)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Initial commit of redis changes from main

* Fix merging

* Spell framework correctly

* Fix redis merge issues

---------

Co-authored-by: Ahmed Helil <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <[email protected]>
Co-authored-by: lrafeei <[email protected]>
Co-authored-by: Tim Pansino <[email protected]>
Co-authored-by: Uma Annamalai <[email protected]>
Co-authored-by: hmstepanek <[email protected]>
Co-authored-by: Timothy Pansino <[email protected]>
Co-authored-by: TimPansino <[email protected]>
Co-authored-by: umaannamalai <[email protected]>

Author: 6 people

codecov.yml

@@ -1,23 +1,25 @@
 ignore:
-  - "newrelic/packages/**/*"
-  - "newrelic/packages/*"
   - "newreilc/hooks/component_sentry.py"
-  - "newrelic/hooks/adapter_meinheld.py"
+  - "newrelic/admin/*"
+  - "newrelic/console.py"
   - "newrelic/hooks/adapter_flup.py"
+  - "newrelic/hooks/adapter_meinheld.py"
   - "newrelic/hooks/adapter_paste.py"
   - "newrelic/hooks/component_piston.py"
+  - "newrelic/hooks/database_oursql.py"
+  - "newrelic/hooks/database_psycopg2ct.py"
+  - "newrelic/hooks/datastore_aioredis.py"
+  - "newrelic/hooks/datastore_aredis.py"
+  - "newrelic/hooks/datastore_motor.py"
   - "newrelic/hooks/datastore_pyelasticsearch.py"
-  - "newrelic/hooks/external_pywapi.py"
+  - "newrelic/hooks/datastore_umemcache.py"
   - "newrelic/hooks/external_dropbox.py"
   - "newrelic/hooks/external_facepy.py"
+  - "newrelic/hooks/external_pywapi.py"
   - "newrelic/hooks/external_xmlrpclib.py"
   - "newrelic/hooks/framework_pylons.py"
   - "newrelic/hooks/framework_web2py.py"
-  - "newrelic/hooks/middleware_weberror.py"
   - "newrelic/hooks/framework_webpy.py"
-  - "newrelic/hooks/datastore_motor.py"
-  - "newrelic/hooks/database_oursql.py"
-  - "newrelic/hooks/database_psycopg2ct.py"
-  - "newrelic/hooks/datastore_umemcache.py"
-  - "newrelic/admin/*"
-  - "newrelic/console.py"
+  - "newrelic/hooks/middleware_weberror.py"
+  - "newrelic/packages/*"
+  - "newrelic/packages/**/*"

newrelic/config.py

@@ -2690,12 +2690,14 @@ def _process_module_builtin_defaults():
         "aioredis.connection", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_connection"
     )
 
+    # Redis v4.2+
     _process_module_definition(
-        "redis.asyncio.client", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_client"
+        "redis.asyncio.client", "newrelic.hooks.datastore_redis", "instrument_asyncio_redis_client"
     )
 
+    # Redis v4.2+
     _process_module_definition(
-        "redis.asyncio.commands", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_client"
+        "redis.asyncio.commands", "newrelic.hooks.datastore_redis", "instrument_asyncio_redis_client"
     )
 
     _process_module_definition(

newrelic/hooks/datastore_aioredis.py

@@ -11,7 +11,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
 from newrelic.api.datastore_trace import DatastoreTrace
 from newrelic.api.time_trace import current_trace
 from newrelic.api.transaction import current_transaction
@@ -60,7 +59,13 @@ def _nr_wrapper_AioRedis_method_(wrapped, instance, args, kwargs):
         # Method will return synchronously without executing,
         # it will be added to the command stack and run later.
         aioredis_version = get_package_version_tuple("aioredis")
-        if aioredis_version and aioredis_version < (2,):
+
+        # This conditional is for versions of aioredis that are outside
+        # New Relic's supportability window but will still work.  New
+        # Relic does not provide testing/support for this.  In order to
+        # keep functionality without affecting coverage metrics, this
+        # segment is excluded from coverage analysis.
+        if aioredis_version and aioredis_version < (2,):  # pragma: no cover
             # AioRedis v1 uses a RedisBuffer instead of a real connection for queueing up pipeline commands
             from aioredis.commands.transaction import _RedisBuffer
 
@@ -72,8 +77,6 @@ def _nr_wrapper_AioRedis_method_(wrapped, instance, args, kwargs):
             # AioRedis v2 uses a Pipeline object for a client and internally queues up pipeline commands
             if aioredis_version:
                 from aioredis.client import Pipeline
-            else:
-                from redis.asyncio.client import Pipeline
             if isinstance(instance, Pipeline):
                 return wrapped(*args, **kwargs)
 
@@ -137,7 +140,12 @@ async def wrap_Connection_send_command(wrapped, instance, args, kwargs):
         return await wrapped(*args, **kwargs)
 
 
-def wrap_RedisConnection_execute(wrapped, instance, args, kwargs):
+# This wrapper is for versions of aioredis that are outside
+# New Relic's supportability window but will still work.  New
+# Relic does not provide testing/support for this.  In order to
+# keep functionality without affecting coverage metrics, this
+# segment is excluded from coverage analysis.
+def wrap_RedisConnection_execute(wrapped, instance, args, kwargs):  # pragma: no cover
     # RedisConnection in aioredis v1 returns a future instead of using coroutines
     transaction = current_transaction()
     if not transaction:
@@ -205,6 +213,11 @@ def instrument_aioredis_connection(module):
         if hasattr(module.Connection, "send_command"):
             wrap_function_wrapper(module, "Connection.send_command", wrap_Connection_send_command)
 
-    if hasattr(module, "RedisConnection"):
+    # This conditional is for versions of aioredis that are outside
+    # New Relic's supportability window but will still work.  New
+    # Relic does not provide testing/support for this.  In order to
+    # keep functionality without affecting coverage metrics, this
+    # segment is excluded from coverage analysis.
+    if hasattr(module, "RedisConnection"):  # pragma: no cover
         if hasattr(module.RedisConnection, "execute"):
             wrap_function_wrapper(module, "RedisConnection.execute", wrap_RedisConnection_execute)