Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ext/pymysql: Add Instrumentor #611

Merged
merged 19 commits into from
Apr 29, 2020
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
a27027b
ext/pymysql: Add test requirements section in setup
mauriciovasquezbernal Apr 23, 2020
6ddbeb5
Use TestBase in integration tests
mauriciovasquezbernal Apr 23, 2020
2ab5e5b
ext/dbapi: Implement unwrap connect
mauriciovasquezbernal Apr 23, 2020
3a54847
ext/pymysql: Implement instrumentor interface
mauriciovasquezbernal Apr 23, 2020
7123ebb
update changelog
mauriciovasquezbernal Apr 24, 2020
7ea6b3b
typo
mauriciovasquezbernal Apr 24, 2020
c76ade0
Add test for custom tracer provider
mauriciovasquezbernal Apr 27, 2020
03fa1d5
Merge branch 'master' into mauricio/instrumentor-pymysql
mauriciovasquezbernal Apr 27, 2020
e0be0de
Merge branch 'master' into mauricio/instrumentor-pymysql
mauriciovasquezbernal Apr 27, 2020
4a4c167
fix tests
mauriciovasquezbernal Apr 27, 2020
7a070af
Merge branch 'master' into mauricio/instrumentor-pymysql
mauriciovasquezbernal Apr 27, 2020
2722ebb
update changelog
mauriciovasquezbernal Apr 27, 2020
3430446
Merge branch 'master' into mauricio/instrumentor-pymysql
mauriciovasquezbernal Apr 27, 2020
9f3120e
enable tests
mauriciovasquezbernal Apr 27, 2020
249d87f
ext/dbapi: Add tests for wrap_connect and unwrap_connect
mauriciovasquezbernal Apr 24, 2020
3625a85
Update ext/opentelemetry-ext-dbapi/src/opentelemetry/ext/dbapi/__init…
mauriciovasquezbernal Apr 28, 2020
7712252
further changes
mauriciovasquezbernal Apr 28, 2020
00e29a3
update class name in entrypoint
mauriciovasquezbernal Apr 28, 2020
bede60b
Merge branch 'master' into mauricio/instrumentor-pymysql
c24t Apr 29, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,15 @@ def wrap_connect_(
logger.warning("Failed to integrate with DB API. %s", str(ex))


def unwrap_connect(
connect_module: typing.Callable[..., any], connect_method_name: str,
):
if hasattr(connect_module, connect_method_name):
conn = getattr(connect_module, connect_method_name)
if isinstance(conn, wrapt.ObjectProxy):
setattr(connect_module, connect_method_name, conn.__wrapped__)


class DatabaseApiIntegration:
Copy link
Member

@hectorhdzg hectorhdzg Apr 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to not use BaseInstrumentor in here as well?, people could be using dbapi instrumentor directly

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BaseInstrumentor is used to implement an interface that allows to instrument all the calls made to a given library, for instance pymysql. AFAIU dbapi is a specification and not a library, so having an instrumentor here doesn't make sense. If a person wants to use dpapi directly, they should use the wrap_connect and unwrap_connect functions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should document this? In the docstrings trace_integration() is the recommended way to instrument. Perhaps we should expose an API method to uninstrument (since there isn't really a semantic similarity between trace_integration and unwrap_connect.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PyMySQL docstrings was wrong, trace_integration doesn't exist anymore.
PymysqlInstrumentor offers two methods, instrument and uninstrument.

I'm not sure what to do with the trace_integration on dpapi, it's very similar to wrap_connect but creates a tracer before. Maybe we could remove it and let the user play directly with wrap_connect...

def __init__(
self,
Expand Down Expand Up @@ -184,7 +193,7 @@ def get_connection_attributes(self, connection):
self.name += "." + self.database
user = self.connection_props.get("user")
if user is not None:
self.span_attributes["db.user"] = user
self.span_attributes["db.user"] = str(user)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

User is not returned as string in some cases?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, PyMySQL returns bytes and it was causing problems in the console exporter. I opened an issue about it #623.

host = self.connection_props.get("host")
if host is not None:
self.span_attributes["net.peer.name"] = host
Expand Down
38 changes: 33 additions & 5 deletions ext/opentelemetry-ext-dbapi/tests/test_dbapi_integration.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.

from unittest import mock

from opentelemetry import trace as trace_api
from opentelemetry.ext.dbapi import DatabaseApiIntegration
from opentelemetry.ext import dbapi
from opentelemetry.test.test_base import TestBase


Expand All @@ -35,7 +37,7 @@ def test_span_succeeded(self):
"host": "server_host",
"user": "user",
}
db_integration = DatabaseApiIntegration(
db_integration = dbapi.DatabaseApiIntegration(
self.tracer, "testcomponent", "testtype", connection_attributes
)
mock_connection = db_integration.wrapped_connection(
Expand Down Expand Up @@ -66,7 +68,9 @@ def test_span_succeeded(self):
)

def test_span_failed(self):
db_integration = DatabaseApiIntegration(self.tracer, "testcomponent")
db_integration = dbapi.DatabaseApiIntegration(
self.tracer, "testcomponent"
)
mock_connection = db_integration.wrapped_connection(
mock_connect, {}, {}
)
Expand All @@ -85,7 +89,9 @@ def test_span_failed(self):
self.assertEqual(span.status.description, "Test Exception")

def test_executemany(self):
db_integration = DatabaseApiIntegration(self.tracer, "testcomponent")
db_integration = dbapi.DatabaseApiIntegration(
self.tracer, "testcomponent"
)
mock_connection = db_integration.wrapped_connection(
mock_connect, {}, {}
)
Expand All @@ -97,7 +103,9 @@ def test_executemany(self):
self.assertEqual(span.attributes["db.statement"], "Test query")

def test_callproc(self):
db_integration = DatabaseApiIntegration(self.tracer, "testcomponent")
db_integration = dbapi.DatabaseApiIntegration(
self.tracer, "testcomponent"
)
mock_connection = db_integration.wrapped_connection(
mock_connect, {}, {}
)
Expand All @@ -110,6 +118,26 @@ def test_callproc(self):
span.attributes["db.statement"], "Test stored procedure"
)

@mock.patch("opentelemetry.ext.dbapi")
def test_wrap_connect(self, mock_dbapi):
dbapi.wrap_connect(self.tracer, mock_dbapi, "connect", "-")
connection = mock_dbapi.connect()
self.assertEqual(mock_dbapi.connect.call_count, 1)
self.assertIsInstance(connection, dbapi.TracedConnectionProxy)
self.assertIsInstance(connection.__wrapped__, mock.Mock)

@mock.patch("opentelemetry.ext.dbapi")
def test_unwrap_connect(self, mock_dbapi):
dbapi.wrap_connect(self.tracer, mock_dbapi, "connect", "-")
connection = mock_dbapi.connect()
self.assertEqual(mock_dbapi.connect.call_count, 1)
self.assertIsInstance(connection, dbapi.TracedConnectionProxy)

dbapi.unwrap_connect(mock_dbapi, "connect")
connection = mock_dbapi.connect()
self.assertEqual(mock_dbapi.connect.call_count, 2)
self.assertIsInstance(connection, mock.Mock)


# pylint: disable=unused-argument
def mock_connect(*args, **kwargs):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,12 @@
# limitations under the License.

import os
import time
import unittest

import pymysql as pymy

from opentelemetry import trace as trace_api
from opentelemetry.ext.pymysql import trace_integration
from opentelemetry.sdk.trace import Tracer, TracerProvider
from opentelemetry.sdk.trace.export import SimpleExportSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import (
InMemorySpanExporter,
)
from opentelemetry.ext.pymysql import PymysqlInstrumentor
from opentelemetry.test.test_base import TestBase

MYSQL_USER = os.getenv("MYSQL_USER ", "testuser")
MYSQL_PASSWORD = os.getenv("MYSQL_PASSWORD ", "testpassword")
Expand All @@ -33,17 +27,14 @@
MYSQL_DB_NAME = os.getenv("MYSQL_DB_NAME ", "opentelemetry-tests")


class TestFunctionalPyMysql(unittest.TestCase):
class TestFunctionalPyMysql(TestBase):
@classmethod
def setUpClass(cls):
super().setUpClass()
cls._connection = None
cls._cursor = None
cls._tracer_provider = TracerProvider()
cls._tracer = Tracer(cls._tracer_provider, None)
cls._span_exporter = InMemorySpanExporter()
cls._span_processor = SimpleExportSpanProcessor(cls._span_exporter)
cls._tracer_provider.add_span_processor(cls._span_processor)
trace_integration(cls._tracer_provider)
cls._tracer = cls.tracer_provider.get_tracer(__name__)
PymysqlInstrumentor().instrument()
cls._connection = pymy.connect(
user=MYSQL_USER,
password=MYSQL_PASSWORD,
Expand All @@ -58,11 +49,8 @@ def tearDownClass(cls):
if cls._connection:
cls._connection.close()

def setUp(self):
self._span_exporter.clear()

def validate_spans(self):
spans = self._span_exporter.get_finished_spans()
spans = self.memory_exporter.get_finished_spans()
self.assertEqual(len(spans), 2)
for span in spans:
if span.name == "rootSpan":
Expand Down
2 changes: 2 additions & 0 deletions ext/opentelemetry-ext-pymysql/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# Changelog

## Unreleased

- Implement instrumentor interface ([#611](https://github.com/open-telemetry/opentelemetry-python/pull/611))
9 changes: 9 additions & 0 deletions ext/opentelemetry-ext-pymysql/setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,16 @@ packages=find_namespace:
install_requires =
opentelemetry-api == 0.7.dev0
opentelemetry-ext-dbapi == 0.7.dev0
opentelemetry-auto-instrumentation == 0.7.dev0
PyMySQL ~= 0.9.3

[options.extras_require]
test =
opentelemetry-test == 0.7.dev0

[options.packages.find]
where = src

[options.entry_points]
opentelemetry_instrumentor =
pymysql = opentelemetry.ext.pymysql:PymysqlInstrumentor
Original file line number Diff line number Diff line change
Expand Up @@ -45,24 +45,30 @@

import pymysql

from opentelemetry.ext.dbapi import wrap_connect
from opentelemetry.auto_instrumentation.instrumentor import BaseInstrumentor
from opentelemetry.ext.dbapi import unwrap_connect, wrap_connect
from opentelemetry.ext.pymysql.version import __version__
from opentelemetry.trace import TracerProvider, get_tracer


def trace_integration(tracer_provider: typing.Optional[TracerProvider] = None):
"""Integrate with the PyMySQL library.
https://github.com/PyMySQL/PyMySQL/
"""
class PymysqlInstrumentor(BaseInstrumentor):
def _instrument(self, **kwargs):
"""Integrate with the PyMySQL library.
https://github.com/PyMySQL/PyMySQL/
"""
tracer_provider = kwargs.get("tracer_provider")

tracer = get_tracer(__name__, __version__, tracer_provider)
tracer = get_tracer(__name__, __version__, tracer_provider)

connection_attributes = {
"database": "db",
"port": "port",
"host": "host",
"user": "user",
}
wrap_connect(
tracer, pymysql, "connect", "mysql", "sql", connection_attributes
)
connection_attributes = {
"database": "db",
"port": "port",
"host": "host",
"user": "user",
}
wrap_connect(
tracer, pymysql, "connect", "mysql", "sql", connection_attributes
)

def _uninstrument(self, **kwargs):
unwrap_connect(pymysql, "connect")
55 changes: 47 additions & 8 deletions ext/opentelemetry-ext-pymysql/tests/test_pymysql_integration.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,22 +17,61 @@
import pymysql

import opentelemetry.ext.pymysql
from opentelemetry.ext.pymysql import trace_integration
from opentelemetry.ext.pymysql import PymysqlInstrumentor
from opentelemetry.sdk import resources
from opentelemetry.test.test_base import TestBase


class TestPyMysqlIntegration(TestBase):
def test_trace_integration(self):
with mock.patch("pymysql.connect"):
trace_integration()
cnx = pymysql.connect(database="test")
cursor = cnx.cursor()
query = "SELECT * FROM test"
cursor.execute(query)
def tearDown(self):
super().tearDown()
# ensure that it's uninstrumented if some of the tests fail
PymysqlInstrumentor().uninstrument()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please enclose this between logging.disable calls to suppress unnecessary logging:

from logging import disable, WARNING, NOTSET

...

    def tearDown(self):
        ...
        disable(WARNING)
        PymysqlInstrumentor().uninstrument()
        disable(NOTSET)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I followed a similar approach.


@mock.patch("pymysql.connect")
# pylint: disable=unused-argument
def test_instrumentor(self, mock_connect):
PymysqlInstrumentor().instrument()

cnx = pymysql.connect(database="test")
cursor = cnx.cursor()
query = "SELECT * FROM test"
cursor.execute(query)

spans_list = self.memory_exporter.get_finished_spans()
self.assertEqual(len(spans_list), 1)
span = spans_list[0]

# Check version and name in span's instrumentation info
self.check_span_instrumentation_info(span, opentelemetry.ext.pymysql)

# check that no spans are generated after uninstrument
PymysqlInstrumentor().uninstrument()

cnx = pymysql.connect(database="test")
cursor = cnx.cursor()
query = "SELECT * FROM test"
cursor.execute(query)

spans_list = self.memory_exporter.get_finished_spans()
self.assertEqual(len(spans_list), 1)

@mock.patch("pymysql.connect")
# pylint: disable=unused-argument
def test_custom_tracer_provider(self, mock_connect):
resource = resources.Resource.create({})
result = self.create_tracer_provider(resource=resource)
tracer_provider, exporter = result

PymysqlInstrumentor().instrument(tracer_provider=tracer_provider)

cnx = pymysql.connect(database="test")
cursor = cnx.cursor()
query = "SELECT * FROM test"
cursor.execute(query)

spans_list = exporter.get_finished_spans()
self.assertEqual(len(spans_list), 1)
span = spans_list[0]

self.assertIs(span.resource, resource)
3 changes: 2 additions & 1 deletion tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -183,8 +183,9 @@ commands_pre =
psycopg2: pip install {toxinidir}/ext/opentelemetry-ext-dbapi
psycopg2: pip install {toxinidir}/ext/opentelemetry-ext-psycopg2

pymysql: pip install {toxinidir}/opentelemetry-auto-instrumentation
pymysql: pip install {toxinidir}/ext/opentelemetry-ext-dbapi
pymysql: pip install {toxinidir}/ext/opentelemetry-ext-pymysql
pymysql: pip install {toxinidir}/ext/opentelemetry-ext-pymysql[test]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the [test] for using TestBase? How does this work?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a way to tell pip to install opentelemetry-ext-pymysql and the dependencies on the extra test section: https://github.com/open-telemetry/opentelemetry-python/pull/611/files#diff-3874eac15d9a093a3db07c9283d97d12R48


redis: pip install {toxinidir}/opentelemetry-auto-instrumentation
redis: pip install {toxinidir}/ext/opentelemetry-ext-redis[test]
Expand Down