Skip to content

Replace jsonpickle with json to serialize entity #275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Mar 22, 2021
Merged

Replace jsonpickle with json to serialize entity #275

merged 7 commits into from
Mar 22, 2021

Conversation

lupengamzn
Copy link
Contributor

Issue #, if available:

Security issue regarding decode API in jsonpickle: jsonpickle/jsonpickle#335
X-Ray Python SDK doesn't use decode API, but better find a replacement

Description of changes:

Replace jsonpickle with json to serialize entity data

Benchmark:

Some initial benchmark results using pytest-benchmark on Python 2.7:

benchmark

Will automate the benchmark across the library shortly

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@codecov-io
Copy link

codecov-io commented Mar 17, 2021

Codecov Report

Merging #275 (7abff72) into master (508f929) will increase coverage by 0.17%.
The diff coverage is 94.02%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #275      +/-   ##
==========================================
+ Coverage   79.34%   79.52%   +0.17%     
==========================================
  Files          82       83       +1     
  Lines        3249     3277      +28     
==========================================
+ Hits         2578     2606      +28     
  Misses        671      671              
Impacted Files Coverage Δ
aws_xray_sdk/core/utils/conversion.py 85.71% <85.71%> (ø)
aws_xray_sdk/core/models/entity.py 92.85% <100.00%> (+0.23%) ⬆️
aws_xray_sdk/core/models/segment.py 97.14% <100.00%> (-0.12%) ⬇️
aws_xray_sdk/core/models/subsegment.py 96.82% <100.00%> (-0.15%) ⬇️
aws_xray_sdk/core/models/throwable.py 89.13% <100.00%> (+9.13%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 508f929...7abff72. Read the comment docs.

Copy link
Contributor

@srprash srprash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Some minor comments.

@@ -18,8 +18,8 @@ deps =
requests
bottle >= 0.10
flask >= 0.10
sqlalchemy
Flask-SQLAlchemy
sqlalchemy==1.3.*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should only pin for py 3.4 and 3.5 and should still test with the latest version of sqlalchemy and flask-sqlalchemy for the supported python. Ideally, the SDK should work with every sqlalchemy version supported for each python version.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other py SDKs will fail the test with the latest sqlalchemy and flask-sqlalchemy version.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does the X-Ray SDK tests fail for the latest version of sqlalchemy? we are not using any of the internal APIs of sqlalchemy in our tests right? Does that mean the SDK is not compatible with latest sqlalchemy for supported python versions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of the unit tests are using the API(for example), which will fail the test for the latest sqlalchemy and flask-sqlalchemy. Not sure if the Python SDK is compatible with the latest sqlalchemy or not, but at least some unit tests need to update the API they use from sqlalchemy and flask-sqlalchemy.

Copy link
Contributor

@srprash srprash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@bhautikpip bhautikpip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also add performance results with this PR? This would help customers to understand the performance improvement in next release because of this serialization logic change.


for key, value in vars(self).items():
if isinstance(value, bool) or value:
if key == 'subsegments':
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we handle the case if key is annotations, aws or http ? I see that you're handling metadata case separately. Shouldn't we do the same with annotations too? since in that case also we don't know how many key-value pairs would be there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

annotations, aws or http are all dict with serializable values like string, int or boolean, so there is no need to handle these cases.

@lupengamzn
Copy link
Contributor Author

Should we also add performance results with this PR? This would help customers to understand the performance improvement in next release because of this serialization logic change.

We can do a follow-up PR to address this in future as seems like currently the pytest-benchmark library doesn't support all of the Python platforms supported by X-Ray Python SDK. So will need some workaround.

@lupengamzn lupengamzn merged commit 266bc82 into aws:master Mar 22, 2021
BastianZim added a commit to regro-cf-autotick-bot/aws-xray-sdk-feedstock that referenced this pull request Mar 24, 2021
archlinux-github pushed a commit to archlinux/svntogit-community that referenced this pull request Mar 26, 2021
Removal of jsonpickle: aws/aws-xray-sdk-python#275


git-svn-id: file:///srv/repos/svn-community/svn@904452 9fca08f4-af9d-4005-b8df-a31f2cc04f65
archlinux-github pushed a commit to archlinux/svntogit-community that referenced this pull request Mar 26, 2021
Removal of jsonpickle: aws/aws-xray-sdk-python#275

git-svn-id: file:///srv/repos/svn-community/svn@904452 9fca08f4-af9d-4005-b8df-a31f2cc04f65
entity_dict[key] = subsegments
elif key == 'cause':
entity_dict[key] = {}
entity_dict[key]['working_directory'] = self.cause['working_directory']

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lupengamzn

I couldn't dive into this implementation yet to provide a fix, but this is causing a regression in the AWS Lambda Powertools Tracer feature leading to a TypeError: string must be a list of integers

aws-powertools/powertools-lambda-python#383

Hargrav3s pushed a commit to Gavant/aws-xray-sdk-python that referenced this pull request Mar 22, 2022
* Replace jsonpickle with json to serialize entity

* Added workflow to create release tag

* Pinned sqlalchemy and Flask-SQLAlchemy for unit test

* Fixed version

* Changed log to debug level

* Update logging

* Added empty line
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants