Failed to send entity to Daemon with Streaming_Threshold=0

[klofton]

Whenever my lambda is triggered, I receive the following errors from XRay:
"[ERROR] 2022-02-08T02:32:12.228Z 564a4269-1437-496c-a824-943fa6bad466 Failed to send entity to Daemon.
Traceback (most recent call last):
File "/var/task/aws_xray_sdk/core/emitters/udp_emitter.py", line 41, in send_entity
self._send_data(message)
File "/var/task/aws_xray_sdk/core/emitters/udp_emitter.py", line 63, in _send_data
self._socket.sendto(data.encode('utf-8'), (self._ip, self._port))
OSError: [Errno 90] Message too long"

The lambda functions as expected.

I've followed similar issues related to "Failed to send entity to X-Ray" for resolution
https://github.com/awslabs/aws-lambda-powertools-python/issues/283
https://github.com/aws/aws-xray-sdk-python/issues/21
https://github.com/aws/aws-xray-sdk-python/issues/21

After setting "xray_recorder.configure(streaming_threshold=0)", I continue to receive the same errors.

Environment

AWS Lambda function runtime: Python 3.8
AWS_XRAY_TRACING_NAME is define
APPNAME_TRACE_XRAY is Y
Trigger APIGATEWAY

CODE*******
def trace_flask(app):
if not os.environ.get("APPNAME_TRACE_XRAY"):
return app

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.ext.flask.middleware import XRayMiddleware

xray_recorder.configure(streaming_threshold=0)

XRayMiddleware(app, xray_recorder)

return app

Any help would be greatly appreciated.

[srprash]

Hi @klofton
Running inside a lambda function, the X-Ray SDK should already have the streaming threshold set to 0. This means that the SDK will not batch subsegments to send over UDP, rather it will immediately emit subsegments as soon as they are closed.

From the error you're seeing, it is most likely the size of a subsegment your lambda function may be creating is more than 64KB. Can you provide details about the subsegments you may be creating inside your lambda function? What is it trying to capture and what does it look like? Are you attaching lots of metadata or annotations to it within a loop or something?

If you can provide the complete code of your lambda function, it would be great.

Thanks!

[klofton]

Hi @srprash,

The size is definitely larger than 64K. The lambda function does a few things. It makes calls to STS, IAM, API Gateway and RDS cluster to collect data and process it. Assuming its more then 64K, is this a bug or we just cant use the Xray SDK?

[bhautikpip]

I would just simply call it as a SDK side limitation to not able to send over data more than 64KB. So just to be clear the size of one subsegment inside lambda function is more than 64 kb in your case? Are you using annotations and metadata which is making subsegment bulkier?

[allanlewis]

This was fixed in #205 and released in v2.5.0.