CodeCommit: SerializationError upon event processing

[antoineco]

Upon pushing a commit to the observed repository, the source fails with a SerializationError:

time="2020-04-07T19:54:35Z" level=info msg="Push Events Enabled!"
time="2020-04-07T19:54:36Z" level=info msg="Pull Request Events Enabled!"
time="2020-04-07T19:58:06Z" level=error msg="SerializationError: failed to unmarshal response error\n\tstatus code: 400
, request id: 8b8b7d42-6ad1-484d-82aa-229a602e9403\ncaused by: UnmarshalError: failed decoding error message\n\t0000000
0  3c 54 68 72 6f 74 74 6c  69 6e 67 45 78 63 65 70  |<ThrottlingExcep|\n00000010  74 69 6f 6e 3e 0a 20 20  3c 4d 65 73
 73 61 67 65  |tion>.  <Message|\n00000020  3e 52 61 74 65 20 65 78  63 65 65 64 65 64 3c 2f  |>Rate exceeded</|\n00000
030  4d 65 73 73 61 67 65 3e  0a 3c 2f 54 68 72 6f 74  |Message>.</Throt|\n00000040  74 6c 69 6e 67 45 78 63  65 70 74
69 6f 6e 3e 0a  |tlingException>.|\n\ncaused by: invalid character '<' looking for beginning of value"
time="2020-04-07T20:41:10Z" level=error msg="SerializationError: failed to unmarshal response error\n\tstatus code: 400
, request id: 7676f2b8-ab78-47a3-bd23-139983973551\ncaused by: UnmarshalError: failed decoding error message\n\t0000000
0  3c 54 68 72 6f 74 74 6c  69 6e 67 45 78 63 65 70  |<ThrottlingExcep|\n00000010  74 69 6f 6e 3e 0a 20 20  3c 4d 65 73
 73 61 67 65  |tion>.  <Message|\n00000020  3e 52 61 74 65 20 65 78  63 65 65 64 65 64 3c 2f  |>Rate exceeded</|\n00000
030  4d 65 73 73 61 67 65 3e  0a 3c 2f 54 68 72 6f 74  |Message>.</Throt|\n00000040  74 6c 69 6e 67 45 78 63  65 70 74
69 6f 6e 3e 0a  |tlingException>.|\n\ncaused by: invalid character '<' looking for beginning of value"
time="2020-04-07T20:42:44Z" level=info msg="send Commit Event"

Pod manifest

spec:
  containers:
  - name: adapter
    image: gcr.io/triggermesh/awscodecommit
    args:
    - -sink=http://default-broker.dev.svc.cluster.local
    env:
    - name: REPO
      value: source-test
    - name: BRANCH
      value: master
    - name: AWS_REGION
      value: eu-central-1
    - name: EVENTS
      value: pull_request,push
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          key: aws_access_key_id
          name: awscreds
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          key: aws_secret_access_key
          name: awscreds

[antoineco]

After some quick analysis, it seems like we get rate limited due to aggressive polling and don't handle the error properly.

[sebgoa]

@tzununbekov could you check that one tomorrow.

Assuming your exponenatial backoff solved the rate limiting we should not see this error anymore.
but there might also be better error handling that we can get in before the release of v0.2.0

[tzununbekov]

With backoff mechanism in place issue is not reproducible, although we must remember that requests rate limit covers all requests to all services in AWS account (ref). That is, even exponential backoff may not help in some edge cases, but I think we should handle such issues as they come.

[antoineco]

One thing that was happening if I remember correctly was that errors were verbose and poorly handled, like if the API response was not a commit we would just fail at unmarshaling instead of handling the error returned by the AWS SDK. Is that still the case? If not, I agree we should close this.

[antoineco]

@tzununbekov I went over the code and all errors seem to be handled correctly now. Closing!

[tzununbekov]

yep, I think that error was there when we had custom types for every AWS event and tried to unmarshal incoming data. I have a tiny piece of improvement for codecommit source - there is a for loop with incorrect break condition that I don't think is used anyhow. PR is coming in a moment