Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SQS payload empty when trying to access from sensor #3534

Open
gavin-crane opened this issue Mar 24, 2025 · 0 comments
Open

SQS payload empty when trying to access from sensor #3534

gavin-crane opened this issue Mar 24, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@gavin-crane
Copy link

gavin-crane commented Mar 24, 2025 •
edited
Loading

Describe the bug
In a WSL environment. I have a k3s cluster setup with a configured event source and sensor for an SQS queue that is attached to an s3 bucket for object create events. My main goal is to process the SQS event payload message, which will contain metadata about the created object. Right now, I am simply trying to echo the payload body to the terminal to verify that the data is being transmitted across the bus and into any pods/workflows afterwards,

To Reproduce
Steps to reproduce the behavior:

  1. Set up a k3s cluster using the curl -sfL https://get.k3s.io | sh - quick install script
  2. Setup an SQS queue with an IAM user access, record access key and secret key
  3. Download and apply argo events to the argo-events namespace, (use versions below)
  4. Create and apply aws-credentials.yaml secret (file contents below), containing access key and secret key for SQS queue access
  5. Create and apply sqs-source.yaml event source (file contents below)
  6. Create and apply s3-sensor.yaml (file contents below)
  7. Send a message on the SQS manually through AWS dashboard.
  8. Check for the generated pod within the argo-events namespace
  9. View the logs of the pod, The logs should show the SQS payload body, but nothing is shown:
># kubectl -n argo-events get pods
NAME                                         READY   STATUS      RESTARTS      AGE
aws-sqs-eventsource-flcxc-7754d979c6-wc2gd   1/1     Running     3 (78m ago)   10d
controller-manager-7c585bd5c-6fz57           1/1     Running     3 (78m ago)   10d
eventbus-default-stan-0                      2/2     Running     6 (78m ago)   10d
eventbus-default-stan-1                      2/2     Running     6 (78m ago)   10d
eventbus-default-stan-2                      2/2     Running     6 (78m ago)   10d
log-pod-4qjqm                                0/1     Completed   0             40m
log-pod-h87nj                                0/1     Completed   0             2m57s
log-pod-h96sv                                0/1     Completed   0             21m
log-pod-nk7sq                                0/1     Completed   0             4s <-- SQS event triggers pod generation
log-pod-tz4wb                                0/1     Completed   0             8m3s
s3-sensor-sensor-wqzrp-7df9cdcddd-69j52      1/1     Running     0             3m34s

EXPECTED:

># kubectl logs log-pod-nk7sq -n argo-events
S3 file upload event received!!!:
<body message data from sqs message>
>#

RESULT:

># kubectl logs log-pod-nk7sq -n argo-events
S3 file upload event received!!!:

>#

Files:
aws-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
type: Opaque
data:
  accesskey: <b64 encoded key>
  secretkey: <b64 encoded key>

s3-sensor.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: s3-sensor
spec:
  template:
    serviceAccountName: argo-events-sa
  dependencies:
    - name: test-dep
      eventSourceName: aws-sqs
      eventName: example
  triggers:
    - template:
        name: log-trigger
        k8s:
          group: ""
          version: v1
          resource: pods
          operation: create
          source:
            resource:
              apiVersion: v1
              kind: Pod
              metadata:
                generateName: log-pod-
                labels: log-pod
              spec:
                containers:
                  - name: log-container
                    image: alpine
                    command: ["echo"]
                    args: ["S3 file upload event received:\n", ""]
                restartPolicy: Never
        parameters:
          - src:
              dependencyName: test-dep
              dataKey: body
              dest: spec.containers.0.args.1

sqs-source.yaml:

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: aws-sqs
spec:
  sqs:
    example:
      # jsonBody specifies that all event body payload coming from this
      # source will be JSON
      jsonBody: true
      # accessKey contains information about K8s secret that stores the access key
      accessKey:
        # Key within the K8s secret whose corresponding value (must be base64 encoded) is access key
        key: accesskey
        # Name of the K8s secret that contains the access key
        name: aws-secret
      # secretKey contains information about K8s secret that stores the secret key
      secretKey:
        # Key within the K8s secret whose corresponding value (must be base64 encoded) is secret key
        key: secretkey
        # Name of the K8s secret that contains the secret key
        name: aws-secret
      # aws region
      region: us-east-1
      # name of the queue. The eventsource resolves the url of the queue from the queue name.
      queue: sqsqueue
      # The duration (in seconds) for which the call waits for a message to arrive in the queue before returning.
      # MUST BE > 0 AND <= 20
      waitTimeSeconds: 20

Environment

  • Ubuntu 24.04.2 LTS (WSL)
  • k3s version v1.31.6+k3s1 (6ab750f9)
  • Go version go1.22.12
  • Argo: v3.6.5
  • Argo-events: v1.9.4

Additional context
Sensor logs:

># kubectl logs s3-sensor-sensor-wqzrp-7df9cdcddd-69j52 -n argo-events
{"level":"info","ts":1742830674.6684031,"logger":"argo-events.sensor","caller":"cmd/start.go:83","msg":"starting sensor server","sensorName":"s3-sensor","version":"v1.9.4"}
{"time":"2025-03-24T15:37:54.668554216Z","level":"INFO","msg":"trying to resolve absolute path","path":"/etc/eventbus/auth"}
{"time":"2025-03-24T15:37:54.668690013Z","level":"INFO","msg":"adding path to search paths","path":"/etc/eventbus/auth"}
{"time":"2025-03-24T15:37:54.668697581Z","level":"INFO","msg":"attempting to read in config file"}
{"time":"2025-03-24T15:37:54.668701153Z","level":"INFO","msg":"searching for config in paths","paths":["/etc/eventbus/auth"]}
{"level":"info","ts":1742830674.6687393,"logger":"argo-events.sensor","caller":"metrics/metrics.go:205","msg":"starting metrics server","sensorName":"s3-sensor"}
{"level":"info","ts":1742830674.6717188,"logger":"argo-events.sensor","caller":"leaderelection/leaderelection.go:166","msg":"Not the LEADER, stand by ...","sensorName":"s3-sensor"}
{"level":"info","ts":1742830675.5685213,"logger":"argo-events.sensor","caller":"leaderelection/leaderelection.go:175","msg":"Becoming a Candidate, stand by ...","sensorName":"s3-sensor"}
{"level":"info","ts":1742830675.5687103,"logger":"argo-events.sensor","caller":"leaderelection/leaderelection.go:172","msg":"I'm the LEADER, starting ...","sensorName":"s3-sensor"}
{"time":"2025-03-24T15:37:55.56877041Z","level":"INFO","msg":"trying to resolve absolute path","path":"/etc/eventbus/auth"}
{"time":"2025-03-24T15:37:55.568800544Z","level":"INFO","msg":"adding path to search paths","path":"/etc/eventbus/auth"}
{"time":"2025-03-24T15:37:55.568802876Z","level":"INFO","msg":"attempting to read in config file"}
{"time":"2025-03-24T15:37:55.568805218Z","level":"INFO","msg":"searching for config in paths","paths":["/etc/eventbus/auth"]}
{"level":"info","ts":1742830675.5690393,"logger":"argo-events.sensor","caller":"sensors/listener.go:353","msg":"Sensor started.","sensorName":"s3-sensor"}
{"level":"info","ts":1742830675.5690708,"logger":"argo-events.sensor","caller":"sensors/listener.go:505","msg":"Dependency expression for trigger log-trigger: test-dep","sensorName":"s3-sensor"}
{"level":"info","ts":1742830675.5690994,"logger":"argo-events.sensor","caller":"base/stan.go:49","msg":"NATS auth strategy: Token","sensorName":"s3-sensor","clientID":"client-1156381081-69"}
{"level":"info","ts":1742830675.571017,"logger":"argo-events.sensor","caller":"base/stan.go:61","msg":"Connected to NATS server.","sensorName":"s3-sensor","clientID":"client-1156381081-69"}
{"level":"info","ts":1742830675.573095,"logger":"argo-events.sensor","caller":"base/stan.go:74","msg":"Connected to NATS streaming server.","sensorName":"s3-sensor","clientID":"client-1156381081-69"}
{"level":"debug","ts":1742830675.5731585,"logger":"argo-events.sensor","caller":"sensors/listener.go:165","msg":"just created connection 0xc000f8e650, STANTriggerConn{ClientID:client-1156381081-69,Sensor:s3-sensor,Trigger:log-trigger}","sensorName":"s3-sensor","triggerName":"log-trigger"}
{"level":"info","ts":1742830675.5731812,"logger":"argo-events.sensor","caller":"sensors/listener.go:316","msg":"starting eventbus connection daemon for client STANTriggerConn{ClientID:client-1156381081-69,Sensor:s3-sensor,Trigger:log-trigger}...","sensorName":"s3-sensor","triggerName":"log-trigger"}
{"level":"info","ts":1742830675.5732145,"logger":"argo-events.sensor","caller":"sensors/listener.go:302","msg":"started subscribing to events for trigger log-trigger with client connection STANTriggerConn{ClientID:client-1156381081-69,Sensor:s3-sensor,Trigger:log-trigger}","sensorName":"s3-sensor","triggerName":"log-trigger"}
{"level":"info","ts":1742830675.5754392,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:106","msg":"Subscribed to subject eventbus-argo-events using durable name client-1156381081","sensorName":"s3-sensor","triggerName":"log-trigger","clientID":"client-1156381081-69"}
{"level":"info","ts":1742830675.575538,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:114","msg":"starting ExactOnce cache clean up daemon ...","sensorName":"s3-sensor","triggerName":"log-trigger","clientID":"client-1156381081-69"}
{"level":"debug","ts":1742830710.722548,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:178","msg":"New incoming Event Source Message, dependency name=test-dep","sensorName":"s3-sensor","triggerName":"log-trigger","clientID":"client-1156381081-69"}
{"level":"debug","ts":1742830710.722679,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:281","msg":"Triggering actions for client client-1156381081-69","sensorName":"s3-sensor","triggerName":"log-trigger","clientID":"client-1156381081-69"}
{"level":"debug","ts":1742830710.7228324,"logger":"argo-events.sensor","caller":"sensors/listener.go:416","msg":"resolving the trigger implementation","sensorName":"s3-sensor","triggerName":"log-trigger"}
{"level":"debug","ts":1742830710.7229478,"logger":"argo-events.sensor","caller":"sensors/listener.go:423","msg":"fetching trigger resource if any","sensorName":"s3-sensor"}
{"level":"debug","ts":1742830710.7230136,"logger":"argo-events","caller":"artifacts/resource.go:55","msg":"reading artifact from resource template","resource":{"apiVersion":"v1","kind":"Pod","metadata":{"generateName":"log-pod-","labels":"log-pod"},"spec":{"containers":[{"args":["S3 file upload event received!!!:\n",""],"command":["echo"],"image":"alpine","name":"log-container"}],"restartPolicy":"Never"}}}
{"level":"debug","ts":1742830710.7244658,"logger":"argo-events.sensor","caller":"sensors/listener.go:432","msg":"applying resource parameters if any","sensorName":"s3-sensor","triggerName":"log-trigger","triggerType":"Kubernetes"}
{"level":"debug","ts":1742830710.7244983,"logger":"argo-events.sensor","caller":"sensors/listener.go:438","msg":"executing the trigger resource","sensorName":"s3-sensor","triggerName":"log-trigger","triggerType":"Kubernetes"}
{"level":"info","ts":1742830710.7245097,"logger":"argo-events.sensor","caller":"standard-k8s/standard-k8s.go:157","msg":"creating the object...","sensorName":"s3-sensor","triggerName":"log-trigger","triggerType":"Kubernetes"}
{"level":"debug","ts":1742830710.733335,"logger":"argo-events.sensor","caller":"sensors/listener.go:443","msg":"trigger resource successfully executed","sensorName":"s3-sensor","triggerName":"log-trigger","triggerType":"Kubernetes"}
{"level":"debug","ts":1742830710.7333958,"logger":"argo-events.sensor","caller":"sensors/listener.go:445","msg":"applying trigger policy","sensorName":"s3-sensor","triggerName":"log-trigger","triggerType":"Kubernetes"}
{"level":"info","ts":1742830710.7334106,"logger":"argo-events.sensor","caller":"sensors/listener.go:449","msg":"Successfully processed trigger 'log-trigger'","sensorName":"s3-sensor","triggerName":"log-trigger","triggerType":"Kubernetes","triggeredBy":["test-dep"],"triggeredByEvents":["191b755677b7418197f54df003be0d16"]}
{"level":"debug","ts":1742830741.4391618,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:135","msg":"finished evicting 0 cached IDs, time cost: 0 ms","sensorName":"s3-sensor","triggerName":"log-trigger","clientID":"client-1156381081-69"}

Event source logs:

># kubectl logs -n argo-events aws-sqs-eventsource-flcxc-7754d979c6-wc2gd
{"level":"info","ts":1742828014.06327,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"1754f4065f6640e795a067a9c4570850"}
{"level":"info","ts":1742828014.3436458,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"a9a3650fb4bc40f29080dfc949829b6f"}
{"level":"info","ts":1742828014.5282779,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"3853455a089b4237b9bc4f295d8e9e94"}
{"level":"info","ts":1742828015.139669,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"c29a5c624b5b44c89fc256905a97e796"}
{"level":"info","ts":1742828015.3078299,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"a1edc68d13d048eeb1d073742e0bdd63"}
{"level":"info","ts":1742828015.4751337,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"823b64e52e0f4acc9ec569a7d8d53115"}
{"level":"info","ts":1742828015.9374697,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"6bf18fb4cb9c4fb583f635b038729a2c"}
{"level":"info","ts":1742828016.0121534,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"c16756c47a424e4fac0452da24882390"}
{"level":"info","ts":1742828016.0851908,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"e8d0e618da7543daade4dc46a6d7336f"}
{"level":"info","ts":1742828016.616361,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"c995a8c856ab446badf8964b4680577f"}
{"level":"info","ts":1742828016.7819896,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"4847ca11f44045f9bb0d97189c518a2a"}
{"level":"info","ts":1742828016.9738164,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"159051bb3d014419a547dea10aaec847"}
{"level":"info","ts":1742828428.4833825,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"a24b93bc4ad14ebaabe0317af1d6da16"}
{"level":"info","ts":1742829578.0527687,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"2e5743f372b34f36a3d9306e79edc36c"}
{"level":"info","ts":1742830404.459951,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"1c13a3e07de34ea3a954205c0919c48c"}
{"level":"info","ts":1742830710.722201,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"191b755677b7418197f54df003be0d16"}
{"level":"info","ts":1742830883.6820583,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"ef734ad27d074b309b4c3d95b43ca68c"}
{"level":"info","ts":1742832417.5998452,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:590","msg":"Succeeded to publish an event","eventSourceName":"aws-sqs","eventName":"example","eventSourceType":"sqs","eventID":"0ed4fd22bfe5469ca6885eb6ae919aac"}

Message from the maintainers:
@gavin-crane gavin-crane added the bug Something isn't working label Mar 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gavin-crane