feat(bigTent slo): Custom Validate function for big tent SLO schema

[Leo-DiCara]

Please Don't merge this until all work in https://github.com/grafana/slo/issues/2697 is complete. We want to ensure that the SLO Plugin API is merged and running in prod before we merge terraform-provider changes.

This adds a custom validate function for queries that checks to ensure basic form of big SLO JSON is being followed before we allow the user to roundtrip it to the API. This checks for:

• Basic JSON structure
• refID field
• datasource field
• uid and type subfields in datasource

Closes https://github.com/grafana/slo/issues/2702, https://github.com/grafana/slo/issues/2876, https://github.com/grafana/slo/issues/2875

[github-actions]

In order to lower resource usage and have a faster runtime, PRs will not run Cloud tests automatically.
To do so, a Grafana Labs employee must trigger the cloud acceptance tests workflow manually.

[elainevuong]

I'd like to see if we can remove the warnings for valid Prometheus queries when we validate the query field. It might not be a fantastic customer experience for existing users, to suddenly get warnings from the Provider for queries that they haven't changed.

Perhaps we can investigate if this DiffSuppressFunc is an option to pursue?

[elainevuong]

Just want to circle back - I verified that if we use the Terraform Provider as is, it DOES give warnings that show up, even if there isn't a diff in the field.

My last thought is I'm not even sure validate runs if a tf plan doesnt generate a diff for that field. This would mean unchanged SLOs wouldn't generate warnings. tf validate would generate these warnings everytime.

With the way this PR is set up currently, we've see this warning in both Ratio SLOs (due to the ValidateBigTent on the success and total metrics) AND in the Freeform Prom QL SLOs. These warnings appear on both "CREATE" of a new SLO, as well as on "UPDATE" of an existing SLO.

I think we need two changes:

• drop the ValidateBigTent() on both the success_metric and total_metric
• remove the warning if we can't parse as JSON

screenshots if anyone is interested in viewing what the user experience is like.

CREATE - Ratio SLO

CREATE - Adv Freeform Prom SLO

UPDATE - Adv Freeform Prom SLO

[elainevuong]

issue: can we use a Big Tent SLO example that matches the format and RefID of how the frontend generates it? i.e. using the Success and Total RefIDs, so it's easier for users to follow along? the example currently used looks to be a non-standard SLO that we had used for initial testing

maybe something like this

[
  {
    "datasource": {
      "type": "graphite",
      "uid": "datasource-uid"
    },
    "refId": "Success",
    "target": "groupByNode(perSecond(web.*.http.2xx_success.*.*), 3, 'avg'')"
  },
  {
    "datasource": {
      "type": "graphite",
      "uid": "datasource-uid"
    },
    "refId": "Total",
    "target": "groupByNode(perSecond(web.*.http.5xx_errors.*.*), 3, 'avg')"
  },
  {
    "datasource": {
      "type": "__expr__",
      "uid": "__expr__"
    },
    "expression": "$Success / $Total",
    "refId": "Expression",
    "type": "math"
  }
]

[elainevuong]

issue: do we want to use "Big Tent" language in our error messages? how about "supported datasource must specify a datasource: uid" and or "datasource: type"

nit: did you want to specify datasource: uid rather than just uid? same comment for datasource: type above. since the uid and type fields are part of the datasource struct. this is what it looks like in the Terraform terminal (I removed the type field and then I removed the uid field - wanted to see how the error message appeared to a user)

[elainevuong]

Sample Unhappy grafanaquery response from the API

notice how we have three apostrophes in the Graphite query

this returns a:

{
    "code": 400,
    "error": "SLO failed validation: invalid parameters on query please check your input values"
}

---

[
  {
    "datasource": {
      "type": "graphite",
      "uid": "becy9yvjmuz9ca"
    },
    "refId": "Success",
    "target": "groupByNode(perSecond(web.*.http.2xx_success.*.*), 1, 'avg''')"
  },
  {
    "datasource": {
      "type": "graphite",
      "uid": "becy9yvjmuz9ca"
    },
    "refId": "Total",
    "target": "groupByNode(perSecond(web.*.http.*.*.*), 1, 'avg')"
  },
  {
    "datasource": {
      "type": "__expr__",
      "uid": "__expr__"
    },
    "expression": "$Success / $Total",
    "refId": "Expression",
    "type": "math"
  }
]

[ellisda]

LGTM - we'll have some minor doc additions to add here with instructions to prototype queries in a dashboard panel and copy/paste here, but we can add that later as well

[ellisda]

praise: we've been meaning to swap / fix this for awhile. Not part of big-tent, but a good improvement

[ellisda]

issue: the extra ' char was from testing, is invalid in current form

Suggested change

	target : "groupByNode(perSecond(web.*.http.2xx_success.*.*), 3, 'avg'')"
	target : "groupByNode(perSecond(web.*.http.2xx_success.*.*), 3, 'avg')"

[ellisda]

suggestion - 5xx errors isn't really a great "total" query, we should give a better example ... but I think we may have used this one on website too, so need to fix both

[Leo-DiCara]

Yeah this is the exact same query.

[ellisda]

nit / suggestion: most of our docs seem to try to have more informative link titles than "docs" (though I say "docs" internally all the time)

Suggested change

	For a list of currently supported datasources review the [documentation](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/set-up/additionaldatasources/#supported-data-sources).
	For a complete list, see [supported data sources](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/set-up/additionaldatasources/#supported-data-sources).

[ellisda]

Suggested change

Description: "Array for holding a set of grafana queries",

the other query types seem to skip this outer description

[ellisda]

Suggested change

	Grafana Queries use the grafana_queries field. It expects a JSON string list of valid grafana query JSON objects.
	Grafana Queries use the grafana_queries field. It expects a JSON string list of valid grafana query JSON objects, the same as you'll find assigned to a Grafana Dashboard panel `targets` field.