Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: cluster key rate limit enhancement #1036

Merged
merged 18 commits into from
Jun 17, 2024
Merged

feat: cluster key rate limit enhancement #1036

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
0f828be
feat: cluster rate limit by client ip
hanxiantao May 26, 2024
6c07b58
feat: cluster key rate limit
hanxiantao Jun 2, 2024
55b5c04
Merge branch 'alibaba:main' into key-cluster-rate-limit
hanxiantao Jun 2, 2024
d91c337
Merge branch 'main' into key-cluster-rate-limit
hanxiantao Jun 6, 2024
a335fc6
simplify the redis configuration method
hanxiantao Jun 9, 2024
484ad2d
Merge branch 'alibaba:main' into key-cluster-rate-limit
hanxiantao Jun 12, 2024
8ae7a1f
cluster key rate limit enhancement
hanxiantao Jun 13, 2024
92670ca
Update README.md
johnlanni Jun 13, 2024
66da298
cluster key rate limit enhancement
hanxiantao Jun 13, 2024
fa0a76d
Update README.md
hanxiantao Jun 13, 2024
cd8843a
fix limit_by_per_consumer
hanxiantao Jun 14, 2024
f1e785d
Update README.md
johnlanni Jun 14, 2024
a485bf6
support batch configuration of rate limiting rules
hanxiantao Jun 15, 2024
0df978a
Update README.md
hanxiantao Jun 15, 2024
75a2263
Update README.md
hanxiantao Jun 15, 2024
78e73f8
adjust the rules for construct redis keys
hanxiantao Jun 15, 2024
7a9d6e3
modify the comment
hanxiantao Jun 15, 2024
e9d2c29
Update README.md
johnlanni Jun 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 48 additions & 14 deletions plugins/wasm-go/extensions/cluster-key-rate-limit/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 功能说明

`key-cluster-rate-limit`插件实现了基于特定键值实现集群限流，键值来源可以是 URL 参数、HTTP 请求头、客户端 IP 地址
`key-cluster-rate-limit`插件实现了基于特定键值实现集群限流，键值来源可以是 URL 参数、HTTP 请求头、客户端 IP 地址、consumer 名称



Expand All @@ -9,9 +9,10 @@
| 配置项 | 类型 | 必填 | 默认值 | 说明 |
| ----------------------- | ------ | ---- | ------ | ---- |
| rule_name | string | 是 | - | 限流规则名称，根据限流规则名称和限流的客户端IP段来拼装redis key |
| limit_by_header | string | 否，`limit_by_header`,`limit_by_param`,`limit_by_per_ip` 中选填一项 | - | 配置获取限流键值的来源 http 请求头名称 |
| limit_by_param | string | 否，`limit_by_header`,`limit_by_param`,`limit_by_per_ip` 中选填一项 | - | 配置获取限流键值的来源 URL 参数名称 |
| limit_by_per_ip | string | 否，`limit_by_header`,`limit_by_param`,`limit_by_per_ip` 中选填一项 | - | 配置获取限流键值的来源 IP 参数名称，从请求头获取，以`from-header-对应的header名`，示例：`from-header-x-forwarded-for`，直接获取对端socket ip，配置为`from-remote-addr` |
| limit_by_header | string | 否，`limit_by_header`,`limit_by_param`,`limit_by_per_ip`,`limit_by_consumer` 中选填一项 | - | 配置获取限流键值的来源 http 请求头名称 |
| limit_by_param | string | 否，`limit_by_header`,`limit_by_param`,`limit_by_per_ip`,`limit_by_consumer` 中选填一项 | - | 配置获取限流键值的来源 URL 参数名称 |
| limit_by_per_ip | string | 否，`limit_by_header`,`limit_by_param`,`limit_by_per_ip`,`limit_by_consumer` 中选填一项 | - | 配置获取限流键值的来源 IP 参数名称，从请求头获取，以`from-header-对应的header名`，示例：`from-header-x-forwarded-for`，直接获取对端socket ip，配置为`from-remote-addr` |
| limit_by_consumer | string | 否，`limit_by_header`,`limit_by_param`,`limit_by_per_ip`,`limit_by_consumer` 中选填一项 | - | 无需添加实际值 |
| limit_keys | array of object | 是 | - | 配置匹配键值后的限流次数 |
| show_limit_quota_header | bool | 否 | false | 响应头中是否显示`X-RateLimit-Limit`（限制的总请求数）和`X-RateLimit-Remaining`（剩余还可以发送的请求数） |
| rejected_code | int | 否 | 429 | 请求被限流时，返回的HTTP状态码 |
Expand All @@ -20,13 +21,13 @@

`limit_keys`中每一项的配置字段说明

| 配置项 | 类型 | 必填 | 默认值 | 说明 |
| ---------------- | ------ | ------------------------------------------------------------ | ------ | ------------------ |
| key | string | 是 | - | 匹配的键值 |
| query_per_second | int | 否，`query_per_second`,`query_per_minute`,`query_per_hour`,`query_per_day` 中选填一项 | - | 允许每秒请求次数 |
| query_per_minute | int | 否，`query_per_second`,`query_per_minute`,`query_per_hour`,`query_per_day` 中选填一项 | - | 允许每分钟请求次数 |
| query_per_hour | int | 否，`query_per_second`,`query_per_minute`,`query_per_hour`,`query_per_day` 中选填一项 | - | 允许每小时请求次数 |
| query_per_day | int | 否，`query_per_second`,`query_per_minute`,`query_per_hour`,`query_per_day` 中选填一项 | - | 允许每天请求次数 |
| 配置项 | 类型 | 必填 | 默认值 | 说明 |
| ---------------- | ------ | ------------------------------------------------------------ | ------ | ------------------------------------------------------------ |
| key | string | 是 | - | 匹配的键值，针对`limit_by_header`,`limit_by_param`,`limit_by_consumer` 支持配置正则表达式（以regexp:开头后面跟实际正则表达式）或者*（代表所有），正则表达式示例：`regexp:^d.*`（以d开头的所有字符串） |
| query_per_second | int | 否，`query_per_second`,`query_per_minute`,`query_per_hour`,`query_per_day` 中选填一项 | - | 允许每秒请求次数 |
| query_per_minute | int | 否，`query_per_second`,`query_per_minute`,`query_per_hour`,`query_per_day` 中选填一项 | - | 允许每分钟请求次数 |
| query_per_hour | int | 否，`query_per_second`,`query_per_minute`,`query_per_hour`,`query_per_day` 中选填一项 | - | 允许每小时请求次数 |
| query_per_day | int | 否，`query_per_second`,`query_per_minute`,`query_per_hour`,`query_per_day` 中选填一项 | - | 允许每天请求次数 |

`redis`中每一项的配置字段说明

Expand All @@ -51,6 +52,12 @@ limit_keys:
query_per_second: 10
- key: a6a6d7f2-ba8a-11ec-bec2-00163e1250b5
query_per_minute: 100
# 正则表达式，以b开头的所有字符串
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里会有歧义，是匹配到的所有统一限制1000，还是匹配到的每一个分别限制1000。

所以我建议增加新的配置项: limit_by_per_header, limit_by_per_param

用来表示匹配到的每一个分别限制这一含义。

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里会有歧义，是匹配到的所有统一限制1000，还是匹配到的每一个分别限制1000。

所以我建议增加新的配置项: limit_by_per_header, limit_by_per_param

用来表示匹配到的每一个分别限制这一含义。

好的，那应该就是limit_by_header和limit_by_param只支持精确匹配的，limit_by_per_header和limit_by_per_param只支持正则匹配和*的方式

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以的

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以的

已调整，目前支持以下几种方式：
limit_by_header、limit_by_param、limit_by_consumer、limit_by_cookie（匹配cookie中的key=value） 支持精准匹配
limit_by_per_ip 支持按照ip和ip段配置
limit_by_peer_header、limit_by_peer_param、limit_by_peer_consumer、limit_by_per_cookie 支持正则匹配和*

- key: "regexp:^b.*"
query_per_hour: 1000
# 兜底用，代表所有，即每个带apikey请求参数的请求10000qdh
- key: "*"
query_per_hour: 10000
redis:
service_name: redis.static
show_limit_quota_header: true
Expand All @@ -65,6 +72,12 @@ limit_keys:
query_per_second: 10
- key: 308239
query_per_hour: 10
# 正则表达式，以c开头的所有字符串
- key: "regexp:^c.*"
query_per_hour: 1000
# 兜底用，代表所有，即每个带x-ca-key请求头的请求10000qdh
- key: "*"
query_per_hour: 10000
redis:
service_name: redis.static
show_limit_quota_header: true
Expand All @@ -76,20 +89,41 @@ show_limit_quota_header: true
rule_name: limit_by_per_ip_from-header-x-forwarded-for
limit_by_per_ip: from-header-x-forwarded-for
limit_keys:
# 精确ip
# 精确ip
- key: 1.1.1.1
query_per_day: 10
# ip段，符合这个ip段的ip，每个ip 100qps
# ip段，符合这个ip段的ip，每个ip 100qpd
- key: 1.1.1.0/24
query_per_day: 100
# 兜底用，即默认每个ip 1000qps
# 兜底用，即默认每个ip 1000qpd
- key: 0.0.0.0/0
query_per_day: 1000
redis:
service_name: redis.static
show_limit_quota_header: true
```

## 识别consumer，进行区别限流

```yaml
rule_name: default_limit_by_consumer
limit_by_consumer:
limit_keys:
- key: consumer1
query_per_hour: 10
# 正则表达式，以d开头的所有字符串
- key: "regexp:^d.*"
query_per_hour: 1000
# 兜底用，代表所有，即每个consumer 10000qdh
- key: "*"
query_per_hour: 10000
redis:
service_name: redis.static
show_limit_quota_header: true
```



## 对特定路由或域名开启

```yaml
Expand Down
126 changes: 74 additions & 52 deletions plugins/wasm-go/extensions/cluster-key-rate-limit/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,30 @@ package main

import (
"errors"
"fmt"
"github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
"github.com/tidwall/gjson"
regexp "github.com/wasilibs/go-re2"
"github.com/zmap/go-iptree/iptree"
"strings"
)

// 限流规则类型
type limitRuleType string

// 限流配置项key类型
type limitItemType string

const (
limitByHeaderType limitRuleType = "limitByHeader"
limitByParamType limitRuleType = "limitByParam"
limitByPerIpType limitRuleType = "limitByPerIp"
limitByHeaderType limitRuleType = "limit_by_header"
limitByParamType limitRuleType = "limit_by_param"
limitByPerIpType limitRuleType = "limit_by_per_ip"
limitByConsumer limitRuleType = "limit_by_consumer"

exactType limitItemType = "exact" // 精确匹配
regexpType limitItemType = "regexp" // 正则表达式
allType limitItemType = "*" // 匹配所有情况
ipNetType limitItemType = "ipNet" // ip段

RemoteAddrSourceType = "remote-addr"
HeaderSourceType = "header"
Expand All @@ -28,6 +39,13 @@ const (
SecondsPerDay = 24 * SecondsPerHour
)

var timeWindows = map[string]int64{
"query_per_second": Second,
"query_per_minute": SecondsPerMinute,
"query_per_hour": SecondsPerHour,
"query_per_day": SecondsPerDay,
}

type ClusterKeyRateLimitConfig struct {
ruleName string // 限流规则名称
limitType limitRuleType // 限流类型
Expand All @@ -47,8 +65,10 @@ type LimitByPerIp struct {
}

type LimitItem struct {
itemType limitItemType // 限流配置项key类型
key string // 限流key
ipNet *iptree.IPTree // 限流key转换的ip地址或者ip段
ipNet *iptree.IPTree // 限流key转换的ip地址或者ip段,仅用于itemType为ipNetType
re *regexp.Regexp // 正则表达式,仅用于itemType为regexpType
count int64 // 指定时间窗口内的总请求数量阈值
timeWindow int64 // 时间窗口大小
}
Expand Down Expand Up @@ -84,7 +104,7 @@ func initRedisClusterClient(json gjson.Result, config *ClusterKeyRateLimitConfig
return config.redisClient.Init(username, password, int64(timeout))
}

func parseClusterKeyRateLimitConfig(json gjson.Result, config *ClusterKeyRateLimitConfig, log wrapper.Log) error {
func parseClusterKeyRateLimitConfig(json gjson.Result, config *ClusterKeyRateLimitConfig) error {
ruleName := json.Get("rule_name")
if !ruleName.Exists() {
return errors.New("missing rule_name in config")
Expand Down Expand Up @@ -127,13 +147,17 @@ func parseClusterKeyRateLimitConfig(json gjson.Result, config *ClusterKeyRateLim
}
limitType = limitByPerIpType
}
limitByConsumerResult := json.Get("limit_by_consumer")
if limitByConsumerResult.Exists() {
limitType = limitByConsumer
}
if limitType == "" {
return errors.New("only one of 'limit_by_header' and 'limit_by_param' and 'limit_by_per_ip' can be set")
return errors.New("only one of 'limit_by_header' and 'limit_by_param' and 'limit_by_per_ip' and 'limit_by_consumer' can be set")
}
config.limitType = limitType

// 初始化LimitItem
err := initLimitItems(json, config, log)
err := initLimitItems(json, config)
if err != nil {
return err
}
Expand All @@ -158,7 +182,7 @@ func parseClusterKeyRateLimitConfig(json gjson.Result, config *ClusterKeyRateLim
return nil
}

func initLimitItems(json gjson.Result, config *ClusterKeyRateLimitConfig, log wrapper.Log) error {
func initLimitItems(json gjson.Result, config *ClusterKeyRateLimitConfig) error {
limitKeys := json.Get("limit_keys")
if !limitKeys.Exists() {
return errors.New("missing limit_keys in config")
Expand All @@ -172,59 +196,57 @@ func initLimitItems(json gjson.Result, config *ClusterKeyRateLimitConfig, log wr
if !key.Exists() || key.String() == "" {
return errors.New("limit_keys key is required")
}
var ipNet *iptree.IPTree

var (
itemKey = key.String()
itemType limitItemType
ipNet *iptree.IPTree
re *regexp.Regexp
)
if config.limitType == limitByPerIpType {
var err error
ipNet, err = parseIPNet(key.String())
ipNet, err = parseIPNet(itemKey)
if err != nil {
log.Errorf("parseIPNet error: %v", err)
return err
return fmt.Errorf("failed to parse IPNet for key '%s': %w", itemKey, err)
}
itemType = ipNetType
} else if itemKey == "*" {
itemType = allType
} else if strings.HasPrefix(itemKey, "regexp:") {
regexpStr := itemKey[len("regexp:"):]
var err error
re, err = regexp.Compile(regexpStr)
if err != nil {
return fmt.Errorf("failed to compile regex for key '%s': %w", itemKey, err)
}
itemType = regexpType
} else {
ipNet = nil
itemType = exactType
}

qps := item.Get("query_per_second")
if qps.Exists() && qps.Int() > 0 {
limitItems = append(limitItems, LimitItem{
key: key.String(),
ipNet: ipNet,
count: qps.Int(),
timeWindow: Second,
})
continue
}
qpm := item.Get("query_per_minute")
if qpm.Exists() && qpm.Int() > 0 {
limitItems = append(limitItems, LimitItem{
key: key.String(),
ipNet: ipNet,
count: qpm.Int(),
timeWindow: SecondsPerMinute,
})
continue
}
qph := item.Get("query_per_hour")
if qph.Exists() && qph.Int() > 0 {
limitItems = append(limitItems, LimitItem{
key: key.String(),
ipNet: ipNet,
count: qph.Int(),
timeWindow: SecondsPerHour,
})
continue
}
qpd := item.Get("query_per_day")
if qpd.Exists() && qpd.Int() > 0 {
limitItems = append(limitItems, LimitItem{
key: key.String(),
ipNet: ipNet,
count: qpd.Int(),
timeWindow: SecondsPerDay,
})
continue
if limitItem, err := createLimitItemFromRate(item, itemType, itemKey, ipNet, re); err != nil {
return err
} else if limitItem != nil {
limitItems = append(limitItems, *limitItem)
}
}
config.limitItems = limitItems
return nil
}

func createLimitItemFromRate(item gjson.Result, itemType limitItemType, key string, ipNet *iptree.IPTree, re *regexp.Regexp) (*LimitItem, error) {
for timeWindowKey, duration := range timeWindows {
q := item.Get(timeWindowKey)
if q.Exists() && q.Int() > 0 {
return &LimitItem{
itemType: itemType,
key: key,
ipNet: ipNet,
re: re,
count: q.Int(),
timeWindow: duration,
}, nil
}
}
return nil, errors.New("one of 'query_per_second', 'query_per_minute', 'query_per_hour', or 'query_per_day' must be set for key: " + key)
}
2 changes: 2 additions & 0 deletions plugins/wasm-go/extensions/cluster-key-rate-limit/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ require (
github.com/google/uuid v1.3.0 // indirect
github.com/higress-group/nottinygc v0.0.0-20231101025119-e93c4c2f8520 // indirect
github.com/magefile/mage v1.14.0 // indirect
github.com/tetratelabs/wazero v1.7.1 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.0 // indirect
github.com/wasilibs/go-re2 v1.5.3 // indirect
)
4 changes: 4 additions & 0 deletions plugins/wasm-go/extensions/cluster-key-rate-limit/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ github.com/magefile/mage v1.14.0 h1:6QDX3g6z1YvJ4olPhT1wksUcSa/V0a1B+pJb73fBjyo=
github.com/magefile/mage v1.14.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/tetratelabs/wazero v1.7.1 h1:QtSfd6KLc41DIMpDYlJdoMc6k7QTN246DM2+n2Y/Dx8=
github.com/tetratelabs/wazero v1.7.1/go.mod h1:ytl6Zuh20R/eROuyDaGPkp82O9C/DJfXAwJfQ3X6/7Y=
github.com/tidwall/gjson v1.14.3 h1:9jvXn7olKEHU1S9vwoMGliaT8jq1vJ7IH/n9zD9Dnlw=
github.com/tidwall/gjson v1.14.3/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
Expand All @@ -19,6 +21,8 @@ github.com/tidwall/pretty v1.2.0 h1:RWIZEg2iJ8/g6fDDYzMpobmaoGh5OLl4AXtGUGPcqCs=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/tidwall/resp v0.1.1 h1:Ly20wkhqKTmDUPlyM1S7pWo5kk0tDu8OoC/vFArXmwE=
github.com/tidwall/resp v0.1.1/go.mod h1:3/FrruOBAxPTPtundW0VXgmsQ4ZBA0Aw714lVYgwFa0=
github.com/wasilibs/go-re2 v1.5.3 h1:wiuTcgDZdLhu8NG8oqF5sF5Q3yIU14lPAvXqeYzDK3g=
github.com/wasilibs/go-re2 v1.5.3/go.mod h1:PzpVPsBdFC7vM8QJbbEnOeTmwA0DGE783d/Gex8eCV8=
github.com/zmap/go-iptree v0.0.0-20210731043055-d4e632617837 h1:DjHnADS2r2zynZ3WkCFAQ+PNYngMSNceRROi0pO6c3M=
github.com/zmap/go-iptree v0.0.0-20210731043055-d4e632617837/go.mod h1:9vp0bxqozzQwcjBwenEXfKVq8+mYbwHkQ1NF9Ap0DMw=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
Loading
Loading