first stab at RESP3; no actual parse code at the moment - this is API-centric

- introduce `Resp2Type` and `Resp3Type` shims (`Resp2Type` has reduced types)
- mark existing `Type` as `[Obsolete]`, and proxy to `Resp2Type` for compat
- deal with null handling differences
- deal with `Boolean`, which works very differently (`t`/`f` instead of `1`/`0`)

remove RedisResult.Type from Shipped.txt

- handle RESP3 types
- handle [+|-]{inf|nan}
- avoid alloc when parsing doubles

made the RedisResult constructor non-public (fix accidental API)

remove RawResult.Type; fix broken loop

format

incorrect attribute check

- nomenclature: MultiBulk => Array
- efficiency: use bit-packing to RESP2 type conversion is a bit mask

fix RawResult.HasValue

fix null array return (EmptyMultiBulk is no longer helpful)

add a ToString to Replica (other bits were local test setup issues)

simplify switch in TryParseDouble

- protocol configuration parsing
- avoid inbuilt equality/comparison operations on Version
- rules for when to try resp3

configuration documentation

words

fix ConfigurationOptions.Clone

actually connect via RESP3

demand redis6 in the RESP3 connect test

remove unnecessary directives

Lua results

more tests and tweaks to fix tests

simplify handshake

fallback connect

move SETNAME back into HELLO message; add lots of documentation about *why*

DEBUG PROTOCOL tests; some failures to look at

fix protocol tests

add missing "hide me" attribs

add docs and release notes

tyop

redundant

re-enable to get server-maintenance notifications

- ConnectWithBrokenHello is inconclusive if not a v6 server
- allow non-RESP3 tests on non-v6 servers

"DEBUG PROTOCOL" tests are inconclusive on non-v6

fix TryConnect (CLIENT ID) not always available

save all

counting is hard

expose IAsyncEnumerable on ChannelMessageQueue (#2402)

* expose IAsyncEnumerable on ChannelMessageQueue
fix #2400

* PR number

* move ChannelMessageQueue.GetAsyncEnumerator to shipped

LUA conversions

version Lua RESP conversions

true/false handling depends on setresp(3)

revert "if" split in ResultProcessor

use enum for RedisProtocol

reinstate parameterless RedisResult .ctor

fix resp 2/3 inversion snafu from enumification

fix resp dependent connection reuse issue

add failing Execute test re RESP2 vs RESP3 delta

ValuePairInterleavedProcessorBase should auto-handle responses that have become jagged in RESP3

pattern match is easier to read here

- move IsResp3; that is a PhysicalConnection thing, not a ServerEndPoint thing
- allow RawResult to know whether it is RESP3; involved moving some flags (which removes a bit hack we were using, so: yay)
- make the interleave un-jaggedify only apply on RESP3

add more RESP3 API change tests

compensate for XREAD having a different shape in RESP3

disable implicit RESP3 based on target server version

# Conflicts:
#	docs/Configuration.md
#	docs/ReleaseNotes.md
#	src/StackExchange.Redis/ConfigurationOptions.cs
#	src/StackExchange.Redis/ConnectionMultiplexer.cs
#	src/StackExchange.Redis/Interfaces/IConnectionMultiplexer.cs
#	src/StackExchange.Redis/ServerEndPoint.cs
#	tests/StackExchange.Redis.Tests/PubSubTests.cs
#	tests/StackExchange.Redis.Tests/TestBase.cs

Author: mgravell

StackExchange.Redis.sln

@@ -119,6 +119,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "docs", "docs", "{153A10E4-E
 		docs\Profiling_v2.md = docs\Profiling_v2.md
 		docs\PubSubOrder.md = docs\PubSubOrder.md
 		docs\ReleaseNotes.md = docs\ReleaseNotes.md
+		docs\Resp3.md = docs\Resp3.md
 		docs\Scripting.md = docs\Scripting.md
 		docs\Server.md = docs\Server.md
 		docs\Testing.md = docs\Testing.md

docs/Configuration.md

@@ -1,4 +1,4 @@
-Configuration
+# Configuration
 ===
 
 When connecting to Redis version 6 or above with an ACL configured, your ACL user needs to at least have permissions to run the ECHO command. We run this command to verify that we have a valid connection to the Redis service.
@@ -15,7 +15,7 @@ The `configuration` here can be either:
 
 The latter is *basically* a tokenized form of the former.
 
-Basic Configuration Strings
+## Basic Configuration Strings
 -
 
 The *simplest* configuration example is just the host name:
@@ -66,7 +66,7 @@ Microsoft Azure Redis example with password
 var conn = ConnectionMultiplexer.Connect("contoso5.redis.cache.windows.net,ssl=true,password=...");
 ```
 
-Configuration Options
+## Configuration Options
 ---
 
 The `ConfigurationOptions` object has a wide range of properties, all of which are fully documented in intellisense. Some of the more common options to use include:
@@ -98,6 +98,7 @@ The `ConfigurationOptions` object has a wide range of properties, all of which a
 | version={string}       | `DefaultVersion`       | (`4.0` in Azure, else `2.0`) | Redis version level (useful when the server does not make this available)                                 |
 | tunnel={string}        | `Tunnel`               | `null`                       | Tunnel for connections (use `http:{proxy url}` for "connect"-based proxy server)                          |
 | setlib={bool}          | `SetClientLibrary`     | `true`                       | Whether to attempt to use `CLIENT SETINFO` to set the library name/version on the connection              |
+| protocol={string}      | `Protocol`             | `null`                       | Redis protocol to use; see section below                                                                  |
 
 Additional code-only options:
 - ReconnectRetryPolicy (`IReconnectRetryPolicy`) - Default: `ReconnectRetryPolicy = ExponentialRetry(ConnectTimeout / 2);`
@@ -121,16 +122,17 @@ Additional code-only options:
 Tokens in the configuration string are comma-separated; any without an `=` sign are assumed to be redis server endpoints. Endpoints without an explicit port will use 6379 if ssl is not enabled, and 6380 if ssl is enabled.
 Tokens starting with `$` are taken to represent command maps, for example: `$config=cfg`.
 
-Obsolete Configuration Options
+## Obsolete Configuration Options
 ---
+
 These options are parsed in connection strings for backwards compatibility (meaning they do not error as invalid), but no longer have any effect.
 
 | Configuration string   | `ConfigurationOptions` | Previous Default | Previous Meaning |
 | ---------------------- | ---------------------- | ---------------------------- | --------------------------------------------------------------------------------------------------------- |
 | responseTimeout={int} | `ResponseTimeout` | `SyncTimeout` | Time (ms) to decide whether the socket is unhealthy |
 | writeBuffer={int} | `WriteBuffer` | `4096` | Size of the output buffer |
 
-Automatic and Manual Configuration
+## Automatic and Manual Configuration
 ---
 
 In many common scenarios, StackExchange.Redis will automatically configure a lot of settings, including the server type and version, connection timeouts, and primary/replica relationships. Sometimes, though, the commands for this have been disabled on the redis server. In this case, it is useful to provide more information:
@@ -159,7 +161,8 @@ Which is equivalent to the command string:
 ```config
 redis0:6379,redis1:6380,keepAlive=180,version=2.8.8,$CLIENT=,$CLUSTER=,$CONFIG=,$ECHO=,$INFO=,$PING=
 ```
-Renaming Commands
+
+## Renaming Commands
 ---
 
 A slightly unusual feature of redis is that you can disable and/or rename individual commands. As per the previous example, this is done via the `CommandMap`, but instead of passing a `HashSet<string>` to `Create()` (to indicate the available or unavailable commands), you pass a `Dictionary<string,string>`. All commands not mentioned in the dictionary are assumed to be enabled and not renamed. A `null` or blank value records that the command is disabled. For example:
@@ -182,8 +185,9 @@ The above is equivalent to (in the connection string):
 $INFO=,$SELECT=use
 ```
 
-Redis Server Permissions
+## Redis Server Permissions
 ---
+
 If the user you're connecting to Redis with is limited, it still needs to have certain commands enabled for the StackExchange.Redis to succeed in connecting. The client uses:
 - `AUTH` to authenticate
 - `CLIENT` to set the client name
@@ -203,7 +207,7 @@ For example, a common _very_ minimal configuration ACL on the server (non-cluste
 
 Note that if you choose to disable access to the above commands, it needs to be done via the `CommandMap` and not only the ACL on the server (otherwise we'll attempt the command and fail the handshake). Also, if any of the these commands are disabled, some functionality may be diminished or broken.
 
-twemproxy
+## twemproxy
 ---
 
 [twemproxy](https://github.com/twitter/twemproxy) is a tool that allows multiple redis instances to be used as though it were a single server, with inbuilt sharding and fault tolerance (much like redis cluster, but implemented separately). The feature-set available to Twemproxy is reduced. To avoid having to configure this manually, the `Proxy` option can be used:
@@ -216,8 +220,9 @@ var options = new ConfigurationOptions
 };
 ```
 
-envoyproxy
+##envoyproxy
 ---
+
 [Envoyproxy](https://github.com/envoyproxy/envoy) is a tool that allows to front a redis cluster with a set of proxies, with inbuilt discovery and fault tolerance. The feature-set available to Envoyproxy is reduced. To avoid having to configure this manually, the `Proxy` option can be used:
 ```csharp
 var options = new ConfigurationOptions+{
@@ -227,7 +232,7 @@ var options = new ConfigurationOptions+{
 ```
 
 
-Tiebreakers and Configuration Change Announcements
+## Tiebreakers and Configuration Change Announcements
 ---
 
 Normally StackExchange.Redis will resolve primary/replica nodes automatically. However, if you are not using a management tool such as redis-sentinel or redis cluster, there is a chance that occasionally you will get multiple primary nodes (for example, while resetting a node for maintenance it may reappear on the network as a primary). To help with this, StackExchange.Redis can use the notion of a *tie-breaker* - which is only used when multiple primaries are detected (not including redis cluster, where multiple primaries are *expected*). For compatibility with BookSleeve, this defaults to the key named `"__Booksleeve_TieBreak"` (always in database 0). This is used as a crude voting mechanism to help determine the *preferred* primary, so that work is routed correctly.
@@ -238,8 +243,9 @@ Both options can be customized or disabled (set to `""`), via the `.Configuratio
 
 These settings are also used by the `IServer.MakeMaster()` method, which can set the tie-breaker in the database and broadcast the configuration change message. The configuration message can also be used separately to primary/replica changes simply to request all nodes to refresh their configurations, via the `ConnectionMultiplexer.PublishReconfigure` method.
 
-ReconnectRetryPolicy
+## ReconnectRetryPolicy
 ---
+
 StackExchange.Redis automatically tries to reconnect in the background when the connection is lost for any reason. It keeps retrying  until the connection has been restored. It would use ReconnectRetryPolicy to decide how long it should wait between the retries.
 ReconnectRetryPolicy can be exponential (default), linear or a custom retry policy.
 
@@ -264,3 +270,20 @@ config.ReconnectRetryPolicy = new LinearRetry(5000);
 //5	        5000
 //6	        5000
 ```
+
+## Redis protocol
+```
+
+Without any additional prompting, StackExchange.Redis will use the RESP2 protocol; this means that pub/sub requires a separatate connection to the server. RESP3 is a newer protocol
+(usually, but not always, available on v6 servers and above) which allows (smong other changes) pub/sub messages to be communicated on the *same* connection - which can be very
+desirable in servers with a large number of clients. The protocol handshake needs to happen very early in the connection, so *by default* the library does not attempt a RESP3 connection
+unless it has reason to expect it to work:
+
+This can be considered, in order:
+
+- the `HELLO` command has been disabled: RESP2 is used
+- a protocol *other than* `resp3` or `3` is specified: RESP2 is used
+- a protocol of `resp3` or `3` is specified: RESP3 is attempted (with fallback if it fails)
+- a version of at least 6 is specified: RESP3 is attempted (with fallback if it fails)
+- in all other scenarios: RESP2 is used
+

docs/ReleaseNotes.md

@@ -8,6 +8,7 @@ Current package versions:
 
 ## Unreleased
 
+- Adds: RESP3 support ([#2396 by mgravell](https://github.com/StackExchange/StackExchange.Redis/pull/2396)) - see https://stackexchange.github.io/StackExchange.Redis/Resp3
 - Fix [#2507](https://github.com/StackExchange/StackExchange.Redis/issues/2507): Pub/sub with multi-item payloads should be usable ([#2508 by mgravell](https://github.com/StackExchange/StackExchange.Redis/pull/2508))
 - Add: connection-id tracking (internal only, no public API) ([#2508 by mgravell](https://github.com/StackExchange/StackExchange.Redis/pull/2508))
 

docs/Resp3.md

@@ -0,0 +1,39 @@
+# RESP3 and StackExchange.Redis
+
+RESP2 and RESP3 are evolutions of the Redis protocol; the main differences are:
+
+1. RESP3 can carry out-of-band / "push" messages on a single connection, where-as RESP2 requires a separate connection for these messages
+2. RESP3 can (when appropriate) convey additional semantic meaning about returned payloads
+
+For most people, the first point is the main reason to consider RESP3, as in high-usage servers, this can halve the number of connections required.
+This is particularly useful in hosted environments where the number of inbound connections to the server is capped as part of a service plan.
+Alternatively, where users are currently choosing to disable the out-of-band connection to achieve this, they may now be able to re-enable this
+(for example, to receive server maintenance notifications) *without* incurring any additional connection overhead.
+
+There are no significant other differences, i.e. security, performance, etc all perform identically under both RESP2 and RESP3.
+
+RESP3 requires a Redis server version 6 or above; since the library cannot automatically know the server version *before* it has successfully connected,
+the library currently requires a hint to enable this mode, in particular, configuring the `ConfigurationOptions.Version` property to 6 (or above), or using
+`,version=6.0` (or above) on the configuration string.
+
+When using StackExchange.Redis, the second point only applies to:
+
+- Lua scripts that are invoked via the `ScriptEvaluate[Async](...)` or related APIs, that either:
+  - uses the `redis.setresp(3)` API and returns a value from `redis.[p]call(...)`
+  - returns a value that satisfies the [LUA to RESP3 type conversion rules](https://redis.io/docs/manual/programmability/lua-api/#lua-to-resp3-type-conversion)
+- ad-hoc commands (in particular: *modules*) that are invoked via the `Execute[Async](string command, ...)` API
+
+both which return `RedisResult`. **If you are not using these APIs, you do not need to do anything.**
+
+Historically, you could use the `RedisResult.Type` property to query the type of data returned (integer, string, etc). In particular:
+
+- two new properties are added: `RedisResult.Resp2Type` and `RedisResult.Resp3Type`
+  - the `Resp3Type` property exposes the new semantic data (when using RESP3), for example it can indicate that a value is a double-precision number, a boolean, a map, etc (types that did not historically exist)
+  - the `Resp2Type` property exposes the same value that *would* have been returned if this data had been returned over RESP2
+  - the `Type` property is now marked obsolete, but functions identically to `Resp2Type`, so that pre-existing code (for example, that has a `switch` on the type) is not impacted by RESP3
+- the `ResultType.MultiBulk` is superseded by `ResultType.Array` (this is a nomenclature change only; they are the same value and function identically)
+
+No changes to existing code are *required*, but:
+
+1. to prevent build warnings, replace usage of `ResultType.MultiBulk` with `ResultType.Array`, and usage of `RedisResult.Type` with `RedisResult.Resp2Type`
+2. if you wish to exploit the additional semantic data when using RESP3, use `RedisResult.Resp3Type` where appropriate

docs/index.md

@@ -39,6 +39,7 @@ Documentation
 - [Transactions](Transactions) - how atomic transactions work in redis
 - [Events](Events) - the events available for logging / information purposes
 - [Pub/Sub Message Order](PubSubOrder) - advice on sequential and concurrent processing
+- [Using RESP3](Resp3) - information on using RESP3
 - [ServerMaintenanceEvent](ServerMaintenanceEvent) - how to listen and prepare for hosted server maintenance (e.g. Azure Cache for Redis)
 - [Streams](Streams) - how to use the Stream data type
 - [Where are `KEYS` / `SCAN` / `FLUSH*`?](KeysScan) - how to use server-based commands

src/StackExchange.Redis/APITypes/LatencyHistoryEntry.cs

@@ -13,7 +13,7 @@ private sealed class Processor : ArrayResultProcessor<LatencyHistoryEntry>
     {
         protected override bool TryParse(in RawResult raw, out LatencyHistoryEntry parsed)
         {
-            if (raw.Type == ResultType.MultiBulk)
+            if (raw.Resp2TypeArray == ResultType.Array)
             {
                 var items = raw.GetItems();
                 if (items.Length >= 2

src/StackExchange.Redis/APITypes/LatencyLatestEntry.cs

@@ -13,7 +13,7 @@ private sealed class Processor : ArrayResultProcessor<LatencyLatestEntry>
     {
         protected override bool TryParse(in RawResult raw, out LatencyLatestEntry parsed)
         {
-            if (raw.Type == ResultType.MultiBulk)
+            if (raw.Resp2TypeArray == ResultType.Array)
             {
                 var items = raw.GetItems();
                 if (items.Length >= 4

src/StackExchange.Redis/ChannelMessageQueue.cs

@@ -1,6 +1,7 @@
 using System;
 using System.Collections.Generic;
 using System.Reflection;
+using System.Runtime.CompilerServices;
 using System.Threading;
 using System.Threading.Channels;
 using System.Threading.Tasks;

src/StackExchange.Redis/ClientInfo.cs

@@ -280,7 +280,7 @@ private class ClientInfoProcessor : ResultProcessor<ClientInfo[]>
         {
             protected override bool SetResultCore(PhysicalConnection connection, Message message, in RawResult result)
             {
-                switch(result.Type)
+                switch(result.Resp2TypeBulkString)
                 {
                     case ResultType.BulkString:
                         var raw = result.GetString();

src/StackExchange.Redis/CommandTrace.cs

@@ -73,9 +73,9 @@ private class CommandTraceProcessor : ResultProcessor<CommandTrace[]>
         {
             protected override bool SetResultCore(PhysicalConnection connection, Message message, in RawResult result)
             {
-                switch(result.Type)
+                switch(result.Resp2TypeArray)
                 {
-                    case ResultType.MultiBulk:
+                    case ResultType.Array:
                         var parts = result.GetItems();
                         CommandTrace[] arr = new CommandTrace[parts.Length];
                         int i = 0;

src/StackExchange.Redis/Condition.cs

@@ -563,7 +563,7 @@ internal override bool TryValidate(in RawResult result, out bool value)
                         return true;
 
                     default:
-                        switch (result.Type)
+                        switch (result.Resp2TypeBulkString)
                         {
                             case ResultType.BulkString:
                             case ResultType.SimpleString:
@@ -619,7 +619,7 @@ internal sealed override IEnumerable<Message> CreateMessages(int db, IResultBox?
 
             internal override bool TryValidate(in RawResult result, out bool value)
             {
-                switch (result.Type)
+                switch (result.Resp2TypeBulkString)
                 {
                     case ResultType.BulkString:
                     case ResultType.SimpleString:
@@ -692,7 +692,7 @@ internal sealed override IEnumerable<Message> CreateMessages(int db, IResultBox?
 
             internal override bool TryValidate(in RawResult result, out bool value)
             {
-                switch (result.Type)
+                switch (result.Resp2TypeBulkString)
                 {
                     case ResultType.BulkString:
                     case ResultType.SimpleString:
@@ -749,7 +749,7 @@ internal sealed override IEnumerable<Message> CreateMessages(int db, IResultBox?
 
             internal override bool TryValidate(in RawResult result, out bool value)
             {
-                switch (result.Type)
+                switch (result.Resp2TypeBulkString)
                 {
                     case ResultType.BulkString:
                     case ResultType.SimpleString:
@@ -806,7 +806,7 @@ internal sealed override IEnumerable<Message> CreateMessages(int db, IResultBox?
 
             internal override bool TryValidate(in RawResult result, out bool value)
             {
-                switch (result.Type)
+                switch (result.Resp2TypeBulkString)
                 {
                     case ResultType.Integer:
                         var parsedValue = result.AsRedisValue();