Skip to content

Commit 806e455

Browse files
erikjohnstonerikjohnston
committed
Simplified sliding sync
1 parent 863c694 commit 806e455

File tree

1 file changed

+345
-0
lines changed

1 file changed

+345
-0
lines changed

Diff for: ‎proposals/4186-simplified-sliding-sync.md

+345
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,345 @@
1+
# MSC4186: Simplified Sliding Sync
2+
3+
The current `/sync` endpoint scales badly as the number of rooms on an account increases. It scales badly because all
4+
rooms are returned to the client, incremental syncs are unbounded and slow down based on how long the user has been
5+
offline, and clients cannot opt-out of a large amount of extraneous data such as receipts. On large accounts with
6+
thousands of rooms, the initial sync operation can take tens of minutes to perform. This significantly delays the
7+
initial login to Matrix clients, and also makes incremental sync very heavy when resuming after any significant pause in
8+
usage.
9+
10+
Note: this is a “simplified” version of the sliding sync API proposed in
11+
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575), based on paring back that API based
12+
on real world use cases and usages.
13+
14+
15+
# Goals
16+
17+
This improved `/sync` mechanism has a number of goals:
18+
19+
- Sync time should be independent of the number of rooms you are in.
20+
- Time from launch to confident usability should be as low as possible.
21+
- Time from login on existing accounts to usability should be as low as possible.
22+
- Bandwidth should be minimized.
23+
- Support lazy-loading of things like read receipts (and avoid sending unnecessary data to the client)
24+
- Support informing the client when room state changes from under it, due to state resolution.
25+
- Clients should be able to work correctly without ever syncing in the full set of rooms they’re in.
26+
- Don’t incremental sync rooms you don’t care about.
27+
- Servers should not need to store all past since tokens. If a since token has been discarded we should gracefully
28+
degrade to initial sync.
29+
30+
These goals shaped the design of this proposal.
31+
32+
33+
# Proposal
34+
35+
The core differences between sync v2 and simplified sliding sync are:
36+
37+
- The server initially only sends the most recent N rooms to the client (where N is specified by the lcient), which then
38+
can paginate in older rooms in subsequent requests
39+
- The client can configure which information the server will return for different sets of rooms (e.g. a smaller timeline
40+
limit for older rooms).
41+
- The client can filter what rooms it is interested in
42+
- The client can maintain multiple sync loops (with some caveats)
43+
44+
The basic operation is similar between sync v2 and simplified sliding sync: both use long-polling with tokens to fetch
45+
updates from the server. I.e., the basic operation of both APIs is to do an “initial” request and then repeatedly call
46+
the API supplying the token returned in the previous response in the subsequent “incremental” request.
47+
48+
49+
## Lists and room subscriptions
50+
51+
The core component of a sliding sync request is “lists”, which specify what information to return about which rooms.
52+
Each list specifies some filters on rooms (e.g. ignore spaces), the range of filtered rooms to select (e.g. the most
53+
recent 20 filtered rooms), and the config for the data to return for those rooms (e.g. the required state, timeline
54+
limit, etc). The order of rooms is always done based on when the server received the most recent event for the room.
55+
56+
The client can also specify config for specific rooms if it has their room ID, these are known as room subscriptions.
57+
58+
Multiple lists and subscriptions can be specified in a request. If a room matches multiple lists/subscriptions then the
59+
config is “combined” to be the superset of all configs (e.g. take the maximum timeline limit). See below for the exact
60+
algorithm.
61+
62+
The server tracks what data has been sent to the client in which rooms. If a room matches a list or subscription that
63+
hasn’t been sent down before, then the server will respond with the full metadata about the room indicated by `initial:
64+
true`. If a room stops matching a list (i.e. it falls out of range) then no further updates will be sent until it starts
65+
matching a list again, at which point the missing updates (limited by the `timeline_limit`) will be sent down. However,
66+
as clients are now expected to paginate all rooms in the room list in the background (in order to correctly order and
67+
search them), the act of a room falling out of range is a temporary edge-case.
68+
69+
70+
## Pagination
71+
72+
Pagination is achieved by the client increasing the ranges of one (or more) lists.
73+
74+
For example an initial request might have a list called `all_rooms` specifying a range of `0..20` in the initial
75+
request, and the server will respond with the top 20 rooms (by most recently updated). On the second request the client
76+
may change the range to `0..100`, at which point the server will respond with the top 100 rooms that either a) weren’t
77+
sent down in the first request, or b) have updates since the first request.
78+
79+
Clients can increase and decrease the ranges as they see fit. A common approach would be to start with a small window
80+
and grow that until the range covers all the rooms. After some threshold of the app being offline it may reduce the
81+
range back down and incrementally grow it again. This allows for ensuring that a limited amount of data is requested at
82+
once, to improve response times.
83+
84+
85+
## Connections
86+
87+
Clients can have multiple “connections” (i.e. sync loops) with the server, so long as each connection has a different
88+
`conn_id` set in the request.
89+
90+
Clients must only have a single request in-flight at any time per connection (clients can have multiple connections by
91+
specifying a unique `conn_id`). If a client needs to send another request before receiving a response to an in-flight
92+
request (e.g. for retries or to change parameters) the client *must* cancel the in-flight request and *not* process any
93+
response it receives for it.
94+
95+
In particular, a client must use the returned `pos` value in a response as the `since` param in exactly one request that
96+
the client will process the response for. Clients must be careful to ensure that when processing a response any new
97+
requests use the new `pos`, and any in-flight requests using an old `pos` are canceled.
98+
99+
The server cannot assume that a client has received a response until it receives a new request with the `since` token
100+
set to the `pos` it returned in the response. The server must ensure that any per-connection state it tracks correctly
101+
handles receiving multiple requests with the same `since` token (e.g. the client retries the request or decides to
102+
cancel and resend a request with different parameters).
103+
104+
A server may decide to “expire” connections, either to free resources or because the server thinks it would be faster
105+
for the client to start from scratch (e.g. because there are many updates to send down). This is done by responding with
106+
a 400 HTTP status and an error code of `M_UNKNOWN_POS`.
107+
108+
## List configuration
109+
110+
**TODO**, these are the same as in [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
111+
112+
- Required state format
113+
- The filters
114+
- Lazy loading of members
115+
- Combining room config
116+
117+
## Room config changes
118+
119+
When a room comes in and out of different lists or subscriptions, the effective `timeline_limit` and `required_state`
120+
parameters may change. This section outlines how the server should handle these cases.
121+
122+
If the `timeline_limit` *increases* then the server *may* choose to send down more historic data. This is to support the
123+
ability to get more history for certain rooms, e.g. when subscribing to the currently visible rooms in the list to
124+
precache their history. This is done by setting `unstable_expanded_timeline` to true and sending down the last N events
125+
(this may include events that have already been sent down). The server may choose not to do this if it believes it has
126+
already sent down the appropriate number of events.
127+
128+
If new entries are added to `required_state` then the server must send down matching current state events.
129+
130+
\[NOTE (erikj): we probably want to replace the convoluted nature of “trickling events” via expanding the timeline with
131+
a “batch /messages” API, or otherwise rethink how we trickle down events.\]
132+
133+
## Extensions
134+
135+
We anticipate that as more features land in Matrix, different kinds of data will also want to be synced to clients. Sync
136+
v2 did not have any first-class support to opt-in to new data. Sliding Sync does have support for this via "extensions".
137+
Extensions also allow this proposal to be broken up into more manageable sections. Extensions are requested by the
138+
client in a dedicated extensions block.
139+
140+
In an effort to reduce the size of this proposal, extensions will be done in separate MSCs. There will be extensions
141+
for:
142+
143+
- To Device Messaging \- MSC3885
144+
- End-to-End Encryption \- MSC3884
145+
- Typing Notifications \- MSC3961
146+
- Receipts \- MSC3960
147+
- Presence \- presence in sync v2: spec
148+
- Account Data \- account\_data in sync v2: MSC3959
149+
- Threads
150+
151+
**TODO** explain how these interact with the room lists, this is the same as in
152+
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575)
153+
154+
## Request format
155+
156+
```javascript
157+
{
158+
"conn_id": "<conn_id>", // Client chosen ID of the connection, c.f. "Connections"
159+
160+
// The set of room lists
161+
"lists": {
162+
// An arbitrary string which the client is using to refer to this list for this connection.
163+
"<list-name>": {
164+
165+
// Sliding window ranges, c.f. Lists and room subscriptions
166+
"ranges": [[0, 10]],
167+
168+
// Filters to apply to the list.
169+
"filters": {
170+
// Flag which only returns rooms present (or not) in the DM section of account data.
171+
// If unset, both DM rooms and non-DM rooms are returned. If false, only non-DM rooms
172+
// are returned. If true, only DM rooms are returned.
173+
"is_dm": true|false|null,
174+
175+
// Flag which only returns rooms which have an `m.room.encryption` state event. If unset,
176+
// both encrypted and unencrypted rooms are returned. If false, only unencrypted rooms
177+
// are returned. If true, only encrypted rooms are returned.
178+
"is_encrypted": true|false|null,
179+
180+
// Flag which only returns rooms which have an `m.room.encryption` state event. If unset,
181+
// both encrypted and unencrypted rooms are returned. If false, only unencrypted rooms
182+
// are returned. If true, only encrypted rooms are returned.
183+
"is_invite": true|false|null,
184+
185+
// If specified, only rooms where the `m.room.create` event has a `type` matching one
186+
// of the strings in this array will be returned. If this field is unset, all rooms are
187+
// returned regardless of type. This can be used to get the initial set of spaces for an account.
188+
// For rooms which do not have a room type, use 'null' to include them.
189+
"room_types": [ ... ],
190+
191+
// Same as "room_types" but inverted. This can be used to filter out spaces from the room list.
192+
// If a type is in both room_types and not_room_types, then not_room_types wins and they are
193+
// not included in the result.
194+
"not_room_types": [ ... ],
195+
},
196+
197+
// The maximum number of timeline events to return per response.
198+
"timeline_limit": 10,
199+
200+
// Required state for each room returned. An array of event type and state key tuples.
201+
// Elements in this array are ORd together to produce the final set of state events
202+
// to return. One unique exception is when you request all state events via ["*", "*"]. When used,
203+
// all state events are returned by default, and additional entries FILTER OUT the returned set
204+
// of state events. These additional entries cannot use '*' themselves.
205+
// For example, ["*", "*"], ["m.room.member", "@alice:example.com"] will _exclude_ every m.room.member
206+
// event _except_ for @alice:example.com, and include every other state event.
207+
// In addition, ["*", "*"], ["m.space.child", "*"] is an error, the m.space.child filter is not
208+
// required as it would have been returned anyway.
209+
"required_state": [ ... ],
210+
}
211+
},
212+
213+
// The set of room subscriptions
214+
"room_subscriptions": {
215+
// The key is the room to subscribe to.
216+
"!foo:example.com": {
217+
// These have the same meaning as in `lists` section
218+
"timeline_limit": 10,
219+
"required_state": [ ... ],
220+
}
221+
},
222+
223+
// c.f. "Extensions"
224+
"extensions": {
225+
}
226+
}
227+
```
228+
229+
230+
## Response format
231+
232+
```javascript
233+
{
234+
// The position to use as the `since` token in the next sliding sync request.
235+
// c.f. Connections.
236+
"pos": "<opaque string>",
237+
238+
// Information about the lists supplied in the request.
239+
"lists": {
240+
// Matches the list name supplied by the client in the request
241+
"<list-name>" {
242+
// The total number of rooms that match the list's filter.
243+
"count": 1234,
244+
}
245+
},
246+
247+
// Aggregated rooms from lists and room subscriptions. There will be one entry per room, even if
248+
// the room appears in multiple lists and/or room subscriptions.
249+
"rooms": {
250+
"!foo:example.com": {
251+
// The room name, if one exists. Only sent initially and when it changes.
252+
"name": str|null,
253+
// The room avatar, if one exists. Only sent initially and when it changes.
254+
"avatar_url": str|null,
255+
// The "heroes" for the room, if there is no room name. Only sent initially and when it changes.
256+
"heroes": [
257+
{"user_id":"@alice:example.com","displayname":"Alice","avatar_url":"mxc://..."},
258+
],
259+
260+
// Flag which is set when this is the first time the server is sending this data on this connection.
261+
// When set the client must replace any stored metadata for the room with the new data. In
262+
// particular, the state must be replaced with the state in `required_state`.
263+
"initial": true|null,
264+
265+
// Same as in sync v2. Indicates whether there are more events to fetch than those in the timeline.
266+
"limited:" true|null,
267+
// Indicates if we have "expanded" the timeline due to the timeline_limit changing, c.f. Room config
268+
// changes above.
269+
"unstable_expanded_timeline": true|null,
270+
// The list of events, sorted least to most recent.
271+
"timeline": [ ... ],
272+
// The current state of the room as a list of events
273+
"required_state": [ ... ],
274+
// The number of timeline events which have just occurred and are not historical.
275+
// The last N events are 'live' and should be treated as such.
276+
"num_live": 1,
277+
// Same as sync v2, passed to `/messages` to fetch more past events.
278+
"prev_batch": "...",
279+
280+
// For invites this is the stripped state of the room at the time of invite
281+
"invite_state": [ .. ],
282+
283+
// For knocks this is the stripped state of the room at time of knock
284+
"knock_state": [ .. ],
285+
286+
// Whether the room is a DM room.
287+
"is_dm": true|null,
288+
289+
// An opaque integer that can be used to sort the rooms by "Bump Stamp"
290+
"bump_stamp": 1,
291+
292+
// These are the same as sync v2.
293+
"joined_count": 1,
294+
"invited_count": 1,
295+
"notification_count": 1,
296+
"highlight_count": 1,
297+
}
298+
},
299+
300+
"extensions": {
301+
},
302+
}
303+
```
304+
305+
306+
# Alternatives / changes
307+
308+
There are a number of potential changes that we could make.
309+
310+
## Pagination
311+
312+
In practice, having the client specify the ranges to use for the lists is often sub-optimal. The client generally wants
313+
to have the sync request return as quickly as possible, but it doesn't know how much data the server has to return and
314+
so whether to increase or decrease the range.
315+
316+
An alternative is for the client to specify a `page_size`, where the server sends down at most `page_size` number of
317+
rooms. If there are more rooms to send to the client (beyond `page_size`), then the client can request to "paginate" in
318+
these missed updates in subsequent updates.
319+
320+
Since this would require client side changes, this should be explored in a separate MSC.
321+
322+
## Timeline event trickling
323+
324+
If the `timeline_limit` is increased then the server will send down historic data (c.f. "Room config changes"), which
325+
allows the clients to easily preload more history in recent rooms.
326+
327+
This mechanism is fiddly to implement, and ends up resending down events that we have previously sent to the client.
328+
329+
A simpler alternative is to use `/messages` to fetch the history. This has two main problems: 1) clients generally want
330+
to preload history for multiple rooms at once, and 2) `/messages` can be slow if it tries to backfill over federation.
331+
332+
We could implement a bulk `/messages` endpoint, where the client would specify multiple rooms and `prev_batch` tokens.
333+
We can also add a flag to disable attempting to backfill over pagination (to match the behaviour of the sync timeline).
334+
335+
## `required_state` response format
336+
337+
The format of returned state in `required_state` is a list of events. This does now allow the server to indicate if a
338+
"state reset" has happened which removed an entry from the state entirely (rather than it being replaced with another
339+
event).
340+
341+
This is particularly problematic if the user gets "state reset" out of the room, where the server has no mechanism to
342+
indicate to the client that the user has effectively left the room (the server has no leave event to return).
343+
344+
We may want to allow special entries in the `required_state` list of the form
345+
`{"type": .., "state_key": .., content: null}` to indicate that the state entry has been removed.

0 commit comments

Comments
 (0)