Missing sync events during historical batch imports

[Fizzadar]

Split out from #9424 (comment) onwards. I have inlined the relevant context from that issue below.

We have re-created the following issue; I've included the event list and trimmed sync responses in this gist: https://gist.github.com/Fizzadar/316dabfe9be30ebaa48e288de2380de1. Some notes:

• The first sync containing the invite seems to correctly pull everything
• The second sync includes member information but misses out the prior state
• The invite also appears to arrive after the room creation event
• Prior syncs do not reference the room at all (can provide JSON dumps if useful)

Notably, this is only occuring during initial backfill using MSC2716 batch sends. When this is disabled on the bridge we have not yet replicated the problem seen here.

The DAG:

synapse=> SELECT stream_ordering, type, state_key, event_id, (ej.json::json)->'prev_events' AS prev_events FROM events
LEFT JOIN event_json AS ej USING (event_id)
WHERE events.room_id ='!SjekxtmsBigrUMcMkX:beeper-dev.com' ORDER BY stream_ordering ASC;
 stream_ordering |             type             |                          state_key                           |                   event_id                   |                   prev_events
-----------------+------------------------------+--------------------------------------------------------------+----------------------------------------------+--------------------------------------------------
         -113203 | org.matrix.msc2716.insertion |                                                    | $G5GLwX6lc8MXKqwauoM1p9lkLD6S5GfwcosFlPra0ws | []
         -113202 | m.room.encrypted             |                                                    | $I3PsF44AMu5I_XOZQ2Hm1cOhcdLJCbA8NcAMS69hLEE | ["$G5GLwX6lc8MXKqwauoM1p9lkLD6S5GfwcosFlPra0ws"]
         -113201 | m.room.encrypted             |                                                    | $hIO0ZI-ZYr8gWGkWtvBiPfCA0jKdjiZP7PCnEfuutTY | ["$I3PsF44AMu5I_XOZQ2Hm1cOhcdLJCbA8NcAMS69hLEE"]
         -113200 | m.room.encrypted             |                                                    | $-jWSmA7NJ1dbl5b86K8FAKN2ydt-qoOw8ftCdSEN4A4 | ["$hIO0ZI-ZYr8gWGkWtvBiPfCA0jKdjiZP7PCnEfuutTY"]
         -113199 | m.room.encrypted             |                                                    | $5EtpKKW7g5HV5oCweU6vlbGjptR095p-kZSSSy2tBgs | ["$-jWSmA7NJ1dbl5b86K8FAKN2ydt-qoOw8ftCdSEN4A4"]
         -113198 | m.room.encrypted             |                                                    | $5aVzqGvybXzV3Xj3AwA8SQKqp2X2Z7RLH7IGX5VBeX4 | ["$5EtpKKW7g5HV5oCweU6vlbGjptR095p-kZSSSy2tBgs"]
         -113197 | org.matrix.msc2716.batch     |                                                    | $KkSRlRidPiOkkLEuiHs3gWLyCqCdQTHAVYJxzhDIzcM | ["$5aVzqGvybXzV3Xj3AwA8SQKqp2X2Z7RLH7IGX5VBeX4"]
         -111135 | org.matrix.msc2716.insertion |                                                    | $LQBf_c9-1stBXkPOLqZwkJgMg5YBCp448qcUwKLN6ks | []
         -111134 | m.room.encrypted             |                                                    | $RzKCB8rV0a9jfQVfbJeGvgGIZtT_X7ODb-78LUhXpXs | ["$LQBf_c9-1stBXkPOLqZwkJgMg5YBCp448qcUwKLN6ks"]
         -111133 | org.matrix.msc2716.batch     |                                                    | $l_jGmPdlA73wDzS3_gRwC67co1pTnx9-QBufta10xWI | ["$RzKCB8rV0a9jfQVfbJeGvgGIZtT_X7ODb-78LUhXpXs"]
         -111132 | org.matrix.msc2716.insertion |                                                    | $_mT6Awgb70IjwseWpCUeU0O4tm_iT7OBe4IPLJYUHUU | ["$a6xyChJTR8wJLsfYfGI8Tu1W5cXrtohkBRdwCSsyG68"]
         2633404 | m.room.create                |                                                    | $ZMCw8gIYOeNmC2mGNKiAdxbRdwuhDiA9gNaM8EwDlRM | []
         2633421 | m.room.member                | @_adam4_whatsapp_xxx:beeper-dev.com                | $a6LZbYMvr5yYRcFc0uethSHLVC_0l7LRUr0nCZXB2lQ | ["$ZMCw8gIYOeNmC2mGNKiAdxbRdwuhDiA9gNaM8EwDlRM"]
         2633434 | m.room.power_levels          |                                                    | $S2rnt0m3dfbgC0ySGI3RGSdIjNqhgjjilZ2Gm7jGtvg | ["$a6LZbYMvr5yYRcFc0uethSHLVC_0l7LRUr0nCZXB2lQ"]
         2633448 | m.room.join_rules            |                                                    | $5hkAty1mFIMtuFINEh8wA3OyIgnO6Qxh1HgCTT9c1p0 | ["$S2rnt0m3dfbgC0ySGI3RGSdIjNqhgjjilZ2Gm7jGtvg"]
         2633461 | m.room.history_visibility    |                                                    | $nNR6iQSvWPpoKRvfpXJXU_sWpnk8eCwTFJD9zwC2rfo | ["$5hkAty1mFIMtuFINEh8wA3OyIgnO6Qxh1HgCTT9c1p0"]
         2633469 | m.room.guest_access          |                                                    | $iNJMDkjnFwBekoCZASqZmiNtH8UMLsPc_1VHJD6IFpU | ["$nNR6iQSvWPpoKRvfpXJXU_sWpnk8eCwTFJD9zwC2rfo"]
         2633478 | m.bridge                     | net.maunium.whatsapp://whatsapp/xxx@s.whatsapp.net | $p5fs4ik0Lnj-mIbw-cjPHX_aWzx0x6O_rIrhSi8_1C8 | ["$iNJMDkjnFwBekoCZASqZmiNtH8UMLsPc_1VHJD6IFpU"]
         2633492 | uk.half-shot.bridge          | net.maunium.whatsapp://whatsapp/xxx@s.whatsapp.net | $tAZ3y00piOU_54qs0MuK-HRPivXNRBmN8x6yY5KWXkk | ["$p5fs4ik0Lnj-mIbw-cjPHX_aWzx0x6O_rIrhSi8_1C8"]
         2633503 | m.room.avatar                |                                                    | $BZOaOEhozp28cCJtdh8A6R2hKgXiuDmd1NWI2vEuyPs | ["$tAZ3y00piOU_54qs0MuK-HRPivXNRBmN8x6yY5KWXkk"]
         2633515 | m.room.encryption            |                                                    | $qTv8IJZIdlH_3S7atbGTrd-7p7UEt4siocS0vz8-K3Y | ["$BZOaOEhozp28cCJtdh8A6R2hKgXiuDmd1NWI2vEuyPs"]
         2633523 | m.room.name                  |                                                    | $joClbFehqiqiVOuYYDWGJeYaozJnaccZgVog8j4bB6g | ["$qTv8IJZIdlH_3S7atbGTrd-7p7UEt4siocS0vz8-K3Y"]
         2633533 | m.room.topic                 |                                                    | $WP0ALeVvT2oCRSt9qs4_bE2O5AKsu_4GqbTCPFwaKa4 | ["$joClbFehqiqiVOuYYDWGJeYaozJnaccZgVog8j4bB6g"]
         2633547 | m.room.member                | @_adam4_whatsapp_bot:beeper-dev.com                | $hq4kZZPt4ItYdBF_I_AVITFHOZOZ2b_Av8ss3F06byg | ["$WP0ALeVvT2oCRSt9qs4_bE2O5AKsu_4GqbTCPFwaKa4"]
         2633571 | m.room.member                | @adam4:beeper-dev.com                              | $DyFoSr8CbRUyexbx6_yuZxgS4S7lAEHDYgldiRGnppc | ["$hq4kZZPt4ItYdBF_I_AVITFHOZOZ2b_Av8ss3F06byg"]
         2633582 | m.room.member                | @adam4:beeper-dev.com                              | $51gzIcw5d1fvbNLqYXdBwma669-H8DZlz0UzsavMBbU | ["$DyFoSr8CbRUyexbx6_yuZxgS4S7lAEHDYgldiRGnppc"]
         2633590 | m.room.member                | @_adam4_whatsapp_bot:beeper-dev.com                | $Sx9mswlxUJs1rRL4UyHARmY7V0h8_cH_c_ZDnwTn3cc | ["$51gzIcw5d1fvbNLqYXdBwma669-H8DZlz0UzsavMBbU"]
         2633601 | fi.mau.dummy.portal_created  |                                                    | $a6xyChJTR8wJLsfYfGI8Tu1W5cXrtohkBRdwCSsyG68 | ["$Sx9mswlxUJs1rRL4UyHARmY7V0h8_cH_c_ZDnwTn3cc"]
         2633614 | fi.mau.dummy.pre_backfill    |                                                    | $E97WApcmVmCJ6nt2WL3TnuxhyCLnPl1dt-t6p25ZLwY | ["$a6xyChJTR8wJLsfYfGI8Tu1W5cXrtohkBRdwCSsyG68"]
         2633633 | fi.mau.dummy.backfill_end    |                                                    | $NILDi9SfVDgaW72alIEsc5ESzdJSoLip_c7klMAeuPo | ["$E97WApcmVmCJ6nt2WL3TnuxhyCLnPl1dt-t6p25ZLwY"]
(30 rows)

Note that this DAG looks similar to rooms where this problem did not occur.

[Fizzadar]

Based upon reviewing the MSC and room batch code the prev_event field should be present for all insertion batches. I think this is possibly a separate issue.

For the sync itself it seems something is confusing the sync handler because none of the batch events should ever come down via sync (noted as an issue on the MSC). In our live tests we do see batches come down with sync requests which I now believe to be incorrect.

Have just checked and replaying the same sync request (account is untouched since the original issue) does include the missing m.bridge events as expected. It also includes the very first MSC2716 event, the only one with any prev_event content ($_mT6Awgb70IjwseWpCUeU0O4tm_iT7OBe4IPLJYUHUU).

[reivilibre]

Have just checked and replaying the same sync request (account is untouched since the original issue) does include the missing m.bridge events as expected. It also includes the very first MSC2716 event, the only one with any prev_event content ($_mT6Awgb70IjwseWpCUeU0O4tm_iT7OBe4IPLJYUHUU).

Very curious, so you're saying that replaying the sync request is giving you all the expected state events now?

Would you be happy to share a new trimmed extract of it?

Also, for the sake of tracking what's going on, would you be able to post the event content of the MSC2716 batch and insertion events? I don't think they're critical but it may help to show how this fits together; I'm still getting up to speed with MSC2716 so having the real world example would be beneficial personally.

[reivilibre]

I've been making scribbles on this graph of the DAG (→ = prev-events of, so newer events are generally at the top), thought attaching it here may be of use if others want to look at this too.

DAG

The scribbles themselves are probably less useful to others for the moment..

[Fizzadar]

Very curious, so you're saying that replaying the sync request is giving you all the expected state events now?
Would you be happy to share a new trimmed extract of it?

My apologies it does miss the same events (phew!), so the issue is consistent. Must've copied the wrong since token somewhere. Here's the sync response when I re-create it today: https://gist.github.com/Fizzadar/316dabfe9be30ebaa48e288de2380de1#file-2022-03-24-trimmed-sync-s2633573_17_338_6732159_1082514_541479_274711_265584_1-json

Working on getting the event content too will update here.

[Fizzadar]

Also, for the sake of tracking what's going on, would you be able to post the event content of the MSC2716 batch and insertion events? I don't think they're critical but it may help to show how this fits together; I'm still getting up to speed with MSC2716 so having the real world example would be beneficial personally.

👍 Just pulled this down: https://gist.github.com/Fizzadar/316dabfe9be30ebaa48e288de2380de1#file-events_content-json

[MadLittleMods]

@Fizzadar It's very hard to follow that gist and what's expected vs missing. Is it possible to more summarize that in terms of requests that happened.

ex.

1. Room create
2. /batch_send?prev_event_id=(which event)
   • Includes a few messages
3. /batch_send?batch_id=continue
4. /sync
   • Everything as expected
5. /sync?since=a
   • Missing $xxx (m.room.member invite)
6. etc (with more detail)

Then I can try to create a Complement test to reproduce and understand better.

---

For the sync itself it seems something is confusing the sync handler because none of the batch events should ever come down via sync (noted as an issue on the MSC). In our live tests we do see batches come down with sync requests which I now believe to be incorrect.

They can come down an initial sync. But they shouldn't come down an incremental sync (with ?since) because the stream-ordering token will be after the negative stream_ordering historical batch

[Fizzadar]

@MadLittleMods sure thing, describing the order below and I've also included the raw bridge logs specific for the room in question:

1. Client starts, initalSync happens before any bridge or backfill
2. Bridge starts, client signin/etc flow
3. Bridge starts backfill process - note this creates + backfills up to 20 rooms in parallel
4. Room create, initial state includes the m.bridge event as expected
5. /sync?since=s2633508_17_338_6732159_1082514_541479_274711_265584_1
   a. this file in the gist, contains all the room state as expected in the invite state dict
6. Bridge adds the user to the room via m.room.member state
7. First /batch_send?prev_event_id=$a6xyChJTR8wJLsfYfGI8Tu1W5cXrtohkBRdwCSsyG68
8. /sync?since=s2633573_17_338_6732159_1082514_541479_274711_265584_1
   a. this file in the gist, this is where the m.bridge state event ($p5fs4ik0Lnj-mIbw-cjPHX_aWzx0x6O_rIrhSi8_1C8 is missing + more)
   b. replaying this request today gives the same result
9. Second /batch_send?batch_id=IepWImCQ&prev_event_id=$a6xyChJTR8wJLsfYfGI8Tu1W5cXrtohkBRdwCSsyG68

Note that this is not consistent - of 20 rooms being created + backfilled in parallel there may be one or two that have missing state as described above.

I've added the raw logs from the bridge specific to the affected room in the gist.

[bradtgmurray]

I'm curious if the issue is using a sync token at a particular point in the event graph, and the inconsistency of whether we can reproduce it or not is just the frequency of the client syncing and where the graph for a particular room gets divided between syncs. I wonder if we could take a room that "worked" for us and then manually bump back the sync token up the event graph until we get a sync that doesn't include the m.bridge event anymore and should.

[MadLittleMods]

I'm curious if the issue is using a sync token at a particular point in the event graph, and the inconsistency of whether we can reproduce it or not is just the frequency of the client syncing and where the graph for a particular room gets divided between syncs. I wonder if we could take a room that "worked" for us and then manually bump back the sync token up the event graph until we get a sync that doesn't include the m.bridge event anymore and should.

This sounds pretty plausible. rooms.join.<roomId>.state.events should only include:

Updates to the state, between the time indicated by the since parameter, and the start of the timeline (or all state up to the start of the timeline, if since is not given, or full_state is true).

-- https://spec.matrix.org/v1.1/client-server-api/#get_matrixclientv3sync

When I look at the Synapse database dump of events to get the full picture and compare with the /sync responses, I'm not quite seeing that behavior. I've included and sorted in the stream tokens from (sync 1 ...) and (sync 2 ...) into the list. And I've added (1) and (2) if they appeared in the /sync state list accordingly (invite_state in the first sync case).

It's possible I'm interpreting those stream tokens wrong too 🤷

• s2633508_17_338_6732159_1082514_541479_274711_265584_1 -> 2633508
• s2633573_17_338_6732159_1082514_541479_274711_265584_1 ->2633573

synapse/synapse/types.py

Lines 436 to 439 in 3c41d87

	Live tokens start with an "s" followed by the "stream_ordering" id of the
	event it comes after. Historic tokens start with a "t" followed by the
	"topological_ordering" id of the event it comes after, followed by "-",
	followed by the "stream_ordering" id of the event it comes after.

synapse=> SELECT stream_ordering, type, state_key, event_id, (ej.json::json)->'prev_events' AS prev_events FROM events
LEFT JOIN event_json AS ej USING (event_id)
WHERE events.room_id ='!SjekxtmsBigrUMcMkX:beeper-dev.com' ORDER BY stream_ordering ASC;
 stream_ordering |             type             |                          state_key                           |                   event_id                   |                   prev_events
-----------------+------------------------------+--------------------------------------------------------------+----------------------------------------------+--------------------------------------------------
         -113203 | org.matrix.msc2716.insertion |                                                    | $G5GLwX6lc8MXKqwauoM1p9lkLD6S5GfwcosFlPra0ws | []
         -113202 | m.room.encrypted             |                                                    | $I3PsF44AMu5I_XOZQ2Hm1cOhcdLJCbA8NcAMS69hLEE | ["$G5GLwX6lc8MXKqwauoM1p9lkLD6S5GfwcosFlPra0ws"]
         -113201 | m.room.encrypted             |                                                    | $hIO0ZI-ZYr8gWGkWtvBiPfCA0jKdjiZP7PCnEfuutTY | ["$I3PsF44AMu5I_XOZQ2Hm1cOhcdLJCbA8NcAMS69hLEE"]
         -113200 | m.room.encrypted             |                                                    | $-jWSmA7NJ1dbl5b86K8FAKN2ydt-qoOw8ftCdSEN4A4 | ["$hIO0ZI-ZYr8gWGkWtvBiPfCA0jKdjiZP7PCnEfuutTY"]
         -113199 | m.room.encrypted             |                                                    | $5EtpKKW7g5HV5oCweU6vlbGjptR095p-kZSSSy2tBgs | ["$-jWSmA7NJ1dbl5b86K8FAKN2ydt-qoOw8ftCdSEN4A4"]
         -113198 | m.room.encrypted             |                                                    | $5aVzqGvybXzV3Xj3AwA8SQKqp2X2Z7RLH7IGX5VBeX4 | ["$5EtpKKW7g5HV5oCweU6vlbGjptR095p-kZSSSy2tBgs"]
         -113197 | org.matrix.msc2716.batch     |                                                    | $KkSRlRidPiOkkLEuiHs3gWLyCqCdQTHAVYJxzhDIzcM | ["$5aVzqGvybXzV3Xj3AwA8SQKqp2X2Z7RLH7IGX5VBeX4"]
         -111135 | org.matrix.msc2716.insertion |                                                    | $LQBf_c9-1stBXkPOLqZwkJgMg5YBCp448qcUwKLN6ks | []
         -111134 | m.room.encrypted             |                                                    | $RzKCB8rV0a9jfQVfbJeGvgGIZtT_X7ODb-78LUhXpXs | ["$LQBf_c9-1stBXkPOLqZwkJgMg5YBCp448qcUwKLN6ks"]
         -111133 | org.matrix.msc2716.batch     |                                                    | $l_jGmPdlA73wDzS3_gRwC67co1pTnx9-QBufta10xWI | ["$RzKCB8rV0a9jfQVfbJeGvgGIZtT_X7ODb-78LUhXpXs"]
         -111132 | org.matrix.msc2716.insertion |                                                    | $_mT6Awgb70IjwseWpCUeU0O4tm_iT7OBe4IPLJYUHUU | ["$a6xyChJTR8wJLsfYfGI8Tu1W5cXrtohkBRdwCSsyG68"]
         2633404 | m.room.create (1)            |                                                    | $ZMCw8gIYOeNmC2mGNKiAdxbRdwuhDiA9gNaM8EwDlRM | []
         2633421 | m.room.member (1) (2)        | @_adam4_whatsapp_xxx:beeper-dev.com                | $a6LZbYMvr5yYRcFc0uethSHLVC_0l7LRUr0nCZXB2lQ | ["$ZMCw8gIYOeNmC2mGNKiAdxbRdwuhDiA9gNaM8EwDlRM"]
         2633434 | m.room.power_levels          |                                                    | $S2rnt0m3dfbgC0ySGI3RGSdIjNqhgjjilZ2Gm7jGtvg | ["$a6LZbYMvr5yYRcFc0uethSHLVC_0l7LRUr0nCZXB2lQ"]
         2633448 | m.room.join_rules (1)        |                                                    | $5hkAty1mFIMtuFINEh8wA3OyIgnO6Qxh1HgCTT9c1p0 | ["$S2rnt0m3dfbgC0ySGI3RGSdIjNqhgjjilZ2Gm7jGtvg"]
         2633461 | m.room.history_visibility    |                                                    | $nNR6iQSvWPpoKRvfpXJXU_sWpnk8eCwTFJD9zwC2rfo | ["$5hkAty1mFIMtuFINEh8wA3OyIgnO6Qxh1HgCTT9c1p0"]
         2633469 | m.room.guest_access          |                                                    | $iNJMDkjnFwBekoCZASqZmiNtH8UMLsPc_1VHJD6IFpU | ["$nNR6iQSvWPpoKRvfpXJXU_sWpnk8eCwTFJD9zwC2rfo"]
         2633478 | m.bridge (1)                 | net.maunium.whatsapp://whatsapp/xxx@s.whatsapp.net | $p5fs4ik0Lnj-mIbw-cjPHX_aWzx0x6O_rIrhSi8_1C8 | ["$iNJMDkjnFwBekoCZASqZmiNtH8UMLsPc_1VHJD6IFpU"]
         2633492 | uk.half-shot.bridge (1)      | net.maunium.whatsapp://whatsapp/xxx@s.whatsapp.net | $tAZ3y00piOU_54qs0MuK-HRPivXNRBmN8x6yY5KWXkk | ["$p5fs4ik0Lnj-mIbw-cjPHX_aWzx0x6O_rIrhSi8_1C8"]
         2633503 | m.room.avatar (1)            |                                                    | $BZOaOEhozp28cCJtdh8A6R2hKgXiuDmd1NWI2vEuyPs | ["$tAZ3y00piOU_54qs0MuK-HRPivXNRBmN8x6yY5KWXkk"]
-------- 2633508 (sync 1 while invited)
         2633515 | m.room.encryption (1)        |                                                    | $qTv8IJZIdlH_3S7atbGTrd-7p7UEt4siocS0vz8-K3Y | ["$BZOaOEhozp28cCJtdh8A6R2hKgXiuDmd1NWI2vEuyPs"]
         2633523 | m.room.name (1)              |                                                    | $joClbFehqiqiVOuYYDWGJeYaozJnaccZgVog8j4bB6g | ["$qTv8IJZIdlH_3S7atbGTrd-7p7UEt4siocS0vz8-K3Y"]
         2633533 | m.room.topic (1)             |                                                    | $WP0ALeVvT2oCRSt9qs4_bE2O5AKsu_4GqbTCPFwaKa4 | ["$joClbFehqiqiVOuYYDWGJeYaozJnaccZgVog8j4bB6g"]
         2633547 | m.room.member (2)            | @_adam4_whatsapp_bot:beeper-dev.com                | $hq4kZZPt4ItYdBF_I_AVITFHOZOZ2b_Av8ss3F06byg | ["$WP0ALeVvT2oCRSt9qs4_bE2O5AKsu_4GqbTCPFwaKa4"]
         2633571 | m.room.member (1)            | @adam4:beeper-dev.com                              | $DyFoSr8CbRUyexbx6_yuZxgS4S7lAEHDYgldiRGnppc | ["$hq4kZZPt4ItYdBF_I_AVITFHOZOZ2b_Av8ss3F06byg"]
-------- 2633573 (sync 2 after being joined)
         2633582 | m.room.member (2)            | @adam4:beeper-dev.com                              | $51gzIcw5d1fvbNLqYXdBwma669-H8DZlz0UzsavMBbU | ["$DyFoSr8CbRUyexbx6_yuZxgS4S7lAEHDYgldiRGnppc"]
         2633590 | m.room.member                | @_adam4_whatsapp_bot:beeper-dev.com                | $Sx9mswlxUJs1rRL4UyHARmY7V0h8_cH_c_ZDnwTn3cc | ["$51gzIcw5d1fvbNLqYXdBwma669-H8DZlz0UzsavMBbU"]
         2633601 | fi.mau.dummy.portal_created  |                                                    | $a6xyChJTR8wJLsfYfGI8Tu1W5cXrtohkBRdwCSsyG68 | ["$Sx9mswlxUJs1rRL4UyHARmY7V0h8_cH_c_ZDnwTn3cc"]
         2633614 | fi.mau.dummy.pre_backfill    |                                                    | $E97WApcmVmCJ6nt2WL3TnuxhyCLnPl1dt-t6p25ZLwY | ["$a6xyChJTR8wJLsfYfGI8Tu1W5cXrtohkBRdwCSsyG68"]
         2633633 | fi.mau.dummy.backfill_end    |                                                    | $NILDi9SfVDgaW72alIEsc5ESzdJSoLip_c7klMAeuPo | ["$E97WApcmVmCJ6nt2WL3TnuxhyCLnPl1dt-t6p25ZLwY"]
(30 rows)

@Fizzadar I'm not confident in the correct case though. Can you post the Synapse db dump and the two /sync results when it's working correctly?

Actual reproduction gist questions

I'm still fuzzy on some details:

1. Who is the creator of the room?
   • I assume @_adam4_whatsapp_xxx:beeper-dev.com (is this the same as @_adam4_whatsapp_bot:beeper-dev.com?)
2. Who is the bot?
   • I assume @_adam4_whatsapp_bot:beeper-dev.com
3. Who is doing the /sync'ing?
   • I assume @adam4:beeper-dev.com
4. Are we using the invite field with /createRoom?
5. In the /sync responses, are we expecting the m.bridge event to be in rooms.join.<roomId>.timeline.events or rooms.join.<roomId>.state.events?

Complement test

I wrote a Complement test to try to reproduce the problem but I am seeing some differences:

1. Does the test setup look correct to what you're doing? Test case to try to reproduce missing state in sync (possibly related to MSC2716) complement#352
2. With the current test, in the first incremental sync, I'm not seeing the m.bridge as part of the first rooms.invite.<roomId>.invite_state.events like you do.
   • Is this necessary?
   • The spec isn't clear to me what should be included in invite_state

[Fizzadar]

1. Who is the creator of the room?
   * I assume `@_adam4_whatsapp_xxx:beeper-dev.com` (is this the same as `@_adam4_whatsapp_bot:beeper-dev.com`?)
2. Who is the bot?
   * I assume `@_adam4_whatsapp_bot:beeper-dev.com`
3. Who is doing the `/sync`'ing? 
   * I assume `@adam4:beeper-dev.com`

All correct.

4. Are we using the `invite` field with `/createRoom`?

Yes - for the bot user, the sync user @adam4:beeper-dev.com is invited after the room is created.

5. In the `/sync` responses, are we expecting the `m.bridge` event to be in `rooms.join.<roomId>.timeline.events` or `rooms.join.<roomId>.state.events`?

I expect to see the m.bridge state within the rooms.join.<roomid>.state.events - this is the behavior we see elsewhere when not using MSC2716.

However this actually appears to be ad-odds with the spec. According to the spec I would not expect to see either the m.bridge event nor $a6LZbYMvr5yYRcFc0uethSHLVC_0l7LRUr0nCZXB2lQ (which is included). I'm unsure how, if the spec is followed exactly, an invited room member is ever able to receive any room state prior to their join (assuming one should ignore invite_state). Maybe I'm misinterpreting something here?

I wrote a Complement test to try to reproduce the problem but I am seeing some differences:

1. Does the test setup look correct to what you're doing? [Test case to try to reproduce missing state in sync (possibly related to MSC2716) complement#352](https://github.com/matrix-org/complement/pull/352)

This looks correct - I think the only thing missing is the state_key for the m.bridge event which would explain why it never appears in the sync state I think.

   * The spec isn't clear to me what should be included in `invite_state`

Agreed, I think it's just for display purposes but it's not clear. What is clear is any invite_state should not replace rooms.join.<roomid>.state.events.