Skip to content

Add VisibilityBitmap to TableMapEvent in replication #813

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 24, 2023
Merged

Add VisibilityBitmap to TableMapEvent in replication #813

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ccabd26
Add `VisibilityBitmap` to `TableMapEvent` for MySQL 8.0.23+ Invisible…
dongwook-chan Aug 15, 2023
6260982
Refactor bitmap iteration to align with MySQL source code
dongwook-chan Aug 15, 2023
cac73e6
Add test for column visibility in table map
dongwook-chan Aug 19, 2023
b55ff1d
Add test case 2 for column visibility in table map
dongwook-chan Aug 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions replication/const.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,7 @@ const (
TABLE_MAP_OPT_META_PRIMARY_KEY_WITH_PREFIX
TABLE_MAP_OPT_META_ENUM_AND_SET_DEFAULT_CHARSET
TABLE_MAP_OPT_META_ENUM_AND_SET_COLUMN_CHARSET
TABLE_MAP_OPT_META_COLUMN_VISIBILITY
)

type IntVarEventType byte
Expand Down
26 changes: 26 additions & 0 deletions replication/row_event.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ type TableMapEvent struct {
// EnumSetDefaultCharset/EnumSetColumnCharset is similar to DefaultCharset/ColumnCharset but for enum/set columns.
EnumSetDefaultCharset []uint64
EnumSetColumnCharset []uint64

// VisibilityBitmap stores bits that are set if corresponding column is not invisible (MySQL 8.0.23+)
VisibilityBitmap []byte
}

func (e *TableMapEvent) Decode(data []byte) error {
Expand Down Expand Up @@ -312,6 +315,9 @@ func (e *TableMapEvent) decodeOptionalMeta(data []byte) (err error) {
return err
}

case TABLE_MAP_OPT_META_COLUMN_VISIBILITY:
e.VisibilityBitmap = v

default:
// Ignore for future extension
}
Expand Down Expand Up @@ -421,6 +427,7 @@ func (e *TableMapEvent) Dump(w io.Writer) {
fmt.Fprintf(w, "Primary key prefix: %v\n", e.PrimaryKeyPrefix)
fmt.Fprintf(w, "Enum/set default charset: %v\n", e.EnumSetDefaultCharset)
fmt.Fprintf(w, "Enum/set column charset: %v\n", e.EnumSetColumnCharset)
fmt.Fprintf(w, "Invisible Column bitmap: \n%s", hex.Dump(e.VisibilityBitmap))

unsignedMap := e.UnsignedMap()
fmt.Fprintf(w, "UnsignedMap: %#v\n", unsignedMap)
Expand All @@ -440,6 +447,9 @@ func (e *TableMapEvent) Dump(w io.Writer) {
geometryTypeMap := e.GeometryTypeMap()
fmt.Fprintf(w, "GeometryTypeMap: %#v\n", geometryTypeMap)

visibilityMap := e.VisibilityMap()
fmt.Fprintf(w, "VisibilityMap: %#v\n", visibilityMap)

nameMaxLen := 0
for _, name := range e.ColumnName {
if len(name) > nameMaxLen {
Expand Down Expand Up @@ -730,6 +740,22 @@ func (e *TableMapEvent) GeometryTypeMap() map[int]uint64 {
return ret
}

// VisibilityMap returns a map: column index -> visiblity.
// Invisible column was introduced in MySQL 8.0.23
// nil is returned if not available.
func (e *TableMapEvent) VisibilityMap() map[int]bool {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As for testing, I think maybe you can capture/construct some VisibilityBitmap bytes and test the result of TableMapEvent.VisibilityMap is expected.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lance6716
Sure. I think I should create a separate function. Just as TestTableMapOptMetaPrimaryKey is defined solely for primary key type meta data. At first, I thought about squeezing into TestTableMapOptMetaNames or TestTableMapHelperMaps, but it's too packed and I saw no room for any addition. Your advice would be help me a lot. 😀

if len(e.VisibilityBitmap) == 0 {
return nil
}
p := 0
ret := make(map[int]bool)
for i := 0; i < int(e.ColumnCount); i++ {
ret[i] = e.VisibilityBitmap[p/8]&(1<<uint(7-p%8)) != 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Maybe it's a little easier to understand the original code with countdown instead of 1<<uint(7-p%8)

Copy link
Contributor Author

@dongwook-chan dongwook-chan Aug 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@atercattus
I appreciate your review. Also, I agree with you and I can get this fixed.
Actually VisibilityMap is copy and paste of existing UnsignedMap. Should I fix UnsignedMap too?
I could create a separate PR it if it seems irrelevant to this PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. @lance6716 what do you think about this? Apply the Boy Scout Rule or not?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we decide to improve, the code will look like below.

func (e *TableMapEvent) VisibilityMap() map[int]bool {
	if len(e.VisibilityBitmap) == 0 {
		return nil
	}
	ret := make(map[int]bool)
	for _, field := range e.VisibilityBitmap {
		for c, i := 0x80, 0; c != 0; c, i = c>>1, i+1 {
			ret[i] = field&byte(c) != 0
		}
	}
	return ret
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old code is really not easy to understand 😢 I also hope we start to improve the readability. As for UnsignedMap, I'm OK that you also modify it in this PR or another PR. WDYT @atercattus

Your new code snippet seems not optimal. The status of i should be kept when iterate bytes of VisibilityBitmap. And we can early stop when the column count is not a multiple of 8 (though the original code seems not optimize about this). Maybe we can

...
i := 0
for e.VisibilityBitmap {
  for c := 0x80; ... {
    ret[i] = ...
    i++
    if i >= e.ColumnCount { return }
  }
}
...

Welcome to find different code

Copy link
Contributor Author

@dongwook-chan dongwook-chan Aug 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad. i should indeed be initialized outside of the inner loop.

p++
}
return ret
}

// Below realType and IsXXXColumn are base from:
// table_def::type in sql/rpl_utility.h
// Table_map_log_event::print_columns in mysql-8.0/sql/log_event.cc and mariadb-10.5/sql/log_event_client.cc
Expand Down