Skip to content

Optimize heatmap query #33853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Mar 20, 2025
Merged

Conversation

lunny
Copy link
Member

@lunny lunny commented Mar 11, 2025

When there are over 5M records on action table, the heatmap on dashboard is very slow as below SQL.

database duration=1.8881s db.sql="SELECT created_unix DIV 900 * 900 AS timestamp, count(user_id) as contributions FROM `action` WHERE user_id=? AND act_user_id=? AND (created_unix > ?) GROUP BY timestamp ORDER BY timestamp"

This PR add a new index for action table with columns user_id, act_user_id and created_unix so that this query will become about 6 times faster than before.

@lunny lunny added the performance/speed performance issues with slow downs label Mar 11, 2025
@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Mar 11, 2025
@github-actions github-actions bot added modifies/go Pull requests that update Go code modifies/migrations labels Mar 11, 2025
@lunny lunny added this to the 1.24.0 milestone Mar 11, 2025
@silverwind
Copy link
Member

are those index names consistent with previous indexes? maybe better to spell out the column names wholly.

@lunny
Copy link
Member Author

lunny commented Mar 14, 2025

are those index names consistent with previous indexes? maybe better to spell out the column names wholly.

ce57b3b will follow previous index naming pattern.

@silverwind
Copy link
Member

I mean why not use act_user_id_created_unix_user_id instead of au_c_u? Note I don't know who other indexes were named.

@lunny
Copy link
Member Author

lunny commented Mar 14, 2025

I mean why not use act_user_id_created_unix_user_id instead of au_c_u? Note I don't know who other indexes were named.

All other indexes in this table uses a first character of the columns in this index. au == act_user_id, c == created_unix and u == user_id.

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Mar 15, 2025
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Mar 18, 2025
@lunny lunny merged commit 4a7ab0a into go-gitea:main Mar 20, 2025
26 checks passed
@lunny lunny deleted the lunny/optimaze_dashboard_heatmap branch March 20, 2025 16:30
zjjhot added a commit to zjjhot/gitea that referenced this pull request Mar 24, 2025
* giteaofficial/main:
  [skip ci] Updated translations via Crowdin
  Only use prev and next buttons for pagination on user dashboard (go-gitea#33981)
  update jwt and redis packages (go-gitea#33984)
  [skip ci] Updated translations via Crowdin
  Improve oauth2 error handling (go-gitea#33969)
  [skip ci] Updated translations via Crowdin
  Cover `go.mod` and `go.sum` in `.editorconfig` (go-gitea#33960)
  Drop timeout for requests made to the internal hook api (go-gitea#33947)
  Fix file name could not be searched if the file was not a text file when using the Bleve indexer (go-gitea#33959)
  Fix oauth2 auth and UI (go-gitea#33961)
  Allow filtering issues by any assignee (go-gitea#33343)
  Optimize total count of feed when loading activities in user dashboard. (go-gitea#33841)
  Extract code to their own functions for push update (go-gitea#33944)
  Optimize heatmap query (go-gitea#33853)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. modifies/go Pull requests that update Go code modifies/migrations performance/speed performance issues with slow downs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants