Skip to content

Fix for count being very slow on large Parse Classes' collections (Postgres) #5330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 25 commits into from
Apr 8, 2019
Merged

Fix for count being very slow on large Parse Classes' collections (Postgres) #5330

Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
ec332d7
Changed count to be approximate. Should help with postgres slowness
Jan 28, 2019
9e2e8cd
refactored last commit to only fall back to estimate if no complex query
Jan 28, 2019
2ac2e82
handlign variables correctly
Jan 28, 2019
7bf3679
Trying again because it was casting to lowercase table names which do…
Jan 28, 2019
bd6e816
syntax error
Jan 28, 2019
3209b5d
Adding quotations to pg query
Jan 28, 2019
fb47e0f
hopefully final pg fix
Jan 28, 2019
1d90ed4
Postgres will now use an approximate count unless there is a more com…
Jan 28, 2019
87f9add
handling edge case
Jan 29, 2019
909243d
Fix for count being very slow on large Parse Classes' collections in …
Jan 29, 2019
6be92f1
Fixed silly spelling error resulting from copying over notes
Jan 29, 2019
f464f51
Lint fixes
Jan 29, 2019
7b55d87
limiting results to 1 on approximation
Jan 30, 2019
662ed81
suppress test that we can no longer run for postgres
Jan 30, 2019
b29d93b
removed tests from Postgres that no longer apply
Jan 30, 2019
15991a3
made changes requested by dplewis
behrangs Feb 1, 2019
7401970
fixed count errors
behrangs Mar 27, 2019
9136b20
updated package.json
behrangs Mar 27, 2019
fcf63bd
Merge branch 'master' into bugfix-slowPostgresCount
CoderickLamar Mar 31, 2019
25597b0
removed test exclude for pg
behrangs Apr 2, 2019
9bc37cf
removed object types from method
behrangs Apr 2, 2019
00c23c4
test disabled for postgres
behrangs Apr 2, 2019
7d63f80
returned type
behrangs Apr 2, 2019
f397c8a
add estimate count test
dplewis Apr 3, 2019
8951580
fix mongo test
dplewis Apr 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 61 additions & 14 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions spec/AudienceRouter.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ describe('AudiencesRouter', () => {
});
});

it('query installations with count = 1', done => {
it_exclude_dbs(['postgres'])('query installations with count = 1', done => {
const config = Config.get('test');
const androidAudienceRequest = {
name: 'Android Users',
Expand Down Expand Up @@ -189,7 +189,7 @@ describe('AudiencesRouter', () => {
});
});

it('query installations with limit = 0 and count = 1', done => {
it_exclude_dbs(['postgres'])('query installations with limit = 0 and count = 1', done => {
const config = Config.get('test');
const androidAudienceRequest = {
name: 'Android Users',
Expand Down
4 changes: 2 additions & 2 deletions spec/InstallationsRouter.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ describe('InstallationsRouter', () => {
});
});

it('query installations with count = 1', done => {
it_exclude_dbs(['postgres'])('query installations with count = 1', done => {
const config = Config.get('test');
const androidDeviceRequest = {
installationId: '12345678-abcd-abcd-abcd-123456789abc',
Expand Down Expand Up @@ -209,7 +209,7 @@ describe('InstallationsRouter', () => {
});
});

it('query installations with limit = 0 and count = 1', done => {
it_exclude_dbs(['postgres'])('query installations with limit = 0 and count = 1', done => {
const config = Config.get('test');
const androidDeviceRequest = {
installationId: '12345678-abcd-abcd-abcd-123456789abc',
Expand Down
19 changes: 17 additions & 2 deletions src/Adapters/Storage/Postgres/PostgresStorageAdapter.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1970,9 +1970,24 @@ export class PostgresStorageAdapter implements StorageAdapter {

const wherePattern =
where.pattern.length > 0 ? `WHERE ${where.pattern}` : '';
const qs = `SELECT count(*) FROM $1:name ${wherePattern}`;
let tempQs = '';

if (where.pattern.length > 0 || !query) {
tempQs = `SELECT count(*) FROM $1:name ${wherePattern}`;
} else {
tempQs =
'SELECT reltuples AS approximate_row_count FROM pg_class WHERE relname = $1';
}
const qs = tempQs;

return this._client
.one(qs, values, a => +a.count)
.one(qs, values, a => {
if (a.approximate_row_count != null) {
return +a.approximate_row_count;
} else {
return +a.count;
}
})
.catch(error => {
if (error.code !== PostgresRelationDoesNotExistError) {
throw error;
Expand Down