Skip to content

Commit 5116c32

Browse files
rickstaaMatteoPierroanuraghazra
authored and
Otard95
committed
feat: enable multi page star fetching for private vercel instances (anuraghazra#2159)
* feat: enable multi-page stars' fetching for private vercel instances This commit enables multi-page stars' support from fetching on private Vercel instances. This feature can be disabled on the public Vercel instance by adding the `FETCH_SINGLE_PAGE_STARS=true` as an env variable in the public Vercel instance. This variable will not be present when people deploy their own Vercel instance, causing the code to fetch multiple star pages. * fix: improve stats multi-page fetching behavoir This commit makes sure that the GraphQL api is only called one time per 100 repositories. The old method added one unnecesairy GraphQL call. * docs: update documentation * style: improve code syntax Co-authored-by: Matteo Pierro <[email protected]> * lol happy new year * docs: remove rate limit documentation for now Remove the `FETCH_SINGLE_PAGE_STARS` from documentation for now since it might confuse people. * fix: fix error in automatic merge * feat: make sure env variable is read Co-authored-by: Matteo Pierro <[email protected]> Co-authored-by: Anurag <[email protected]>
1 parent c63dd1b commit 5116c32

File tree

4 files changed

+210
-166
lines changed

4 files changed

+210
-166
lines changed

‎readme.md

+4-4
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ Visit <https://indiafightscorona.giveindia.org> and make a small donation to hel
9393
- [Language Card Exclusive Options](#language-card-exclusive-options)
9494
- [Wakatime Card Exclusive Option](#wakatime-card-exclusive-options)
9595
- [Deploy Yourself](#deploy-on-your-own-vercel-instance)
96-
- [Keep your fork up to date](#keep-your-fork-up-to-date)
96+
- [Keep your fork up to date](#keep-your-fork-up-to-date)
9797

9898
# GitHub Stats Card
9999

@@ -264,7 +264,7 @@ You can customize the appearance of your `Stats Card` or `Repo Card` however you
264264
- `border_radius` - Corner rounding on the card. Default: `4.5`.
265265

266266
> **Warning**
267-
> We use caching to decrease the load on our servers (see https://github.com/anuraghazra/github-readme-stats/issues/1471#issuecomment-1271551425). Our cards have a default cache of 4 hours (14400 seconds). Also, note that the cache is clamped to a minimum of 4 hours and a maximum of 24 hours.
267+
> We use caching to decrease the load on our servers (see <https://github.com/anuraghazra/github-readme-stats/issues/1471#issuecomment-1271551425>). Our cards have a default cache of 4 hours (14400 seconds). Also, note that the cache is clamped to a minimum of 4 hours and a maximum of 24 hours.
268268
269269
##### Gradient in bg_color
270270

@@ -354,7 +354,7 @@ Use [show_owner](#customization) variable to include the repo's owner username
354354
The top languages card shows a GitHub user's most frequently used top language.
355355

356356
> **Note**
357-
> Top Languages does not indicate my skill level or anything like that; it's a GitHub metric to determine which languages have the most code on GitHub. It is a new feature of github-readme-stats._
357+
> Top Languages does not indicate my skill level or anything like that; it's a GitHub metric to determine which languages have the most code on GitHub. It is a new feature of github-readme-stats.
358358
359359
### Usage
360360

@@ -508,7 +508,7 @@ How you host this is now up to you, be it a server in your home, AWS or otherwis
508508

509509
## Deploy on your own Vercel instance (recommended)
510510

511-
#### [Check Out Step By Step Video Tutorial By @codeSTACKr](https://youtu.be/n6d4KHSKqGk?t=107)
511+
#### :film_projector: [Check Out Step By Step Video Tutorial By @codeSTACKr](https://youtu.be/n6d4KHSKqGk?t=107)
512512

513513
> **Warning**
514514
> If you are on the [hobby (i.e. free)](https://vercel.com/pricing) Vercel plan, please make sure you change the `maxDuration` parameter in the [vercel.json](https://github.com/anuraghazra/github-readme-stats/blob/master/vercel.json) file from `30` to `10` (see [#1416](https://github.com/anuraghazra/github-readme-stats/issues/1416#issuecomment-950275476) for more information).

‎src/fetchers/stats-fetcher.js

+103-104
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
// @ts-check
22
import axios from "axios";
3+
import * as dotenv from "dotenv";
34
import githubUsernameRegex from "github-username-regex";
45
import { calculateRank } from "../calculateRank.js";
56
import { retryer } from "../common/retryer.js";
@@ -11,46 +12,74 @@ import {
1112
wrapTextMultiline,
1213
} from "../common/utils.js";
1314

15+
dotenv.config();
16+
17+
// GraphQL queries.
18+
const GRAPHQL_REPOS_FIELD = `
19+
repositories(first: 100, ownerAffiliations: OWNER, orderBy: {direction: DESC, field: STARGAZERS}, after: $after) {
20+
totalCount
21+
nodes {
22+
name
23+
stargazers {
24+
totalCount
25+
}
26+
}
27+
pageInfo {
28+
hasNextPage
29+
endCursor
30+
}
31+
}
32+
`;
33+
34+
const GRAPHQL_REPOS_QUERY = `
35+
query userInfo($login: String!, $after: String) {
36+
user(login: $login) {
37+
${GRAPHQL_REPOS_FIELD}
38+
}
39+
}
40+
`;
41+
42+
const GRAPHQL_STATS_QUERY = `
43+
query userInfo($login: String!, $after: String) {
44+
user(login: $login) {
45+
name
46+
login
47+
contributionsCollection {
48+
totalCommitContributions
49+
restrictedContributionsCount
50+
}
51+
repositoriesContributedTo(first: 1, contributionTypes: [COMMIT, ISSUE, PULL_REQUEST, REPOSITORY]) {
52+
totalCount
53+
}
54+
pullRequests(first: 1) {
55+
totalCount
56+
}
57+
openIssues: issues(states: OPEN) {
58+
totalCount
59+
}
60+
closedIssues: issues(states: CLOSED) {
61+
totalCount
62+
}
63+
followers {
64+
totalCount
65+
}
66+
${GRAPHQL_REPOS_FIELD}
67+
}
68+
}
69+
`;
70+
1471
/**
1572
* Stats fetcher object.
1673
*
1774
* @param {import('axios').AxiosRequestHeaders} variables Fetcher variables.
1875
* @param {string} token GitHub token.
19-
* @returns {Promise<import('../common/types').StatsFetcherResponse>} Stats fetcher response.
76+
* @returns {Promise<import('../common/types').Fetcher>} Stats fetcher response.
2077
*/
2178
const fetcher = (variables, token) => {
79+
const query = !variables.after ? GRAPHQL_STATS_QUERY : GRAPHQL_REPOS_QUERY;
2280
return request(
2381
{
24-
query: `
25-
query userInfo($login: String!) {
26-
user(login: $login) {
27-
name
28-
login
29-
contributionsCollection {
30-
totalCommitContributions
31-
restrictedContributionsCount
32-
}
33-
repositoriesContributedTo(contributionTypes: [COMMIT, ISSUE, PULL_REQUEST, REPOSITORY]) {
34-
totalCount
35-
}
36-
pullRequests {
37-
totalCount
38-
}
39-
openIssues: issues(states: OPEN) {
40-
totalCount
41-
}
42-
closedIssues: issues(states: CLOSED) {
43-
totalCount
44-
}
45-
followers {
46-
totalCount
47-
}
48-
repositories(ownerAffiliations: OWNER) {
49-
totalCount
50-
}
51-
}
52-
}
53-
`,
82+
query,
5483
variables,
5584
},
5685
{
@@ -60,39 +89,42 @@ const fetcher = (variables, token) => {
6089
};
6190

6291
/**
63-
* Fetch first 100 repositories for a given username.
92+
* Fetch stats information for a given username.
6493
*
65-
* @param {import('axios').AxiosRequestHeaders} variables Fetcher variables.
66-
* @param {string} token GitHub token.
67-
* @returns {Promise<import('../common/types').StatsFetcherResponse>} Repositories fetcher response.
94+
* @param {string} username Github username.
95+
* @returns {Promise<import('../common/types').StatsFetcher>} GraphQL Stats object.
96+
*
97+
* @description This function supports multi-page fetching if the 'FETCH_MULTI_PAGE_STARS' environment variable is set to true.
6898
*/
69-
const repositoriesFetcher = (variables, token) => {
70-
return request(
71-
{
72-
query: `
73-
query userInfo($login: String!, $after: String) {
74-
user(login: $login) {
75-
repositories(first: 100, ownerAffiliations: OWNER, orderBy: {direction: DESC, field: STARGAZERS}, after: $after) {
76-
nodes {
77-
name
78-
stargazers {
79-
totalCount
80-
}
81-
}
82-
pageInfo {
83-
hasNextPage
84-
endCursor
85-
}
86-
}
87-
}
88-
}
89-
`,
90-
variables,
91-
},
92-
{
93-
Authorization: `bearer ${token}`,
94-
},
95-
);
99+
const statsFetcher = async (username) => {
100+
let stats;
101+
let hasNextPage = true;
102+
let endCursor = null;
103+
while (hasNextPage) {
104+
const variables = { login: username, first: 100, after: endCursor };
105+
let res = await retryer(fetcher, variables);
106+
if (res.data.errors) return res;
107+
108+
// Store stats data.
109+
const repoNodes = res.data.data.user.repositories.nodes;
110+
if (!stats) {
111+
stats = res;
112+
} else {
113+
stats.data.data.user.repositories.nodes.push(...repoNodes);
114+
}
115+
116+
// Disable multi page fetching on public Vercel instance due to rate limits.
117+
const repoNodesWithStars = repoNodes.filter(
118+
(node) => node.stargazers.totalCount !== 0,
119+
);
120+
hasNextPage =
121+
process.env.FETCH_MULTI_PAGE_STARS === "true" &&
122+
repoNodes.length === repoNodesWithStars.length &&
123+
res.data.data.user.repositories.pageInfo.hasNextPage;
124+
endCursor = res.data.data.user.repositories.pageInfo.endCursor;
125+
}
126+
127+
return stats;
96128
};
97129

98130
/**
@@ -137,46 +169,6 @@ const totalCommitsFetcher = async (username) => {
137169
return 0;
138170
};
139171

140-
/**
141-
* Fetch all the stars for all the repositories of a given username.
142-
*
143-
* @param {string} username GitHub username.
144-
* @param {array} repoToHide Repositories to hide.
145-
* @returns {Promise<number>} Total stars.
146-
*/
147-
const totalStarsFetcher = async (username, repoToHide) => {
148-
let nodes = [];
149-
let hasNextPage = true;
150-
let endCursor = null;
151-
while (hasNextPage) {
152-
const variables = { login: username, first: 100, after: endCursor };
153-
let res = await retryer(repositoriesFetcher, variables);
154-
155-
if (res.data.errors) {
156-
logger.error(res.data.errors);
157-
throw new CustomError(
158-
res.data.errors[0].message || "Could not fetch user",
159-
CustomError.USER_NOT_FOUND,
160-
);
161-
}
162-
163-
const allNodes = res.data.data.user.repositories.nodes;
164-
const nodesWithStars = allNodes.filter(
165-
(node) => node.stargazers.totalCount !== 0,
166-
);
167-
nodes.push(...nodesWithStars);
168-
// hasNextPage =
169-
// allNodes.length === nodesWithStars.length &&
170-
// res.data.data.user.repositories.pageInfo.hasNextPage;
171-
hasNextPage = false; // NOTE: Temporarily disable fetching of multiple pages. Done because of #2130.
172-
endCursor = res.data.data.user.repositories.pageInfo.endCursor;
173-
}
174-
175-
return nodes
176-
.filter((data) => !repoToHide[data.name])
177-
.reduce((prev, curr) => prev + curr.stargazers.totalCount, 0);
178-
};
179-
180172
/**
181173
* Fetch stats for a given username.
182174
*
@@ -203,7 +195,7 @@ const fetchStats = async (
203195
rank: { level: "C", score: 0 },
204196
};
205197

206-
let res = await retryer(fetcher, { login: username });
198+
let res = await statsFetcher(username);
207199

208200
// Catch GraphQL errors.
209201
if (res.data.errors) {
@@ -259,8 +251,15 @@ const fetchStats = async (
259251
stats.contributedTo = user.repositoriesContributedTo.totalCount;
260252

261253
// Retrieve stars while filtering out repositories to be hidden
262-
stats.totalStars = await totalStarsFetcher(username, repoToHide);
254+
stats.totalStars = user.repositories.nodes
255+
.filter((data) => {
256+
return !repoToHide[data.name];
257+
})
258+
.reduce((prev, curr) => {
259+
return prev + curr.stargazers.totalCount;
260+
}, 0);
263261

262+
// @ts-ignore // TODO: Fix this.
264263
stats.rank = calculateRank({
265264
totalCommits: stats.totalCommits,
266265
totalRepos: user.repositories.totalCount,

0 commit comments

Comments
 (0)