Skip to content

Add local registry with most popular crates from crates.io #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 30, 2021
Merged

Add local registry with most popular crates from crates.io #27

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
bf4d406
Download common crates to a local repository
dhovart Sep 21, 2021
0d83318
Add local registry regeneration script
dhovart Sep 21, 2021
53da89f
Minor Dockerfile improvements
dhovart Sep 24, 2021
3fdd095
Generate regitry from a list of crates
dhovart Sep 25, 2021
74629fa
Minor Dockerfile update
dhovart Sep 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 27 additions & 3 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,39 @@ RUN rm src/lib.rs
COPY src/* src/
# build the executable
RUN cargo build --release
# download jq
RUN mkdir -p ${wd}/bin
RUN curl -L -o /usr/local/bin/jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \
&& chmod +x /usr/local/bin/jq
# download cargo-local-registry (we don't use cargo install because we want it portable: we'll copy it later to the image that is run)
RUN curl -L -o clr.tar.gz https://github.com/ChrisGreenaway/cargo-local-registry/releases/download/0.2.1/cargo-local-registry-0.2.1-x86_64-unknown-linux-musl.tar.gz \
&& tar xvzf clr.tar.gz && chmod +x cargo-local-registry && mv cargo-local-registry /usr/local/cargo/bin
# download and build popular crates to local registry
RUN mkdir /local-registry
WORKDIR /local-registry
COPY local-registry/* ./
RUN curl "https://crates.io/api/v1/crates?page=1&per_page=100&sort=downloads" | \
jq -r '.crates[] | .id + "=\"" + .max_stable_version + "\""' >> Cargo.toml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How big are crates? Is it worth doing 1000 instead of 100? Or are they big?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the top 100 crates it's just 17MB

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be up for expanding it to the top 500 then? @ErikSchierboom?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤷 It all depends on how likely it is those crates are used. I have no data to back up any claims though. @coriolinus @dhovart how likely is it that students would used a crate that is not in the top 100 crates?

Copy link
Contributor Author

@dhovart dhovart Sep 24, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have absolutely no idea, sorry! I just know that this list should maybe be fine-tuned. There are crates we pull that make no sense in the context of exercism, such as for gui development.
Maybe there should be a message for students somewhere (below the list of supported crates maybe) that say that if they want support for a given crate, they should open a GH issue?

Copy link
Contributor Author

@dhovart dhovart Sep 24, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of the crates.io API, we would get the list from somewhere else, where it was edited.
I guess I can add an endpoint to exercism's site API. But I'm not familiar with the site codebase yet, is there some administration interface where someone could create this list ? (if my proposal makes sense)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have absolutely no idea, sorry! I just know that this list should maybe be fine-tuned.

I think having a curated list makes the most sense. We definitely don't need to have any GUI crates included, and that would also allow us to cherry-pick crates that are not in the top 100, but still useful for students in the context of Exercism.

Instead of the crates.io API, we would get the list from somewhere else, where it was edited.

We haven't yet come up with a spec for this. I think we want this list to be in the track repo at exercism/rust. Having it there means that we'll be able to programmatically access it from the website. The next question then becomes: where to store it? I think there are basically two options:

  1. Add a section to the root config.json file
  2. Add a new file just to list the supported libraries

I don't yet have a real preference, but the first option might be best.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be fine as MVP to get it just as a file in this repo that's read when Docker builds. That should be quite trivial. Then we can improve it later. In my eyes this is a real priority, so the quickest solution is probably the best one! :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be fine as MVP to get it just as a file in this repo that's read when Docker builds.

I wouldn't want to touch the existing Docker push workflow (we want to sync that across all repos at some point) and secondly, I'd prefer it to be in exercism/rust as there will be more visibility with maintainers (and students will figure out that location sooner than they will this repo).

RUN cargo generate-lockfile && \
cargo local-registry --sync Cargo.lock .

# As of Dec 2019, we need to use the nightly toolchain to get JSON test output
FROM rustlang/rust:nightly AS test
ENV wd /opt/test-runner
RUN mkdir -p ${wd}/bin
WORKDIR ${wd}
# download jq
RUN curl -L -o bin/jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \
&& chmod +x bin/jq
COPY --from=build /rust-test-runner/target/release/transform-output bin
COPY --from=build /usr/local/bin/jq bin
# configure local-registry
COPY --from=build /local-registry local-registry/
RUN echo '[source.crates-io]\n\
registry = "https://github.com/rust-lang/crates.io-index"\n\
replace-with = "local-registry"\n\
\n\
[source.local-registry]\n\
local-registry = "/opt/test-runner/local-registry/"\n' >> $CARGO_HOME/config.toml
# set entrypoint
COPY bin/run.sh bin
COPY --from=build /usr/local/cargo/bin/cargo-local-registry /usr/local/cargo/bin/
COPY bin/regenerate-registry.sh bin
ENTRYPOINT ["bin/run.sh"]
10 changes: 10 additions & 0 deletions bin/regenerate-registry.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/usr/bin/env bash

export RUSTUP_HOME=/usr/local/rustup
export CARGO_HOME=/usr/local/cargo
export PATH=/usr/local/cargo/bin:$PATH

cd /opt/test-runner/local-registry
perl -i -p0e 's/(\[dependencies\]\n).*/$1/se' Cargo.toml
curl -s -N "https://crates.io/api/v1/crates?page=1&per_page=100&sort=downloads" | ../bin/jq -r '.crates[] | .id + "=\"" + .max_stable_version + "\""' >> Cargo.toml
cargo generate-lockfile && cargo local-registry --sync Cargo.lock .
7 changes: 1 addition & 6 deletions bin/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,7 @@ if [ -z "$slug" ] || [ -z "$solution_path" ]; then
fi

cd "$solution_path"
if [ -e Cargo.lock ]; then
if [ "$(grep -c '\[\[package\]\]' Cargo.lock)" -gt 1 ]; then
echo "{\"status\":\"error\",\"message\":\"building $slug: external crates not supported\",\"tests\":[]}" > "$output_path"/results.json
exit
fi
else
if [ ! -e Cargo.lock ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if at some later point there would be an error messageif the student uses a non-supported crate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was thinking about this. I'll think of a way to support this.

echo "WARNING: student did not upload Cargo.lock. This may cause build errors." | tee -a "$output_path/results.out"
fi

Expand Down
8 changes: 8 additions & 0 deletions local-registry/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[package]
name = "dummy_package"
version="0.1.0"

[lib]
path = "dummy.rs"

[dependencies]
Empty file added local-registry/dummry.rs
View file
Open in desktop
Empty file.