Use different library for mime on windows #600
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jyn514
reviewed
Feb 9, 2020
Zexbe
force-pushed
the
mime-alternative
branch
from
3877ac0 to
ce566fa
Compare
jyn514
reviewed
Feb 11, 2020
Zexbe
force-pushed
the
mime-alternative
branch
from
ce566fa to
0518a0d
Compare
jyn514
reviewed
Feb 13, 2020
jyn514
added a commit
to jyn514/docs.rs
that referenced
this pull request
Mar 28, 2020
This uses the Rust tree_magic package instead. Most work was already done in rust-lang#600. This just sets tree_magic to build unconditionally instead of always on Windows. This improves the memory usage of docs.rs by over a factor of 5 when uploading files in some cases. This can be tested locally like so: Run minio in the background: ``` docker run -p 9000:9000 -e MINIO_ACCESS_KEY=password -e MINIO_SECRET_KEY=password minio/minio server /data ``` Navigate to localhost:9000/ and set both the access key and secret key to 'password'. Add a new bucket called `rust-docs-rs`. Now in another terminal, run docs.rs. This assumes you already have the CRATESFYI_DATABASE_URL set up locally as described in [developing without docker-compose](https://forge.rust-lang.org/docs-rs/no-docker-compose.html): ``` # set up a fairly large file mkdir -p ignored/tmp yes | head -n $((1024 * 1024 * 50)) > ignored/tmp/100MB git checkout master cargo build --release RUST_LOG=cratesfyi,info S3_ENDPOINT=http://localhost:9000 AWS_ACCESS_KEY_ID=password AWS_SECRET_ACCESS_KEY=password valgrind --tool=massif --massif-out-file=libmagic.massif.out target/release/cratesfyi database add-directory ignored/tmp/ # this will show ~1.5 GB used, depending on your system ms_print libmagic.massif.out | less # now try without libmagic git checkout druid cargo build --release RUST_LOG=cratesfyi,info S3_ENDPOINT=http://localhost:9000 AWS_ACCESS_KEY_ID=password AWS_SECRET_ACCESS_KEY=password valgrind --tool=massif --massif-out-file=no_libmagic.massif.out target/release/cratesfyi database add-directory ignored/tmp/ # this will show ~250 MB used, depending on your system ms_print no_libmagic.massif.out | less ```
jyn514
added a commit
that referenced
this pull request
Mar 28, 2020
This uses the Rust tree_magic package instead. Most work was already done in #600. This just sets tree_magic to build unconditionally instead of always on Windows. This improves the memory usage of docs.rs by over a factor of 5 when uploading files in some cases. This can be tested locally like so: Run minio in the background: ``` docker run -p 9000:9000 -e MINIO_ACCESS_KEY=password -e MINIO_SECRET_KEY=password minio/minio server /data ``` Navigate to localhost:9000/ and set both the access key and secret key to 'password'. Add a new bucket called `rust-docs-rs`. Now in another terminal, run docs.rs. This assumes you already have the CRATESFYI_DATABASE_URL set up locally as described in [developing without docker-compose](https://forge.rust-lang.org/docs-rs/no-docker-compose.html): ``` # set up a fairly large file mkdir -p ignored/tmp yes | head -n $((1024 * 1024 * 50)) > ignored/tmp/100MB git checkout master cargo build --release RUST_LOG=cratesfyi,info S3_ENDPOINT=http://localhost:9000 AWS_ACCESS_KEY_ID=password AWS_SECRET_ACCESS_KEY=password valgrind --tool=massif --massif-out-file=libmagic.massif.out target/release/cratesfyi database add-directory ignored/tmp/ # this will show ~1.5 GB used, depending on your system ms_print libmagic.massif.out | less # now try without libmagic git checkout druid cargo build --release RUST_LOG=cratesfyi,info S3_ENDPOINT=http://localhost:9000 AWS_ACCESS_KEY_ID=password AWS_SECRET_ACCESS_KEY=password valgrind --tool=massif --massif-out-file=no_libmagic.massif.out target/release/cratesfyi database add-directory ignored/tmp/ # this will show ~250 MB used, depending on your system ms_print no_libmagic.massif.out | less ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Windows can't build with magic, so this gives it an alternative crate to use.
While adding unit tests, and checking what magic returned, I found a few mimes it was returning incorrectly, and have also corrected those.