-
Notifications
You must be signed in to change notification settings - Fork 229
Collect NPM #101 #121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Collect NPM #101 #121
Conversation
Signed-off-by: Navonil Das <[email protected]>
d73f582
to
c95c4f8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@NavonilDas thank you ++ for this. This is a very good start!
A few global comments before I can review further:
- can you ensure you rebase your branch on the latest develop branch?
- please use pep8 formatting
- use a docstring for function documentation rather than a # comment before the function
- most important of all, we need some unit tests for all this
import semantic_version | ||
import re | ||
import json | ||
from urllib.request import urlopen |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mind to sort imports?
Also can you add a standard license header as in the other files?
vulnerabilities/scraper/npm.py
Outdated
NPM_URL = 'https://registry.npmjs.org{}' | ||
PAGE = '/-/npm/v1/security/advisories?page=1' | ||
|
||
# Removes spaces and v Charecter in front of version |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you make this a docstring rather than a comment? Also plase make sure you use pep8-like formatting (e.g. 2 empty lines between top level members)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, I will change the requested things.
vulnerabilities/scraper/npm.py
Outdated
|
||
# Removes spaces and v Charecter in front of version | ||
def remove_spaces(x): | ||
x = re.sub(r" +"," ",x) # Replace Multiple spaces to one |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please avoid end of line comments if possible. And use single ' quotes unless for docstrings.
That said, what is the purpose of this function? Are spaces an issue for the semantic version library?
And this is a rather complex function, so we absolutely need many tests for this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This function removes multiple spaces and v character in front of the version (like v2.1.3). The semantic version module throws parsing exception due to these conditions.
yes, tests would be a good idea.
@NavonilDas Thanks for making those changes. However, your last commit is missing the "Signed-off-by" bit. Could you please add it? You can automate that for future commits with a git hook:
Your branch also still needs to be rebased on current Let me know if you need any help. I can probably also do the rebase for you if you prefer that. |
@haikoschol Thanks for the script. After writing a few Tests I will rebase it to develop branch. |
Signed-off-by: Navonil Das <[email protected]>
Signed-off-by: Navonil Das <[email protected]>
Signed-off-by: Navonil Das <[email protected]>
Signed-off-by: Navonil Das <[email protected]>
Signed-off-by: Navonil Das <[email protected]>
Signed-off-by: Navonil Das <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the update and the progress... See my comments inline.
README.md
Outdated
@@ -68,6 +68,37 @@ DJANGO_DEV=1 python manage.py test vulnerabilities/tests | |||
DJANGO_DEV=1 python manage.py import --all | |||
``` | |||
|
|||
If you want to run the import periodically, you can use a systemd timer: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure why this (which is from #124 ) ends up in your PR.... di you merge rather than rebase on develop?
vulnerabilities/data_dump.py
Outdated
@@ -127,7 +127,17 @@ def archlinux_dump(extract_data): | |||
VulnerabilityReference.objects.create( | |||
vulnerability=vulnerability, | |||
reference_id=vulnerability_id, | |||
<<<<<<< HEAD |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You have an unresolved merge conflict here... but why would your npm code be related to archlinux code?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is because we both made some trivial changes to the same lines of the archlinux code. That's what I was referring to at the end of my earlier comment (#121 (comment)):
The conflicting code on both branches does the same thing, just with different syntax (string formatting with '{}'.format(...) vs f'{...}'). I prefer the f'{...}' syntax, but pick whichever one you want.
Signed-off-by: Navonil Das <[email protected]>
bd6a008
to
0cd3f38
Compare
It looks like all review comments have been addressed. There is only one commit failing the DCO check. Since most of the commits on this branch only contain styling or merge fixes, I'll squash them and merge the PR. |
Collecting vulnerabilities from npm registry.