feat: added a function to utilize purl integration

[inosmeet]

This implementation resolves name collision for docutils #3152.
I have temporarily added the database related code in parsers directory for demonstration until we include it in cache.
For the simplicity's sake, I have only modified python parser. After this one gets green flag, I'll add the other.

@terriko @anthonyharrison

[inosmeet]

I'm not sure what to do about this

[anthonyharrison]

What is location being used for? How is the path established? As written, this won't work on Windows.

I think the location in the ProductInfo is the file location of the application which is being scanned.

[inosmeet]

I too thought that it's the location of the product binary being scanned, but wasn't sure so I put a default string similar to find_vendor() method.

Apart from windows, it should be working on the Linux, however it's not able to find the installed database in the cache.

[terriko]

From your email, it sounds like we just need to add purl2cpe in as a data source. It's not quite as complicated as the other data sources because right now we aren't parsing data or anything, just grabbing the db file and going, so it's going to be a slightly awkward fit but I think we can make it work.

I'd suggest you do it rather than me, but since I was just in the sources updating stuff let me lay out what I think needs to happen

Here's what you'd need to do:

1. Make the file cve_bin_tool/data_sources/purl2cpe.py
2. Make a Data_Source for purl2cpe inside that file. You can look at the other data sources for how the __init__ function parameters are set up, but don't bother duplicating much of the rest of it.
   • The only function that actually matters here is the get_cve_data one. That's the one that's called in cvedb when the data sources are updated.
   • The function name becomes slightly nonsensical here since we're not actually getting CVE data but rather purl/cpe data, but I don't think it's worth you going and refactoring it to be get_data() right now in part because it'll break my EPSS PR that's still in progress. But pretend it's called get_data if it makes it feel less weird.
3. Inside the get_cve_data() function, you want to download the purl2cpe database and stick it in the .cache directory. You should be able to figure that out from the other data sources; make sure to include error checking for failures during the request call so it'll fail gracefully if there's a timeout or something. You don't need to do parsing or anythign for this one, just download and make sure it's in the correct place.
4. Go to cli.py and add the ability to enable/disable the data source.
5. Go to cvedb.py and edit the populatedb() function so it doesn't try to do anything further with the purl2cpe data.
   • again, I think we probably need to refactor the sources so this works better eventually, but not today!
6. If you can think of any tests to go with this, that would be great, but it's ok if we don't have any for now. I'll be writing a "check if it's disabled" type test for the EPSS stuff and we can hook into that when it's ready, at least.

Writing this makes me think maybe I should also add some docs on how to write a data source, so I'll do the docs writing, but you should write the code for purl2cpe specifically!

[inosmeet]

This should be working now. After removing cache manually and running the tool, it works fine locally. Failing tests too, pass.

[anthonyharrison]

Added some comments.

Need to create some test scripts for PURL2CPE data source.

[anthonyharrison]

What is location being used for? How is the path established? As written, this won't work on Windows.

I think the location in the ProductInfo is the file location of the application which is being scanned.

[inosmeet]

Added some comments.

Need to create some test scripts for PURL2CPE data source.

Yeah, I'll be working on the tests this week, after we're satisfied with the code.

[terriko]

Hm, so... our repo settings won't let me merge this because the tests don't pass, but the tests aren't passing because the cache doesn't have the purl2cpe database yet.

Can we maybe split out the part that adds the database so I can merge that and then update the cache, and then we'll be able to merge the rest?

[terriko]

The cache update is running now. I think you can see it here if you want to track when it finishes successfully and we can re-run tests here and hopefully have them all pass:

https://github.com/intel/cve-bin-tool/actions/workflows/update-cache.yml

[terriko]

Whoops, didn't mean to mark that as approved before the tests run but whatever.

Cache appears to have updated cleanly, I've resolved the merge conflict, and I believe the tests should pass this time. I've got to run to a meeting but I'll be back to check on them later.

[terriko]

Okay, tests are passing, let's review for real!

I think we're going to need to refactor decode_cp23 out of here and where it is in sbom manager and make a single function that we call, probably in util for now. Since you've probably gone to bed and this should solve at least the docutils issue, I think I'm going to err on the side of merging as is and I'll file a separate issue for a refactor.