Add ecosystem specific inclusions or exclusions

[AyanSinhaMahapatra]

Also ignore specific files paths containing metadata in ruby gems.

Reference: #1438
Reference: #1476

[pombredanne]

Looking fine, the name "config.py" is too generic though for now.

[AyanSinhaMahapatra]

@tdruez fixed the tests, this is ready for your review. 😄

[tdruez]

See some suggestions about refining how we declare the configs.
Now, I'm struggling to understand the way we load and then use the config values on the pipeline. The implementation is probably a bit too complex and we should look into ways to simplify this so it's not too hard to maintain in the future.

[tdruez]

Is this a leftover or are we planning to duplicate those methods?

[tdruez]

Should this exist directly in the d2d_config module instead?

[tdruez]

Could we benefit from using a @DataClass here for EcosystemConfig?

For example:

from dataclasses import dataclass, field

@dataclass
class EcosystemConfig:
    """
    Base class for ecosystem-specific configurations to be defined
    for each ecosystem.
    """

    # This should be defined for each ecosystem which
    # are options in the pipelines
    ecosystem_option: str = "Default"

    # These are extensions for packages of this ecosystem which
    # need to be matched from purldb
    purldb_package_extensions: list = field(default_factory=list)

    # These are extensions for resources of this ecosystem which
    # need to be matched from purldb
    purldb_resource_extensions: list = field(default_factory=list)

    # Extensions for document files which do not require review
    doc_extensions: list = field(default_factory=list)

    # Paths in the deployed binaries/archives (on the to/ side) which
    # do not need review even if they are not matched to the source side
    deployed_resource_path_exclusions: list = field(default_factory=list)

    # Paths in the development/source archive (on the from/ side) which
    # should not be considered even if unmapped to the deployed side when
    # assessing what to review on the deployed side
    devel_resource_path_exclusions: list = field(default_factory=list)

    # Symbols which are found in ecosystem-specific standard libraries
    # which are not so useful in mapping
    standard_symbols_to_exclude: list = field(default_factory=list)


# Dictionary of ecosystem configurations
ECOSYSTEM_CONFIGS = {
    "Default": EcosystemConfig(
        purldb_package_extensions=[".zip", ".tar.gz", ".tar.xz"],
        devel_resource_path_exclusions=["*/tests/*"],
        doc_extensions=[
            ".pdf",
            ".doc",
            ".docx",
            ".ppt",
            ".pptx",
            ".tex",
            ".odt",
            ".odp",
        ],
    ),
    "Java": EcosystemConfig(
        ecosystem_option="Java",
        purldb_package_extensions=[".jar", ".war"],
        purldb_resource_extensions=[".class"],
    ),
    "JavaScript": EcosystemConfig(
        ecosystem_option="JavaScript",
        purldb_resource_extensions=[
            ".map",
            ".js",
            ".mjs",
            ".ts",
            ".d.ts",
            ".jsx",
            ".tsx",
            ".css",
            ".scss",
            ".less",
            ".sass",
            ".soy",
        ],
    ),
    "Go": EcosystemConfig(
        ecosystem_option="Go",
        purldb_resource_extensions=[".go"],
    ),
    "Rust": EcosystemConfig(
        ecosystem_option="Rust",
        purldb_resource_extensions=[".rs"],
    ),
    "Ruby": EcosystemConfig(
        ecosystem_option="Ruby",
        purldb_package_extensions=[".gem"],
        purldb_resource_extensions=[".rb"],
        deployed_resource_path_exclusions=["*checksums.yaml.gz*", "*metadata.gz*"],
    ),
}

def get_ecosystem_config(ecosystem):
    """Return the ``ecosystem`` config."""
    return ECOSYSTEM_CONFIGS.get(ecosystem, ECOSYSTEM_CONFIGS["Default"])

[tdruez]

I think we should make this work without the need for the extra mixing. See further suggestions.

[tdruez]

This could be replaced by a def get_ecosystem_config(ecosystem) imported from the d2d_config module. See the implementation suggestion below.

[tdruez]

We need to provide more details about what's actually happening when "adding" a config.

[tdruez]

Missing a detailed docstring.

[tdruez]

Not ideal to set values from all the way down here, shouldn't we return those to a higher location that will explicitly set the values?