externalize RPM metadata and make it version based

[tomerd]

motivation: share package metadata for swift versions and platforms

changes:

• introduce new directory for share metadata
• create metadata file for rpms
• update centos 8 spec and script to use the shared metadata

[tomerd]

@shahmishal this builds on top of #32, see 48f2f7e for the actual changes, once we merge #32 I will rebase on top

[tomerd]

@shahmishal @tachoknight @futurejones wdyt about this technical approach to achieve some of the ideas mentioned in #23 (comment) ? we still need to figure out the actual values, so focus on the technical aspect of how this is wired up

[futurejones]

@tomerd, I think externalizing the metadata is the right approach but trying do this across all platforms and package managers in one global file may not be practical.
RPM packages and DNF or YUM package managers are a very different beast to DEB packages and the APT package manager.
The rules for including packages in the respective official repositories also differ greatly.
I really think that this project should be separated into the different package manager families of distributions.
We could have a global overview containing a project road map and a basic design document etc, and then drop down into the separate package manager specifics.

swift-installer-scripts
  └ platforms
        ├ Linux
        │   ├ README
        │   ├ Docs - contains master design docs, road map etc.
        │   ├ DEB
        │   │   ├ README
        │   │   ├ Docs - contains information specific to .deb packages
        │   │   ├ Ubuntu - contains information specific to the distribution
        │   │   │   ├ 18.04 - contains information specific to the version
        │   │   │   ├ 20.04
        │   │   │   ├ 21.04
        │   │   │   └ ...
        │   │   ├ Debian
        │   │   │   ├ 10
        │   │   │   ├ 11
        │   │   │   └ ...
        │   │   └ ...
        │   │
        │   ├ RPM
        │   │   ├ README
        │   │   ├ Docs - contains information specific to .rpm packages
        │   │   ├ Centos
        │   │   │   ├ 8
        │   │   │   └ ...
        │   │   ├ Fedora
        │   │   │   ├ 33
        │   │   │   ├ 34
        │   │   │   └ ...
        │   │   └ ...
        │   └ ...
        └ Windows
            └ ...

[tachoknight]

I really think that this project should be separated into the different package manager families of distributions.

This seems like just an additional level of indirection; if I want to build Swift for Ubuntu or Fedora, why not just go to the appropriate directory directly, without having to be know about the package type?

[futurejones]

@tachoknight, what I am discussing is not about building swift. Its about packaging the swift toolchain for distribution using the native packaging system.
Knowing the package type and what the different package managers require is the key point and the most relevant information. I wouldn't have thought of this as an 'additional level of indirection'.
If you want to build the swift toolchain then this is already covered at https://github.com/apple/swift.

[tomerd]

@futurejones i like the proposed directory structure and totally agree with your assessment

note this PR does not attempt to canonicalize the metadata across different package managers - the proposed shared metadata file is qualified to be for RPM only, so it can be shared across centos, amazon linux, etc

[tomerd]

what I am discussing is not about building swift. Its about packaging the swift toolchain for distribution using the native packaging system.
Knowing the package type and what the different package managers require is the key point and the most relevant information. I wouldn't have thought of this as an 'additional level of indirection'.

i suspect @tachoknight point was the explicit directory representing the type of package (eg RPM) is redundant as it can be implicitly derived from the linux distro

personally i can go either way, having that directory can help with sharing packager specific files like proposed in this PR

[futurejones]

Using DEB as an example, my thought was that there is metadata and relative information which is specific to the package system i.e. DEB and the same across all distributions and versions.

If you drop immediately down to the distribution level you are already starting to duplicate yourself unnecessarily.
There are no distribution level package differences between Ubuntu, Debian or other derivatives.
Essentially the only differences between the debian packages is on the distribution-version level.

Also I feel that putting all the RPM and DEB distributions side by side and in the one basket is wrong as they come from entirely different linux ecosystems.

[tomerd]

since we dont expect end-users to use this repo directly this is more of a concern to the maintainers, for which the additional directory could be useful. as such, i suggest we adopt the structure suggested by @futurejones. we can always change it later if we feel it's redundant.

@shahmishal @tachoknight wdyt?

[tomerd]

@futurejones @tachoknight @shahmishal updated to reflect the directory structure suggested by @futurejones which I think makes this nice.

should we move forward with this?

[tachoknight]

Admittedly I'm not sure how it makes it better than just going to the appropriate distro directory, but if that's how folks like it, then ¯_(ツ)_/¯.

[shahmishal]

The directory structure suggested by @futurejones works for me

[tomerd]

@shahmishal please merge when you are ready

[soloturn]

@shahmishal @tomerd for the arch linux and family build what @tachoknight suggests seems slimmer. the package scripts are so small that a directory structure imo over-complicates it.

[soloturn]

what i'd really would have loved this cool metadata idea would solve, that the dependencies can be easier extracted. e.g.

• a list of software and version which the build depends on, e.g. ninja, ICU, i.e. non-apple
• a list of urls to build swift, e.g. llvm-project
• a list of urls to be built when swift is built, swift source code, e.g. swift-crypto
• the list of URLs in representation git-url, maybe even a git repo with submodules so one can do a shallow clone with depth 1.
• the list of URLs in represenation binary download

because currently i do the following check update_checkout/update-checkout-config.json, and need to check if it is tag or branch as arch wants the direct https url to feed into "git clone".

[tomerd]

@shahmishal @tomerd for the arch linux and family build what @tachoknight suggests seems slimmer. the package scripts are so small that a directory structure imo over-complicates it.

it makes it easier to share metadata in a format that is agnostic to the package manager. the original PR had it in a separate place, but this makes it easy to reason about. in any case, this is something we can change later, so I suggest we continue with this for the sake of making progress

what i'd really would have loved this cool metadata idea would solve, that the dependencies can be easier extracted. e.g.
a list of software and version which the build depends on, e.g. ninja, ICU, i.e. non-apple
a list of urls to build swift, e.g. llvm-project
a list of urls to be built when swift is built, swift source code, e.g. swift-crypto
the list of URLs in representation git-url, maybe even a git repo with submodules so one can do a shallow clone with depth 1.
the list of URLs in represenation binary download
because currently i do the following check update_checkout/update-checkout-config.json, and need to check if it is tag or > branch as arch wants the direct https url to feed into "git clone".

this is the general direction I would like to go to, but this PR starts with the minimal and easy to extract and share metadata. I can see follow on PRs to extract these as well, especially since some of this may be version specific

[soloturn]

@shahmishal @tomerd for the arch linux and family build what @tachoknight suggests seems slimmer. the package scripts are so small that a directory structure imo over-complicates it.

it makes it easier to share metadata in a format that is agnostic to the package manager. the original PR had it in a separate place, but this makes it easy to reason about. in any case, this is something we can change later, so I suggest we continue with this for the sake of making progress

agreed, metadata should be as close as possible to the usage. but the structure seems really complex, let us digest this path: platforms/Linux/RPM/shared/metadata/5.5.0/metadata.inc. why it cannot be simply: rpm/metadata.inc, and rpm/fedora-8/package.spec, msi/whatebber ?

lets then look at why rpm should be there. you have the problem of operating systems where one could have multiple formats, e.g. for windows you have msi, chocolatey, winget, exe, msys pkg, cygwin and zip. for fedora you have rpm, and tar ball. arch linux based distros use shell scripting, just like tar ball.

another decision point is the version number. 5.5.0 is in the path while the whole structure is stored in a source code repo, where versions are usually tags, not in the path. of course there may be good reasons for doing it. a new version of swift-formater comes, and you want to change it for 5.5.x and 5.4.x in one go instead of checking out 20 branches and re-tag. but IFF you decide that this repo should not follow the same versioning scheme as other swift repos, why not making it so that you can drop prehistoric stuff easily later on, and put the version on top: 5.5.0/rpm/metadata.inc.

another aspect is the change rate. ideally this metadata contains volatile data like version numbers which is a pain. but - here i agree with you that it can be fixed any time.

in a summary, a simple directory structure would be cool. extracting later on is easy in case somebody finds it too much work to maintain. something flexible like:

swift-installer-scripts
  ├ fedora-33
  │   ├ package.spec
  │   └ ...
  ├ shared
  │   ├ rpm
  │   │   ├ metadata.inc
  │   │   └ ...
  │   └ ...
  ├ windows-11
  │   ├ winget
  │   │   ├ swiftlang.json
  │   │   └ ...
  │   ├ msi
  │   └ ...
  └ ...

in this proposal it is convention that fedora uses shared rpm. if you really like it, we might put 5.5.0 as top level directory.

[shahmishal]

There is empty readme - platforms/Linux/RPM/readme.md

Also, can we update the file name for platforms/Linux/RPM/centos/8/readme.md to README.md to be consistent with other readme's in the repo?