Remove GHC 8.10.2, 8.10.3, 8.10.4 and 8.10.5 (after the hls release including GHC 8.10.7)

[Ailrun]

Currently, GHC 8.10.5 in macOS experiences https://gitlab.haskell.org/ghc/ghc/-/issues/19968 in our CI (and in @konn's Mac), so we cannot replace GHC 8.10.2 with GHC 8.10.5 as planned in #1898.

We may be able to remove GHC 8.10.2 after GHC 8.10.6 with the fix above.

[mouse07410]

The mentioned problem only surfaces if Xcode Command Line Tools package is installed. On machines with Xcode.app only (which also provides command line tools!) this problem https://gitlab.haskell.org/ghc/ghc/-/issues/19968 is not present.

[Ailrun]

@mouse07410 Thank you for the information! @konn, could you check you can build HLS without that package? Or is that package necessary for some other things?

[mouse07410]

. . . build HLS without that package?

Trivial - with one caveat. On MacOS you need either Xcode.app or CLT installed.

Or is that package necessary for some other things?

No, CLT package is not necessary - unless one insists on not installing Xcode.app.

Something has to provide Clang, header files, and libraries - they come either from Xcode, or from CLT. It turns out that CLT (for reasons beyond my understanding) is often (always?) more buggy and less usable than the "main" app that also adds GUI IDE...

[jneira]

Mmm, we are using github action envs to build the release and it failed so we could check if that library is installed there.
In that case we could:

• try to change the libraries installed in the images used by gha
• build locally the version in some macos and upload it manually to the release (like we do with 8.8.4 and windows)

[mouse07410]

we are using github action envs to build the release

Unfortunately, I've zero experience/expertise on GHA. But with some digging, it seems that GHA provides

Xcode Command Line Tools 12.4.0.0.1.1610135815

But it also seems to have Homebrew Clang-12 and Xcode_12.4.app - see the screenshot below.

Also, I tried with CLT 12.5. Your GHA used CLT 12.4. Both failed.

Proving that CLT is the problem, for which we need either a workaround (I've no idea how), or a bypass.

IMHO, you could be either referring to stuff in /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk (underneath this directory path, look for usr/include/ and usr/lib), or using Homebrew-installed Clang-12 as shown above.

. . . and it failed

Would be nice to see what the cause/complaint was. Perhaps you could post the summary/excerpt?

. . . so we could check if that library is installed there

Perhaps one can search in the above places for that library (which library is it, BTW?)?

[konn]

@mouse07410 Thank you for your information! That helps a lot!

@Ailrun I've just confirmed that network package just builds after sudo rm -rf /Library/Developer/CommandLineTools (which is the official way of uninstalling CLT).
With this workaround, it seems that HLS also builds with GHC 8.10.5 on my environment.

[Ailrun]

@konn Awesome, could you upload that version for this release?

[konn]

Sure. Just uploading executable built with stack with the proper name would suffice? Is there any description for the uploading procedure?

[Ailrun]

I guess @jneira knows well about that, as he uploads Windows binary regularly.

[jneira]

Sure. Just uploading executable built with stack with the proper name would suffice? Is there any description for the uploading procedure?

Yeah, the tar and the file contained in it have to follow the exact name schema as other ones. You can take as example the names of the existing one for ghc-8.10.4 and change the version.

Then you edit the release and attach the new file and the end of the editing form.

[konn]

@jneira Thank you for the explanation!

the tar and the file contained in it have to follow the exact name schema as other ones.

Changing the exe name after the build would suffice, or should I edit the name in .cabal file?
It seems there are two kinds of gzs:

1. A tarball of HLSs for multiple versions and wrapper, and
2. A .gz that contains only single executable for each specific os and GHC version.

So I must update (1) to include the exe for 8.10.5 and upload new (2) for 8.10.5, right?

[jneira]

Changing the exe name after the build would suffice, or should I edit the name in .cabal file?

changing the name after the build would be fine

So I must update (1) to include the exe for 8.10.5 and upload new (2) for 8.10.5, right?

Yeah, (1) is the used one by the automatic download of the vscode extension and (2) is used by ghcup to install hls for all ghc versions. When (2) is available the maintainer of ghcup, @hasufell could include it.

[jneira]

@mouse07410 Many thanks for hunting the bug and provide a workaround to build hls. Your insights has been indispensable to add support to macos+ghc8.10.5 quickly (i see you have updated the ghc ticket 😸 )

In the reddit announce for the new version hls some user had noted that hls could be built in some macos envs, and i was a little bit intrigued but i had no clue about what could be the cause.

And thanks @konn for build locally the new version.

[konn]

I finally uploaded binaries for HLS 1.2.0 + GHC 8.10.5. If anyone could use these binaries, I would re-edit release text in 1.2.0.

[jneira]

@konn not sure if the name of the binary is the correct one, it should be haskell-language-server-8.10.5 but it seems to be haskell-language-server-macOS-8.10.5 (see the contents of haskell-language-server-macOS-8.10.4.gz f.e.)

[konn]

@jneira Oops! I will change the name and re-upload .gz file shortly.

[jneira]

No worries, i am preparing a pr to add specific instructions for the manual upload to help in the process

[konn]

Hmm, to me, it seems that -1.2.0.tar.gz contains exes without -macOS suffix, but macOS-8.10.4 .gz and/or -linux-8.10.4.gz contains the ones WITH OS.

[jneira]

mmm, i've just double check haskell-language-server-macOS-8.10.4.gz and haskell-language-server-Linux-8.8.4.gz and i dont see the os in the binary file inside them, but maybe i am being totally dislexic or windows is cheating me 😄 , could someone check it as well?

[konn]

@jneira Oh, I forgot to pass -Noption to gunzip. Then I get the exe WITHOUT os name. Sorry for confusion...

[konn]

OK, I've just uploaded .gz file with the correct name.

[jneira]

@mouse07410

. . . and it failed

Would be nice to see what the cause/complaint was. Perhaps you could post the summary/excerpt?

The builds on macOsS + ghc-8.10.5 failed due to a timeout with no concrete error i am afraid, see https://github.com/haskell/haskell-language-server/runs/2770148984?check_suite_focus=true f.e. It was cancelled after 5h 55m

. . . so we could check if that library is installed there

Perhaps one can search in the above places for that library (which library is it, BTW?)?

I was referring to CLT/Clang thing you already have investigated

So the possible solutions would be:

• Wait to a workaround coming from the ghc ticket
• Try to tweak our build script to make ghc use the correct c toolchain (we can name it this way, no?). I dont have any experience on macOS to even know how to start that (maybe @konn could reproduce the steps followed in the local machine?)

[konn]

• Try to tweak our build script to make ghc use the correct c toolchain (we can name it this way, no?). I dont have any experience on macOS to even know how to start that (maybe @konn could reproduce the steps followed in the local machine?)

👍 for this option.

I think we can adopt the workaround to remove CLT as the tentative workaround for GHC 8.10.5 on macOS.
As a considerable amount of people successfully uses GHC 8.10.5 without CLT, I think we should continue to support GHC 8.10.5 + macOS in HLS for a while.

[konn]

I'm trying this #1931.

[mouse07410]

Something seems wrong with the tarballs. ghcup complains:

$ ghcup install hls
[ Info  ] downloading: https://github.com/haskell/haskell-language-server/releases/download/1.2.0/haskell-language-server-macOS-1.2.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   650  100   650    0     0    182      0  0:00:03  0:00:03 --:--:--   183
100  352M  100  352M    0     0  37.1M      0  0:00:09  0:00:09 --:--:-- 64.4M
[ Info  ] verifying digest of: haskell-language-server-macOS-1.2.0.tar.gz
[ Error ] Digest error: expected "43d2ef356fb8cbd8e27acf70f94c079157258916bb1220751547584513584aaa", but got "9863f00928f72cf0a16bd3f3578ac55704c721bc77148273d909fd895d05c783"
[ Error ] Also check the logs in /Users/ur20980/.ghcup/logs

and the log content:

Debug:  Identified Platform as: Darwin
Debug:  Unable to get/parse Last-Modified header
Debug:  downloading: https://www.haskell.org/ghcup/data/ghcup-0.0.5.yaml
Debug:  Requested to install hls version Version {_vEpoch = Nothing, _vChunks = (Digits 1 :| []) :| [Digits 2 :| [],Digits 0 :| []], _vMeta = [], _vRel = []}
Info: downloading: https://github.com/haskell/haskell-language-server/releases/download/1.2.0/haskell-language-server-macOS-1.2.0.tar.gz
Error: Download failed: Process "curl" with arguments ["-fL","-o","/private/var/folders/c6/lnc_0m093ys8w16md_fm1mnxhtfnj8/T/ghcup-eef831a335f5628a/haskell-language-server-macOS-1.2.0.tar.gz","https://github.com/haskell/haskell-language-server/releases/download/1.2.0/haskell-language-server-macOS-1.2.0.tar.gz"] failed with exit code 22.
Error: Also check the logs in /Users/ur20980/.ghcup/logs

Also reported at https://gitlab.haskell.org/haskell/ghcup-hs/-/issues/149

[konn]

Oops, I'm just re-uploading the stuff now, so it must be the cause. It took a while to fully upload new tar ball.

[konn]

OK, all the gzips and tarballs replaced, so ghcup needs to sync the digests.

[jneira]

OK, all the gzips and tarballs replaced, so ghcup needs to sync the digests.

We have to be in touch with @maerwald when changing release artifacts.
Alternatively we can do a pull request to update the digests like this one: https://gitlab.haskell.org/haskell/ghcup-hs/-/merge_requests/83

[mouse07410]

Is there a way to get the digests updated in a more-or-less timely manner?

[jneira]

Is there a way to get the digests updated in a more-or-less timely manner?

do you mean in an automatic way? i think we talked about that with @maerwald but iirc the manual pr was preferred

[mouse07410]

do you mean in an automatic way? i think we talked about that with @maerwald but iirc the manual pr was preferred

I think I understand the security implications well.

What's the usual time lag between the appearance of a release and it's availability?

And in any case, it seems wrong to update one but not the other.

I'd like to eliminate the current situation when the updater (ghcup, to be specific) get broken and is unable to install HLS at all - the tarball is already "new" but the digest is still "old".

[jneira]

Well, we have to make a balance between make available the general release to users asap even without some versions and to keep in sync the different ways to distribute hls . Windows+ghc-8.8.4 for example was uploaded after the release as we usually do. But we did not have support in ghcup and windows until very recently.

The macOS affair was a little bit different: we assumed we would have no chance to have a version for 1.2.0 and 8.10.5 but finally we got it with your help, once the release was closed.

But i hope it has been a extraordinary situation. However it could happen again and we should be ready to handle it, with not much maintainer effort if possible.

For example we could wait to have the pr for ghcup merged to mark the release as definitive and sync both releases. But other users will have to wait to download it automatically with the extension. ghcup is not the only way to distribute hls, we have a brew formula, a chocolatey package, etc. Keep in sync all ways to distribute it will be a little bit crazy (although arguably ghcup is the second more used way to get hls so it could be privileged)

As for the updated releases we could stick to not change them never and make new releases each time we want to change artifacts. That way we will not break any distribution based on hashes. But that would suppose maintainer effort and bump versions everywhere.

So i think it is not an easy decision 🙂

[mouse07410]

The update is still broken - now it shows a different digest from the one the PR asked to change to:

$ ghcup install hls
[ Info  ] downloading: https://github.com/haskell/haskell-language-server/releases/download/1.2.0/haskell-language-server-macOS-1.2.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   650  100   650    0     0   3048      0 --:--:-- --:--:-- --:--:--  3170
100  351M  100  351M    0     0  40.5M      0  0:00:08  0:00:08 --:--:-- 52.8M
[ Info  ] verifying digest of: haskell-language-server-macOS-1.2.0.tar.gz
[ Error ] Digest error: expected "9863f00928f72cf0a16bd3f3578ac55704c721bc77148273d909fd895d05c783", but got "a310d8a3e9c5c4218210f750682c74a0f82ad0f59995adde0dbe775115b1e357"
[ Error ] Also check the logs in /Users/ur20980/.ghcup/logs
$

And the log:

Debug:  Identified Platform as: Darwin
Debug:  Unable to get/parse Last-Modified header
Debug:  downloading: https://www.haskell.org/ghcup/data/ghcup-0.0.5.yaml
Debug:  Requested to install hls version Version {_vEpoch = Nothing, _vChunks = (Digits 1 :| []) :| [Digits 2 :| [],Digits 0 :| []], _vMeta = [], _vRel = []}
Info: downloading: https://github.com/haskell/haskell-language-server/releases/download/1.2.0/haskell-language-server-macOS-1.2.0.tar.gz
Info: verifying digest of: haskell-language-server-macOS-1.2.0.tar.gz
Error: Digest error: expected "9863f00928f72cf0a16bd3f3578ac55704c721bc77148273d909fd895d05c783", but got "a310d8a3e9c5c4218210f750682c74a0f82ad0f59995adde0dbe775115b1e357"
Error: Also check the logs in /Users/ur20980/.ghcup/logs

For example we could wait to have the pr for ghcup merged to mark the release as definitive and sync both releases. But other users will have to wait to download it automatically with the extension.

In general, there should be a way to update neither or both: the tarball and its digest. Unless they're in lock-step, digest validation loses it's meaning and purpose.

ghcup is not the only way to distribute hls, we have a brew formula, a chocolatey package, etc.

Definitely not the only way - but I assume rather few people would install Haskell platform via Brew, Macports (yes, it's available there - I never touched it and don't intend to), Chocolatey, etc. To me it's not a good or common way.

IMHO (and I may be biased), sane people install Rust via rustup, Haskell - via ghcup, etc.

Keep in sync all ways to distribute it will be a little bit crazy

The point here is - making sure the the available tarballs and their corresponding digests are in sync.

(although arguably ghcup is the second more used way to get hls so it could be privileged)

Unless you have the numbers to back your point (and I don't need to see them), I say that now ghcup is the most used way to get the Haskell platform - GHC, HLS, Cabal, and now Stack. So, I do recommend making it "privileged".

As for the updated releases we could stick to not change them never and make new releases each time we want to change artifacts. That way we will not break any distribution based on hashes. But that would suppose maintainer effort and bump versions everywhere.

Unless you think that the maintainers manually update their stuff, rather than having their code pulling what's present on the distribution sites, I see no extra load on the maintainers. Oh, wait - Macports and Brew require creating a new version manually. But they're usually/always behind anyway. So, if releases are coming out too fas for them, they can skip some. ;-)

[hasufell]

As for the updated releases we could stick to not change them never and make new releases each time we want to change artifacts.

No need... you can do haskell-language-server-macOS-1.2.0_p1.tar.gz (patch level) or haskell-language-server-macOS-1.2.0-r1.tar.gz (revision) and upload them to the current release assets, which is common practice. Just avoid updating the existing tarball in-place.

---

Another discussion: https://gitlab.haskell.org/haskell/ghcup-hs/-/issues/156 but I'm really on the fence on this one

[mouse07410]

i hope it has been a extraordinary situation. However it could happen again . . .

@jneira this extraordinary situation has already happened again. The current tarball and its presumed digest are out of sync again. I.e., security-induced Denial of Service.

[jneira]

Ok, thanks for your thoughts and proposals.
So from now on i think we could:

• Not touch tarballs generated for ghcup. If we have to update them like this time with macOS we will upload a new tarball on the same release with -r1, following @hasufell suggestion
• Try to keep in sync our releases and their publish in ghcup being in touch with ghcup maintainers and sharing the needed work (making the pr basically)

If you agree i included both points in the release document.

//cc @Ailrun @berberman, as you have done the hard work of cutting releases lately

[mouse07410]

I like both of your suggestions, and think that's the way forward.

[jneira]

Ok, we are about to add ghc-8.10.6 support with many fixes for macos. I would wait to make a release with actual 8.10 versions set and remove all but 8.10.5 and 8.10.6 (so 8.10.2, 8.10.3 and 8.10.4)

[mouse07410]

Would the fixes apply to the 9.0.1 version?

Since 8.10.6 had been released - is it time to release the HLS update?

[jneira]

I hope there will be a 9.0.2 release with the backport see
https://gitlab.haskell.org/ghc/ghc/-/milestones/367
A new release with both would be needed, yeah

[jneira]

8.10.7 will be released soon, as 8.10.6 did not contain one of the fixes for macos, see #2096 (comment)

[tmcgilchrist]

Any chance of getting HLS releases for 8.10.7 added to https://github.com/haskell/haskell-language-server/releases/tag/1.3.0 now that it is released?

[jneira]

@tmcgilchrist we could try it, but add to the ghcup release will be a little bit more complex as we have to assemble a new tar with all ghcs including the new one and update ghcup config

we are making releases each month so 1.4.0 will not take a long time (i hope! 🙂). In the meanwhile the workaround could be build from source (once the ghc-8.10.7 support is added)

[CThuleHansen]

@jneira so it is expected to be part of 1.4.0?

[jneira]

@jneira so it is expected to be part of 1.4.0?

yeah

[jneira]

closed in favour of the generic #2168