Add tests

[lukel97]

Copy over most of the functional tests from https://github.com/haskell/haskell-ide-engine. The vast majority of these shouldn't need any changes.

We also need some general smoke tests for things like implicit cradle discovery etc

[alanz]

I think the initial basic task is to just set up the basics of having tests, perhaps with the ones for diagnostics, as those are already present in ghcide.

One philosophical discussion we need to have is about test frameworks. ghcide uses tasty, hie uses hspec.

Should we harmonise on a single one? If so, which one?

ping @cocreature @ndmitchell @jneira @fendor @bubba

[fendor]

Yes we should harmonise!
I dont have a preference, though, I like both frameworks.
EDIT: I am fine with almost anything, as long as I dont have to look at @?= operators (HUnit).

[alanz]

I think the choice is between tasty and hspec. And I am pretty sure that Digital Asset has a lot of other software built on top of ghcide, and around it, using tasty. And this is likely to tip the balance.

For the record, I am happy with either of them too.

[ndmitchell]

We did discuss which test framework to use, but I can't remember why we landed on the one we did - I've got no real preference.

[alanz]

Decision time: we will use tasty.

a) This is already in use for ghcide, so there is current and future commonality
b) The hie tests are going to have to be reworked anyway, to bring them over
c) hie is going to be deprecated in future, so ghcide / hls compatibility is more important

Postscript: (2020-05-10). Shortly after posting this we reversed the decision, and went for hspec. Apologies to all for confusion caused by not communicating this properly.

[jeffwindsor]

I can help on this ticket, let me know which tests are still to be ported over. I see Main, FormatSpec, FunctionalSpec and Utils (PluginSpec seems to be new), just don't want to step on any efforts in flight.

[fendor]

@jeffwindsor
Stuff I can think of:

• Refactor format tests
  • Unify test cases
  • Add more parsing examples
  • Make it easier to extend (e.g. if we want to add another formatter)
• Pragma Tests
• Any unit-tests that are applicable

I dont think there are any ongoing efforts regarding this. Also, feel free to port any plugin from hie to hls that you want.

[jeffwindsor]

Sounds good, I started a structure over in my fork (https://github.com/jeffwindsor/haskell-language-server/tree/issue-5/test/functional) and will put up a pull request as soon as I have something significant.

[jeffwindsor]

I have all the tests converted and moved, but ran out of time before i was able to wire the CI / config / runner. I went ahead and put up the pull request for comment. Locally I am not able to get the test to connect to the server it is looking for (or perhaps run server properly), so although all tests compile and run, most are failing due to timeouts. Any suggestions are welcome

[jneira]

Hi, the build with ghc-8.8.3, cabal and index-state: 2020-05-17T20:25:21Z in the cabal.project is failing in master with:

Failed to build tasty-ant-xml-1.0.0.5.
Build log (
C:\Users\atrey\AppData\Roaming\cabal\logs\ghc-8.8.3\tasty-ant-xml-1.0.0.5-9f5994c1308e334cdf9ece3d9c7c4e1ad0ac7d59.log       
):
Preprocessing library for tasty-ant-xml-1.0.0.5..
Building library for tasty-ant-xml-1.0.0.5..
[1 of 1] Compiling Test.Tasty.Runners.AntXML ( Test\Tasty\Runners\AntXML.hs, dist\build\Test\Tasty\Runners\AntXML.o )        

Test\Tasty\Runners\AntXML.hs:105:15: error:
    Not in scope: data constructor ‘Tasty.Exception’
    Perhaps you meant one of these:
      ‘Tasty.Option’ (imported from Test.Tasty.Options),
      ‘Tasty.Executing’ (imported from Test.Tasty.Runners),
      variable ‘Tasty.setOption’ (imported from Test.Tasty.Options)
    Neither ‘Test.Tasty.Options’,
            ‘Test.Tasty.Providers’ nor ‘Test.Tasty.Runners’ exports ‘Exception’.
    |
105 |               Tasty.Exception e -> pure $
    |               ^^^^^^^^^^^^^^^
cabal.exe: Failed to build tasty-ant-xml-1.0.0.5 (which is required by
test:func-test from haskell-language-server-0.1.0.0). See the build log above
for details.

Maybe a recent incompatible version bump?

It is picking a really old lib version, stack picks tasty-ant-xml-1.1.6

[jneira]

cabal freeze -w ghc-8.8.3 is picking those versions:

             any.tasty ==1.3.1,
             tasty +clock,
             any.tasty-ant-xml ==1.0.0.5,
             any.tasty-expected-failure ==0.11.1.2,
             any.tasty-hunit ==0.10.0.2,
             any.tasty-quickcheck ==0.10.1.1,
             any.tasty-rerun ==1.1.17,

The build succeded in ci after adding tasty: https://circleci.com/gh/haskell/haskell-language-server/379?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link
My next commit to master only changed the index state 🤔

[jneira]

Last issue resolved by #119

[jneira]

Tests already exist, although most of them are ignored. However i think we can close this one and opened a new one.