Issues porting a large, legacy Fortran codebase to Meson

[AndreasKempfNEC]

Dear Meson team,

as a preamble, I would like to mention that I am still pretty unfamiliar with Meson and its code in particular. Some of the stuff below might be a result of my insufficient understanding.

While trying to port a large, old Fortran codebase (Fortran 90+ mixed with some Fortran 77) to the Meson build system, I hit three problems. Old Fortran codes often do not follow any kind of consistent style so they are kind of a worst-case scenario.

Firstly, the Fortran module dependency generator seems to be case-sensitive. This leads to the following problem,

$ cat test1.f90
MODULE TEST1
  CONTAINS
    SUBROUTINE T1
    END SUBROUTINE
END MODULE
$ cat test2.f90
module test2
  contains
    subroutine t2
    end subroutine
end module'
$ cat main.f90
program prog
  use test1
  use TEST2
end program
$ cat meson.build
project('testing', 'fortran')

exe = executable('test.meson.exe', ['main.f90', 'test1.f90', 'test2.f90'])
$ meson build
...
$ cat build/meson
grep main.f90 build/build.ninja 
build test.meson.exe@exe/main.f90.o: fortran_COMPILER ../main.f90
 DEPFILE = test.meson.exe@exe/main.f90.o.d
build test.meson.exe: fortran_LINKER test.meson.exe@exe/main.f90.o test.meson.exe@exe/test1.f90.o test.meson.exe@exe/test2.f90.o

As can be seen, the module dependencies are not picked up by Meson, the line should be,

build test.meson.exe@exe/main.f90.o: fortran_COMPILER ../main.f90 | test.meson.exe@exe/test1.mod test.meson.exe@exe/test2.mod

(assuming, at least, that the filenames are consistently lower case...)

For a simple fix, I just changed two lines in ninjabackend.py so that everything is lower case, although there is probably a better solution,

$ diff ninjabackend-orig.py ninjabackend-test.py 
1643c1643
<                         module_files[modname] = s
---
>                         module_files[modname.lower()] = s
1655c1655
<                     usename = usematch.group(1)
---
>                     usename = usematch.group(1).lower()

Secondly, mixed Fortran+C codes seem to be linked using the C linker instead of the Fortran linker. As far as I know, it is usually recommended to use the Fortran linker [1]. I just switched the order of clike_langs in compilers.py, which seems to be all that is required:

$ diff ~/build/meson-0.39.1/mesonbuild/compilers.py ~/compilers.py 
52c52
< clike_langs = ('objcpp', 'objc', 'd', 'cpp', 'c', 'fortran',)
---
> clike_langs = ('objcpp', 'objc', 'd', 'fortran', 'cpp', 'c',)

Finally, I needed to work with preprocessed Fortran files, which quickly leads to pretty ugly edge cases. I assume the only way to solve this problem cleanly would be to use some kind of generator and to preprocess all files in a separate step (which probably brings its own set of problems). This might all be related to the following comment in compilers.py,

def get_dependency_gen_args(self, outtarget, outfile):
    # Disabled until this is fixed:
    # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=62162
    # return ['-cpp', '-MMD', '-MQ', outtarget]
    return []

I guess that this would no longer be an issue for GCC if this long-standing gfortran bug were to be fixed at some point? However, there are other compilers that might not generate Fortran dependencies cleanly, as well.

For now, I tried the following changes to ninjabuild.py which would pick up "use ..." statements in included headers,

$ diff ninjabackend-orig.py ninjabackend.py
1643c1643
<                         module_files[modname] = s
---
>                         module_files[modname.lower()] = s
1648a1649
>         incre = re.compile(r'#include\s+"([\w.]+)"')
1655c1656
<                     usename = usematch.group(1)
---
>                     usename = usematch.group(1).lower()
1676a1678,1690
>                 incmatch = incre.match(line)
>                 if incmatch is not None:
>                     incfile = incmatch.group(1)
>                     inc_same_dir = os.path.join(os.path.dirname(src), incfile)
>                     if os.path.exists(inc_same_dir):
>                         mod_files += self.get_fortran_deps(compiler, inc_same_dir, target)
>                         continue
>                     for dirs in target.include_dirs:
>                         for incdir in dirs.get_incdirs():
>                             fullpath = os.path.join(incdir, incfile)
>                             if os.path.exists(fullpath):
>                                 mod_files += self.get_fortran_deps(compiler, fullpath, target)

But this is certainly an ugly hack for an ugly problem. Moreover, it still does not pick up dependencies on the header files themselves and #ifdefs are ignored. The former issue could probably be hacked in a similar way although I am not sure how to handle absolute/relative paths and include directories. For my porting work, this at least allows for a proper build. It will probably break as soon as a header file is touched, though. Building Fortran is often pretty grisly but I worry that C++ might run into similar issues if modules were to be introduced at some point [1]. (cmake somehow seems to cope with preprocessed Fortran but I have no idea how they do it)

All of this is, of course only relevant if Fortran is supposed to be well supported by Meson and if these problems are not caused by my own stupidity.

Apart from the issues above, I noticed that there can be problems with long build commands that I reported as a Ninja bug.
ninja-build/ninja#1261
In retrospect, this could really be an issue with Meson. Depending on the reaction by the Ninja devs, I might open a separate issue here for that.

[1] https://software.intel.com/en-us/node/678447
[2] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3347.pdf

[nirbheek]

Great to see your work here! It looks like most of your analysis is indeed correct. Could you please open a PR with these changes as separate commits? That way we can do code review and discussion in a much nicer way. We do intend to support Fortran and would be glad to make things work well for you.

The problem with long build commands might be related to #671. Could you paste one such command line here?

[nirbheek]

What else is left here that you need to make Meson work for you?

[AndreasKempfNEC]

Sorry, I have not been especially responsive. Evaluating Meson for use in Fortran is mostly a side project for me.

The current version of Meson already works much better, thank you!

The issue I have left is the one with long build lines. I am not comfortable posting the exact line I have the problem with because it would reveal the code I am working on. It is, however very simple in principle: Whenever the command is extremely long (> 128k, e.g., trying to pack 10000 object files into a .a archive), it can hit the MAX_ARG_STRLEN limit and the command just fails. Apparently, .rsp files are needed for that to work and they are currently only used on Windows systems, as far as I can see.
If I replace the condition in the following line with True, an rsp file is used even on my Linux system and the archive can be created properly:

if mesonlib.is_windows() and not isinstance(static_linker, ArLinker):

[nirbheek]

RSP files are great for skirting around quoting issues and argument length limits, but they are also bad for usability because you don't get to see the command that was run. We should fetch $ getconf ARG_MAX (in pure python if possible), and if the argument length is greater than that, we should use an rsp file for that command.

[scivision]

@AndreasKempfNEC there has been a lot of Fortran functionality added to Meson even this year. Would you be willing to try Meson again and see if these or other issues are still a problem?

[scivision]

J believe this type of issue has been fixes in the past year in Meson, including by #4978 link_language option.

[burp]

Unfortunately this patch does not seem to work and the supplied test case does not seem to be a proper test of the functionality. This is because the C component is only supplied as a library and it will therefore select the Fortran compiler for linking. From a discussion in #mesonbuild we figured out that "as far as executable() can tell, it is fortran-exclusive code, and shared libraries don't have a language type."

A minimal example is:

project('tproj', ['fortran','cpp'])
executable('run', ['test.f90','test.cpp'], link_language: 'fortran')

then it generates rule cpp_LINKER with the gcc command instead of gfortran