The numpy python package is not available (but is available in a normal shell)

[paulovieira]

Describe the bug

I want to use the gdal_calc.py utility, which is part of the gdal nixpkg.

gdal_calc.py imports the numpy python package: https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/gdal_calc.py

To Reproduce

This plain old shell.nix works as expected with nix-shell

{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {

  packages = with pkgs; [
    gdal
    python3
    python3Packages.numpy 
  ];

  shellHook = ''
    which python3;
    which gdal_calc.py;
    python3 -c "import sys; print('\n'.join(sys.path))";
    python3 -c "import numpy; print(numpy.__version__);  print(numpy.__file__)";
    gdal_calc.py --help;
  '';
}

However this other shell.nix, which uses the same packages via devshell, does not work:

{ 
  system ? builtins.currentSystem,
  pkgs ? import <nixpkgs> {},
  devshell ? import (fetchTarball "https://github.com/numtide/devshell/archive/master.tar.gz") { inherit system; }
}:

devshell.mkShell {
  
  devshell.packages = with pkgs; [
    gdal
    python3
    python3Packages.numpy 
  ];

  bash = {
    interactive = ''
      which python3;
      which gdal_calc.py;
      python3 -c "import sys; print('\n'.join(sys.path))";
      python3 -c "import numpy; print(numpy.__version__);  print(numpy.__file__)";
      gdal_calc.py --help;
    '';
  };

}

When executing gdal_calc.py I get the error "ModuleNotFoundError: No module named 'numpy'"

Expected behavior

I expected that the 'numpy' python module was available.

System information

$ nix-shell -p nix-info --command "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 4.4.0-210-generic, Ubuntu, 16.04.7 LTS (Xenial Xerus)`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.7.0`
 - channels(pvieira): `"nixpkgs"`
 - nixpkgs: `/home/pvieira/.nix-defexpr/channels/nixpkgs`

Additional context

The problem remains when using the packages attribute as

packages = with pkgs; [
  gdal
  python3
  (python3.withPackages(p: [ p.numpy ]))
];

That is, works as expected with pkgs.mkShell but doesn't work with devshell.mkShell. In fact I have to use (hiPrio python3) instead of python3 because of some collision error, but that seems to be unrelated.

I'm a beginner in the nix world so I don't really understand the difference between using python3Packages.numpy and python3.withPackages(p: [ p.numpy ]).

Looking at the output of sys.path in the 2 shell.nix above, the difference is clear: the second one has only

/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib/python39.zip
/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib/python3.9
/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib/python3.9/lib-dynload
/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib/python3.9/site-packages

while the first one has those same 4 paths and also these extra 2 paths:

/nix/store/07lvs63zjfxb6c63myb3k06xbz77f4fg-gdal-3.4.1/lib/python3.9/site-packages
/nix/store/hzi6fdch87l6hx9yyb6c3c2qbapczpg0-python3.9-numpy-1.21.5/lib/python3.9/site-packages

So I guess my question is: what should I do to have the same sys.path in devshell?

Thanks for your time and for this great tool!

[jfroche]

gdal is python as well, it works if you package python with the two packages:

{ system ? builtins.currentSystem
, pkgs ? import <nixpkgs> { }
, devshell ? import (fetchTarball "https://github.com/numtide/devshell/archive/master.tar.gz") { inherit system; }
}:

devshell.mkShell {

  devshell.packages = with pkgs; [
    (python3.withPackages(p: [ p.numpy p.gdal ]))
  ];

  bash = {
    interactive = ''
      which python3;
      which gdal_calc.py;
      python3 -c "import sys; print('\n'.join(sys.path))";
      python3 -c "import numpy; print(numpy.__version__);  print(numpy.__file__)";
      gdal_calc.py --help;
    '';
  };

}

[paulovieira]

Ah, thank you so much @jfroche!

I had actually cloned NixOS/nixpkgs and searched for some python package named *gdal* in pkgs/development/python-modules. But now I understand that it is defined only in pkgs/top-level/python-packages.nix (using the toPythonModule function).

But I would like to suggest that devshell should somehow support buildInputs, which would probably solve these kind of discrepancies between pkgs.mkShell and devshell.mkShell. I searched the past issues and it looks like it happened before:

• Propagating build inputs #56
• Cannot find dependency headers when trying to install C-based Python package (pylibmc) #172

And it seems that PR #6 could actually implement this!

Thanks.