No such file or directory: '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/torchgen/packaged/ATen/native/native_functions.yaml'

[AkiSakurai]

It is not necessary for the package to be located in get_python_lib.

For example, when compiling PyTorch from source using python3 setup.py develop, the package is located directly in the source folder.

Note that I cannot use the prebuilt PyTorch package, as it is no longer available on Intel Macs.

importlib.util.find_spec('torchgen') might be more portable.

diff --git a/build/Codegen.cmake b/build/Codegen.cmake
index 435b3d2..f772e36 100644
--- a/build/Codegen.cmake
+++ b/build/Codegen.cmake
@@ -79,19 +79,19 @@ function(generate_bindings_for_kernels)
   execute_process(
     COMMAND
       "${PYTHON_EXECUTABLE}" -c
-      "from distutils.sysconfig import get_python_lib;print(get_python_lib())"
+      "import importlib;print(importlib.util.find_spec('torchgen').submodule_search_locations[0])"
     OUTPUT_VARIABLE site-packages-out
     ERROR_VARIABLE site-packages-out-error
     RESULT_VARIABLE site-packages-result
     WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
     OUTPUT_STRIP_TRAILING_WHITESPACE
   )
-  file(GLOB_RECURSE _torchgen_srcs "${site-packages-out}/torchgen/*.py")
+  file(GLOB_RECURSE _torchgen_srcs "${site-packages-out}/*.py")
   set(_gen_command
       "${PYTHON_EXECUTABLE}" -m torchgen.gen_executorch
       --source-path=${EXECUTORCH_ROOT}/codegen --install-dir=${_out_dir}
-      --tags-path=${site-packages-out}/torchgen/packaged/ATen/native/tags.yaml
-      --aten-yaml-path=${site-packages-out}/torchgen/packaged/ATen/native/native_functions.yaml
+      --tags-path=${site-packages-out}/packaged/ATen/native/tags.yaml
+      --aten-yaml-path=${site-packages-out}/packaged/ATen/native/native_functions.yaml
       --op-selection-yaml-path=${_oplist_yaml}
   )

[JacobSzwejbka]

@larryliu0820 @dbort @GregoryComer can you take a look?

[larryliu0820]

Thanks for your suggestion and fix! Marking this as resolved.