Manual cherry-pick to update numpy as well as all dependencies (#7727)

* Install torchao by default (#7470)

Summary:

Now that torchao is not experimental anymore, we can rely on torchao directly via git submodule.

Part of #7460

(cherry picked from commit a861294)

* Upgrade numpy version requirement to 2.0 (#7599)

Upgrade numpy version requirement to

Summary:

1.21 is too old now. For 3.10 or above, we should be able to use 2.0.0

We are already on CoreMLTools 8.1, which is https://github.com/apple/coremltools/blob/8.1/reqs/build.pip

We had to upgrade to the latest transformers as well

(cherry picked from commit a7b5297)

* Update torchtune pin (#7670)

(cherry picked from commit a18f6e8)

* [Arm] TOSA reference model update

* Install release version of torchtune

---------

Co-authored-by: Sicheng Stephen Jia <[email protected]>

Author: mergennachin

.ci/docker/requirements-ci.txt

@@ -1,17 +1,14 @@
 mpmath==1.3.0
-numpy==1.21.3; python_version == '3.10'
-numpy==1.23.2; python_version == '3.11'
-numpy; python_version >= '3.12'
+numpy==2.0.0; python_version >= '3.10'
 PyYAML==6.0.1
 ruamel.yaml==0.17.32
 sympy==1.12
 timm==0.6.13
 tomli==2.0.1
 torchsr==1.0.4
-transformers==4.38.0
+transformers==4.47.1
 zstd==1.5.5.1
-pandas==2.0.3; python_version == '3.10'
-pandas; python_version >= '3.11'
+pandas==2.2.2; python_version >= '3.10'
 pytest==7.2.0
 pytest-cov==4.1.0
 expecttest==0.1.6
@@ -24,7 +21,7 @@ sphinx-gallery==0.14.0
 breathe==4.34.0
 exhale==0.2.3
 docutils==0.16
-matplotlib==3.7.2
+matplotlib==3.9.4
 # PyTorch Theme
 -e git+https://github.com/pytorch/pytorch_sphinx_theme.git#egg=pytorch_sphinx_theme
 myst-parser==0.18.1

backends/apple/coreml/scripts/install_requirements.sh

@@ -47,11 +47,7 @@ cmake --build "$COREMLTOOLS_DIR_PATH/build" --parallel
 
 echo "${green}ExecuTorch: Installing coremltools."
 pip install "$COREMLTOOLS_DIR_PATH"
-# CoreMLTools have started supporting numpy 2.0,
-# but ExecuTorch example model test env is still using older transformers,
-# so for now we will need to downgrade numpy to 1.x
-# TODO: Remove this numpy downgrade once later transformers starts to be used
-pip install numpy==1.26.4
+
 STATUS=$?
 if [ $STATUS -ne 0 ]; then
     echo "${red}ExecuTorch: Failed to install coremltools."

examples/arm/setup.sh

@@ -2,7 +2,7 @@
 # Copyright (c) Meta Platforms, Inc. and affiliates.
 # All rights reserved.
 #
-# Copyright 2023-2024 Arm Limited and/or its affiliates.
+# Copyright 2023-2025 Arm Limited and/or its affiliates.
 #
 # This source code is licensed under the BSD-style license found in the
 # LICENSE file in the root directory of this source tree.
@@ -88,7 +88,7 @@ ethos_u_base_rev="24.08"
 
 # tosa reference model
 tosa_reference_model_url="https://review.mlplatform.org/tosa/reference_model"
-tosa_reference_model_rev="c5570b79e90c3a36ab8c4ddb8ee3fbc2cd3f7c38"
+tosa_reference_model_rev="v0.80.1"
 
 # vela
 vela_repo_url="https://review.mlplatform.org/ml/ethos-u/ethos-u-vela"

examples/models/llama/install_requirements.sh

@@ -9,15 +9,10 @@
 # Install sentencepiece for llama tokenizer
 pip install snakeviz sentencepiece
 
-# Install torchao.
-pip install "$(dirname "$0")/../../../third-party/ao"
-
 # Install lm-eval for Model Evaluation with lm-evalution-harness
 # Install tiktoken for tokenizer
 pip install lm_eval==0.4.5
 pip install tiktoken blobfile
-# Restore numpy if >= 2.0
-pip install "numpy<2.0"
 
 # Call the install helper for further setup
 python examples/models/llama/install_requirement_helper.py

examples/models/llama3_2_vision/install_requirements.sh

@@ -5,10 +5,5 @@
 # This source code is licensed under the BSD-style license found in the
 # LICENSE file in the root directory of this source tree.
 
-NIGHTLY_VERSION="dev20241112"
-
 # Install torchtune nightly for model definitions.
-pip install --pre torchtune==0.4.0.${NIGHTLY_VERSION} --extra-index-url https://download.pytorch.org/whl/nightly/cpu --no-cache-dir
-
-# Install torchao.
-pip install "$(dirname "$0")/../../../third-party/ao"
+pip install torchtune==0.5.0 --no-cache-dir

examples/models/phi-3-mini-lora/install_requirements.sh

@@ -8,6 +8,3 @@
 pip install torchvision
 pip install torchtune
 pip install tiktoken
-
-# Install torchao.
-pip install "$(dirname "$0")/../../../third-party/ao"

install_requirements.py

@@ -133,7 +133,7 @@ def python_is_compatible():
     "timm==1.0.7",
     "torchaudio==2.6.0" if USE_PYTORCH_NIGHTLY else "torchaudio",
     "torchsr==1.0.4",
-    "transformers==4.46.1",
+    "transformers==4.47.1",
 ]
 
 # pip packages needed for development.
@@ -166,6 +166,23 @@ def python_is_compatible():
     check=True,
 )
 
+LOCAL_REQUIREMENTS = [
+    "third-party/ao",  # We need the latest kernels for fast iteration, so not relying on pypi.
+]
+
+# Install packages directly from local copy instead of pypi.
+# This is usually not recommended.
+subprocess.run(
+    [
+        sys.executable,
+        "-m",
+        "pip",
+        "install",
+        *LOCAL_REQUIREMENTS,
+    ],
+    check=True,
+)
+
 #
 # Install executorch pip package. This also makes `flatc` available on the path.
 # The --extra-index-url may be necessary if pyproject.toml has a dependency on a

pyproject.toml

@@ -55,12 +55,9 @@ dependencies=[
   "flatbuffers",
   "hypothesis",
   "mpmath==1.3.0",
-  "numpy==1.21.3; python_version == '3.10'",
-  "numpy==1.23.2; python_version == '3.11'",
-  "numpy; python_version >= '3.12'",
+  "numpy==2.0.0; python_version >= '3.10'",
   "packaging",
-  "pandas==2.0.3; python_version == '3.10'",
-  "pandas; python_version >= '3.11'",
+  "pandas==2.2.2; python_version >= '3.10'",
   "parameterized",
   "pytest",
   "pytest-xdist",