Skip to content

[llvm-exegesis][AArch64] Disable pauth and ldgm as unsupported instructions fixed #136868

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Apr 28, 2025

Conversation

lakshayk-nv
Copy link
Contributor

[llvm-exegesis][AArch64] Disable pauth and ldgm as unsupported instructions.
Skipping AUT and LDGM opcode variants which currently throws "illegal instruction".

Last pull request #132346 got reviewed and merged but builder bot got failed. This was due to undefined PR_PAC_SET_ENABLED_KEYS utilized were not defined in x86 arch, resulting in build failure.

This is followup to merge the changes with following changes to fixup the build failure.

Changes:

  • Fixed up the problem with arch specific check for prctl library import
  • Defining PR_PAC_SET_ENABLED_KEYS if undefined.

1) Checking opcodes specifically for LDGM and AUT opcode instruction variants.
2) Gracefully exiting with "<opcode-name> : Unsupported opcode: isPointerAuth/isUncheckedAccess"
3) Added corresponding test cases to check exit message.
- Replace hardcoded opcode ranges with AArch64::* enum values for pointer auth and
unchecked access instructions.
- Add required AArch64 headers and update tests.
- Style changes for corresponding testcases.
…ctions.

1) Change location of isPointerAuth and isLoadTagMultiple function across file for correct implementation
2) Style nits changes in test cases.
- Included necessary headers for PR_PAC_* constants and modified the getIgnoredOpcodeReasonOrNull function to disabling pointer authentication.
- Adjusted formatting.
…ication and load tag multiple

- Moved isPointerAuth and isLoadTagMultiple functions to AArch64/Target.cpp for better organization.
- Updated getIgnoredOpcodeReasonOrNull to handle pointer authentication and load tag multiple opcodes correctly.
- Removed redundant definitions from Target.cpp and Target.h.
…Linux

- Added platform-specific handling for pointer authentication instructions, enabling the disabling of PAC keys on Linux.
- Ensured that unsupported opcodes return an appropriate message on non-Linux platforms.
To resolve build failure due to undefined PR_PAC_SET_ENABLED_KEYS.
1) Added stricter check for OS requirement.
2) Initialized variable used if still undefined.
@llvmbot
Copy link
Member

llvmbot commented Apr 25, 2025

@llvm/pr-subscribers-tools-llvm-exegesis

Author: Lakshay Kumar (lakshayk-nv)

Changes

[llvm-exegesis][AArch64] Disable pauth and ldgm as unsupported instructions.
Skipping AUT and LDGM opcode variants which currently throws "illegal instruction".

Last pull request #132346 got reviewed and merged but builder bot got failed. This was due to undefined PR_PAC_SET_ENABLED_KEYS utilized were not defined in x86 arch, resulting in build failure.

This is followup to merge the changes with following changes to fixup the build failure.

Changes:

  • Fixed up the problem with arch specific check for prctl library import
  • Defining PR_PAC_SET_ENABLED_KEYS if undefined.

Full diff: https://github.com/llvm/llvm-project/pull/136868.diff

2 Files Affected:

  • (added) llvm/test/tools/llvm-exegesis/AArch64/skip_unsupported_instructions.s (+10)
  • (modified) llvm/tools/llvm-exegesis/lib/AArch64/Target.cpp (+81)
diff --git a/llvm/test/tools/llvm-exegesis/AArch64/skip_unsupported_instructions.s b/llvm/test/tools/llvm-exegesis/AArch64/skip_unsupported_instructions.s
new file mode 100644
index 0000000000000..927ee190e803f
--- /dev/null
+++ b/llvm/test/tools/llvm-exegesis/AArch64/skip_unsupported_instructions.s
@@ -0,0 +1,10 @@
+llvm/test/tools/llvm-exegesis/AArch64/skip_unsupported_instructions.s
+
+# REQUIRES: aarch64-registered-target
+
+# Check for skipping of illegal instruction errors (AUT and LDGM)
+# RUN: llvm-exegesis -mcpu=neoverse-v2 -mode=latency --opcode-name=AUTIA --benchmark-phase=assemble-measured-code 2>&1 | FileCheck %s --check-prefix=CHECK-AUTIA
+# CHECK-AUTIA-NOT: snippet crashed while running: Illegal instruction
+
+# RUN: llvm-exegesis -mcpu=neoverse-v2 -mode=latency --opcode-name=LDGM --benchmark-phase=assemble-measured-code 2>&1 | FileCheck %s --check-prefix=CHECK-LDGM
+# CHECK-LDGM: LDGM: Unsupported opcode: load tag multiple 
\ No newline at end of file
diff --git a/llvm/tools/llvm-exegesis/lib/AArch64/Target.cpp b/llvm/tools/llvm-exegesis/lib/AArch64/Target.cpp
index bf2b053003ce3..6d8e19e5820c7 100644
--- a/llvm/tools/llvm-exegesis/lib/AArch64/Target.cpp
+++ b/llvm/tools/llvm-exegesis/lib/AArch64/Target.cpp
@@ -9,12 +9,64 @@
 #include "AArch64.h"
 #include "AArch64RegisterInfo.h"
 
+#if defined(__aarch64__) && defined(__linux__)
+#include <linux/prctl.h> // For PR_PAC_* constants
+#include <sys/prctl.h>
+#ifndef PR_PAC_SET_ENABLED_KEYS
+#define PR_PAC_SET_ENABLED_KEYS 60
+#endif
+#ifndef PR_PAC_GET_ENABLED_KEYS
+#define PR_PAC_GET_ENABLED_KEYS 61
+#endif
+#endif
+
 #define GET_AVAILABLE_OPCODE_CHECKER
 #include "AArch64GenInstrInfo.inc"
 
 namespace llvm {
 namespace exegesis {
 
+bool isPointerAuth(unsigned Opcode) {
+  switch (Opcode) {
+  default:
+    return false;
+
+  // FIXME: Pointer Authentication instructions.
+  // We would like to measure these instructions, but they can behave
+  // differently on different platforms, and maybe the snippets need to look
+  // different for these instructions,
+  // Platform-specific handling:  On Linux, we disable authentication, may
+  // interfere with measurements. On non-Linux platforms, disable opcodes for
+  // now.
+  case AArch64::AUTDA:
+  case AArch64::AUTDB:
+  case AArch64::AUTDZA:
+  case AArch64::AUTDZB:
+  case AArch64::AUTIA:
+  case AArch64::AUTIA1716:
+  case AArch64::AUTIASP:
+  case AArch64::AUTIAZ:
+  case AArch64::AUTIB:
+  case AArch64::AUTIB1716:
+  case AArch64::AUTIBSP:
+  case AArch64::AUTIBZ:
+  case AArch64::AUTIZA:
+  case AArch64::AUTIZB:
+    return true;
+  }
+}
+
+bool isLoadTagMultiple(unsigned Opcode) {
+  switch (Opcode) {
+  default:
+    return false;
+
+  // Load tag multiple instruction
+  case AArch64::LDGM:
+    return true;
+  }
+}
+
 static unsigned getLoadImmediateOpcode(unsigned RegBitWidth) {
   switch (RegBitWidth) {
   case 32:
@@ -134,6 +186,35 @@ class ExegesisAArch64Target : public ExegesisTarget {
     // Function return is a pseudo-instruction that needs to be expanded
     PM.add(createAArch64ExpandPseudoPass());
   }
+
+  const char *getIgnoredOpcodeReasonOrNull(const LLVMState &State,
+                                           unsigned Opcode) const override {
+    if (const char *Reason =
+            ExegesisTarget::getIgnoredOpcodeReasonOrNull(State, Opcode))
+      return Reason;
+
+    if (isPointerAuth(Opcode)) {
+#if defined(__aarch64__) && defined(__linux__)
+      // Disable all PAC keys. Note that while we expect the measurements to
+      // be the same with PAC keys disabled, they could potentially be lower
+      // since authentication checks are bypassed.
+      if (prctl(PR_PAC_SET_ENABLED_KEYS,
+                PR_PAC_APIAKEY | PR_PAC_APIBKEY | PR_PAC_APDAKEY |
+                    PR_PAC_APDBKEY, // all keys
+                0,                  // disable all
+                0, 0) < 0) {
+        return "Failed to disable PAC keys";
+      }
+#else
+      return "Unsupported opcode: isPointerAuth";
+#endif
+    }
+
+    if (isLoadTagMultiple(Opcode))
+      return "Unsupported opcode: load tag multiple";
+
+    return nullptr;
+  }
 };
 
 } // namespace

Copy link
Collaborator

@davemgreen davemgreen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM to me still. We will have to see if this still has some issues with the buildbots, if we do then we will perhaps need to make it simpler or adjust it via a cmake option. Good luck!

Copy link
Contributor

@boomanaiden154 boomanaiden154 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (carrying over from the last patch).

Were you able to reproduce the issue locally?

@lakshayk-nv
Copy link
Contributor Author

LGTM (carrying over from the last patch).

Were you able to reproduce the issue locally?

Yes, We replicated build failure locally using a x86_64 machine with Ubuntu 20.04.6 LTS. And reproduced build failure `error: ‘PR_PAC_SET_ENABLED_KEYS’ was not declared in this scope.

@boomanaiden154
Copy link
Contributor

Yes, We replicated build failure locally using a x86_64 machine with Ubuntu 20.04.6 LTS. And reproduced build failure `error: ‘PR_PAC_SET_ENABLED_KEYS’ was not declared in this scope.

Sounds good. Seems perfectly reasonable to reland then.

@sjoerdmeijer sjoerdmeijer merged commit 475531b into llvm:main Apr 28, 2025
12 checks passed
@tuliom
Copy link
Contributor

tuliom commented May 5, 2025

We started to see this new test crashing when running on real aarch64:

FAIL: LLVM :: tools/llvm-exegesis/AArch64/skip_unsupported_instructions.s (52758 of 56744)
******************** TEST 'LLVM :: tools/llvm-exegesis/AArch64/skip_unsupported_instructions.s' FAILED ********************
Exit Code: 139

Command Output (stderr):
--
/builddir/build/BUILD/llvm-21.0.0_pre20250505.ged2f89fa00c784-build/llvm-project-ed2f89fa00c784c6a4290955fb2c68dea033fe59/llvm/redhat-linux-build/bin/llvm-exegesis -mcpu=neoverse-v2 -mode=latency --opcode-name=AUTIA --benchmark-phase=assemble-measured-code 2>&1 | /builddir/build/BUILD/llvm-21.0.0_pre20250505.ged2f89fa00c784-build/llvm-project-ed2f89fa00c784c6a4290955fb2c68dea033fe59/llvm/redhat-linux-build/bin/FileCheck /builddir/build/BUILD/llvm-21.0.0_pre20250505.ged2f89fa00c784-build/llvm-project-ed2f89fa00c784c6a4290955fb2c68dea033fe59/llvm/test/tools/llvm-exegesis/AArch64/skip_unsupported_instructions.s --check-prefix=CHECK-AUTIA # RUN: at line 6
+ /builddir/build/BUILD/llvm-21.0.0_pre20250505.ged2f89fa00c784-build/llvm-project-ed2f89fa00c784c6a4290955fb2c68dea033fe59/llvm/redhat-linux-build/bin/llvm-exegesis -mcpu=neoverse-v2 -mode=latency --opcode-name=AUTIA --benchmark-phase=assemble-measured-code
+ /builddir/build/BUILD/llvm-21.0.0_pre20250505.ged2f89fa00c784-build/llvm-project-ed2f89fa00c784c6a4290955fb2c68dea033fe59/llvm/redhat-linux-build/bin/FileCheck /builddir/build/BUILD/llvm-21.0.0_pre20250505.ged2f89fa00c784-build/llvm-project-ed2f89fa00c784c6a4290955fb2c68dea033fe59/llvm/test/tools/llvm-exegesis/AArch64/skip_unsupported_instructions.s --check-prefix=CHECK-AUTIA
/builddir/build/BUILD/llvm-21.0.0_pre20250505.ged2f89fa00c784-build/llvm-project-ed2f89fa00c784c6a4290955fb2c68dea033fe59/llvm/redhat-linux-build/test/tools/llvm-exegesis/AArch64/Output/skip_unsupported_instructions.s.script: line 2: 412142 Segmentation fault      (core dumped) /builddir/build/BUILD/llvm-21.0.0_pre20250505.ged2f89fa00c784-build/llvm-project-ed2f89fa00c784c6a4290955fb2c68dea033fe59/llvm/redhat-linux-build/bin/llvm-exegesis -mcpu=neoverse-v2 -mode=latency --opcode-name=AUTIA --benchmark-phase=assemble-measured-code 2>&1
     412143 Done                    | /builddir/build/BUILD/llvm-21.0.0_pre20250505.ged2f89fa00c784-build/llvm-project-ed2f89fa00c784c6a4290955fb2c68dea033fe59/llvm/redhat-linux-build/bin/FileCheck /builddir/build/BUILD/llvm-21.0.0_pre20250505.ged2f89fa00c784-build/llvm-project-ed2f89fa00c784c6a4290955fb2c68dea033fe59/llvm/test/tools/llvm-exegesis/AArch64/skip_unsupported_instructions.s --check-prefix=CHECK-AUTIA

I've only seen this failing in CPUs where paca and pacg are available.
Any idea what could be causing this crash?

@tuliom
Copy link
Contributor

tuliom commented May 5, 2025

A full build log with the error is available at: https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250505/fedora-rawhide-aarch64/08995642-llvm/builder-live.log.gz

This is the cpu info:

PU info:
Architecture:                         aarch64
CPU op-mode(s):                       32-bit, 64-bit
Byte Order:                           Little Endian
CPU(s):                               4
On-line CPU(s) list:                  0-3
Vendor ID:                            ARM
Model name:                           Neoverse-V1
Model:                                1
Thread(s) per core:                   1
Core(s) per socket:                   4
Socket(s):                            1
Stepping:                             r1p1
BogoMIPS:                             2100.00
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
L1d cache:                            256 KiB (4 instances)
L1i cache:                            256 KiB (4 instances)
L2 cache:                             4 MiB (4 instances)
L3 cache:                             32 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-3
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; CSV2, BHB
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

@abhilash1910
Copy link
Contributor

Looking at the error log, it seems that it is disabling pac (on some systems) without checking if the keys are present/accessible through prctl() . A solution can be to check prior if pac is available in the system and then to disable it .

IanWood1 pushed a commit to IanWood1/llvm-project that referenced this pull request May 6, 2025
…ctions fixed (llvm#136868)

[llvm-exegesis][AArch64] Recommit: Disable pauth and ldgm as unsupported instructions.

Skipping AUT and LDGM opcode variants which currently throws "illegal
instruction".

Last pull request
[llvm#132346](llvm#132346) got reviewed
and merged but builder bot got failed. This was due to undefined
`PR_PAC_SET_ENABLED_KEYS` utilized were not defined in x86 arch,
resulting in build failure.

This is followup to merge the changes with following changes to fixup
the build failure.

Changes: 
- Fixed up the problem with arch specific check for `prctl` library
import
- Defining `PR_PAC_SET_ENABLED_KEYS` if undefined.
IanWood1 pushed a commit to IanWood1/llvm-project that referenced this pull request May 6, 2025
…ctions fixed (llvm#136868)

[llvm-exegesis][AArch64] Recommit: Disable pauth and ldgm as unsupported instructions.

Skipping AUT and LDGM opcode variants which currently throws "illegal
instruction".

Last pull request
[llvm#132346](llvm#132346) got reviewed
and merged but builder bot got failed. This was due to undefined
`PR_PAC_SET_ENABLED_KEYS` utilized were not defined in x86 arch,
resulting in build failure.

This is followup to merge the changes with following changes to fixup
the build failure.

Changes: 
- Fixed up the problem with arch specific check for `prctl` library
import
- Defining `PR_PAC_SET_ENABLED_KEYS` if undefined.
IanWood1 pushed a commit to IanWood1/llvm-project that referenced this pull request May 6, 2025
…ctions fixed (llvm#136868)

[llvm-exegesis][AArch64] Recommit: Disable pauth and ldgm as unsupported instructions.

Skipping AUT and LDGM opcode variants which currently throws "illegal
instruction".

Last pull request
[llvm#132346](llvm#132346) got reviewed
and merged but builder bot got failed. This was due to undefined
`PR_PAC_SET_ENABLED_KEYS` utilized were not defined in x86 arch,
resulting in build failure.

This is followup to merge the changes with following changes to fixup
the build failure.

Changes: 
- Fixed up the problem with arch specific check for `prctl` library
import
- Defining `PR_PAC_SET_ENABLED_KEYS` if undefined.
Ankur-0429 pushed a commit to Ankur-0429/llvm-project that referenced this pull request May 9, 2025
…ctions fixed (llvm#136868)

[llvm-exegesis][AArch64] Recommit: Disable pauth and ldgm as unsupported instructions.

Skipping AUT and LDGM opcode variants which currently throws "illegal
instruction".

Last pull request
[llvm#132346](llvm#132346) got reviewed
and merged but builder bot got failed. This was due to undefined
`PR_PAC_SET_ENABLED_KEYS` utilized were not defined in x86 arch,
resulting in build failure.

This is followup to merge the changes with following changes to fixup
the build failure.

Changes: 
- Fixed up the problem with arch specific check for `prctl` library
import
- Defining `PR_PAC_SET_ENABLED_KEYS` if undefined.
@pcc
Copy link
Contributor

pcc commented May 13, 2025

I'm guessing that the issue is that this change is disabling the IA key while a stack frame built with return PAC has a signed return address. The AUTIASP will act as a NOP and the RET instruction will branch to the signed address, causing the segfault observed in #136868 (comment) . Also, if llvm-exegesis is run on a system that supports PAuth ABI, this prctl will lead to a crash on the next indirect or virtual call.

PR_PAC_SET_ENABLED_KEYS isn't really meant to be used by normal user applications, so I'm not sure if there's a good way to make this work. Maybe you could prevent llvm-exegesis from generating AUT instructions for now?

@boomanaiden154
Copy link
Contributor

I've disabled the test for now while this discussion is ongoing. I thought some of the fixes would be moving a little bit faster.

d46458c

@lakshayk-nv
Copy link
Contributor Author

Maybe you could prevent llvm-exegesis from generating AUT instructions for now?

We can put up a patch for this, let me know :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants