projects/linux·commit 91dbbe6c

Merge tag 'riscv-for-linus-6.13-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux

91dbbe6c · Linus Torvalds <[email protected]> · 2024-11-27 11:19:09 -08:00

Pull RISC-v updates from Palmer Dabbelt:

 - Support for pointer masking in userspace

 - Support for probing vector misaligned access performance

 - Support for qspinlock on systems with Zacas and Zabha

* tag 'riscv-for-linus-6.13-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux: (38 commits)
  RISC-V: Remove unnecessary include from compat.h
  riscv: Fix default misaligned access trap
  riscv: Add qspinlock support
  dt-bindings: riscv: Add Ziccrse ISA extension description
  riscv: Add ISA extension parsing for Ziccrse
  asm-generic: ticket-lock: Add separate ticket-lock.h
  asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock
  riscv: Implement xchg8/16() using Zabha
  riscv: Implement arch_cmpxchg128() using Zacas
  riscv: Improve zacas fully-ordered cmpxchg()
  riscv: Implement cmpxchg8/16() using Zabha
  dt-bindings: riscv: Add Zabha ISA extension description
  riscv: Implement cmpxchg32/64() using Zacas
  riscv: Do not fail to build on byte/halfword operations with Zawrs
  riscv: Move cpufeature.h macros into their own header
  KVM: riscv: selftests: Add Smnpm and Ssnpm to get-reg-list test
  RISC-V: KVM: Allow Smnpm and Ssnpm extensions for guests
  riscv: hwprobe: Export the Supm ISA extension
  riscv: selftests: Add a pointer masking test
  riscv: Allow ptrace control of the tagged address ABI
  ...
Commit
91dbbe6c9ffe5eded9a3e75d773ff92da8d2bc57
Parents
c9469697 · 8d4f1e05

Merge commit. Changes below are compared with the first parent: c9469697.

Changes

53 files changed · +1789 −282

FileStatusAddDelContent
Documentation/arch/riscv/hwprobe.rstmodified+19−0View file
Documentation/arch/riscv/uabi.rstmodified+16−0View file
Documentation/devicetree/bindings/riscv/extensions.yamlmodified+30−0View file
Documentation/features/locking/queued-spinlocks/arch-support.txtmodified+1−1View file
arch/riscv/Kconfigmodified+136−2View file
arch/riscv/Makefilemodified+6−0View file
arch/riscv/configs/defconfigmodified+1−0View file
arch/riscv/include/asm/Kbuildmodified+3−1View file
arch/riscv/include/asm/cmpxchg.hmodified+208−78View file
arch/riscv/include/asm/compat.hmodified+0−1View file
arch/riscv/include/asm/cpufeature-macros.hadded+66−0View file
arch/riscv/include/asm/cpufeature.hmodified+14−59View file
arch/riscv/include/asm/csr.hmodified+16−0View file
arch/riscv/include/asm/entry-common.hmodified+1−0View file
arch/riscv/include/asm/hwcap.hmodified+7−0View file
arch/riscv/include/asm/hwprobe.hmodified+1−1View file
arch/riscv/include/asm/mmu.hmodified+7−0View file
arch/riscv/include/asm/mmu_context.hmodified+13−0View file
arch/riscv/include/asm/processor.hmodified+9−0View file
arch/riscv/include/asm/spinlock.hadded+47−0View file
arch/riscv/include/asm/switch_to.hmodified+19−0View file
arch/riscv/include/asm/uaccess.hmodified+38−5View file
arch/riscv/include/asm/vector.hmodified+2−0View file
arch/riscv/include/uapi/asm/hwprobe.hmodified+6−0View file
arch/riscv/include/uapi/asm/kvm.hmodified+2−0View file
arch/riscv/kernel/Makefilemodified+2−1View file
arch/riscv/kernel/copy-unaligned.hmodified+5−0View file
arch/riscv/kernel/cpufeature.cmodified+13−3View file
arch/riscv/kernel/fpu.Smodified+2−2View file
arch/riscv/kernel/process.cmodified+154−0View file
arch/riscv/kernel/ptrace.cmodified+42−0View file
arch/riscv/kernel/setup.cmodified+37−0View file
arch/riscv/kernel/smpboot.cmodified+0−2View file
arch/riscv/kernel/suspend.cmodified+2−2View file
arch/riscv/kernel/sys_hwprobe.cmodified+44−0View file
arch/riscv/kernel/traps_misaligned.cmodified+128−11View file
arch/riscv/kernel/unaligned_access_speed.cmodified+148−8View file
arch/riscv/kernel/vdso/Makefilemodified+5−4View file
arch/riscv/kernel/vec-copy-unaligned.Sadded+58−0View file
arch/riscv/kernel/vector.cmodified+1−1View file
arch/riscv/kvm/vcpu_onereg.cmodified+4−0View file
drivers/cpuidle/cpuidle-riscv-sbi.cmodified+9−2View file
include/asm-generic/qspinlock.hmodified+2−0View file
include/asm-generic/spinlock.hmodified+1−86View file
include/asm-generic/spinlock_types.hmodified+2−10View file
include/asm-generic/ticket_spinlock.hadded+105−0View file
include/uapi/linux/elf.hmodified+1−0View file
include/uapi/linux/prctl.hmodified+4−1View file
tools/testing/selftests/kvm/riscv/get-reg-list.cmodified+8−0View file
tools/testing/selftests/riscv/Makefilemodified+1−1View file
tools/testing/selftests/riscv/abi/.gitignoreadded+1−0View file
tools/testing/selftests/riscv/abi/Makefileadded+10−0View file
tools/testing/selftests/riscv/abi/pointer_masking.cadded+332−0View file

Only the first 600 diff lines are shown. Download the complete patch.

Documentation/arch/riscv/hwprobe.rst

View file
diff --git a/Documentation/arch/riscv/hwprobe.rst b/Documentation/arch/riscv/hwprobe.rst
index 85b7092..955fbcd 100644
--- a/Documentation/arch/riscv/hwprobe.rst
+++ b/Documentation/arch/riscv/hwprobe.rst
@@ -239,6 +239,9 @@ The following keys are defined:
ratified in commit 98918c844281 ("Merge pull request #1217 from
riscv/zawrs") of riscv-isa-manual.
+ * :c:macro:`RISCV_HWPROBE_EXT_SUPM`: The Supm extension is supported as
+ defined in version 1.0 of the RISC-V Pointer Masking extensions.
+
* :c:macro:`RISCV_HWPROBE_KEY_CPUPERF_0`: Deprecated. Returns similar values to
:c:macro:`RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF`, but the key was
mistakenly classified as a bitmask rather than a value.
@@ -274,3 +277,19 @@ The following keys are defined:
represent the highest userspace virtual address usable.
* :c:macro:`RISCV_HWPROBE_KEY_TIME_CSR_FREQ`: Frequency (in Hz) of `time CSR`.
+
+* :c:macro:`RISCV_HWPROBE_KEY_MISALIGNED_VECTOR_PERF`: An enum value describing the
+ performance of misaligned vector accesses on the selected set of processors.
+
+ * :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN`: The performance of misaligned
+ vector accesses is unknown.
+
+ * :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_SLOW`: 32-bit misaligned accesses using vector
+ registers are slower than the equivalent quantity of byte accesses via vector registers.
+ Misaligned accesses may be supported directly in hardware, or trapped and emulated by software.
+
+ * :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_FAST`: 32-bit misaligned accesses using vector
+ registers are faster than the equivalent quantity of byte accesses via vector registers.
+
+ * :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_UNSUPPORTED`: Misaligned vector accesses are
+ not supported at all and will generate a misaligned address fault.

Documentation/arch/riscv/uabi.rst

View file
diff --git a/Documentation/arch/riscv/uabi.rst b/Documentation/arch/riscv/uabi.rst
index 2b420ba..243e400 100644
--- a/Documentation/arch/riscv/uabi.rst
+++ b/Documentation/arch/riscv/uabi.rst
@@ -68,3 +68,19 @@ Misaligned accesses
Misaligned scalar accesses are supported in userspace, but they may perform
poorly. Misaligned vector accesses are only supported if the Zicclsm extension
is supported.
+
+Pointer masking
+---------------
+
+Support for pointer masking in userspace (the Supm extension) is provided via
+the ``PR_SET_TAGGED_ADDR_CTRL`` and ``PR_GET_TAGGED_ADDR_CTRL`` ``prctl()``
+operations. Pointer masking is disabled by default. To enable it, userspace
+must call ``PR_SET_TAGGED_ADDR_CTRL`` with the ``PR_PMLEN`` field set to the
+number of mask/tag bits needed by the application. ``PR_PMLEN`` is interpreted
+as a lower bound; if the kernel is unable to satisfy the request, the
+``PR_SET_TAGGED_ADDR_CTRL`` operation will fail. The actual number of tag bits
+is returned in ``PR_PMLEN`` by the ``PR_GET_TAGGED_ADDR_CTRL`` operation.
+
+Additionally, when pointer masking is enabled (``PR_PMLEN`` is greater than 0),
+a tagged address ABI is supported, with the same interface and behavior as
+documented for AArch64 (Documentation/arch/arm64/tagged-address-abi.rst).

Documentation/devicetree/bindings/riscv/extensions.yaml

View file
diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Documentation/devicetree/bindings/riscv/extensions.yaml
index 2cf2026..af7e523 100644
--- a/Documentation/devicetree/bindings/riscv/extensions.yaml
+++ b/Documentation/devicetree/bindings/riscv/extensions.yaml
@@ -128,6 +128,18 @@ properties:
changes to interrupts as frozen at commit ccbddab ("Merge pull
request #42 from riscv/jhauser-2023-RC4") of riscv-aia.
+ - const: smmpm
+ description: |
+ The standard Smmpm extension for M-mode pointer masking as
+ ratified at commit d70011dde6c2 ("Update to ratified state")
+ of riscv-j-extension.
+
+ - const: smnpm
+ description: |
+ The standard Smnpm extension for next-mode pointer masking as
+ ratified at commit d70011dde6c2 ("Update to ratified state")
+ of riscv-j-extension.
+
- const: smstateen
description: |
The standard Smstateen extension for controlling access to CSRs
@@ -147,6 +159,12 @@ properties:
and mode-based filtering as ratified at commit 01d1df0 ("Add ability
to manually trigger workflow. (#2)") of riscv-count-overflow.
+ - const: ssnpm
+ description: |
+ The standard Ssnpm extension for next-mode pointer masking as
+ ratified at commit d70011dde6c2 ("Update to ratified state")
+ of riscv-j-extension.
+
- const: sstc
description: |
The standard Sstc supervisor-level extension for time compare as
@@ -178,6 +196,12 @@ properties:
as ratified at commit 4a69197e5617 ("Update to ratified state") of
riscv-svvptc.
+ - const: zabha
+ description: |
+ The Zabha extension for Byte and Halfword Atomic Memory Operations
+ as ratified at commit 49f49c842ff9 ("Update to Rafified state") of
+ riscv-zabha.
+
- const: zacas
description: |
The Zacas extension for Atomic Compare-and-Swap (CAS) instructions
@@ -290,6 +314,12 @@ properties:
in commit 64074bc ("Update version numbers for Zfh/Zfinx") of
riscv-isa-manual.
+ - const: ziccrse
+ description:
+ The standard Ziccrse extension which provides forward progress
+ guarantee on LR/SC sequences, as ratified in commit b1d806605f87
+ ("Updated to ratified state.") of the riscv profiles specification.
+
- const: zk
description:
The standard Zk Standard Scalar cryptography extension as ratified

Documentation/features/locking/queued-spinlocks/arch-support.txt

View file
diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.txt b/Documentation/features/locking/queued-spinlocks/arch-support.txt
index 22f2990..cf26042 100644
--- a/Documentation/features/locking/queued-spinlocks/arch-support.txt
+++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt
@@ -20,7 +20,7 @@
| openrisc: | ok |
| parisc: | TODO |
| powerpc: | ok |
- | riscv: | TODO |
+ | riscv: | ok |
| s390: | TODO |
| sh: | TODO |
| sparc: | ok |

arch/riscv/Kconfig

View file
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index ff1e353..cc63aef 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -83,6 +83,7 @@ config RISCV
select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP
select ARCH_WANTS_NO_INSTR
select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE
+ select ARCH_WEAK_RELEASE_ACQUIRE if ARCH_USE_QUEUED_SPINLOCKS
select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU
select BUILDTIME_TABLE_SORT if MMU
select CLINT_TIMER if RISCV_M_MODE
@@ -116,6 +117,7 @@ config RISCV
select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO
select HARDIRQS_SW_RESEND
select HAS_IOPORT if MMU
+ select HAVE_ALIGNED_STRUCT_PAGE
select HAVE_ARCH_AUDITSYSCALL
select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP
select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT
@@ -507,6 +509,39 @@ config NODES_SHIFT
Specify the maximum number of NUMA Nodes available on the target
system. Increases memory reserved to accommodate various tables.
+choice
+ prompt "RISC-V spinlock type"
+ default RISCV_COMBO_SPINLOCKS
+
+config RISCV_TICKET_SPINLOCKS
+ bool "Using ticket spinlock"
+
+config RISCV_QUEUED_SPINLOCKS
+ bool "Using queued spinlock"
+ depends on SMP && MMU && NONPORTABLE
+ select ARCH_USE_QUEUED_SPINLOCKS
+ help
+ The queued spinlock implementation requires the forward progress
+ guarantee of cmpxchg()/xchg() atomic operations: CAS with Zabha or
+ LR/SC with Ziccrse provide such guarantee.
+
+ Select this if and only if Zabha or Ziccrse is available on your
+ platform, RISCV_QUEUED_SPINLOCKS must not be selected for platforms
+ without one of those extensions.
+
+ If unsure, select RISCV_COMBO_SPINLOCKS, which will use qspinlocks
+ when supported and otherwise ticket spinlocks.
+
+config RISCV_COMBO_SPINLOCKS
+ bool "Using combo spinlock"
+ depends on SMP && MMU
+ select ARCH_USE_QUEUED_SPINLOCKS
+ help
+ Embed both queued spinlock and ticket lock so that the spinlock
+ implementation can be chosen at runtime.
+
+endchoice
+
config RISCV_ALTERNATIVE
bool
depends on !XIP_KERNEL
@@ -532,6 +567,17 @@ config RISCV_ISA_C
If you don't know what to do here, say Y.
+config RISCV_ISA_SUPM
+ bool "Supm extension for userspace pointer masking"
+ depends on 64BIT
+ default y
+ help
+ Add support for pointer masking in userspace (Supm) when the
+ underlying hardware extension (Smnpm or Ssnpm) is detected at boot.
+
+ If this option is disabled, userspace will be unable to use
+ the prctl(PR_{SET,GET}_TAGGED_ADDR_CTRL) API.
+
config RISCV_ISA_SVNAPOT
bool "Svnapot extension support for supervisor mode NAPOT pages"
depends on 64BIT && MMU
@@ -633,6 +679,40 @@ config RISCV_ISA_ZAWRS
use of these instructions in the kernel when the Zawrs extension is
detected at boot.
+config TOOLCHAIN_HAS_ZABHA
+ bool
+ default y
+ depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zabha)
+ depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zabha)
+ depends on AS_HAS_OPTION_ARCH
+
+config RISCV_ISA_ZABHA
+ bool "Zabha extension support for atomic byte/halfword operations"
+ depends on TOOLCHAIN_HAS_ZABHA
+ depends on RISCV_ALTERNATIVE
+ default y
+ help
+ Enable the use of the Zabha ISA-extension to implement kernel
+ byte/halfword atomic memory operations when it is detected at boot.
+
+ If you don't know what to do here, say Y.
+
+config TOOLCHAIN_HAS_ZACAS
+ bool
+ default y
+ depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zacas)
+ depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zacas)
+ depends on AS_HAS_OPTION_ARCH
+
+config RISCV_ISA_ZACAS
+ bool "Zacas extension support for atomic CAS"
+ depends on TOOLCHAIN_HAS_ZACAS
+ depends on RISCV_ALTERNATIVE
+ default y
+ help
+ Enable the use of the Zacas ISA-extension to implement kernel atomic
+ cmpxchg operations when it is detected at boot.
+
If you don't know what to do here, say Y.
config TOOLCHAIN_HAS_ZBB
@@ -786,10 +866,24 @@ config THREAD_SIZE_ORDER
config RISCV_MISALIGNED
bool
+ help
+ Embed support for detecting and emulating misaligned
+ scalar or vector loads and stores.
+
+config RISCV_SCALAR_MISALIGNED
+ bool
+ select RISCV_MISALIGNED
select SYSCTL_ARCH_UNALIGN_ALLOW
help
Embed support for emulating misaligned loads and stores.
+config RISCV_VECTOR_MISALIGNED
+ bool
+ select RISCV_MISALIGNED
+ depends on RISCV_ISA_V
+ help
+ Enable detecting support for vector misaligned loads and stores.
+
choice
prompt "Unaligned Accesses Support"
default RISCV_PROBE_UNALIGNED_ACCESS
@@ -801,7 +895,7 @@ choice
config RISCV_PROBE_UNALIGNED_ACCESS
bool "Probe for hardware unaligned access support"
- select RISCV_MISALIGNED
+ select RISCV_SCALAR_MISALIGNED
help
During boot, the kernel will run a series of tests to determine the
speed of unaligned accesses. This probing will dynamically determine
@@ -812,7 +906,7 @@ config RISCV_PROBE_UNALIGNED_ACCESS
config RISCV_EMULATED_UNALIGNED_ACCESS
bool "Emulate unaligned access where system support is missing"
- select RISCV_MISALIGNED
+ select RISCV_SCALAR_MISALIGNED
help
If unaligned memory accesses trap into the kernel as they are not
supported by the system, the kernel will emulate the unaligned
@@ -841,6 +935,46 @@ config RISCV_EFFICIENT_UNALIGNED_ACCESS
endchoice
+choice
+ prompt "Vector unaligned Accesses Support"
+ depends on RISCV_ISA_V
+ default RISCV_PROBE_VECTOR_UNALIGNED_ACCESS
+ help
+ This determines the level of support for vector unaligned accesses. This
+ information is used by the kernel to perform optimizations. It is also
+ exposed to user space via the hwprobe syscall. The hardware will be
+ probed at boot by default.
+
+config RISCV_PROBE_VECTOR_UNALIGNED_ACCESS
+ bool "Probe speed of vector unaligned accesses"
+ select RISCV_VECTOR_MISALIGNED
+ depends on RISCV_ISA_V
+ help
+ During boot, the kernel will run a series of tests to determine the
+ speed of vector unaligned accesses if they are supported. This probing
+ will dynamically determine the speed of vector unaligned accesses on
+ the underlying system if they are supported.
+
+config RISCV_SLOW_VECTOR_UNALIGNED_ACCESS
+ bool "Assume the system supports slow vector unaligned memory accesses"
+ depends on NONPORTABLE
+ help
+ Assume that the system supports slow vector unaligned memory accesses. The
+ kernel and userspace programs may not be able to run at all on systems
+ that do not support unaligned memory accesses.
+
+config RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS
+ bool "Assume the system supports fast vector unaligned memory accesses"
+ depends on NONPORTABLE
+ help
+ Assume that the system supports fast vector unaligned memory accesses. When
+ enabled, this option improves the performance of the kernel on such
+ systems. However, the kernel and userspace programs will run much more
+ slowly, or will not be able to run at all, on systems that do not
+ support efficient unaligned memory accesses.
+
+endchoice
+
source "arch/riscv/Kconfig.vendor"
endmenu # "Platform type"

arch/riscv/Makefile

View file
diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile
index d469db9..9fe1ee7 100644
--- a/arch/riscv/Makefile
+++ b/arch/riscv/Makefile
@@ -82,6 +82,12 @@ else
riscv-march-$(CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI) := $(riscv-march-y)_zicsr_zifencei
endif
+# Check if the toolchain supports Zacas
+riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas
+
+# Check if the toolchain supports Zabha
+riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) := $(riscv-march-y)_zabha
+
# Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by
# matching non-v and non-multi-letter extensions out with the filter ([^v_]*)
KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/')

arch/riscv/configs/defconfig

View file
diff --git a/arch/riscv/configs/defconfig b/arch/riscv/configs/defconfig
index 1d5e13b..b4a3734 100644
--- a/arch/riscv/configs/defconfig
+++ b/arch/riscv/configs/defconfig
@@ -167,6 +167,7 @@ CONFIG_PINCTRL_SOPHGO_CV1800B=y
CONFIG_PINCTRL_SOPHGO_CV1812H=y
CONFIG_PINCTRL_SOPHGO_SG2000=y
CONFIG_PINCTRL_SOPHGO_SG2002=y
+CONFIG_GPIO_DWAPB=y
CONFIG_GPIO_SIFIVE=y
CONFIG_POWER_RESET_GPIO_RESTART=y
CONFIG_SENSORS_SFCTEMP=m

arch/riscv/include/asm/Kbuild

View file
diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild
index 1461af1..de13d5a 100644
--- a/arch/riscv/include/asm/Kbuild
+++ b/arch/riscv/include/asm/Kbuild
@@ -6,10 +6,12 @@ generic-y += early_ioremap.h
generic-y += flat.h
generic-y += kvm_para.h
generic-y += mmzone.h
+generic-y += mcs_spinlock.h
generic-y += parport.h
-generic-y += spinlock.h
generic-y += spinlock_types.h
+generic-y += ticket_spinlock.h
generic-y += qrwlock.h
generic-y += qrwlock_types.h
+generic-y += qspinlock.h
generic-y += user.h
generic-y += vmlinux.lds.h

arch/riscv/include/asm/cmpxchg.h

View file
diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h
index ebbce13..4cadc56 100644
--- a/arch/riscv/include/asm/cmpxchg.h
+++ b/arch/riscv/include/asm/cmpxchg.h
@@ -12,30 +12,43 @@
#include <asm/fence.h>
#include <asm/hwcap.h>
#include <asm/insn-def.h>
-
-#define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \
-({ \
- u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \
- ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \
- ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \
- << __s; \
- ulong __newx = (ulong)(n) << __s; \
- ulong __retx; \
- ulong __rc; \
- \
- __asm__ __volatile__ ( \
- prepend \
- "0: lr.w %0, %2\n" \
- " and %1, %0, %z4\n" \
- " or %1, %1, %z3\n" \
- " sc.w" sc_sfx " %1, %1, %2\n" \
- " bnez %1, 0b\n" \
- append \
- : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \
- : "rJ" (__newx), "rJ" (~__mask) \
- : "memory"); \
- \
- r = (__typeof__(*(p)))((__retx & __mask) >> __s); \
+#include <asm/cpufeature-macros.h>
+
+#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \
+ swap_append, r, p, n) \
+({ \
+ if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \
+ riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA)) { \
+ __asm__ __volatile__ ( \
+ prepend \
+ " amoswap" swap_sfx " %0, %z2, %1\n" \
+ swap_append \
+ : "=&r" (r), "+A" (*(p)) \
+ : "rJ" (n) \
+ : "memory"); \
+ } else { \
+ u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \
+ ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \
+ ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \
+ << __s; \
+ ulong __newx = (ulong)(n) << __s; \
+ ulong __retx; \
+ ulong __rc; \
+ \
+ __asm__ __volatile__ ( \
+ prepend \
+ "0: lr.w %0, %2\n" \
+ " and %1, %0, %z4\n" \
+ " or %1, %1, %z3\n" \
+ " sc.w" sc_sfx " %1, %1, %2\n" \
+ " bnez %1, 0b\n" \
+ sc_append \
+ : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \
+ : "rJ" (__newx), "rJ" (~__mask) \
+ : "memory"); \
+ \
+ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \
+ } \
})
#define __arch_xchg(sfx, prepend, append, r, p, n) \
@@ -58,8 +71,13 @@
\
switch (sizeof(*__ptr)) { \
case 1: \
+ __arch_xchg_masked(sc_sfx, ".b" swap_sfx, \
+ prepend, sc_append, swap_append, \
+ __ret, __ptr, __new); \
+ break; \
case 2: \
- __arch_xchg_masked(sc_sfx, prepend, sc_append, \
+ __arch_xchg_masked(sc_sfx, ".h" swap_sfx, \
+ prepend, sc_append, swap_append, \
__ret, __ptr, __new); \
break; \
case 4: \
@@ -106,55 +124,90 @@
* store NEW in MEM. Return the initial value in MEM. Success is
* indicated by comparing RETURN with OLD.
*/
-
-#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \
-({ \
- u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \
- ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \
- ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \
- << __s; \
- ulong __newx = (ulong)(n) << __s; \
- ulong __oldx = (ulong)(o) << __s; \
- ulong __retx; \
- ulong __rc; \
- \
- __asm__ __volatile__ ( \
- prepend \
- "0: lr.w %0, %2\n" \
- " and %1, %0, %z5\n" \
- " bne %1, %z3, 1f\n" \
- " and %1, %0, %z6\n" \
- " or %1, %1, %z4\n" \
- " sc.w" sc_sfx " %1, %1, %2\n" \
- " bnez %1, 0b\n" \
- append \
- "1:\n" \
- : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \
- : "rJ" ((long)__oldx), "rJ" (__newx), \
- "rJ" (__mask), "rJ" (~__mask) \
- : "memory"); \
- \
- r = (__typeof__(*(p)))((__retx & __mask) >> __s); \
+#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, \
+ sc_prepend, sc_append, \
+ cas_prepend, cas_append, \
+ r, p, o, n) \
+({ \
+ if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \
+ IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \
+ riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA) && \
+ riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \
+ r = o; \
+ \
+ __asm__ __volatile__ ( \
+ cas_prepend \
+ " amocas" cas_sfx " %0, %z2, %1\n" \
+ cas_append \
+ : "+&r" (r), "+A" (*(p)) \
+ : "rJ" (n) \
+ : "memory"); \
+ } else { \
+ u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \
+ ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \
+ ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \
+ << __s; \
+ ulong __newx = (ulong)(n) << __s; \
+ ulong __oldx = (ulong)(o) << __s; \
+ ulong __retx; \
+ ulong __rc; \
+ \
+ __asm__ __volatile__ ( \
+ sc_prepend \
+ "0: lr.w %0, %2\n" \
+ " and %1, %0, %z5\n" \
+ " bne %1, %z3, 1f\n" \
+ " and %1, %0, %z6\n" \
+ " or %1, %1, %z4\n" \
+ " sc.w" sc_sfx " %1, %1, %2\n" \
+ " bnez %1, 0b\n" \
+ sc_append \
+ "1:\n" \
+ : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \
+ : "rJ" ((long)__oldx), "rJ" (__newx), \
+ "rJ" (__mask), "rJ" (~__mask) \
+ : "memory"); \
+ \
+ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \
+ } \
})
-#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \
+#define __arch_cmpxchg(lr_sfx, sc_sfx, cas_sfx, \
+ sc_prepend, sc_append, \
+ cas_prepend, cas_append, \
+ r, p, co, o, n) \
({ \
- register unsigned int __rc; \
+ if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \
+ riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \
+ r = o; \
\
- __asm__ __volatile__ ( \
- prepend \
- "0: lr" lr_sfx " %0, %2\n" \
- " bne %0, %z3, 1f\n" \
- " sc" sc_sfx " %1, %z4, %2\n" \
- " bnez %1, 0b\n" \
- append \
- "1:\n" \
- : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \
- : "rJ" (co o), "rJ" (n) \
- : "memory"); \
+ __asm__ __volatile__ ( \
+ cas_prepend \
+ " amocas" cas_sfx " %0, %z2, %1\n" \
+ cas_append \
+ : "+&r" (r), "+A" (*(p)) \
+ : "rJ" (n) \
+ : "memory"); \
+ } else { \
+ register unsigned int __rc; \
+ \
+ __asm__ __volatile__ ( \
+ sc_prepend \
+ "0: lr" lr_sfx " %0, %2\n" \
+ " bne %0, %z3, 1f\n" \
+ " sc" sc_sfx " %1, %z4, %2\n" \
+ " bnez %1, 0b\n" \
+ sc_append \
+ "1:\n" \
+ : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \
+ : "rJ" (co o), "rJ" (n) \
+ : "memory"); \

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/compat.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/cpufeature-macros.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/cpufeature.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/csr.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/entry-common.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/hwcap.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/hwprobe.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/mmu.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/mmu_context.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/processor.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/spinlock.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/switch_to.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/uaccess.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/asm/vector.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/uapi/asm/hwprobe.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/include/uapi/asm/kvm.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/Makefile

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/copy-unaligned.h

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/cpufeature.c

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/fpu.S

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/process.c

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/ptrace.c

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/setup.c

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/smpboot.c

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/suspend.c

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/sys_hwprobe.c

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/traps_misaligned.c

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/unaligned_access_speed.c

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/vdso/Makefile

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/vec-copy-unaligned.S

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kernel/vector.c

View file

This diff is incomplete. Download the complete patch.

arch/riscv/kvm/vcpu_onereg.c

View file

This diff is incomplete. Download the complete patch.

drivers/cpuidle/cpuidle-riscv-sbi.c

View file

This diff is incomplete. Download the complete patch.

include/asm-generic/qspinlock.h

View file

This diff is incomplete. Download the complete patch.

include/asm-generic/spinlock.h

View file

This diff is incomplete. Download the complete patch.

include/asm-generic/spinlock_types.h

View file

This diff is incomplete. Download the complete patch.

include/asm-generic/ticket_spinlock.h

View file

This diff is incomplete. Download the complete patch.

include/uapi/linux/elf.h

View file

This diff is incomplete. Download the complete patch.

include/uapi/linux/prctl.h

View file

This diff is incomplete. Download the complete patch.

tools/testing/selftests/kvm/riscv/get-reg-list.c

View file

This diff is incomplete. Download the complete patch.

tools/testing/selftests/riscv/Makefile

View file

This diff is incomplete. Download the complete patch.

tools/testing/selftests/riscv/abi/.gitignore

View file

This diff is incomplete. Download the complete patch.

tools/testing/selftests/riscv/abi/Makefile

View file

This diff is incomplete. Download the complete patch.

tools/testing/selftests/riscv/abi/pointer_masking.c

View file

This diff is incomplete. Download the complete patch.