[0/4] arm64: ensure CPUs are quiescent before patching

Message

Mark Rutland Dec. 3, 2021, 10:47 a.m. UTC

On arm64, certain instructions cannot be patched while they are being
concurrently executed, and in these cases we use stop_machine() to
ensure that while one CPU is patching instructions all other CPUs are in
a quiescent state. We have two distinct sequences for this, one used for
boot-time patching of alternatives, and on used for runtime patching
(e.g. kprobes).

Both sequences wait for patching to be complete before CPUs exit the
quiescent state, but we don't wait for CPUs to be quiescent *before* we
start patching, and so we may patch code which is still being executed
(e.g. portions of stop_machine() itself).

These patches fix this problem by updating the sequences to wait for
CPUs to become quiescent before starting patches. The first two patches
are potentially backportable fixes for the individual sequences, and the
this patch unifies them behind an arm64-specific patch_machine() helper.
The last patch prevents taking asynchronous exceptions out of a
quiescent state (just DAIF for now; I'm not sure exactly how to handle
SDEI).

The architecture documentation is a little vague on how to ensure
completion of prior execution (i.e. when patching from another CPU
cannot possibly affect this and cause UNPREDICTABLE behaviour). For the
moment I'm assuming that an atomic store cannot become visible until all
prior execution has completed, but I suspect that we *might* need to add
barriers into patch_machine() prior to signalling quiescence.

This series does not intend to address the more general problem that out
patching sequences may use directly-patchable or instrumentable code,
and I'm intending that we address those with subsequent patches. Fixing
that will require a more substantial rework (e.g. of the insn code).

Thanks,
Mark.

Mark Rutland (4):
  arm64: alternative: wait for other CPUs before patching
  arm64: insn: wait for other CPUs before patching
  arm64: patching: unify stop_machine() patch synchronization
  arm64: patching: mask exceptions in patch_machine()

 arch/arm64/include/asm/patching.h |  4 ++
 arch/arm64/kernel/alternative.c   | 33 +++--------
 arch/arm64/kernel/patching.c      | 94 +++++++++++++++++++++++++------
 3 files changed, 89 insertions(+), 42 deletions(-)