[v2,00/13] arm64: extable: remove anonymous out-of-line fixups

Message

Mark Rutland Oct. 19, 2021, 4:02 p.m. UTC

We recently realised that out-of-line extable fixups cause a number of problems
for backtracing (mattering both for developers and for RELIABLE_STACKTRACE and
LIVEPATCH). Dmitry spotted a confusing backtrace, which we identified was due
to problems with unwinding fixups, as summarized in:

  https://lore.kernel.org/linux-arm-kernel/[e-mail]/

The gist is that while backtracing through a fixup, the fixup gets symbolized
as an offset from the nearest prior symbol (which happens to be
`__entry_tramp_text_end`), and we the backtrace misses the function that was
being fixed up (because the fixup handling adjusts the PC, then the fixup does
a direct branch back to the original function). We can't reliably map from an
arbitrary PC in the fixup text back to the original function.

The way we create fixups is a bit unfortunate: most fixups are generated from
common templates, and only differ in register to be poked and the address to
branch back to, leading to redundant copies of the same logic that must pollute
Since the fixups are all written in assembly, and duplicated for each fixup
site, we can only perform very simple fixups, and can't handle any complex
triage that we might need for some exceptions (e.g. MTE faults).

This series address these concerns by getting rid of the out-of-line anonymous
fixup logic:

* For plain assembly functions, we move the fixup into the body of
  the function, after the usual return, as we already do for our cache
  routines. This simplifies the source code, and only adds a handful of
  instructions to the main body of `.text`.

  This is handled by the first three patches, which I think are trivial and
  could be queued regardless of the rest of the series.

* For inline assembly, we add specialised handlers which run in exception
  context to update registers, then adjust the PC *within* the faulting
  function. This requires some new glue to capture the handler and metadata in
  struct exception_table_entry (costing 32 bits per fixup), but for any
  non-trivial fixup (which is all of the inline asm cases), this removes at
  least two instructions of out-of-line fixup.

As the fixups are now handled from C code in exception context, we can more
easily extend these in future with more complex triage if necessary.

Overall, this doesn't have an appreciable impact on Image size (in local
testing the size of the Image was identical before/after), but does shift the
boundary between .text and .rodata, making .text smaller and .rodata bigger.
.text somewhat while growing .rodata somewhat.

I've tested this with both GCC and clang (including with clang CFI), and
everything is working as expected.

Other than changes to backtracing, there should be no functional change as a
result of this series.

Since v1 [1]:
* Add Acked-by and Reviewed-by tags
* Add fixup comments to lib/*.S functions
* Drop RO_EXCEPTION_TABLE_ALIGN to 4
* Elaborate on exception_table_entry alignment
* Note concurrent x86 extable rework

Thanks
Mark.

[1] https://lore.kernel.org/r/[e-mail]

Mark Rutland (13):
  arm64: lib: __arch_clear_user(): fold fixups into body
  arm64: lib: __arch_copy_from_user(): fold fixups into body
  arm64: lib: __arch_copy_to_user(): fold fixups into body
  arm64: kvm: use kvm_exception_table_entry
  arm64: factor out GPR numbering helpers
  arm64: gpr-num: support W registers
  arm64: extable: consolidate definitions
  arm64: extable: make fixup_exception() return bool
  arm64: extable: use `ex` for `exception_table_entry`
  arm64: extable: add `type` and `data` fields
  arm64: extable: add a dedicated uaccess handler
  arm64: extable: add load_unaligned_zeropad() handler
  arm64: vmlinux.lds.S: remove `.fixup` section

 arch/arm64/include/asm/asm-extable.h    | 95 +++++++++++++++++++++++++++++++++
 arch/arm64/include/asm/asm-uaccess.h    |  7 ++-
 arch/arm64/include/asm/assembler.h      | 29 +---------
 arch/arm64/include/asm/extable.h        | 23 +++++---
 arch/arm64/include/asm/futex.h          | 25 +++------
 arch/arm64/include/asm/gpr-num.h        | 26 +++++++++
 arch/arm64/include/asm/kvm_asm.h        |  7 +--
 arch/arm64/include/asm/sysreg.h         | 25 +++------
 arch/arm64/include/asm/uaccess.h        | 26 ++-------
 arch/arm64/include/asm/word-at-a-time.h | 21 ++------
 arch/arm64/kernel/armv8_deprecated.c    | 12 ++---
 arch/arm64/kernel/traps.c               |  9 +---
 arch/arm64/kernel/vmlinux.lds.S         |  3 +-
 arch/arm64/kvm/hyp/include/hyp/switch.h | 10 ++--
 arch/arm64/lib/clear_user.S             | 10 ++--
 arch/arm64/lib/copy_from_user.S         |  8 ++-
 arch/arm64/lib/copy_to_user.S           |  8 ++-
 arch/arm64/mm/extable.c                 | 85 +++++++++++++++++++++++++----
 arch/arm64/net/bpf_jit_comp.c           |  9 ++--
 scripts/sorttable.c                     | 30 +++++++++++
 20 files changed, 310 insertions(+), 158 deletions(-)
 create mode 100644 arch/arm64/include/asm/asm-extable.h
 create mode 100644 arch/arm64/include/asm/gpr-num.h

Comments

Will Deacon Oct. 21, 2021, 10:05 a.m. UTC | #1

On Tue, 19 Oct 2021 17:02:06 +0100, Mark Rutland wrote:
> We recently realised that out-of-line extable fixups cause a number of problems
> for backtracing (mattering both for developers and for RELIABLE_STACKTRACE and
> LIVEPATCH). Dmitry spotted a confusing backtrace, which we identified was due
> to problems with unwinding fixups, as summarized in:
> 
>   https://lore.kernel.org/linux-arm-kernel/[e-mail]/
> 
> [...]

Applied to arm64 (for-next/extable), thanks!

[01/13] arm64: lib: __arch_clear_user(): fold fixups into body
        https://git.kernel.org/arm64/c/35d67794b882
[02/13] arm64: lib: __arch_copy_from_user(): fold fixups into body
        https://git.kernel.org/arm64/c/4012e0e22739
[03/13] arm64: lib: __arch_copy_to_user(): fold fixups into body
        https://git.kernel.org/arm64/c/139f9ab73d60
[04/13] arm64: kvm: use kvm_exception_table_entry
        https://git.kernel.org/arm64/c/ae2b2f3384c6
[05/13] arm64: factor out GPR numbering helpers
        https://git.kernel.org/arm64/c/8ed1b498ada6
[06/13] arm64: gpr-num: support W registers
        https://git.kernel.org/arm64/c/286fba6c2a45
[07/13] arm64: extable: consolidate definitions
        https://git.kernel.org/arm64/c/819771cc2892
[08/13] arm64: extable: make fixup_exception() return bool
        https://git.kernel.org/arm64/c/e8c328d7de03
[09/13] arm64: extable: use `ex` for `exception_table_entry`
        https://git.kernel.org/arm64/c/5d0e79051425
[10/13] arm64: extable: add `type` and `data` fields
        https://git.kernel.org/arm64/c/d6e2cc564775
[11/13] arm64: extable: add a dedicated uaccess handler
        https://git.kernel.org/arm64/c/2e77a62cb3a6
[12/13] arm64: extable: add load_unaligned_zeropad() handler
        https://git.kernel.org/arm64/c/753b32368705
[13/13] arm64: vmlinux.lds.S: remove `.fixup` section
        https://git.kernel.org/arm64/c/bf6e667f4738

Cheers,