mbox series

[v6,00/12] livepatch: new features and fixes

Message ID 20191126100801.124844-1-wipawel@amazon.de (mailing list archive)
Headers show
Series livepatch: new features and fixes | expand

Message

Wieczorkiewicz, Pawel Nov. 26, 2019, 10:07 a.m. UTC
This series introduces new features to the livepatch functionality as
briefly discussed during Xen Developer Summit 2019: [a] and [b].
It also provides a few fixes and some small improvements.

Main changes in v6:
- Added missing action pad field zeroing

Main changes in v4:
- Fix various typos and minor issues
- Simplify arch_livepatch_{apply,revert} by using
  common_livepatch_{apply,revert}
- Improve python bindings and fix few issues

Main changes in v3:
- Fix expectation test to work on Arm
- Add test for metadata (Konrad)
- Minor fixes to documentation

Main changes in v2:
- added new features to livepatch documentation
- added livepatch tests
- enabled Arm support for [5]
- make .modinfo optional for [11]
- fixed typos

FEATURES:

1. independent modules (patches: [1], [2])

  * livepatch-build-tools repo dependency [A]

  Livepatch enforces the following buildid-based dependency chain
  between hotpatch modules:
    1) first module depends on given hypervisor buildid
    2) every consecutive module depends on previous module's buildid
  This way proper hotpatch stack order is maintained and enforced.
  While it is important for production hotpatches it limits agility and
  blocks usage of testing or debug hotpatches. These kinds of hotpatch
  modules are typically expected to be loaded at any time irrespective
  of current state of the modules stack.

  [A] livepatch-build: Embed hypervisor build id into every hotpatch

2. pre- and post- apply|revert actions hooks (patches: [3], [4])

  * livepatch-build-tools repo dependency [B]

  This is an implementation of 4 new livepatch module vetoing hooks,
  that can be optionally supplied along with modules.
  Hooks that currently exists in the livepatch mechanism aren't agile
  enough and have various limitations:
  * run only from within a quiescing zone
  * cannot conditionally prevent applying or reverting
  * do not have access to the module context
  To address these limitations the following has been implemented:
  1) pre-apply hook
  2) post-apply hook
  3) pre-revert hook
  4) post-revert hook

  [B] create-diff-object: Handle extra pre-|post- hooks

3. apply|revert actions replacement hooks (patches: [5], [6], [7])

  * livepatch-build-tools repo dependency: [C], [D], [E]

  To increase hotpatching system's agility and provide more flexiable
  long-term hotpatch solution, allow to overwrite the default apply
  and revert action functions with hook-like supplied alternatives.
  The alternative functions are optional and the default functions are
  used by default.

  [C] create-diff-object: Do not create empty .livepatch.funcs section
  [D] create-diff-object: Handle optional apply|revert hooks
  [E] create-diff-object: Add support for applied/reverted marker

4. inline asm hotpatching expectations (patches: [8])

  * livepatch-build-tools repo dependency: [F]

  Expectations are designed as optional feature, since the main use of
  them is planned for inline asm hotpatching.
  The payload structure is modified as each expectation structure is
  part of the livepatch_func structure and hence extends the payload.
  The payload version is bumped to 3 with this change to highlight the
  ABI modification and enforce proper support.
  The expectation is manually enabled during inline asm module
  construction. If enabled, expectation ensures that the expected
  content of memory is to be found at a given patching (old_addr)
  location.

  [F] create-diff-object: Add support for expectations

5. runtime hotpatch metadata support (patches: [9], [10], [11])

  Having detailed hotpatch metadata helps to properly identify module's
  origin and version. It also allows to keep track of the history of
  hotpatch loads in the system (at least within dmesg buffer size
  limits).
  Extend the livepatch list operation to fetch also payloads' metadata.
  This is achieved by extending the sysctl list interface with 2 extra
  guest handles:
  * metadata     - an array of arbitrary size strings
  * metadata_len - an array of metadata strings' lengths (uin32_t each)
  To unify and simplify the interface, handle the modules' name strings
  of arbitrary size by copying them in adhering chunks to the userland.

6. python bindings for livepatch operations (patches: [12])

  Extend the XC python bindings library to support all common livepatch
  operations and actions:
  - status (pyxc_livepatch_status):
  - action (pyxc_livepatch_action):
  - upload (pyxc_livepatch_upload):
  - list (pyxc_livepatch_list):

[a] https://wiki.xenproject.org/wiki/Design_Sessions_2019#LivePatch_improvements_and_features
[b] https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg00846.html

Merged in v1:
  python: Add XC binding for Xen build ID
  livepatch: always print XENLOG_ERR information


Pawel Wieczorkiewicz (12):
  livepatch: Always check hypervisor build ID upon livepatch upload
  livepatch: Allow to override inter-modules buildid dependency
  livepatch: Export payload structure via livepatch_payload.h
  livepatch: Implement pre-|post- apply|revert hooks
  livepatch: Add support for apply|revert action replacement hooks
  livepatch: Do not enforce ELF_LIVEPATCH_FUNC section presence
  livepatch: Add per-function applied/reverted state tracking marker
  livepatch: Add support for inline asm livepatching expectations
  livepatch: Add support for modules .modinfo section metadata
  livepatch: Handle arbitrary size names with the list operation
  livepatch: Add metadata runtime retrieval mechanism
  livepatch: Add python bindings for livepatch operations

 .gitignore                                     |   6 +-
 docs/misc/livepatch.pandoc                     | 248 +++++++++-
 tools/libxc/include/xenctrl.h                  |  68 ++-
 tools/libxc/xc_misc.c                          | 163 ++++--
 tools/misc/xen-livepatch.c                     | 257 +++++++---
 tools/python/xen/lowlevel/xc/xc.c              | 268 ++++++++++
 xen/common/livepatch.c                         | 656 +++++++++++++++++++++----
 xen/include/public/sysctl.h                    |  63 ++-
 xen/include/xen/livepatch.h                    |  43 +-
 xen/include/xen/livepatch_payload.h            |  83 ++++
 xen/test/livepatch/Makefile                    | 121 ++++-
 xen/test/livepatch/xen_action_hooks.c          | 102 ++++
 xen/test/livepatch/xen_action_hooks_marker.c   | 112 +++++
 xen/test/livepatch/xen_action_hooks_noapply.c  | 136 +++++
 xen/test/livepatch/xen_action_hooks_nofunc.c   |  86 ++++
 xen/test/livepatch/xen_action_hooks_norevert.c | 143 ++++++
 xen/test/livepatch/xen_expectations.c          |  41 ++
 xen/test/livepatch/xen_expectations_fail.c     |  42 ++
 xen/test/livepatch/xen_prepost_hooks.c         | 122 +++++
 xen/test/livepatch/xen_prepost_hooks_fail.c    |  75 +++
 20 files changed, 2556 insertions(+), 279 deletions(-)
 create mode 100644 xen/test/livepatch/xen_action_hooks.c
 create mode 100644 xen/test/livepatch/xen_action_hooks_marker.c
 create mode 100644 xen/test/livepatch/xen_action_hooks_noapply.c
 create mode 100644 xen/test/livepatch/xen_action_hooks_nofunc.c
 create mode 100644 xen/test/livepatch/xen_action_hooks_norevert.c
 create mode 100644 xen/test/livepatch/xen_expectations.c
 create mode 100644 xen/test/livepatch/xen_expectations_fail.c
 create mode 100644 xen/test/livepatch/xen_prepost_hooks.c
 create mode 100644 xen/test/livepatch/xen_prepost_hooks_fail.c

Comments

George Dunlap June 9, 2020, 3:01 p.m. UTC | #1
On Tue, Nov 26, 2019 at 10:14 AM Pawel Wieczorkiewicz <wipawel@amazon.de>
wrote:

> This series introduces new features to the livepatch functionality as
> briefly discussed during Xen Developer Summit 2019: [a] and [b].
> It also provides a few fixes and some small improvements.
>
> Main changes in v6:
> - Added missing action pad field zeroing
>
> Main changes in v4:
> - Fix various typos and minor issues
> - Simplify arch_livepatch_{apply,revert} by using
>   common_livepatch_{apply,revert}
> - Improve python bindings and fix few issues
>
> Main changes in v3:
> - Fix expectation test to work on Arm
> - Add test for metadata (Konrad)
> - Minor fixes to documentation
>
> Main changes in v2:
> - added new features to livepatch documentation
> - added livepatch tests
> - enabled Arm support for [5]
> - make .modinfo optional for [11]
> - fixed typos
>
> FEATURES:
>
> 1. independent modules (patches: [1], [2])
>
>   * livepatch-build-tools repo dependency [A]
>
>   Livepatch enforces the following buildid-based dependency chain
>   between hotpatch modules:
>     1) first module depends on given hypervisor buildid
>     2) every consecutive module depends on previous module's buildid
>   This way proper hotpatch stack order is maintained and enforced.
>   While it is important for production hotpatches it limits agility and
>   blocks usage of testing or debug hotpatches. These kinds of hotpatch
>   modules are typically expected to be loaded at any time irrespective
>   of current state of the modules stack.
>
>   [A] livepatch-build: Embed hypervisor build id into every hotpatch
>
> 2. pre- and post- apply|revert actions hooks (patches: [3], [4])
>
>   * livepatch-build-tools repo dependency [B]
>
>   This is an implementation of 4 new livepatch module vetoing hooks,
>   that can be optionally supplied along with modules.
>   Hooks that currently exists in the livepatch mechanism aren't agile
>   enough and have various limitations:
>   * run only from within a quiescing zone
>   * cannot conditionally prevent applying or reverting
>   * do not have access to the module context
>   To address these limitations the following has been implemented:
>   1) pre-apply hook
>   2) post-apply hook
>   3) pre-revert hook
>   4) post-revert hook
>
>   [B] create-diff-object: Handle extra pre-|post- hooks
>
> 3. apply|revert actions replacement hooks (patches: [5], [6], [7])
>
>   * livepatch-build-tools repo dependency: [C], [D], [E]
>
>   To increase hotpatching system's agility and provide more flexiable
>   long-term hotpatch solution, allow to overwrite the default apply
>   and revert action functions with hook-like supplied alternatives.
>   The alternative functions are optional and the default functions are
>   used by default.
>
>   [C] create-diff-object: Do not create empty .livepatch.funcs section
>   [D] create-diff-object: Handle optional apply|revert hooks
>   [E] create-diff-object: Add support for applied/reverted marker
>
> 4. inline asm hotpatching expectations (patches: [8])
>
>   * livepatch-build-tools repo dependency: [F]
>
>   Expectations are designed as optional feature, since the main use of
>   them is planned for inline asm hotpatching.
>   The payload structure is modified as each expectation structure is
>   part of the livepatch_func structure and hence extends the payload.
>   The payload version is bumped to 3 with this change to highlight the
>   ABI modification and enforce proper support.
>   The expectation is manually enabled during inline asm module
>   construction. If enabled, expectation ensures that the expected
>   content of memory is to be found at a given patching (old_addr)
>   location.
>
>   [F] create-diff-object: Add support for expectations
>
> 5. runtime hotpatch metadata support (patches: [9], [10], [11])
>
>   Having detailed hotpatch metadata helps to properly identify module's
>   origin and version. It also allows to keep track of the history of
>   hotpatch loads in the system (at least within dmesg buffer size
>   limits).
>   Extend the livepatch list operation to fetch also payloads' metadata.
>   This is achieved by extending the sysctl list interface with 2 extra
>   guest handles:
>   * metadata     - an array of arbitrary size strings
>   * metadata_len - an array of metadata strings' lengths (uin32_t each)
>   To unify and simplify the interface, handle the modules' name strings
>   of arbitrary size by copying them in adhering chunks to the userland.
>
> 6. python bindings for livepatch operations (patches: [12])
>
>   Extend the XC python bindings library to support all common livepatch
>   operations and actions:
>   - status (pyxc_livepatch_status):
>   - action (pyxc_livepatch_action):
>   - upload (pyxc_livepatch_upload):
>   - list (pyxc_livepatch_list):
>

This series looks like it would be a good candidate for a CHANGELOG.md line.

What about something like this:

- Livepatch improvements: Buildid / hotpatch "stack" restrictions,
Additional {pre,post}-{apply,revert} hooks, inline hotpatching
expectations, runtime hotpatch metdata, python bindings for livepatch
operations

 -George
George Dunlap June 11, 2020, 1:48 p.m. UTC | #2
On Tue, Nov 26, 2019 at 10:14 AM Pawel Wieczorkiewicz <wipawel@amazon.de>
wrote:

> This series introduces new features to the livepatch functionality as
> briefly discussed during Xen Developer Summit 2019: [a] and [b].
> It also provides a few fixes and some small improvements.
>
> Main changes in v6:
> - Added missing action pad field zeroing
>
> Main changes in v4:
> - Fix various typos and minor issues
> - Simplify arch_livepatch_{apply,revert} by using
>   common_livepatch_{apply,revert}
> - Improve python bindings and fix few issues
>
> Main changes in v3:
> - Fix expectation test to work on Arm
> - Add test for metadata (Konrad)
> - Minor fixes to documentation
>
> Main changes in v2:
> - added new features to livepatch documentation
> - added livepatch tests
> - enabled Arm support for [5]
> - make .modinfo optional for [11]
> - fixed typos
>
> FEATURES:
>
> 1. independent modules (patches: [1], [2])
>
>   * livepatch-build-tools repo dependency [A]
>
>   Livepatch enforces the following buildid-based dependency chain
>   between hotpatch modules:
>     1) first module depends on given hypervisor buildid
>     2) every consecutive module depends on previous module's buildid
>   This way proper hotpatch stack order is maintained and enforced.
>   While it is important for production hotpatches it limits agility and
>   blocks usage of testing or debug hotpatches. These kinds of hotpatch
>   modules are typically expected to be loaded at any time irrespective
>   of current state of the modules stack.
>
>   [A] livepatch-build: Embed hypervisor build id into every hotpatch
>
> 2. pre- and post- apply|revert actions hooks (patches: [3], [4])
>
>   * livepatch-build-tools repo dependency [B]
>
>   This is an implementation of 4 new livepatch module vetoing hooks,
>   that can be optionally supplied along with modules.
>   Hooks that currently exists in the livepatch mechanism aren't agile
>   enough and have various limitations:
>   * run only from within a quiescing zone
>   * cannot conditionally prevent applying or reverting
>   * do not have access to the module context
>   To address these limitations the following has been implemented:
>   1) pre-apply hook
>   2) post-apply hook
>   3) pre-revert hook
>   4) post-revert hook
>
>   [B] create-diff-object: Handle extra pre-|post- hooks
>
> 3. apply|revert actions replacement hooks (patches: [5], [6], [7])
>
>   * livepatch-build-tools repo dependency: [C], [D], [E]
>
>   To increase hotpatching system's agility and provide more flexiable
>   long-term hotpatch solution, allow to overwrite the default apply
>   and revert action functions with hook-like supplied alternatives.
>   The alternative functions are optional and the default functions are
>   used by default.
>
>   [C] create-diff-object: Do not create empty .livepatch.funcs section
>   [D] create-diff-object: Handle optional apply|revert hooks
>   [E] create-diff-object: Add support for applied/reverted marker
>
> 4. inline asm hotpatching expectations (patches: [8])
>
>   * livepatch-build-tools repo dependency: [F]
>
>   Expectations are designed as optional feature, since the main use of
>   them is planned for inline asm hotpatching.
>   The payload structure is modified as each expectation structure is
>   part of the livepatch_func structure and hence extends the payload.
>   The payload version is bumped to 3 with this change to highlight the
>   ABI modification and enforce proper support.
>   The expectation is manually enabled during inline asm module
>   construction. If enabled, expectation ensures that the expected
>   content of memory is to be found at a given patching (old_addr)
>   location.
>
>   [F] create-diff-object: Add support for expectations
>
> 5. runtime hotpatch metadata support (patches: [9], [10], [11])
>
>   Having detailed hotpatch metadata helps to properly identify module's
>   origin and version. It also allows to keep track of the history of
>   hotpatch loads in the system (at least within dmesg buffer size
>   limits).
>   Extend the livepatch list operation to fetch also payloads' metadata.
>   This is achieved by extending the sysctl list interface with 2 extra
>   guest handles:
>   * metadata     - an array of arbitrary size strings
>   * metadata_len - an array of metadata strings' lengths (uin32_t each)
>   To unify and simplify the interface, handle the modules' name strings
>   of arbitrary size by copying them in adhering chunks to the userland.
>
> 6. python bindings for livepatch operations (patches: [12])
>
>   Extend the XC python bindings library to support all common livepatch
>   operations and actions:
>   - status (pyxc_livepatch_status):
>   - action (pyxc_livepatch_action):
>   - upload (pyxc_livepatch_upload):
>   - list (pyxc_livepatch_list):
>

This series looks like it would be a good candidate for a CHANGELOG.md line.

What about something like this:

- Livepatch improvements: Buildid / hotpatch "stack" restrictions,
Additional {pre,post}-{apply,revert} hooks, inline hotpatching
expectations, runtime hotpatch metdata, python bindings for livepatch
operations
Wieczorkiewicz, Pawel June 12, 2020, 7:26 a.m. UTC | #3
> On 11. Jun 2020, at 15:48, George Dunlap <dunlapg@umich.edu> wrote:
> 
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> 
> 
> 
> 
> On Tue, Nov 26, 2019 at 10:14 AM Pawel Wieczorkiewicz <wipawel@amazon.de> wrote:
> This series introduces new features to the livepatch functionality as
> briefly discussed during Xen Developer Summit 2019: [a] and [b].
> It also provides a few fixes and some small improvements.
> 
> Main changes in v6:
> - Added missing action pad field zeroing
> 
> Main changes in v4:
> - Fix various typos and minor issues
> - Simplify arch_livepatch_{apply,revert} by using
>   common_livepatch_{apply,revert}
> - Improve python bindings and fix few issues
> 
> Main changes in v3:
> - Fix expectation test to work on Arm
> - Add test for metadata (Konrad)
> - Minor fixes to documentation
> 
> Main changes in v2:
> - added new features to livepatch documentation
> - added livepatch tests
> - enabled Arm support for [5]
> - make .modinfo optional for [11]
> - fixed typos
> 
> FEATURES:
> 
> 1. independent modules (patches: [1], [2])
> 
>   * livepatch-build-tools repo dependency [A]
> 
>   Livepatch enforces the following buildid-based dependency chain
>   between hotpatch modules:
>     1) first module depends on given hypervisor buildid
>     2) every consecutive module depends on previous module's buildid
>   This way proper hotpatch stack order is maintained and enforced.
>   While it is important for production hotpatches it limits agility and
>   blocks usage of testing or debug hotpatches. These kinds of hotpatch
>   modules are typically expected to be loaded at any time irrespective
>   of current state of the modules stack.
> 
>   [A] livepatch-build: Embed hypervisor build id into every hotpatch
> 
> 2. pre- and post- apply|revert actions hooks (patches: [3], [4])
> 
>   * livepatch-build-tools repo dependency [B]
> 
>   This is an implementation of 4 new livepatch module vetoing hooks,
>   that can be optionally supplied along with modules.
>   Hooks that currently exists in the livepatch mechanism aren't agile
>   enough and have various limitations:
>   * run only from within a quiescing zone
>   * cannot conditionally prevent applying or reverting
>   * do not have access to the module context
>   To address these limitations the following has been implemented:
>   1) pre-apply hook
>   2) post-apply hook
>   3) pre-revert hook
>   4) post-revert hook
> 
>   [B] create-diff-object: Handle extra pre-|post- hooks
> 
> 3. apply|revert actions replacement hooks (patches: [5], [6], [7])
> 
>   * livepatch-build-tools repo dependency: [C], [D], [E]
> 
>   To increase hotpatching system's agility and provide more flexiable
>   long-term hotpatch solution, allow to overwrite the default apply
>   and revert action functions with hook-like supplied alternatives.
>   The alternative functions are optional and the default functions are
>   used by default.
> 
>   [C] create-diff-object: Do not create empty .livepatch.funcs section
>   [D] create-diff-object: Handle optional apply|revert hooks
>   [E] create-diff-object: Add support for applied/reverted marker
> 
> 4. inline asm hotpatching expectations (patches: [8])
> 
>   * livepatch-build-tools repo dependency: [F]
> 
>   Expectations are designed as optional feature, since the main use of
>   them is planned for inline asm hotpatching.
>   The payload structure is modified as each expectation structure is
>   part of the livepatch_func structure and hence extends the payload.
>   The payload version is bumped to 3 with this change to highlight the
>   ABI modification and enforce proper support.
>   The expectation is manually enabled during inline asm module
>   construction. If enabled, expectation ensures that the expected
>   content of memory is to be found at a given patching (old_addr)
>   location.
> 
>   [F] create-diff-object: Add support for expectations
> 
> 5. runtime hotpatch metadata support (patches: [9], [10], [11])
> 
>   Having detailed hotpatch metadata helps to properly identify module's
>   origin and version. It also allows to keep track of the history of
>   hotpatch loads in the system (at least within dmesg buffer size
>   limits).
>   Extend the livepatch list operation to fetch also payloads' metadata.
>   This is achieved by extending the sysctl list interface with 2 extra
>   guest handles:
>   * metadata     - an array of arbitrary size strings
>   * metadata_len - an array of metadata strings' lengths (uin32_t each)
>   To unify and simplify the interface, handle the modules' name strings
>   of arbitrary size by copying them in adhering chunks to the userland.
> 
> 6. python bindings for livepatch operations (patches: [12])
> 
>   Extend the XC python bindings library to support all common livepatch
>   operations and actions:
>   - status (pyxc_livepatch_status):
>   - action (pyxc_livepatch_action):
>   - upload (pyxc_livepatch_upload):
>   - list (pyxc_livepatch_list):
> 
> This series looks like it would be a good candidate for a CHANGELOG.md line.
> 
> What about something like this:
> 
> - Livepatch improvements: Buildid / hotpatch "stack" restrictions, Additional {pre,post}-{apply,revert} hooks, inline hotpatching expectations, runtime hotpatch metdata, python bindings for livepatch operations
> 

LGTM, thanks!

Is there anything I have to do?

Best,
Pawel Wieczorkiewicz
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879