mbox series

[00/17] Rust support

Message ID 20210704202756.29107-1-ojeda@kernel.org (mailing list archive)
Headers show
Series Rust support | expand

Message

Miguel Ojeda July 4, 2021, 8:27 p.m. UTC
From: Miguel Ojeda <ojeda@kernel.org>

Rust support

This is the patch series to add support for Rust as a second language
to the Linux kernel.

If you are interested in following this effort, please join us in
the mailing list at:

    rust-for-linux@vger.kernel.org

and take a look at the project itself at:

    https://github.com/Rust-for-Linux

Cheers,
Miguel

--

# Rust support

This cover letter explains the major changes and updates done since
the RFC sent in the previous merge window back in April, plus a few
extra notes and announcements. For the RFC, please see:

    https://lore.kernel.org/lkml/20210414184604.23473-1-ojeda@kernel.org/


## Rust infrastructure updates

There have been several major improvements to the overall Rust
support. The following subsections cover these.


### Removed panicking allocations

We have removed infallible allocations. In order to do so, we have
integrated a subset of the `alloc` standard library crate, with some
additions on top. This allows us to customize things to our needs,
while giving upstream the time they need to evaluate our changes.

Eventually, the goal is to have everything the kernel needs in
upstream `alloc` and drop it from the kernel tree. We have already
started this process and some changes have been already accepted
upstream.

On top of that, `alloc` is now compiled with panicking allocation
methods disabled, thus they cannot be used within the kernel by
mistake either.

Moreover, the documentation for this customized `alloc`  crate (as well
as for `core`) is now generated alongside the rest of the Rust kernel
documentation. Thus kernel developers can now easily browse the subset
that is available within the kernel. Like last time, you can take
a look at a preview of the documentation at:

     https://rust-for-linux.github.io/docs/alloc/

Note that the `compiler_builtins` panicking intrinsics are still
there, but those will be solved by partitioning `core` via feature
gates. The first one, for disabling floating-point functionality,
has just been accepted upstream.


### Beta compiler supported

Up until now, we have been using nightly releases of `rustc` because
we need some of the latest fixes and unstable features.

However, the kernel can now be compiled with beta and stable `rustc`
releases. At the moment, we are using the 1.54-beta1 version as our
reference compiler. At the end of this month, `rustc` 1.54 will be
released, and we will move to that version as our reference.

Note that the kernel still requires unstable features, even if it is
compiled with a stable `rustc` release, thus we cannot guarantee that
future `rustc` versions will work without changes to the kernel tree.

Thus, until all the unstable features we need are stabilized, we will
support a single `rustc` version for each kernel release.


### Testing support

Another big addition has been the support for testing. We now support
the standard Rust `#[test]` attribute to easily write tests, e.g.:

    #[test]
    fn f() {
        let a = 20;
        let b = 22;
        assert_eq!(a + b, 42);
    }

Furthermore, we are now also supporting Rust documentation tests
("doctests"). These allow us to make sure our examples remain up to
date, and also double as tests too, e.g.:

    /// ```
    /// assert_eq!(foo::f(), 42);
    /// ```
    pub fn f() -> i32 {
        42
    }

For the moment, both kinds of tests are run in the host only, but
the goal is to have them running in kernel space, so that we can
test code that depends on kernel features, and to allow any kernel
module to declare and use them.


### Architectures and compiler support

`arm` (i.e. 32-bit) and `riscv` are now also supported.

On compilers, we would like to mention all the work that has been
going in GCC Rust (a GCC frontend for the Rust language) and
`rustc_codegen_gcc` (a `rustc` backend for GCC). The latter now passes
all `core` tests, and the former is now working on traits. We continue
to track their progress as they may become the best way to have
GCC-built kernels with Rust support enabled. Their latest reports
can be found at:

    https://blog.antoyo.xyz/rustc_codegen_gcc-progress-report-1
    https://thephilbert.io/2021/06/28/gcc-rust-weekly-status-report-20/

On top of that, we requested Compiler Explorer to add support to all
the alternative compilers. At the time of writing, they have already
added `mrustc` and GCC Rust; and `rustc_codegen_gcc` is coming soon.
See a live example at:

    https://godbolt.org/z/8o74c57Yj


## Rust abstractions and driver updates

We have developed new Rust abstractions that use the kernel C
implementations: red-black trees, reference-counted objects, file
descriptor creation, tasks, files, io vectors...

Additionally, we have improved driver support: improvements to
`file_operations` (more operations supported, arbitrary state),
reduced boiler-plate code, improved `module!` macro, registration
macros, rudimentary (`probe` and `remove`) platform drivers...

On Binder, there is now support for transferring files descriptors
and LSM hooks; and we are working on preliminary performance numbers.

Moreover, there is ongoing work on a Rust example driver,
`bcm2835-rng`. This is the hardware random-number generator present
on Raspberry Pi Zero(W), Classic, Two, and Three.

There are other small improvements, such as drivers being restricted
on what unstable features they can use.


## Patch series status

Like it was mentioned in the RFC, the Rust support is still to be
considered experimental. However, as noted back in April, support is
good enough that kernel developers can start working on the Rust
abstractions for subsystems and write drivers and other modules.

Please note that the current series have just arrived in `linux-next`,
thus the first run will happen on Tuesday.


## Industry and academia support

We have been in contact with a set of companies and academia members
that would like to use Rust as a second language in the kernel. Some
of them have already started to evaluate Rust for their needs using
the infrastructure we have already in place.

In particular, we have got a few statements from major companies.
In no particular order:

  Microsoft's Linux Systems Group is interested in contributing to
  getting Rust into Linux kernel. Hopefully we will be able to submit
  select Hyper-V drivers written in Rust in the coming months.

  Arm recognises the Rust value proposition and is actively working
  with the Rust community to improve Rust for Arm based systems.
  A good example is Arm’s RFC contribution to the Rust language which
  made Linux on 64-bit Arm systems a Tier-1 Rust supported platform.
  Rustaceans at Arm are excited about the Rust for Linux initiative
  and look forward to assisting in this effort.

  Google supports and contributes directly to the Rust for Linux
  project. Our Android team is evaluating a new Binder implementation
  and considering other drivers where Rust could be adopted.

In addition, IBM contributed the Rust kernel support for PowerPC
which was already included in the RFC.

In addition, from academia, there are already several projects around
Rust for Linux going on. As an example, members of LSE (Systems
Research Laboratory) at EPITA (École pour l'informatique et les
techniques avancées) are developing an SPI Rust driver.

And, of course, special thanks go to ISRG (Internet Security Research
Group) and Google for their financial support on this endeavor.


## Conferences and talks

We have submitted talk proposals for LPC (Linux Plumbers Conference).

The main one will describe the work we have done so far and ideally
will also serve as an introduction for other kernel developers
interested in using Rust in the kernel. It will cover an introduction
of the language within the context of the kernel, how the overall Rust
support works, how code is documented, how tests are written, a tour
of available tooling, an explanation of coding guidelines, how kernel
driver code looks like, etc.

In addition, we would like to announce that we are organizing a new
conference that focuses on Rust and the Linux kernel. The first
edition will be virtual and will take place before LPC. Details will
be announced soon.


## Acknowledgements

The signatures in the main commits correspond to the people that
wrote code that has ended up in them at the present time. For details
on contributions to code and discussions, please see our repository:

    https://github.com/Rust-for-Linux/linux

However, we would like to give credit to everyone that has contributed
in one way or another to the Rust for Linux project. Since the RFC:

  - bjorn3 for all the input on Rust compiler details and all
    the reviews and suggestions.

  - Arthur Cohen, Esteban Blanc and Martin Schmidt for their ongoing
    work on the SPI abstractions and driver.

  - Dan Robertson for his ongoing work on softdeps in the `module!`
    macro and the addition of a few safety comments.

  - Paul Römer for his ongoing experiment on using `NonNull` as much
    as possible.

  - Sladyn Nunes for his ongoing sorting of error constants.

  - Jonathan Corbet and the LPC organizers for lending us the Linux
    Plumbers Conference infrastructure so that we can have an easy
    time setting up the new conference.

  - John Ericson for quickly implementing the `no_global_oom_handling`
    `cfg` feature in upstream `alloc` to easily disable all
    functionality that relies on panicking allocations.

  - Josh Triplett and John Ericson for their input on `alloc` which
    helped us decide what to do with it (i.e. fully custom vs.
    slightly custom in-tree copy vs. upstream), as well as offering
    to help moving forward some needed features on the Rust side.

  - Mark Rousskov for answering some questions about beta backports
    and scheduling, as well as working on unsticking `1.54.0-beta.1`.

  - Philipp Krones for his input on Clippy lints and discussing
    extensions for developing custom lints.

  - Antoni Boucher for his work on `rustc_codegen_gcc`.

  - Philip Herrons (and his supporters Open Source Security and
    Embecosm) for his work on GCC Rust.

  - Marc Poulhiès for his work on Compiler Explorer to add the
    alternative Rust compilers we requested.

  - Many folks that have reported issues, tested the project,
    helped spread the word, joined discussions and contributed in
    other ways! In no particular order: Chenguang Wang, Greg Morenz,
    John Baublitz, Leah Leshchinsky, Caedin Cook, Liam Arzola,
    Fabio Aiuto, Hanqing Zhao, Robin Randhawa, Michal Rostecki,
    Wei Liu...

For additional acknowledgements, please see the RFC from April.

Miguel Ojeda (17):
  kallsyms: support big kernel symbols (2-byte lengths)
  kallsyms: increase maximum kernel symbol length to 512
  Makefile: generate `CLANG_FLAGS` even in GCC builds
  vsprintf: add new `%pA` format specifier
  rust: add C helpers
  rust: add `compiler_builtins` crate
  rust: add `alloc` crate
  rust: add `build_error` crate
  rust: add `macros` crate
  rust: add `kernel` crate
  rust: export generated symbols
  Kbuild: add Rust support
  docs: add Rust documentation
  samples: add Rust examples
  scripts: add `generate_rust_analyzer.py`
  MAINTAINERS: Rust
  Android: Binder IPC in Rust (WIP)

 .gitignore                                  |    5 +
 .rustfmt.toml                               |   12 +
 Documentation/doc-guide/kernel-doc.rst      |    3 +
 Documentation/index.rst                     |    1 +
 Documentation/kbuild/kbuild.rst             |    4 +
 Documentation/process/changes.rst           |   13 +
 Documentation/rust/arch-support.rst         |   35 +
 Documentation/rust/assets/favicon-16x16.png |  Bin 0 -> 798 bytes
 Documentation/rust/assets/favicon-32x32.png |  Bin 0 -> 2076 bytes
 Documentation/rust/assets/rust-logo.png     |  Bin 0 -> 53976 bytes
 Documentation/rust/coding.rst               |   92 +
 Documentation/rust/docs.rst                 |  110 +
 Documentation/rust/index.rst                |   20 +
 Documentation/rust/quick-start.rst          |  222 ++
 MAINTAINERS                                 |   14 +
 Makefile                                    |  176 +-
 arch/arm/rust/target.json                   |   28 +
 arch/arm64/rust/target.json                 |   35 +
 arch/powerpc/rust/target.json               |   30 +
 arch/riscv/Makefile                         |    1 +
 arch/riscv/rust/rv32ima.json                |   37 +
 arch/riscv/rust/rv32imac.json               |   37 +
 arch/riscv/rust/rv64ima.json                |   37 +
 arch/riscv/rust/rv64imac.json               |   37 +
 arch/x86/rust/target.json                   |   37 +
 drivers/android/Kconfig                     |    7 +
 drivers/android/Makefile                    |    2 +
 drivers/android/allocation.rs               |  264 ++
 drivers/android/context.rs                  |   80 +
 drivers/android/defs.rs                     |   99 +
 drivers/android/node.rs                     |  476 +++
 drivers/android/process.rs                  |  972 ++++++
 drivers/android/range_alloc.rs              |  189 ++
 drivers/android/rust_binder.rs              |  114 +
 drivers/android/thread.rs                   |  857 +++++
 drivers/android/transaction.rs              |  328 ++
 include/linux/kallsyms.h                    |    2 +-
 include/linux/spinlock.h                    |   17 +-
 include/uapi/linux/android/binder.h         |   28 +-
 init/Kconfig                                |   28 +
 kernel/kallsyms.c                           |    7 +
 kernel/livepatch/core.c                     |    4 +-
 kernel/printk/printk.c                      |    5 +-
 lib/Kconfig.debug                           |  144 +
 lib/vsprintf.c                              |   12 +
 rust/.gitignore                             |    6 +
 rust/Makefile                               |  316 ++
 rust/alloc/README.md                        |   32 +
 rust/alloc/alloc.rs                         |  425 +++
 rust/alloc/borrow.rs                        |  493 +++
 rust/alloc/boxed.rs                         | 1728 ++++++++++
 rust/alloc/collections/mod.rs               |  116 +
 rust/alloc/fmt.rs                           |  587 ++++
 rust/alloc/lib.rs                           |  197 ++
 rust/alloc/macros.rs                        |  128 +
 rust/alloc/prelude/mod.rs                   |   17 +
 rust/alloc/prelude/v1.rs                    |   16 +
 rust/alloc/raw_vec.rs                       |  612 ++++
 rust/alloc/rc.rs                            | 2539 +++++++++++++++
 rust/alloc/slice.rs                         | 1271 ++++++++
 rust/alloc/str.rs                           |  614 ++++
 rust/alloc/string.rs                        | 2847 ++++++++++++++++
 rust/alloc/sync.rs                          | 2631 +++++++++++++++
 rust/alloc/vec/drain.rs                     |  157 +
 rust/alloc/vec/drain_filter.rs              |  145 +
 rust/alloc/vec/into_iter.rs                 |  296 ++
 rust/alloc/vec/is_zero.rs                   |  106 +
 rust/alloc/vec/mod.rs                       | 3255 +++++++++++++++++++
 rust/alloc/vec/partial_eq.rs                |   49 +
 rust/alloc/vec/set_len_on_drop.rs           |   30 +
 rust/alloc/vec/spec_extend.rs               |  170 +
 rust/bindgen_parameters                     |   13 +
 rust/build_error.rs                         |   33 +
 rust/compiler_builtins.rs                   |  146 +
 rust/exports.c                              |   16 +
 rust/helpers.c                              |  235 ++
 rust/kernel/allocator.rs                    |   63 +
 rust/kernel/bindings.rs                     |   28 +
 rust/kernel/bindings_helper.h               |   24 +
 rust/kernel/buffer.rs                       |   39 +
 rust/kernel/build_assert.rs                 |   80 +
 rust/kernel/c_types.rs                      |  119 +
 rust/kernel/chrdev.rs                       |  212 ++
 rust/kernel/error.rs                        |  272 ++
 rust/kernel/file.rs                         |  130 +
 rust/kernel/file_operations.rs              |  698 ++++
 rust/kernel/io_buffer.rs                    |  153 +
 rust/kernel/iov_iter.rs                     |   95 +
 rust/kernel/lib.rs                          |  220 ++
 rust/kernel/linked_list.rs                  |  245 ++
 rust/kernel/miscdev.rs                      |  113 +
 rust/kernel/module_param.rs                 |  497 +++
 rust/kernel/of.rs                           |  101 +
 rust/kernel/pages.rs                        |  176 +
 rust/kernel/platdev.rs                      |  166 +
 rust/kernel/prelude.rs                      |   28 +
 rust/kernel/print.rs                        |  412 +++
 rust/kernel/random.rs                       |   50 +
 rust/kernel/raw_list.rs                     |  361 ++
 rust/kernel/rbtree.rs                       |  570 ++++
 rust/kernel/security.rs                     |   79 +
 rust/kernel/static_assert.rs                |   39 +
 rust/kernel/str.rs                          |  259 ++
 rust/kernel/sync/arc.rs                     |  227 ++
 rust/kernel/sync/condvar.rs                 |  136 +
 rust/kernel/sync/guard.rs                   |   82 +
 rust/kernel/sync/locked_by.rs               |  112 +
 rust/kernel/sync/mod.rs                     |   84 +
 rust/kernel/sync/mutex.rs                   |  101 +
 rust/kernel/sync/spinlock.rs                |  109 +
 rust/kernel/sysctl.rs                       |  198 ++
 rust/kernel/task.rs                         |  193 ++
 rust/kernel/traits.rs                       |   26 +
 rust/kernel/types.rs                        |  249 ++
 rust/kernel/user_ptr.rs                     |  191 ++
 rust/macros/lib.rs                          |  127 +
 rust/macros/module.rs                       |  754 +++++
 samples/Kconfig                             |    2 +
 samples/Makefile                            |    1 +
 samples/rust/Kconfig                        |  113 +
 samples/rust/Makefile                       |   12 +
 samples/rust/rust_chrdev.rs                 |   51 +
 samples/rust/rust_minimal.rs                |   38 +
 samples/rust/rust_miscdev.rs                |  150 +
 samples/rust/rust_module_parameters.rs      |   72 +
 samples/rust/rust_print.rs                  |   57 +
 samples/rust/rust_random.rs                 |   61 +
 samples/rust/rust_semaphore.rs              |  177 +
 samples/rust/rust_semaphore_c.c             |  212 ++
 samples/rust/rust_stack_probing.rs          |   40 +
 samples/rust/rust_sync.rs                   |   81 +
 scripts/Makefile.build                      |   22 +
 scripts/Makefile.lib                        |   12 +
 scripts/generate_rust_analyzer.py           |  143 +
 scripts/kallsyms.c                          |   33 +-
 scripts/kconfig/confdata.c                  |   67 +-
 scripts/rust-version.sh                     |   31 +
 tools/include/linux/kallsyms.h              |    2 +-
 tools/include/linux/lockdep.h               |    2 +-
 tools/lib/perf/include/perf/event.h         |    2 +-
 tools/lib/symbol/kallsyms.h                 |    2 +-
 141 files changed, 33003 insertions(+), 45 deletions(-)
 create mode 100644 .rustfmt.toml
 create mode 100644 Documentation/rust/arch-support.rst
 create mode 100644 Documentation/rust/assets/favicon-16x16.png
 create mode 100644 Documentation/rust/assets/favicon-32x32.png
 create mode 100644 Documentation/rust/assets/rust-logo.png
 create mode 100644 Documentation/rust/coding.rst
 create mode 100644 Documentation/rust/docs.rst
 create mode 100644 Documentation/rust/index.rst
 create mode 100644 Documentation/rust/quick-start.rst
 create mode 100644 arch/arm/rust/target.json
 create mode 100644 arch/arm64/rust/target.json
 create mode 100644 arch/powerpc/rust/target.json
 create mode 100644 arch/riscv/rust/rv32ima.json
 create mode 100644 arch/riscv/rust/rv32imac.json
 create mode 100644 arch/riscv/rust/rv64ima.json
 create mode 100644 arch/riscv/rust/rv64imac.json
 create mode 100644 arch/x86/rust/target.json
 create mode 100644 drivers/android/allocation.rs
 create mode 100644 drivers/android/context.rs
 create mode 100644 drivers/android/defs.rs
 create mode 100644 drivers/android/node.rs
 create mode 100644 drivers/android/process.rs
 create mode 100644 drivers/android/range_alloc.rs
 create mode 100644 drivers/android/rust_binder.rs
 create mode 100644 drivers/android/thread.rs
 create mode 100644 drivers/android/transaction.rs
 create mode 100644 rust/.gitignore
 create mode 100644 rust/Makefile
 create mode 100644 rust/alloc/README.md
 create mode 100644 rust/alloc/alloc.rs
 create mode 100644 rust/alloc/borrow.rs
 create mode 100644 rust/alloc/boxed.rs
 create mode 100644 rust/alloc/collections/mod.rs
 create mode 100644 rust/alloc/fmt.rs
 create mode 100644 rust/alloc/lib.rs
 create mode 100644 rust/alloc/macros.rs
 create mode 100644 rust/alloc/prelude/mod.rs
 create mode 100644 rust/alloc/prelude/v1.rs
 create mode 100644 rust/alloc/raw_vec.rs
 create mode 100644 rust/alloc/rc.rs
 create mode 100644 rust/alloc/slice.rs
 create mode 100644 rust/alloc/str.rs
 create mode 100644 rust/alloc/string.rs
 create mode 100644 rust/alloc/sync.rs
 create mode 100644 rust/alloc/vec/drain.rs
 create mode 100644 rust/alloc/vec/drain_filter.rs
 create mode 100644 rust/alloc/vec/into_iter.rs
 create mode 100644 rust/alloc/vec/is_zero.rs
 create mode 100644 rust/alloc/vec/mod.rs
 create mode 100644 rust/alloc/vec/partial_eq.rs
 create mode 100644 rust/alloc/vec/set_len_on_drop.rs
 create mode 100644 rust/alloc/vec/spec_extend.rs
 create mode 100644 rust/bindgen_parameters
 create mode 100644 rust/build_error.rs
 create mode 100644 rust/compiler_builtins.rs
 create mode 100644 rust/exports.c
 create mode 100644 rust/helpers.c
 create mode 100644 rust/kernel/allocator.rs
 create mode 100644 rust/kernel/bindings.rs
 create mode 100644 rust/kernel/bindings_helper.h
 create mode 100644 rust/kernel/buffer.rs
 create mode 100644 rust/kernel/build_assert.rs
 create mode 100644 rust/kernel/c_types.rs
 create mode 100644 rust/kernel/chrdev.rs
 create mode 100644 rust/kernel/error.rs
 create mode 100644 rust/kernel/file.rs
 create mode 100644 rust/kernel/file_operations.rs
 create mode 100644 rust/kernel/io_buffer.rs
 create mode 100644 rust/kernel/iov_iter.rs
 create mode 100644 rust/kernel/lib.rs
 create mode 100644 rust/kernel/linked_list.rs
 create mode 100644 rust/kernel/miscdev.rs
 create mode 100644 rust/kernel/module_param.rs
 create mode 100644 rust/kernel/of.rs
 create mode 100644 rust/kernel/pages.rs
 create mode 100644 rust/kernel/platdev.rs
 create mode 100644 rust/kernel/prelude.rs
 create mode 100644 rust/kernel/print.rs
 create mode 100644 rust/kernel/random.rs
 create mode 100644 rust/kernel/raw_list.rs
 create mode 100644 rust/kernel/rbtree.rs
 create mode 100644 rust/kernel/security.rs
 create mode 100644 rust/kernel/static_assert.rs
 create mode 100644 rust/kernel/str.rs
 create mode 100644 rust/kernel/sync/arc.rs
 create mode 100644 rust/kernel/sync/condvar.rs
 create mode 100644 rust/kernel/sync/guard.rs
 create mode 100644 rust/kernel/sync/locked_by.rs
 create mode 100644 rust/kernel/sync/mod.rs
 create mode 100644 rust/kernel/sync/mutex.rs
 create mode 100644 rust/kernel/sync/spinlock.rs
 create mode 100644 rust/kernel/sysctl.rs
 create mode 100644 rust/kernel/task.rs
 create mode 100644 rust/kernel/traits.rs
 create mode 100644 rust/kernel/types.rs
 create mode 100644 rust/kernel/user_ptr.rs
 create mode 100644 rust/macros/lib.rs
 create mode 100644 rust/macros/module.rs
 create mode 100644 samples/rust/Kconfig
 create mode 100644 samples/rust/Makefile
 create mode 100644 samples/rust/rust_chrdev.rs
 create mode 100644 samples/rust/rust_minimal.rs
 create mode 100644 samples/rust/rust_miscdev.rs
 create mode 100644 samples/rust/rust_module_parameters.rs
 create mode 100644 samples/rust/rust_print.rs
 create mode 100644 samples/rust/rust_random.rs
 create mode 100644 samples/rust/rust_semaphore.rs
 create mode 100644 samples/rust/rust_semaphore_c.c
 create mode 100644 samples/rust/rust_stack_probing.rs
 create mode 100644 samples/rust/rust_sync.rs
 create mode 100755 scripts/generate_rust_analyzer.py
 create mode 100755 scripts/rust-version.sh

Comments

Miguel Ojeda July 4, 2021, 11:11 p.m. UTC | #1
On Sun, Jul 4, 2021 at 10:28 PM <ojeda@kernel.org> wrote:
>
> This is the patch series to add support for Rust as a second language
> to the Linux kernel.

In case somebody wonders: patch 07 was particularly big and lore does
not seem to be showing it.

Let me know if you want me to split it; otherwise I will ask
Konstantin when he is back from holidays.

>   - Many folks that have reported issues, tested the project,
>     helped spread the word, joined discussions and contributed in
>     other ways! In no particular order: Chenguang Wang, Greg Morenz,
>     John Baublitz, Leah Leshchinsky, Caedin Cook, Liam Arzola,
>     Fabio Aiuto, Hanqing Zhao, Robin Randhawa, Michal Rostecki,
>     Wei Liu...

I forgot to mention Samantha Miller, Santosh Sivaraj, Vegard Nossum
and Leandro Coutinho.

My apologies!

Cheers,
Miguel
Christoph Hellwig July 7, 2021, 6:43 a.m. UTC | #2
Not sure if this is just a state of the union post or if you actually
want to submit it.  If the later I strongly disagree with merging it,
as you should refined it and prove it actually is useful first.  Where
useful would be a real-life driver like say nvme or a usb host
controller driver that actually works and shows benefits over the
existing one.  Until it shows such usefulness it will just drag
everyone else down.
Marco Elver July 7, 2021, 10:51 a.m. UTC | #3
On Sun, Jul 04, 2021 at 10:27PM +0200, ojeda@kernel.org wrote:
[...]
>   rust: add `alloc` crate
[...]

I think this patch never made it to the mailing list. b4 also doesn't
find it:

	ERROR: missing [7/17]
Miguel Ojeda July 7, 2021, 11:28 a.m. UTC | #4
On Wed, Jul 7, 2021 at 12:51 PM Marco Elver <elver@google.com> wrote:
>
> I think this patch never made it to the mailing list. b4 also doesn't
> find it:
>
>         ERROR: missing [7/17]

Yeah, thanks -- I mentioned in my reply above:

    https://lore.kernel.org/lkml/CANiq72nQq8Y8v9Pyf7JFq6Kf-+doNP+mHAFNzj_cSFBa3KwS5w@mail.gmail.com/

If you want to try the patches, you can pick them from this tree (the
one in linux-next):

    https://github.com/Rust-for-Linux/linux/tree/rust-next

Cheers,
Miguel
Miguel Ojeda July 7, 2021, 12:33 p.m. UTC | #5
On Wed, Jul 7, 2021 at 8:44 AM Christoph Hellwig <hch@infradead.org> wrote:
>
> Not sure if this is just a state of the union post or if you actually
> want to submit it.  If the later I strongly disagree with merging it,

We are submitting it.

> as you should refined it and prove it actually is useful first.  Where
> useful would be a real-life driver like say nvme or a usb host
> controller driver that actually works and shows benefits over the
> existing one.  Until it shows such usefulness it will just drag
> everyone else down.

It is "proven" in the sense we are already starting to get users
downstream and other interested parties have shown support.

However, others are more conservative and will only start investing
into it if we are in mainline, i.e. if the decision of having Rust or
not is already taken.

But it is fair to ask for a hardware driver to see how it looks like,
which is why the cover letter mentions we are working on one.

Now, if you are OK with non-hardware modules, you can take a look at
Rust Binder (last patch in the series) which is a non-trivial module
and it is already working.

Cheers,
Miguel
Greg KH July 7, 2021, 12:50 p.m. UTC | #6
On Wed, Jul 07, 2021 at 02:33:57PM +0200, Miguel Ojeda wrote:
> Now, if you are OK with non-hardware modules, you can take a look at
> Rust Binder (last patch in the series) which is a non-trivial module
> and it is already working.

Cool, does it actually pass the binder self-tests that the Android
systems have for the codebase?  Last I looked at this thing, it was not
feature-complete compared to the in-kernel binder code, has that been
resolved and the needed filesystem changes added?

thanks,

greg k-h
Wedson Almeida Filho July 7, 2021, 2:07 p.m. UTC | #7
On Wed, Jul 07, 2021 at 02:50:54PM +0200, Greg Kroah-Hartman wrote:
> On Wed, Jul 07, 2021 at 02:33:57PM +0200, Miguel Ojeda wrote:
> > Now, if you are OK with non-hardware modules, you can take a look at
> > Rust Binder (last patch in the series) which is a non-trivial module
> > and it is already working.
> 
> Cool, does it actually pass the binder self-tests that the Android
> systems have for the codebase?

We haven't run the Android tests yet because they depend on an Android-specific
service (servicemanager) running and other Android-specific libraries. What we
are doing instead is adding binder tests that don't depend on anything
Android-specific; in fact, we are putting them in tools/testing/selftests/binder
so that they can run on any vanilla system.

The commit is available here:
https://github.com/wedsonaf/linux/commit/f90ec49be9207fa765f07ad1071210ad871712ac

The tests are written in C and run successfully against both C and Rust drivers.
I still have another ~20 tests that I wrote in another harness that I will
convert to selftests soon, but the two together I believe have more coverage
than the ones in Android.

We also have a trivial latency benchmark (ping with no payload) where the Rust
version performs better than the C one.

The benchmark is available here: https://github.com/wedsonaf/linux/commits/ping

> Last I looked at this thing, it was not
> feature-complete compared to the in-kernel binder code, has that been
> resolved and the needed filesystem changes added?

It is not feature-complete in comparison to the C one just yet, it is missing a
few things but not for any fundamental reason -- we were mostly focusing on the
kernel crate and tests.

Miguel's point is that it does implement the vast majority of binder features
and is non-trivial, so it could be used as evidence that useful kernel drivers
can be built with Rust; not just "transpiled" from C, but written with the Rust
safety guarantees.

Cheers,
-Wedson
Christoph Hellwig July 7, 2021, 2:12 p.m. UTC | #8
On Wed, Jul 07, 2021 at 03:07:50PM +0100, Wedson Almeida Filho wrote:
> Miguel's point is that it does implement the vast majority of binder features
> and is non-trivial, so it could be used as evidence that useful kernel drivers
> can be built with Rust; not just "transpiled" from C, but written with the Rust
> safety guarantees.

binder is not a kernel driver.  It is a ill design IPC mechanism
absolutely no representative for other kernel code.

Please write an actual real driver dealing with real (and common) hardware
and come back with the results.
Greg KH July 7, 2021, 3:02 p.m. UTC | #9
On Wed, Jul 07, 2021 at 03:07:50PM +0100, Wedson Almeida Filho wrote:
> > Last I looked at this thing, it was not
> > feature-complete compared to the in-kernel binder code, has that been
> > resolved and the needed filesystem changes added?
> 
> It is not feature-complete in comparison to the C one just yet, it is missing a
> few things but not for any fundamental reason -- we were mostly focusing on the
> kernel crate and tests.

I love it how you call "binderfs is missing" a "few things" :)

> Miguel's point is that it does implement the vast majority of binder features
> and is non-trivial, so it could be used as evidence that useful kernel drivers
> can be built with Rust; not just "transpiled" from C, but written with the Rust
> safety guarantees.

As Christoph said, and I and others have said before, binder is in no
way shape or form anything that resembles any sort of "driver" at all.
It is a crazy IPC mechanism that is tacked onto the side of the kernel.
Not to say that it doesn't have its usages, but the interactions between
binder and the rest of the kernel are very small and specific.
Something that almost no one else will ever write again.

Please work on a real driver to help prove, or disprove, that this all
is going to be able to work properly.  There are huge unanswered
questions that need to be resolved that you will run into when you do
such a thing.

Good luck!

greg k-h
Fiona Behrens July 7, 2021, 8:56 p.m. UTC | #10
> On 7. Jul 2021, at 17:02, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> 
> On Wed, Jul 07, 2021 at 03:07:50PM +0100, Wedson Almeida Filho wrote:
>>> Last I looked at this thing, it was not
>>> feature-complete compared to the in-kernel binder code, has that been
>>> resolved and the needed filesystem changes added?
>> 
>> It is not feature-complete in comparison to the C one just yet, it is missing a
>> few things but not for any fundamental reason -- we were mostly focusing on the
>> kernel crate and tests.
> 
> I love it how you call "binderfs is missing" a "few things" :)
> 
>> Miguel's point is that it does implement the vast majority of binder features
>> and is non-trivial, so it could be used as evidence that useful kernel drivers
>> can be built with Rust; not just "transpiled" from C, but written with the Rust
>> safety guarantees.
> 
> As Christoph said, and I and others have said before, binder is in no
> way shape or form anything that resembles any sort of "driver" at all.
> It is a crazy IPC mechanism that is tacked onto the side of the kernel.
> Not to say that it doesn't have its usages, but the interactions between
> binder and the rest of the kernel are very small and specific.
> Something that almost no one else will ever write again.
> 
> Please work on a real driver to help prove, or disprove, that this all
> is going to be able to work properly.  There are huge unanswered
> questions that need to be resolved that you will run into when you do
> such a thing.
> 

There is a more general use driver (network dummy) still in the making, It is fully operational, just the documentation of the rust bindings are not finished yet, so it is not merged into the rust tree yet, also I have to rebase it.
I testet the network dummy (dummyrs) driver and for stupid tests like Iperf3 or ping it performed slightly better that the C version. See https://github.com/Kloenk/linux/blob/rust-netdevice/drivers/net/dummy_rs.rs for the actually driver.

Yes this driver is still only virtually, but the network apis are probably more general usage than the binder apis.

> Good luck!
> 
> greg k-h

CU,
Finn
Matthew Wilcox (Oracle) July 8, 2021, 12:58 a.m. UTC | #11
On Wed, Jul 07, 2021 at 10:56:57PM +0200, Finn Behrens wrote:
> There is a more general use driver (network dummy) still in the making, It is fully operational, just the documentation of the rust bindings are not finished yet, so it is not merged into the rust tree yet, also I have to rebase it.

Why are you so resistant to writing a real driver that deals with actual
hardware?  A simple NVMe driver is less than a thousand lines of C.
I know the one in the kernel now is ridiculously complicated and has
been thoroughly messed up with abstractions to support NVMeoF instead
of having a separate driver, but it's really a simple interface at heart.
Wedson Almeida Filho July 22, 2021, 10:55 p.m. UTC | #12
Hey Matthew,

On Thu, Jul 08, 2021 at 01:58:32AM +0100, Matthew Wilcox wrote:
> Why are you so resistant to writing a real driver that deals with actual
> hardware?  

I don't think it was so much resistance but rather a prioritisation thing. Have
you by any chance seen the gpio driver I posted a couple of days ago?

> A simple NVMe driver is less than a thousand lines of C.
> I know the one in the kernel now is ridiculously complicated and has
> been thoroughly messed up with abstractions to support NVMeoF instead
> of having a separate driver, but it's really a simple interface at heart.

The latest NVMe spec is 452 pages long, which seems to contradict your claim
that it's simple. In any case, translating less than 1K lines of C shouldn't be
too hard (after I've built the abstractions, of course). Would you mind sharing
the simple driver you mention above?

Thanks,
-Wedson
Matthew Wilcox (Oracle) July 23, 2021, 1:17 a.m. UTC | #13
On Thu, Jul 22, 2021 at 11:55:58PM +0100, Wedson Almeida Filho wrote:
> Hey Matthew,
> 
> On Thu, Jul 08, 2021 at 01:58:32AM +0100, Matthew Wilcox wrote:
> > Why are you so resistant to writing a real driver that deals with actual
> > hardware?  
> 
> I don't think it was so much resistance but rather a prioritisation thing. Have
> you by any chance seen the gpio driver I posted a couple of days ago?

I haven't seen it, no ...

> > A simple NVMe driver is less than a thousand lines of C.
> > I know the one in the kernel now is ridiculously complicated and has
> > been thoroughly messed up with abstractions to support NVMeoF instead
> > of having a separate driver, but it's really a simple interface at heart.
> 
> The latest NVMe spec is 452 pages long, which seems to contradict your claim
> that it's simple.

As I said, they've put all kinds of crap into NVMe these days.
If you look at the 1.0e spec, it's 127 pages.

> In any case, translating less than 1K lines of C shouldn't be
> too hard (after I've built the abstractions, of course). Would you mind sharing
> the simple driver you mention above?

Unfortunately, most of the early versions were lost during the
kernel.org breakin.  If you check out commit 5da273fe3fd1 and look at
drivers/block/nvme.c, you'll see a driver that's about 2000 lines.
Keith Busch July 23, 2021, 3:08 p.m. UTC | #14
On Thu, Jul 22, 2021 at 11:55:58PM +0100, Wedson Almeida Filho wrote:
> On Thu, Jul 08, 2021 at 01:58:32AM +0100, Matthew Wilcox wrote:
> > A simple NVMe driver is less than a thousand lines of C.
> > I know the one in the kernel now is ridiculously complicated and has
> > been thoroughly messed up with abstractions to support NVMeoF instead
> > of having a separate driver, but it's really a simple interface at heart.
> 
> The latest NVMe spec is 452 pages long, which seems to contradict your claim
> that it's simple. In any case, translating less than 1K lines of C shouldn't be
> too hard (after I've built the abstractions, of course). Would you mind sharing
> the simple driver you mention above?

You can use the 1.0 spec, which is much shorter. A 1.0 capable driver
should be forward compatible with newer devices, too.

The current nvme driver became less simple since blk-mq integration and
has only gotten more complicated since then with other transports and
more advanced features. For a simpler example, you can reference an in
kernel version <= 3.16, and ignore the "nvme-scsi" parts.