mbox series

[00/10] Generic USB Display driver

Message ID 20200429124830.27475-1-noralf@tronnes.org (mailing list archive)
Headers show
Series Generic USB Display driver | expand

Message

Noralf Trønnes April 29, 2020, 12:48 p.m. UTC 
Hi,

A while back I had the idea to turn a Raspberry Pi Zero into a $5
USB to HDMI/SDTV/DSI/DPI display adapter.

This series adds a USB host driver and a device/gadget driver to achieve
that.

The reason for calling it 'Generic' is so anyone can make a USB
display/adapter against this driver, all that's needed is to add a USB
vid:pid. I was hoping to have someone working on a microcontroller based
USB display by now, but unfortunately that has been delayed. It would
have been nice to have a microcontroller implementation to ensure that I
haven't made things unnecessary difficult to implement.

Performance
The one thing that decides how useful this all is, is how smooth an
experience it gives. My hope was that it should not be noticeably laggy
with ordinary office use on 1920x1080@RG16. I'm pleased to see that it's
also possible to watch youtube movies, although not in fullscreen.

Some of the main factors that affects performance:
- Display resolution
- Userspace providing damage reports (FB_DAMAGE_CLIPS or
DRM_IOCTL_MODE_DIRTYFB)
- Color depth (DRM_CAP_DUMB_PREFERRED_DEPTH = 16 if RGB565)
- How well the frames compress (lz4)
- Gadget device memory bandwidth, CPU power for decompression
- (Big endian hosts will have to do byte swapping on the frames)

I've tested these:
- xorg-server on Pi4. This was nice and smooth since it uses
DRM_IOCTL_MODE_DIRTYFB and honours DRM_CAP_DUMB_PREFERRED_DEPTH.
- Ubuntu 20.04 GNOME on x86. This was useable, but not so good for
movies. GNOME doesn't look at DRM_CAP_DUMB_PREFERRED_DEPTH and doesn't
set FB_DAMAGE_CLIPS on the pageflips.

I've made a short video to show what it looks like[1].

I have used a Pi4 as the gadget during development since it has much
better memory bandwith (4000 vs 200 MBps)[2] and CPU than the PiZ. They
both have the same gadget controller (dwc2).

I did an RFC [3] of this 2 months ago where I looked at doing a Multi
Function USB device. I gave up on that when I realised how much work the
review process would be. So I stripped down to my original idea. I have
made sure that the drivers will tolerate another USB interface of type
VENDOR, so it's still possible for the display to be part of a multi
function device.

Noralf.

[1] https://youtu.be/AhGZWwUm8JU
[2] https://magpi.raspberrypi.org/articles/raspberry-pi-specs-benchmarks
[3] https://patchwork.freedesktop.org/series/73508/


Noralf Trønnes (10):
  backlight: Add backlight_device_get_by_name()
  drm: Add backlight helper
  drm/client: Add drm_client_init_from_id()
  drm/client: Add drm_client_framebuffer_flush()
  drm/client: Add drm_client_modeset_check()
  drm/client: Add a way to set modeset, properties and rotation
  drm/format-helper: Add drm_fb_swab()
  drm: Add Generic USB Display driver
  drm/gud: Add functionality for the USB gadget side
  usb: gadget: function: Add Generic USB Display support

 .../ABI/testing/configfs-usb-gadget-gud_drm   |   10 +
 Documentation/gpu/drm-kms-helpers.rst         |    6 +
 MAINTAINERS                                   |   10 +
 drivers/gpu/drm/Kconfig                       |    9 +
 drivers/gpu/drm/Makefile                      |    2 +
 drivers/gpu/drm/drm_backlight_helper.c        |  154 +++
 drivers/gpu/drm/drm_client.c                  |   79 +-
 drivers/gpu/drm/drm_client_modeset.c          |  174 ++-
 drivers/gpu/drm/drm_format_helper.c           |   61 +-
 drivers/gpu/drm/drm_mipi_dbi.c                |    2 +-
 drivers/gpu/drm/gud/Kconfig                   |   20 +
 drivers/gpu/drm/gud/Makefile                  |    5 +
 drivers/gpu/drm/gud/gud_drm_connector.c       |  629 +++++++++
 drivers/gpu/drm/gud/gud_drm_drv.c             |  624 +++++++++
 drivers/gpu/drm/gud/gud_drm_gadget.c          | 1169 +++++++++++++++++
 drivers/gpu/drm/gud/gud_drm_internal.h        |   66 +
 drivers/gpu/drm/gud/gud_drm_pipe.c            |  423 ++++++
 drivers/usb/gadget/Kconfig                    |   12 +
 drivers/usb/gadget/function/Makefile          |    2 +
 drivers/usb/gadget/function/f_gud_drm.c       |  678 ++++++++++
 drivers/video/backlight/backlight.c           |   21 +
 include/drm/drm_backlight_helper.h            |    9 +
 include/drm/drm_client.h                      |   44 +-
 include/drm/drm_connector.h                   |   10 +
 include/drm/drm_format_helper.h               |    4 +-
 include/drm/gud_drm.h                         |  369 ++++++
 include/linux/backlight.h                     |    1 +
 27 files changed, 4563 insertions(+), 30 deletions(-)
 create mode 100644 Documentation/ABI/testing/configfs-usb-gadget-gud_drm
 create mode 100644 drivers/gpu/drm/drm_backlight_helper.c
 create mode 100644 drivers/gpu/drm/gud/Kconfig
 create mode 100644 drivers/gpu/drm/gud/Makefile
 create mode 100644 drivers/gpu/drm/gud/gud_drm_connector.c
 create mode 100644 drivers/gpu/drm/gud/gud_drm_drv.c
 create mode 100644 drivers/gpu/drm/gud/gud_drm_gadget.c
 create mode 100644 drivers/gpu/drm/gud/gud_drm_internal.h
 create mode 100644 drivers/gpu/drm/gud/gud_drm_pipe.c
 create mode 100644 drivers/usb/gadget/function/f_gud_drm.c
 create mode 100644 include/drm/drm_backlight_helper.h
 create mode 100644 include/drm/gud_drm.h

Comments

Noralf Trønnes May 1, 2020, 1:22 p.m. UTC | #1 
Den 29.04.2020 14.48, skrev Noralf Trønnes:
> Hi,
> 
> A while back I had the idea to turn a Raspberry Pi Zero into a $5
> USB to HDMI/SDTV/DSI/DPI display adapter.
> 
> This series adds a USB host driver and a device/gadget driver to achieve
> that.
> 
> The reason for calling it 'Generic' is so anyone can make a USB
> display/adapter against this driver, all that's needed is to add a USB
> vid:pid. I was hoping to have someone working on a microcontroller based
> USB display by now, but unfortunately that has been delayed. It would
> have been nice to have a microcontroller implementation to ensure that I
> haven't made things unnecessary difficult to implement.
> 
> Performance
> The one thing that decides how useful this all is, is how smooth an
> experience it gives. My hope was that it should not be noticeably laggy
> with ordinary office use on 1920x1080@RG16. I'm pleased to see that it's
> also possible to watch youtube movies, although not in fullscreen.
> 
> Some of the main factors that affects performance:
> - Display resolution
> - Userspace providing damage reports (FB_DAMAGE_CLIPS or
> DRM_IOCTL_MODE_DIRTYFB)
> - Color depth (DRM_CAP_DUMB_PREFERRED_DEPTH = 16 if RGB565)
> - How well the frames compress (lz4)
> - Gadget device memory bandwidth, CPU power for decompression
> - (Big endian hosts will have to do byte swapping on the frames)

One factor that I forgot is USB2 vs USB3.
The Pi's have a USB2 Device controller (dwc2). I couldn't find a cheap
board with a USB3 Device controller that could run mainline Linux, so I
haven't tried that.

> 
> I've tested these:
> - xorg-server on Pi4. This was nice and smooth since it uses
> DRM_IOCTL_MODE_DIRTYFB and honours DRM_CAP_DUMB_PREFERRED_DEPTH.
> - Ubuntu 20.04 GNOME on x86. This was useable, but not so good for
> movies. GNOME doesn't look at DRM_CAP_DUMB_PREFERRED_DEPTH and doesn't
> set FB_DAMAGE_CLIPS on the pageflips.
> 
> I've made a short video to show what it looks like[1].

I got a question if this would work with usbip[4], USB over IP.
I did a quick test with two Pi4's connected by cable to the same network
switch (1Gb). Showing a movie in a window like my previous test didn't
show much of a difference. Maybe some occasional glitching, hard to tell
without proper tests.

There's no pageflipping on the device side, so it could be tearing that
I saw.

Noralf.

[4] tools/usb/usbip/README

> 
> I have used a Pi4 as the gadget during development since it has much
> better memory bandwith (4000 vs 200 MBps)[2] and CPU than the PiZ. They
> both have the same gadget controller (dwc2).
> 
> I did an RFC [3] of this 2 months ago where I looked at doing a Multi
> Function USB device. I gave up on that when I realised how much work the
> review process would be. So I stripped down to my original idea. I have
> made sure that the drivers will tolerate another USB interface of type
> VENDOR, so it's still possible for the display to be part of a multi
> function device.
> 
> Noralf.
> 
> [1] https://youtu.be/AhGZWwUm8JU
> [2] https://magpi.raspberrypi.org/articles/raspberry-pi-specs-benchmarks
> [3] https://patchwork.freedesktop.org/series/73508/
>