Plan and objectives

Weekly activity reports

This page contains weekly reports of the Virtual Open Systems activity about the "AGL Native VIRTIO phase 2" project.

week 16

GPU:

This week we identified the GPU rendering issues we were experiencing as dependent on the virtio-gpu driver to allow access on the framebuffer resource. There is then a need to remove kernel-space dependencies and fully render GPU graphics in user-space inside the adapter application. As a result, we are now working in that direction, updating the adapter's vhost-user-gpu support with the SDL rendering pipeline of Qemu. This could lead to a small delay in the delivery of the GPU support (initially planned for Oct. 31st). We will provide more info with a new deadline next week.

CAN:     
Finish an initial draft version of vhost-user-can under the adapter. A new vhost-user-can device was implemented under adapter which compiles without error and it is ready to be tested with a backend device. In parallel, continue working with vhost-user-can under rust-vmm project. An initial draft version, was already in place but the device is not mature enough to be compiled and tested with adapter's component. Next week, we target to have an first communication between adapter and backend device, exchanging virtio and vhost-user features, queue number, sizes etc.

week 15

GPU:

Updated the virtio-loopback-adapter GPU's support and split the frames-rendering part to a standalone application. This is intended to result in a viewer-application that uses standardized DRM/KMS APIs to retrieve and render frames from the character device that is probed by the virtio-gpu driver. Performed tests to verify the virtio-gpu probed driver state and the message exchanges with vhost-user-gpu. Tested the virtqueues management and the events/irqs received by the virtio-loopback-adapter threads with success.
Next steps include completing and resolving the issues on the viewer-application for virtio-gpu.

CAN:     
We are following last week's plan and we started working on the first implementation of the vhost-user-can interface for the virtio-loopback-adapter. This is the part of the adapter that handles the communications between [virtio-can](https://lwn.net/Articles/934187/) with the vhost-user-can backend in rust (will be implemented in parallel). As of today, since there is no vhost-user-can front-end device in QEMU project, the development will be based on older adapter's device (ex. GPIO) and the [virtio-can](https://lwn.net/Articles/934187/) QEMU device which is still under RFC phase. Next week we will continue working on the adapter's component, while starting to build an initial vhost-user-can backend device under rust-vmm.

week 14

GPU:
We are working on the partial rendering issue we have with the glmark2 application. In the debug we are particularly focusing on the virtio-loopback-adapter and the virtqueues notification event mechanism which bridges the vhost-user-gpu and the virtio-gpu driver.

Next steps include resolving the rendering issue and polishing the changes on virtio-loopback-adapter and virtio-loopback-driver.

CAN:     
We started working on the CAN support and after some investigations we prepared an initial plan. In general, there is no vhost-user device available publicly, and the plan includes the development of a rust vhost-user-can device that will be shared with the rust-vmm community.
Plan milestones:
1) Implement Vhost-user-sound Adapter interface   (deadline 23/10/2023)

- Based on other adapter's device implementation: [virtio-can RFC](https://lwn.net/Articles/934187/)
2) Vhost-user-can backend in Rust (deadline 01/12/2023)
- Implement vhost-user-can based on vhost-device-rng & vhost-device-gpio
- [vhost-device](https://github.com/rust-vmm/vhost-device/tree/main)

3) Testing & Prepare the release: (deadline 08/12/2023)
- Polish/Push the epsilon-release version by the addition of vhost-user-can

week 13

GPU:
We moved the development environment on a host (we were working inside a guest) and we resolved the latest stability issues. Now we are able to load/unload the driver multiple times etc. Rendering is still partial with the glmark2 application. It needs more investigation that will be done next week.

Sound:      

Vhost-user-sound device is added in virtio-loopback, tested and pushed (epsilon-release).
- [adapter](https://git.virtualopensystems.com/virtio-loopback/adapter_app/-/tree/epsilon-release)
- [loopback driver](https://git.virtualopensystems.com/virtio-loopback/loopback_driver/-/tree/epsilon-release)
- [vhost-device](https://git.virtualopensystems.com/virtio-loopback/vhost-user-rng-rust/vhost-device/-/tree/epsilon-release) (sound & gpio)

Note: The vhost-user-sound & vhost-user-gpio backend implementations were taken by the developer repo [vhost-device](https://github.com/virtio-sound/vhost-device) (since vhost-user-sound is under development).

We integrated new sound and GPIO devices in the epsilon release and we will be pushing this new version of the 29225 patches early next week.

Week 12

GPU:
The effort on the GPU task was spent on finalizing the visualization of 3D applications. Continued the tests with the glmark2 OpenGL application. At the current stage the visualization is not yet stable and we are working to identify the issue.
Next step is to find the current issue and finalize window rendering..

Sound:

In the context of the audio tests, we updated the adapter and loopback driver exchange notifications and interrupts. After the updates, the behavior of the sound device seems to be stable, and we able to get audio out of the speakers when using the "pipewire" backend. Tests were applied for the rest of the vhost-user-sound's backends as "null" and "alsa".
Next week we will continue testing the virtio-sound behavior on top of virtio-sound, and start preparing a preliminary release of the code under the "epsilon-release" branches.

GPIO:

Vhost-user-gpio device is added in virtio-loopback, tested and pushed (epsilon-release).
    - [adapter](https://git.virtualopensystems.com/virtio-loopback/adapter_app/-/tree/epsilon-release)
    - [loopback driver](https://git.virtualopensystems.com/virtio-loopback/loopback_driver/-/tree/epsilon-release)
The vhost-user-gpio backend implementation was taken by the upstream [rust-vmm repository](https://github.com/rust-vmm/vhost-device/tree/main).
Next step is to embed the device in the yocto recipe.

Week 11

GPU:
Continued the work on the visualization of virtio-gpu framebuffers. Tested the merge of static contextes (background framebuffer) and GL contextes from the 3D pipeline with a test application. Started to test the example 3D applications from mesa-demos to check for incompatibilities.
Next work includes the finalization of visualization/window rendering and getting information about the performance with a GPU benchmark application.

Sound:
As mentioned last week, we are working in parallel to evaluate two scenarios:
1) Testing vhost-user-sound with Qemu:
Started testing vhost-user-sound with both "ALSA" and "Pipewire" backend and Qemu. We weren't able to make it work, we tried both linaro's and redhat's versions of QEMU. On "pipewire" case, when we tried to play audio with "aplay audio.wav" (or mpg123 audio.mp3), we get a short sound from the speaker and an debug messages "Dropping IOmessage". Lastly, on the "ALSA" case, an error message is being printed, followed by the same "Dropping IOmessage" log and the stream is closed.
2) Testing vhost-user-sound on virtio-loopback:
Also in this scenario, we tried to output audio by running vhost-user-sound on top of virtio-loopback. By using the "pipewire" backend and running "aplay audio.wav" we were able to get a full song played on the speakers. During the execution, logs "Dropping IOmessage" were printed, audio came out only from one of the speakers and we noticed that the stream was not closed successfully after the song's end. This might be caused due to the "stop" message doesn't reach vhost-user-sound successfully. "ALSA" backend did not yet work successfully also on "virtio-loopback" case.
Next week we will continue to investigate the ALSA and Pipewire backends on both "QEMU - vhost-user", "Virtio-loopback - vhost-user" scenarios. Our first priority, would be to understand which of the errors noticed are derived by virtio-loopback implementation and correct them, targeting to play a sound stream on the speakers successfully.

GPIO:
We started polishing the new adapter's gpio device and working on a new release.
Next week we plan to finalize and push the gpio release on our git servers. The gerrit patches will arrive the week after.

Week 10

Activity

GPU:
Worked on the visualization part of the virtio-gpu DRM device. Continued the investigation on how to combine framebuffers from multiple sources.
Explored the QEMU display handling code (high-level) for potential similarities and implementation specific constraints.

Sound:

Testing vhost-user-sound on virtio-loopback: This week we were able to solve the issue and eliminate any behavioral differences between the scenarios "QEMU - vhost-user-sound" and "Virtio-loopback - vhost-user-sound". Seems that on virtio-loopback case, for big audio files we observe more stable behavior when SMP is enabled. We continue our investigation by start testing sound device with ALSA backend where we found some errors printed from the device side when try to play an audio file.
Next week we will continue to investigate the ALSA case and in parallel start testing sound with "Pipewire" backend which is still under active development to evaluate the current status of the device.

GPIO:
Testing vhost-user-gpio on virtio-loopback: Continue developing the vhost-user-gpio frontend into the adapter and testing it with "gpio-mockup" pseudo device in the kernel. We are able to observe notification and data being exchanged between virtio-gpio and vhost-user-gpio. In more details, we are able to read GPIO lines and bits by using "gpioget" command, but not able to change those values ("gpioset"). After a short investigation, "gpioset" was not able to change gpio value even directly on "gpio-mockup" device (no virtio-loopback in place), so this proves was a issue unrelated to the virito-loopback solution.
Next week we plan to continue testing the virtio-gpio device, and start polishing the implementation if no new issues occur.

Week 9

GPU:
Worked on the finalization effort for enabling the virtio-gpu DRM renderer device. Created a simple OpenGL workload application to test the functionality of the DRM renderer device and the handling of GL calls by 'virgl'.
Successfully managed to run the 3D application without exceptions anywhere from the software-stack. Verified the GL operation by tracing the calls on vhost-user-gpu and the handling code in 'virgl'. Begin to extend the software stack in order to visualize the frames of the framebuffer of the virtio-gpu device. At the moment this part will be integrated on the 'Adapter' application.
The next steps are to continue the development of the visualization code for the virtio-gpu frames, and explore how to deal with multiple contexts when the 3D acceleration is enabled.

sound:
This week we tested vhost-user-sound with the new adapter sound interface, comparing the behavior of vhost-user-sound between the scenarios a) virtio-loopback (non-hypervisor) and b) QEMU (host-guest) to test similarities and differences. During the tests we noticed that some notifications from multiple virtqueues coming from the kernel collided. We updated the notification mechanism for multiple virtqueues between the driver and adapter to make sure that all of them are delivered and with the same order.
Next week we will continue to investigate the above behavior and go one step closer to finalize the vhost-user-sound adapter frontend before moving to experiment with other sound backend as "alsa" and "pipewire".

gpio:

We continued investigations on the linux kernel side (`gpio_virtio.ko`) and on the vhost-user-gpio device. Additionally, we added support for a new generic device in the adapter (to be extended as a gpio device next week). The development environment has been prepared, and it includes the gpio-mockup kernel module, which is going to be used to test the implementation.

Next week, we plan to extend the existing adapter support to gpio (today it is just a generic device).

Week 8

Activity

GPU:

Worked on the Adapter application to debug and trace the enablement of the virtio-gpu DRM Renderer device. The device is part of the virtio-gpu driver, handles memory requests from the user-space driver library, and exposes the render node via '/dev/dri/renderDxxx'
Updated the Adapter to inform the 'virtio-mmio' driver with the correct parameters in order to register the virtio-gpu DRM driver with the renderer feature enabled. Successfully made the virtio-gpu driver to register both the display and renderer devices.

Our plan for next week are on stabilizing the virtio-gpu DRM driver with the renderer feature and test an OpenGL application

sound:

Development of the first draft of the virtio-loopback-adapter sound interface complete. Now, a new virtio-sound device is being registered successfully into the kernel and the virtio-loopback-adapter sound interface is able to communicate (exchange notifications) with the vhost-user-sound device. The vhost-user-sound backend when using the "null" backend seems to be able to access the shared data space without any error and read data. This proves that the compatibility layer for rust devices developed the previous week works as expected also for vhost-user-sound. 

Next steps will be to check the validity of the data read by vhost-user-sound from the share data space, and soon after, start experimenting with alsa and pipewire.

gpio:

We started the activity with investigations on the current gpio driver status (vhost user device available, developed in rust).

Next week we plan to start adding gpio support in the adapter application.

Week 7

Activity

GPU:

This week we focused on running the weston compositor on virtio-gpu with virtio-looback. Tested the user-space DRM char-devices with libdrm examples and proceeded on running the weston compositor. At this moment, there is no visual output for the frames rendered by virtio-gpu, thus the frames are checked by dumping the contents of the framebuffer.

Next week: the steps are to enable the 3D acceleration on vhost-user-gpu and make sure that the virtio-gpu DRM device probes the renderer device as well. More testing will be done by creating an OpenGL workload application to debug & trace the overall stack.

sound:

Continue working on the adapter sound front-end and upgrade the data sharing mechanism in order to exchange data with vhost-user-sound backend (in Rust).

The target of this week, was to: a) create a new compatibility layer in the adapter so vhost-user sound device can access the shared space without the requirement of any modification of vhost-user-sound, b) continue the development of adapter vhost-user-sound front-end, so a new linux virtio-sound device will be register successfully at the end of this process. The updated adapter version (a) has the new compatibility layer integrated, and it was tested with vhost-user-rng device in rust, and in C to cross-validate it (testing it with vhost-user-sound will be done later at a later stage). About the "device registration" objective (b), virtio-sound in linux seem to not register a new device yet, so the adapter sound front-end part still needs further development.

Next week: we will continue working on the front-end sound part and target an initial version of the adapter which will be ready to register the virtio sound device in the kernel successfully.

Week 6

Activity

sound: Established an initial communication between adapter and vhost-user-sound backend. During this communication, the two user-space components exchange virtio features, vhost-user features, queue number, queue size, etc. but they do not exchange any data. We developed some missing parts of the adapter in order to handle all messages (excluding for now data related messages) i.e., send/receive to/from the vhost-user-sound device using the "null" backend.

Plan for next week Next week our effort will be focused on the data exchange process between the two components (adapter, vhost-user-sound backend).

Week 5

Activity

GPU:
We continued the effort on enabling all features of the virtio-gpu DRM driver. At this stage 'vhost-user-gpu' runs without the 'virgl' flag, thus meaning that only the display part of the virtio-gpu DRM driver is enabled.
Started testing the functionality of the virtio-gpu driver over virtio-loopback. Primarily, the tests are using the libdrm library to ensure the expected functionality from the virtio-gpu DRM driver.
The foreseen development and testing plan consist of running the weston compositor on the virtio-gpu and enabling the 3D translation path on vhost-user-gpu, Adapter and the virtio-gpu driver.

sound:
We are working at the first implementation of the vhost-user-sound interface for the virtio-loopback-adapter. This is the part of the adapter that handles the communications with the vhost-user-sound backend. Now the adapter successfully creates an instance of the sound device, and starts the message exchange with the vhost-user backend device. As of today, we based this development on the RedHat solution because we found it as more complete than Linaro's implementation (see weekly report 4 for more info). We are constantly monitoring the vhost-user-sound developments to make sure we do not miss any update and minimize the impact of changes on our work with virtio-loopback.

Plan for next week Next week we will continue working on the adapter component, while we are testing its communication with vhost-user-sound backend and developing any missing parts.

Week 4

Activity

GPU: We continued the effort on enabling all features of the virtio-gpu DRM driver. At this stage 'vhost-user-gpu' runs without the 'virgl' flag, thus meaning that only the display part of the virtio-gpu DRM driver is enabled. Started testing the functionality of the virtio-gpu driver over virtio-loopback. Primarily, the tests are using the libdrm library to ensure the expected functionality from the virtio-gpu DRM driver.
The foreseen development and testing plan for next week consists of running the weston compositor on the virtio-gpu and enabling the 3D translation path on vhost-user-gpu, Adapter and the virtio-gpu driver.

Sound: This week we explored the initialization procedures for devices with existing support on 'virtio-loopback' (input, blk, rng), and started a preliminary skeleton implementation for 'vhost-user-sound'  on the 'Adapter' application. 

Moreover, we continued the investigations and we found two alternative implementations of vhost-user-sound (one from RedHat, and one from linaro). Since none of the two is upstream, we will try to understand what's the most advanced and the most likely to be accepted upstream. Similarly to what we prepared with GPU, the development plan for sound is the following:

Part 1: Work on the virtio-loopback Adapter application
Add support for the vhost-user-sound device on the adapter application.
This will handle the new command-line configuration for vhost-user-sound, handle the socket connection with 'vhost-user-sound' backend and to instantiate/load the 'virtio-sound' driver module At this stage it will be possible to start and communicate with the rust backend device, but no way to hear the output on the speakers.
Estimation: ~4.5 weeks

Part 2: Update vhost-user-sound device from rust-vmm
This task targets to fill any development gaps existing at the time of vhost-user-sound backend and establish a successful communication with the Adapter and eventually with virtio-sound. The above-mentioned repositories already implement a vhost-user-sound backend in rust, although they might not be very stable as they are under development. This task should carefully evaluate the two to decide which way to go and integrate one of the two as AGL solution. Fixes on top of the selected existing solution, if needed, will be implemented.
Estimation: ~4.5 weeks

Part 3: Integrate/testing vhost-user-sound, virtio-loopback with Pipewire/ALSA
This task will focus on establishing a successful communication between virtio-sound and Pipewire via virtio-loopback. Putting together parts 1 and 2, the outcome of this task is an application will be able through virtio-loopback and vhost-user device to communicate with Pipewrire/Alsa and output to the speakers.
Estimation: ~3 weeks

Part 4: Integration/testing with virtio-loopback upstream and AGL yocto layers
This task targets the upstreaming effort to update the upstream version of virtio-loopback.
Estimation: 1 week

Week 3

Activity

GPU:We focused on the Adapter app debugging the interactions and data shared on the socket interface connected to the vhost-user-gpu. We managed to resolve the registration of the Virtio-GPU DRM driver issue mentioned last week. Now the virtio-GPU DRM driver is loaded correctly by the loopback-driver.
We are now about to test on the Virtio-GPU DRM character device to verify and match the functionality that is needed. At the same time, we started the activities related to the sound as discussed during the last EG call.

Plan for next week Plan for next week is to continue the work for testing the DRM driver, as well as to progress with the sound.

Week 2

Activity

GPU: The activities advancement for this week are mostly on the 'Adapter' application. We progressed with the development of the virtio-gpu interface on the 'Adapter' and investigated the code-path until the virtio-loopback driver requests to register the virtio-gpu driver. At this stage the virtio-gpu driver registration takes place with invalid parameters and the DRM driver is not loaded correctly. We then continued on debugging the communication data from 'vhost-user-gpu' to the 'Adapter' as well as the features set passed when running the 'vhost-user-gpu' with the '-virgl' flag.

Plan for next week The next steps include the successful registration of the virtio-gpu driver, continue the development of the 'Adapter' and initiate a plan to test the functionality of the virtio-gpu driver. The goal is to run a compositor (weston) on the character-device exposed by the virtio-gpu and stress the functionality of the solution. Similar tests will be carried on at a lower-API level using the 'libdrm' library.

Week 1

Activity

GPU: we continued the investigation on 'vhost-user-gpu' and the vhost-user protocol. We started the development of the virtio-gpu agnostic code on the 'Adapter' application. Explored the communication messages sent by the 'vhost-user-gpu' application to the 'Adapter' via the socket interface as well as messages from devices with existing support (virtio-rng) to compare. Traced the feature-set provided by 'vhost-user-gpu', the vhost-user handling code and continued with the support of the gpu interface on the 'Adapter' application.

Plan for next week is to investigate the virtio-gpu driver registration and continue the development of virtio-gpu specific code on the 'Adapter' application.

Week 0 (started 2023-07-03)

ActivityT

GPU:his week we started the investigations on the 'vhost-user-gpu' implementation of Qemu and its socket interface. This qemu component provides the device emulation for 'virtio-gpu' and allows an external application to connect via socket (virtio-loopback 'Adapter' in our case). Moreover, we explored the initialization procedures for devices with existing support on 'virtio-loopback' (input, rng), and started a preliminary skeleton implementation for 'vhost-user-gpu'  on the 'Adapter' application. As a result of these investigations, we prepared a development plan for the next weeks:

virtio-GPU development plan

• • • Part 1: decouple the Virtio-GPU device from the QEMU

This task targets to explore and understand the virtio-gpu device (vhost-user-gpu) from the QEMU project. The functionality is packed in an application and later used via socket by the adapter application. This includes the graphics (rendering) part of the GPU as well as the display portion.

Estimation: ~4 weeks

• • • Part 2: Work on the virtio-loopback Adapter application

Add support for the virtio-gpu device on the adapter application.
This will handle the new command-line configuration for virtio-gpu, handle the socket connection with 'vhost-user-gpu' and to instantiatiate/load the 'virtio-gpu' driver module At this stage it will be possible to start and communicate with the GPU device, but no way to see the output on a screen/window.

Estimation: ~4 weeks

• • • Part 3: Wiring the adapter with virglrenderer and integration with the display server

By concept, the GPU commands, data are proxied and handled by the virglrenderer library. On this task we are wiring the virglrenderer context with the adapter application to allow the client-virtio-GPU context to be forwarded to the real-GPU context. As a proof-of-concept the display server will be a Wayland based compositor.

Estimation: ~4 weeks

• • • Part 4: Integration/testing with virtio-loopback upstream and AGL yocto layers

Estimation: ~1 weeks

Plan for next week is to go deeper in Part 1 with the addition of the 'vhost-user-gpu' interface to the 'Adapter' and the instantiation of the 'virtio-gpu' driver by the loopback kernel module.