virtio-loopback: multi device vhost-user-sound support for CES2024

# Running two virtio-sound devices on top of virtio-loopback on AGL Reference Hardware platform (RefHw)

## An overview
This guide contains instructions and sources/binaries to manually compile, test and experiment virtio-loopback with two vhost-user sound devices on **AGL reference hardware**. This initial multi-device support was developed to demonstrate the flexibility of virtio-loopback, but it is by no means stable and optimized.
The yocto integration of virtio-loopback and vhost-user-sound in AGL is ongoing and is targeting single device support only. Future work idea is to properly support multi-devices in 2024.

This work was presented at OSS Japan by Virtual Open Systems (https://ossjapan2022.sched.com/event/1D127/virtio-loopback-an-hardware-abstraction-layer-hal-for-non-virtualized-environments-michele-paolino-virtual-open-systems).

The code of this demo diverges a bit from the single-device that is targeting upstream in AGL. For instance, it contains makefiles to build two identical virtio-loopback-drivers with different names. For this reason, we placed all the sources/binaries in a temporary public repository available [here](https://git.virtualopensystems.com/virtio-loopback/docs/-/tree/multi-sound-guide).

 *[Figure 2]: Description of two virtio-sound device running on top of virtio-loopback architecture*


## Components
The architecture we want to put in place is composed by the following components:
- [Virtio Loopback Transport](https://git.virtualopensystems.com/virtio-loopback/loopback_driver/-/tree/multi-sound-demo), a Linux module which acts as a transport for the virtio-devices and redirects the virtio messages to the user-space (virtio-loopback-adapter).
- [Virtio Loopback Adapter](https://git.virtualopensystems.com/virtio-loopback/adapter_app/-/tree/multi-sound-demo) is a user-space application which is the intermediate between virtio-loopback-transport and vhost-user-rng device and bridges the communication between them.
- [vhost-device-sound](https://github.com/rust-vmm/vhost-device.git) is the upstream vhost-user-sound device

# Instructions for reproducing the VOSYS OSS23 demo

## 1. Prerequisites

Before start compiling and running the components of the architecture you need:
- **RefHw Linux kernel** dependencies:
  + The following configuration flags need to be enabled: *CONFIG_VIRTIO*, *CONFIG_SND_VIRTIO*
- **RefHw filesystem** dependencies:
  + **alsa-utils** so to use *aplay* command, in order to feed audio stream to a sound device of the system
- **Host system** dependencies
  + **git** to fetch the sources and patch them before compilation

Since RefHw current kernel version is older than v5.13, the virtio-sound driver is not available in the kernel. In that case, patches provided in the *[virtio_snd_patches/](https://git.virtualopensystems.com/virtio-loopback/docs/-/tree/multi-sound-guide/virtio_snd_patches)* subdirectory of the current repo should be applied to the kernel sources. Please find more information about that in the next section (3.1).


## 2. Build virtio-loopback components

This section focuses on the steps need to be followed in order all the required components of the architecture be compiled.

**NOTE**: For simplicity, we will consider to execute all the operations in "$HOME/virtio-loopback-test".

### 2.1 Backport virtio-sound on kernel v5.10
 Patches which backport the virtio-sound driver on kernel v5.10 (used by AGL reference HW) can be found under "virtio_snd_patches/" of the current repo. To apply them to the target kernel (assuming you have sources in the "linux_kernel_source" folder) the following steps are required:
- `cd linux_kernel_source`
- `git am $HOME/virtio-loopback-test/virtio_loopback_guide/virtio_snd_patches/000*`
- `make sound/virtio/virtio_snd.ko`
    - For cross-compile please use: `make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- sound/virtio/virtio_snd.ko`

> **NOTE:** A **prebuilt** virtio_snd.ko module can be found under "[prebuilt_binaries](https://git.virtualopensystems.com/virtio-loopback/docs/-/tree/multi-sound-guide/prebuilt_binaries)" directory.

### 2.2 virtio-loopback driver

The loopback driver is one of two core components of the presented design. It is a kernel module responsible of re-routing virtio messages to the user-space and provides a way to user-space applications to access the vrings. The instructions about how to clone the corresponding repository and build it are the following:

Clone and build loopback driver by executing the following commands:
- `cd $HOME/virtio-loopback-test`
- `git clone git@git.virtualopensystems.com:virtio-loopback/loopback_driver.git`
- `git checkout multi-sound-demo`
- `cd loopback_driver/`
- `./make_loopback.sh` or `./make_loopback.sh 1` for enabling logs

>More in details, the `make` command (used inside the make_loopback.sh script) takes as an argument:
>- 'DEBUG=1' which defines if the driver is going to print logs.

**Note:** Please update the variable *LINUX_DIR* in the "Makefile", pointing to the target linux sources.

> A prebuilt version of loopback modules can be found under "[prebuilt_binaries](https://git.virtualopensystems.com/virtio-loopback/docs/-/tree/multi-sound-guide/prebuilt_binaries)":
>  - loopback_driver0.ko, *logs disabled*
>  - loopback_driver1.ko, *logs disabled*
>  - loopback_driver0_logs.ko
>  - loopback_driver1_logs.ko

The kernel side component of the virtio-loopback architecture is now ready to be used. Before proceeding, let's build the user-space component virtio-loopback adapter.

### 2.3 virtio-loopback adapter

The adapter is the second important component of the presented design and it is responsible of bridging the communication between the vhost-user-device (user-space) and the loopback driver (described in the previous section).

Instructions to clone and build this component are the following:
- `cd $HOME/virtio-loopback-test`
- `git clone git@git.virtualopensystems.com:virtio-loopback/adapter_app.git`
- `git checkout multi-sound-demo`
- `cd adapter_app/`
- `make ARCH=arm64`

More in details, the `make` command takes as an argument:
- 'DEBUG=1' which defines if the adapter is going to print logs.

>**NOTE:** A prebuilt version of adapter's binary can be found under "[prebuilt_binaries](https://git.virtualopensystems.com/virtio-loopback/docs/-/tree/multi-sound-guide/prebuilt_binaries)" directory:
>  - adapter, *logs disabled*
>  - adapter_logs

Next step is to compile vhost-user-sound device to be able to test the architecture.

### 2.4 Vhost-user-sound

The third component of the virtio-loopback architecture is the vhost-user-device. Vhost-user-device is the component with whom virtio driver communicates via the path "loopback driver" <-> "adapter". Essentially, it is a user-space driver which virtio tries communicate with.

Instructions to clone and build this modified version are the following:
- `cd $HOME/`
- `git clone https://github.com/rust-vmm/vhost-device.git`
- `cd ./vhost-device/staging/`
- `cargo build --bin vhost-device-sound`

>**NOTE:** A prebuilt version of vhost-device-sound can be found under "[prebuilt_binaries/vhost-device/](https://git.virtualopensystems.com/virtio-loopback/docs/-/tree/multi-sound-guide/prebuilt_binaries/vhost-device)" directory:
>  - vhost-device-sound
>  - vhost-device-sound.d
>
>Both files needs to be in the same directory

After the above process is completed we should be able to locate vhost-device-sound to the following paths:
- $HOME/vhost-device/staging/target/debug/vhost-device-sound

### 2.4 Application plays audio stream to a sound system device

In order to perform the multi-virtio-sound demo, we need a simple user-space application that plays a sound stream to a sound device. This application is "aplay" command included in the Pike rootfs with target agl-demo-platform. Also sound samples played during that demo can be found in the subdirectory *sound-samples/*.

## 3. Run the demonstration

At this point all the components required to start the demo should have been downloaded or built.

### System Initialization

This section is an example of how to run two vhost-user-sound on top of the virtio-loopback architecture. It consists of three steps:
- Insert the virtio-sound and virtio-loopback-transport drivers into the kernel
    - `insmod` the *virtio_snd.ko* binary
    - `insmod $HOME/loopback_driver/virtio-loopback0.ko`
    - `insmod $HOME/loopback_driver/virtio-loopback1.ko`
> By using insmod `virtio-loopback*_logs.ko` prebuilt modules, driver's logs can be seen with `dmesg` but this **will** affect performance.

As a result of that process we can see that two new virtio-loopback-driver has been registered in the kernel. This can be verified by running:
```
$ ls /dev/| grep "loopback-"
loopback-X
loopback-Y
```
> X and Y are used to discriminate the two virtio-loopback driver instances.

- Run the vhost-user-devices
    - `$HOME/vhost-device/staging/target/debug/vhost-device-sound --socket $HOME/sound.sock0 --backend pipewire`
    - `$HOME/vhost-device/staging/target/debug/vhost-device-sound --socket $HOME/sound.sock1 --backend pipewire`
> The "vhost-device-sound" has two arguments "`--socket-path=`" and "`--backend`". The socket argument is the connection between and instance of the vhost-user-device and the virtio-loopback-adapter. Lastly, the backend can be one of the three choices: "null", "alsa", "pipewire". The current demo focuses on "pipewire".
>
> By using `RUST_LOG=trace` you will be able to see vhost-device-sound's logs but this might affect performance.
- Start two instances of virtio-loopback-adapter
    - `$HOME/adapter_app/adapter -s /home/root/agl/sound.sock0 -d vhusnd -l X`
    - `$HOME/adapter_app/adapter -s /home/root/agl/sound.sock1 -d vhusnd -l Y`
> By using `adapter_logs` prebuilt binary, adapter's logs can be seen in the same terminal but this might affect performance.


After executing those steps, two new virtio-sound devices should appear in the system. We can verify that and find their names by running:
```
$ aplay -l
card 1: SoundCard [VirtIO SoundCard], device 0: virtio-snd [VirtIO PCM 0]      
  Subdevices: 1/1                                                              
  Subdevice #0: subdevice #0                                                   
card 2: SoundCard_1 [VirtIO SoundCard], device 0: virtio-snd [VirtIO PCM 0]    
  Subdevices: 1/1
  Subdevice #0: subdevice #0                                                   

```

### Playing audio on Speakers

All the virtio-loopback infrastructure is now in place. Every request to the virtio-sound driver in kernel space, will be forwarded through virtio-loopback driver and adapter components to the vhost-device-sound device running in user space. At last the vhost-device-sound device via "pipewire" and "ALSA" outputs the audio on speakers.

Playing audio to the speakers is done via two simple commands:
- `aplay -D sysdefault:CARD=SoundCard $HOME/sound-pacthes/agl_demo.wav`
    + Playing sound to the speakers via virtio-sound first instance (SoundCard)
- `aplay -D sysdefault:CARD=SoundCard_1 $HOME/sound-pacthes/agl_ic.wav`
    + Playing sound to the speakers via virtio-sound second instance (SoundCard_1)