C#•13mo ago

Avalonia app with WASM-based UI-plugins; looking to bounce some ideas around

Hey everyone.

I'm currently working on a plugin system as a side project to my main Avalonia app. The plugins are meant to be compiled WASM core modules and my app will consume them. The idea is to provide a sandboxed environment that any language (that compiles valid WASM core modules) can interact with. This is to allow developers some freedoms in what language they can choose to build the plugins.

Interop between the app and the plugin happens via Extism, a WASM plugin<->host service available for many languages. It allows for the following:

Sending packages from and to the plugin
- Extism can allocate a block of memory to place the data in, which is then read by the other side
- This can include any JSON-serialized object, which is then deserialized on the other side
- I have not found (or not checked enough for) a way to send unmanaged structs without serializing
Plugins can import and export unmanaged methods
- The app can define unmanaged callbacks, which the plugin can then
```
DllImport
```
```
DllImport
```
  and call at any time (with some limited arguments)
- The plugin must define some expected exports (
```
[UnmanagedCallersOnly]
```
```
[UnmanagedCallersOnly]
```
  ) that the app will then call to get certain data

These plugins should now gain the ability to draw on and update the UI of my app.
For some context on that: each plugin gets a dedicated slot on my app. One whole grid row that spans the entire width of the app. The height is determined by the content that the plugin wants to present (or a fixed height, if provided).

eroOP•11/25/24, 10:06 PM

As an example: I will have a "Title" plugin, which will display some information about the app's current state. It may include just one line, or two, or other arbitrary information;

|                          |
|     "Current Title"      |
|                          |

|                          |
|     "Current Title"      |
|                          |

|                          |
|     "Current Title"      |
| "Additional Information" |
|                          |

|                          |
|     "Current Title"      |
| "Additional Information" |
|                          |

|                          |
| "Current Title"      0/0 |
| "Additional Information" |
|                          |

|                          |
| "Current Title"      0/0 |
| "Additional Information" |
|                          |

My question is the following:
How can I design a good API that allows users to build this UI from their plugin simply by sending some JSON-serialized data containing the UI information?
How can I provide a good API that allows users to update single cells within their UI? The updates must happen often (at least 60 times a second), and serializing a new object doesn't sound ideal for that.

Klarth•11/25/24, 10:47 PM

This is a cursed problem if I ever saw one.

eroOP•11/25/24, 10:59 PM

:yep:

Klarth•11/25/24, 11:01 PM

Trying to serialize state and synchronize it with the visual tree will be a nightmare.

Klarth•11/25/24, 11:02 PM

It's hard enough to do well as a one-time thing, let alone 60 fps, let alone being effectively defined from WASI.

Klarth•11/25/24, 11:02 PM

High freq updates across your app is for immediate mode renderers, not retained mode renderers like Avalonia/WPF/UWP.

eroOP•11/25/24, 11:13 PM

My idea there was to have a separate thread for each plugin. Is something like that feasible?

Klarth•11/25/24, 11:35 PM

Avalonia, like most other GUI frameworks, is single threaded for updates. There's a second thread for rendering.

Klarth•11/25/24, 11:36 PM

So you would have to marshal all of your updates onto the UI thread.

Klarth•11/25/24, 11:37 PM

You're also really thread-limited in WASM. I'm not sure if each plugin being its own thing bypasses that limit.

Klarth•11/25/24, 11:38 PM

Async isn't great in .NET WASM either. I think they're aiming for .NET 10 for "good" support although technically there already is some degree of support for the past couple versions.

Klarth•11/25/24, 11:39 PM

Though I guess at this point, your plugins could be written in anything since it's all interop via serialization.

KKlarth You're also really thread-limited in WASM. I'm not sure if each plugin being its...

eroOP•11/25/24, 11:39 PM

The app isn't WASM, if that's your concern.

eroOP•11/25/24, 11:40 PM

The app is plain ol .NET, where I should be able to have a thread per plugin just fine.

Eero The app isn't WASM, if that's your concern.

Klarth•11/25/24, 11:42 PM

I was worried about there being some sort of singular WASI host that may have some thread limit. I'm not familiar with that area though.

eroOP•11/25/24, 11:44 PM

Not sure it works that way with Extism

KKlarth Avalonia, like most other GUI frameworks, is single threaded for updates. There'...

eroOP•11/26/24, 11:05 AM

The "idea" was to send the information to the plugin's viewmodel, which would then update on the UI via bindings. I imagine this may lead to strange behavior if there are many plugins doing the work, ending up with the UI struggling to keep up?

Klarth•11/26/24, 11:07 AM

Sure, all of the updates have to occur on the UI thread as I mentioned before so there's a bottleneck there.

Klarth•11/26/24, 11:07 AM

By "update", I mean setting of a binded property or modifying of a binded collection.

Klarth•11/26/24, 11:08 AM

Since that necessarily triggers the UI to update (which happens on the same thread).

eroOP•11/26/24, 11:09 AM

well i mean 1/60 is over 16 ms, which i feel like is plenty of time for a lot of updates...

eroOP•11/26/24, 11:10 AM

but it can't work forever

eroOP•11/26/24, 11:10 AM

well either way i'm not really keen on changing my approach away from wasm

eroOP•11/26/24, 11:11 AM

so i'd just like to brainstorm the api, data sent, how to receive it, and how to convert it in a way that avalonia can render it

Klarth•11/26/24, 11:11 AM

It depends on what you're updating, how much you're updating, and how much an exact 60 fps matters without frame drops.

Klarth•11/26/24, 11:12 AM

As I said, retained mode renderers aren't your friend if you need locked 60fps. They're made for software apps on business machines, not for games.

eroOP•11/26/24, 11:13 AM

it doesn't need to be perfectly 60fps, it's not really a game. i'm fine with having some PeriodicTimers that skip a frame if we're not ready for the next one yet

eroOP•11/26/24, 11:15 AM

i would like to allow the user to choose their refresh rate on the app, can be 1hz, can be synced to the monitor refresh rate

eroOP•11/26/24, 11:15 AM

maybe even on a per-plugin basis

Klarth•11/26/24, 11:16 AM

PeriodicTimer isn't going to be a friend. Your updates need to happen on the UI thread.

Klarth•11/26/24, 11:16 AM

So you're either using

DispatcherTimer

DispatcherTimer

TopLevel.RequestAnimationFrame

TopLevel.RequestAnimationFrame

to tie into the compositor's frame rate.

Klarth•11/26/24, 11:16 AM

The latter being better, but harder to use.

KKlarth PeriodicTimer isn't going to be a friend. Your updates need to happen on the UI ...

eroOP•11/26/24, 11:21 AM

my thinking was that i can dispatch the data received on the plugin thread to a viewmodel on the main thread?

eroOP•11/26/24, 11:22 AM

and then avalonia can do whatever with it?

Klarth•11/26/24, 11:27 AM

Dispatch the data? So then you're back to serialization? Or at least passing objects and performing state synchronization.

eroOP•11/26/24, 11:27 AM

the latter was my thinking

Klarth•11/26/24, 11:29 AM

I'm not even sure if the latter is possible. Are your plugins not going to define their own types/schema on the WASI side? So how is the C# / Avalonia side going to have a properly typed ViewModel to explicitly synchronize properties with?

KKlarth I'm not even sure if the latter is possible. Are your plugins not going to defin...

eroOP•11/26/24, 11:32 AM

well, the idea was to have a unified api. otherwise, i'm not sure how i would even reveice the data.

eroOP•11/26/24, 11:33 AM

unified schemas that can be deserialized back into concrete types on the app side

eroOP•11/26/24, 11:37 AM

so something like

{
  cells: [
    [
      {
        "content": "Current Title",
        "horizontalAlignment": "center"
      }
    ],
    [
      {
        "content": "Additional Information",
        "horizontalAlignment": "center"
      }
    ]
  ]
}

{
  cells: [
    [
      {
        "content": "Current Title",
        "horizontalAlignment": "center"
      }
    ],
    [
      {
        "content": "Additional Information",
        "horizontalAlignment": "center"
      }
    ]
  ]
}