C
C#7d ago
ero

Avalonia app with WASM-based UI-plugins; looking to bounce some ideas around

Hey everyone. I'm currently working on a plugin system as a side project to my main Avalonia app. The plugins are meant to be compiled WASM core modules and my app will consume them. The idea is to provide a sandboxed environment that any language (that compiles valid WASM core modules) can interact with. This is to allow developers some freedoms in what language they can choose to build the plugins. Interop between the app and the plugin happens via Extism, a WASM plugin<->host service available for many languages. It allows for the following: 1. Sending packages from and to the plugin - Extism can allocate a block of memory to place the data in, which is then read by the other side - This can include any JSON-serialized object, which is then deserialized on the other side - I have not found (or not checked enough for) a way to send unmanaged structs without serializing 2. Plugins can import and export unmanaged methods - The app can define unmanaged callbacks, which the plugin can then DllImport and call at any time (with some limited arguments) - The plugin must define some expected exports ([UnmanagedCallersOnly]) that the app will then call to get certain data These plugins should now gain the ability to draw on and update the UI of my app. For some context on that: each plugin gets a dedicated slot on my app. One whole grid row that spans the entire width of the app. The height is determined by the content that the plugin wants to present (or a fixed height, if provided).
25 Replies
ero
eroOP7d ago
As an example: I will have a "Title" plugin, which will display some information about the app's current state. It may include just one line, or two, or other arbitrary information;
| |
| "Current Title" |
| |
| |
| "Current Title" |
| |
| |
| "Current Title" |
| "Additional Information" |
| |
| |
| "Current Title" |
| "Additional Information" |
| |
| |
| "Current Title" 0/0 |
| "Additional Information" |
| |
| |
| "Current Title" 0/0 |
| "Additional Information" |
| |
My question is the following: How can I design a good API that allows users to build this UI from their plugin simply by sending some JSON-serialized data containing the UI information? How can I provide a good API that allows users to update single cells within their UI? The updates must happen often (at least 60 times a second), and serializing a new object doesn't sound ideal for that.
Klarth
Klarth7d ago
This is a cursed problem if I ever saw one.
ero
eroOP7d ago
:yep:
Klarth
Klarth7d ago
Trying to serialize state and synchronize it with the visual tree will be a nightmare. It's hard enough to do well as a one-time thing, let alone 60 fps, let alone being effectively defined from WASI. High freq updates across your app is for immediate mode renderers, not retained mode renderers like Avalonia/WPF/UWP.
ero
eroOP7d ago
My idea there was to have a separate thread for each plugin. Is something like that feasible?
Klarth
Klarth7d ago
Avalonia, like most other GUI frameworks, is single threaded for updates. There's a second thread for rendering. So you would have to marshal all of your updates onto the UI thread. You're also really thread-limited in WASM. I'm not sure if each plugin being its own thing bypasses that limit. Async isn't great in .NET WASM either. I think they're aiming for .NET 10 for "good" support although technically there already is some degree of support for the past couple versions. Though I guess at this point, your plugins could be written in anything since it's all interop via serialization. 🤷‍♂️
ero
eroOP7d ago
The app isn't WASM, if that's your concern. The app is plain ol .NET, where I should be able to have a thread per plugin just fine.
Klarth
Klarth7d ago
I was worried about there being some sort of singular WASI host that may have some thread limit. I'm not familiar with that area though.
ero
eroOP6d ago
Not sure it works that way with Extism The "idea" was to send the information to the plugin's viewmodel, which would then update on the UI via bindings. I imagine this may lead to strange behavior if there are many plugins doing the work, ending up with the UI struggling to keep up?
Klarth
Klarth6d ago
Sure, all of the updates have to occur on the UI thread as I mentioned before so there's a bottleneck there. By "update", I mean setting of a binded property or modifying of a binded collection. Since that necessarily triggers the UI to update (which happens on the same thread).
ero
eroOP6d ago
well i mean 1/60 is over 16 ms, which i feel like is plenty of time for a lot of updates... but it can't work forever well either way i'm not really keen on changing my approach away from wasm so i'd just like to brainstorm the api, data sent, how to receive it, and how to convert it in a way that avalonia can render it
Klarth
Klarth6d ago
It depends on what you're updating, how much you're updating, and how much an exact 60 fps matters without frame drops. As I said, retained mode renderers aren't your friend if you need locked 60fps. They're made for software apps on business machines, not for games.
ero
eroOP6d ago
it doesn't need to be perfectly 60fps, it's not really a game. i'm fine with having some PeriodicTimers that skip a frame if we're not ready for the next one yet i would like to allow the user to choose their refresh rate on the app, can be 1hz, can be synced to the monitor refresh rate maybe even on a per-plugin basis
Klarth
Klarth6d ago
PeriodicTimer isn't going to be a friend. Your updates need to happen on the UI thread. So you're either using DispatcherTimer or TopLevel.RequestAnimationFrame to tie into the compositor's frame rate. The latter being better, but harder to use.
ero
eroOP6d ago
my thinking was that i can dispatch the data received on the plugin thread to a viewmodel on the main thread? and then avalonia can do whatever with it?
Klarth
Klarth6d ago
Dispatch the data? So then you're back to serialization? Or at least passing objects and performing state synchronization.
ero
eroOP6d ago
the latter was my thinking
Klarth
Klarth6d ago
I'm not even sure if the latter is possible. Are your plugins not going to define their own types/schema on the WASI side? So how is the C# / Avalonia side going to have a properly typed ViewModel to explicitly synchronize properties with?
ero
eroOP6d ago
well, the idea was to have a unified api. otherwise, i'm not sure how i would even reveice the data. unified schemas that can be deserialized back into concrete types on the app side so something like
{
cells: [
[
{
"content": "Current Title",
"horizontalAlignment": "center"
}
],
[
{
"content": "Additional Information",
"horizontalAlignment": "center"
}
]
]
}
{
cells: [
[
{
"content": "Current Title",
"horizontalAlignment": "center"
}
],
[
{
"content": "Additional Information",
"horizontalAlignment": "center"
}
]
]
}
Klarth
Klarth6d ago
This is just a pretty awful idea all around.
ero
eroOP6d ago
everything for the user :yep:
Klarth
Klarth6d ago
The implementation is going to be very hard, IMO.
ero
eroOP6d ago
yup! which is really why i made this thread, i need all the ideas i can get 😅
Klarth
Klarth6d ago
If there's no bidirectional state transferred, then a WebView with iframes could be better and just let the plugin render themself.
ero
eroOP5d ago
there is... the app has some info that the plugins will need to access. which i now realize will have even more problems with synchronization hmm. maybe i can use a handle sort of system. create "handles" for cells and pass those handles when setting the cell's content
Want results from more Discord servers?
Add your server