Camera API
1. Add send/receive lists of file-descriptors through any stardust message
2. Create Camera Item type: pass projection, resolution, render-target (pass through file-descriptor e.g. dma-buf, supporting any pixel format)
264 Replies
glImportMemoryFdEXT or dmabuf
or maybe some third thing i didn't know of
oh and we gotta make sure the client can't ever crash the server with an invalid texture or such
very important
Should have no response from the server
https://discord.com/channels/830752623765749780/830752623765749783/1133622362789642383
https://github.com/StardustXR/core/blob/main/core/src/messenger.rs#L321
https://doc.rust-lang.org/stable/std/os/fd/struct.BorrowedFd.html
https://discord.com/channels/500028886025895936/500336333500448798/1120768446939009176
https://discord.gg/tokio
https://www.phind.com/agent?cache=clkjdbj0y002gl208lmnc1vzc
GPT-4; be wary of hallucinations.
Phind: AI Programming Assistant
Get instant answers, explanations, and examples for all of your technical questions.
seems kinda reasonable to me
also i was mistaken
we need dmabuf for this indeed
arcan didn't find an alternative
good to know
oh also the gl memory fd extension actually uses dmabuf under the hood
so we might as well just use dmabuf directly
why? the gl memory looks a lot easier to use if you're already doing gl
but we can't guarantee that
someone might be using vulkan on their client
the FD is the same
ok, but what about the formats?
what about GLES that doesn't support the external mem fd
GLES does
It's an extension for both: https://registry.khronos.org/OpenGL/extensions/EXT/EXT_external_objects_fd.txt
usable only with POSIX, of course
Some reason I thought you'd shared that to me
They use it specifically as a way to go between OpenGL and Vulkan
ooooh
hmm
so...
what about formats?
it's not sharing a texture
it's sharing a memory object
which might be a texture
?
oh well uh
so it's whatever formats your texture is
we should share a texture
yes
right but how do we sync formats
you don't have to
like, we need to make sure that this will eventually work for HDR
it's just a texture (if that's what you've shared)
i'm confused
so ok we have a texture
now
which has whatever formats that texture has, including HDR
the client has sent over the texture
HDR is just color gamut
nothing really new
ok, so on the server how do we make sure whatever format it is, it can be rnedered to
afaik
wait what, then why is it such a big deal requiring tons of protocols
point me to them?
oh huh
i guess it is just color management
ok, you know more about this than me for sure
if we can make a system that will work into the future with HDR and other new texture formats in an easy way that doesn't require a ton of back and forth
then i'll be happy
I think we should be good
On Naming: Perhaps Cameras should be physical sensors?
we can go with 'Viewpoint', since that's what SK calls it
Or Viewport, but that has another meaning in GL
(Viewpoints can have Viewports, even)
hmm
i like camera
besides, to stardust a physical vs virtual camera will be the same thing
So you could ask for access to a physical camera's stream?
no, more like you can attach a camera item that represents a physical camera to your object
Isn't that what I just said?
oh
uh
idk
ask for access implies a permission dialog
the client asking the server for access, whether it's presented to the user in a UI or not
You can't have security popups for every capability conferred
On Security: SK Viewpoints support a Render Layer filter, which could be used to control what is shown to a Virtual Camera. Unfortunately, they only support 9 layers š¦
ehh, i don't want stardust to have that anyway
what i do want is to have near/far depthbuffer clipping
is that part of the projection matrix?
I think so
yes
if anything it would be "only show these objects" but that's really difficult
because in stardust you can't really see all the objects
because that'd be privacy issues
My thought was it wouldn't be so much render layers as containers the cameras have permission to see
oh yea also
can you think of a good way to make sure that when a camera item is released out of an item acceptor
that the client who made the item acceptor can't just keep getting frames from it?
oh wait
you could just call a
frame method every frame on the item
duhhhthat's backwards
the server is rendering to the buffer
right
when it stops rendering, the client just won't get anything new
But, I don't think any of this will work if you have to 'physically' put cameras into acceptors
The 2d screen client needs to be able to boot up and configure its cameras without interaction
ok but how can i trust the 2D screen client
that's why I said to use render layers
then it can only see the objects shared with it or that it creates
how would you share objects
it's just... really complicated
well, then you have to just trust the client
also, your 2D screen client could be a camera item UI handler
in which case it would have access to the cameras
because I'm not going to reconfigure everything everytime I boot up Stardust
that's what the camera item UI handler is for
I guess I don't understand yet.
The 2D screen client creates a Camera that it wants to look through
Positions it and gives it a rendertarget
What's the User Interaction/Interface there?
if the 2D screen client is an item UI handler, there is no UI
it just does it
as long as that's possible, I'm happy
So, what is UI handler?
something that has permission to interact with system UI stuff?
it's an object that the first client to create it gets it
and it lets the client both access all items of a type anytime and acceptors
and create UIs for all the items of that type
and manages when the items are accepted into item acceptors
look at orbit for a panel UI handler
as well as flatland
so, there can only be one Camera Item UI Handler?
yea
the 2d screen manager would be the only thing that can create cameras?
uh... yea
well
you could also just have a camera item UI that you can grab cameras out of and drop them on the screens too
and your screens are camera item acceptors
which wouldn't be very hard to do
that's a no go
when stardust has persistence, it won't be an issue
the camera items will automatically be put back on after a restart
you have to have a programatic way to set it up
even for the first time
"why is my screen black?" "Oh, you have to drag this camera thing onto it" "..."
That's why I suggested render layers. You could have up to 9 camera system UI handlers, each with their own set of objects they can see
if we have only 9 then what happens when you have more
it'll just break
we'd have to fix StereoKit to allow more
also, now you have to explain to users "this camera can only see these things"
which is very challenging
it's a ton easier to say "this camera can view this far"
How were you hoping to show the camera as something you could drag into a camera item acceptor?
2 ways
1. have the camera item UI handler render a camera ghost on all acceptors
2. have that ghost automatically make a tracer to where you can summon more cameras
but that's the thing
for now we can just have your screen handler be a camera item UI handler and just get access to cameras directly
there's no problem with that
ok
I don't think you ever want to see the cameras though
the camera UI handler would hide them whenever they're accepted
there might be a client that lets you have a camera you can drag around
It feels like we'll just need to have a camera system ui handler that presents the pop-up "this app wants to use a camera; allow? remember?"
someday
and until then you can only use one client that uses cameras
You need a way to revoke the privilege too
basically, I think we've established that camera management is tricky enough it actually does need to be handled by a handler
that handler may choose to show ghosts
yep, you can grab the cameras right out of their sockets
or such
if they were visible
usually they won't be
really it just depends on the item UI handler
you can have a virtual tool for that
like a glove that lets you pick up intangible stuff
nice
'camera vision goggles'
that'd use pulse receivers and such to send a message to the UI handler to go "we want to pick this up"
or such
can also have a dumb ui-less handler that has a config file that says "this client is allowed to make and use a camera"
haha that'd be neat
not this though
i really don't like hidden config files
not hidden
~/.configs
those are hidden
remember, not everyone likes UIs
if i have to hunt in some folder
then it's hidden
I think config files can be easier to use
i def disagree
config files are the WORSTT
better than the Windows Registry
and guess what, the Windows Registry was created to store configurations for system UIs
aAAAAAAAAA
seriously though, writing things down is a good way to remember things
you can't have persistence without remembering things, and I'd much rather remember them in a way a human can deal with
hence, config files
this is a great example of object oriented code though
i'd rather just have persistence with some good UI
See, I think a text file is a good UI
You can make GUIs that work on them too
yes, GUIs that lack context spatially
storing state for user-configurable things as binary, not good
like I said, you have to store the configuration somehow. Put the good UI on top of it
Anyways, the idea was that you could have a System UI Handler Bus of sorts that lets you run multiple System UI Handlers through it
Maybe it would be a Stardust server itself š¤
please no
that sounds so convoluted
That way the Camera Handler could manage, present UI for, and remember permissions, keeping the config files outside of the core server, giving permission to the 2D Screen Manager to make however many cameras it wants as if it were a SystemUI itself, but still allowing e.g. screenshare to ask for permission each time with a good UI
that's... a LOT for 1 client
permission management and permission UI?
doesn't seem like too much
permission management delegated to a client
because you didn't want to do it in the server
You'd have to do all of that in the server if you didn't delegate it to a client
that's what the item system is for
You would have to support multiple system UI handlers in the server if you don't have the single system UI handler to handle that sort of thing
why?
well, you just said that I could only have one System UI handler that could create it's own cameras (for the screens, e.g.). What if I want another client that is able to e.g. to video recordings?
or display your ghost cameras
camera item acceptors
and when persistence is implemented in stardust, those will automatically be filled with cameras if they were before
That requires physical interaction
please don't make me have to drag a camera onto my virtual dslr everytime I want to use it
only the first time
configuration is much better than persistence
also, there's no reason your camera UI handler couldn't make it so you can just press on the camera item acceptor to place a camera in there instead of dragging it over
every time I create a fake dslr, it's the first time for that dslr
...
any sort of interaction there is going to be very annoying
look, for most people if you don't have something in their face it doesn't exist
you gotta understand that configuration without spatial context instantly is in the ether
it's in a nowhere place
another universe
etc.
use your ether-goggles then
but it's on a computer. It doesn't exist
sure, but in order to make it actually obvious to the person using it you gotta pretend like it does
map the abstract nonsense to 3D objects so people can see it
the fake DSLR in my example is that object
a ghost camera doesn't make any sense when I've got a (virtual) DSLR right here!
Why do I have to drag a ghost camera to possess my DSLR?
because you don't want some other client making a camera without your knowledge
hence a system ui handler manager that can ask you
one of which may show ghost cameras
for those people who really want to see them
that's what the camera item UI handler can do
yes
anyway, if you really want one with configuration
then you can make that
i definitely won't though :p
the thing is, the 2D screen manager needs to be able to act as a system UI handler
but I don't want it to do the permission management
for now, you can make it so it just doesn't worry about camera UI handlers at all
but then it's super easy to update later to be camera item acceptors
and your camera item UI handler to give it to them properly
@UnderSampled https://docs.rs/withfd/0.1.1/withfd/
withfd - Rust
withfd allows passing file descriptors through Unix sockets.UnixStream in tokio::net - Rust
A structure representing a connected Unix socket.
Vectored I/O
In computing, vectored I/O, also known as scatter/gather I/O, is a method of input and output by which a single procedure call sequentially reads data from multiple buffers and writes it to a single data stream (gather), or reads data from a data stream and writes it to multiple buffers (scatter), as defined in a vector of buffers. Scatter/gathe...
@Nova The message header we are adding the FDs into has a 'scatter/gather array' in it
@Nova Sorry for the confusion I made. There is no message header actually sent with iovecs: iovecs tell the system-call where to read or write the data to/from
the 'header' I thought I understood was actually just a structure to provide more parameters to the system call
it does allow you to split up the bytes you read into multiple buffers, but they're all fixed
so if you want to stream the data with a variable length, I think you'll have to continue doing it as you do
The other option is to choose a max length, and use two iovecs to split it into the length and the content. The advantage of this is that you can reuse the same buffers each time, which could be faster
@Nova PR available! https://github.com/StardustXR/core/pull/14
GitHub
Add Support for sending/recieving file-descriptors in the core mess...
This is untested, but it compiles.
I was not able to use sendFd because it didn't do async, but I was able to learn from it's source how it worked with Tokio and was able to do that myself as well
Originally, I had planned to include the FDs when the header is sent, but moved it to the body so that the number of FDs could be sent in the header too
the number to receive is limited by the OS; modern Linux is limited to 253 FDS
withFd just always made the buffer for receiving the FDs big enough to receive all 253, so I could have done it that way instead and used the suggested
cmsg_space macro (I had to reimplement it to accept a variable number of expected FDs, but it was only a few lines)
I brought the FDs all the way up to the scenegraph and mostly stopped there, thinking you'll want to take over from there. I also recommend turning the method response return value into a tuple (Vec<u8>, Vec<RawFd>)
Also: You have to send some data in the body if you're sending an FD. If that doesn't work for you, we can switch it back to the header message and fixed 253, since the header always has something.
I did not test it at all, but it did compile.@UnderSampled what data do we have to send in the body?
It just can't be null
Sorry, I should have added a check
@UnderSampled making a type safe api for this with
OwnedFd
so you actually send owned fds to the other sideI wondered if you might š
@Nova how's it working out?
haven't fully updated everything yet but you should take a look at the API
see if i missed anything
check core main branch @UnderSampled
make sure my owned fd logic is sound?
I think, even though they should be the same size (being as they are the same data), it isn't good to use
size_of::<OwnedFd>() for the csmghdr buffer
I could be convinced otherwise
but the libc function is outputting raw fds into that bufferidk where that was set
but... yea
idk, anything you want to fix?
199
still reading the rest
but it's recieving rawfds, not owned ones
well yea, doesn't it have to recieve raw fds?
instead of owned ones?
yes
anyway, i figure you know more than me... i'll integrate this into the server
test and hope it works
if it does, we'll need to update... all the clients right?
since the message header is double length breaking the spec?
should be
oh oopsies
should i edit that?
yes
I'm still reading the rest
and, yes. I wasn't really thinking about compatibility...
if you don't want to change the header length, we can have a fixed buffer for the cmsgs
orrr we could add it to the end of the message struct... right?
hmm maybe not actually
no
wait why do we need a fixed buffer?
well, you could, but you'd have to do a third message for any fds
couldn't we just have like
idk
what even is a message header anyway????
like, what are cmsgs??
some side channel of communication?
no
well, sorta?
I think the data probably comes through the same channel, but it's parsed out
anyways, the cmsgs are just places for it to put the extra data
like the iovecs are where you tell it to put the data it's reading off the socket
can you shove... any data through the socket?
you can shove any data through the socket, yess
like i mean in cmsgs
no
remember, the fds aren't normal data, because when you tell the kernel to send it on the socket, it creates a new fd for the other process
so it needs this special way to send it
so we can't just stuff the count through the cmsgs on the length
shoot :((
so, the way withfd recieves them, is it creates a buffer that can accept up to the maximum 253 (which is the max supported by the linux kernel)
if you don't receive that many, that's fine, it will just be a bigger buffer than necessary
253*4 bytes
so, a 1k buffer. woo
not necessarily great to keep allocating it each time though
to be efficient, you'd just reuse the same buffer, probably
though maybe that's too much optimization
stuff like that matters more on e.g. microcontrollers
Would you like me to just allocate 1k each time, and see what fds we get?
mind making a benchmark?
or is that too much?
you don't have any tests
i guess
How important is compatibility at the moment?
i guess not much
If we do the fixed max 1k buffer, this
becomes
and it wouldn't have the
if for whether to handle fds or not
I'd also probably send them when sending the header. That was my original plan, which I switched around when I tried to make it dynamic
in theory, instead of 253, it should be SCM_MAX_FD, but I can't figure where that is availableidk, what do you think is best?
probably keeping compatibility
@UnderSampled cool! i can give you access to the repo if you like to help fix that
i'm really tired right now
okay
ok @UnderSampled invited you
sweet!
The core send/recieve should be quite different, and the header will be smaller again (I'll keep the struct though), but everything else should be about the same
you can start using FDs in things now š
or, maybe writing tests to see if you can š
so, backwards compatibility or not?
it should be backwards compatible, I'm just leaving the code for fancy headers in place
if you want to break compatibility now and leave extra room for future stuff in the header, now's your chance
š
(I can't think of what might be in the header besides length though, if we're not putting fd-count)
yea i think that's good
like, what we have now
i mean, what other data can you even stuff in a header??
often things like recipient/channel, if you're sharing a stream with something else
error correction
TCP has all sorts of stuff
do you want me not to keep the Header struct?
idk tbh
this is a part of the code i really just want to get fixed and never touch again :p
I noticed that you probably used an autoreformatter on some things
So I figured I'd run it too
I'm not sure it came up with the same results?
oh, it's probably because I removed an indent
cargo fmt aka rustfmthmm
I see it in GitHub, but when I go to push locally it doesn't like it
@Nova I have it back in my branch
oof lemme see
i sent you an invite now @UnderSampled
k
Did you want to review?
I left it sending the fds on the body message, since I didn't have to change the code around as much, and if you ever do need to add the header stuff back you don't have to change it back again
(technically, its just writing/reading it with the same call as the body is added to the stream; there are no 'packets')
oh weeeird
what?
... I literally just pushed
oh
well uh
ok
thank you!
is it all good?
(besides never being tested)
haha yea looks good so far
š
:3
So, what's next?
well, i JUST got the server syntax errors fixed
so time to test!!!
VERY GOOD SIGN @UnderSampled
i just loaded it up and BAM
woah!
works as normal
backwards compatible indeed!
I mean, we haven't successfully tried sending FDs
but, I didn't break it!
woot!
i'll save that for the camera item
yeah
good news is that this feature can go on the server and client independently
yeah, old clients will still work
and most of them won't need the feature
new ones will still be checking for FDs with every message
e.g. with a buffer to catch them š
fixing versions and uploading
is that a problem?
will it slow it down enough to matter?
no
the compiler might even notice that it's going to be used a lot and prepare for that
being the same size each time will actually help with that
oh yea rust's whole design is STUPID EFFICIENT
- Add send/receive lists of file-descriptors through any stardust message
- Create Camera Item type: pass projection, resolution, render-target (pass through file-descriptor e.g. dma-buf, supporting any pixel format)
well, not quite done with send/recieve lists
we have to actually upload it :p
well, nothing uses them now, so how would you know?
we won't know until we either write a unit-test or create a Camera Item
i have faith in your code and the power of rust
š¬
we shall see
still, what do we need to do to make a Camera Item?
ok so! we wanted to do the variable length buffers right?
we need to figure out how to:
a. use dmabufs at all in rust
b. render to a dmabuf texture that could be... basically any format with stereokit render_to
I thought we were just going to use https://registry.khronos.org/OpenGL/extensions/EXT/EXT_external_objects.txt
followed by https://registry.khronos.org/OpenGL/extensions/EXT/EXT_external_objects_fd.txt
(the first for exporting, the second for importing)
it does use dmabuf under the hood
which would mean we don't have to figure out dmabuf, yeah?
maybe... can you help figure out how you negotiate formats with this?
and if you can use the same fd with dmabuf?
i just don't want this to become an issue later
why does it have to work with dmabuf?
because dmabuf gives finer grained control that may be necessary for some applications
i want to avoid this
even if we have to use dmabuf, I don't think that's the first thing to tackle
first, we need the Camera Item to exist to start playing with it
fair fair
it does seem like DMA-buf might be better documented. The external_objects thing is more focused on vulkan->opengl stuff
Blaztinn's Blog
Inter-Process Texture Sharing with DMA-BUF
How to share textures across processes on Linux.
so, we can use dma buf, sure š
still need a camera first
right ok so i'm just publishing the 2 client crates
needed to get the server updated
lol
now there's a type name š
PFNEGLEXPORTDMABUFIMAGEQUERYMESAPROCoh god
PFN EGL export DMABUF image query Mesa Process?
uhhh idk
it's just a place to load the function from the extension
I don't think we'll see that in Rust
yea... where do we even deal with this in rust
do you do anything with GL at the moment?
(besides use SterepKit)
presumably something for panels?
i mean... barely
smithay does most of it
but yea smithay has GL bindings... and i guess needs to use dmabuf!
I can look at their code š
we might actually be able to just reuse it
GitHub
smithay/src/backend/renderer/gles/mod.rs at b77ef9a7ee655ad021004ab...
A smithy for rusty wayland compositors. Contribute to Smithay/smithay development by creating an account on GitHub.
@UnderSampled made the camera item... should it be an immediate mode API or not?
cool!
What do you mean?
like call
frame() on it every frame
and feed it a dmabufhmm
and it'll render into it then return a success or such value
actually wait wait
no multiple dmabufs is a bad idea...
hmm
so, judging from https://blaztinn.gitlab.io/post/dmabuf-texture-sharing/, I don't think we'll need multiple
I'm hopeful
right, but we might want multiple
i might just ask the arcan people
k
the question is really just about if screen-tearing or access management are things we need to worry about
if dmabuf handles that (and I think it does?) we don't need to worry about it, and one is enough
but
I think RenderTo has to be called each frame
We'll have to upgrade StereoKit if we want a RenderTo equivalent that stays around
screen tearing isn't handled by dmabuf at all
access management is done by the client giving the server the dmabuf it should render into
so... screen tearing is what happens when you swap a texture right in the middle of rendering to a newer one
or you change content in a CPU buffer or such
that can be avoided by having multiple buffers, drawing into 1, then swapping them and drawing into another, at the cost of a teeeny bit of latency
I don't think so
what do you mean?
from https://www.kernel.org/doc/html/latest/driver-api/dma-buf.html
The three main components of this are: (1) dma-buf, representing a sg_table and exposed to userspace as a file descriptor to allow passing between devices, (2) fence, which provides a mechanism to signal when one device has finished access, and (3) reservation, which manages the shared or exclusive fence(s) associated with the buffer.
fence being the functionality in question
both devices have to be able to use the same buffer. just because you sent it to the other over an FD doesn't mean they don't both still need it
they just can't use it at the same time
(without screen tearing or maybe worse things)it wouldn't be worse
yes, it easily culd be
how? it's not changing in terms of allocation
knowing which texture to use when?
remember, the client is running at a different framerate
but, I go back to this: RenderTo has to be called each frame
if that logic runs in the CameraItem, then it needs to receive the frame signal
no ok here's how it would work
when the client calls
frame() on the camera item it just saves your settings into the camera item for the next xr frame on the server, renders to the dmabuf with those settings, then will send the method return backCould you say that again, defining 'you'?
And 'next time it can render'?
I don't know those parts well enough to understand who's talking to it when, and how it knows about when it can render
ok yea did thata
That sounds good š
After some review, it seems that Smithay exposes it's dmabuf utilities as a toolbox specifically for this sort of use case š
Not only that, but it also provides the tools to make the GL contexts, swapchains, create windows, or even work with DRM and logind!
yep
basically, everything needed to make the client š
okay
well, hopefully we can work out a double buffer that makes sense then
or maybe your last solution as a sort of semaphore will be enough
š¤
idk what a semaphore is
Semaphore (programming)
In computer science, a semaphore is a variable or abstract data type used to control access to a common resource by multiple threads and avoid critical section problems in a concurrent system such as a multitasking operating system. Semaphores are a type of synchronization primitive. A trivial semaphore is a plain variable that is changed (for e...
a message that says "your turn"
(or "my turn")
ahh
Next steps:
1. Make initial client screen manager with: https://docs.rs/smithay/latest/smithay/backend/winit/index.html
2. Add buffer format capability info request method
3. Create and export a pair of buffers from screen manager renderer into dmabufs
4. Add camera setup/update method to Import the dmabufs on the server into render targets compatible with glBindTexture and change transforms.
5. Add camera render method with preloaded-buffer index as target (e.g. render camera to buffer 1)
6. In client: on frame event, present frame n, then ask for render to
n=(n+1)%bufferCountsmithay::backend::winit - Rust
Implementation of backend traits for types provided by
winitthat sounds a lot more complicated :S
Than what?
the idea i had of it
i'm not saying i know a better way
just... :((
If you'd like, we can start with virtual screens:
1. Add a method to create a panel with a render target compatible with glBindTexture (or create one for an existing panel? Panel shell?) and export a secure ref for it to a client.
2. Add camera setup/update method to choose buffers (provide the single texture-ref) and change transforms.
3. Add camera render method with preloaded-buffer index as target (e.g. render camera to buffer 1)
4. In client: on frame event, ask for render to buffer 1
huh... maybe
security might be an issue with that though
Agreed, however consider that the same thing can be done with dmabufs
The trick that the texture ref sent over to the client should only be usable by that client
You could make it a FD š
that's the idea
Not dmabufs, just using FDs to make sure only thay process has access to it
To save the step of reimporting the dmabuf to a rendertarget when it comes back to the server
(Since the rendertarget texture already exists there)
@Nova I think I'm at a place where doing some pair programming to get your rust knowledge would help me immensely
I'm free the rest of the evening
i'm pretty fatigued right now, but idk how i can make that better
so whatever, might as well just do what i normally would
sleep won't fix this kind of fatigue
how about in 30min?
I just took a long nap š
30 mins should be fine
though I'll have to get food sometime around then
Another thing that can help with fatigue is exercise, though I'm not a good role-model there
hi @UnderSampled
hey š
vc?