Open Brush, Open Blocks, Icosa Gallery•16mo ago

btw. to explain https://xrfragment.org from a icosagallery/ `.tilt`-fileformat perspective: `.tilt

btw. to explain https://xrfragment.org from a icosagallery/

.tilt

.tilt

-fileformat perspective:

.tilt

.tilt

is an (open) 3D file(format) for XR experiences.
XR Fragments (XRF) spec is about (implicit+explicit) metadata (found in a 3D file) which hints the viewer-application (icosagallery e.g.) to enhance XR experiences, for example:

navigate in-VR to other 3D file by make objects with URL-metada clickable
load/project video on a plane with videoURL-metadata
switch camera/position/animation by clicking an objectIn theory, the XRF js-parser can be included in THREE.js to leverage those features (it just scans the `.userData`-fields of a loaded `.tilt|glb|obj` 3D file).(I'm writing an NLnet proposal atm, and am thinking of making a XRF>Icosagallery PR a milestone)

Lleon_van_kammen#4615 btw. to explain https://xrfragment.org from a icosagallery/ `.tilt`-fileformat p...

andybak•9/18/24, 7:09 PM

OK. I think there's an important distinction between "editing a scene" and "viewing/interacting with a scene". We do try and cover both in Open Brush but the lines are a bit blurry and the "view only" side of things is a bit unfinished

andybak•9/18/24, 7:10 PM

I think the Icosa Gallery is a better target for everything related to "Viewing/Interacting with" - I'm currently wondering if it's worth incorprating parts of Spoke (the Mozilla Hubs scene composer/editor) or the three.js editor into Icosa.

andybak•9/18/24, 7:11 PM

However - "marking up" a sketch in Open Brush so that extra capabilities are enabled when viewing is totally in scope.

leon_van_kammen#4615OP•9/18/24, 7:22 PM

These thoughts make sense.
I actually studied Spoke & Three.js editor during drafting the XR Fragments spec.
The idea was to create a fileformat-agnostic version (first I wanted to bet on Spoke, until it got abandoned by mozilla).
The texturescrolling- and link-features (of XR Fragments) were inspired by Spoke extensions, and can be used by other fileformats too now.
The deeplinking/teleporting-via-URL aspects of XR Fragments were something I've always longed for with Spoke/Hubs.

leon_van_kammen#4615OP•9/18/24, 7:27 PM

Btw. is there a drop-in THREEjs viewer-library of

.tilt

.tilt

files?
I'd love to add support for

.tilt

.tilt

-files on https://xrfragment.org
The fact that it is an open portable format is important for XR Fragments to promote.

Lleon_van_kammen#4615 Btw. is there a drop-in THREEjs viewer-library of `.tilt` files? I'd love to add...

andybak•9/18/24, 8:35 PM

We don't really intend tilt files themselves to be portable. We would rather put effort into making our gltf exports portable.

andybak•9/18/24, 8:36 PM

There is a tilt viewer for threejs but it's very rudimentary. There is far too much magic baked into our brushes for tilt files to be easily rendered outside of the app.

andybak•9/18/24, 8:37 PM

But the gltf files can have enough metadata to be rendered with varying levels of fidelity across many apps.

leon_van_kammen#4615OP•9/19/24, 8:13 AM

ah like progressive enhancement

leon_van_kammen#4615OP•9/19/24, 8:15 AM

indeed, I noticed a very cool scene in icosagallery which had some sort of illuminating fog/nebula.
I figured that perhaps some sort of shadermagic was going on.

Lleon_van_kammen#4615 indeed, I noticed a very cool scene in icosagallery which had some sort of illum...

andybak•9/19/24, 11:37 AM

Yep. Currently only the more basic brushes look correct in other apps (unless they can directly use our Unity or WebGL shaders).

I am trying to improve the export (with the help of @Nick ) to get closer to what is theoretically possible. The lack of additive blending in GLTF is a hindrance but I want to propose an extension to cover this and try and get a few other apps to implement it.

leon_van_kammen#4615OP•9/19/24, 12:29 PM

Blending modes in GLTF would enable so much more possibilities.
I wonder if .usdz (which Apple seems to bet on) has anything for that.
Upload works btw: https://icosa.gallery/view/coderofsalvation/rJ6NfwHhWlo

icosa

leon_van_kammen#4615OP•9/19/24, 12:33 PM

btw. once icosagallery has a search-feature, I can implement a searx-plugin for it, to include in this opensource WebXR searchengine (which I also maintain) : https://searxr.me/

andybak•9/19/24, 2:57 PM

Bear in mind that https://icosa.gallery/ hasn't been touched in several years - the new site is still in development although we're fairly close to launching a beta

andybak•9/19/24, 2:58 PM

We are going to implement oEmbed as it's fairly simple. Activitypub is something we're looking into but in general I'm interested in supporting as many open formats and protocols as possible.

andybak•9/19/24, 2:58 PM

So - what would a minimal integration with XR fragments look like? What's the starting point?

leon_van_kammen#4615OP•9/20/24, 6:24 PM

A minimal integration could be done on various levels:

**EASY:** render an AFRAME div-playerframe for non-`.tilt` uploaded assets (basically .glb/.usdz etc) with the XR Fragments AFRAME plugin (https://xrfragment.org/#%F0%9F%A7%B0%20AFRAME) like in this codepen: https://codepen.io/coderofsalvation/pen/yLwedvX**Pros**: drop-in solution including VR/AR & hand-controls (I can do an example PR next week)**Cons**: no cons
**LESS EASY:** integrate the XRF THREE-plugin into the icosa-viewercode: https://xrfragment.org/#%F0%9F%A7%B0%20THREE.js which adds metadata-listeners onto THREE loaders (and enhances objects accordingly, making them clickable e.g.).**Pros**: would enable XRF-features/metadata for `.tilt` assets too**Cons**: extra layer of complexity, no hand-control interactions, no AR-mode
**HARDCORE**: integrate the XRF parser-library into the icosa-viewer, which only scans/parses/validates the metadata to aid further implementation: https://xrfragment.org/#%F0%9F%94%A8%20XR%20Fragments%20parser

andybak•9/20/24, 6:35 PM

we're pretty wedded to our threejs based viewer at the moment - although it's pretty lacking in XR navigation features.

A transition to A-Frame is interesting but I'm not sure how much work it would be. Is this something you might want to help with? We could put some of the NLNet funding towards it, if that would help.

Or at least maybe advise on - a few hours consultancy might save me a ton of time.

leon_van_kammen#4615OP•9/20/24, 6:42 PM

yes that's definately possible.
TBH it's an interesting conversation to have.
AFRAME is basically THREE.js with batteries-included XR goodies.
Last year, I had no time to get into the nitty gritty of THREE.js XR controls/navigation/local teleporting e.g., so I quickly wrapped the XRF THREE.js-code into an AFRAME component.
I don't know if much changed, but last year I could not really find community-maintained XR goodies for THREE.js (other than AFRAME)

andybak•9/20/24, 6:44 PM

No - I was a bit shocked how little XR character controller stuff was available for vanilla threejs.

andybak•9/20/24, 6:45 PM

i could probably just do a simple "fly" controller - but getting teleport right is fiddly.

mikesky•9/20/24, 6:45 PM

Personally I’d prefer to stick at threejs level

andybak•9/20/24, 6:48 PM

maybe the next batch of nlnet funding could be used to write a decent XR nav component.

andybak•9/20/24, 6:48 PM

it's mainly about making teleport non-ugly and not complex

andybak•9/20/24, 6:48 PM

the old Steam implementation was nearly perfect.

leon_van_kammen#4615OP•9/20/24, 7:17 PM

Right. So the teleport-via-button-clicks are already implemented by the three.js-plugin.
When a model gets parsed by any THREE.js-loader, it will:

scan for

someobject.userData.href

someobject.userData.href

occurences (with value

#platformB

#platformB

https://icosa.gallery/john/l2k34.glb

https://icosa.gallery/john/l2k34.glb

e.g.)

emit a 'click' event for that object (currently mouse and XRController)
setup basic click-listener (overridable) which sets the camera location to objectname
```
#platformB
```
```
#platformB
```
or replace the current scene with
```
https://icosa.gallery/john/l2k34.glb
```
```
https://icosa.gallery/john/l2k34.glb
```
it adds a hashtag to the current url so the location can be shared: https://icosa.gallery/view/coderofsalvation/wHhWlo#platformB

These teleports are already working, and might already be enough (especially for roomscale VR and gaze-interactions).
I mean I used https://github.com/jure/aframe-blink-controls for free-form teleporting, but this is not really needed for most experiences (which I usually guide with fixed teleporting points anyways).

andybak•9/20/24, 7:19 PM

It would be nice to offer both.

leon_van_kammen#4615OP•9/20/24, 7:20 PM

haha @andybak your avatar makes me smile with every sentence you write

leon_van_kammen#4615OP•9/20/24, 7:20 PM

isn't that the scientist from the muppet show?

andybak•9/20/24, 7:21 PM

https://www.youtube.com/watch?v=lTC2Ihnsrf0

YouTubeSpookiedoofus

Beaker has coffee.

Oh dear, Beaker!

andybak•10/25/24, 10:19 AM

Initial test of Spoke/Hubs integration: https://www.youtube.com/watch?v=hoPletmElkg

YouTubeAndy Baker

Icosa Gallery Spoke Integration 2

andybak•10/25/24, 10:20 AM

Currently not merged to the official Hubs repo but it shouldn't be far off.

andybak•10/25/24, 10:22 AM

most of the work is thanks to @Shoseki

yank•11/14/24, 2:58 PM

@andybak
RE: Standalone godot app. What are the use cases for something like this?

andybak•11/14/24, 2:59 PM

I can think of a few uses - plus it helps raise visibility for Icosa with people that don't use Open Brush or Blocks

yank•11/14/24, 2:59 PM

That window I’m working on there can be put into the godot editor as a built in store, or it could be put onto a website as a wasm app.

Or it could be just an exe.. or the other ones for each platform

andybak•11/14/24, 2:59 PM

i'm thinking native android, iOS etc

yank•11/14/24, 2:59 PM

Oh that too

andybak•11/14/24, 3:00 PM

places where people tend to what to use apps over websites

yank•11/14/24, 3:00 PM

iOS is a pain because of the fees and hoops

yank•11/14/24, 3:00 PM

But android is pretty novel

andybak•11/14/24, 3:00 PM

we already publish to iOS so we've done some of the hard work up front

andybak•11/14/24, 3:01 PM

Steam is an interesting target as well - especially with the Steam Deck

andybak•11/14/24, 3:01 PM

once we get our shaders ported it also gives us a native viewer for tilt files

yank•11/14/24, 3:01 PM

Oh yeah I have one of those

yank•11/14/24, 3:01 PM

Love me steam deck haha

andybak•11/14/24, 3:02 PM

yeah. open brush already runs in 2d mode but there's not much UI in that mode

andybak•11/14/24, 3:03 PM

hey = probably move the tech chatter to #icosa-gallery-development

andybak•11/14/24, 3:03 PM

but no biggie

yank•11/14/24, 3:03 PM

On it