We're talking about creating gaussian splats from video - not generative AI. That's a separate debat
We're talking about creating gaussian splats from video - not generative AI. That's a separate debate and would belong in #offtopic anyway.
A(but I'll just say that generative and procedural techniques have been part of art since the first cave painting used pigment blown through a straw - it's a difference in degree not in kind...)
Tand luma AI is a generative AI company, that's what i was commenting on
Aonly recently. before dream machine they were working on gaussian splats
A(which is what we were talking about)
A(which is AI - and to some degree "generative" but probably not what you mean by "generative AI")
A
Ttook a bit of hunting to find that, all the links are going to their dream machine crap
Tthe nerf/splat thing is pretty useful though
JOh wow what a pain… feel free to share the video with me & I’ll process & export in your chosen formats when I have time. See attached screencap showing options.
JKeep in mind it’s only UE & splat export formats that retain the view dependent effects like reflections
JExample high poly Luma GLTF export @AncientWorlds
textured_mesh_glb.glb35.31MB
JPretty nuts… 5min to process on Luma iPhone app. Sorry to hear droid app is not the same…
AI suspect its because I'm not a paid subscriber? I systematically cancelled everything over the last couple of months.
Almost everything is something I use once in a while, hard to justify keeping them all active.
That looks really cool. I often feel primitive in android world.
Those scan extractions look great though.
Might take you up on that, going to look at the mesh you sent when I fire up the headset
Almost everything is something I use once in a while, hard to justify keeping them all active.
That looks really cool. I often feel primitive in android world.
Those scan extractions look great though.
Might take you up on that, going to look at the mesh you sent when I fire up the headset
JCool feel free - also keep in mind that mesh is high poly - there is also medium & low poly options. You may have more luck with web upload & processing. Luma used to allow it though I haven't checked recently. Also if interested see below. @andybak no doubt you've seen these & think we discussed a while back but resurfacing with all the Icosa progress of late - as well as increasing public interest in nerfs/splats/3D scans:) https://github.com/aras-p/UnityGaussianSplatting https://github.com/antimatter15/splat
GitHub
Toy Gaussian Splatting visualization in Unity. Contribute to aras-p/UnityGaussianSplatting development by creating an account on GitHub.
GitHub
WebGL 3D Gaussian Splat Viewer. Contribute to antimatter15/splat development by creating an account on GitHub.
JI should mention this came through trusted rec of master Keijiro https://github.com/keijiro/SplatVFX
GitHub
3D Gaussian Splatting with Unity VFX Graph. Contribute to keijiro/SplatVFX development by creating an account on GitHub.
J
J
PHi guys, I'm uncertain if this is the right channel. But is there any site that hosts .tilt files that can be downloaded?
AThat's exactly what Icosa Gallery is for - we're a few weeks away from a release with that functionality (well - you can already manually upload and download .tilt files from your own account but I suspect you want to make them available for other people as well?)
PThat's what I meant. I want for us all to be able to share our .tilt files.
AYep. Stay tuned then. We're nearly there. Everything works in the app like it used to with Google Poly
LLbtw. to explain https://xrfragment.org from a icosagallery/ -fileformat perspective:
is an (open) 3D file(format) for XR experiences.
XR Fragments (XRF) spec is about (implicit+explicit) metadata (found in a 3D file) which hints the viewer-application (icosagallery e.g.) to enhance XR experiences, for example:
is an (open) 3D file(format) for XR experiences.
XR Fragments (XRF) spec is about (implicit+explicit) metadata (found in a 3D file) which hints the viewer-application (icosagallery e.g.) to enhance XR experiences, for example:
- navigate in-VR to other 3D file by make objects with URL-metada clickable
- load/project video on a plane with videoURL-metadata
- switch camera/position/animation by clicking an objectIn theory, the XRF js-parser can be included in THREE.js to leverage those features (it just scans the `.userData`-fields of a loaded `.tilt|glb|obj` 3D file).(I'm writing an NLnet proposal atm, and am thinking of making a XRF>Icosagallery PR a milestone)
AOK. I think there's an important distinction between "editing a scene" and "viewing/interacting with a scene". We do try and cover both in Open Brush but the lines are a bit blurry and the "view only" side of things is a bit unfinished
AI think the Icosa Gallery is a better target for everything related to "Viewing/Interacting with" - I'm currently wondering if it's worth incorprating parts of Spoke (the Mozilla Hubs scene composer/editor) or the three.js editor into Icosa.
AHowever - "marking up" a sketch in Open Brush so that extra capabilities are enabled when viewing is totally in scope.
LThese thoughts make sense.
I actually studied Spoke & Three.js editor during drafting the XR Fragments spec.
The idea was to create a fileformat-agnostic version (first I wanted to bet on Spoke, until it got abandoned by mozilla).
The texturescrolling- and link-features (of XR Fragments) were inspired by Spoke extensions, and can be used by other fileformats too now.
The deeplinking/teleporting-via-URL aspects of XR Fragments were something I've always longed for with Spoke/Hubs.
I actually studied Spoke & Three.js editor during drafting the XR Fragments spec.
The idea was to create a fileformat-agnostic version (first I wanted to bet on Spoke, until it got abandoned by mozilla).
The texturescrolling- and link-features (of XR Fragments) were inspired by Spoke extensions, and can be used by other fileformats too now.
The deeplinking/teleporting-via-URL aspects of XR Fragments were something I've always longed for with Spoke/Hubs.
LBtw. is there a drop-in THREEjs viewer-library of files?
I'd love to add support for -files on https://xrfragment.org
The fact that it is an open portable format is important for XR Fragments to promote.
I'd love to add support for -files on https://xrfragment.org
The fact that it is an open portable format is important for XR Fragments to promote.
AWe don't really intend tilt files themselves to be portable. We would rather put effort into making our gltf exports portable.
AThere is a tilt viewer for threejs but it's very rudimentary. There is far too much magic baked into our brushes for tilt files to be easily rendered outside of the app.
ABut the gltf files can have enough metadata to be rendered with varying levels of fidelity across many apps.
Lah like progressive enhancement
Lindeed, I noticed a very cool scene in icosagallery which had some sort of illuminating fog/nebula.
I figured that perhaps some sort of shadermagic was going on.
I figured that perhaps some sort of shadermagic was going on.
AYep. Currently only the more basic brushes look correct in other apps (unless they can directly use our Unity or WebGL shaders).
I am trying to improve the export (with the help of @Nick ) to get closer to what is theoretically possible. The lack of additive blending in GLTF is a hindrance but I want to propose an extension to cover this and try and get a few other apps to implement it.
I am trying to improve the export (with the help of @Nick ) to get closer to what is theoretically possible. The lack of additive blending in GLTF is a hindrance but I want to propose an extension to cover this and try and get a few other apps to implement it.
LBlending modes in GLTF would enable so much more possibilities.
I wonder if .usdz (which Apple seems to bet on) has anything for that.
Upload works btw: https://icosa.gallery/view/coderofsalvation/rJ6NfwHhWlo
I wonder if .usdz (which Apple seems to bet on) has anything for that.
Upload works btw: https://icosa.gallery/view/coderofsalvation/rJ6NfwHhWlo
icosa
Lbtw. once icosagallery has a search-feature, I can implement a searx-plugin for it, to include in this opensource WebXR searchengine (which I also maintain) : https://searxr.me/
ABear in mind that https://icosa.gallery/ hasn't been touched in several years - the new site is still in development although we're fairly close to launching a beta
AWe are going to implement oEmbed as it's fairly simple. Activitypub is something we're looking into but in general I'm interested in supporting as many open formats and protocols as possible.
ASo - what would a minimal integration with XR fragments look like? What's the starting point?
LA minimal integration could be done on various levels:
- **EASY:** render an AFRAME div-playerframe for non-`.tilt` uploaded assets (basically .glb/.usdz etc) with the XR Fragments AFRAME plugin (https://xrfragment.org/#%F0%9F%A7%B0%20AFRAME) like in this codepen: https://codepen.io/coderofsalvation/pen/yLwedvX**Pros**: drop-in solution including VR/AR & hand-controls (I can do an example PR next week)**Cons**: no cons
- **LESS EASY:** integrate the XRF THREE-plugin into the icosa-viewercode: https://xrfragment.org/#%F0%9F%A7%B0%20THREE.js which adds metadata-listeners onto THREE loaders (and enhances objects accordingly, making them clickable e.g.).**Pros**: would enable XRF-features/metadata for `.tilt` assets too**Cons**: extra layer of complexity, no hand-control interactions, no AR-mode
- **HARDCORE**: integrate the XRF parser-library into the icosa-viewer, which only scans/parses/validates the metadata to aid further implementation: https://xrfragment.org/#%F0%9F%94%A8%20XR%20Fragments%20parser
Awe're pretty wedded to our threejs based viewer at the moment - although it's pretty lacking in XR navigation features.
A transition to A-Frame is interesting but I'm not sure how much work it would be. Is this something you might want to help with? We could put some of the NLNet funding towards it, if that would help.
Or at least maybe advise on - a few hours consultancy might save me a ton of time.
A transition to A-Frame is interesting but I'm not sure how much work it would be. Is this something you might want to help with? We could put some of the NLNet funding towards it, if that would help.
Or at least maybe advise on - a few hours consultancy might save me a ton of time.
Lyes that's definately possible.
TBH it's an interesting conversation to have.
AFRAME is basically THREE.js with batteries-included XR goodies.
Last year, I had no time to get into the nitty gritty of THREE.js XR controls/navigation/local teleporting e.g., so I quickly wrapped the XRF THREE.js-code into an AFRAME component.
I don't know if much changed, but last year I could not really find community-maintained XR goodies for THREE.js (other than AFRAME)
TBH it's an interesting conversation to have.
AFRAME is basically THREE.js with batteries-included XR goodies.
Last year, I had no time to get into the nitty gritty of THREE.js XR controls/navigation/local teleporting e.g., so I quickly wrapped the XRF THREE.js-code into an AFRAME component.
I don't know if much changed, but last year I could not really find community-maintained XR goodies for THREE.js (other than AFRAME)
ANo - I was a bit shocked how little XR character controller stuff was available for vanilla threejs.
Ai could probably just do a simple "fly" controller - but getting teleport right is fiddly.
MPersonally I’d prefer to stick at threejs level
Amaybe the next batch of nlnet funding could be used to write a decent XR nav component.
Ait's mainly about making teleport non-ugly and not complex
Athe old Steam implementation was nearly perfect.
LRight. So the teleport-via-button-clicks are already implemented by the three.js-plugin.
When a model gets parsed by any THREE.js-loader, it will:
I mean I used https://github.com/jure/aframe-blink-controls for free-form teleporting, but this is not really needed for most experiences (which I usually guide with fixed teleporting points anyways).
When a model gets parsed by any THREE.js-loader, it will:
- scan for occurences (with value or e.g.)
- emit a 'click' event for that object (currently mouse and XRController)
- setup basic click-listener (overridable) which sets the camera location to objectname or replace the current scene with
- it adds a hashtag to the current url so the location can be shared: https://icosa.gallery/view/coderofsalvation/wHhWlo#platformB
I mean I used https://github.com/jure/aframe-blink-controls for free-form teleporting, but this is not really needed for most experiences (which I usually guide with fixed teleporting points anyways).
AIt would be nice to offer both.