Godot addon
I’m going to get the godot addon for the icosa gallery working and production ready so I’m making this thread just to put everything in an organised spot
287 Replies
💪🤖
hey, whats these
- doing ? @andybak
at first I thought with the formats it was filtering them, so
-GLTF would remove gltf from the result
but, orderBy has them and its throwing me off
is that most and least?sometimes you get "Google" instead of the artist name
maybe it should be like, "archived by Google" ?
or something
Good point. Minus means two different things. For filters it means (as you thought it did) "exclude this"
But for sorting it means "in reverse order"
Do you think this is confusing? We'd be open to suggestions to improve it.
That is the "artist"'s name - it's the models that Google seeded the collection with so it was filed under their account. (It used to be "Poly by Google" but we changed it to just "Google" as it seemed a bit clearer
@James R ☝️ interesting feedback
yeah by reverse order i meant most and least, so it made perfect sense!
ah, so it really was google that made those guns?
nitpick but pageSize is probably an int?
Thanks for the feedback @yank I agree, using
- for both negation and reverse ordering is a bit confusing. I'd like to use ! which also doesn't need url encoding. I'll see what other do for this.
You're right, good spot! This is currently a "won't fix" bug with the rest framework we are using that won't accept nulls for integers so I had to make it a string. The same goes for pageToken, which is also an int. I'll document this somewhere.that would be good as
! denotes not in programmer speak.
I understood it fine in the end, just wanted to double check if my assumption was correct.
question for @andybak . what would be the "front page" request? something with orderBy=BEST&curated=true I'd imagine. if you have something you'd like to show first let me know
I'm going to default that stuff on from what I remember you saying a while ago
what is the default page size?
it seems to be 20yep
plus maybe a licence query? CCBY or similar
let me check what we do in open brush
the other question is whether to include tilt brush sketches by default. They are kinda niche for the general godot or blender user. not very low poly and need specific materials.
-tilt on format handles that mostly
yeah thats a good shout
category is an enum, but what if you wanted to search multiple categories, does that actually work?
same with the format filter
@James R - we don't have multiple categories or any chaining of filters yet, do we?
it's on our list
from the first godot one I made a while back, I think I built it with little radio switches to turn on and off for each category, and I half remember it working, but it was a while ago
no, but as you say, it's on the list
I'll build it with this in mind then, whats the syntax for chaining filters like that?
also, another nitpick but I noticed that author_name is snake case and the rest is camel case
it should accept both for backwards compat reasons. i think it's just the docs that are wrong here. @James R ?
whats the syntax for chaining filters like that?We haven't decided yet. What's your thoughts? query params don't have to be unique so one approach is ?format=GLTF&format=FBX
I was going to suggest that, that sounds like the best way to do it
rather than something weird like parsing sub-items you just parse another item
@yank - the best default for the licence filter is probably "REMIXABLE" or "ALL_CC" which are shorthand for:
REMIXABLE probably makes the most sense as it excludes "No Derivatives". It seems to make sense to exclude those from loading up in Blender because surely the reason for loading them up would be to make a derivative work?
Ah sweet. I just got all the api working and it’s all neat and tidy. Now I’m just doing the interfaces and stuff
("i am not a lawyer yadda yadda")
In the blender plugin i'm going to exclude assets that contain tilt files by default with format=-TILT
but i'll make it so you can override it if you want to
I’ve done the same, I have a default search query with -TILT, BEST and the licence thing but I will include some info about what you said, I didn’t know they represented a group of licenses
Everything is going to plan anyway, and from my last rendition and these improvements it’s gonna be awesome. I should be finishing up late tonight in UTC
Also, a bonus is that there’s a way to export to an application rather than embedded in editor, and it still works the same. This also means people could embed it into a game or app/platform
Oh that's pretty cool
Didn’t get a chance to finish last night, Now im working on the final part, downloading assets 🕺
I added a few little fancy fade animations to give it some life. It’s very subtle
You own the repo for this @andybak so when you get a chance tomorrow or something have a look at this:
https://docs.godotengine.org/en/stable/community/asset_library/submitting_to_assetlib.html
when it’s done done we can add it to the official godot assetlib as an addon, so you never have to leave the editor:
- create empty godot project
- download icosa gallery on the assetlib tab in the editor
- download models
Godot Engine documentation
Submitting to the Asset Library
Introduction: This tutorial aims to serve as a guide on how you can submit your own assets to the Godot Asset Library and share them with the Godot community. As mentioned in the Using the Asset Li...
yep, good spot. I've corrected this.
@andybak
Looks great!
seems really fast
thanks! I'm quite happy with how its turned out, and I tried to keep the code very neat and tidy which ended up taking the most time.
I used Arial as the font to give it a bit of a modern feel.
I still have a couple things left to do, actually downloading assets, search options and pagination
i wrote the code for all of these and started over because it was a bit hard to follow
heres a sample of the API. I just put all the data / endpoints in here, and a few helper functions to turn it into processable godot objects, seperate from any GUI code
oh. I need to do something about this as well. got some great feedback from godot users, they will definitely want to see license info for piece of mind.
Already talking about all the game jams that this is gonna be great for - this is great for godot users.
I think it’s going to be a big hit for that !
suggestion from me. file size search options
Hmmm. I figured the poly count filter ticked that box. I guess it doesn't cover textures. Was that your main concern?
That’s one reason - textures can be massive sometimes.
another reason is that it’s nice info to have on hand. Don’t want to accidentally download something thats 500mb into a tiny project
And displaying poly count is only approximating the byte size
Yeah. We currently don't track total file size. And it's a bit tricky because "over the wire", "disk usage" and ram or game engine size can all be very different.
godot has a way of downloading with http built in, but it has no "progress" which is why I was asking about byte sizes. I wanted to add progress to the download so the user can see how long it would be or if its really large.
it does have a low level http class so I've been working hard trying to get it working from scratch, and I think the archive.org links are throwing it off or something. or I'm just doing it wrong
either way I might have to continue without download progress, and show a little spinning thing or something
The blender plugin doesn't have progress and neither does the web viewer. Maybe I should consider all of these things holistically.
I think we'd need to precalculate the total file size and store it under each format (format being the collection of resources you download. Assets have Formats which have Resources...)
The only snag is this requires a lot of HTTP requests to Internet Archive which has some really fierce rate limiting!
I suppose its more expected in an editor like blender, in the browser it calculates this for you in your downloads area.
but yeah, with the archive.org requests I've been super careful
damn. archive.org links are being a huge pain for godot. I even tested the older browser I made and it doesnt seem to be working on there anymore.
first I had a problem with double case of
https:// in the url, then when sanitizing it for godot, i make a request and nothing happens. no errors or warnings or prints.What error codes do you get?
Invalid paramater
so a 400 (or 40-something)?
can you give me the exact url that returns that? (and the actual response if you can get it)
Yeah hold on. Was trying to get the source from godot
It’s not getting past godot
That’s the issue, and when I uri encode it, it doesn’t seem to change anything
GitHub
godot/core/io/http_client_tcp.cpp at master · godotengine/godot
Godot Engine – Multi-platform 2D and 3D game engine - godotengine/godot
Here’s the function it fails on, because it thinks the ‘double’ link that you get from web archive is a ‘paramater’ for the URL
I’ll get the error from the browser
I’ve tried maybe 30 different configurations of the URL and request to juggle around problem areas but I got nowhere
this is the error I get when I pass an URL like this:
https://web.archive.org/web/poly.googleusercontent.com%2Fdownloads%2Fc%2Ffp%2F1623061107783986%2F9L2Lt-sxzdp%2F9CCR_YwpE17%2Fmodel.gltf
so here's the url as we store it:
https://web.archive.org/web/https://poly.googleusercontent.com/downloads/c/fp/1623061107783986/9L2Lt-sxzdp/9CCR_YwpE17/model.gltf
yup, I tried to process that
it's a valid url so it should work as is - no need to quote or unquote anything
https://web.archive.org/web/https:// because there are 2 https:// i think the colon throws off godot because it recognises it as a URL parameter, and not a valid oneurl parameters should just be anything after ? separated by &
colons aren't special
so I tried URL encoding it like this - and this actually works in my browser but godot will not take the request
I'm probably going to submit it as a bug report today
I've tried all kinds of stuff. for hours and hours. its the finish line for this project so I was expecting it to work like last time. and I was like, what did I do different a few months ago? turns out nothing
so I tried the browser from a few months ago, nothing would download, just got lots of failed errors
we do have a non-internet archive url for all assets
(we need it for the web viewer because of CORS)
but... it's not always the one you'd want Godot to use
i.e. for legacy tilt files it might be GLTF v1
yeah, thats probably no good. I'm going to file a bug report because it shouldnt be this hard. made loads of http requests in the past. asset data and thumbnails are grabbed fine
let's go through this carefully.
the path is wrong. it's losing everything after .org and before //
/web/https://
I told you, if theres another
: in the URL, godot just wont make the request for some reason
so I can't put that there, but when I URL encode it, it also failsah
this is wrong:
if you percent encode the original it should be:
this link works in my browser. maybe thats not telling me much
oh hold on
Internet archive doesn't care about the http.
so - this works:
https://web.archive.org/web/poly.googleusercontent.com/downloads/c/fp/1623061107783986/9L2Lt-sxzdp/9CCR_YwpE17/model.gltf
which has no colon!
E 0:00:08:245 request: Condition "!_check_request_url(p_method, p_url)" is true. Returning: ERR_INVALID_PARAMETER
<C++ Source> core/io/http_client_tcp.cpp:152 @ request()
but it's not the colon
I dont understand what it means by parameter, even after looking at the source
ERR_FAIL_COND_V(!_check_request_url(p_method, p_url), ERR_INVALID_PARAMETER);
calls line 131
yeah I found this too
Checks for a colon (:) not at the start or end.wrong port there. extra colon too
https://web.archive.org:443/web/poly.googleusercontent.com/downloads/c/fp/1623061107783986/9L2Lt-sxzdp/9CCR_YwpE17/model.gltf
ooops
https://web.archive.org:443/web/poly.googleusercontent.com/downloads/c/fp/1623061107783986/9L2Lt-sxzdp/9CCR_YwpE17/model.gltf
fuck discord
i meant port 443
but discord is removing it
oh yeah and discord strips stuff from URL
put it in
thisTY!
phone call. gotta dash
E 0:00:08:219 request: Condition "!_check_request_url(p_method, p_url)" is true. Returning: ERR_INVALID_PARAMETER
<C++ Source> core/io/http_client_tcp.cpp:152 @ request()
GitHub
cannot download specific link with
HTTPRequest or HTTPClient ·...Tested versions 4.4 stable System information Win10 Issue description I am trying to download into godot from this url: https://web.archive.org/web/https://poly.googleusercontent.com/downloads/c/fp...
woah. already got a bunch of feedback on this
ah! i did wonder if it was about redirects
i even half typed a reply here when that bloody phone call came in
oh I do that so often XD
@yank despite there seems to be an issue here, i would still recommend against using an url whose reachability you dont control. If you release a game and this file suddenly is no longer available for any reason, your game will become unplayable. I would recommend to download the model and reupload it to some location you control.Stupidest suggestion ever
I know I'm already a bit frustrated so that rubbed me the wrong way LOL
So - most http clients have an option. Something like request.GET(url, follow_redirects=True)
but seems like in godot you need to do this yourself
but there still appears to be a bug in godot : https://github.com/godotengine/godot/issues/104651#issuecomment-2754840342
GitHub
cannot download specific link with
HTTPRequest or HTTPClient ·...Tested versions 4.4 stable System information Win10 Issue description I am trying to download into godot from this url: https://web.archive.org/web/https://poly.googleusercontent.com/downloads/c/fp...
(we probably also need to account for a "429 Too Many Requests" response - which is when we hit the rate limit)
(although I haven't done this in the other clients)
yeah. I was being careful with the requests at first
now im just trying to get it working so i've been giving it loads
i think the limit is 15 requests per minute?
oh we good
haha
godot wont let me half the time because of invalid params anyway
or most of the time
all of the time
hopefully someone on the godot networking team can see this today and shine some light
I'm so ready for this to just work now XD
Weirdly, this happens only when: requesting the original url that gives a redirect (https://web.archive.org/web/https://poly.googleusercontent.com/downloads/c/fp/1623061107783986/9L2Lt-sxzdp/9CCR_YwpE17/model.gltf) redirects are turned on (max_redirects > 0`) Requesting the target URL directly causes no issues, and disabling redirects on the initial url causes no issues. Trying to recreate this issue using nginx (return 302 <url>) without success, maybe something about the sent headers?this implies there's a setting for redirects!? if so - turning it off and handling the redirect ourselves should work
i will ask swarkin to be more verbose, a friend of mine
someone who does stuff with sockets on the web etc
I'm currently trying to download ~30,000 files from Internet Archive.
😦
oof
maybe you could try torrents?
archive seem to have torrents available
i actually downloaded the archive files but they are in a format that's hard to parse. it was actually taking about the same time as it did to download each file directly
they are great if you want everything but they suck if you want specific files
got it working @andybak
Cool!
damn
new problem
hot off the press
but, I cant open them
sorry for the wall of jargon
the gltf file seems correct, can open it and read it
so im not sure whats up
oh. I know whats up. I broke the refrence
nevermind
WROKING
yayyyy
@andybak you around? can show you
just heading off to cook dinner. but if you want to post a screen cap I can take a look on my phone?
yeah, its no biggie just excited that its fully working now
😄
me too!
Ran some tests, got like 80% files downloading correctly
It’s the ones by Google that have a bin that is named something unique instead of model.bin
Send me an example of a broken one
I think it would be awesome to make a big demo scene of all those assets on icosa gallery.
Like a nice little kitbash
I’m gonna list all the stuff the browser still needs/some suggestions/ideas:
- get 100% success of downloads working. The problem is that I’m messing with the file names and it’s breaking refrences so that’s my issue, not the files
- search options (should be very easy, an hour or so)
- download progress: I found a way to do it but it’s rather complicated. I did manage to get total files downloaded progress though 👍
- more centering of divs, little more polish
- some small internal cache of downloaded assets, and putting a check there so you know you have it already. That wouldn’t be too crazy to add either
- licence info displayed in an obvious place with some information tooltips / helpful icons etc
- some browser internal settings to manage the .gitignore, owned assets db
- using a module for godot supported gltf extensions, if required, which I made already for another project
- endgame could be logging in, uploading and using the rest of the api that isn’t just downloading and viewing browser
some browser internal settings to manage the .gitignore, owned assets dbNot sure I follow?
endgame could be logging in, uploading and using the rest of the api that isn’t just downloading and viewing browserYeah. Uploading is the biggie. We have it in Blocks and Brush - but I haven't tackled it in Blender yet. (Blender has login - which gives you "my models" and "my liked models" - I mostly got that for free because of existing code from Sketchfab)
get 100% success of downloads working. The problem is that I’m messing with the file names and it’s breaking refrences so that’s my issue, not the files100% is gonna take a while! I'm aiming for 80-90%. There's about 150,000 files and the data is a mess. I'm focusing on the most "liked/downloaded/curated". It's going to be an ongoing project but hopefully - if we can flag "preferred download format" you don't have to worry beyond some initial troubleshooting The models are going to follow a "long tail" distribution. 90 or 95% will be rarely of interest to anyone. It's just hard to know which ones are in that 90% 😉 I've already filtered out some sportwear company that uploaded 5000 models of sports tops. And a company with several hundred moulded brackets. and a bunch of pentesters and god-knows what else
like settings for the godot users experience using the browser
maybe adding a gitignore, and some other helpful things. debugging
i was thinking in the folder of the icosa godot addon, a user can store a small db file like:
then when browsing, it will check it for you
just a nice little touch isnt it
yeah downloading off web archive is a PITA
im not sure if I broke something or what but I can't download anyting again, getting a 500 response this time
i think it was me lol
but hopefully it will be fine. it was fine when it was fine
downloading off web archive is weird because I'm making a request, then its redirecting me so I have to make another request
but I have to do it manually
godot has something built in for redirects, but this gives tht invalid parameter thing
anyway, I mean 100% sucess rate meaning, yeah, most things that people might want, a pleasant experience
so more like 80%
oh - one question for you as well - what is the max range for tri count? if we are counting complexity, then it would be good to know the upper limit
Rather worryingly - there's no upper limit!
At the moment - the only limit is the longest we allow a connection to last for a single upload. The server timing out on POST is probably the only limit. We should probably fix that!
(we haven't documented the upload API and so the only way to do uploads at present would be to reverse engineer what Open Brush or Blocks do...)
makes me wonder why people dont use torrents more
you can pick it up and drop it off whenever
i guess everything would need a torrent client service built in though
its handy that archive.org uses that
ive torrented a lot of big things off there, and sometimes i delete it and then just restart the torrent to get it back
mainly music samples
ah
I know why my downloads arent working
web archive is down
Downfor
Is Internet Archive down? Live status and problems past 24 hours
Live problems for Internet Archive. Error received? Down? Slow? Check what is going on.
LOL
I spent a while trying to fix this
relieving though
i've asked aaronfranke why this will not open in godot, but it will in blender
not sure whats up with these particular files, the ones with "Google" as the author and stylized in similar quality.
the mesh doesnt show up as the gltf is invalid
Aaron said that the obj2gltf converter is making invalid gltf files
@andybak you know anything about this?
all of those assets made by "Google" have this problem - but they are really good and all fit into a specific style so they would be GREAT to have
We have multiple formats and variants for each asset. Plus the obj which is fine for low poly stuff. It's just a case of ensuring we select the right version
I'll take a look tomorrow and see what's happening. Blender is mostly working for low poly stuff
yes it does load in blender, but not in godot
if we can get it loading in godot that would be soo sick
@andybak
is there any error or debugging output from the godot gltf loader?
this file loads in every other app i've tried - it would be helpful to pin down exactly what godot doesn't like and either preprocess our file (if that's possible) or at least file a bug report for godot
in the meantime - the obj file should work but i can't get it to recognise the mtl file
does godot support obj+mtl?
managed to fix the gltf manually
godot is really being too fussy here. it's choking on an error it could safely ignore
the first error is saying that accessor 4 is invalid. but the third error is saying it's unused. so it's safe to ignore the first error
and accessor 4 points to bufferview 0 which is the source of the second severe error. so if we remove accessor 4 then bufferview 0 is unused and that error is safe to ignore
if i manually remove accessor 4 and fix bufferview 0 then it loads fine.
@yank
if we can get godot to report the error then we could try modifying the gltf and retry. but if not then we'd need to guess for every file that fails to load
but - as i said before - this is really better fixed in godot if we can persuade them that it's their issue. they can always fall back on "your file is invalid" which is true - but there's invalid and there's invalid. Importers should be forgiving where possible.
The obj file is interesting in this case. It's converted the vertex colors to a normal texture map - and baked in some lighting. Which is kinda weird but maybe useful for some people.
but it's not a direct equivalent to the gltf
this is the file i used as a test: https://icosa.gallery/view/10u8FYPC5Br
Fox | Icosa Gallery
icosa
my hunch is that the above will apply to all the google supplied scenes (of which there are 3019 in the Icosa library)
How do you fix bufferview 0? That’s the only thing I don’t understand about all this
I should be able to ‘patch’ the glTF as it comes through with the addon
I would write to the file and remove the unused thing before giving it to the user
If it’s not used it’s not needed ?
And saying that, it might be interesting to see if I can shave off all the empty nodes, lights and cameras, unsupported Google extentions.
These 3000 models seem to be that worth it
As they all fit in this one unified style
I found a bunch of problems with how some models are split up. I think the gltf1 to gltf2 converter might of done something
Basically, if you look for any ‘kits’ where it’s an arrangement of a bunch of models, the meshes are not split by islands, but some random nonsensical order.
And it seems to have done it for complete models that are a bit complex
I will send a pic
I don’t think we can do anything about that one
Or at least I wouldn’t know
Ohhh they are both unused. Sorry I just woke up haha
Another problem I encountered is that sometimes the root is offset. I guess I downloaded a frog - and was expecting the root to be zeroed at center of mass
So it would be really useful to just strip the mesh out of all those empty nodes + camera + lights.
But this should be a user choice, maybe on by default
I’m super busy today but I’ll be looking at this in the afternoon. I was able to download and view basically all the gltf2 I tried except the Google ones. Got the split thing and the offset thing on some others
The unusually split meshes are the ones I can’t think of an easy fix
hard to say why these are split up, thought it was by material, but if you look at the wheel you can see that its split the wheels in 2, diagonally, so thats odd
this is called "jeep" but, its most obvious with the asset: "knights character kit"
a lot of "kits" have this issue
anyway
godot does, but I fear I may have broken some references. I was just targeting gltf2 first, and I think I will highly suggest gltf2 if there is one to the users, but I will defintiely look at fbx and obj after gltf2.
godot has no usd support, and I will end up hiding .blocks and .tilt files, or hiding the asset altogether if there are no valid extensions
Yeah the mesh island thing is a problem for many assets. A blender script might be the best fix. There's a precedent for Godot calling or to Blender I believe? One day I'd like to fix them en masse
In this case I think we should warn and continue loading
It’s an interesting problem because of ‘nested’ errors, if the error checking was more methodical it could pick up on that and then do what blender seems to be doing
I totally forgot I need to test godot icosa in Mac and Linux
my friend is testing it in Linux at the moment to see how it does in VR
OK. I think I've got an idea of the processing steps that will make these invalid files load in Godot.
1. Loop through each member of the "accessors" array and look for an accessor where the count property is 0. Replace it with any other valid accessor (these are hopefully unused so we're just putting "anything valid" in there as a filler. We could remove that entry but then we'd have to change the index-based references to it elsewhere which seems like more effort)
3. Loop through each member of the "bufferViews" array and if "byteStride" is 0, remove the byteStride property (Again - these are all hopefully unused so it won't break anything)
I did it manually on a couple and it seemed to work. Should be trivial to code up in GDScript. The Python would be something like this:
@yank @aaronfranke
I can probably fix all the existing ones this way as it's only a few thousand. But that assumes only files uploaded by Google are affected by this. I might do a random sampling of other files to see if this is the case.
But considering this only affects Godot - it would save me a job if you can do it in the importer?
Especially as this is a fairly hacky fix and I'm not confident enough to want to apply it irrevocably to the originals
Seems reasonable yeah, those Google assets are really nice
Ended up being busy today so I will have a look at that tomorrow
Thanks for the python code Thats super helpful because gdscript is so similar
Basically all I need to do is plug in your fix tonorrow
@andybak @yank Sorry for the slow reply, but I've got a fix for the byteStride being 0, both an engine fix and a GDScript workaround.
Fixing that is enough to get the Toco Toucan file Yank sent to load.
However, I haven't touched the accessor count of 0 yet, is that also something that needs to be fixed for some files?
The fox example loads correctly in Godot without changing the accessor, but indeed an accessor count of 0 is still invalid.
Maybe it's the byteStride that matters. I just fixed both because they were reported with the top severity by the validator.
(although so was "unrecognised extension" which seemed a silly thing to fail on)
Really? VS Code's built-in validator reports that as "note" severity. (the lowest severity)
i'm using https://github.khronos.org/glTF-Validator/
oh. my bad
severity 2 - but it was listed first so i assumed it was the worst
It seems that Godot's code just turns a zero count accessor into an array of size zero so I think it's good.
it's not sorted by severity. doh!
cool. so bytestride is the only fatal error
GDScript workaround for
byteStride, @yank knows how to use this.Very impressed you got an engine fix done that quickly! Will it get merged soon?
Usually minor bug fixes like this don't take too long, should be merged in less than a week. https://github.com/godotengine/godot/pull/104780
This is what the file will show up as with this PR.
looks spot on
It will print error messages because the file is invalid. If you want those errors to go away, you have to fix the files on Icoas's end.
i mean - the files are invalid so it's probably fair
and we should fix them at source one day...
but i've got 20,000 tilt files that don't even load in anything to find a workaround for first!
(that's a bit of a simplification but if gives you a flavour of what i'm dealing with)
Is the tilt format documented anywhere?
We could build some kind of Godot importer and then use Godot as a tool to convert them to glTF.
i actually meant "gltf's generated from tilt brush" - tilt files themselves are a whole different world of pain. @Nick Fox-Gieg has done some good work in this area.
extracting the paths is fairly achievable - but the actual mesh generation is a gnarly beast with only a single current implementation (open brush itself)
Also feel free to keep throwing me files that 1) Godot doesn't load properly and 2) You think are possible to load properly (either the file is valid, or the invalid things can be worked around).
yeah. I'm taking a "if it loads in blender and/or UnityGLTF then it's probably worth saving" approach
UnityGLTF is a good benchmark implementation. It's conformant and forgiving..
Postel's Law. 🙂
Wow, thanks Aaron for that fix.
A lot of people are gonna love these particular assets I think
I will make sure to credit you for that
Also, I think I can extend this particular fixer script to clean up a lot of other stuff, like looking for GOOGLE_ in extras and cleaning up the cameras and sun nodes
But just for cleanliness I will keep it as one script
I have made a gltf extension library addon for godot which is a opinionated fork of the OMI-godot repo
That’s where I like to have a load order setup in the editor, and also reloading of the extensions.
If we can get godot users uploading to the icosa gallery, with godot related extensions from OMI / elsewhere built into them. That might be really interesting
It’s something I’d love to anticipate, even if I’m thinking way far ahead.
Yeah. I want to start supporting Hubs or OMI extensions in the web viewer. Being able to link scenes together with portals would be fantastic.
I'd also love a browsing mode that kept you entirely in VR.
hm. tried to add your fix @aaronfranke, this file crashes godot with this strange error message. just curious if it does the same. havent been able to get your fix to work on my end
crash on import
hmm. seems to be working now
great!
nope. there are some assets that are crashing the editor
@aaronfranke not sure why this is crashing the editor, some other models are taking the fix and still working, but these particular ones don't, and crash the editor in every situation
I'll take a look.
no,
I think this is for andy to look into
i get this trying to open it in blender
it suggests that the file is completely broken
yeah, i cant open this model anywhere
its not a godot issue
i guess if its crashing the editor thats bad
It's still bad if it crashes Godot, it should at worst fail without crashing.
@andybak, we should look into all the "Google" authored assets. seems like some of them are broke in a different way
gotcha. I guess I'm too forgiving with editor crashes in godot lol, but the gltf import is causing it here. i posted some logs earlier https://discord.com/channels/783806589991780412/1352344141203243088/1355939052586668114
For the Knife file
Accessor 13 contains 0 elements, which does not match the amount of elements in accessor 11 or 12 (48 elements).
glTF spec requires all of these to contain the same number of elements.
This means that every vertex needs a normal and a texcoord, or none of the vertices may have a texcoord.
is this 1.0 spec aswell?
well, it doesnt matter i guess
I am reluctant to fix this bug because I have another PR that touches mesh attributes. Yet another PR conflicting with myself.
:aaronscrem: :aaronannoyed:
I need people to review my Godot PRs.
well, it could be that this bug is so rare its not worth fixing in engine
if its only these icosa assets which have gltfs, I can just hook into them with gdscript no problem. its a big problem is if it crashes before we get a chance
but its only a problem for these particular assets, which andy might be able to fix
or really, somewhere along the revival process they have got a bit malformed
will some comments help? I can't review godot PRs unfortuntely
What do you mean you "can't review" them?
Everyone can review any PR on GitHub.
I can??
Yeah it will just show up as a grey checkmark if you approve them instead of a green checkmark.
I thought that I don’t have the authority
Everyone has the authority to review any PR on the whole site in any public repo.
Oh weird, I don’t know why I thought that
Just make sure that if you review the PR, you state what you did in the review.
Did you test it? Did you look over the code? Do you need the PR personally for a specific use case, does it fix a specific problem with a file you have? Did you look at the code style, the code structure, compliance with a spec?
A review that's just "This looks good to me, I tested it and it works." is not as valuable as a more detailed review but is still valuable if you can't be bothered to do a more detailed review. But you should review in as much detail as you can, I suppose.
If you think of any edge cases you can test them, and/or leave a comment.
I try my best in OMI, but mostly I see you building stuff in real time so I don’t have much to say in the review I haven’t said in voice already, but in godot i can write a bunch of comments for sure
Also, even if your checkmark is grey, if the Godot maintainers see a consistent stream of valuable accurate reviews (i.e. not approving a PR with issues in it), they will gradually begin to weigh your thoughts more heavily over time.
The same applies to pretty much all repos.
Cool! I often report bugs when I can so I get how godot is moving at the moment.
I’d personally like to see this icosa godot project alive sooner than later, I think it’s a great benefit for godot so if there are any related PRs like the one you mentioned post them here and I will review them. I have a good motivation to review these so I can be very methodical
Maybe we can free up some of the pipes to get things moving
I can’t really upload this to assetlib if there are assets people will use and it crashes the engine, and won’t let you open the project (even in the new recovery mode)
I did a huge update yesterday where I added all the search options available, license info on the asset, and started cleaning up the console spam a bit
It’s getting really close to production use and it’s exciting to see it through
I'll link a few specific ones immediately relevant then (simplest on top)
This simple PR is holding up other changes https://github.com/godotengine/godot/pull/101270
This fixes the byteStride issue Icosa is experiencing https://github.com/godotengine/godot/pull/104780
The PR I mentioned touching the texcoord attribute https://github.com/godotengine/godot/pull/96748
ah, just checked and the bytestride one already got reviewed by calinou just now. so I will have a look at the simple PR
calinou also beat me to the padding issue
I opened the review page before calinou got to it and i guess beat me to the review, but i made one anyway! lol
that just leaves the texcoord one
when it comes to UV maps and all that stuff I get confused pretty quickly. I could get some help from unfa to look it over and leave a review
cleaning up this has been a bit tricky. can easily get rid of the camera and the lights, but im making a big mess of the mesh. and also, whats the best practice? I was thinking getting the
Node3.transform and applying it to the mesh node, and adding it the scene root. I got a few crashes while trying to do this @aaronfranke
adding the mesh with the transform to the scene root means you can instance the glTF in your 3D godot scene, and you ONLY get the root and the mesh. I thought that would be the best idea
You can still review it though. 3 reviews is better than 2.
Yeah I will do, it was cool to get sniped that fast. Godot is moving so quickly now
Calinou has seen the gallery, I posted in the func_godot discord because it's mapping related, and they recognised it as the old Google poly archive so i think we have some really decent support for this across the board
So, Google's converter seems to have added a bunch of nodes. However, if we know in advance we only want the mesh, we could just write a GLTFDocumentExtension to generate new glTF files with only a mesh, and upload those back to Icosa.
The only tricky thing is, how to determine whether or not this should be done.
Should the code just check for the Obj2Gltf generator?
Actually I like that idea
Yank I'll write some code for you shortly but I'm busy at this exact moment.
I was just going to hotfix them from the gallery - uploading back into icosa might be too much work. Writing to the glTF might be a little more acceptable
I skipped all of that and was doing it post import
Well actually, if we can programatically determine when to do this, then we can just have the gallery apply this on download.
But it's problematic to remove those nodes
So then there's no need to upload back to Icosa (but that would still be good)
I want to prevent them from generating in the first place
Yeah the godot gallery and fixing the icosa gallery seem to be 2 parrelel projects but there could be a way to use the gallery and godots glTF exporter
To fix icosa
I mean we can use the godot addon to export the correct mesh and upload it, but we'd have to download every mesh and upload them all back
thank you, absolutely if you have time I'd appreciate the help but don't let me bother you too much if your busy this is all the long game bugs that take a little bit to sort out
I'm just reporting them as I see them. I'm still waiting for Julian Todd to come in here and talk about his ideas for icosa, he wants to network the runtime app (the gallery works as a scene just fine + runtime gltf loading)
So basically 2 people can look at the same window and interact with it and upload download assets into the scene. He said it could be like resonates inventory system.
Resonite* sorry
I think it's time I will start using the issues part of this project on github to track what's left to do and what assets don't work.
So far I've got everything coming in as gltf and working except for those ones I sent that are crashing the editor.
I will do that tomorrow anyway I'm going to bed soon
this cleans up the assets without destruction to the gltf file or the transforms. the mesh already has its origin in the correct position, so no transforms are needed.
but also, it crashes the editor the first time it imports like this. probably because its really hacky to remove nodes post import like this, as essentially the code hits the first Node3D and removes everything and then later I add the mesh back. i guess something is crashing it because of that
This is a much cleaner alternative to what you have there, and I think I've included enough checks to ensure this only runs for Google Obj converted assets
ohhh this is even better !!
yes
this is very awesome
Essentially this completely ignores everything in
nodes and scenes
It just checks, if this is a glTF with Google extensions converted from OBJ with exactly one mesh, let's only generate one node that uses the one mesh.yeah this is exactly what we need, google is adding transforms and lighting for the thumbnail, and its baked into the gltf. so when you take all that out, you get a mesh, and its origin is set in a good spot. very exciting!!!
it makes sense now why all that extra stuff is in there
Note that the asset could still use further manual tweaking, for example the knife and mug are giant but the toucan is the correct size
LOL yeah. thats what im looking at now. I found the scale to be good at 0.01. but i cant tell what the scale is supposed to be
I think most of these bad files are centimeters.
A hack you could do is check if the model is over 10 meters in diameter or something, then scale it by 0.01
However this will break models that are supposed to be big, like space ships.
im thinking there is probably some identifier that is making the mesh large
or something else unrelated in the text might tell us
That would be ideal
well. it could just be generated that way
wait. how did you get the knife and mug opening in godot? those crash my editor
Oh sorry I was running this together with my byteStride PR
ended up testing it out a bit. works quite well for making nice scenes !
hey @aaronfranke - those fixes loaded with GTLFDocumentExtension, I'm assuming those don't work at runtime?
I tried to enable the addon and then import to runtime but it comes in blank because the fixes are missing
They do work if you register the GLTFDocumentExtension at runtime
The EditorPlugin script doesn't run outside the editor.
I seeeeee yep, we just got that working
Any other files that are causing issues?
I have an issue in Blender that is specific to files that contain the GOOGLE_tilt_brush_techniques extension. That won't be relevant to any of the low-poly stuff but let me know if you hit the same issue in Godot
did you manage to fix the ones crashing the editor??
i may have missed that
you did mention something about not wanting to touch the TEX_COORD issue
I did mention that but I took a look anyway and the fix was really easy so I just added that to the buffer stride PR.
So no crashes in 4.5 then? Awesome
Great work
This is really coming together now, maybe we can anticipate getting it on the assetlib for 4.5
I haven’t found anymore dodgy gltf files. The other formats don’t interest me much so I haven’t been testing them really
gltf works so good with godot that fbx and obj are just a bit redundant
And those other strange formats are just none of godots concern
If you’re fine with it @andybak , I would prefer to obscure the other formats away from a default, so that when the user clicks the download button, it’s gltf by default, but that can be changed to drop down if the user asks
I think that’s better for godot users. And to sort of let them know - for game dev and godot, gltf is going to be the optimal format
We're still working out the details but we want to tag formats so that clients can choose the best one for their use-case. It will usually be the GLTF you're currently getting but there might be outliers and exceptions. (for a start - users will be allowed to upload their own files and when we relax the current restrictions they might not have a GLTF)
So - for now I think you're doing the right thing - but it might change.
Yes I think for godot it makes sense, you can also export and upload gltf from godot.
In the beginning - I thought uploading might not be what godot users want to do, but there's actually a lot of potential I didn't anticipate, like creating a scene out of lots of other models and then uploading that. Or even generating a level using some scripts and uploading that. Also, there might be a way to use godot to make "game ready" assets using gltf
Shouldn't there always be a GLTF, even if auto-converted?
I think some of the assets don't have a gltf, but it was really rare to see that
Yeah but then we can convert them ourselves using some automated tool.
Currently that's mostly true. The 155 were uploads to the original Icosa gallery rather than Poly and probably aren't really valid.
But I want to allow people to upload other formats eventually. The web viewer currently supports , obj, ply, stl, usdx and vox via builtin three.js loaders.
150k assets! Wow
The quality drops off sharply after the first 10,000 or so. 😉
I actually went on a bit of a cull where a single user had uploaded tons of stuff of debateable merit. Like 5000 cycling jersey's. And several hundred different types of plastic bottle.
I did however relent at the company who uploaded hundreds of different decorative mouldings. Because I thought they were actually quite cool.
There's also a few insanely big uploads that might have to go. The top 10% of big uploads is probably accounting for an unreasonably large chunk of storage. So unless they are incredible works of genius, I'm tempted to lose them. But that does raise the question of limits moving forward. How do we prevent abuse? What do we do when our hosting costs get unreasonable? When does "reasonable usage" tip over into "taking the [insert acceptable local colloquialism here]" ?
I think we're going to have to be stricter for stuff uploaded with non-creative-commons licences - because we aren't here to cover the expenses for people's private portfolios. We could create some payment structure but the costs (both financial and mental) of handling that are pretty high. We're here to provide a public good rather than create a commercially viable service.
Lots of things to figure out as we go. One good samaritan who wants to fund ongoing hosting (hello Internet Archive!) would make this a non-issue. It's pocket change to most companies.
We're using Backblaze B2 for storage and they are among the most reasonably priced. Someone like that or maybe Cloudflare could host us in perpetuity without even blinking.
Hmm. Never thought about this, I'm on the other end opening the floodgates.
Now you mention it, I can think of loads of ideas to trial limits. As you are a foundation of purely good will, it would be unsavoury behaviour to "take the piss" (local colloquialism)
you could try something like, if nobody downloads this after X amount of time - it gets removed. Was it really worth sharing if nobody downloads it?
You could rate limit uploads globally per day - then spend the next day scolding and blocking whomever attempted said piss take.
As for anything over reasonable x megabytes - you gotta pay. This isn't a free lunch
As a content distributor, you are going to be a target for abuse. If you want to start anticipating abuse I can also do this in the godot addon
There's also rating systems, which is another thing I'd be happy to implement as this would help with godot curation.
In fact I think I wanted to mention this - if you can give us godot users any little way to curate the gallery I can make it happen
A rating system could work. Even just hearts/likes would be fine
failing that I could just curate it with a select few people. I've already pegged the Google assets as 'game ready' just by working on the gallery
There's also a bunch of other ones. It could work by adding GODOT to orderBy in the API
Just to be clear, I only removed stuff from legacy Poly users. i.e. petite who already assumed their stuff had gone away years ago. Removing content uploaded by actual Icosa users is a whole different kettle of fish and we'd hopefully have an actual conversation with the person involved
Ah OK, I suppose after all that work preserving stuff it's not really worth deleting much. But if only 10% are quality - then people are sifting through lots of things to find something worthwhile to them.
It got me thinking about this:
https://www.thebasemesh.com
Maybe it could be great to start adding to the gallery from other sources - and absorbing quality assets, in order to "seed" more good assets from users who might upload even better stuff and so on
TheBaseMesh
The Base Mesh | 100% Free CC0 Assets
- 3D Models
- TheBaseMesh
- Every model adheres to real-world scale and comes with basic UVs, so you can get to making those all-important creative decisions faster.
Not talking about just repurposing but maybe the owner would be happy if you asked. Seems like it's one feller making all these assets here (untextured but AAA quality)
There's probably treasure all over the Internet. Kenney also comes to mind
Can a submission have multiple files attached to it, so we could make a manually curated "game ready" glTF alongside the original file, and then be able to filter by game ready assets?
We already have multiple "formats" per "asset". Format is more than just gltf, obj etc. For example most assets have "original" and "updated" gltfs with the latter having potentially modified metadata for background, lighting etc. Blocks exports two types of obj (one triangulated and the other with n-gons).
There's also some stuff I haven't fully deciphered yet. Old files from tilt brush usually have 3 or 4 GLTF/GLBs and I haven't yet worked out exactly what the pattern behind that is. We have a field for each format that I've called "Role" which currently has some arbitrary names where I couldn't find a clear name in the Poly scrape data.
I'm currently tweaking the heuristics for which ones we serve up via the API and the web download UI but my priority so far has mainly been making sure we get them displaying correctly in the web viewer.
I'm up for attaching more meaningful names to "roles" but some detective work is required first. And there are other quirks - like Blender having a very fussy importer that chokes on files that load fine elsewhere.
(I definitely got too bogged down in the preservation and restoration aspects of this. I wanted to save as much Poly stuff as possible and it's had a cost in terms of the cleanliness of the codebase and data for new stuff. Gradually fixing that...)
Good stuff to know!
I did ping Kenney a DM on social media but he didn't respond.
His licence allows us to use them but I'd be much happier getting his blessing first.
It's a worthwhile effort I think. The fact that you made these assets all work and can be searched and stuff. It's a great start, and it will encourage people to add the library
I had a little browse last night of all these similar 3D galleries with CC0 / BY licensing
A lot of them seem to be using high def models, but really good quality and I was surprised by what is available.
@yank told me to introduce myself. I've got multiplayer VR working as a plugin. If I could make this gallery plugin work with it, then we'd have a minimal version of Mozilla Hubs, where you used to spawn objects into the world from the google poly website.
Thanks Julian, thought this was a cool idea @andybak, and I didn't think at first about cache stuff but it became rather obvious because of how godot projects works.
In godot it's easy, we have a Resouce class built in which can hold ALL the data including a thumbnail and the model, or a path to it in the project. Then we can basically share those instead over a multiplayer scenario instead of 10 people downloading from source. So one person makes the download and sends it to the others.
But the cache made sense way before that scenario.. for example tracking which assets you have downloaded already, and caching thumbnails too.
This is a byproduct of making the gallery independent of running in the godot editor or in someone's project (runtime app). It could be an entire app by itself, or it could be embedded in someone's godot app. I didn't expect that to happen so quickly as it did
yep. it's worth noting that both Open Brush and Blocks aggressively cache these assets too. it's one of the reasons we moved towards making published assets on the Icosa gallery immutable (because a) that's how Poly worked and b) changing that would require reworking a bunch of stuff in Open Brush/Blocks)
But worried that we'll get pushback from artists about this - but let's see.
(actually - only Creative Commons assets are immutable. People can upload "all rights reserved" and do what they want. But we don't show those in api searches or public listers so there's less scope for breaking things for other people)
(related - we recently decided to rename visibility status "PRIVATE" to "DRAFT" as it better matches how we are using it. So it's public, unlisted and draft with the first two also being immutable)
@JulianTodd - hi by the way! i love the minimal multiplayer thing
will it also work in a browser? that would be a great feature
I'd love to host it as part of icosa.gallery at some point
I was looking into web multiplayer over the weekend. Networked AFrame hasn't been updated much for a while and alternatives to it seem a bit heavy.
godot has a html export, so the gallery could be exported to html too. i'd probably have to write a bit of extra stuff but not much. maybe not
godot is extremely everywhere. its really sank its teeth into getting on every platform and doing all this weird stuff lol
on a tangent here but worth a mention. was talking yesterday about using a godot export in html has this "JavaScriptBridge" where you can access the DOM of a webpage using gdscript and also evaluate javascript. so if someone was clever enough - they could gut out a lot of classes in godot and get the runtime down to around 6MB of WASM code and then you have godot as a web runtime that interacts with html elements. pretty crazy stuff for a game engine.
I really wish I had time to dive into Godot more.
I'd love to port this to the dotnet branch of Godot: https://github.com/IxxyXR/polyhydra-core
It's mostly vanilla C# currently - the Unity specific parts would be easy to shim out
Also note, you can contribute back changes that allow the project to support both Godot and Unity.
https://github.com/godotengine/godot/pull/104780 has been merged.
Thank you Aaron 😊
Impressive how coordinated godot is to get this fix pulled in so fast as well
I always wish it was faster but yeah 2 weeks is pretty good all things considered.
how does this endpoint work? It requires no parameters. always get a 401 on it
oh i see. I pass a token here
so far ive been able to login with the API, but not tested uploads or anything. currently making all the GUI for user browsing
@yank when it comes to uploads via the API, we currently have two endpoints. One is very much designed to work with Open Blocks, but the other, more general endpoint is: POST a zip file to /v1/users/me/assets much like you might do on the web front end. Let me know if that works out for you.
gotcha, I will try and get this working
POST
https://api.icosa.gallery/v1/users/me/assets
<zip file contents in bytes>
-H "User-Agent: Icosa Gallery Godot Engine / 1.0",
-H "accept: application/json",
-H "Authorization: Bearer (token)",
-H "Content-Type: application/zip"
I managed to upload a few assets, but I cant see it on the web to delete them
I got a 201 response each time, but I think I just made an empty file
nice, I think my request is bogus, I am just figuring out how to do this in godot
want me to delete those ones for you?
no, its ok, rather, I'm not sure how to use the upload endpoint
I call it like this, passing in the zip file as bytes into the request_raw function
do I need to pass any more data?
ask @James R tomorrow
us old people tend not work evenings 😛
of course! I have a bit of a spring in my step the last couple of days, managed to get loads done
been staying up way too late, though
morning @yank here's a curl command that works for me:
I assume you have a token. If not let me know.
OK, thanks I will try this today
got it working.
https://icosa.gallery/view/9HuUnG2ksXE
@andybak
lots more UX improvements to make on my end, but I have everything working now, searching, downloading, user profile, uploading, and also deleting assets :]
I guess the one thing missing here is being able to edit an assets parameters, like its licensing, name, description.
I can upload an asset, but I can't edit anything about it
unless there is more hidden API for this @James R ?
@yank Glad you got it working! We've not yet implemented
PATCH or PUT fully for asset properties. For now you need to use the web UI.
I'll let you know when this is done; pretty straightforward but there is some interesting logic around licensing.yes, I see you are unable to retroactively choose a more permissive license
that is great, I was thinking of making a nicer uploading GUI with a preview and thumbnail photo-taker, as well as using multiple files and packing a zip manually. so I could work on that in anticipation
the uploader i have now is just a simple file dialog that takes a zip, without any checking, upload progress, etc. so i have a a bit to do there to make it really nice
once we have
PATCH et al, you will probably be able to make an empty POST to /users/me/assets and then progressively PUT to your asset.Oh, I did notice there is a 500 error when I try to PATCH user data btw, but honestly, I think this will be the only endpoint i dont use. i think that stuff is better done on the site
i was just messing in the docs. with the example execution
I saw that in sentry, I'm working on that soon.
yeah no worries, I wont be using that one just so you know
sweet, I'll take my time on it 🙂
how long does a token last for btw?
This is fixed btw.
Just want to check, is it only the metadata like license, name etc that you need to edit here? Are you happy being able to upload a zip of all the other assets in one go?
Yes. and yes 👍
Being able to edit that metadata means godot users wont need to visit the site because everything is included then
There's some potentially complex logic and UI around changing license and visibility where we need to ensure we communicate clearly with the user as to "what" and "why". Unless you want to implement this in your addon and keep it up to date while it evolves - I'd suggest you don't allow editing of this. Just let them easily open up the edit page on the website for changes like that. It's how we're going to handle it in Open Brush and Open Blocks
Yeah if the license is the only thing you need to change on the site it could be fine
Why do you have this logic? What if someone makes a mistake?
example. assets that don't allow remixing are hidden from search results and listers
otherwise people will assume they can use them in their stuff
there's also the fact that you can't revoke licenses after assigning them. that needs to be explained carefully
we currently also have some rules around changing visibility that I can't remember off the top of my head - but we're still thinking through these
That’s what I’m asking. Why do this? Is this a real legal thing? I thought you could do that
It would be good to know in the future in case I make the same mistake
Before licensing Before you apply a CC license or CC0 to your work, there are some important things to consider: The licenses and CC0 cannot be revoked. This means once you apply a CC license to your material, anyone who receives it may rely on that license for as long as the material is protected by copyright, even if you later stop distributing it.https://creativecommons.org/share-your-work/cclicenses/ It's how CC licences are designed. Imagine if you could revoke them. How would you inform everyone that has ever used your asset? it would be unenforceable.
Ahh ok. I’m sure I’ve seen code projects change licenses half way
But no that makes sense
usually with code you can relicense it but people can stay on the version they had and stick to the license it had at that time
Ahh I see yeah. Assets usually only have 1 version
In a way
Not an evolving project
I see I see
So technically then, if someone uploads an asset it’s automatically ‘all rights reserved’
And they must make this choice to license it, and they must understand the choice and the consequences
I suppose the godot addon has limited legal responsibilities right now.
I’ve clearly marked each asset with CC stickers thing to help users understand what they are getting
Yeah OK, I would like to edit the metadata of an uploaded asset, except for the licensing
i think we default to "no license chosen" usually (which amounts to the same as "all rights reserved" legally)
Yeah that makes sense
Better wording
Will there be anything like timesDownloaded in the api @andybak ?
we don't distinguish "views" vs "downloads" (because we have no idea whether people are viewing or downloading in many cases). We have historical data for views. We track new views on the website. Not sure if we do the same for "views" via the api. @James R - i think it's tricky isn't it?
the download url is in the lister and might be hosted anywhere so we can't track it easily.
if you replace the download button on the website with an API call, then everyone using the gallery will be making API calls and you can track them
I supply a user agent header with Godot Icosa written on it
So you can track godot downloads at least
I suppose you can’t say if the user downloaded something successfully, but you can track when someone clicks the button at least
I got all the features working from the API on the gallery side of the addon now, not a lot left to do there.
I think it has a slight visual problem, it looks ok, but in the editor it has unexpected issues with theming
so i think I have to dig into that to make it look a bit nicer
Cool! I'm away today but around on Thursday to have a proper look.