Lancer system & TRL

Let's move the conversation over to a forum post as I have a feeling there could be a few messages. When I'm done w/ lunch I'll finish this initial post. 😄

Yeah.. I was going to comment on that.. ;P I've taken a peak at the system repo from time to time to see what is going on.. SvelteKit is definitely cool, but doesn't really have a place in respect to Foundry dev as FVTT dev is all 3rd party client code launched from the Foundry server.

I am interest though in svelte-package / part of SvelteKit to see if that will work with component library distribution. https://kit.svelte.dev/docs/packaging

When I mention getting TRL to run on & off Foundry running on top of SvelteKit independently is where I'm headed with that; IE TRL providing the final mile / client side desktop app like API.

No.. It's pure Svelte... SvelteKit is more of a way to run a server / routing / SSR for a web app. I'm sure they will be adding authentication soon. TRL is just a plain Svelte component library + other goodies all bundled together to provide a cohesive app dev loop.

For Foundry dev w/ Svelte it's definitely not something I think fits into the picture. I'm certainly glad they are progressing w/ the final release though as a lot of the Svelte maintainers switched to working on SvelteKit and Svelte itself needs a little more love and attention and should be getting some PRs and new support in.. I have a PR for adding container query support that should get in soon.

It's good to experience the core API and push it to where it doesn't seem like a good fit. I had this experience w/ fully building out and polishing FQL / Forien's Quest Log. It's about as complex of a module that you'd want to build w/ the core APIs.

Heh heh.. I'm going to finish lunch though and BRB for more chat in ~20-30 mins.. ;P

regarding authentication:

NextAuth.js has recently rebranded as Auth.js and is currently in the process of porting the entire library over to WebStandard API (ditching any and all node deps) and framework agnosticity. In the wake of this, Auth.js Svelte has been released a few months ago

I see that LANCER uses Typescript. I couldn't get typescript to work with TRL. I can not find setting that'll let me import @typhonjs-fvtt/runtime/application

There is typedefs for TyphonJS in the NPM package, but I can't for the life of me actually get them to work

If there were accurate and up to date Foundry community types that would make it much easier to enable TS declarations for the application sub-module export. IE the one that has SvelteApplication (extends Application). Besides not being up to date the community types are built in a very heavy handed way pulling in all sorts of unnecessary dependencies behind the scene for everything that Foundry exposes. IE if the community types were bundled and had better tooling instead of the kitchen sink approach that is the current arrangement things would be a lot smoother.

I just can't tightly couple the community types w/ TRL... What do I do only ship v9 types / hard couple that to a specific release of TRL?

Essentially everything else in TRL has types except the meeting point between TRL & Foundry. I built a pretty slick ESM to TS declaration tool that generates bundled types from ESM code / JSDocs.

The problem I am facing is that I can not import the existing available typedef for TRLs own code

As such I am unable to import TRL at all in a .ts file

A package.json export that has no "types" value and contains only ".js" files appears to be invisible to TSC, at least with the setting options I found

Also the heaviness of foundry types is in large parts due to the unfortunate shift in type declaration management that lead to @types being largely abandonned by the community. Typedefs are now found directly in the npm module, forcing fvtt-types to import those modules in their entirety

I'm not an expert w/ TS either though do use it from time to time. One of the external libraries that I wrote that is included in TRL was a perfect fit for TS. My general concern is where this can fit on the list of engineering priorities. It's unclear if the Foundry community types are going to survive in the future. v10 not done yet and v11 already here imminently, etc.

I'd have to do some engineering analysis on what it would take to get something effectively working for the Foundry bridge part. My general concern is that right now that could be ~1+ month of effort where there are more valuable engineering aspects that have greater impact.

The problem I am facing is that I can not import the existing available typedef for TRLs own code

We'll definitely have to confirm / discuss this separately. TRL does use NPM sub-package exports w/ types defined correctly.

You can see in the TRL main distribution that types are defined for all sub-package exports except for the parts that touch Foundry / extend from Foundry classes.
https://github.com/typhonjs-fvtt-lib/typhonjs/blob/main/package.json#L49-L57

Typedefs are now found directly in the npm module, forcing fvtt-types to import those modules in their entirety

You can be more clever and bundle types w/ Rollup and other tooling. IE you have them as dev dependencies and a build step in the community types to bundle all of the external packages into a single distributable of just the relevant type declarations.

Definitely a bit optimistic and too strict approach where. It's more like the community types were a philosophical take on exactness than something that is practical and usable.

They appear to be focusing a lot on type inference sugar instead of prioritizing just getting functioning types out

In any case, the issue with using TRL in a TS based project, like LANCER, is that I can not find a setting that'll let me import those three submodules

The best way forward may be like you suggested to have some sort of facade types that are minimal that are replaced with whatever actual full types / the community types or otherwise. There is not a whole lot of things that need this facade, so if there was a way to not tightly couple any given set of types for the parts that touch Foundry that would be ideal.

Hard to say on any code being cracked. The dependency angle / not bundled distribution for the community types IMHO is a hard blocker too. I played around with seeing if I could get bundling of the community types to work, but a lot of esoteric TS foo is applied in the community types that seemingly made that not work out of the box. It would take some time to untangle that aspect.

My expertise of mix-and-match types is a bit limited, but it should theoretically be possible to decalre types in a way that allow a manually imported fvtt-types to complete the typing for you

A bare bones approach could be something like this (Plan C): https://medium.com/@steveruiz/using-a-javascript-library-without-type-declarations-in-a-typescript-project-3643490015f3

Yeah funnily enough, TS being TS, Plan C does not work with the clean module import because TSC decides that exports with no corresponding typedef dont exist

it'd only work if you directly imported the .js file from the node_modules directory

I probably have to add that bare bones declaration as part of the TRL NPM package / define the types field in the sub-package export.

What you could do is import the typedef for Application from fvtt-types and define it as an optional dependency

Or rather, not actually import it because it's put into global

I believe that optional dependencies are still installed by NPM, but I'd have to verify.

it depends strongly on package manager.
optional peerDeps as defined via peerDependenciesMeta should not be installed automatically

some npm versions appear to errorneously do it anyway, but that apparently is a bug

In general I think the takeaway is that yes, I do want to make TS possible w/ TRL & Foundry. A prior big problem until recently w/ TS was that it didn't work at all w/ NPM sub-package exports and that only got resolved recently. I think TS ~4.8.5 or so; this support got pushed back several major TS versions... It can certainly help to have those that are interested in TS to help test and come up with a game plan.

I'll gladly help out. Although personally i've been looking into replacing TS with JSDoc, TS is still quite nice

TRL is mostly JSDoc, but accurate JSDoc unlike some Foundry core aspects.

Of interest is my build tool that generates TS declarations from ESM source code: https://github.com/typhonjs-node-build-test/esm-d-ts

I think until we have full TS compatibility, this thread is at its limit though

-------

Getting back to the main points that you raised earlier:

- Ideally we'd do this in as piecemeal a way as possible
- As mentioned, our sheets are very large

There is backward compatible legacy support for Handlebars for Application / FormApplication by HandlebarsApplication & HandlebarsFormApplication. This keeps the App v1 handlebars rendering intact, but gives you the power of the TRL application shell / full reactivity for most things regarding the application shell.
https://github.com/typhonjs-fvtt-lib/svelte/tree/main/src/application/legacy

More or less it should work by replacing Application / FormApplication w/ the above and no other changes. Indeed each app would need to be piecemeal re-implemented as pure Svelte eventually, but there are initial benefits to the TRL legacy support over stock App v1 API.

- Our rough plan is to essentially just convert any handlebars helper into a Svelte component.

TRL already provides a couple of converted Foundry handlebars helpers. I can add more as necessary:
https://github.com/typhonjs-fvtt-lib/svelte/tree/main/src/helper/legacy

- Event handlers seem like they would be problematic, as any applyListeners step would likely not survive well under svelte re-generating the DOM.

- I don't really know a tidy way around this aspect

It's going to be a lot of work, but if done in a piecemeal manner quite possible. I'd certainly like to hear more about the current refactor objectives that you mentioned regarding the current rewrite of LANCER.

-------

The next TRL release is focusing on some of the remaining aspects useful for system dev. I already have implemented "reactive embedded collections", but am adding capability to do drag & drop with them the "Svelte way". I'm also buffing up the reactive document wrapper / TJSDocument to provide a mechanism to automatically create stores for individual document properties / auto-update / make it easy to connect document properties to Svelte components transparently without the management overhead. These things should be the last major aspects that really will supercharge Svelte actor / item sheets, etc.

Yeah.. Let's stay in touch. I'm all for finding that pathway for TS packages. If I had to guess ~May would be the approximate time if not sooner that something can get put together. It's a bit hard to schedule / prioritize TS angles w/ the complications from TS itself (no support for subpackage NPM exports until recently), no reliable / solid Foundry TS types and such.

Working on actual new features for TRL always edged out finalizing a TS strategy / implementation. I think a rough estimate is that there are less than 10% of packages on Foundry that use TS, so there is that too; a smaller community to serve, etc.

----

So one of the problems that I encountered w/ trying to add types to the application sub-package exports is that in WebStorm this removes intellisense w/ linking the Foundry client code. IE WebStorm seemingly always tries to open up the type declarations and not the linked Foundry source code which is not desirable for the majority working in ESM.

I do auto-generate albeit not precise types for these sub-package exports and they are in _types.

There may be a way to point to them from tsconfig from the local project, but won't be an immediate solution as the auto-generated types do not include any Foundry API. IE SvelteApplication extending "Application" the auto-generation has no knowledge of the Foundry level. This is where perhaps very minimal definition of Foundry types could come into play. I suppose declaration merging is the thing to look into.

https://github.com/typhonjs-fvtt-lib/typhonjs/tree/main/_types/svelte/application

And yeah.. It looks like there is no declaration merging for classes, so this likely shoots down any minimal definitions in TRL that could merge w/ say the Foundry community types. https://www.typescriptlang.org/docs/handbook/declaration-merging.html#disallowed-merges

posting this in the right place... I put it in the chat lounge:

To be fair I didn't update those separated types w/ the latest TRL releases. So they will be old. It was kind of a first attempt to get something running. I don't think it will work well or at all w/ what is currently there and needs some work likely through a combination of auto-generation and by hand.

You can checkout: https://github.com/typhonjs-fvtt-lib/svelte
- npm install
- uncomment these lines for the auto-generation: https://github.com/typhonjs-fvtt-lib/svelte/blob/main/rollupAll.js#L294-L308
- run the NPM build script
- copy the _types to your local project ./node_modules/@typhonjs-fvtt/runtime/_types directory

In general the problem comes down to it likely taking quite a bit of effort / time to get this working for a handful of folks using TS. Without knowing all the details this could be a massive time sink. What definitely could be helpful is getting folks that need TS support more looped into the process of finding a solution. I know no-one wants to put in any extra time on this... Just a bummer as I fear it could take me ~1+ months just to arrive on a solution when there are other higher priority things to finish off at this time.

Yeah... Not only is what is currently there out of date, but it's not going to work well at all as the JSDocs for that section of code (which the current types are auto-generated from) reference the library linked Foundry code via WebStorm / similar to symlinking foundry.js, so not TS Foundry types. This all works fine and dandy for ESM developers (90%+ of the 3rd party dev community). A further complication last time I tried to link things in package.json is that it messes up the ESM dev experience by jumping to types instead of the source code. Those sub-package exports are not bundled / raw source and intended for folks to be able to click through to the source code and then to the Foundry code if you set up a library in WebStorm. It gives you full intellisense w/ ESM through TRL to the Foundry client code.

The previous discussion is finding out some sort of facade / minimal types that are merged with whatever actual Foundry community types or otherwise might be used, but there are likely complications to get both the ESM dev experience top notch at the same time as TS.. That is why I think it is a month+ of effort. It's not just getting TS support, but making sure both the ESM & TS support is top notch.

Also... A good thing to know... TS requires the types field to be the first entry otherwise it doesn't work...

    "./svelte/application": {
      "types": "./_types/svelte/application/index.d.mts",
      "import": "./_dist/svelte/application/index.js"
    },

Yeah... It's pretty effed... And not clear at all; not a lot of good resources besides scouring tons of TS issues related to sub-package exports; which I have done during the drawn out implementation for sub-package exports support being added to TS.

On a positive front there has been quite a bit of improvements in WebStorm in the past couple of releases for handling types through NPM packages and a lot of other IDE related intellisense issues and is improving every release. I haven't had the time to setup VSCode to test anything.

This definitely is one of those final mile to "beta" things to resolve which is an ~end of the year thing more or less or early next year. "beta" is hit when I have TRL running on and off Foundry, so lots more work taking priority before that final stretch for my direct involvement. For the Foundry version of TRL anything that touches / extends Foundry code is just a pain to get working for TS right now from a library construction perspective without official up to date types.

Heh heh.. Join the dark side... WebStorm... At least worth trying it out via the OSS license just to give it a spin. Lots of standard dev setup / linking Foundry code / debugging setup w/ TRL / your project code & Foundry client code is rock solid & easy to setup.

There are challenges for both IDEs though in respect to the discussion related to Foundry types. I can use help in prototyping what a potential minimal / dummy Foundry types arrangement could look like that is merged w/ say the community types. If that step could be demonstrated / other dev get something minimally functional then that could kick start things. It's just a big context switch for where I'm at w/ the rest of the engineering higher up on the TODO list.

At the end of the day... In general because there are no official Foundry types and the future of the community types is in question it's generally reasonable to consider switching to ESM. At least w/ WebStorm everything works great and as long as you write good JSDocs I have had zero problems w/ intellisense or the types of things folks like Typescript for... IE no bugs in all of TRL, but that is just diligent coding. TS does make sense at a larger organization with novice coders in the mix when all the libraries / code has TS types, etc.

For the off Foundry aspect of TRL the larger Svelte community will expect full types and that will be easy to provide when Foundry is not in the mix.

The not using the exports path thing is definitely a TS quirk as technically anything not defined in the "exports" rules of "package.json" should not be allowed. Though that is more a NodeJS runtime aspect.

The types currently in @typhonjs-fvtt/runtime/_types are old as mentioned here with general instructions on how to regenerate them though it's not going to be the solution: https://discord.com/channels/737953117999726592/1078418639461560363/1078486893278285866

_types was my initial attempt at getting this working, but clearly is not the way re: auto-generation.

If you have the time the two big areas of general types / TS config to research are:

1) Can a minimal shim FVTT types package be put together with just what is required to cleanly do declaration merging with the full community FVTT types. A minimal demo / proof of concept would be great!

2) Is it possible to configure TSC / tsconfig.json to recognize side-loaded types for those sub-package exports without assigning a types field to the export entry in package.json.

-----------

The idea is to auto-generate the types from ESM; hand modify them w/ the FVTT types shim, be able to side-load them for a sub-package export and merge w/ the community types.

This may be necessary for the ESM / core library linking aspects for IDE discovery w/ Foundry in particular. I'll investigate the best approach given #1 above though as perhaps it is possible to use the types field.

The copying of the application packages makes it easier to read the code when you click through to it from the IDE rather than one single bundled file.

The copying of the component package is so that the .svelte files are compiled for the proper runtime in your project. When you run the dev server you can't use pre-compiled Svelte components as the way automatic Svelte HMR works is by using a proxy SvelteComponent parent during compilation.

A fair amount of the other sub-package exports import various libraries together so are bundled themselves to pull in everything.

At the end of a production build in your project you of course control the bundling / minification, so there is no reason to necessarily bundle / compress / pre-process TRL and such where not necessary.

I will be changing things around for the Svelte components using SvelteKit package later this year.

----

Re: package.json in each dist / sub-package. It should neither hurt / nor hinder anything. I got started on TRL just before sub-package exports. It could be left over from those early days, but the Svelte library also does the same, so um maybe I copied that when running into problems in the early days of sub-package export support. It certainly can be evaluated if necessary and if not removed now that there is more tooling support.

esm-d-ts is pretty solid and works well. It does use TSC after all. For the application sub-packages that touch Foundry the actual global references for Foundry API are used in JSDoc. This works for ESM especially well in WebStorm when linking the Foundry client code to your project, but doing this does not produce anything meaningful for Typescript for auto-generation from that JSDoc.

That explains why the #1 & #2 "research" steps are being pursued as described.

Auto-generate the declarations for the application sub-package exports, manually hand modify them to add the shim / correct things, then hopefully those declarations can be merged w/ whatever Foundry types are in the end developer downstream project build.

That is Rollup related artifact as the types are bundled. This doesn't occur in the other auto-generated / bundled sub-packages. I haven't re-generated those types since updating a lot the backing tech / switching from Rollup 2 to 3 and such, so something to look into. IE the existing old _types doesn't show this symptom.

That is a very old issue for a very early version of Rollup. It could be related, but yes it is a naming conflict thing. I recall having to fix something like that before w/ the auto-generation, but it could be a combination of different things causing it now. While useful to correct it won't solve the issue at hand described by #1 / #2 above.

You can see the unbundled d.ts files in the ./dts directory that are generated. The last one generated is what remains afterward.

Types that can be associated w/ the sub-package export without explicitly defining the "types" field in the export rule.

---

This is more to do w/ maintaining the ESM dev experience from the IDE and being able to click directly through to the source code & have the JSDocs w/ the linked Foundry client code all behave correctly.

It's possible that this might be fixed up w/ recent versions of WebStorm / better support of sub-package exports. So need to revisit things again.

WebStorm understands the "types" field and defaults to going to the types instead of the ESM source or at least it did back when support for sub-package exports was initially supported. You could be doing ESM coding click on SvelteApplication and it would only take you to the "broken" types w/ no option to view the source instead.

Currently, if you click through it goes to the source code then you can click on Application and w/ the Foundry client code linked it jumps to the Foundry client code.

The reason to look into #1 though is that it should be possible to create a standalone proof of concept shim to merge w/ the community Foundry types even if it was for just a single Foundry API class say Application.

I'll look into the $1 thing this upcoming week though.