Compendium loading

hmm ghost, I added some console.time tags here and there to measure the difference between loading only SRD data vs bigger compendiums (I tried the compendiums I have with stuff imported from ddbi) and I noticed that times lower significantly after the first calls on each of the compendium groups, these are cached midway I guess ? I am not aware of how the foundry infrastructure works inside. I mean.. I tried multiple times like this: 1) using only SRD compendiums, Times X 2) using DDBImporter compendiums, Times Y > X 3) repeat test 2 (without reloading client), Times Y² < Y thought Y² > X 4) repeat test 3, Times Y³ ~= Y²
C
ccjmk949d ago
moving this here: loading SRD compendiums on the tool took around 1177ms total, for all data loaded. DDBI compendiums took 1965ms. DDBI compendiums retry without client reload took 111ms
E
Ethaks949d ago
Yeah, that sounds like the caching Things to consider re: compendium loading. Once upon a time, when Foundry still did not load packs on init at all, there was a module called Compendium Browser that requested all packs entities. This caused the server to load all of them, which resulted in OOM issues and servers dying left and right. Thus, the module was nuked from existence 😄 Nowadays. the server already loads all packs to generate an index, which means everything gets loaded either way – there are no spikes afterwards though. So requesting all documents up front has less of a chance of outright killing the server, but requesting all documents will still add to the loading time by requesting data and then preparing the actual documents from that
C
ccjmk949d ago
hmm.. i see; i think my initial idea is not really good here, because I knew there might be some repetition (request the same compendium contents for more than one category, e.g. Races and Race Features are saved on the same compendium on the SRD, so i'm essentially getting those twice and parsing them, but I need to get them for each case, because the user might have its races and race features on different compendiums, i can't know beforehand) so i thought about loading all compendiums mentioned on Any of the data sources, and then dealing them to whatever source requested them to avoid repetition... eeh but the cache seems to be doing a lot better job than what I could do, probably haaha
E
Ethaks949d ago
This was for some kind of character creator or something like that, right?
C
ccjmk949d ago
yup, basicaly a charactermancer so i ask user to checkbox-select which compendiums on the world have Races, which have Classes, etc... and right now, and loading all that crap when the application starts (Application, the module, not foundry) plus, im still a little sidewinded with the whole Documents and Indexes and all the shenanigans from 8+... hmm.. would it be ok if I paste the code I'm using to load stuff ? it's basically a Function X by Ghost 😛 my app's lifecycle is rather dumb i think.. when the Application is created, I go through each tab and call some dataPrep() then render() methods where I call this Function X, and draw all I need to draw on screen. maybe it can shed some light to some alternatives
E
Ethaks949d ago
I had to leave for quite a while, sorry for that 😄 And sure, paste some code For your case you might be able to work some magic by requesting specific indexes and then prepare only that data, but if your app ends up scouring every single pack either way to get access to all features a character could need, that might not help all that much
C
ccjmk949d ago
np! so did I hahah so, as a mega quick overview of the lifecycle of my module: when you click the button to open the window, the Application is created and it iterates over each "tab" to load any data that tab needs, then uses that to draw stuff; that data load method for each tab is the one loading stuff from compendiums. There's several "data sources" set up in the module settings, so you can say "get Feats from here, but Classes from here, and Spells from there.. " and so on; in general, I expect each to be different compendiums (mostly because anything not-class nor-spell is a generic Feature item so far, so I need to split those semantically, rather than programatically), but specially in the case of Races|Subraces|Racial traits those can live on the same compendium (i was semi-forced to do this by the SRD 5e races compendium having all 3 stuff together 🤷‍♂️ ). Each data source is made by a list of all compendiums in the world, filtered by those the user selected the checkbox for. and when the time comes to load stuff, I use these methods:
/**
* @param source what type of data source it's been loaded - compendium names are fetched from the settings
*/
export async function getSources(source: keyof Source) {
const propValue: Source = (await game.settings.get(Constants.MODULE_NAME, SettingKeys.SOURCES)) as Source;
const packs = propValue[source];
const selectedPacks = Object.keys(packs).filter((p: string) => (packs as any)[p]);
const ret = await getItemListFromPackListByNames(selectedPacks); // this is Ghost's Function X
return ret;
}

// where Source is..
export interface Source {
races: any;
racialFeatures: any;
classes: any;
classFeatures: any;
backgroundFeatures: any;
spells: any;
feats: any;
}
/**
* @param source what type of data source it's been loaded - compendium names are fetched from the settings
*/
export async function getSources(source: keyof Source) {
const propValue: Source = (await game.settings.get(Constants.MODULE_NAME, SettingKeys.SOURCES)) as Source;
const packs = propValue[source];
const selectedPacks = Object.keys(packs).filter((p: string) => (packs as any)[p]);
const ret = await getItemListFromPackListByNames(selectedPacks); // this is Ghost's Function X
return ret;
}

// where Source is..
export interface Source {
races: any;
racialFeatures: any;
classes: any;
classFeatures: any;
backgroundFeatures: any;
spells: any;
feats: any;
}
and the actual data loading..
export async function getItemListFromPackListByNames(packNames: string[]) {
const allItems = [];
for (const compendiumName of packNames) {
const pack = game.packs.get(compendiumName);
const worldItems = game.items;
if (!worldItems) throw new Error('game.items not initialized yet');
if (!pack) ui.notifications?.warn(`No pack for name [${compendiumName}]!`);
if (pack?.documentName !== 'Item') throw new Error(`${compendiumName} is not an Item pack`);
const itemPack = pack as CompendiumCollection<CompendiumCollection.Metadata & { entity: 'Item' }>;
const itemsPromises: Promise<Item | null | undefined>[] = [];
for (const itemIndex of pack.index.keys()) {
const item = itemPack.getDocument(itemIndex);
itemsPromises.push(item);
}
const items = await Promise.all(itemsPromises);
allItems.push(
...items
.filter((item): item is Item => !!item)
.map((item) => {
const itemFromCompendium = worldItems.fromCompendium(item);
// intentionally adding the flag without using the API as I don't want to persist this flag
// this should be enough and more lightweight - this id/link is used to create the entity-link popups
// as freshly created items or some ddbi imported ones don't seem to have the right core flag
itemFromCompendium.flags.hct = {
link: {
id: item.id,
pack: item.pack,
},
};
return itemFromCompendium;
}),
);
}
return allItems;
}
export async function getItemListFromPackListByNames(packNames: string[]) {
const allItems = [];
for (const compendiumName of packNames) {
const pack = game.packs.get(compendiumName);
const worldItems = game.items;
if (!worldItems) throw new Error('game.items not initialized yet');
if (!pack) ui.notifications?.warn(`No pack for name [${compendiumName}]!`);
if (pack?.documentName !== 'Item') throw new Error(`${compendiumName} is not an Item pack`);
const itemPack = pack as CompendiumCollection<CompendiumCollection.Metadata & { entity: 'Item' }>;
const itemsPromises: Promise<Item | null | undefined>[] = [];
for (const itemIndex of pack.index.keys()) {
const item = itemPack.getDocument(itemIndex);
itemsPromises.push(item);
}
const items = await Promise.all(itemsPromises);
allItems.push(
...items
.filter((item): item is Item => !!item)
.map((item) => {
const itemFromCompendium = worldItems.fromCompendium(item);
// intentionally adding the flag without using the API as I don't want to persist this flag
// this should be enough and more lightweight - this id/link is used to create the entity-link popups
// as freshly created items or some ddbi imported ones don't seem to have the right core flag
itemFromCompendium.flags.hct = {
link: {
id: item.id,
pack: item.pack,
},
};
return itemFromCompendium;
}),
);
}
return allItems;
}
E
Ethaks948d ago
I got a tiny little bit side tracked by another module I wanted to implement a feature for and might have forgotten about this after I initially read the message this morning 😅 How does 5e distinguish between races and features, is there a field in the item's data that shows "This is a race, and that is a racial feature"?
C
ccjmk948d ago
there is Nada I figured it out (sort of, there's some forced cases*) in that Elf has no requirement, so its a parent class (or a class without subclasses, like Dragonlord or Half-Elf), and subclasses have the parent class as Requirement, so like High Elf has "Elf" as requirement, while ALSO having "Elf" as part of the name .. High Elf, and features also have the race name as Requirement but don't have it in their names, like Fey Ancestry having 'Elf' as requirement, or Tinker that has 'Rock Gnome' thought 'Gnome Cunning' and 'Halfling Nimbleness' are exceptions because they do have a race name on the name, but are not subraces :/ I just filtered those out by hand so: * not-a-subrace race: data.requirement empty * subrace: data.requirement has a not-a-subrace-name && name.contains(the-same-not-a-subrace-name) * race trait: data.requirement has a not-a-subrace-name && name.NOTcontains(the-same-not-a-subrace-name)
Want results from more Discord servers?
Add your server
More Posts
Skill and Ability bonusesA massive enhancement of Skills and Ability Checks/Saves allows them to be individually affected by 150 Sheet Changes@sdenec @lordzeel Sheet Changes required: - New Cog-menu for Ability Scores and Skills to support neDocumentData shennaneginsTIL: It is possible to `update` an Actor or Item's `data.data` with arbitrary information that isn'tflag shenanneginsTIL you can set the `flags` key on a document to whatever you want to (e.g. a string). This is a surraaeAfter some discussion yesterday in #active-effects , I decided to try to create a system-agnostic modevMode json changed warningI checked the module repo and I honestly can't figure out where I would plug in my code. I'd be guesS3 File Picker SettingsCan someone with an S3 configuration give me a test of the FilePicker settings api and tell me if a item preCreate@sol.folango @mrprimate (pinging you two in particular because you do import stuff involving existinItem Macro Compendium WorkflowOkay, here's a long one that's a bit of a doozy. I'm looking at setting up some sort of tooling/workItem Specific Crit DetailsOh that critical hit thing is gonna hit MRE too isn't it... hrm...1.5.x 72%@dnd5e No action required (but suggested 🙂 ) The 1.5.0 milestone is ~72% complete. It has a due daV9 Tabs IssueIf you never figured this out, here's why this happened: A small change in `Tabs` during v9 causes sDeck Creation MacroI created a macro to fill out a 52-card deck, you set the ID of the deck and the base URL of the foldevmode-issues@arcanist figured out, the flex layout elements have `pointer-events: none`, so anything injected wifunction vs methodAnyone know of a good guide that explains the difference between a function and a method? I don't reActive Effect HelpAnd it should be the actual ActiveEffect document for the effect, not just the id. I don't have theTOL JEs@badgerwerks I haven't had a chance to poke around in TOB/BoL yet, but I wanted to ask how you did tCI 2 minute windowAlright, I've got a Gitlab CI pipeline I'm reasonably pleased with as a result of yesterday's effortGitLab CI PipelinesToday's project is Gitlab CI. Wow me with your pipelines if you got em.Storybook Shennanegins@wyrmisis I have a headcannon for you, one who also uses Storybook. _What if_ we could load foundry