Compendium loading

hmm ghost, I added some console.time tags here and there to measure the difference between loading only SRD data vs bigger compendiums (I tried the compendiums I have with stuff imported from ddbi) and I noticed that times lower significantly after the first calls on each of the compendium groups, these are cached midway I guess ? I am not aware of how the foundry infrastructure works inside. I mean.. I tried multiple times like this: 1) using only SRD compendiums, Times X 2) using DDBImporter compendiums, Times Y > X 3) repeat test 2 (without reloading client), Times Y² < Y thought Y² > X 4) repeat test 3, Times Y³ ~= Y²
9 Replies
ccjmk
ccjmk3y ago
moving this here: loading SRD compendiums on the tool took around 1177ms total, for all data loaded. DDBI compendiums took 1965ms. DDBI compendiums retry without client reload took 111ms
Ethaks
Ethaks3y ago
Yeah, that sounds like the caching Things to consider re: compendium loading. Once upon a time, when Foundry still did not load packs on init at all, there was a module called Compendium Browser that requested all packs entities. This caused the server to load all of them, which resulted in OOM issues and servers dying left and right. Thus, the module was nuked from existence 😄 Nowadays. the server already loads all packs to generate an index, which means everything gets loaded either way – there are no spikes afterwards though. So requesting all documents up front has less of a chance of outright killing the server, but requesting all documents will still add to the loading time by requesting data and then preparing the actual documents from that
ccjmk
ccjmk3y ago
hmm.. i see; i think my initial idea is not really good here, because I knew there might be some repetition (request the same compendium contents for more than one category, e.g. Races and Race Features are saved on the same compendium on the SRD, so i'm essentially getting those twice and parsing them, but I need to get them for each case, because the user might have its races and race features on different compendiums, i can't know beforehand) so i thought about loading all compendiums mentioned on Any of the data sources, and then dealing them to whatever source requested them to avoid repetition... eeh but the cache seems to be doing a lot better job than what I could do, probably haaha
Ethaks
Ethaks3y ago
This was for some kind of character creator or something like that, right?
ccjmk
ccjmk3y ago
yup, basicaly a charactermancer so i ask user to checkbox-select which compendiums on the world have Races, which have Classes, etc... and right now, and loading all that crap when the application starts (Application, the module, not foundry) plus, im still a little sidewinded with the whole Documents and Indexes and all the shenanigans from 8+... hmm.. would it be ok if I paste the code I'm using to load stuff ? it's basically a Function X by Ghost 😛 my app's lifecycle is rather dumb i think.. when the Application is created, I go through each tab and call some dataPrep() then render() methods where I call this Function X, and draw all I need to draw on screen. maybe it can shed some light to some alternatives
Ethaks
Ethaks3y ago
I had to leave for quite a while, sorry for that 😄 And sure, paste some code For your case you might be able to work some magic by requesting specific indexes and then prepare only that data, but if your app ends up scouring every single pack either way to get access to all features a character could need, that might not help all that much
ccjmk
ccjmk3y ago
np! so did I hahah so, as a mega quick overview of the lifecycle of my module: when you click the button to open the window, the Application is created and it iterates over each "tab" to load any data that tab needs, then uses that to draw stuff; that data load method for each tab is the one loading stuff from compendiums. There's several "data sources" set up in the module settings, so you can say "get Feats from here, but Classes from here, and Spells from there.. " and so on; in general, I expect each to be different compendiums (mostly because anything not-class nor-spell is a generic Feature item so far, so I need to split those semantically, rather than programatically), but specially in the case of Races|Subraces|Racial traits those can live on the same compendium (i was semi-forced to do this by the SRD 5e races compendium having all 3 stuff together 🤷‍♂️ ). Each data source is made by a list of all compendiums in the world, filtered by those the user selected the checkbox for. and when the time comes to load stuff, I use these methods:
/**
* @param source what type of data source it's been loaded - compendium names are fetched from the settings
*/
export async function getSources(source: keyof Source) {
const propValue: Source = (await game.settings.get(Constants.MODULE_NAME, SettingKeys.SOURCES)) as Source;
const packs = propValue[source];
const selectedPacks = Object.keys(packs).filter((p: string) => (packs as any)[p]);
const ret = await getItemListFromPackListByNames(selectedPacks); // this is Ghost's Function X
return ret;
}

// where Source is..
export interface Source {
races: any;
racialFeatures: any;
classes: any;
classFeatures: any;
backgroundFeatures: any;
spells: any;
feats: any;
}
/**
* @param source what type of data source it's been loaded - compendium names are fetched from the settings
*/
export async function getSources(source: keyof Source) {
const propValue: Source = (await game.settings.get(Constants.MODULE_NAME, SettingKeys.SOURCES)) as Source;
const packs = propValue[source];
const selectedPacks = Object.keys(packs).filter((p: string) => (packs as any)[p]);
const ret = await getItemListFromPackListByNames(selectedPacks); // this is Ghost's Function X
return ret;
}

// where Source is..
export interface Source {
races: any;
racialFeatures: any;
classes: any;
classFeatures: any;
backgroundFeatures: any;
spells: any;
feats: any;
}
and the actual data loading..
export async function getItemListFromPackListByNames(packNames: string[]) {
const allItems = [];
for (const compendiumName of packNames) {
const pack = game.packs.get(compendiumName);
const worldItems = game.items;
if (!worldItems) throw new Error('game.items not initialized yet');
if (!pack) ui.notifications?.warn(`No pack for name [${compendiumName}]!`);
if (pack?.documentName !== 'Item') throw new Error(`${compendiumName} is not an Item pack`);
const itemPack = pack as CompendiumCollection<CompendiumCollection.Metadata & { entity: 'Item' }>;
const itemsPromises: Promise<Item | null | undefined>[] = [];
for (const itemIndex of pack.index.keys()) {
const item = itemPack.getDocument(itemIndex);
itemsPromises.push(item);
}
const items = await Promise.all(itemsPromises);
allItems.push(
...items
.filter((item): item is Item => !!item)
.map((item) => {
const itemFromCompendium = worldItems.fromCompendium(item);
// intentionally adding the flag without using the API as I don't want to persist this flag
// this should be enough and more lightweight - this id/link is used to create the entity-link popups
// as freshly created items or some ddbi imported ones don't seem to have the right core flag
itemFromCompendium.flags.hct = {
link: {
id: item.id,
pack: item.pack,
},
};
return itemFromCompendium;
}),
);
}
return allItems;
}
export async function getItemListFromPackListByNames(packNames: string[]) {
const allItems = [];
for (const compendiumName of packNames) {
const pack = game.packs.get(compendiumName);
const worldItems = game.items;
if (!worldItems) throw new Error('game.items not initialized yet');
if (!pack) ui.notifications?.warn(`No pack for name [${compendiumName}]!`);
if (pack?.documentName !== 'Item') throw new Error(`${compendiumName} is not an Item pack`);
const itemPack = pack as CompendiumCollection<CompendiumCollection.Metadata & { entity: 'Item' }>;
const itemsPromises: Promise<Item | null | undefined>[] = [];
for (const itemIndex of pack.index.keys()) {
const item = itemPack.getDocument(itemIndex);
itemsPromises.push(item);
}
const items = await Promise.all(itemsPromises);
allItems.push(
...items
.filter((item): item is Item => !!item)
.map((item) => {
const itemFromCompendium = worldItems.fromCompendium(item);
// intentionally adding the flag without using the API as I don't want to persist this flag
// this should be enough and more lightweight - this id/link is used to create the entity-link popups
// as freshly created items or some ddbi imported ones don't seem to have the right core flag
itemFromCompendium.flags.hct = {
link: {
id: item.id,
pack: item.pack,
},
};
return itemFromCompendium;
}),
);
}
return allItems;
}
Ethaks
Ethaks3y ago
I got a tiny little bit side tracked by another module I wanted to implement a feature for and might have forgotten about this after I initially read the message this morning 😅 How does 5e distinguish between races and features, is there a field in the item's data that shows "This is a race, and that is a racial feature"?
ccjmk
ccjmk3y ago
there is Nada I figured it out (sort of, there's some forced cases*) in that Elf has no requirement, so its a parent class (or a class without subclasses, like Dragonlord or Half-Elf), and subclasses have the parent class as Requirement, so like High Elf has "Elf" as requirement, while ALSO having "Elf" as part of the name .. High Elf, and features also have the race name as Requirement but don't have it in their names, like Fey Ancestry having 'Elf' as requirement, or Tinker that has 'Rock Gnome' thought 'Gnome Cunning' and 'Halfling Nimbleness' are exceptions because they do have a race name on the name, but are not subraces :/ I just filtered those out by hand so: * not-a-subrace race: data.requirement empty * subrace: data.requirement has a not-a-subrace-name && name.contains(the-same-not-a-subrace-name) * race trait: data.requirement has a not-a-subrace-name && name.NOTcontains(the-same-not-a-subrace-name)