C#•3mo ago

✅ Writing a source generator for this use case

I have some JSON data I want to load at startup, but image size has become a concern since in the published image these files are taking up 80MB. The general pattern I am using at the minute is

public static class MasterAsset
{
    public static MasterAssetData<Charas, CharaData> CharaData { get; private set; } = null!;

    public static async Task LoadAsync()
    {
        ValueTask<MasterAssetData<Charas, CharaData>> charaData = MasterAssetData.LoadAsync<
            Charas,
            CharaData
        >("CharaData.json", x => x.Id);

        ValuteTask<MasterAssetData<Other, Other>> other = ...
        
        CharaData = await charaData;
        Other = await other;
    }
}

public static class MasterAsset
{
    public static MasterAssetData<Charas, CharaData> CharaData { get; private set; } = null!;

    public static async Task LoadAsync()
    {
        ValueTask<MasterAssetData<Charas, CharaData>> charaData = MasterAssetData.LoadAsync<
            Charas,
            CharaData
        >("CharaData.json", x => x.Id);

        ValuteTask<MasterAssetData<Other, Other>> other = ...
        
        CharaData = await charaData;
        Other = await other;
    }
}

Note that there are about 100 such properties and I am interleaving the async calls to load them in parallel. Here the MasterAssetData.LoadAsync factory takes the JSON file which is an array and processes it as a dict:

public static class MasterAssetData
{
    public static async ValueTask<MasterAssetData<TKey, TItem>> LoadAsync<TKey, TItem>(
        string path,
        Func<TItem, TKey> keySelector
    )
        where TItem : class
        where TKey : notnull
    {
        await using FileStream fs = File.OpenRead(path);

        List<TItem> items =
            await JsonSerializer.DeserializeAsync<List<TItem>>(fs);

        FrozenDictionary<TKey, TItem> frozenDict = items
            .ToDictionary(keySelector, x => x)
            .ToFrozenDictionary();

        return new MasterAssetData<TKey, TItem>(frozenDict);
    }
}

public static class MasterAssetData
{
    public static async ValueTask<MasterAssetData<TKey, TItem>> LoadAsync<TKey, TItem>(
        string path,
        Func<TItem, TKey> keySelector
    )
        where TItem : class
        where TKey : notnull
    {
        await using FileStream fs = File.OpenRead(path);

        List<TItem> items =
            await JsonSerializer.DeserializeAsync<List<TItem>>(fs);

        FrozenDictionary<TKey, TItem> frozenDict = items
            .ToDictionary(keySelector, x => x)
            .ToFrozenDictionary();

        return new MasterAssetData<TKey, TItem>(frozenDict);
    }
}

I wanted to write a console app to run at build time, to convert all the json files to MessagePack with LZ4 compression. I initially used Roslyn to attempt to parse the contents of the MasterAsset class as a standalone cs file (so no symbol info) to build an association between json filepath -> type name, e.g. CharaData.json -> CharaData, so that I could first load up the JSON to deserialize it before re-serializing as binary, but I ran into a number of stumbling blocks trying to go from that type name to an actual Type loaded via reflection, in the presence of things like generic types e.g. MasterAssetData<int, EventItem<BuildEventItem>>. I think it would be easier if I rethought this approach and wrote a source generator so that I had

[GenerateMasterAsset<Charas, CharaData>("CharaData.json", nameof(CharaData.Id))]
public static partial class MasterAsset
{
}

[GenerateMasterAsset<Charas, CharaData>("CharaData.json", nameof(CharaData.Id))]
public static partial class MasterAsset
{
}

which could then generate a property and LoadAsync method for you, like in my second code block. This would then allow me in my console app to get the json path via reflection of the metadata of MasterAsset rather than trying to parse the syntax tree for the function call, and it would save some typing when new entries are to be added. Downside is it feels a bit overkill to write an SG for just one class. I'd appreciate any thoughts, maybe the whole approach is horrific or maybe I'm on to something 🙂

11 Replies

dreadfullydistinct•3mo ago

@viceroypenguin | 🦋🐧🌟 let me know if anything doesn't make sense

viceroypenguin•3mo ago

i've written SG's for one class before if you can save time writing 100 of those, then i say it's worth it honestly, that's a pretty simple SG to write too

dreadfullydistinct•3mo ago

It also just occured to me that if I could get symbol info (e.g. actual Types from InvocationExpressionSyntax) in my console app that would be another solution. Is that possible for a standalone console app? I'm not sure how you could hook into the compilation

viceroypenguin•3mo ago

just do a pre/post-build step. i've got one for my scaffolding that builds the orm model from the db as a build step

dreadfullydistinct•3mo ago

and that can access the compilation? how I'm just doing

SyntaxTree syntaxTree = CSharpSyntaxTree.ParseText(
    File.ReadAllText(Path.Join(sharedProjectPath, "/MasterAsset/MasterAsset.cs"))
);
CompilationUnitSyntax root = syntaxTree.GetCompilationUnitRoot();

MasterAssetWalker walker = new();
walker.Visit(root);

SyntaxTree syntaxTree = CSharpSyntaxTree.ParseText(
    File.ReadAllText(Path.Join(sharedProjectPath, "/MasterAsset/MasterAsset.cs"))
);
CompilationUnitSyntax root = syntaxTree.GetCompilationUnitRoot();

MasterAssetWalker walker = new();
walker.Visit(root);

lol Tbh though it sounds like an SG would save a lot of time here so maybe it is the best route, thanks for the input

viceroypenguin•3mo ago

just read $ig

MODiX•3mo ago

If you want to make an incremental source generator, please make sure to read through both the design document and the cookbook before starting.

viceroypenguin•3mo ago

should jumpstart your journey feel free to ask more questions here or in #roslyn when you don't understand something about SGs

dreadfullydistinct•3mo ago

Did have a quick read of those but should probably take a closer look I noticed that it says you should use a wrapper around stringbuilder to generate syntax with the proper indentation Is there somewhere I can look at an example of that It doesn’t sound like the framework offers one

viceroypenguin•3mo ago

fw doesn't offer one, correct. i just use Scriban instead https://github.com/viceroypenguin/Immediate.Handlers you can see how i put stuff together, including the templates

dreadfullydistinct•3mo ago

Will take a look thanks

Gaming

Programming

✅ Writing a source generator for this use case