mix ash_postgres.gen.resources is slow for large codebases
We're investigating using Ash with our existing codebase, which is quite large. We're looking to generate resources for existing Ecto schemas using mix ash_postgres.gen.resources, but we're experiencing slow performance with that command. It seems others have experienced this also (see https://elixirforum.com/t/integrating-ash-into-an-existing-project/66927/6).
I've added some debug logging in ash_postgres, and it seems that the majority of the time is being spent in Ash.Resource.Igniter.list_resources(igniter):
Generating resources from [OgatApp.Repo]
[PERF] Migrations: false
[PERF] Igniter.compose_task("ash.gen.domain", ...): 14947ms
[PERF] list_resources: 607837ms
[PERF] filter resources with Code.ensure_loaded?: 16ms
[PERF] include_all_elixir_files: 0ms
[PERF] Generating for 1 repo(s)
[PERF] table_specs query: 236ms
[PERF] build_attributes(pupils): 101ms
[PERF] Processing 1 tables
[PERF] add_foreign_keys(pupils): 18ms
[PERF] add_indexes(pupils): 13ms
[PERF] add_check_constraints(pupils): 3ms
[PERF] tables(OgatApp.Repo): 1645ms
[PERF] Spec.tables(OgatApp.Repo): 1657ms
[PERF] Spec.tables for all repos: 1667ms
[PERF] Adding relationships for 1 specs
[PERF] do_add_relationships(OgatApp.Repo, 1 specs): 2ms
[PERF] add_relationships (total): 2ms
[PERF] Spec.add_relationships: 2ms
[PERF] Generating 1 resources
[PERF] build resource template(pupils): 0ms
[PERF] add_resource_reference(pupils): 17100ms
[PERF] create_module(pupils): 173ms
[PERF] table_to_resource(pupils): 17273ms
[PERF] table_to_resource for all specs: 17273ms
[PERF] generate (total): 626795ms
[PERF] AshPostgres.ResourceGenerator.generate(...): 626795ms
[PERF] mix task igniter/1 (total): 641748ms
It looks like this might be scanning all existing files to determine which are resources. Is there anything that can be done to speed this up? Happy to help with a PR.
12 Replies
🤔 possibly. Its hard to say. We can maybe parallelize it in some way but it is relatively expensive. We've talked about also adding caching to that but I'm not sure it would really help.
Could also revisit that task to see if its actually necessary to list resources.
Is there a need for it to scan the uncompiled files to find resources? Could it instead list the current domains based on the app’s config, and retrieve the resources from those?
It depends. It could check if the igniter has any changed files, and if not it could list resources the way you described
I think I can see how to get the resources via the Ash Info modules. Is there an existing function to check if Igniter has any changed files?
It's extremely slow for me too. I used to favor generators until it became unwieldy. Now I just try to write the resources myself and compile everything together
The main performance issues seem to be Ash.Resource.Igniter.list_resources (which takes around 600 seconds on average on our codebase), and Ash.Domain.Igniter.list_domains (which takes around 15 seconds, but is called twice during the ash_postgres.gen.resources task, for a total of 30 seconds).
If I swap out the call to list_resources to instead do:
resources =
Ash.Info.domains_and_resources(List.first(repos).config()[:otp_app])
|> Map.values()
|> List.flatten()
{igniter, resources}
then this takes the runtime down to around 30 seconds for ash_postgres.gen.resources (which is essentially then the 2x ~15 second calls to list_domains).
Whilst we're in the early stages of trying out Ash, the ash_postgres.gen.resources is our main bottleneck as we bring in resources for existing tables, but the slowness also seems be an issue in other generators (which at a glance also rely on these calls).
It'd be great if we could find a way to speed this up and improve the experience of using Ash on large codebases. It's particularly painful to try and adopt into existing projects due to this at the moment.
How many resources do you have?
Roughly, would be interesting to have a point of comparison
We only have 4 Ash resources at the moment, but a large number of Elixir modules in general, which is what the list_resources/list_domain functions iterate over at the moment.
i see
@elliotb you can look at the size of
igniter.rewrite.sources@Zach Thanks, I’ll take a look later this week.
@Zach I've submitted a proof of concept PR for this at https://github.com/ash-project/ash/pull/2371
GitHub
improvement: optimize Ash.Resource.Igniter.list_resources and Ash.D...
This follows a discussion in Discord about the performance of mix ash_postgres.gen.resources with codebases with large numbers of existing Elixir modules.
Having added some debug logging to the tas...