Ash Framework•2mo ago

Showing paginated data inside generic actions, works in normal call but i can not show in json

Hi my friends, Since my application's structure includes two types of users — one group being masters (who don't have a site_id and have access to everything), and the other group being regular users (who do have a site_id and can only access their own data) — this setup has led to a few challenges for me. The main issue I'm facing due to this structure is that I often need to create generic actions. Unfortunately, I'm running into two major problems with this approach: 1. I'm ending up writing a lot of repetitive code and running extra queries, which I feel defeats the purpose of using Ash in the first place. 2. Another issue is that, for example, in the code below I can’t display media and its pagination properly in the JSON response when using generic actions — even though everything works fine in a normal call. This problem is recurring throughout my entire application. If you have any suggestions or guidance, I would really appreciate it. Thanks in advance. Code: https://gist.github.com/shahryarjb/0be03b650741abfbeb7cce0678a38ea5 It is related to https://discord.com/channels/711271361523351632/1019647368196534283/threads/1425967219036393582 But since it was a general issue, I separated it." By the way, every where i face this issue i am forced to create separate calculate 🥴😵‍💫🥲

Solution:

its not much money but we aren't loaded 😆

Jump to solution

16 Replies

ZachDaniel•2mo ago

I think choosing to use multitenancy when you have hierarchical setup may be your undoing here TBH. It was never designed with your use case in mind Your actions should probably just take arguments indicating what kind of data to fetch and apply filters for example If you do proceed this route, likely what you are missing is that you need to create a type that represents a page of results And then say that your action returns that type With that said you may also just need paginated relationship support in ash_json_api We support it in ash_graphql and ash core

ShahryarOP•2mo ago

Hi dear Zach, I hope you're doing well. Fortunately or unfortunately, I've written a lot of code and I think I need to keep going. I'll be opening a feature request in AshJson soon. I even tried using Related, but I wasn’t able to implement it fully, and in the end, I had to create a generic function. Because I couldn’t display the output in the JSON API, I added it to the metadata and then retrieved and displayed it using calculate, as an example. Inside the generic function itself:

Ash.Resource.put_metadata(category, :paginated_media_data, media_page_result)

Ash.Resource.put_metadata(category, :paginated_media_data, media_page_result)

And this code is also related to the calculate:

calculate :paginated_media, :map do
      public? true

      calculation fn records, _context ->
        Enum.map(records, fn record ->
          case Ash.Resource.get_metadata(record, :paginated_media_data) do
            %Ash.Page.Offset{} = page ->
              results =
                MishkaCms.Runtime.Media
                |> MishkaCms.Runtime.Helpers.take_public_attributes(page.results)

              %{results: results, limit: page.limit, offset: page.offset, more?: page.more?}

            _ ->
              %{results: [], limit: 20, offset: 0, more?: false}
          end
        end)
      end

      description "Returns complete paginated media data including results and pagination metadata"
    end

calculate :paginated_media, :map do
      public? true

      calculation fn records, _context ->
        Enum.map(records, fn record ->
          case Ash.Resource.get_metadata(record, :paginated_media_data) do
            %Ash.Page.Offset{} = page ->
              results =
                MishkaCms.Runtime.Media
                |> MishkaCms.Runtime.Helpers.take_public_attributes(page.results)

              %{results: results, limit: page.limit, offset: page.offset, more?: page.more?}

            _ ->
              %{results: [], limit: 20, offset: 0, more?: false}
          end
        end)
      end

      description "Returns complete paginated media data including results and pagination metadata"
    end

This is the easiest way i could find and can be re-usable one (as a separate module for every where i have it) But as you saw in the Gist, the code is quite a lot. I'm not sure if Ash has better ways to simplify things, or if this is the best approach for now!?

ZachDaniel•2mo ago

You'll honestly probably have an easier time if you figure out how to support paginated relationships in AshJsonApi 😄 You can potentially simplify things via a generic calculation, i.e

calculate :paginated_meta, :map, {PaginatedRelationship, relationship: :media}

calculate :paginated_meta, :map, {PaginatedRelationship, relationship: :media}

There are many cases where people hitting certain road blocks have found it easier to extend the core tooling. Like supporting this in AshJsonApi would probably look something like: ?included_page[foo.bar][limit]=10 Then, you'd adjust the serialize to support receiving a page of data I don't really think it would be that hard? You may need to add to the DSL to say which includes can be paginated as well i.e paginated_includes [[:foo], [:foo, :bar]] i.e "these paths of includes can be paginated" You can do relationship pagination in Ash core with this kind of pattern:

load(..., relationship: Ash.Query.page(Related, limit: 10))

load(..., relationship: Ash.Query.page(Related, limit: 10))

ShahryarOP•2mo ago

Thank you for the explanation. I'm honestly a bit lost in the sea of information right now, and I still haven’t been able to fully control the number of queries being made to the database in Ash. Your explanation really helps improve my understanding.

ZachDaniel•2mo ago

Ash typically makes as many queries as relationships have been loaded are you seeing otherwise? i.e

User
|> Ash.Query.load([:friends, :enemies])
|> Ash.read!()

User
|> Ash.Query.load([:friends, :enemies])
|> Ash.read!()

would make 3 queries, one for the users, one for the friends, one for the enemies.

ShahryarOP•2mo ago

I think it's mostly due to my own skill issues, both database and in Ash. For example, in many cases I eventually gave up and used raw queries — like in the example below.

action :load_page_builder_resources_raw, :struct do
      argument :site_id, :uuid do
        allow_nil? true

        description "The ID of the site to load with its resources. If nil, returns only global resources"
      end

      run fn input, _ ->
        site_id = input.arguments.site_id

        # Convert UUID only if site_id is not nil
        uuid_binary =
          if is_nil(site_id) do
            nil
          else
            {:ok, binary} = Ecto.UUID.dump(site_id)
            binary
          end

        sql = """
        SELECT
          (SELECT row_to_json(s.*) FROM sites s WHERE s.id = $1 AND s.archived_at IS NULL) as site,
          (
            SELECT COALESCE(json_agg(c.*), '[]'::json)
            FROM components c
            WHERE ($1 IS NULL AND c.site_id IS NULL) OR (c.site_id = $1 OR c.site_id IS NULL)
              AND c.archived_at IS NULL
              AND c.active = true
          ) as components,
          (
            SELECT COALESCE(json_agg(l.*), '[]'::json)
            FROM layouts l
            WHERE ($1 IS NULL AND l.site_id IS NULL) OR (l.site_id = $1 OR l.site_id IS NULL)
              AND l.archived_at IS NULL
              AND l.active = true
          ) as layouts
        """

        case Ecto.Adapters.SQL.query(MishkaCms.Repo, sql, [uuid_binary]) do
          {:ok, %{rows: [[site, components, layouts]]}} ->
            {:ok,
             %{
               "site" => site,
               "components" => Helpers.builder_encoded_ast(components, "component"),
               "layouts" => Helpers.builder_encoded_ast(layouts, "layout")
             }}

          _ ->
            {:ok, %{"site" => nil, "components" => [], "layouts" => []}}
        end
      end

      description "Loads a site with all its resources in a single query"
    end

action :load_page_builder_resources_raw, :struct do
      argument :site_id, :uuid do
        allow_nil? true

        description "The ID of the site to load with its resources. If nil, returns only global resources"
      end

      run fn input, _ ->
        site_id = input.arguments.site_id

        # Convert UUID only if site_id is not nil
        uuid_binary =
          if is_nil(site_id) do
            nil
          else
            {:ok, binary} = Ecto.UUID.dump(site_id)
            binary
          end

        sql = """
        SELECT
          (SELECT row_to_json(s.*) FROM sites s WHERE s.id = $1 AND s.archived_at IS NULL) as site,
          (
            SELECT COALESCE(json_agg(c.*), '[]'::json)
            FROM components c
            WHERE ($1 IS NULL AND c.site_id IS NULL) OR (c.site_id = $1 OR c.site_id IS NULL)
              AND c.archived_at IS NULL
              AND c.active = true
          ) as components,
          (
            SELECT COALESCE(json_agg(l.*), '[]'::json)
            FROM layouts l
            WHERE ($1 IS NULL AND l.site_id IS NULL) OR (l.site_id = $1 OR l.site_id IS NULL)
              AND l.archived_at IS NULL
              AND l.active = true
          ) as layouts
        """

        case Ecto.Adapters.SQL.query(MishkaCms.Repo, sql, [uuid_binary]) do
          {:ok, %{rows: [[site, components, layouts]]}} ->
            {:ok,
             %{
               "site" => site,
               "components" => Helpers.builder_encoded_ast(components, "component"),
               "layouts" => Helpers.builder_encoded_ast(layouts, "layout")
             }}

          _ ->
            {:ok, %{"site" => nil, "components" => [], "layouts" => []}}
        end
      end

      description "Loads a site with all its resources in a single query"
    end

ZachDaniel•2mo ago

It might help to know that you can just use Ash resources as ecto schemas if you need to write ecto queries

ShahryarOP•2mo ago

it is not just relations , sorry it is my skill issues :))

ZachDaniel•2mo ago

YourResource |> Ecto.Repo.all() from row in YourResource, where: .... etc. Its not necessarily a problem to write code in Ash actions like that TBH. THe most important thing in Ash is just typed actions living in resources. The rest is convenience. With all that said: I'm willing to bet that if you pushed through, learned how AshJsonApi works under the hood (its not very complicated TBH), and made a PR for paginated relationships support, that you would learn enough about Ash and the internals to be more successful building your own application. In fact: I want this feature too 😆 Maybe I'll put a bounty on it. We don't have much in the Ash open collective, but I could do $300 for a PR

ShahryarOP•2mo ago

I think the CMS will be finished soon — and it could probably be presented as the worst code ever written with Ash! 😄 Thanks again, your explanation was very thorough and helpful. I still don’t fully understand everything yet, and I’m moving forward with my code a bit using AI. But I’ll go ahead and open the issue. If it’s within my ability, I’ll definitely try to submit a PR as well.

ZachDaniel•2mo ago

👍 you should claim the bounty if you want to give it a shot otherwise I'll see if others feel like doing it

Solution

ZachDaniel•2mo ago

its not much money but we aren't loaded 😆

ShahryarOP•2mo ago

HHAHAHA, open-source and the low-power donors — hahahaha! Thank you, first i try to learn how it works, if i will be okey , sure i send pr ,

ZachDaniel•2mo ago

@Shahryar looks like someone beat you to it: https://github.com/ash-project/ash_json_api/pull/392

GitHub

feat: Add pagination support for included relationships by srikanth...

This commit implements the first two steps of paginated relationships: DSL Configuration Add paginated_includes option to json_api resource schema Users can specify which relationship paths allo...

ZachDaniel•2mo ago

I will review that soon, but perhaps until then, you can try that branch?

ShahryarOP•2mo ago

Hi dear Zach, Ohh man you are so kind ♥️♥️, Even if I could fix it, I would never take any bounty or money for it. I owe everything I do to your work — it’s all come to us through you. I think it will take me a long time to even try, I’m very tied up with CMS work right now. By the way, I posted something about it on Bluesky. https://bsky.app/profile/shahryar-tbiz.bsky.social/post/3m2vsg5s5ck26 I hope he done it very well specially with supporting multi tendency (active and inactive)

Shahryar Tavakkoli (@shahryar-tbiz.bsky.social)

Ash lover, if you have time and you are a user of AshJson, there is a Feature-requested issue, which needs to be implemented 🤭 Issue link: github.com/ash-project/... #ElixirLang

Bluesky

Gaming

Programming

Showing paginated data inside generic actions, works in normal call but i can not show in json

Did you find this page helpful?