How to Ash-idiomatically get all unique values of an attribute across all records?
App.Api.Item is a resource with an attribute :name.
How can I go from the set of all records of Item (App.Api.Item |> App.Api.read!()) to a list of all unique values of the :name attribute in the most idiomatic way?50 Replies
Hello! Ash does actually have a way to do that as of recently 🙂
App.Api.Item |> App.Api.list(:name) is how you'd get the list of names
however, you should be able to pass uniq?: true as an option to that to get unique values, but due to a bug in the way we handle those options you can't
so I'll push up a fix
but this was added like last week
FWIW there are still some optimizations to be made to the generated SQL there, but it won't change how they are used so you should be goodI'm getting the following (
P is my aliased App.Api which is actually named Bouketo.Portfolio):
The list in here is actually the contents I want to get:
It also fails on aggregates:
Can you update to the latest ash and ash postgres? If it’s still broken try ash postgres main
Updated to latest ash and ash_postgres; same results.
Updated ash_postgres to main; same results.
🤔 okay I'll take a look now
thanks
Let me check is something is weird with my data (I'm importing a CSV file into tables)
I can get all records of a resource and pick one of them.
...works fine.
Once I try to
load!(), I get an unknown error:
@Zach Daniel I don't want to send you on a wild goose chase within ash_postgres in case I have defined something the wrong way... what could be a cause of the unknown error if not ash_postgres?
Notably, this doesn't happen in all resources.What are you loading?
That error is pretty garbage 😢
the CSV file representing a flat database of sales items, each with its item code, manufacturer part number, product category, product family, list price, etc., and splitting it into resources with relationships.
I've been writing the logic in the
seeds.exs file, and allegedly all records are generated OK.
I wonder whether I might have some circularity?
What about this error?So the original error you showed me should definitely go away on the latest ash_postgres main
are you still getting that error?
Well, sorry, it won't necessarily go away
but the SQL it generates would be different for sure
i.e from doing this:
P.Brand |> P.list(:name)
Yes, the SQL is different
Interestingly, ash_admin (branch phoenix-1.7) works well, I can see all records listed, relationships load fine on a record.
🤔 what version of postgres are you on?
okay, I just pushed a fix up to ash_postgres main for that error
I think the version of postgres I'm testing with for whatever reason doesn't surface that error
or some other dependency version, whatever it is
but the fix should be there
So once that is fixed, how did you get this error?: https://discord.com/channels/711271361523351632/1074452397176393808/1074754246169350174
postgres v13 from debian/bullseye
which version are you on?
14.6
either way, it was "broken" in both places
just for some reason I wasn't getting the error
ash_postgres main should do the trick now
I'm trying
P.Whatever |> P.read!() |> Enum.at(0) |> P.load!(:somerelationship) on all Whatever resources I have defined. It works for some, but not for all, so maybe PEBKAC...If you can show me a specific one maybe we can hunt down the issue and find out.
is invalid is unideal but might just be hinting at the thing you tried to load not actually existingP.Brand has_many P.Family
works.
P.Category has_many P.Family
works.
P.Family belongs_to P.Category and P.Brand, and has_many P.Variant
works.
P.Variant belongs_to P.Family and has_one P.Item
works.
P.Sequence belongs_to P.Item
works.
I think I found it
P.Item belongs_to one P.Variant
fails with the Unknown Error.
The [debug] output shows a SELECT query like so, and it's truncated.
can I see the relationship definition? And the full error?
https://pastebin.com/3zd85kv7
This seems fishy to me
The way that we do loads at the moment (up for change/optimization for cases like this later, i.e loading a
belongs_to/has_one) is not via joins
We do it in separate queries, passing down all the idsGot it, hence the list of uuids
that pastebin is gone
ah dang it
lemme redo
https://pastebin.com/JhE4Usjk expires in a week
I'll try postgres v15 to see if it makes a difference
Still says not found 😆
same here, pastebin.com is broken...???
https://0bin.net/paste/Q1A56FyN#g9SHFrcNjXZt+9+NO98cvGXXF0PNARJRHPzD5JEDOiH
0bin - encrypted pastebin
0bin is a client-side-encrypted alternative pastebin. You can store code/text/images online for a set period of time and share with the world. Featuring burn after reading, history, clipboard.
okay that works 😆
Do you have any custom actions on the variant resource?
no
Can you read the variants directly?
Yeah, I see that you can
something strange is going on there for sure
Would
P.Item |> P.read!() |> P.load!(:variant) run OK even if one of the records is somehow broken?
Maybe the issue is with one or more records not properly associated, e.g. due to dirty data in the CSVOh, yknow
So it works on
Enum.at(0) that I use as a quick test, but not across all records!
Hmmmmmm :thinkies: Let me check...Something weird is going on there for sure
Gimme a few minutes... mix deps.compile takes a while on my i5-2520M ...
Interesting. The first call leads to a different result to the second one.
Now on Postgres 15. Reimported all the data.
🤔
there is definitely a dependency version issue there
well...maybe not I guess? Something weird is happening though that much is sure
mix hex.info ash and mix hex.info ash_postgres what do those show?
And in mix.exs:
oh hey, it works now! wth?
wut 🥴
😢
this works
okay, dunno whats up with that
but this is still broken?
P.Item |> P.read!() |> P.load!(:variantyep, just checked, this one is still broken
oh man, thank you so much for that
Ash.Api.list() !okay, can you do something like this:
that will print out the offending record
Indeed, it choked after a few records. Let me check.
The offending record can be read, but loading
:variant leads to the unknown error. I'll check my data.
Actually it chokes at the very first record.can I see it?
0bin - encrypted pastebin
0bin is a client-side-encrypted alternative pastebin. You can store code/text/images online for a set period of time and share with the world. Featuring burn after reading, history, clipboard.
I don't see anything wrong with that...
something really strange is happening here
This is super weird.
This works:
This also works:
This fails with unknown error:
Can you do this:
will print out lots of output, but we can see what the last things it prints before it dies is
Interestingly,
P.Item has an aggregate first :variant_code, :variant, :code, and
So, :variant_code based on the related :variant is successfully prepared.0bin - encrypted pastebin
0bin is a client-side-encrypted alternative pastebin. You can store code/text/images online for a set period of time and share with the world. Featuring burn after reading, history, clipboard.
So, the
:code calculation fails
I'll check all the dependent values of this
All fail with:
Weird, because the :code preparation works as intended, and P.Item |> P.read!() delivers records with code: "9381310" etc., as intended.
@Zach Daniel verified.
I remove :code from the preparations:
P.Item |> P.read!() |> P.load!(:variant) -> works.
P.Item |> P.read!() |> P.load!(:code) -> unknown error.
The other calculation, calculate :tagline, :string, expr(variant_name) , works with P.Item |> P.read!() |> P.load!(:tagline).is
code on variant an aggregate?
You can't aggregate over aggregates currently, which is why P.Item |> P.list!(:sequence_prefix) doesn't work
I've lost track a bit of the specific issues here
Could you make a separate thread for each one please?
It seems like there is an issue w/ loading certain aggregates, and an issue loading a specific relationship?code is an attribute of P.Variant, accessible within P.Item as aggregate variant_code.
code is an attribute of P.Category, accessible within P.Item as aggregate category_code.
P_item contains a calculation named code that concatenates aggregates variant_code and category_code.
Apologies for the confusion and thank you for taking the time! I will split the issues, as it's not anymore about P.list().