Ash ai prompt wrapper
Moving the question here from the ash development channel:
Boris - BackedBy — 08:13
Every time I submit an LLM request, I also submit evaluation criteria. After receiving the main response, I have a validation model respond with 'true' or 'false' re whether the response fits the evaluation criteria. The logprob of the 'true' token is then used as a numerical score for how well the LLM responded. What would be the best approach for extending the ash ai prompt-backed action to have optional criteria: [list of strings] and validation_model: string definitions to run the validation step?
Zach Daniel — 08:15
As it stands right now, your best bet would likely be to create a "wrapper" around the prompt action implementation
Or even just copy it and use its implementation, as its not a huge thing really
If there is an example of a wrapper around a the prompt action available for me to take a look at that would be very helpful. Thank you!
Boris - BackedBy — 08:13
Every time I submit an LLM request, I also submit evaluation criteria. After receiving the main response, I have a validation model respond with 'true' or 'false' re whether the response fits the evaluation criteria. The logprob of the 'true' token is then used as a numerical score for how well the LLM responded. What would be the best approach for extending the ash ai prompt-backed action to have optional criteria: [list of strings] and validation_model: string definitions to run the validation step?
Zach Daniel — 08:15
As it stands right now, your best bet would likely be to create a "wrapper" around the prompt action implementation
Or even just copy it and use its implementation, as its not a huge thing really
If there is an example of a wrapper around a the prompt action available for me to take a look at that would be very helpful. Thank you!
The Elixir backend framework for unparalleled productivity. Declarative tools that let you stop wasting time. Use with Phoenix LiveView or build APIs in minutes for your front-end of choice.
2,193Members
Resources
Similar Threads
Was this page helpful?
