AE
Ash Elixir•2mo ago
Shaba

AshAI json schema usage

Any examples passing in json_schema to: 
    action :parse_raw, :map do
      argument :raw_content, :string, allow_nil?: false

      run prompt(
        fn _input, _context ->
          LangChain.ChatModels.ChatOpenAI.new!(%{
            model: "gpt-4o-mini",
            api_key: System.get_env("OPENAI_API_KEY"),
            temperature: 0.1,
            json_schema: JobListingSchema.schema()
          })
        end,
        prompt: """
        Parse this job listing into structured data following the exact schema.
        Extract all available information and return as JSON:

        <%= @input.arguments.raw_content %>
        """,
        tools: false
      )
    end
    action :parse_raw, :map do
      argument :raw_content, :string, allow_nil?: false

      run prompt(
        fn _input, _context ->
          LangChain.ChatModels.ChatOpenAI.new!(%{
            model: "gpt-4o-mini",
            api_key: System.get_env("OPENAI_API_KEY"),
            temperature: 0.1,
            json_schema: JobListingSchema.schema()
          })
        end,
        prompt: """
        Parse this job listing into structured data following the exact schema.
        Extract all available information and return as JSON:

        <%= @input.arguments.raw_content %>
        """,
        tools: false
      )
    end
I keep getting 
[error] Received error from API: "Invalid schema for response_format 'result': In context=('properties', 'result'), 'additionalProperties' is required to be supplied and to be false."
[error] Error during chat call. Reason: %LangChain.LangChainError{type: nil, message: "Invalid schema for response_format 'result': In context=('properties', 'result'), 'additionalProperties' is required to be supplied and to be false.", original: nil}
[error] Received error from API: "Invalid schema for response_format 'result': In context=('properties', 'result'), 'additionalProperties' is required to be supplied and to be false."
[error] Error during chat call. Reason: %LangChain.LangChainError{type: nil, message: "Invalid schema for response_format 'result': In context=('properties', 'result'), 'additionalProperties' is required to be supplied and to be false.", original: nil}
I tested the same schema in openai playground and it has worked fine
83 Replies
ZachDaniel
ZachDaniel•2mo ago
We are almost certiainly overriding this: json_schema: JobListingSchema.schema() with the empty 'map' schema To use structured outputs, you have to tell it exactly what the fields are, so you'd need to put in like a custom map type w/ use Ash.Type.NewType, subtype_of: :map (or use the new TypedStruct thing that I released like 10 minutes ago) Alternatively you could use a different adapter, like RequestJson
Shaba
ShabaOP•2mo ago
how would StructuredOutput look like in this example?
ZachDaniel
ZachDaniel•2mo ago
What is this: JobListingSchema.schema()? I think maybe you're missing a piece here which is how much Ash will do for you automatically 
defmodule JobListing do
  use Ash.TypedStruct

  typed_struct do
    field :name, :string, allow_nil?: false
  end
end
defmodule JobListing do
  use Ash.TypedStruct

  typed_struct do
    field :name, :string, allow_nil?: false
  end
end
Shaba
ShabaOP•2mo ago
Just a module to read in json schema file and convert it to a map: 
defmodule Snowmass.JobMarket.Schemas.JobListingSchema do
  @moduledoc """
  JSON schema for job listing parsing with LLM.
  """

  @json_schema_file Path.join([__DIR__, "job_listing.json"])
  @external_resource @json_schema_file

  @schema @json_schema_file
               |> File.read!()
               |> Jason.decode!()

  def schema do
    @schema
  end
end
defmodule Snowmass.JobMarket.Schemas.JobListingSchema do
  @moduledoc """
  JSON schema for job listing parsing with LLM.
  """

  @json_schema_file Path.join([__DIR__, "job_listing.json"])
  @external_resource @json_schema_file

  @schema @json_schema_file
               |> File.read!()
               |> Jason.decode!()

  def schema do
    @schema
  end
end
Probably unnecessary
ZachDaniel
ZachDaniel•2mo ago
    action :parse_raw, JobListing do
      argument :raw_content, :string, allow_nil?: false

      run prompt(
        fn _input, _context ->
          LangChain.ChatModels.ChatOpenAI.new!(%{
            model: "gpt-4o-mini",
            api_key: System.get_env("OPENAI_API_KEY"),
            temperature: 0.1
          })
        end,
        prompt: """
        Parse this job listing into structured data following the exact schema.
        Extract all available information and return as JSON:

        <%= @input.arguments.raw_content %>
        """,
        tools: false
      )
    end
    action :parse_raw, JobListing do
      argument :raw_content, :string, allow_nil?: false

      run prompt(
        fn _input, _context ->
          LangChain.ChatModels.ChatOpenAI.new!(%{
            model: "gpt-4o-mini",
            api_key: System.get_env("OPENAI_API_KEY"),
            temperature: 0.1
          })
        end,
        prompt: """
        Parse this job listing into structured data following the exact schema.
        Extract all available information and return as JSON:

        <%= @input.arguments.raw_content %>
        """,
        tools: false
      )
    end
Shaba
ShabaOP•2mo ago
Ah oh so in infers the schema from the return type which you updated from :map to the new Ash.TypedStruct
ZachDaniel
ZachDaniel•2mo ago
Yep!
Shaba
ShabaOP•2mo ago
Ok so im thinking of adding this to a types directory in the related domain
ZachDaniel
ZachDaniel•2mo ago
Yeah /domain/types/foo.ex is a normal pattern
Shaba
ShabaOP•2mo ago
👍 awesome, yeah not many examples for LLMs to explain this lol
ZachDaniel
ZachDaniel•2mo ago
updating docs & usage-rules.md
Shaba
ShabaOP•2mo ago
iex(2)> test = Ash.run_action(Ash.ActionInput.for_action(Snowmass.JobMarket.JobListing, :parse_raw, %{raw_content: raw}))
{:ok,
 %Snowmass.JobMarket.Types.JobListingParseResult{name: "Solutions Engineer"}}
iex(3)> 
iex(2)> test = Ash.run_action(Ash.ActionInput.for_action(Snowmass.JobMarket.JobListing, :parse_raw, %{raw_content: raw}))
{:ok,
 %Snowmass.JobMarket.Types.JobListingParseResult{name: "Solutions Engineer"}}
iex(3)>
awesome time to write out that TypedStruct One thing I noticed is I cant set allow_nil? on a field: 
[error] Received error from API: "Invalid schema for response_format 'result': In context=('properties', 'result'), 'required' is required to be supplied and to be an array including every key in properties. Missing 'title'."
[error] Error during chat call. Reason: %LangChain.LangChainError{type: nil, message: "Invalid schema for response_format 'result': In context=('properties', 'result'), 'required' is required to be supplied and to be an array including every key in properties. Missing 'title'.", original: nil}
[error] Received error from API: "Invalid schema for response_format 'result': In context=('properties', 'result'), 'required' is required to be supplied and to be an array including every key in properties. Missing 'title'."
[error] Error during chat call. Reason: %LangChain.LangChainError{type: nil, message: "Invalid schema for response_format 'result': In context=('properties', 'result'), 'required' is required to be supplied and to be an array including every key in properties. Missing 'title'.", original: nil}
In my original json schema I normall set it to two possible types: 
        "max_annual_salary": {
          "type": ["number", "null"],
          "description": "The maximum annual salary for the job. Leave null if job is paid hourly"
        }
        "max_annual_salary": {
          "type": ["number", "null"],
          "description": "The maximum annual salary for the job. Leave null if job is paid hourly"
        }
But I can still list it under required: 
      "required": [
...
        "max_annual_salary",
      "required": [
...
        "max_annual_salary",
ZachDaniel
ZachDaniel•2mo ago
That is a limitation of structured outputs in open ai AFAIK
Shaba
ShabaOP•2mo ago
on the Ash/langchain side of things? Because from my understanding, required only defines what fields have to be present in the response, regardless of whether it is null or not Or at least thats how it was working in practice for me
Shaba
ShabaOP•2mo ago
Shaba
ShabaOP•2mo ago
Maybe it is related to defp add_null_for_non_required(%{required: required} = schema): https://github.com/ash-project/ash_ai/blob/4678d6868f828c857b4e34c76ad3a71c571d27f3/lib/ash_ai/open_api.ex#L148
GitHub
ash_ai/lib/ash_ai/open_api.ex at 4678d6868f828c857b4e34c76ad3a71c57...
Structured outputs, vectorization and tool calling for your Ash application - ash-project/ash_ai
Shaba
ShabaOP•2mo ago
Would there be a way to see the schema that gets passed to the LLM?
ZachDaniel
ZachDaniel•2mo ago
There isn't currently but we could make one Could be as simple as not overwriting a non-nil schema Right, we should change our json schema implementation to show all fields as required always but to allow them to have null values
Shaba
ShabaOP•2mo ago
I can try to take a look, I am having trouble understanding where the TypedStruct gets referenced and then converted to a json schema AI: 
The key flow is: prompt action → get_json_schema/1 → AshAi.OpenApi.resource_write_attribute_type/3 → handles embedded resources by extracting their attributes into JSON schema format.
The key flow is: prompt action → get_json_schema/1 → AshAi.OpenApi.resource_write_attribute_type/3 → handles embedded resources by extracting their attributes into JSON schema format.
ZachDaniel
ZachDaniel•2mo ago
yeah so it would be in AshAi.OpenApi
Shaba
ShabaOP•2mo ago
hmmm 
iex(16)> AshAi.OpenApi.resource_write_attribute_type(  %{name: :result, type: Snowmass.JobMarket.Types.JobListingParseResult, constraints: []},  nil,  :create)
%{}
iex(16)> AshAi.OpenApi.resource_write_attribute_type(  %{name: :result, type: Snowmass.JobMarket.Types.JobListingParseResult, constraints: []},  nil,  :create)
%{}
ZachDaniel
ZachDaniel•2mo ago
🤔 that doesn't seem right to me
Shaba
ShabaOP•2mo ago
This is giving me a bit more info: 
:dbg.tracer()
:dbg.p(:all, :c)
:dbg.tpl(AshAi.OpenApi, :resource_write_attribute_type, [])
# Then run your action
:dbg.tracer()
:dbg.p(:all, :c)
:dbg.tpl(AshAi.OpenApi, :resource_write_attribute_type, [])
# Then run your action
iex(35)> test = Ash.run_action(Ash.ActionInput.for_action(Snowmass.JobMarket.JobListing, :parse_raw, %{raw_content: raw}))
(<0.442.0>) call 'Elixir.AshAi.OpenApi':resource_write_attribute_type(#{name => result,
  type => 'Elixir.Snowmass.JobMarket.Types.JobListingParseResult',
  constraints =>
      [{fields,
           [{min_years_of_experience,
                [{'allow_nil?',false},
                 {constraints,[]},
                 {type,'Elixir.Ash.Type.Integer'}]}]},
       {instance_of,'Elixir.Snowmass.JobMarket.Types.JobListingParseResult'},
       {'preserve_nil_values?',false}]},nil,create)
(<0.442.0>) call 'Elixir.AshAi.OpenApi':resource_write_attribute_type(#{name => result,type => 'Elixir.Ash.Type.Struct',
  constraints =>
      [{fields,
           [{min_years_of_experience,
                [{'allow_nil?',false},
                 {constraints,[]},
                 {type,'Elixir.Ash.Type.Integer'}]}]},
       {instance_of,'Elixir.Snowmass.JobMarket.Types.JobListingParseResult'},
       {'preserve_nil_values?',false}]},nil,create)
(<0.442.0>) call 'Elixir.AshAi.OpenApi':resource_write_attribute_type(#{name => result,type => 'Elixir.Ash.Type.Map',
  constraints =>
      [{fields,
           [{min_years_of_experience,
                [{'allow_nil?',false},
                 {constraints,[]},
                 {type,'Elixir.Ash.Type.Integer'}]}]},
       {instance_of,'Elixir.Snowmass.JobMarket.Types.JobListingParseResult'},
       {'preserve_nil_values?',false}]},nil,create)
(<0.442.0>) call 'Elixir.AshAi.OpenApi':resource_write_attribute_type(#{name => result,type => 'Elixir.Ash.Type.Integer',description => nil,
  constraints => []},nil,create)
{:ok,
 %Snowmass.JobMarket.Types.JobListingParseResult{min_years_of_experience: 1}}
iex(35)> test = Ash.run_action(Ash.ActionInput.for_action(Snowmass.JobMarket.JobListing, :parse_raw, %{raw_content: raw}))
(<0.442.0>) call 'Elixir.AshAi.OpenApi':resource_write_attribute_type(#{name => result,
  type => 'Elixir.Snowmass.JobMarket.Types.JobListingParseResult',
  constraints =>
      [{fields,
           [{min_years_of_experience,
                [{'allow_nil?',false},
                 {constraints,[]},
                 {type,'Elixir.Ash.Type.Integer'}]}]},
       {instance_of,'Elixir.Snowmass.JobMarket.Types.JobListingParseResult'},
       {'preserve_nil_values?',false}]},nil,create)
(<0.442.0>) call 'Elixir.AshAi.OpenApi':resource_write_attribute_type(#{name => result,type => 'Elixir.Ash.Type.Struct',
  constraints =>
      [{fields,
           [{min_years_of_experience,
                [{'allow_nil?',false},
                 {constraints,[]},
                 {type,'Elixir.Ash.Type.Integer'}]}]},
       {instance_of,'Elixir.Snowmass.JobMarket.Types.JobListingParseResult'},
       {'preserve_nil_values?',false}]},nil,create)
(<0.442.0>) call 'Elixir.AshAi.OpenApi':resource_write_attribute_type(#{name => result,type => 'Elixir.Ash.Type.Map',
  constraints =>
      [{fields,
           [{min_years_of_experience,
                [{'allow_nil?',false},
                 {constraints,[]},
                 {type,'Elixir.Ash.Type.Integer'}]}]},
       {instance_of,'Elixir.Snowmass.JobMarket.Types.JobListingParseResult'},
       {'preserve_nil_values?',false}]},nil,create)
(<0.442.0>) call 'Elixir.AshAi.OpenApi':resource_write_attribute_type(#{name => result,type => 'Elixir.Ash.Type.Integer',description => nil,
  constraints => []},nil,create)
{:ok,
 %Snowmass.JobMarket.Types.JobListingParseResult{min_years_of_experience: 1}}
Its probably: 
      required:
          constraints[:fields]
          |> Enum.filter(fn {_, config} -> !config[:allow_nil?] end)
          |> Enum.map(&elem(&1, 0))
      required:
          constraints[:fields]
          |> Enum.filter(fn {_, config} -> !config[:allow_nil?] end)
          |> Enum.map(&elem(&1, 0))
ZachDaniel
ZachDaniel•2mo ago
Yep. That can just but all fields
Shaba
ShabaOP•2mo ago
      %{
        type: :object,
        properties: Map.new(fields),
        additionalProperties: false
        # required: required Missing?
      }
      %{
        type: :object,
        properties: Map.new(fields),
        additionalProperties: false
        # required: required Missing?
      }
ZachDaniel
ZachDaniel•2mo ago
required: Keyword.keys(constraints[:fields])
Shaba
ShabaOP•2mo ago
Its working now: 
iex(11)> test = Ash.run_action(Ash.ActionInput.for_action(Snowmass.JobMarket.JobListing, :parse_raw, %{raw_content: raw}))
{:ok,
 %Snowmass.JobMarket.Types.JobListingParseResult{
   is_hybrid: false,
   is_oncall_required: false,
   is_management_role: false,
   is_remote: true,
   is_onsite: false,
   is_travel_required: false,
   is_security_clearance_required: false,
   max_annual_salary: 0,
   min_annual_salary: 0,
   min_hourly: 0,
   max_hourly: 0,
   is_hourly: false,
   raw_salary: nil,
iex(11)> test = Ash.run_action(Ash.ActionInput.for_action(Snowmass.JobMarket.JobListing, :parse_raw, %{raw_content: raw}))
{:ok,
 %Snowmass.JobMarket.Types.JobListingParseResult{
   is_hybrid: false,
   is_oncall_required: false,
   is_management_role: false,
   is_remote: true,
   is_onsite: false,
   is_travel_required: false,
   is_security_clearance_required: false,
   max_annual_salary: 0,
   min_annual_salary: 0,
   min_hourly: 0,
   max_hourly: 0,
   is_hourly: false,
   raw_salary: nil,
I think its also the LLM ignoring setting field to 0 rather than null in the response
ZachDaniel
ZachDaniel•2mo ago
🤷‍♂️ nothing we can do about that part 😆 you can prompt it not to w/ field descriptions
Shaba
ShabaOP•2mo ago
I do 😢 I even tested it: 
    field :max_annual_salary, :integer, description: "make sure this is null", allow_nil?: true
    field :max_annual_salary, :integer, description: "make sure this is null", allow_nil?: true
haha
ZachDaniel
ZachDaniel•2mo ago
Try thsi then
Shaba
ShabaOP•2mo ago
Just to recap, I changed the following in openapi: ``` required: constraints[:fields] - |> Enum.filter(fn {, config} -> !config[:allownil?] end) + # |> Enum.filter(fn {, config} -> !config[:allow_nil?] end) |> Enum.map(&elem(&1, 0)) and if fields == [] do nil else %{ type: :object, properties: Map.new(fields), additionalProperties: false, - # required: required Missing? + required: Keyword.keys(constraints[:fields]) } |> with_attribute_description(attribute_or_aggregate) ```
ZachDaniel
ZachDaniel•2mo ago
    field :max_annual_salary, :integer do
      description "make sure this is null if it is unknown"
      allow_nil? true
      constraints min: 1
    end
    field :max_annual_salary, :integer do
      description "make sure this is null if it is unknown"
      allow_nil? true
      constraints min: 1
    end
Shaba
ShabaOP•2mo ago
iex(13)> test = Ash.run_action(Ash.ActionInput.for_action(Snowmass.JobMarket.JobListing, :parse_raw, %{raw_content: raw}))
{:error,
 %Ash.Error.Unknown{
   errors: [
     %Ash.Error.Unknown.UnknownError{
       error: "Failed to cast LLM response to expected type: %Ash.Error.Unknown.UnknownError{error: \"must be more than or equal to %{min}\", field: nil, value: [field: :max_annual_salary, message: \"must be more than or equal to %{min}\", min: 1], splode: Ash.Error, bread_crumbs: [], vars: [], path: [], stacktrace: #Splode.Stacktrace<>, class: :unknown}. Raw LLM Response: \"{\\\"result\\\":{\\\"is_hourly\\\":false,\\\"is_hybrid\\\":false,\\\"is_management_role\\\":false,\\\"is_oncall_required\\\":false,\\\"is_onsite\\\":false,\\\"is_remote\\\":true,\\\"is_security_clearance_required\\\":false,\\\"is_travel_required\\\":false,\\\"max_annual_salary\\\":0,\\\"max_hourly\\\":0,\\\"min_annual_salary\\\":0,\\\"min_hourly\\\":0,\\\"min_years_of_experience\\\":1,\\\"raw_salary\\\":\\\"\\\",\\\"requirements_summary\\\":\\\"1+ redacted....\\\",\\\"title\\\":\\\"Solutions Engineer\\\"}}\"",
iex(13)> test = Ash.run_action(Ash.ActionInput.for_action(Snowmass.JobMarket.JobListing, :parse_raw, %{raw_content: raw}))
{:error,
 %Ash.Error.Unknown{
   errors: [
     %Ash.Error.Unknown.UnknownError{
       error: "Failed to cast LLM response to expected type: %Ash.Error.Unknown.UnknownError{error: \"must be more than or equal to %{min}\", field: nil, value: [field: :max_annual_salary, message: \"must be more than or equal to %{min}\", min: 1], splode: Ash.Error, bread_crumbs: [], vars: [], path: [], stacktrace: #Splode.Stacktrace<>, class: :unknown}. Raw LLM Response: \"{\\\"result\\\":{\\\"is_hourly\\\":false,\\\"is_hybrid\\\":false,\\\"is_management_role\\\":false,\\\"is_oncall_required\\\":false,\\\"is_onsite\\\":false,\\\"is_remote\\\":true,\\\"is_security_clearance_required\\\":false,\\\"is_travel_required\\\":false,\\\"max_annual_salary\\\":0,\\\"max_hourly\\\":0,\\\"min_annual_salary\\\":0,\\\"min_hourly\\\":0,\\\"min_years_of_experience\\\":1,\\\"raw_salary\\\":\\\"\\\",\\\"requirements_summary\\\":\\\"1+ redacted....\\\",\\\"title\\\":\\\"Solutions Engineer\\\"}}\"",
ZachDaniel
ZachDaniel•2mo ago
I think thats fine It should tell the agent that and it can call it again right? Oh, right nvm.
Shaba
ShabaOP•2mo ago
Yeah i guess if it was agent that would be useful as tool call
ZachDaniel
ZachDaniel•2mo ago
Strange though
Shaba
ShabaOP•2mo ago
Well its returning 0
ZachDaniel
ZachDaniel•2mo ago
Maybe the description isn't making it in?
Shaba
ShabaOP•2mo ago
I have suspicion the description is not being sent Yes I have processed 50k listings before in my previous phoenix app and never hit this issue, it was very smart
ZachDaniel
ZachDaniel•2mo ago
You can check by converting your type to json schema
Shaba
ShabaOP•2mo ago
Its not when checking the request: 
:dbg.tracer()
:dbg.p(:all, :c)
:dbg.tpl(:httpc, :request, [])
:dbg.tracer()
:dbg.p(:all, :c)
:dbg.tpl(:httpc, :request, [])
  [[]|<<"is_travel_required">>],
             <<"\":">>,
             [<<"{\"">>,
              [[]|<<"type">>],
              <<"\":">>,
              [34,[[]|<<"boolean">>],34],
              125],
             <<",\"">>,
             [[]|<<"max_annual_salary">>],
             <<"\":">>,
             [<<"{\"">>,
              [[]|<<"type">>],
              <<"\":">>,
              [34,[[]|<<"integer">>],34],
              125],
             <<",\"">>,
             [[]|<<"max_hourly">>],
             <<"\":">>,
             [<<"{\"">>,
              [[]|<<"type">>],
              <<"\":">>,
  [[]|<<"is_travel_required">>],
             <<"\":">>,
             [<<"{\"">>,
              [[]|<<"type">>],
              <<"\":">>,
              [34,[[]|<<"boolean">>],34],
              125],
             <<",\"">>,
             [[]|<<"max_annual_salary">>],
             <<"\":">>,
             [<<"{\"">>,
              [[]|<<"type">>],
              <<"\":">>,
              [34,[[]|<<"integer">>],34],
              125],
             <<",\"">>,
             [[]|<<"max_hourly">>],
             <<"\":">>,
             [<<"{\"">>,
              [[]|<<"type">>],
              <<"\":">>,
ZachDaniel
ZachDaniel•2mo ago
That is one way to do it 😄 Looking into it can you PR your required fields fix?
Shaba
ShabaOP•2mo ago
AI:
The TypedStruct isn't including the description field in the constraints. This is likely a bug in how Ash.TypedStruct generates the field constraints - it's not preserving the description metadat
The TypedStruct isn't including the description field in the constraints. This is likely a bug in how Ash.TypedStruct generates the field constraints - it's not preserving the description metadat
ZachDaniel
ZachDaniel•2mo ago
I'm working on it, but please keep in mind that if you are responding w/ AI generated content it must be clearly marked as such
Shaba
ShabaOP•2mo ago
Sorry I put it in backticks but I should be more clear Im "discussing/exploring" ash_ai with amp Ok updated my previous comments to reflect that
ZachDaniel
ZachDaniel•2mo ago
Thanks, its no big deal just letting you know Looking into the issue @Shaba please use the latest version of ash core, I just published a fix.
Shaba
ShabaOP•2mo ago
Forked ash_ai and setup postgres with pgvector and ran tests everything is working fine But making those changes breaks a bunch of tests I can take a closer look tmm Let me test your changes
ZachDaniel
ZachDaniel•2mo ago
Go ahead and open a PR and I can take a look, even w/ the failing tests
Shaba
ShabaOP•2mo ago
https://github.com/ash-project/ash_ai/pull/93
GitHub
fix: update nil-typed struct handling in OpenAPI schema generation ...
Warning tests are broken @zachdaniel Remove filter excluding allow_nil? fields from required list Properly set required fields based on constraints[:fields] keys This ensures nullable fields are ...
Shaba
ShabaOP•2mo ago
Why is this file named OpenAPI?
ZachDaniel
ZachDaniel•2mo ago
its a relic 😄 We extracted it from AshJsonApi which was building open api specs We're mid-process of reworking it to be a better fit for Ash AI
Shaba
ShabaOP•2mo ago
Ill see if I can contribute some dev setup for this? There arent really any instructions regarding the DB and the extension needed if im not mistaken I reused my devenv config but I think this relies on asdf
ZachDaniel
ZachDaniel•2mo ago
That would be wonderful 😄
Shaba
ShabaOP•2mo ago
Thank you, confirmed your update: 
            <<",\"">>,
             [[]|<<"max_annual_salary">>],
             <<"\":">>,
             [<<"{\"">>,
              [[]|<<"description">>],
              <<"\":">>,
              [34,
               [[]|
                <<"make sure this is null if it is unknown, otherwise it should be minimum 1">>],
               34],
              <<",\"">>,
              [[]|<<"type">>],
              <<"\":">>,
              [34,[[]|<<"integer">>],34],
              125],
             <<",\"">>,
            <<",\"">>,
             [[]|<<"max_annual_salary">>],
             <<"\":">>,
             [<<"{\"">>,
              [[]|<<"description">>],
              <<"\":">>,
              [34,
               [[]|
                <<"make sure this is null if it is unknown, otherwise it should be minimum 1">>],
               34],
              <<",\"">>,
              [[]|<<"type">>],
              <<"\":">>,
              [34,[[]|<<"integer">>],34],
              125],
             <<",\"">>,
ZachDaniel
ZachDaniel•2mo ago
🥳
Shaba
ShabaOP•2mo ago
I feel like there is some default conversion from null to whatever nullish value is for the type
Shaba
ShabaOP•2mo ago
No description
Shaba
ShabaOP•2mo ago
Testing in the openai playground im getting the expected result
ZachDaniel
ZachDaniel•2mo ago
I don't think there are We do "" -> null by default
Shaba
ShabaOP•2mo ago
Hmmmm even the cheapest nano model is still returning null in playground
No description
ZachDaniel
ZachDaniel•2mo ago
Sorry, what is the issue specifically?
Shaba
ShabaOP•2mo ago
My LLM json responses are including null when making them in OpenAI playground but when using AshAI they are set to nullish values 
 %Snowmass.JobMarket.Types.JobListingParseResult{
   is_hybrid: false,
   is_oncall_required: false,
   is_management_role: false,
   is_remote: true,
   is_onsite: false,
   is_travel_required: false,
   is_security_clearance_required: false,
   max_annual_salary: 0,
   min_annual_salary: 0,
   min_hourly: 0,
   max_hourly: 0,
   is_hourly: false,
   raw_salary: nil,
 %Snowmass.JobMarket.Types.JobListingParseResult{
   is_hybrid: false,
   is_oncall_required: false,
   is_management_role: false,
   is_remote: true,
   is_onsite: false,
   is_travel_required: false,
   is_security_clearance_required: false,
   max_annual_salary: 0,
   min_annual_salary: 0,
   min_hourly: 0,
   max_hourly: 0,
   is_hourly: false,
   raw_salary: nil,
ZachDaniel
ZachDaniel•2mo ago
Do you have defaults set?
Shaba
ShabaOP•2mo ago
no
ZachDaniel
ZachDaniel•2mo ago
🤔 add verbose?: true option to your promtpt action opts so you can see all the requests and responses to see whats up
Shaba
ShabaOP•2mo ago
MESSAGE PROCESSED: %LangChain.Message{
  content: "{\"result\":{\"is_hourly\":false,\"is_hybrid\":false,\"is_management_role\":false,\"is_oncall_required\":false,\"is_onsite\":false,\"is_remote\":true,\"is_security_clearance_required\":false,\"is_travel_required\":false,\"max_annual_salary\":0,\"max_hourly\":0,\"min_annual_salary\":0,\"min_hourly\":0,\"min_years_of_experience\":1,\"raw_salary\":\"\" ,\"requirements_summary\":\"Requires 1+ year in senior technical customer-facing roles, strong skills in automation tools (e.g., Zapier, Power Automate), SaaS platforms, APIs/webhooks knowledge, problem-solving, and excellent communication with technical and non-technical audiences.\",\"responsibilities_summary\":\"Design and build POCs/MVPs, lead live demos, run technical discovery calls, translate requirements into no-code workflows, build integrations, advise sales and customers, and provide product feedback.\",\"title\":\"Solutions Engineer\"}}",
  processed_content: nil,
  index: 0,
  status: :complete,
  role: :assistant,
  name: nil,
  tool_calls: [],
  tool_results: nil,
  metadata: nil
}
MESSAGE PROCESSED: %LangChain.Message{
  content: "{\"result\":{\"is_hourly\":false,\"is_hybrid\":false,\"is_management_role\":false,\"is_oncall_required\":false,\"is_onsite\":false,\"is_remote\":true,\"is_security_clearance_required\":false,\"is_travel_required\":false,\"max_annual_salary\":0,\"max_hourly\":0,\"min_annual_salary\":0,\"min_hourly\":0,\"min_years_of_experience\":1,\"raw_salary\":\"\" ,\"requirements_summary\":\"Requires 1+ year in senior technical customer-facing roles, strong skills in automation tools (e.g., Zapier, Power Automate), SaaS platforms, APIs/webhooks knowledge, problem-solving, and excellent communication with technical and non-technical audiences.\",\"responsibilities_summary\":\"Design and build POCs/MVPs, lead live demos, run technical discovery calls, translate requirements into no-code workflows, build integrations, advise sales and customers, and provide product feedback.\",\"title\":\"Solutions Engineer\"}}",
  processed_content: nil,
  index: 0,
  status: :complete,
  role: :assistant,
  name: nil,
  tool_calls: [],
  tool_results: nil,
  metadata: nil
}
Super strange I see the problem the type is only defining 1 type 
 json_schema: %{
    "name" => "result",
    "schema" => %{
      "additionalProperties" => false,
      "properties" => %{
        "result" => %{
          "additionalProperties" => false,
          "properties" => %{
            "is_hourly" => %{
              "description" => "Indicates if the job is paid hourly.",
              "type" => "boolean"
            },
            "is_hybrid" => %{
              "description" => "Indicates if the job is hybrid.",
              "type" => "boolean"
            },
            "is_management_role" => %{
              "description" => "Indicates if job is people management role.",
              "type" => "boolean"
            },
            "is_oncall_required" => %{
              "description" => "Indicates if on-call work is required for the job.",
              "type" => "boolean"
            }
...
 json_schema: %{
    "name" => "result",
    "schema" => %{
      "additionalProperties" => false,
      "properties" => %{
        "result" => %{
          "additionalProperties" => false,
          "properties" => %{
            "is_hourly" => %{
              "description" => "Indicates if the job is paid hourly.",
              "type" => "boolean"
            },
            "is_hybrid" => %{
              "description" => "Indicates if the job is hybrid.",
              "type" => "boolean"
            },
            "is_management_role" => %{
              "description" => "Indicates if job is people management role.",
              "type" => "boolean"
            },
            "is_oncall_required" => %{
              "description" => "Indicates if on-call work is required for the job.",
              "type" => "boolean"
            }
...
if allow_nil? is true in the TypedStruct, then we need the type to also be updated to: 
 [type, "null"],

ie. ["string", "null"],
 [type, "null"],

ie. ["string", "null"],
The LLM is being instructed to return that type and I guess for raw_salary, since its a string, its returning "" which gets auto converted to null in Ash Probably has to do with add_null_for_non_required Well I guess not to do with but we are only adding null for non required Im thinking this would make more sense as add_null_for_allowed_nil or whatever name Oh yeahhhh 
{:ok,
 %Snowmass.JobMarket.Types.JobListingParseResult{
   is_hybrid: false,
   is_oncall_required: nil,
   is_management_role: false,
   is_remote: true,
   is_onsite: false,
   is_travel_required: nil,
   is_security_clearance_required: false,
   max_annual_salary: nil,
   min_annual_salary: nil,
   min_hourly: nil,
   max_hourly: nil,
   is_hourly: nil,
   raw_salary: nil,
{:ok,
 %Snowmass.JobMarket.Types.JobListingParseResult{
   is_hybrid: false,
   is_oncall_required: nil,
   is_management_role: false,
   is_remote: true,
   is_onsite: false,
   is_travel_required: nil,
   is_security_clearance_required: false,
   max_annual_salary: nil,
   min_annual_salary: nil,
   min_hourly: nil,
   max_hourly: nil,
   is_hourly: nil,
   raw_salary: nil,
 json_schema: %{
    "name" => "result",
    "schema" => %{
      "additionalProperties" => false,
      "properties" => %{
        "result" => %{
          "additionalProperties" => false,
          "properties" => %{
            "is_hourly" => %{
              "anyOf" => [%{"type" => "boolean"}, %{"type" => "null"}],
              "description" => "Indicates if the job is paid hourly."
            },
            "is_hybrid" => %{
              "anyOf" => [%{"type" => "boolean"}, %{"type" => "null"}],
              "description" => "Indicates if the job is hybrid."
            },
            "is_management_role" => %{
              "anyOf" => [%{"type" => "boolean"}, %{"type" => "null"}],
              "description" => "Indicates if job is people management role."
            },
            "is_oncall_required" => %{
              "anyOf" => [%{"type" => "boolean"}, %{"type" => "null"}],
              "description" => "Indicates if on-call work is required for the job."
            },
            "is_travel_required" => %{
              "anyOf" => [%{"type" => "boolean"}, %{"type" => "null"}],
              "description" => "Indicates if travel is required for the job."
            }
 json_schema: %{
    "name" => "result",
    "schema" => %{
      "additionalProperties" => false,
      "properties" => %{
        "result" => %{
          "additionalProperties" => false,
          "properties" => %{
            "is_hourly" => %{
              "anyOf" => [%{"type" => "boolean"}, %{"type" => "null"}],
              "description" => "Indicates if the job is paid hourly."
            },
            "is_hybrid" => %{
              "anyOf" => [%{"type" => "boolean"}, %{"type" => "null"}],
              "description" => "Indicates if the job is hybrid."
            },
            "is_management_role" => %{
              "anyOf" => [%{"type" => "boolean"}, %{"type" => "null"}],
              "description" => "Indicates if job is people management role."
            },
            "is_oncall_required" => %{
              "anyOf" => [%{"type" => "boolean"}, %{"type" => "null"}],
              "description" => "Indicates if on-call work is required for the job."
            },
            "is_travel_required" => %{
              "anyOf" => [%{"type" => "boolean"}, %{"type" => "null"}],
              "description" => "Indicates if travel is required for the job."
            }
Its pretty hacky and doesnt take into account if the field is allow_nil? but essentially: 
  defp add_null_for_non_required(%{required: required} = schema)
       when is_list(required) do
    Map.update!(schema, :properties, fn
      properties when is_map(properties) ->
        Enum.reduce(properties, %{}, fn {key, value}, acc ->
          # if Enum.member?(required, key) do
          #   Map.put(acc, key, value)
          # else
  defp add_null_for_non_required(%{required: required} = schema)
       when is_list(required) do
    Map.update!(schema, :properties, fn
      properties when is_map(properties) ->
        Enum.reduce(properties, %{}, fn {key, value}, acc ->
          # if Enum.member?(required, key) do
          #   Map.put(acc, key, value)
          # else
ZachDaniel
ZachDaniel•2mo ago
Strange, its supposed to do that by default Oh, I see. We need to add null for non required before marking everying as required effectively yeah, okay
Shaba
ShabaOP•2mo ago
Im noticing the constraint is also not being represented in the schema 
"max_hourly" => %{ "description" => "If the job is paid hourly, the max hourly rate for the job.",     "type" => "integer"},
"max_hourly" => %{ "description" => "If the job is paid hourly, the max hourly rate for the job.",     "type" => "integer"},
    field :max_hourly, :integer, description: "If the job is paid hourly, the max hourly rate for the job.", allow_nil?: true, default: nil, constraints: [min: 10]
    field :max_hourly, :integer, description: "If the job is paid hourly, the max hourly rate for the job.", allow_nil?: true, default: nil, constraints: [min: 10]
might be a rabbit hole to implement this Trying to enumerate all possible constraints but idk how they map to Ash Type contraints ie. 
    "raw_salary": {
        "anyOf" : [{"type": "string", "minLength": 5 }],
        "description": "The raw format of the salary information."
      },
    "raw_salary": {
        "anyOf" : [{"type": "string", "minLength": 5 }],
        "description": "The raw format of the salary information."
      },
The LLM seems to respect it: 
  "min_hourly": null,
  "max_hourly": null,
  "is_hourly": false,
  "raw_salary": "null to be determined",
  "min_hourly": null,
  "max_hourly": null,
  "is_hourly": false,
  "raw_salary": "null to be determined",
ZachDaniel
ZachDaniel•2mo ago
I think its a good idea to implement the core constraints like that 👍
Shaba
ShabaOP•2mo ago
So im assuming this probably needs to be updated: 
  defp resource_attribute_type(%{type: Ash.Type.String}, _resource) do
    %{type: :string}
  end

  defp resource_attribute_type(%{type: Ash.Type.CiString}, _resource) do
    %{type: :string}
  end

  defp resource_attribute_type(%{type: Ash.Type.Boolean}, _resource) do
    %{type: :boolean}
  end

  defp resource_attribute_type(%{type: Ash.Type.Decimal}, _resource) do
    %{type: :string}
  end

  defp resource_attribute_type(%{type: Ash.Type.Integer}, _resource) do
    %{type: :integer}
  end
  defp resource_attribute_type(%{type: Ash.Type.String}, _resource) do
    %{type: :string}
  end

  defp resource_attribute_type(%{type: Ash.Type.CiString}, _resource) do
    %{type: :string}
  end

  defp resource_attribute_type(%{type: Ash.Type.Boolean}, _resource) do
    %{type: :boolean}
  end

  defp resource_attribute_type(%{type: Ash.Type.Decimal}, _resource) do
    %{type: :string}
  end

  defp resource_attribute_type(%{type: Ash.Type.Integer}, _resource) do
    %{type: :integer}
  end
Maybe not since thats the based schema Honestly the file is pretty confusing for me
Shaba
ShabaOP•2mo ago
Wonder if https://github.com/jonasschmidt/ex_json_schema/tree/master would be of any help
GitHub
GitHub - jonasschmidt/ex_json_schema: An Elixir JSON Schema validator
An Elixir JSON Schema validator. Contribute to jonasschmidt/ex_json_schema development by creating an account on GitHub.
Shaba
ShabaOP•2mo ago
or https://github.com/lud/jsv
GitHub
GitHub - lud/jsv: Json Schema Validator for Elixir with full spec s...
Json Schema Validator for Elixir with full spec support - lud/jsv
ZachDaniel
ZachDaniel•2mo ago
I don't think so. It should be pretty straightforward to pull the constraints out in those function heads and use them to build the apppropriate schema If you open an issue I can look into it 🙂
Shaba
ShabaOP•2mo ago
Sort of like this? AI generated: 
  defp resource_attribute_type(%{type: Ash.Type.Integer, constraints: constraints}, _resource) 
       when is_list(constraints) and constraints != [] do
    Enum.reduce(constraints, %{type: :integer}, fn
      {:min, value}, acc -> Map.put(acc, :minimum, value)
      {:max, value}, acc -> Map.put(acc, :maximum, value)
      _, acc -> acc
    end)
  end
  defp resource_attribute_type(%{type: Ash.Type.Integer, constraints: constraints}, _resource) 
       when is_list(constraints) and constraints != [] do
    Enum.reduce(constraints, %{type: :integer}, fn
      {:min, value}, acc -> Map.put(acc, :minimum, value)
      {:max, value}, acc -> Map.put(acc, :maximum, value)
      _, acc -> acc
    end)
  end
ZachDaniel
ZachDaniel•2mo ago
Yep, pretty much not sure if thats what json schema expects but otherwise yes
Shaba
ShabaOP•2mo ago
Closed the PR since its bit of mess but I pushed another change to branch where I had let the LLM run for a bit to see how far it can get. Its a bit of a mess however I think there might be some useful stuff to pull from it
Shaba
ShabaOP•2mo ago
https://github.com/ash-project/ash_ai/pull/93
GitHub
fix: update nil-typed struct handling in OpenAPI schema generation ...
Warning tests are broken @zachdaniel Remove filter excluding allow_nil? fields from required list Properly set required fields based on constraints[:fields] keys This ensures nullable fields are ...
Shaba
ShabaOP•2mo ago
I can open up an issue and also reference the PR, but if you interested and/or willing I am available/interested to pair on this
ZachDaniel
ZachDaniel•2mo ago
I wouldn't have time to pair unfortunately, either an issue describing what the problem is or a PR w/ a (vetted) fix would need to be the next steps here.
Shaba
ShabaOP•2mo ago
Cool I added an issue: https://github.com/ash-project/ash_ai/issues/95
GitHub
Incomplete JSON Schema Generation · Issue #95 · ash-project/ash_ai
Code of Conduct I agree to follow this project&#39;s Code of Conduct AI Policy I agree to follow this project&#39;s AI Policy, or I agree that AI was not used while creating this issue. Versions el...
Shaba
ShabaOP•2mo ago
@Zach Daniel Thanks for the implementing this ❤️ Taking a look at your PR, I was overcomplicating my approach to this. But yesterday when testing with Gemini, my valid json schema for openai was not valid for gemini. I am guessing these transformations may end up needing to become vendor specific Obviously something to worry about if actually becomes an issue
ZachDaniel
ZachDaniel•2mo ago
Yeah, gemini has different requirements we need to adapt that code to take options that will explain what to do And when using Gemini we can pass different options down
Shaba
ShabaOP•2mo ago
https://github.com/piotrmaciejbednarski/structllm
GitHub
GitHub - piotrmaciejbednarski/structllm: Universal Python library f...
Universal Python library for Structured Outputs with any LLM provider - piotrmaciejbednarski/structllm

Did you find this page helpful?