C#•8h ago

ClangSharpPInvokeGenerator emitting duplicate/out of order macro bindings

I am invoking ClangSharpPInvokeGenerator on a bunch of headers that altogether constitute ~30,000 #defines and nothing else. The important bit is that these constants are split over tens of headers that are to be amalgamated into a single class/file. The settings are:

--config
generate-file-scoped-namespaces
generate-macro-bindings
log-potential-typedef-remappings
multi-file
--headerFile
header.txt

--config
generate-file-scoped-namespaces
generate-macro-bindings
log-potential-typedef-remappings
multi-file
--headerFile
header.txt

and the input .rsp is (note that remap.rsp contains no entries relating to this)

@../../../../../settings.rsp
@../../../../../remap.rsp
--file
c001.ath
c002.ath
c003.ath
c004.ath
c005.ath
c006.ath
c007.ath
c008.ath
c041.ath
c043.ath
--methodClassName
CharacterModelElementId
--namespace
Fahrenheit.Core.FFX
--output
../../../../../../core/ffx/ids
--define-macro
CHRATH_INCLUDE_IGNORE

@../../../../../settings.rsp
@../../../../../remap.rsp
--file
c001.ath
c002.ath
c003.ath
c004.ath
c005.ath
c006.ath
c007.ath
c008.ath
c041.ath
c043.ath
--methodClassName
CharacterModelElementId
--namespace
Fahrenheit.Core.FFX
--output
../../../../../../core/ffx/ids
--define-macro
CHRATH_INCLUDE_IGNORE

which results in a single CharacterModelElementId.cs. However, the output regularly contains redefinition errors of the type: The type 'CharacterModelElementId' already contains a definition for 'c104_elem_i_weapon_eff' - even though there is no such duplication/redefinition in the source. Am I doing something prohibited/inadvisable by specifying multiple --file args? Should I write my own header that #includes all the ones I want to consume and only feed ClangSharpPInvokeGenerator that?

42 Replies

peppyOP•8h ago

An example of the input header contents:

#ifndef    __c001_ath
#define    __c001_ath

// elements.
#define c001_elem_nb    136
#define    c001_elem_c001    0
#define    c001_elem_tida_root    1
#define    c001_elem_chest1_chn    2
#define    c001_elem_chest1    3
#define    c001_elem_chest1_eff    4

#ifndef    __c001_ath
#define    __c001_ath

// elements.
#define c001_elem_nb    136
#define    c001_elem_c001    0
#define    c001_elem_tida_root    1
#define    c001_elem_chest1_chn    2
#define    c001_elem_chest1    3
#define    c001_elem_chest1_eff    4

This is for version 20.1.2.1 since I'm not ready to migrate to .NET 10 yet.

PS D:\fh\tools> dotnet tool list --global
Package Id                      Version       Commands
------------------------------------------------------------------------
clangsharppinvokegenerator      20.1.2.1      ClangSharpPInvokeGenerator

PS D:\fh\tools> dotnet tool list --global
Package Id                      Version       Commands
------------------------------------------------------------------------
clangsharppinvokegenerator      20.1.2.1      ClangSharpPInvokeGenerator

The alarming bit is that the amount of produced errors varies by script run. I modeled the way I invoke it on TerraFX, and the script performing traversal/invocation is:

function Generate() {
  $generationDir = Join-Path -Path $RepoRoot -ChildPath "generation"
  $generateRspFiles = Get-ChildItem -Path "$generationDir" -Recurse -Filter "generate*.rsp"

  $generateRspFiles | ForEach-Object {
    Push-Location -Path $_.DirectoryName
    & ClangSharpPInvokeGenerator "@$($_.Name)"
    Pop-Location
  }
}

try {
  $RepoRoot = Join-Path -Path $PSScriptRoot -ChildPath ".."
  Generate
}
catch {
  Write-Host -Object $_
  Write-Host -Object $_.Exception
  Write-Host -Object $_.ScriptStackTrace
  exit 1
}

function Generate() {
  $generationDir = Join-Path -Path $RepoRoot -ChildPath "generation"
  $generateRspFiles = Get-ChildItem -Path "$generationDir" -Recurse -Filter "generate*.rsp"

  $generateRspFiles | ForEach-Object {
    Push-Location -Path $_.DirectoryName
    & ClangSharpPInvokeGenerator "@$($_.Name)"
    Pop-Location
  }
}

try {
  $RepoRoot = Join-Path -Path $PSScriptRoot -ChildPath ".."
  Generate
}
catch {
  Write-Host -Object $_
  Write-Host -Object $_.Exception
  Write-Host -Object $_.ScriptStackTrace
  exit 1
}

Petris•8h ago

cc @Tanner Gooding

peppyOP•8h ago

It also varies per script invocation which entries get duplicated, for what it's worth.

tannergooding•8h ago

Redefinitions likely depend on how your files are setup and include eachother if you process a.h and then separately process b.h but that has #include a.h then it may end up processing a "twice" it's hard to say for sure without seeing the headers for TerraFX, I typically give a single file that #include <...> each file I need as a dependency or want to support and then specify --traverse to indicate what files I actually want to generate for, to avoid the recursive inclusion problem

peppyOP•8h ago

in this case, they are only a bunch of #defines with no #includes

peppyOP•8h ago

occasionally, I also observe spurious failure to process, which has the following trace:

message.txt

peppyOP•8h ago

(never in the same header twice)

tannergooding•8h ago

what version of the tool are you using and are you sure you have a matching version of libClang?

peppyOP•8h ago

20.1.2.1, and I haven't checked beyond simply installing the tool globally

tannergooding•8h ago

.3 has some patches, so I'd try with that first but should otherwise work are you on Windows?

peppyOP•8h ago

I am on Windows, 11 24H2, and I think I went for this version because the later ones demand .NET 10

tannergooding•8h ago

right, which is "production ready" and has a number of perf improvements around it (plus just general fixes to the ClangSharp generator were made)

Petris•8h ago

well issue is that linux distros, package managers and such don't have RCs and I guess corporate policies could block non releases too

tannergooding•8h ago

good thing you can get the binaries manually and this isn't on Linux and this is considered production ready so it shouldn't trigger any corporate policy

peppyOP•8h ago

I would just have to install the RC1 SDK from the .NET website and then the tool can update?

tannergooding•8h ago

yep

peppyOP•8h ago

Sure, I'll go do that and report back; in the meantime, how I can check this thing you asked about having a matching libClang? Or is that not applicable in this situation

tannergooding•8h ago

it shouldn't matter for this case, since you're on Windows and 20.1 or later

Petris•8h ago

does the tool not check if the libclang is matching automatically??

peppyOP•8h ago

I would expect it to, but I am asking all the same since I'm unfamiliar with the subject matter

tannergooding•8h ago

it does, but there are still quirks on Linux where that can get wonky which is why I asked all the questions

Petris•8h ago

i'd have assumed libclang does have some get version export

tannergooding•8h ago

yes, but that isn't fully accurate to the degree needed because they make breaks even in patch releases in some cases

Petris•8h ago

does that export not provide the patch number?

peppyOP•8h ago

Tool 'clangsharppinvokegenerator' was successfully updated from version '20.1.2.1' to version '20.1.2.3'. - rerunning now

tannergooding•8h ago

Not always, no. Again, the Linux builds get quite wonky and there's a ton of special handling required in some cases which only goes so far

peppyOP•7h ago

message.txt

peppyOP•7h ago

a throw of this form is now given

peppyOP•7h ago

for the input header

anc_c105.ath

peppyOP•7h ago

another form of throw occurs for other headers:

Diagnostics for binding generation of mot_n356.ath:
    Error: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at ClangSharp.PInvokeGenerator.GetSourceRangeContents(CXTranslationUnit, CXSourceRange) + 0x167
   at ClangSharp.PInvokeGenerator.VisitMacroDefinitionRecord(MacroDefinitionRecord) + 0x414
   at ClangSharp.PInvokeGenerator.VisitPreprocessingDirective(PreprocessingDirective) + 0x4c
   at ClangSharp.PInvokeGenerator.VisitPreprocessedEntity(PreprocessedEntity) + 0x160
   at ClangSharp.PInvokeGenerator.GenerateBindings(TranslationUnit, String, String[], CXTranslationUnit_Flags) + 0x2af

Diagnostics for binding generation of mot_n356.ath:
    Error: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
   at ClangSharp.PInvokeGenerator.GetSourceRangeContents(CXTranslationUnit, CXSourceRange) + 0x167
   at ClangSharp.PInvokeGenerator.VisitMacroDefinitionRecord(MacroDefinitionRecord) + 0x414
   at ClangSharp.PInvokeGenerator.VisitPreprocessingDirective(PreprocessingDirective) + 0x4c
   at ClangSharp.PInvokeGenerator.VisitPreprocessedEntity(PreprocessedEntity) + 0x160
   at ClangSharp.PInvokeGenerator.GenerateBindings(TranslationUnit, String, String[], CXTranslationUnit_Flags) + 0x2af

(considering that 20.1.2.1 did give valid results after being run enough times to avoid the error, I wouldn't say the input is malformed) there is still no consistency for which header will fail in which of these ways

tannergooding•7h ago

does it repro if you only have a single --file?

peppyOP•7h ago

by single --file you mean one that #includes all current input, or literally just one of them

tannergooding•7h ago

literally just one of them

peppyOP•7h ago

it does not, in five invocations

tannergooding•7h ago

so maybe I've got a leak or something for multiple files that'd be a pita 😄

peppyOP•7h ago

nor has any of the generate.rsps I have that use single --file args ever reproed not to my recollection, anyway would a workaround for this be to do what you do in TerraFX and #include-all in one header

tannergooding•7h ago

yeah, just doing a single file that includes all the files and specifying those as traverse instead would workaround this I just got a repro, but it requires release build so let me see if I can figure it out Looks like the in memory file buffer gets corrupted somehow think I found the issue... there's a CXFile to contents cache but the entry isn't removed after the file is closed and Clang may then reuse the file handle

peppyOP•7h ago

:catfine: I'm glad to hear it didn't take an eternity to find, at least Thank you for looking into it

tannergooding•7h ago

This should fix it: https://github.com/dotnet/ClangSharp/pull/637. I won't have a new release up until later, but if you want to build locally and use the tool instead of working around, that would be fine

peppyOP•6h ago

it can wait for the next release the important thing is that it is fixed, so I can rely on specifying --file X Y Z W {...} since I do have a lot of headers for which I'd have to manually apply the workaround (and the sample was abridged; it's usually ~30 headers per "unit" or .rsp, so it adds up)

tannergooding•6h ago

right, and the more files you have the more likely you'll hit a recycled id

peppyOP•6h ago

thank you very much for taking the time

Gaming

Programming

ClangSharpPInvokeGenerator emitting duplicate/out of order macro bindings

Did you find this page helpful?