C#•2w ago

I have a program that crashes the CLR. How do I go about debugging the root cause?

I have a program that sometimes, but not always, crashes the CLR. It might be something to do with accessing an enumerator from a catch block, but that might be a red-herring and it might be coming from a library I'm using. I'd like to work out whether it's my own code or the library before I raise a bug against the library. So I need some help on how to diagnose what exactly is happening. Most of the time it crashes with no output at all, sometimes the only output is Fatal error. , and once, I got Fatal error. 0xC0000005 which is Access Violation. Just one, but I've not seen it since, was an actual mini CLR stack trace in which the error message was something like "CLR Crash" but I don't remember the detail.

34 Replies

Dongle•2w ago

Do you or the library use DllImport?

etermOP•2w ago

I'm pretty sure the library does, afaik it's just a wrapper / bindings around the C version of the library. I've checked and yes it does, extensively. DuckDB.NET is the library.

Dongle•2w ago

Yeah you should report it, access violations are usually the result of bad DllImport/unsafe code.

etermOP•2w ago

Is there anything I can do to make it more reproducible? It only sometimes happens.

Dongle•2w ago

I can’t really think of any, Access Violations are usually very random. Maybe try enabling GCStress? You do that by setting the DOTNET_GCStress environment variable to one of https://github.com/dotnet/runtime/blob/main/docs/design/coreclr/jit/investigate-stress.md#gc-hole-stress I’d say 0x3 based on descriptions.

GitHub

runtime/docs/design/coreclr/jit/investigate-stress.md at main · do...

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps. - dotnet/runtime

Dongle•2w ago

Note that the app will run very slowly

etermOP•2w ago

Thanks, I'll give that a go. I actually managed to get the mini stack trace again now which is enough for me to be happier to report it to the duckDB.NET library.

Fatal error. Internal CLR error. (0x80131506)
   at DuckDB.NET.Data.DuckDBAppender.ThrowLastError(DuckDB.NET.Native.DuckDBAppender)
   at DuckDB.NET.Data.DuckDBAppender.AppendDataChunk()
   at DuckDB.NET.Data.DuckDBAppender.CreateRow()
   at Program.<Main>$(System.String[])

Fatal error. Internal CLR error. (0x80131506)
   at DuckDB.NET.Data.DuckDBAppender.ThrowLastError(DuckDB.NET.Native.DuckDBAppender)
   at DuckDB.NET.Data.DuckDBAppender.AppendDataChunk()
   at DuckDB.NET.Data.DuckDBAppender.CreateRow()
   at Program.<Main>$(System.String[])

I hope this isn't a sign of my memory going bad, I probably ought to do a mem test too to make sure.

Petris•2w ago

Huh

Petris•2w ago

Does Internal CLR error ignore stacktracehidden?

Lex Li•2w ago

If the crash was related to native memory issues, then very likely you need very experienced professionals to assist on troubleshooting (Some typical scenarios have been documented by experts like Tess Ferrandez, https://www.tessferrandez.com/postindex/). You might reach out to DuckDB.NET authors and see if they have the capacity to work on this (Their GitHub issue track does show similar issues resolved in the past).

Petris•2w ago

Anyway I checked the error method itself and it didn't seem wrong So seems like some memory corruption Whether it's on your end or somewhere else in the library is another matter

Petris•2w ago

Lol

Dongle•2w ago

Oh my god lmao. Definitely on the library lol.

etermOP•2w ago

I've uploaded a minimal reproduction to GitHub ( https://github.com/richardcocks/CLRCrashReproduction ) , I'm a bit curious whether it crashes for anyone else. It doesn't crash for me when running inside WSL. I guess there's a chance it really is my memory. I'll go download memtest86; I haven't done that since it fit on a floppy disk.

GitHub

GitHub - richardcocks/CLRCrashReproduction

Contribute to richardcocks/CLRCrashReproduction development by creating an account on GitHub.

Giorgi•2w ago

@Petris Can you suggest how I should fix it? That will be more helpful for me than laughing 🙃

Giorgi•2w ago

@Dongle @Petris This is the source for that message: https://github.com/Giorgi/DuckDB.NET/issues/178#issuecomment-2195789680 Can you suggest a fix for that?

GitHub

Named parameters throw AccessViolationException when debugging · I...

When running a query like: command.CommandText = "SELECT * FROM 'some.parquet' WHERE some_field IN ($p0, $p1)"; command.Parameters.Add(new DuckDBParameter("p1", things[0...

Giorgi•2w ago

@eterm I figured out why it's not working the way you expect it to work. I don't know why the CLR is crashing but the app doesn't work the way you expect because of a type mismatch

etermOP•2w ago

@Giorgi - That's great to hear, thanks. By the way, I tried to clone the repo to diagnose it myself, but when I build with dotnet build, it doesn't download the duckdb.dll library. Is there a particular msbuild command I need to run to make sure it includes the download? I noticed there's some targets set up, but I'm not sure how to make use of them?

Giorgi•2w ago

You need to pass /p:BuildType=Full The reason it doesn't work is that you have a UINTEGER column but you are inserting long values Changing the table to CREATE TABLE IF NOT EXISTS integers(sequence BIGINT, value BIGINT UNIQUE); gets rid of the CLR crash When using the appender API, it's your responsibility to make sure that data types are matching But I don't know why it wasn't failing in WSL

etermOP•2w ago

Oh that's interesting, I'm inserting long but the long values are only ever positive, so they should fit in a UINT. I dind't realise the types had to be an exact match. Oh rather, the're in the range (0, int.maxvalue-1 ) because they're coming from Random.Next

Giorgi•2w ago

UINTEGER is 4 bytes long is 8 bytes It doesn't matter what the actual value of the long variables are. If I change the table to use UINTEGER column and just insert 1 as long it will still crash

etermOP•2w ago

Oh right, that's interesting, I guess that's just the nature of how the appender works is it? I wonder why it sometimes works and sometimes doesn't then.

Giorgi•2w ago

When using the appender I write directly to the underlying memory:

internal bool AppendValueInternal<T>(T value, ulong rowIndex) where T : unmanaged
{
    ((T*)vectorData)[rowIndex] = value;
    return true;
}

internal bool AppendValueInternal<T>(T value, ulong rowIndex) where T : unmanaged
{
    ((T*)vectorData)[rowIndex] = value;
    return true;
}

When you call it for long (8 bytes) but the underlying vector is a pointer to uint (4 bytes) you are causing AV

etermOP•2w ago

Right, that does explain it, thanks 👍 Does that cause problems the other way too, if it were a UBIGINT but you wrote an UINT?

Giorgi•2w ago

I'm not sure but it's better to have a 100% match between column types and value types It probably makes sense to add a source generator to generate type safe appender for specific types

etermOP•2w ago

It'd be worth putting up a warning in the bulk-data-loading docs page too, especially since it looks like using too-short types causes "random" memory to be written to the database.

Giorgi•2w ago

@eterm By the way, I created this request for your other issue: https://github.com/duckdb/duckdb/discussions/17532

GitHub

C API - Get error type from appender · duckdb duckdb · Discussion...

We have duckdb_result_error_type for getting error type from a duckdb_result but when using Appender API there is no way to get error type from an appender. Would be helpful to have duckdb_appender...

Giorgi•2w ago

You can send a PR for the docs too: https://github.com/Giorgi/DuckDB.NET-Docs

GitHub

GitHub - Giorgi/DuckDB.NET-Docs: DuckDB.NET project documentation.

DuckDB.NET project documentation. Contribute to Giorgi/DuckDB.NET-Docs development by creating an account on GitHub.

etermOP•2w ago

OK, will do, thanks.

Giorgi•2w ago

There is a dotnet channel in the duckdb discord server. If you have any questions in future, feel free to ping me there

Giorgi•2w ago

@eterm https://discord.duckdb.org/

Discord

Join the DuckDB Discord Server!

Check out the DuckDB community on Discord – hang out with 7238 other members and enjoy free voice and text chat.

Dongle•2w ago

I’d recommend that for P/Invoke scenarios you use LibraryImport or ClangSharpPInvokeGenerator. The former makes it easier to determine how marshaling works and if there are any incorrect signatures. The latter will ensure correct signatures.

Giorgi•2w ago

I'll move to LibraryImport when I drop .net standard support.

Dongle•2w ago

Note that it will not immediately solve your problem, just makes it easier to see where it is. ClangSharpPInvokeGenerator will (for most cases) guarantee correct P/Invoke methods, although is harder to use + will most likely require a breaking change

Gaming

Programming

I have a program that crashes the CLR. How do I go about debugging the root cause?

Did you find this page helpful?