C#•4y ago

Can't get items to be added to update

I can't seem to get the "update" List filled..
I have this Json data https://hastebin.com/bazadowese.json
What I am trying to do, is everything that is not in the targetJson should be added to the "inject" list, and the items that are in the targetJson should be checked against the items in the sourceJson, if they are not in there, done. If they are check if they have differences, if they have differences add that item to the "update" list.

Currently it just adds everything from the sourceJson into the "inject" list and doesn't do squat for the update list.

Here's the function:

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
LookupInputs data = JsonConvert.DeserializeObject<LookupInputs>(requestBody);
if (data == null)
    return new BadRequestObjectResult("There was no data provided, or the data was not formatted properly");

Dictionary<string, List<Dictionary<string, JToken>>> result = new()
{
    { "inject", new() },
    { "update", new() }
};

if (data.TargetJson.Count == 0)
{
    result["inject"] = (from item in data.SourceJson select item).ToList();
}
else
{
    result["inject"] = (from source in data.SourceJson
        from target in data.TargetJson
        where source[data.CommonIdentifier] != target[data.CommonIdentifier] && !source.DictEquals(target)
        select source).ToList();

    result["update"] = (from source in data.SourceJson
        from target in data.TargetJson
        where source[data.CommonIdentifier] == target[data.CommonIdentifier] // && !source.DictEquals(target)
        select target).ToList();
}

return new OkObjectResult(results);

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
LookupInputs data = JsonConvert.DeserializeObject<LookupInputs>(requestBody);
if (data == null)
    return new BadRequestObjectResult("There was no data provided, or the data was not formatted properly");

Dictionary<string, List<Dictionary<string, JToken>>> result = new()
{
    { "inject", new() },
    { "update", new() }
};

if (data.TargetJson.Count == 0)
{
    result["inject"] = (from item in data.SourceJson select item).ToList();
}
else
{
    result["inject"] = (from source in data.SourceJson
        from target in data.TargetJson
        where source[data.CommonIdentifier] != target[data.CommonIdentifier] && !source.DictEquals(target)
        select source).ToList();

    result["update"] = (from source in data.SourceJson
        from target in data.TargetJson
        where source[data.CommonIdentifier] == target[data.CommonIdentifier] // && !source.DictEquals(target)
        select target).ToList();
}

return new OkObjectResult(results);

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
LookupInputs data = JsonConvert.DeserializeObject<LookupInputs>(requestBody);
if (data == null)
    return new BadRequestObjectResult("There was no data provided, or the data was not formatted properly");

Dictionary<string, List<Dictionary<string, JToken>>> result = new()
{
    { "inject", new() },
    { "update", new() }
};

if (data.TargetJson.Count == 0)
{
    result["inject"] = (from item in data.SourceJson select item).ToList();
}
else
{
    result["inject"] = (from source in data.SourceJson
        from target in data.TargetJson
        where source[data.CommonIdentifier] != target[data.CommonIdentifier] && !source.DictEquals(target)
        select source).ToList();

    result["update"] = (from source in data.SourceJson
        from target in data.TargetJson
        where source[data.CommonIdentifier] == target[data.CommonIdentifier] // && !source.DictEquals(target)
        select target).ToList();
}

return new OkObjectResult(results);

string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
LookupInputs data = JsonConvert.DeserializeObject<LookupInputs>(requestBody);
if (data == null)
    return new BadRequestObjectResult("There was no data provided, or the data was not formatted properly");

Dictionary<string, List<Dictionary<string, JToken>>> result = new()
{
    { "inject", new() },
    { "update", new() }
};

if (data.TargetJson.Count == 0)
{
    result["inject"] = (from item in data.SourceJson select item).ToList();
}
else
{
    result["inject"] = (from source in data.SourceJson
        from target in data.TargetJson
        where source[data.CommonIdentifier] != target[data.CommonIdentifier] && !source.DictEquals(target)
        select source).ToList();

    result["update"] = (from source in data.SourceJson
        from target in data.TargetJson
        where source[data.CommonIdentifier] == target[data.CommonIdentifier] // && !source.DictEquals(target)
        select target).ToList();
}

return new OkObjectResult(results);

this is the current result: https://hastebin.com/buxaroneda.json
the result I'm expecting is that the item "BookID": "6" is left out of the "inject" list, and added to the "update" list

becquerel•9/30/22, 10:57 AM

Have you confirmed that data.TargetJson.Countdata.TargetJson.Count is greater than 0?

malkavOP•9/30/22, 10:57 AM

yes, I've been logging all over the place

the data.TargetJson.Countdata.TargetJson.Count == 1

becquerel•9/30/22, 10:58 AM

ok - didn't think it would be that simple

might take me a min to parse your logic...

malkavOP•9/30/22, 10:58 AM

sorry if it's a mess to sort through

becquerel•9/30/22, 10:59 AM

something should be in update if:

it is in targetJson
is also in sourceJson
has differences to the corresponding item in sourceJson

becquerel•9/30/22, 10:59 AM

is that correct?

becquerel•9/30/22, 11:00 AM

if so, don't you want that !source.DictEquals comment added back in?

becquerel•9/30/22, 11:00 AM

just as a tip - there is an explicit joinjoin syntax in LINQ you can use to make it a bit cleaner

malkavOP•9/30/22, 11:01 AM

Let me "Try" and help

if (data.TargetJson.Count == 0)if (data.TargetJson.Count == 0)
this part just adds all the items from data.SourceJsondata.SourceJson into the "inject" list of the results

now comes the fun part lol
result["inject"] = ...result["inject"] = ... is supposed to iterate through both data.TargetJsondata.TargetJson and data.SourceJsondata.SourceJson and return all the items from data.SourceJsondata.SourceJson that are not in data.TargetJsondata.TargetJson (though I see that it is returning the item if only one property of the item in the Dictionary is different because they won't equal, but whatever)

result["update'] = ...result["update'] = ... is supposed to return the items from data.TargetJsondata.TargetJson that have the same CommonIdentifierCommonIdentifier as in data.SourceJsondata.SourceJson but have the rest of the properties different (so it needs updating)

becquerel•9/30/22, 11:01 AM

var x = from source in data.SourceJson
        join target in data.TargetJson on // blah
        select ...

var x = from source in data.SourceJson
        join target in data.TargetJson on // blah
        select ...

Bbecquerel if so, don't you want that !source.DictEquals comment added back in?

malkavOP•9/30/22, 11:01 AM

yea I was testing if that resulted in something different

Bbecquerel ```cs var x = from source in data.SourceJson join target in data.TargetJ...

malkavOP•9/30/22, 11:02 AM

not sure if I want to join, since I want the item to be pushed to a new List in the resultsresults variable? but I'm not sure what join would do in this line of code

becquerel•9/30/22, 11:02 AM

return all the items from data.SourceJson that are not in data.TargetJson

would data.SourceJson.Except(data.TargetJson)data.SourceJson.Except(data.TargetJson) work here?

malkavOP•9/30/22, 11:03 AM

I have that in my DictEqualsDictEquals method
Wait, here's that method:

public static bool DictEquals<TKey, TValue>(this Dictionary<TKey, TValue> source, Dictionary<TKey, TValue> target) 
        => source.Count == target.Count && !source.Except(target).Any();

public static bool DictEquals<TKey, TValue>(this Dictionary<TKey, TValue> source, Dictionary<TKey, TValue> target) 
        => source.Count == target.Count && !source.Except(target).Any();

malkavOP•9/30/22, 11:03 AM

I tested, and that acutally does the Equals for me quite well so far

Mmalkav not sure if I want to join, since I want the item to be pushed to a new List in ...

becquerel•9/30/22, 11:03 AM

the join is equivalent to your thing of using two fromfroms with a

where

where

on the commonidentifier

Bbecquerel the join is equivalent to your thing of using two `from`s with a `where` on the ...

malkavOP•9/30/22, 11:04 AM

Ah fair enough, so it pretty much would do the same but cleaner?

TeBeCo•9/30/22, 11:04 AM

i still don't get why the original code invole JToken and Dictionary

TeBeCo•9/30/22, 11:04 AM

¯\_(ツ)_/¯

TeBeCo•9/30/22, 11:04 AM

deserialize to C# POCO ... map / add to REAL list => reserialize

TTeBeCo **__i still don't get why the original code invole JToken and Dictionary__**

malkavOP•9/30/22, 11:04 AM

Because the JToken represents whatever data in the JSON object would be the value of the key.
The string is the key

TeBeCo•9/30/22, 11:04 AM

no JSON magic nowhere

TeBeCo•9/30/22, 11:04 AM

yeah, don't

TeBeCo•9/30/22, 11:04 AM

use POCO

TeBeCo•9/30/22, 11:04 AM

work on that

TeBeCo•9/30/22, 11:04 AM

simple C#

TeBeCo•9/30/22, 11:05 AM

your data is uniform

malkavOP•9/30/22, 11:05 AM

wait, lemme explain, and you can correct me on that

malkavOP•9/30/22, 11:06 AM

the example JSON I sent is not the data I will always get...
What I will receive is a JSON array of objects. These objects have keys (string) and values (unknown)
so it was either dynamic, JValue, JToken, or whichever other type of dynamic data because the values can be int, string, objects, or whatever data type

TeBeCo•9/30/22, 11:06 AM

public record Book(string BookID, string BookTitle, sintrg BookPrice);
public record WeirdJson(Book[] sourceJson, Book[] targetJson, string commonIdentifier);

var weirdStuff = STJ.Deserialize<WeirdJson>(inputJson);

// work with C# here

public record Book(string BookID, string BookTitle, sintrg BookPrice);
public record WeirdJson(Book[] sourceJson, Book[] targetJson, string commonIdentifier);

var weirdStuff = STJ.Deserialize<WeirdJson>(inputJson);

// work with C# here

malkavOP•9/30/22, 11:07 AM

No, no, the example is uniform but the data I will receive is not

TeBeCo•9/30/22, 11:07 AM

oO so you mean the code and data you sent is not like the actual one ?

malkavOP•9/30/22, 11:08 AM

the code is the actual code I intend to use, but the data I send is not the actual one.
the data I will receive is based on whatever the user needs to do a Lookup TransformationLookup Transformation on

malkavOP•9/30/22, 11:08 AM

but in SSIS and not in Informatica

malkavOP•9/30/22, 11:11 AM

What I'm finding so far is that source[data.CommonIdentifier] == target[data.CommonIdentifier]source[data.CommonIdentifier] == target[data.CommonIdentifier] isn't returning a value. While I'm sure that in the example data I sent both have a BookIDBookID of "6""6", so I'm a little confused why it won't return a value, not even in this loop:

foreach (Dictionary<string, JToken> source in data.SourceJson)
{
    log.LogDebug("Iterating through both dictionaries to check if the target even gets through");
    foreach (Dictionary<string, JToken> target in data.TargetJson.Where(target =>
                 source[data.CommonIdentifier] == target[data.CommonIdentifier]))
    {
        log.LogDebug("Target item: {T}, {S}", target[data.CommonIdentifier], source[data.CommonIdentifier]);
    }
}

foreach (Dictionary<string, JToken> source in data.SourceJson)
{
    log.LogDebug("Iterating through both dictionaries to check if the target even gets through");
    foreach (Dictionary<string, JToken> target in data.TargetJson.Where(target =>
                 source[data.CommonIdentifier] == target[data.CommonIdentifier]))
    {
        log.LogDebug("Target item: {T}, {S}", target[data.CommonIdentifier], source[data.CommonIdentifier]);
    }
}

TeBeCo•9/30/22, 11:12 AM

the code is the actual code I intend to use, but the data I send is not the actual one.

I mean if you send data that is not the why you'll use, the code is irrelevant

TeBeCo•9/30/22, 11:12 AM

proof being my suggestion to remove 99% of your code since it would have been useless

malkavOP•9/30/22, 11:12 AM

because I had to have some test data don't I

TeBeCo•9/30/22, 11:12 AM

¯\_(ツ)_/¯

TeBeCo•9/30/22, 11:13 AM

it's probably impossible to ask "how to parse data" is no one know the data format

TeBeCo•9/30/22, 11:13 AM

(or have the wrong format)

malkavOP•9/30/22, 11:14 AM

how would you go about this then?
I get three values from the user

source

source

target

target

, and commonIdentifiercommonIdentifier
source and target are both an Array of objects
and commonIdentifier is the key to which the value should match

TeBeCo•9/30/22, 11:14 AM

if the data is inconsistent in your JSON then either ... it's trully random and yes you use JObject/JToken (if you use Newtonsoft)

TeBeCo•9/30/22, 11:14 AM

or it's not really random but more like inheritance

malkavOP•9/30/22, 11:14 AM

I use Newtonsoft

TeBeCo•9/30/22, 11:14 AM

and then you use json differenciator

TeBeCo•9/30/22, 11:14 AM

and you could then still use C#

Mmalkav I use Newtonsoft

TeBeCo•9/30/22, 11:15 AM

STJ also have support for JSON DOM API

TeBeCo•9/30/22, 11:15 AM

that;s all I mean

TeBeCo•9/30/22, 11:15 AM

Utf8JsonReaderUtf8JsonReader

Can't get items to be added to update

Similar Threads

Similar Threads

Similar Threads