❔ Format parsing problem with character collision

Asher · 2023-09-13T13:38:02.988Z

Hi everyone, assume I have the following bencode list`l4:spam5:helloe` (where the last `e` signifies the end of the list) Apart from reading each element until there is an e which would create code duplication for me, is there any way to easily identify the difference between an e inside of a string and an e as the end of the list object? (or other objects such as integers for that matter)

A

AsherOP•9/13/23, 1:39 PM

int indexOfNextE = str.IndexOf('e', i);
                    // TODO: add 0 check
                    BencodeInteger newInteger = DecodeInteger(str.Substring(i + 1, indexOfNextE - 1));
                    list.Add(newInteger);

int indexOfNextE = str.IndexOf('e', i);
                    // TODO: add 0 check
                    BencodeInteger newInteger = DecodeInteger(str.Substring(i + 1, indexOfNextE - 1));
                    list.Add(newInteger);

int indexOfNextE = str.IndexOf('e', i);
                    // TODO: add 0 check
                    BencodeInteger newInteger = DecodeInteger(str.Substring(i + 1, indexOfNextE - 1));
                    list.Add(newInteger);

int indexOfNextE = str.IndexOf('e', i);
                    // TODO: add 0 check
                    BencodeInteger newInteger = DecodeInteger(str.Substring(i + 1, indexOfNextE - 1));
                    list.Add(newInteger);

this is an example of what I mean important to note, here I can be sure that the next

will be the end of the object

A

AsherOP•9/13/23, 1:40 PM

https://wiki.theory.org/BitTorrentSpecification#Bencoding <- the format I'm working with

P

Pobiega•9/13/23, 2:00 PM

Not really. bencoding relies on the length of each item, as I'm sure you know

P

Pobiega•9/13/23, 2:01 PM

the only reason we know that

ends the list is that

hello

hello

hello

hello was length-prefixed

P

Pobiega•9/13/23, 2:01 PM

Does your parser not handle this for you?

A

AsherOP•9/13/23, 2:01 PM

I'm building one from scratch

P

Pobiega•9/13/23, 2:01 PM

Thats fair, but again, why are you not handling this?

P

Pobiega•9/13/23, 2:02 PM

Not sure how this would result in code duplication. A simple parser combinator would handle bencoding fairly well I imagine

A

AsherOP•9/13/23, 2:03 PM

well, I have 4 separate functions for parsing each type of object str int dict & list

P

Pobiega•9/13/23, 2:03 PM

sure, yep

A

AsherOP•9/13/23, 2:03 PM

string and int were easy enough obviously

A

AsherOP•9/13/23, 2:03 PM

but a list can contain a list

A

AsherOP•9/13/23, 2:03 PM

in which case, I'd prob need to call the function recursively

P

Pobiega•9/13/23, 2:03 PM

Makes sense, yep

A

AsherOP•9/13/23, 2:04 PM

the way I built my functions, they get a substring that they parse in their own way and return (my version) some BencodeObject derived class

A

AsherOP•9/13/23, 2:05 PM

for example, in the list parse function, this is how I parse a string

A

AsherOP•9/13/23, 2:05 PM

            for (int i = 0; i < str.Length; i++)
            {
                char c = str[i];

                // BencodeString
                if(char.IsNumber(c))
                {
                    int j = i;
                    while(str[j] != ':')
                    {
                        j++;
                    }

                    int numCharacters = int.Parse(str.Substring(i, j - 1));

                    BencodeString newString = DecodeString(str.Substring(i, j + numCharacters));
                    list.Add(newString);
                    i = j + numCharacters;
                }
                ... // rest of the objects

            for (int i = 0; i < str.Length; i++)
            {
                char c = str[i];

                // BencodeString
                if(char.IsNumber(c))
                {
                    int j = i;
                    while(str[j] != ':')
                    {
                        j++;
                    }

                    int numCharacters = int.Parse(str.Substring(i, j - 1));

                    BencodeString newString = DecodeString(str.Substring(i, j + numCharacters));
                    list.Add(newString);
                    i = j + numCharacters;
                }
                ... // rest of the objects

            for (int i = 0; i < str.Length; i++)
            {
                char c = str[i];

                // BencodeString
                if(char.IsNumber(c))
                {
                    int j = i;
                    while(str[j] != ':')
                    {
                        j++;
                    }

                    int numCharacters = int.Parse(str.Substring(i, j - 1));

                    BencodeString newString = DecodeString(str.Substring(i, j + numCharacters));
                    list.Add(newString);
                    i = j + numCharacters;
                }
                ... // rest of the objects

            for (int i = 0; i < str.Length; i++)
            {
                char c = str[i];

                // BencodeString
                if(char.IsNumber(c))
                {
                    int j = i;
                    while(str[j] != ':')
                    {
                        j++;
                    }

                    int numCharacters = int.Parse(str.Substring(i, j - 1));

                    BencodeString newString = DecodeString(str.Substring(i, j + numCharacters));
                    list.Add(newString);
                    i = j + numCharacters;
                }
                ... // rest of the objects

A

AsherOP•9/13/23, 2:05 PM

list

list

list

list is the list that will eventually be returned from this function

A

AsherOP•9/13/23, 2:05 PM

if that makes sense

A

AsherOP•9/13/23, 2:06 PM

so assume that i=0 is right where the list starts and i = str.length - 1 is an e that ends the list

P

Pobiega•9/13/23, 2:06 PM

A fairly common approach to parsers is to return two things

P

Pobiega•9/13/23, 2:07 PM

first the result of the parse (the list, string, int etc) and secondly.. the rest of the string

P

Pobiega•9/13/23, 2:07 PM

ie, where our parser stopped

A

AsherOP•9/13/23, 2:09 PM

I assume this is where I do

i = j + numCharacters

i = j + numCharacters

i = j + numCharacters

i = j + numCharacters

A

AsherOP•9/13/23, 2:09 PM

but in way of a return statement

P

Pobiega•9/13/23, 2:09 PM

sort of, I guess

P

Pobiega•9/13/23, 2:09 PM

Let me just say that I'm in no way an expert on parsers, and I've only ever written parser combinators before

P

Pobiega•9/13/23, 2:09 PM

but its a very neat way of doing it

A

AsherOP•9/13/23, 2:11 PM

I will mention I don't know what parser combinators are

A

AsherOP•9/13/23, 2:11 PM

but you did give me a different idea

P

Pobiega•9/13/23, 2:12 PM

well, you can google it and I suggest you do, there are plenty of articles and videos on the topic. Its really interesting too, imho

A

AsherOP•9/13/23, 2:12 PM

I was thinking of possibly having a recursive function for every element and it combines it into the root

P

Pobiega•9/13/23, 2:12 PM

but the idea is that we can write a parser that parses a single character

P

Pobiega•9/13/23, 2:12 PM

and if we combine several of those, we can parse a word

P

Pobiega•9/13/23, 2:12 PM

etc

P

Pobiega•9/13/23, 2:12 PM

so you build progressively larger parsers, by combining smaller ones

P

Pobiega•9/13/23, 2:13 PM

it eventually leads to some very pretty code, where you declare a list as "zero or more valid bencode tokens"

P

Pobiega•9/13/23, 2:13 PM

and a "bencode token" is declared as "a bencoded list, dictionary, byte, string or int"

P

Pobiega•9/13/23, 2:13 PM

etc

A

AsherOP•9/13/23, 2:13 PM

that sounds like what I need

A

AsherOP•9/13/23, 2:14 PM

I'm gonna look at some articles

A

AsherOP•9/13/23, 2:14 PM

might try my idea aswell but this seems like it's perfectly suited for me

P

Pobiega•9/13/23, 2:14 PM

there is a library for C# called

Pidgin

Pidgin

Pidgin

Pidgin that helps you write parsers

P

Pobiega•9/13/23, 2:14 PM

but its also entirely doable to make something from scratch, if you want to

A

AsherOP•9/13/23, 2:15 PM

I'll see that aswell

P

Pobiega•9/13/23, 3:09 PM

Got it, using Pidgin

P

Pobiega•9/13/23, 3:13 PM

dictionary: d4:spam4:eggs2:hii16ee
spam: "eggs"
hi: 16

dictionary: d4:spam4:eggs2:hii16ee
spam: "eggs"
hi: 16

dictionary: d4:spam4:eggs2:hii16ee
spam: "eggs"
hi: 16

dictionary: d4:spam4:eggs2:hii16ee
spam: "eggs"
hi: 16

A

AsherOP•9/13/23, 3:14 PM

mind sharing your code? I won't copy as I want to do this myself, haven't had the time to look at Pidgin yet either

A

AsherOP•9/13/23, 3:14 PM

just as a general idea

P

Pobiega•9/13/23, 3:15 PM

public abstract record BToken
{
}

public record BString(string Value) : BToken
{
    public override string ToString() => $"\"{Value}\"";
}

public record BInteger(int Value) : BToken
{
    public override string ToString() => Value.ToString();
}

public record BList(ImmutableArray<BToken> Elements) : BToken;

public record BDictionary(ImmutableDictionary<string, BToken> Elements) : BToken;

public abstract record BToken
{
}

public record BString(string Value) : BToken
{
    public override string ToString() => $"\"{Value}\"";
}

public record BInteger(int Value) : BToken
{
    public override string ToString() => Value.ToString();
}

public record BList(ImmutableArray<BToken> Elements) : BToken;

public record BDictionary(ImmutableDictionary<string, BToken> Elements) : BToken;

public abstract record BToken
{
}

public record BString(string Value) : BToken
{
    public override string ToString() => $"\"{Value}\"";
}

public record BInteger(int Value) : BToken
{
    public override string ToString() => Value.ToString();
}

public record BList(ImmutableArray<BToken> Elements) : BToken;

public record BDictionary(ImmutableDictionary<string, BToken> Elements) : BToken;

public abstract record BToken
{
}

public record BString(string Value) : BToken
{
    public override string ToString() => $"\"{Value}\"";
}

public record BInteger(int Value) : BToken
{
    public override string ToString() => Value.ToString();
}

public record BList(ImmutableArray<BToken> Elements) : BToken;

public record BDictionary(ImmutableDictionary<string, BToken> Elements) : BToken;

❔ Format parsing problem with character collision

Similar Threads

Similar Threads

Similar Threads