C
C#

help

A generic and efficient way to feed in text for a lexer or scanner

LLPeter199710/8/2022
In the past I was lazy and always just shoved a string into my lexical analyzer as input, but what's the "ideal" way to read source text? It should be relatively efficient as the lexer observes each character. It should also support peeking forward without consumption. The thing is, source code can come from many different places, like a file, a console REPL as lines, edit diffs from a language client, ... I'd like some way to read tokens from essentially any source without paying for virtual calls for each character read or something. Maybe I'd need a view-type like a ReadOnlyMemory or even ReadOnlySpan? Thing is, we have so many options. string, IEnumerable<char>, ReadOnlyMemory<char>, ReadOnlySpan<char>, TextReader, IO pipelines, ...
CChiyoko_S10/8/2022
a stream with an internal buffer?
PPheubel10/8/2022
something i am doing is making use of a TextReader, it reads a stream of data going forwards only. there is a small problem with it tho, sometimes when you call Peek() on it, it will assume that there is nothing to look for anymore, even though if you were to call Read() it would return the next character. What i did was create a simple wrapper around it with a small internal buffer it can use to peek into.
LLPeter199710/8/2022
A stream + an internal buffer doesn't sound bad but in practice it's a pain to work with streams IIRC
PPheubel10/8/2022
so far it was pretty painless, you just need to know what you are doing.
LLPeter199710/8/2022
TextReader is a whole different topic, I'm talking about Stream here Readers are a way friendlier way to tackle this problem
PPheubel10/8/2022
i did make some extra functions to make it a little bit easier to use, for example peek for a specific character and move the reader forwards if it matches
LLPeter199710/8/2022
Sure, you just preallocate a buffer and you got all those functions I'm still not sure if it's the ideal thing to use but I might not have anything better
PPheubel10/8/2022
is there any reason you could not use a reader?
LLPeter199710/8/2022
Oh absolutely, I'm planning to have non-sequential reads and I'd be happy if I didn't have to reimplement my lexer for that Thing is, I'd want to extend my lexer to be optionally incremental, and then I can't afford re-reading the entire source A Stream can at least seek in this case
PPheubel10/8/2022
what i would do is instead of going over the source file again, go over it once and tokenize it and structure it in a tree immediately. then if you want to do multiple passes you can go to the tree instead and transform it
LLPeter199710/8/2022
That has no relation to the problem, yes streaming lexers are a thing For now I've decided to roll with the following interface:
public interface ISourceReader
{
public bool IsEnd { get; }
// Settable to be able to seek on incremental lexing
public int Position { get; set; }
public char Peek(int offset = 0, char @default = '\0');
public void Advanve(int length = 1);
}
public interface ISourceReader
{
public bool IsEnd { get; }
// Settable to be able to seek on incremental lexing
public int Position { get; set; }
public char Peek(int offset = 0, char @default = '\0');
public void Advanve(int length = 1);
}
And I'll make my lexer take a generic parameter, and wrap my source readers as structs so I can guarantee the compiler will inline each type, no vcalls Setting the position will just throw on sources that can't support it, I think that's reasonable, as some configurations just can't allow for that My lexer will be a very thin type with a source reader and a single Next() method anyway, then I'll wrap that for different configurations, like incremental/nonincremental and streaming/non-streaming

Looking for more? Join the community!

Want results from more Discord servers?
Add your server
Recommended Posts
Blazor Re-render [SOLVED]Hello, I have a property stored in static class, and I would like to re-render some components that Restarting audio with MediaPlayer?I've made a button that uses MediaPlayer to play a custom MP3 file that I've made. ```cs MediaPlayerWhat's the meaning of CS0659?```'class' overrides Object.Equals(object o) but does not override Object.GetHashCode()``` I got thiHow does Newtonsoft.Json.JsonConverter handle nullable structs?I'm presently writing a JsonConverter for a third-party struct, and in my json models that field canMost convenient method to handle 3D vectors?I'm looking for a struct/class in which magnitude(L2 norm), normalize, multiplication with a scalar,can anybody recommend where I can learn about file format encoding internals e.g pdf, doc, epubscan anybody recommend where I can learn about file format encoding internals e.g pdf, doc, epubsFiles on desktop [Repost]I want to check files that have been added or edited (I thought of using fileSystemWatcher but everyIndex was out of range. How to check if list has items without triggering this error?Hello, I have a list, with a list inside. ```cs List<List<dynamic>> characterData ``` This is becaWebsocket buffer size performance and latencyI'm trying to figure out what buffer size to have, does having a lower buffer size increase or decreFiles on desktopI want to check files that have been added or edited (I thought of using fileSystemWatcher but everySQL and Visual Studio Retrieving a TableHi - I'm trying to select all the "emails" in a table called "users" with c# in visual studio. CurreBest way to do a discriminated union structI have a struct + enum that looks essentially like this. ```cs public readonly struct Value { puWhat's the purpose of MarshalByRefObject?A couple of classes in the standard library inherit from `MarshalByRefObject`, but what is the purpoHow to deserialize dictionary using Newtonsoft [Answered]I have a Json file with a bunch of pokemon data. I am having trouble deserializing the information. The maximum length of the indexI have such a model: ```cs [Index(nameof(Name), IsUnique = true)] public class Variables { publiEF Core migrations script failing but not enough details!Hey devs, I am trying to run a migrations script to generate a sql file for the migration but I am rfor loop returning nothing```cs using System; using static System.Console; namespace ColouredTriangles { class Program Best API for low latencyI have a small client application that does simple calculations. Now i want to make it report the re[ASP.NET Core 6] Performance when registering a lot of singleton as servicesHi, Does having a lot of classes registered as singleton (_builder.Services.AddSingleton(...)) reducFile Packages ?So I was doing some research because I wanted to make a file extension like .docx which is basically