Recreating Dunes for YCombuster
Design Choices
Abstract Syntax Trees
The reason I am considering Abstract Syntax Trees to chunk code is because I believe that given the general design / flow of a code LLMs are smart enough to convert that design into comprehensible code.
If not at this level they will be soon.
The whole point of a RAG pipeline is to provide LLMs context for them to be able to provide better responses and I think this is one approach that I am keen on exploring and will get better overtime as LLMs get better