Building an Interpreter: Journey so far...

If it isn't tested, it is broken

Our tree walker interpreter has come quite far at this point. We have implemented

1. A basic lexer for generating tokens from our sox source code,

2. A parser for building an AST from a stream of tokens,

3. A resolver for resolving location of local variables,

4. An Environment for holding name object bindings,

5. An interpreter evaluator for walking the AST and evaluating the stamements,

6. A very basic REPL for testing single line statements.

We have covered alot of ground here but have moved forward with scant regard for testing till now. As the saying goes, “code that is not tested, is broken”. To assist with testing, I have taken some of the tests from the Crafting Interpreters tests suite and adapted to our language Sox. The inclusion of the tests has been a rather humbling experience. The tests discovered a slew of issues and bugs with our implementation and the code base is better for it as whole. The integration of the tests led to the following -

1. Improvement to our REPL - there was a bug in the simple REPL which prevented the correct display of evaluated values. This was fixed by changing the logic for executing the AST so that it prints the last None value result from executing the AST.

2. Rework of our Environment data structure - Our implementation of closures had a bug due to a misunderstanding of the the Crafting Interpreters text. When capturing the environment, we made a copy of the environment for the closure rather than using a reference. This meant that changes to the environment outside the closure were invisble to the closure when they should not have been. Resolving this bug led to a rearchitecture of the data structure for represting an environment.

3. Improvement in displaying values - We have also improved how we display values. Rather than just print the default string representation of a rust value, we define a trait Representable that all types must implement. This trait has a single method repr which each implementing type defines to explain how it should be printed.

4. Improvement in how we compare values - When doing equality we currently did inline comparison and only supported numeric values. We expanded equality comparison to other types and rather than defining equality inline in the interpreter visit_binary_expression method, we move equality into the type implementation where they are implemented as methods that go into the type slots so equality logic is now a case of checking for presence of a method in the equality slot for a given type and then using whatever method is in that slot for comparison if such exists.

5. Improvement to our tokens with inclusion of an id field - We have added a monotic id field to our Token type so that each token is unique regardless of if the lexeme and line numbers referenced is the same as another token.

With the suite of tests in place, we can now confidently go ahead with the next steps to upgrade our tree walker interpreter to a virtual machine based interpreter.