Just installed Bonobo Git Server for Windows on Windows Server 2008 R2 x64 without any problem. It is pretty neat web-application that you can install on your IIS to manage your personal git repositories. You don't even need to install git on your server. Everything is included. Easy to install, easy to use - it is just what I need. 

Last several months in my spare time I was working on implementation of Rodrigo's vision about new simplified layer of frontend AST for Boo. I used existing unit tests for Boo Parser as base for unit tests for my new project which has working name TinyAST. I added my project to my branch of the project Boo-Extensions on GitHub.

Couple days ago I finished the first phase of the project. I would raither call it "Alpha version" than "Proof of Concept" because I was able to implement about 90% of Boo syntax using the new approach. During this phase I wanted to find an answers to the following questions:

1. Which AST classes should be added to the new API?
2. Which syntax features could be implemented using the new approach?
3. How simple the parser for the new AST could be?
4. How to integrate the new API with existing compiler pipeline?
5. How to do conversion from TinyAst to Boo AST?
6. How to deal with unsupported syntax?

Below are the answers in the corresponding order.

Which AST classes should be included to the new API?

Currently I came up with the following set of AST classes which is almost identical to Rodrigo's one

ast Form = \
    Identifier(Name as string, IsKeyword as bool, IsSymbol as bool) | \
    Brackets(Form, Type as BracketsType)  | \
    Literal(Value as object, astLiteral as LiteralExpression) | \
    Infix(Operator as Identifier, Left, Right) | \
    Prefix(Operator, Operand, IsPostfix as bool) | \
    Tuple(Forms as (Form)) | \
    Pair(Left, Right, IsMultiline as bool, Doc as string) | \
    Block(Forms as (Form))

Which syntax features could be implemented using the new approach?

This is the list of syntax features which are currently not supported:

1. Enumerable type shortcut
2. Generic placeholders
3. Goto and goto labels
4. Hash literals
5. Regular expressions
6. Slicing
7. Timespan literals

How simple the parser for new AST could be?

Currently the new parser includes about 50 rules excluding rules related to integration with compiler pipeline, tokens and literals. This is the list of the rules:

form_stmt, block, begin_block, begin_block_with_doc, form, infixr closure_separation, inline_block, prefix_expression, pair, single_line_pair, tuple, pair_expression, brakets_prefix, assignment, high_pr_pair, infix or_expression, infix and_expression, prefix_keyword not_expression, infix membership_expression, infix identity_test_expression, infix isa_expression, infix comparison, infix bitwise_or_expression, infix bitwise_xor_expression, infix bitwise_and_expression, infix term, infix factor, infix bitwise_shift_expression, prefix_symbol signalled_expression, prefix_symbol ones_complement_expression, infix exponentiation_expression, infixr as_operator, infix cast_operator, postfix_operator, infix member_reference, prefix_symbol splice, prefix at_operator, high_priority_prefix, hp_tuple, prefix_of_brackets, infixr of_operator, sticky_brackets, atom = exp_in_brackets | prefix_of_brackets | identifier | literal, identifier, exp_in_brackets, paren_brackets, qq_brackets, square_brackets, curly_brackets

The source code of the parser is in the file TinyAstParser.boo.

I think that the next big challenge is to make the parser simpler by creating more special macroses which will help to declare rules inside OMeta macros. I believe all rules could be formalized using notions token, keyword, atom, infix, prefix and mechanism overloading default operator priorities.

How to integrate the new API with existing compiler pipeline?

To integrate the new API the following has been done:

1. Standard parser step is replaced with the new one.
2. The new parser converts everything except namespaces and imports to special macros tinyAst and add it to module's globals. AST objects are stored in the macros annotation.
3. Reference to tinyAst assembly is added to the compiler references so MacroExpansion step can find macros tinyAst.
4. Macros tinyAst converts new front end AST to standard AST using TinyAstEvaluator.
5. New class Module is created with overriden function ToCodeString(). It is necessary for testing purposes. Also a new class TinyAstPrinterVisitor has been created to support printing of code provided by method ToCodeString().

How to do conversion from TinyAst to Boo AST?

I used Boo.OMeta to transform object graph of TinyAst to object graph of Boo AST. Several minor changes has been done to the Boo.OMeta to improve ObjectMatching.

After spending several months experimenting with object graph transformations and keeping in mind that scenario I working on is pretty complex I can assert that Boo.OMeta is very powerful tool for this purpose. It is actually subject for a separate article. The code doing transformations is in class TinyAstEvaluator. When I started creating transformation rules for a object graph (or tree) I always had to  deal with tree structures. It required to create hard to read rules which mimic structure of the tree they try to recognise. For example this is the rule for statement for:

stmt_for = Prefix(
			Operator: FOR,
			Operand: 
				Pair(
					Left:
						Infix(
							Operator: IN,
							Left: declaration_list >> dl,
							Right: rvalue >> r												
						),
					Right:
						block >> body
				)
		) \
		, or_block >> orBlock \
		, then_block >> thenBlock ^ newForStatement(dl, r, body, orBlock, thenBlock)

Finally I came up with a set of auxiliary rules which now allows to create production rules in a regular maner. For example this is how class definition is implemented using auxiliary rules here, next, prefix, optional_prefix_operand, prefix_or_rule, optional, nothing.

class_def = --attributes_line >> att, here >> i, inline_attributes >> in_att, member_modifiers >> mod, prefix[CLASS], class_body >> body \
					, optional_prefix_operand[super_types] >> superTypes, prefix_or_rule[id] >> className, optional[generic_parameters] >> gp, nothing \
					, next[i] ^ newClass([att, in_att], mod, className, gp, superTypes, body)

Rules here, next and nothing can be used in any object pattern matching scenario. Other rules were created specially for TinyAst object tree but this approach can be used for other scenarios.

Dealing with unsupported syntax

This subject is not currently completely resolved but there are already some rules which can be used to deal with the problem. In case the statement cannot be parsed by TinyAstParser, it could call original Boo.OMeta.Parser using function callExternalParser which is currently implemented in TinyAstEvaluator. This approach was used to deal with string interpolation during evalutaion of TinyAst tree:

booparser_string_interpolation = $(callExternalParser("string_interpolation", "Boo.OMeta.Parser.BooParser", input)) \
							| $(callExternalParser("string_literal", "Boo.OMeta.Parser.BooParser", input))

I think that original parser could be always called in parallel with TinyAst parser and results could be compared in some way to figure out if there was a problem with parsing. Also result of original parser execution could be stored in annotations of Form to be used in case evaluator can't recognise the statement. And of course as an option the problematic syntax also could be removed from the language to improve its metaprogramming features.

To do

1. Create examples of use cases for new AST. Now when minimal functionality of the new API is implemented it is possible to test it in real world scenarios.
2. Simplify TinyAst parser or even create a framework allowing developers easily create parsers for the new API.
3. 1,000,000 other things

Conclusions

I hope this project showed some perspectives of Rodrigo's Idea. I think that it could significantly improve metaprogramming features of Boo. Currently code is not polished and sometimes it looks like a battlefield but I hope it is possible to get the idea. It is only a first step and it would be great to be sure that it is done in right directions so opinion of Boo comrades would be very helpfull.

Special thanks to Rodrigo who created such a powerfull framework as Boo.OMeta using Alessandro Warth's OMeta specification.

New SharpDevelop 4.2 is awesome! Even Beta version was very stable for me.

Creation of macro in Boo is a complicated thing. One of the things making it complicated is the fact that it is hard to debug. It is not possible to set a breakpoint in your macro because macro is executed in the compile time. Therefore to debug a macro we need to call Boo's compilation services in a runtime. In order to do it we need to create a test case compiling a source code which uses macro we want to debug.

First thing we should do in our test case is to create instance of BooCompiler.

compiler = Boo.Lang.Compiler.BooCompiler()

Then we should initialize compiler's output writer and add references. Macro we want to debug has to be added as one of the references. Let's assume that our macro name is "my" and we are using macro "macro" to implement it. Keeping all that in mind we will add the following to our code:

compiler.Parameters.OutputWriter = StringWriter()
compiler.Parameters.References.Add(typeof(MacroMacro).Assembly)
compiler.Parameters.References.Add(typeof(MyMacro).Assembly)
...

There can be several choices to initialize the compilers pipeline. You can use Boo.Lang.Compiler.Pipelines.Compile or Boo.Lang.Compiler.Pipelines.ExpandMacros. The former works longer and it will produce all compilation errors and warnings. The later works faster and produce error messages related to Parsing or Macro Expansion only. Let's use the first pipeline to be able to see if our macro is compiled successfuly.

compiler.Parameters.Pipeline = Boo.Lang.Compiler.Pipelines.Compile()

Before we can go ahead and run compilation we should specify input for the compiler. It can be either Boo.Lang.Compiler.IO.StringInput or Boo.Lang.Compiler.IO.FileInput. Choose first one if you want to keep a sorce code in the string variable. Use triple quote string to store your code in the string variable and do not forget to escape ("\") splice symbol $. The second option is more convinient if your source code is big. We can add the source file to the project and set up its Build Action as Content and Copy to output directory as Always. After we done with it we shouldn't care about path to the file because file will be copied to the output directory. Let's use second option to initialize compiler input.

compiler.Parameters.Input.Add(FileInput("ClassUsingMyMacro.boo"))

Now we can execute compilation. When this line of code is executed we can debug expansion of macro using breakpoints and tracing.

context = compiler.Run()

Beside ability to trace macro we can use context variable which contains results of compilation and AST.

Printing compilation errors:

print context.Errors

Printing AST:

module = context.CompileUnit.Modules[0]
print module.ToCodeString()

This is what we will have after combining all lines of codes together.

[Test]
def GenGrammarCode():
 compiler = Boo.Lang.Compiler.BooCompiler()
 
 compiler.Parameters.OutputWriter = StringWriter()

 compiler.Parameters.References.Add(typeof(MacroMacro).Assembly)
 compiler.Parameters.References.Add(typeof(MyMacroExpansion).Assembly)

 compiler.Parameters.Pipeline = Boo.Lang.Compiler.Pipelines.Compile()

 compiler.Parameters.Input.Add(FileInput("ClassUsingMyMacro.boo"))

 context = compiler.Run()

 print context.Errors

 module = context.CompileUnit.Modules[0]
 print module.ToCodeString()

Happy debugging!

I added some facilities to track SourceLocation in Boo.OMeta.Parser.
I assume that there can be three ways to track it:
1) We can use special rules like (here and prev) to track start/end of expression
2) We can use variable "input" in production method call. For example:
        here = "" ^ makeToken("here", null, input, input)
So input means "current input".
3) We save start of every token and then we use this value in our rules.
My changes are here.

I just finished implementing fix for BOO-854 (Anonymous callable types fail when signature contains generic parameters).

It was really hard task because it required understanding of almost every part of boo compiler pipeline. So now satisfaction is directly proportional to applied efforts. I'm enjoying minutes of fame in the company of my PC and myself.

Author of the articale thinks that these are essential behavioral skills of real innovator:

• Questioning
• Observing
• Networking
• Experimenting

More details are in the article.

I was experimenting with Boo grammar in Boo.OMeta.Parser. I tried to create different flavor of Boo using overload of grammar rules. I wanted to make member referencing to be White Space Agnostic so I could reference methods and properties the same way as we can do it in C#.
For example this is correct syntax for my flavor of Boo:

c = A()
       .B().
                  C().
       D()
.
   E().
            F()

I changed existing rule for member-reference from:
    member_reference = ((member_reference >> e, DOT, ID >> name ^ newMemberReference(e, name)) \
        | slicing) >> e, (INCREMENT | DECREMENT | "") >> postOp ^ addSuffixUnaryOperator(e, postOp)
to
       member_reference = ((member_reference >> e, enterWhitespaceAgnosticRegion, DOT, ID >> name, leaveWhitespaceAgnosticRegion ^ newMemberReference(e, name)) \
        | slicing) >> e, (INCREMENT | DECREMENT | "") >> postOp ^ addSuffixUnaryOperator(e, postOp)   

But it actually didn't work out. After some debugging I found out that algorithm tracking indents is not always works correctly for some scenarios because sometimes changes to this stack should be roll-backed.
I improved the algorithm to store indent stack in the OMetaInputWithMemo so that stack value will automatically rollback if rule generating INDENT/DEDENT finally fails.
My new syntax started to work after implementing the new algorithm. So I think this fix could improve ability to use WSA mode in Boo.OMeta.Parser.

Is it a good way to store the stack in the OMetaInputWithMemo or maybe it is better to create property in OMetaInput?

I also created test for simple grammar which works with the new algorithm and doesn't work with the old one.

Test and fix are in the repository of boo extensions.

The surprising truth about what motivates us.

HTML 5 & Silverlight 5