tommy.blog

Making your password TextBox more secure

2008-04-21T15:24:00.001+02:00

If you use a Windows Forms TextBox to let your users enter a password, you should know it's not very secure: external applications can get the password from the TextBox by sending it the WM_GETTEXT message. There are even applications written specifically to do this. If you want to prevent this, you can use the following control that is derived from System.Windows.Forms.TextBox and that prevents external applications from getting the password via WM_GETTEXT. Just use it instead of the regular TextBox for password fields.

using System;
using System.ComponentModel;
using System.Drawing;
using System.Windows.Forms;

namespace TC.WinForms
{
    /// <summary>Represents a text box control for entering passwords.</summary>
    [ToolboxItem(true), ToolboxBitmap(typeof(TextBox))]
    public class PasswordTextBox : TextBox
    {
        /// <summary>Initializes a new instance of the <see cref="T:PasswordTextBox" /> class.</summary>
        public PasswordTextBox()
        {
            base.UseSystemPasswordChar = true;
        }

        bool fAccessText;
        /// <summary>Gets or sets the current text in the <see cref="T:TextBox"/>.</summary>
        /// <returns>The text displayed in the control.</returns>
        public override string Text
        {
            get
            {
                fAccessText = true;
                try { return base.Text; }
                finally { fAccessText = false; }
            }
            set
            {
                fAccessText = true;
                try { base.Text = value; }
                finally { fAccessText = false; }
            }
        }

        /// <summary>Gets the length of text in the control.</summary>
        /// <returns>The number of characters contained in the text of the control.</returns>
        public override int TextLength
        {
            get
            {
                fAccessText = true;
                try { return base.TextLength; }
                finally { fAccessText = false; }
            }
        }

        /// <summary>Processes Windows message.</summary>
        /// <param name="m">The Windows <see cref="T:Message" /> to process.</param>
        protected override void WndProc(ref Message m)
        {
            switch (m.Msg)
            {
                case WM_GETTEXT:
                case WM_GETTEXTLENGTH:
                    if (!fAccessText)
                    {
                        m.Result = IntPtr.Zero;
                        return;
                    }
                    else break;
                case EM_SETPASSWORDCHAR: return;
            }

            base.WndProc(ref m);
        }

        const int WM_GETTEXT = 0x000D, WM_GETTEXTLENGTH = 0x000E, EM_SETPASSWORDCHAR = 0x00CC;
    }
}

EDIT: I added code to also ignore EM_SETPASSWORDCHAR which can also be used to make the password visible.
EDIT 2 (2008-04-22): I fixed the documentation tags in the code.
EDIT 3 (2008-04-22): I fixed some bugs that denied access to the password from within your program.

Technorati Tags: Windows Forms, WinForms, password, TextBox, security, C#.

Silverlight Rehab

2008-04-01T08:01:00.001+02:00

I previously linked to a video made by some Microsoft employees. Now they created another funny one, this time about Silverlight addicts.

Technorati Tags: Microsoft, Channel 10, comedy, Silverlight.

Opera Widget tip: zooming in

2008-03-29T18:57:00.001+01:00

I just discovered something cool: you can make your Opera Widgets larger by zooming in. I just opened my favorite widget (touchTheSky: a weather forecast widget), and the text was a bit small. Usually, when I'm using Opera and I'm reading a web site with a small font, I press Ctrl and use the mouse wheel to zoom in. I just noticed that I unconsciously did the same with the touchTheSky widget and to my surprise it worked. Cool feature.

Technorati Tags: Opera, Widgets, Opera Widgets, touchTheSky.

Life at Microsoft

2008-02-13T07:53:00.001+01:00

The guys over at Channel 10 made an awesome video to show what life at Microsoft is really like. Very funny. Go watch it.

Technorati Tags: Microsoft, Channel 10, comedy.

Video: 12 years of Opera

2008-02-07T07:35:00.001+01:00

Daniel Goldman has posted a cool little video of Opera's 12-year history. In just 2 minutes it shows the long road Opera has walked, the many versions of its desktop browser, mobile browser, mini browser and browser for devices.

Technorati Tags: Opera, Web Browser, Opera Mobile, Opera Mini, video, history.

PowerShell + Speech API

2008-02-01T13:10:00.000+01:00

I just read a blog post about using the Speech API in command-line scripts on Marcelo's Weblog. So I had to try it but instead of using JScript, I wrote it in PowerShell. It's so easy and fun:

$v = New-Object -ComObject SAPI.SpVoice
$v.Speak("Who let the dogs out?")

Technorati Tags: PowerShell, Speech, Speech API.

Garden Gnome Carnage

2008-01-27T12:46:00.001+01:00

I just watched a video about a game called Garden Gnome Carnage. I haven't downloaded it yet, but it looks crazy: the video that explains how to play GGC is pretty weird. Don't worry if you experience some WTF-moments while watching it.

Technorati Tags: Garden Gnome Carnage, GGC, game, video, indy games.

The Making of Peggle

2007-11-11T11:46:00.001+01:00

PopCap released an article, complete with pictures and videos, about how they created their awesome game Peggle (download or play online). This cool article clearly explains the entire development process of a great game. Go read The Making of Peggle.

Technorati Tags: PopCap, Peggle, game development, casual gaming.

Opera Mini 4 released

2007-11-08T07:45:00.001+01:00

Opera has released Opera Mini 4. New features include Opera Link (a way of synchronizing the bookmarks on your mobile phone with the bookmarks on your desktop PC), a virtual mouse cursor, enhanced Small Screen Rendering and much more.

Technorati Tags: Opera, Opera Mini, Web Browser, Java, Mobile Phone.

Cool new Opera skin

2007-11-01T12:09:00.001+01:00

I just downloaded this cool new Opera skin called Vista Black and Black. It combines elements from Vista, IE7 and Office 2007. There are a lot of skins that emulate the Vista-look or IE7 or Office 2007, but none of them really looked fine for me. But I think this one looks awesome.

Technorati Tags: Opera, skin, Vista, IE7, Office 2007.

TC Animation Library

2007-11-01T11:00:00.001+01:00

I just released the source code of a Windows Forms animation library I wrote last weekend. I first didn't want to release the source code because the animations looked terrible.

I knew the code I wrote was good and should work, but it ran horribly. The UI would just freeze when 1 animation ran; running multiple animations simultaneously was impossible. This morning I woke up and had a mad idea: to slow down the animation. The reason the UI would freeze was because all the animation steps were performed on the UI-thread and apparently the animation system flooded the UI-thread with work to do, so the normal UI-activities (painting controls, handling input, ...) were severely delayed. By adding 1 line of code (Thread.Sleep(5);) to the animation loop, I managed to make the animations run fluently while keeping the UI responsive.

You can find the source code in the Channel 9 Sandbox, here.

Technorati Tags: Windows Forms, WinForms, animation, source code, C#, open source.

.NET Framework source code released

2007-10-03T22:34:00.001+02:00

I wasn't going to blog about this, but I just find it remarkable how many people did. The last few hours I received an unusual amount of newsfeed messages about it:

Technorati Tags: .NET Framework, source code.

Opera 9.5: preview of new features

2007-09-03T18:55:00.001+02:00

CyberNet has just published an article with a video that shows some of the new features of Opera 9.5. Apparently they made the rendering engine even faster than it already is, while consuming even less memory and at the same time adding a lot of CSS, JavaScript, AJAX and compatibility features. Wow.

Functional new features include an easy way to open the current page in a different browser, restoring closed windows, synchronizing bookmarks with My Opera, full history search, major improvements to the mail client, etc...

Technorati Tags: Opera, Web Browser.

Transceiver

2007-08-18T11:27:00.001+02:00

For the past 3 years, our company has been developing a new technology for secure file delivery. This week we officially released the technology called Transceiver. It works a bit like e-mail, but for transmitting files instead of mail messages. Here are some of the features:

Designed to be secure: end-to-end security is the primary goal.
Designed to be private: network sniffing will not reveal who is sending files to who.
Uses a white-list system to avoid the problem of spam: you will only receive files from people you add to your list of Trusted Addresses.
No file size limitation (of course it's limited to the available disk space).
If you send a file to someone, you can track the entire transmission: you can see exactly where the file is, and when it moves from machine to machine.
We're developing different client applications for different scenarios. All these client applications can be downloaded for free:
- Transceiver Communicator: an interactive Windows application that works like a regular e-mail client. On top of Communicator, we've also developed add-ins for Word 2003, Word 2007, Excel 2003 and Excel 2007 that talk to Communicator. We'll also release a .NET 2.0 library for other developers to integrate with Communicator.
- Transceiver Automator: an application that runs as a Windows service and that can transmit and receive files automatically. It can be configured to handle incoming files or transmitting files to predefined destinations.
- Transceiver Client Library: a .NET 2.0 library that acts as a Transceiver client and that can be used to communicate with a Transceiver Server to transmit and receive files.

Like I said above, the clients are for free. What we will be selling, is the server software. We encourage people to try out the Transceiver technology for free: we've deployed a test server and a website where you can create a temporary test account (just select Step 1 on the left-hand side). The only client application that can already be downloaded is Transceiver Communicator (together with the Office add-ins). The other client applications will follow by the end of the year.

Technorati Tags: Transceiver, Transceiver Communicator, Transceiver Automator, Transceiver Client Library, security, file delivery, .NET.

Calctor 2.1.1

2007-07-01T10:38:00.001+02:00

This is a small update to Calctor. No additional functionality has been added. I just made some modifications so you can install it on Windows Vista.

You can download it here.

Technorati Tags: Calctor, Calculator, Freeware, Windows Vista.

New version of Windows Live Writer

2007-06-03T12:50:00.001+02:00

I just installed the new version of Windows Live Writer. It's still beta, but it's a lot better than the previous beta version. It seemed for a while that development of WLW had stopped (no updates or news on the WLW blog since November from last year), but I'm glad they're still working on it. This has to be one of the coolest apps Microsoft is currently developing.

Technorati tags: Windows Live Writer, blogging.

Buzzword screencast

2007-05-18T16:18:00.001+02:00

I just watched a screencast of Buzzword (recorded by Dan Bricklin), Virtual Ubiquity's not-yet-released word processor for the web that looks very promising. It's a Flash/Flex application that looks so much better than any web word processor I've seen before. I'll be keeping an eye on this.

Technorati tags: Buzzword, Virtual Ubiquity, word processor, Flash, Flex.

slimCODE against diabetes

2007-05-06T12:42:00.001+02:00

I just read on Martin Plante's blog that he's donating this week's earnings of his slimKEYS application to Team Hanselman to fight diabetes. slimKEYS is a cool product, and this is a great initiative.

Technorati tags: slimCODE, slimKEYS, diabetes, Team Hanselman.

Writing a parser: overview

2007-05-03T22:11:00.001+02:00

Here's a quick overview of the different blog posts in the series Writing a parser:

Technorati tags: Parser, ADL, Acronymic Demonstrational Language, Programming Language, Visual Studio, C#.

Writing a parser: ADL Parser - part 2

2007-05-03T22:03:00.001+02:00

In the previous part we wrote the first Parser methods, now we'll write the methods for parsing expressions. In the introduction to ADL, I showed the order in which expressions are evaluated. To parse expressions, we'll reverse the order and assume that everything is an AND-expression (lowest precedence):

Expression ParseExpression()
{
    return ParseAndExpression();
}

An AND-expression consists of one or more OR-expressions, separated by and operators:

Expression ParseAndExpression()
{
    Expression lNode = ParseOrExpression();

    while (!AtEndOfSource && fCurrentToken.Equals(TokenType.Word, "and"))
    {
        ReadNextToken(); // skip 'and'
        lNode = new AndExpression(lNode, ParseOrExpression());
    }

    return lNode;
}

An OR-expression consists of one or more comparisons, separated by or operators:

Expression ParseOrExpression()
{
    Expression lNode = ParseComparison();

    while (!AtEndOfSource && fCurrentToken.Equals(TokenType.Word, "or"))
    {
        ReadNextToken(); // skip 'or'
        lNode = new OrExpression(lNode, ParseComparison());
    }

    return lNode;
}

A comparison is an additive expression, or 2 additive expressions separated by a comparison operator:

Expression ParseComparison()
{
    Expression lNode = ParseAdditiveExpression();

    if (!AtEndOfSource && fCurrentToken.Type == TokenType.Symbol)
    {
        ComparisonOperator lOperator;
        switch (fCurrentToken.Value)
        {
            case "==": lOperator = ComparisonOperator.Equal; break;
            case "<>": lOperator = ComparisonOperator.NotEquals; break;
            case "<":  lOperator = ComparisonOperator.LessThan; break;
            case ">":  lOperator = ComparisonOperator.GreaterThan; break;
            case "<=": lOperator = ComparisonOperator.LessThanOrEqual; break;
            case ">=": lOperator = ComparisonOperator.GreaterThanOrEqual; break;
            default: return lNode;
        }
        ReadNextToken(); // skip comparison operator
        return new Comparison(lOperator, lNode, ParseAdditiveExpression());
    }
    else return lNode;
}

An additive expression consists of one or more multiplicative expressions, separated by + or - operators:

Expression ParseAdditiveExpression()
{
    Expression lNode = ParseMultiplicativeExpression();

    while (!AtEndOfSource)
    {
        if (fCurrentToken.Equals(TokenType.Symbol, "+"))
        {
            ReadNextToken(); // skip '+'
            lNode = new Addition(lNode, ParseMultiplicativeExpression());
        }
        else if (fCurrentToken.Equals(TokenType.Symbol, "-"))
        {
            ReadNextToken(); // skip '-'
            lNode = new Subtraction(lNode, ParseMultiplicativeExpression());
        }
        else break;
    }

    return lNode;
}

A multiplicative expression consists of one or more unary expressions, separated by * or / operators:

Expression ParseMultiplicativeExpression()
{
    Expression lNode = ParseUnaryExpression();

    while (!AtEndOfSource)
    {
        if (fCurrentToken.Equals(TokenType.Symbol, "*"))
        {
            ReadNextToken(); // skip '*'
            lNode = new Multiplication(lNode, ParseUnaryExpression());
        }
        else if (fCurrentToken.Equals(TokenType.Symbol, "/"))
        {
            ReadNextToken(); // skip '/'
            lNode = new Division(lNode, ParseUnaryExpression());
        }
        else break;
    }

    return lNode;
}

A unary expression is a base expression, optionally prefixed by a -, + or not operator:

Expression ParseUnaryExpression()
{
    CheckForUnexpectedEndOfSource();

    if (fCurrentToken.Equals(TokenType.Symbol, "-"))
    {
        ReadNextToken(); // skip '-'
        return new Negation(ParseBaseExpression());
    }
    else if (fCurrentToken.Equals(TokenType.Word, "not"))
    {
        ReadNextToken(); // skip 'not'
        return new NotExpression(ParseBaseExpression());
    }
    else if (fCurrentToken.Equals(TokenType.Symbol, "+"))
        ReadNextToken(); // skip '+'

    return ParseBaseExpression();
}

A base expression is either an integer constant, a string constant, a variable, a function call or a group expression:

Expression ParseBaseExpression()
{
    CheckForUnexpectedEndOfSource();

    switch(fCurrentToken.Type)
    {
        case TokenType.Integer: return ParseIntegerConstant();
        case TokenType.String: return ParseStringConstant();
        case TokenType.Word: return ParseVariableOrFunctionCall();
        default: // TokenType.Symbol
            if (fCurrentToken.Value == "(")
                return ParseGroupExpression();
            else throw new ParserException("Expected an expression.");
    }
}

A group expression is an expression between parenthesis:

Expression ParseGroupExpression()
{
    ReadNextToken(); // skip '('
    Expression lExpression = ParseExpression();
    SkipExpected(TokenType.Symbol, ")"); // skip ')'
    return lExpression;
}

A variable and a function call both start with an identifier. To disambiguate, we'll have to read the next token:

Expression ParseVariableOrFunctionCall()
{
    string lName = fCurrentToken.Value;
    ReadNextToken();
    if (!AtEndOfSource && fCurrentToken.Equals(TokenType.Symbol, "("))
        return ParseFunctionCall(lName);
    else return new Variable(lName);
}

A string constant is just the read string token:

Expression ParseStringConstant()
{
    string lValue = fCurrentToken.Value;
    ReadNextToken(); // skip string constant
    return new StringConstant(lValue);
}

An integer constant has to be parsed from the string value:

Expression ParseIntegerConstant()
{
    int lValue;
    if (int.TryParse(fCurrentToken.Value, out lValue))
    {
        ReadNextToken(); // skip integer constant
        return new IntegerConstant(lValue);
    }
    else throw new ParserException("Invalid integer constant " + fCurrentToken.Value);
}

A function call consists of an identifier, followed by zero or more arguments between parentheses:

FunctionCall ParseFunctionCall(string name)
{
    ReadNextToken(); // skip '('
    CheckForUnexpectedEndOfSource();

    List lArguments = new List();
    if (!fCurrentToken.Equals(TokenType.Symbol, ")")
    {
        lArguments.Add(ParseExpression());
        CheckForUnexpectedEndOfSource();

        while (fCurrentToken.Equals(TokenType.Symbol, ","))
        {
            ReadNextToken(); // skip ','
            lArguments.Add(ParseExpression());
            CheckForUnexpectedEndOfSource();
        }

        if (!fCurrentToken.Equals(TokenType.Symbol, ")")
            throw new ParserException("Expected ')'.");
    }

    ReadNextToken(); // skip ')'
    return new FunctionCall(name, lArguments.ToArray());
}

Now our parser is ready. To test the parser, I've created a test application that reads code from a TextBox, parses it and shows the parse tree in a TreeView. I'm not going to show the source code of the test application in this post, but you can download the complete Visual Studio 2005 solution with both projects (TC.Adl and TC.Adl.Test) here.

Technorati tags: Parser, ADL, Acronymic Demonstrational Language, Programming Language, Visual Studio, C#.

Writing a parser: ADL Parser - part 1

2007-05-03T21:00:00.001+02:00

We'll now write the Parser class:

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using TC.Adl.ParserNodes;

namespace TC.Adl
{
    public class Parser
    {
        Tokenizer fTokenizer;
        Token fCurrentToken;
    }
}

Our Parser class has only 2 fields:

fTokenizer: The Tokenizer to read tokens from.
fCurrentToken: The current token (most recently read).

The constructor of the Parser class will accept a TextReader argument, create a Tokenizer that uses that TextReader, store it in fTokenizer and read the first token:

public Parser(TextReader source)
{
    if (source == null) throw new ArgumentNullException("source");

    fTokenizer = new Tokenizer(source);
    ReadNextToken();
}

Now we'll add some private helper methods.

Reading a token is simple. just call Tokenizer.ReadNextToken(), which returns a Token or null at the end of the source code.

void ReadNextToken() { fCurrentToken = fTokenizer.ReadNextToken(); }

To determine if we're at the end of the source, we just have to check the current token for null:

bool AtEndOfSource { get { return fCurrentToken == null; } }

We'll need a method that throws an exception when the end of the source has been reached unexpectedly:

void CheckForUnexpectedEndOfSource()
{
    if (AtEndOfSource)
        throw new ParserException("Unexpected end of source.");
}

We'll also need a method that verifies the current token and skips it:

void SkipExpected(TokenType type, string value)
{
    CheckForUnexpectedEndOfSource();
    if (!fCurrentToken.Equals(type, value))
        throw new ParserException("Expected '" + value + "'.");
    ReadNextToken();
}

Now that we've written the private helper methods, we can write the only public method: the ReadNextStatement method. This methods reads a statement and returns it. If we've reached the end of the source, we'll return null, else we'll check the first token to determine the type of statement:

If the current token is the word if, it's an if-statement.
If the current token is the word while, it's a while-statement.
If the current token is the word for, it's a for-statement.
If it's any other word, we assume it's an assignment or a function call.

public Statement ReadNextStatement()
{
    if (AtEndOfSource)
        return null;

    // all the statements start with a word
    if (fCurrentToken.Type != TokenType.Word)
        throw new ParserException("Expected a statement.");

    if (fCurrentToken.Value == "if")
        return ParseIfStatement();

    if (fCurrentToken.Value == "while")
        return ParseWhileStatement();

    if (fCurrentToken.Value == "for")
        return ParseForStatement();

    return ParseAssignmentOrFunctionCallStatement();
}

An if-statement starts with the word if, followed by a condition, the word then, a block of statements, an optional block of statements prefixed with the word else and the words end if:

IfStatement ParseIfStatement()
{
    ReadNextToken(); // skip 'if'

    Expression lCondition = ParseExpression();

    SkipExpected(TokenType.Word, "then"); // skip 'then'

    List lTrueStatements = new List();
    List lFalseStatements = new List();
    List lStatements = lTrueStatements;
    Statement lStatement;

    CheckForUnexpectedEndOfSource();
    while (!fCurrentToken.Equals(TokenType.Word, "end"))
    {
        if (fCurrentToken.Equals(TokenType.Word, "else"))
        {
            ReadNextToken(); // skip 'else'
            CheckForUnexpectedEndOfSource();
            lStatements = lFalseStatements;
        }

        if ((lStatement = ReadNextStatement()) != null)
            lStatements.Add(lStatement);
        else throw new ParserException("Unexpected end of source.");
    }

    ReadNextToken(); // skip 'end'
    SkipExpected(TokenType.Word, "if"); // skip 'if'

    return new IfStatement(lCondition
        , new StatementCollection(lTrueStatements)
        , new StatementCollection(lFalseStatements));
}

A while-statement starts with the word while, followed by a condition, the word do, a block of statements and the words end while:

WhileStatement ParseWhileStatement()
{
    ReadNextToken(); // skip 'while'

    Expression lCondition = ParseExpression();

    SkipExpected(TokenType.Word, "do"); // skip 'do'

    List lStatements = new List();
    Statement lStatement;
    CheckForUnexpectedEndOfSource();
    while (!fCurrentToken.Equals(TokenType.Word, "end"))
    {
        if ((lStatement = ReadNextStatement()) != null)
            lStatements.Add(lStatement);
        else throw new ParserException("Unexpected end of source.");
    }

    ReadNextToken(); // skip 'end'
    SkipExpected(TokenType.Word, "while"); // skip 'while'

    return new WhileStatement(lCondition, new StatementCollection(lStatements));
}

A for-statement starts with the word for, followed by a variable, the symbol :=, a start-value, the word to, an end-value, optionally the word by with a step-size, the word do, a block of statements and the words end for:

ForStatement ParseForStatement()
{
    ReadNextToken(); // skip 'for'
    CheckForUnexpectedEndOfSource();

    if (fCurrentToken.Type != TokenType.Word)
        throw new ParserException("Expected a variable.");

    Variable lVariable = new Variable(fCurrentToken.Value);
    ReadNextToken();

    SkipExpected(TokenType.Symbol, ":="); // skip ':='
    Expression lStartValue = ParseExpression();

    SkipExpected(TokenType.Word, "to"); // skip 'to'
    Expression lEndValue = ParseExpression();
    CheckForUnexpectedEndOfSource();

    Expression lStepSize;
    if (fCurrentToken.Equals(TokenType.Word, "by"))
    {
        ReadNextToken(); // skip 'by'
        lStepSize = ParseExpression();
    }
    else lStepSize = new IntegerConstant(1);

    SkipExpected(TokenType.Word, "do");
    List lStatements = new List();
    Statement lStatement;
    CheckForUnexpectedEndOfSource();
    while (!fCurrentToken.Equals(TokenType.Word, "end"))
    {
        if ((lStatement = ReadNextStatement()) != null)
            lStatements.Add(lStatement);
        else throw new ParserException("Unexpected end of source.");
    }

    ReadNextToken(); // skip 'end'
    SkipExpected(TokenType.Word, "for"); // skip 'for'

    return new ForStatement(lVariable, lStartValue, lEndValue, lStepSize, new StatementCollection(lStatements));
}

An assignment and a function call statement both start with an identifier, so we'll have to read the next token to determine if it's an assignment or a function call statement:

Statement ParseAssignmentOrFunctionCallStatement()
{
    Token lToken = fCurrentToken;
    ReadNextToken();
    CheckForUnexpectedEndOfSource();

    if (fCurrentToken.Equals(TokenType.Symbol, ":="))
        return ParseAssignment(new Variable(lToken.Value));

    if (fCurrentToken.Equals(TokenType.Symbol, "("))
        return new FunctionCallStatement(ParseFunctionCall(lToken.Value));

    throw new ParserException("Expected a statement.");
}

An assignment just has an expression after the :=:

Assignment ParseAssignment(Variable variable)
{
    ReadNextToken(); // skip ':='
    return new Assignment(variable, ParseExpression());
}

In the next post, we'll write the methods for parsing expression.

Technorati tags: Parser, ADL, Acronymic Demonstrational Language, Programming Language, Visual Studio, C#.

Writing a parser: ADL parser node types

2007-04-28T18:28:00.001+02:00

In this post, I'll show you the different types of parser nodes. If you remember from one of the first posts I made in this series, the parser analyzes the tokens it gets from the tokenizer and turns it into structured trees. The nodes in this tree represent the different constructs (additions, multiplications, comparisons, ...).

ParserNode
- Statement
  - Assignment: x := ...
  - IfStatement: if ... then ... else ... end if
  - WhileStatement: while ... do ... end while
  - ForStatement: for i := ... to ... by ... do ... end for
  - FunctionCallStatement: f(...)
- Expression
  - UnaryExpression
    - Negation: -x
    - NotExpression: not x
  - BinaryExpression
    - Addition: x + y
    - Subtraction: x - y
    - Multiplication: x * y
    - Division: x / y
    - AndExpression: x and y
    - OrExpression: x or y
    - Comparison: x < y or x > y or x == y or ...
  - Variable: x
  - IntegerConstant: 123
  - StringConstant: "abc"
  - FunctionCall: f(...)

A statement is an instruction that does not return a value. An expression is an instruction that does return a value. A unary expression has an operator and a single child expression. A binary expression has an operator and 2 child expressions.

You might have noticed that I defined both a FunctionCall and a FunctionCallStatement. The difference is that a FunctionCall is an expression that returns the value of the function call, and a FunctionCallStatement is a statement that ignores the value of the function call.

The basic types look like this:

public abstract class ParserNode
{
}

public abstract class Statement : ParserNode
{
}

public abstract class Expression : ParserNode
{
}

As you can see, they don't contain any code. Because I'm just explaining how parsers work, they don't have to contain code. You should add code if you use these classes to build an interpreter or a compiler. For an interpreter, the Statement class could have a method Execute to execute the statement, and the Expression class could have a method Evaluate to evaluate the expression and return the result.

There's no point in showing the source code in this blog post for all the classes, so you can download them here. Just create a new folder ParserNodes in the TC.Adl project and include the downloaded C# files.

Next time, we'll write the actual parser and a test application.

Technorati tags: Parser, ADL, Acronymic Demonstrational Language, Programming Language, Visual Studio, C#.

Writing a parser: ADL Tokenizer correction

2007-04-25T07:33:00.001+02:00

Sushovan made a great remark on my last post. He asked me why I didn't use the method char.IsWhiteSpace() to test for white-space. The reason I didn't, was ignorance: I had totally forgot about that method. So it's probably a good idea to change the method SkipWhitespace to this:

void SkipWhitespace()
{
    while (char.IsWhiteSpace(fCurrentChar))
        ReadNextChar();
}

Writing a parser: ADL Tokenizer

2007-04-24T21:30:00.001+02:00

We'll now start writing the tokenizer.

One thing I haven't explained yet is how we will deal with white-space. The easiest solution is to just skip the white-space between tokens, and that's what we'll do. Usually white-space is defined as one of the following characters: space (' '), tab ('\t'), new-line ('\n') or carriage-return ('\r'). But I've noticed that any character below space (' ') is never used as text. So we can just consider any character <= ' ' to be white-space, which makes it easier to test a character.

Because we want our tokenizer to read characters from strings as well as streams, we'll use a TextReader. TextReader has 2 derived classes: StringReader (to read text from a string) and StreamReader (to read text from a stream). The only method in TextReader we will use is Read(), which returns the next character or -1 if the end of the stream or string has been reached.

To extract a token, we read the first character and decide what type of token it will be. For each token, we will have to store the characters it contains. We'll use a StringBuilder for this.

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;

namespace TC.Adl
{
    public class Tokenizer
    {
        TextReader fSource;
        char fCurrentChar;
        StringBuilder fTokenValueBuffer;
    }
}

Our Tokenizer class has only 3 fields:

fSource: The TextReader to read characters from.
fCurrentChar: The current character (most recently read).
fTokenValueBuffer: The StringBuilder to store the characters of the current token.

The constructor of the Tokenizer class will accept a TextReader argument, store it in fSource, initialize fTokenValueBuffer and read the first character:

public Tokenizer(TextReader source)
{
    if (source == null) throw new ArgumentNullException("source");

    fSource = source;
    fTokenValueBuffer = new StringBuilder();
    ReadNextChar();
}

Now we'll add some private helper methods.

Reading a character is simple. Just call TextReader.Read(), which returns an int. Because a char cannot be -1 (which is returned by TextReader.Read() at the end of the stream), we will use character '\0' as the end-of-stream character.

void ReadNextChar()
{
    int lChar = fSource.Read();
    if (lChar > 0)
        fCurrentChar = (char)lChar;
    else fCurrentChar = '\0';
}

Skipping white-space is also easy: just keep reading characters until we get a character that is not white-space:

void SkipWhitespace()
{
    while (fCurrentChar > '\0' && fCurrentChar <= ' ')
        ReadNextChar();
}

To determine if we're at the end of the source, we just have to check the current character for '\0':

bool AtEndOfSource { get { return fCurrentChar == '\0'; } }

We'll also write a simple method to store the current character and read the next:

void StoreCurrentCharAndReadNext()
{
    fTokenValueBuffer.Append(fCurrentChar);
    ReadNextChar();
}

And a method to extract the stored characters and clear the buffer:

string ExtractStoredChars()
{
    string lValue = fTokenValueBuffer.ToString();
    fTokenValueBuffer.Length = 0;
    return lValue;
}

Because the source code can have errors, we'll need some methods that throw exceptions:

void CheckForUnexpectedEndOfSource()
{
    if (AtEndOfSource)
        throw new ParserException("Unexpected end of source.");
}

void ThrowInvalidCharException()
{
    if (fTokenValueBuffer.Length == 0)
        throw new ParserException("Invalid character '" + fCurrentChar.ToString() + "'.");
    else
    {
        throw new ParserException("Invalid character '" 
                + fCurrentChar.ToString() + "' after '" 
                + fTokenValueBuffer.ToString() + "'.");
    }
}

Which introduces our ParserException class (a standard Exception class):

using System;
using System.Collections.Generic;
using System.Text;
using System.Runtime.Serialization;

namespace TC.Adl
{
    [Serializable]
    public class ParserException : Exception
    {
        public ParserException() { }
        public ParserException(string message) : base(message) { }
        public ParserException(string message, Exception innerException) : base(message, innerException) { }
        protected ParserException(SerializationInfo info, StreamingContext context) : base(info, context) { }
    }
}

Now that we've written the private helper methods, we can write the only public method: the ReadNextToken method. This method reads a token and returns it. First we skip the initial white-space. If we've reached the end of the source, we'll return null, else we'll check the first character to determine the type of token:

If the token starts with a letter, it's a word.
If the token starts with a digit, it's an integer constant.
If the token starts with a quote, it's a string constant.
If it's any other character, we assume it's a symbol.

public Token ReadNextToken()
{
    SkipWhitespace();

    if (AtEndOfSource)
        return null;

    if (char.IsLetter(fCurrentChar))
        return ReadWord();

    if (char.IsDigit(fCurrentChar))
        return ReadIntegerConstant();

    if (fCurrentChar == '"')
        return ReadStringConstant();

    return ReadSymbol();
}

A word starts with a letter (already tested in ReadNextToken()) followed by zero or more letters or digits. So all we have to do is keep reading until we've reached a character that is not a letter or digit:

Token ReadWord()
{
    do
    {
        StoreCurrentCharAndReadNext();
    }
    while (char.IsLetterOrDigit(fCurrentChar));

    return new Token(TokenType.Word, ExtractStoredChars());
}

An integer constant just contains digits:

Token ReadIntegerConstant()
{
    do
    {
        StoreCurrentCharAndReadNext();
    }
    while (char.IsDigit(fCurrentChar));

    return new Token(TokenType.Integer, ExtractStoredChars());
}

A string constant contains a sequence of characters, enclosed in quotes. Because the quote character is used as a delimiter, the characters in between cannot be quotes. But all other characters are allowed. If the end of the source is reached before the closing quote, we throw an exception. We don't want the quotes to be included in the value of the token, so we'll skip them with ReadNextChar.

Token ReadStringConstant()
{
    ReadNextChar();
    while (!AtEndOfSource && fCurrentChar != '"')
    {
        StoreCurrentCharAndReadNext();
    }

    CheckForUnexpectedEndOfSource();
    ReadNextChar();

    return new Token(TokenType.String, ExtractStoredChars());
}

Reading a symbol is more complicated. A symbol can be one or two characters, so we'll have to treat each case individually:

Token ReadSymbol()
{
    switch (fCurrentChar)
    {
        // the symbols + - * / ( ) ,
        case '+':
        case '-':
        case '*':
        case '/':
        case '(':
        case ')':
        case ',':
            StoreCurrentCharAndReadNext();
            return new Token(TokenType.Symbol, ExtractStoredChars());
        // the symbols := ==
        case ':':
        case '=':
            StoreCurrentCharAndReadNext();
            if (fCurrentChar == '=')
            {
                StoreCurrentCharAndReadNext();
                return new Token(TokenType.Symbol, ExtractStoredChars());
            }

            CheckForUnexpectedEndOfSource();
            ThrowInvalidCharException();
            break;
        // the symbols < <> <=
        case '<':
            StoreCurrentCharAndReadNext();
            if (fCurrentChar == '>' || fCurrentChar == '=')
            {
                StoreCurrentCharAndReadNext();
            }
            return new Token(TokenType.Symbol, ExtractStoredChars());
        // the symbols > >=
        case '>':
            StoreCurrentCharAndReadNext();
            if (fCurrentChar == '=')
            {
                StoreCurrentCharAndReadNext();
            }
            return new Token(TokenType.Symbol, ExtractStoredChars());
        default:
            CheckForUnexpectedEndOfSource();
            ThrowInvalidCharException();
            break;

    }

    return null;
}

That's it, our Tokenizer class is done. To test it, we'll write a small application. Add a new Form to the project TC.Adl.Test, named TokenizerTest. It will have a multiline TextBox where we can type in source code (TextBoxSource), a Button to tokenize the source code (ButtonTokenize), and a ListBox that will display the tokens (ListBoxTokens). It should look like this (follow link to enlarge):

The Click event handler of the button creates a new Tokenizer, reads all the tokens, and adds them to the ListBox:

private void ButtonTokenize_Click(object sender, EventArgs e)
{
    ListBoxTokens.Items.Clear();
    ListBoxTokens.BeginUpdate();
    try
    {
        using (StringReader lSource = new StringReader(TextBoxSource.Text))
        {
            Tokenizer lTokenizer = new Tokenizer(lSource);

            Token lToken = lTokenizer.ReadNextToken();
            while (lToken != null)
            {
                ListBoxTokens.Items.Add(lToken.Type.ToString() + ":\t" + lToken.Value);
                lToken = lTokenizer.ReadNextToken();
            }
        }
    }
    catch (ParserException lException)
    {
        MessageBox.Show(this, lException.Message, this.Text, MessageBoxButtons.OK, MessageBoxIcon.Error);
    }
    finally { ListBoxTokens.EndUpdate(); }
}

Now you can start the application, run the TokenizerTest and try out some code to see what tokens are generated. Try out this sample:

for i := 1 to 100 do
    if i < 50 then
        print(i, " < 50")
    else
        print(i, " >= 50")
    end if
end for

As you can see, converting characters to tokens is not very complicated. You just have to get used to it.

Technorati tags: Parser, Tokenizer, ADL, Acronymic Demonstrational Language.

Writing a parser: ADL tokens

2007-04-22T18:29:00.001+02:00

ADL only has 4 different tokens:

Word: A word starts with a letter, followed by zero or more letters or numbers.; Examples: x, abc, f2, else.
Integer: An integer is a sequence of one or more digits.; Examples: 1, 42, 3141592654.
String: A string is a sequence of characters enclosed in quotes.; Examples: "x", "abc", "Quid pro quo.".
Symbol: A symbol is one of the following sequences:; + - * / ( ) , := == < > <> <= >=

To identify tokens, we'll use an enum called TokenType:

namespace TC.Adl
{
    public enum TokenType
    {
        None = 0,
        Word,
        Integer,
        String,
        Symbol
    }
}

(The value None is the default value and should not occur.)

A token has a type (of type TokenType) and a value (of type string). The value is the sequence of characters that represent the token.

using System;
using System.Collections.Generic;
using System.Text;

namespace TC.Adl
{
    public class Token
    {
        public Token(TokenType type, string value)
        {
            fType = type;
            fValue = value;
        }

        readonly TokenType fType;
        public TokenType Type { get { return fType; } }

        readonly string fValue;
        public string Value { get { return fValue; } }

        public bool Equals(TokenType type, string value)
        {
            return fType == type && fValue == value;
        }
    }
}

You may have noticed that there are no comments or argument validation code. This is just to make the code simpler and easier to understand at first sight. The code I'm writing in Visual Studio is fully commented and has all the necessary argument validation code. I'll release the entire library afterwards.

Next time, we'll write the tokenizer.