Comments on: Interesting problem

By: aaron

aaron — Wed, 18 Oct 2006 03:09:08 +0000

That’s next on my list, as soon as I finish the draft manual. We’ve had good success building a replacement ckit parser. I am hoping to provide an analog to ckit for SML, which would be pretty exciting given that the new parser supports extensible grammars. Anyway, I should have a chance to play with an SML grammar before the release, so I’ll send you whatever I end up with.

By: washburn

washburn — Wed, 18 Oct 2006 02:56:41 +0000

Sounds great. Have you experimented with converting the SML grammar from LALR(1) to LL(k) yet? If so that would be a invaluable reference for me to work from.

By: aaron

aaron — Wed, 18 Oct 2006 02:33:19 +0000

I know that it’s been *forever*, but in the meantime we’ve written a completely new, LL(k) parser generator. Actually this week we are planning to put out 110.60, which will include the new tools. I will send you mail when this happens.

By: washburn

washburn — Wed, 18 Oct 2006 01:54:03 +0000

Ah, I had been meaning to send you some e-mail about this. I just grabbed the latest version of SML/NJ this afternoon and I didn’t see anything about Unicode lexing. Though right now, I’m only going to add a couple tokens to start with (I’ve run out of delimiters…) so I can probably just hardcode the UTF-8 for those.

By: aaron

aaron — Wed, 18 Oct 2006 01:46:00 +0000

We thought about this for ml-ulex, but decided it was much simpler to put a hand-written UTF-8 -> Int decoder in front of the lexer, and then have the lexer (internally) work over words rather than bytes…

By: washburn

washburn — Tue, 17 Oct 2006 23:18:36 +0000

A little. It gives a regular expression for matching valid UTF-8 sequences. But it will take some additional thinking to pare that down to specific glyph ranges.

By: kitby

kitby — Tue, 17 Oct 2006 23:15:08 +0000

Is this page at all helpful?