module Netulex:Support module for Alain Frisch's
ULBis a Unicode-based lexing buffer that reads encoded strings and makes them available to the lexer as both Unicode arrays and UTF-8 strings.
The sub module
Ulexing is a replacement for the module
ulex with the same name. It uses
ULB to represent
the main lexing buffer. It is much faster than the original
Ulexing implementation when the scanned text is UTF-8
Ulexing.utf8_lexeme is frequently called to
get the lexeme strings. Furthermore, it can process input
data of all encodings available to
Netconversion. It is,
however, no drop-in replacement as it has a different
To enable this version of
Ulexing, simply put an
open Netulex before using the
Of course, you need Alain Frisch's
ulex utility first. It installs
itself under the name
ulex as findlib library.
Next, write your lexer, e.g. (line numbers in brackets):
 open Netulex  let digits = lexer  | ['0'-'9']+ -> `Number(int_of_string(Ulexing.utf8_lexeme lexbuf))  | 8364 -> `Euro_sign (* Code point #8364 in Unicode *)
This is a very trivial example. The lexer accepts sequences of digits,
and returns them as `Number tokens. Furthermore, the euro sign is
recognized and returned as `Euro_sign. Note that in the first case
is called to retrieve the current lexeme as UTF-8 string. (Well,
digits are a bad example, as they are only ASCII, and UTF-8 is not
really needed. Imagine you want to extend the scanner to other
number systems represented in the Unicode character set.)
Line 1 is quite important. If you don't open
Netulex, the generated
ulex code will use the version of the
Ulexing module coming with
and not this one.
Call the lexer as follows (line numbers in brackets):
 let sample = "42543\226\130\172";;  let ulb = Netulex.ULB.from_string `Enc_utf8 sample;;  let lexbuf = Netulex.Ulexing.from_ulb_lexbuf ulb;;  let first_token = digits lexbuf;;
`Number 42543. After
 let second_token = digits lexbuf;;
this variable is set to
`Euro_sign, because the three-byte sequence
"\226\130\172" represents the euro sign in UTF-8.
In line 6, the encoding
`Enc_utf8 selects that
sample is an
UTF-8 string. You can pass here any encoding the
If you would like to scan from another source, just change line 6, e.g.
[6'] let ulb = Netulex.ULB.from_in_obj_channel ch
ch is any input channel the
Netchannels module supports.
For example, to read from a file:
let ch = new Netchannels.input_channel (open_in "filename")
You should compile the examples with
ocamlfind ... -package ulex,netstring -syntax camlp4o ...
For the syntax of the lexer rules, see the documentation coming