module Neturl: sig .. end

Contents

• Interface
• Tutorial

Interface

This module provides functions to parse URLs, to print URLs, to store URLs, to modify URLs, and to apply relative URLs.

URLs are strings formed according to pattern (1) or (2):

1. scheme://user;userparams:password@host:port/path;params?query#fragment
2. scheme:other;params?query#fragment

Not all characters are allowed in a URL. Some characters are allowed, but have the special task to separate the various parts of the URL (reserved characters). However, it is possible to include even invalid or reserved characters as normal content by applying the %-encoding on these characters: A '%' indicates that an encoded character follows, and the character is denoted by a two-digit hexadecimal number (e.g. %2f for '/'). In the following descriptions, the term "encoded string" means a string containing such %-encoded characters, and the "decoded string" means a string not containing such characters. See the module Netencoding.Url for functions encoding or decoding strings.

The type url describes values storing the components of a URL, and the url_syntax for the URL. In general, the components are stored as encoded strings; however, not for all components the %-encoding is applicable.

For convenience, the functions creating, modifying, and accessing URLs can handle both encoded and decoded strings. In order to avoid errors, the functions pass strings even in their decoded form.

Note that there is currently no function to compare URLs. The canoncical comparison ( = ) is not applicable because the same URL may be written in different ways.

Note that nothing is said about the character set/encoding of URLs. Some protocols and standards prefer UTF-8 as fundamental encoding and apply the %-encoding on top of it; i.e. the byte sequence representing a character in UTF-8 is %-encoded.

Standards Compliance

This module implements RFC 1738 and RFC 1808. There is also a newer RFC, 2396, updating the former RFCs, but this module is not fully compatible with RFC 2396. The following (minor) problems may occur:

• The module escapes more characters than needed. All characters that are "unsafe" or "reserved" in either RFC document are escaped.
• URL parameters (appended with a ";") are handled as in RFCs 1738/1808. In RFC 2396, every path component may have parameters, and the algorithm to resolve relative URLs is different in this point. If it is required to apply RFC 2396, one can disable URL parameters in the syntax, and extract them from the path by a self-written postprocessor. Usually, this is only required for imap URLs.

• Authorities may be terminated by a question mark, as in "http://host?query". This is illegal in RFC 1738. The consequence is, however, that question marks in user strings must be escaped.

exception Malformed_URL

val extract_url_scheme : string -> string

type url 

val url_syntax_is_valid : url_syntax -> bool

val partial_url_syntax : url_syntax -> url_syntax

val null_url_syntax : url_syntax

 let my_syntax = { null_url_syntax with
                     url_enable_host = Url_part_required; ... }
 

val ip_url_syntax : url_syntax

val common_url_syntax : (string, url_syntax) Hashtbl.t

val null_url : url

val make_url : ?encoded:bool ->
       ?scheme:string ->
       ?user:string ->
       ?user_param:string list ->
       ?password:string ->
       ?host:string ->
       ?port:int ->
       ?path:string list ->
       ?param:string list ->
       ?query:string ->
       ?fragment:string -> ?other:string -> url_syntax -> url

val modify_url : ?syntax:url_syntax ->
       ?encoded:bool ->
       ?scheme:string ->
       ?user:string ->
       ?user_param:string list ->
       ?password:string ->
       ?host:string ->
       ?port:int ->
       ?path:string list ->
       ?param:string list ->
       ?query:string ->
       ?fragment:string -> ?other:string -> url -> url

val remove_from_url : ?scheme:bool ->
       ?user:bool ->
       ?user_param:bool ->
       ?password:bool ->
       ?host:bool ->
       ?port:bool ->
       ?path:bool ->
       ?param:bool ->
       ?query:bool -> ?fragment:bool -> ?other:bool -> url -> url

val default_url : ?encoded:bool ->
       ?scheme:string ->
       ?user:string ->
       ?user_param:string list ->
       ?password:string ->
       ?host:string ->
       ?port:int ->
       ?path:string list ->
       ?param:string list ->
       ?query:string ->
       ?fragment:string -> ?other:string -> url -> url

val undefault_url : ?scheme:string ->
       ?user:string ->
       ?user_param:string list ->
       ?password:string ->
       ?host:string ->
       ?port:int ->
       ?path:string list ->
       ?param:string list ->
       ?query:string ->
       ?fragment:string -> ?other:string -> url -> url

val url_syntax_of_url : url -> url_syntax

val url_of_string : url_syntax -> string -> url

val string_of_url : url -> string

val parse_url : ?schemes:(string, url_syntax) Hashtbl.t ->
       ?base_syntax:url_syntax ->
       ?accept_8bits:bool -> ?enable_fragment:bool -> string -> url

val fixup_url_string : string -> string

val url_provides : ?scheme:bool ->
       ?user:bool ->
       ?user_param:bool ->
       ?password:bool ->
       ?host:bool ->
       ?port:bool ->
       ?path:bool ->
       ?param:bool ->
       ?query:bool -> ?fragment:bool -> ?other:bool -> url -> bool

val url_scheme : url -> string

val url_user : ?encoded:bool -> url -> string

val url_user_param : ?encoded:bool -> url -> string list

val url_password : ?encoded:bool -> url -> string

val url_host : url -> string

val url_port : url -> int

val url_path : ?encoded:bool -> url -> string list

val url_param : ?encoded:bool -> url -> string list

val url_query : ?encoded:bool -> url -> string

val url_fragment : ?encoded:bool -> url -> string

val url_other : ?encoded:bool -> url -> string

val split_path : string -> string list

 split_path "a/b/c" = [ "a"; "b"; "c" ],
 split_path "/a/b"  = [ ""; "a"; "b" ],
 split_path "a/b/"  = [ "a"; "b"; "" ] 

val join_path : string list -> string

val norm_path : string list -> string list

val apply_relative_url : url -> url -> url

val ensure_absolute_url : ?base:url -> url -> url

val file_url_of_local_path : ?getcwd:(unit -> string) -> string -> url

val local_path_of_file_url : url -> string

val print_url : url -> unit

Tutorial

This module is a quite flexible parser for various kinds of URLs occuring in practice. The syntax is configurable such that one URL module can handle a lot of URL types in a generic way.

Generic Parsing

In order to parse an absolute URL (beginning with a scheme identifier like "http:...") of unknown type just call

 
 let url = parse_url "http://me@server/directory"
 

By default, this function can parse all URL types listed at Neturl.common_url_syntax. However, the default configuration implies also that

• relative URLs cannot be parsed
• fragment identifiers are rejected (i.e. the part after the hash mark like in "http://server/document#location")
• characters are rejected when the most significant bit (MSB) is set

The restriction that relative URLs are rejected has to do with the problem that context information is missing. Because the scheme identifier (like "http") is not available, the function does not know which syntax the relative URL should have. For example, the relative URL dir/file?x=1 is differently parsed when it is taken relative to an http URL and when it is interpreted relative to an ftp URL. In the first case, the path component of the URL is "dir/file" and the query component is "?x=1", but in the latter case the path component is "dir/file?x=1", and a query component is not allowed.

The solution is that the syntax of the base URL, relative to which the URL is seen, must be passed as additional argument. Under the assumption that base_url is the base URL, use

 let url = parse_url 
             ~base_syntax:(url_syntax_of_url base_url) 
             "/dir/file?x=1"
 

Parsing For a Certain Syntax

The function url_of_string is also a parser, but you must pass the URL syntax as argument, e.g.

 let url = url_of_string syntax "ipp://server/printer"
 

 let syntax = Hashtbl.find common_url_syntax "ipp" 

Printing URLs

This is much easier, just call string_of_url to convert an URL to a string. It is ensured that every URL always has an unambiguous representation as string.

URL Components

Internally, the parsed URL is decomposed into its components. This module supports two ways of decomposition:

1. scheme://user;userparams:password@host:port/path;params?query#fragment
2. scheme:other;params?query#fragment

Examples:

• http://me:abrakadabra@server/dir?x=5#section1

  scheme="http", user="me", password="abrakadabra", host="server", path="/dir", query="x=5", fragment="section1"

• pop://you;auth=digest-md5@mail

  scheme="pop", user="you", user_params=["auth=digest-md5"], host="mail"

• mailto:gerd@gerd-stolpmann.de?cc=you@domain.com

  scheme="mailto", other="gerd@gerd-stolpmann.de", query="cc=you@domain.com"

It is important to mention that the decomposition is not fully performed, but only down to a certain level. For example, the query "x=5" could be further analysed and be split into the syntactic parts "x" and "5". However, this is not done, just because the author seeked a compromise between the depth of analysis and the genericy of application.

URL Escaping

In order to represent the so-called unsafe characters, one can use %-escaping in URLs. For example, this URL contains a password with @, an unsafe character encoded as %40:

 http://user:!$%40?@server 

It is surprising that the URL parser does not decode these escaped forms (it checks, however, whether they are syntactically correct). Internally, the components are stored as parsed, and one can even retrieve them in their original form. The function url_password returns the password component. Applied to the above URL, one can get the password in its original, "encoded" form, or as decoded string:

• url_password ~encoded:true url returns "!$%40?"
• url_password url returns "!$@?"

The URL components can be retrieved with the functions

• url_scheme
• url_user
• url_user_param
• url_password
• url_host
• url_port
• url_path
• url_param
• url_query
• url_fragment
• url_other

The path component (url_path) has a non-obvious representation. The path is represented as string list, e.g. "a/b/c" is represented as ["a";"b";"c"] . Note, however, that absolute paths have an empty string at the beginning of the list, e.g. "/a/b/" is [""; "a"; "b"; "" ] . In most cases, the paths found in URLs are absolute, and because of this it is quite common to find this empty string at the beginning of the path list. The corner cases are:

• [] is used when the path is missing in the URL
• [ "" ] is "/"
• [ ""; "" ] is considered as illegal

There is the helper function split_path to convert the string representation of paths into the list representation.

The parameters (url_user_param and url_param) are lists, too. A parameter starts with a semicolon as delimiter and runs until the next component, which can be another parameter. The contents, i.e. the values after the semicolons are put into the list. For example, the parameter ";auth=unix;type=i" is represented as ["auth=unix"; "type=i"] .

Hint: Getting Query Arguments

The query component is represented as a single string. When queries use the standard syntax "name1=value1&name2=value2&...", one can parse this string using

 let args = Netencoding.Url.dest_url_encoded_parameters
              (url_query ~encoded:true url)
 

Creating and Modifying URLs

In order to create a URL for a certain syntax, call make_url:

 let url = make_url 
             ~scheme:"http"
             ~user:"user"
             ~password:"!$@?"
             ~host:"server"
             syntax
 

The functions modify_url, default_url, undefault_url, and remove_from_url can be used to modify an existing URL.

Relative URLs

A URL is relative when the scheme identifier at the beginning is omitted. In this case, the URL can be transformed to an absolute URL when the base URL is known. The algorithm for this is defined in RFC 1808, and quite complicated. It is implemented in apply_relative_url, but usually Neturl.ensure_absolute_url is the more convenient function. Just call

 let url' = ensure_absolute_url ~base url