Component "cgi"
Netcgi
 Advanced connectors


RELATED
Reference Manual
   
Netcgi

CGI is a standard way of connecting a web application to a web server. When the server gets a request that is bound to a CGI-based web application, a new process is started running the application, and the request is transferred to the new process, which is expected to generate a reply. Finally, the process terminates. This means that CGI deals with the protocol between the server and the application, and with a certain process model. It does neither deal with the underlying Internet protocol (HTTP), nor with the type of language used for the transferred contents. For these reasons, we call CGI a connector, and should keep in mind that there are alternate connectors, e.g. the Ajp protocol.

Some features in Netcgi are connector-specific, but most are not. It was a design principle of Netcgi to separate both domains from each other. The benefit for the programmer is that it is quite easy to change the used connector, or even to create programs that support several connectors. The following explanations are generic and valid for all connectors, except when a certain connector is explicitly mentioned.

There are two ways of calling a web application:

  • Call by link: A hyperlink can point to a URL that is bound to the web application. The URL can contain additional parameters. The HTTP method is always GET. It is recommended to use GET only for actions that do not involve side effects (e.g. modifications of databases), because browsers may repeat the HTTP requests without asking the user, and they may look into their cache for the page without sending a request at all.

    HTML example:

    <A HREF="http://myserver/cgi/myapplication?param=value">Start my web application</A>
    

  • Call by form: A form can also point to a URL that is bound to the web application. The contents of the form elements (buttons, text boxes etc.) are sent to the application as parameters. The HTTP method can be selected, and is either GET or POST. As already mentioned, GET requests must not have side effects, whereas POST requests are allowed to have them. When the browser sends a POST request again (e.g. because the user presses the reload button), usually a dialog pops up asking the user whether he wants to send the request again. POST requests have another advantage over GET requests: There is no size limit for parameters.

    HTML example:

    <FORM ACTION="http://myserver/cgi/myapplication" METHOD="POST">
      Enter your search query:
      <INPUT TYPE="TEXT" NAME="query">
      <INPUT TYPE="SUBMIT" NAME="submit" VALUE="Go!">
    </FORM>
    

When the user clicks at the link, or presses the button, the browser sends a HTTP request to the web server, and the server forwards the request to the web application. The application generates an HTML page as reply, and this page is displayed next by the browser. As explained, only POST requests should have side effects.

The Netcgi library contains functions to analyze arriving web requests, and to produce a formally correct reply. Normally, there are three O'Caml modules involved:

  • Netcgi_types: This module defines some fundamental types including cgi_argument for the passed parameters, and cgi_activation representing a single request/reply cycle. (Note: Although the name includes the prefix "cgi" these types can also be used for other connectors. The names simply reflect the fact that the CGI protocol serves as a reference for all other connection methods, and that it is the default connector for Netcgi. Nevertheless, most of the values and types with "cgi" in their name can be used for other connectors, too, you are just reminded that things work like they do in CGI.)

  • Netcgi_env: This module defines a type for the execution environment, and two implementations: one for the CGI environment, and one for testing. The environment implementation effectively selects the connector to use.

  • Netcgi: This module contains the implementations for cgi_argument and cgi_activation that are normally used. These implementations are connector-independent.

The simplest possible CGI application

This application looks as follows:

let actobj = new Netcgi.std_activation() in
actobj # set_header();
actobj # output # output_string "<HTML><BODY>Hello world!</BODY></HTML>\n";
actobj # output # commit_work() ;;

The first line creates a new activation object. This means to create a new environment object, too, and to initialize these objects from the real environment. Effectively, the web request has been read in, and the program could now continue by analyzing the request.

Well, this is the simplest possible web application, and because of this, we do not look at the request. Instead, the reply is prepared in the second line by setting the header (of the reply). Note that this does not mean to send the header immediately, Netcgi may defer this until the "right moment" has come. When the header is set, one can choose to transfer additional information in the header. Of course, the simplest possible web application uses a standard header without extra features.

The third line outputs the contents of the reply. actobj # output is a so-called netchannel, an object-oriented encapsulation of I/O channels. See the chapter about netchannels for details. Netchannels have a method output_string appending the passed string to the channel, which means here that the string is appended to the reply.

The fourth statement commits the data appended to the used netchannel. This means that the data are valid and must be sent to the server. In this example, the statement has only the effect that the internal buffers are flushed. However, the statement can mean much more, as there is a special mode where you can say afterwards that the data are invalid and must not be sent - the next chapter explains this special transactional mode.

Note that the output channel is normally not closed. This is usually the task of the connector, although it would be harmless here.

These four lines work only with the CGI connector. The first statement implicitly selects the CGI connector which is the default if nothing else is specified. The other three statements are connector-independent.

Out of scope: After you have compiled these four lines, the question remains how to configure your web server such that this application is bound to a certain URL. This depends on the web server and cannot be answered here.

Command-line: You can start this application on the command-line. It detects automatically that it runs outside a CGI environment and enters the test mode. You can input test parameters, and a single dot (".") lets the program continue. This prints the reply on the screen. There are even command-line options, use -help to get a summary.

Bad style: Don't output the reply directly on stdout! This is incompatible with a number of Netcgi features, including generating a good header, the web transactions, and of course non-standard connectors.

Direct replies vs. transactions

By default, the activation object is configured such that it sends the reply immediately to the web server (at least when you flush the internal buffer of the output channel). This is called the direct mode. It has a disadvantage: When an error happens in the middle of page generation, a part of the page is already sent to the server, and the only thing you can do is to append the error message to the end of the already sent text. The consequences: Users often do not see that there is an error at all because the error message is at the bottom of the page or even invisible. It becomes even possible that incompletely initialized forms can be submitted, leading to follow-up errors.

For reliable applications it is better to select the transactional mode. Instead of being immediately sent to the server the reply is completely buffered until it is committed (by calling commit_work). Alternatively, it is also possible to discard the whole reply (by calling rollback_work), and to generate a completely different one.

The following scheme should be used to enable transactions:

let process (actobj:cgi_activation) =
  try
    actobj # set_header ... ();
    ...
    Code that writes to  actobj # output
    ...
    actobj # output # commit_work();
  with
    error ->
      actobj # output # rollback_work();
      actobj # set_header ~status:`Internal_server_error ();
      ...
      Code that writes the error message to  actobj # output
      ...
      actobj # output # commit_work()
in

let operating_type = Netcgi.buffered_transactional_optype in

process (new Netcgi.std_activation ~operating_type ())

The transactional mode is selected by passing the argument ~operating_type. There are two implementations, one keeping the buffer in the main memory (buffered_transactional_optype), one using an external file as buffer (tempfile_transactional_optype). The latter should be used for very large replies (one megabyte and up).

Normally, the same statements are executed as in the direct case: set_header, then the reply is sent to the actobj # output channel, and finally commit_work is called to flag the data as valid, and to flush the whole transaction buffer.

When an exception is raised, the try...with... block catches it. First, the transaction buffer is discarded (rollback_work), and a different header is set. Here, we set the status code to `Internal_server_error. This is important, because pages with an error status are differently treated as pages with a successful status; mainly the browser is not allowed to put them into its cache. You can send other status codes as long as they indicate either a client or a server error (e.g. you can respond with `Forbidden when the user is not allowed to access the page).

Next, the error message is written into the transaction buffer, and finally, the buffer is committed.

Some details to better understand what is happening:

  • The set_header statement does not send the header immediately. This is deferred until the first commit_work. You can change the header as often as you want until the first commit_work is executed.

  • It is possible to call commit_work and rollback_work several times. Comitting the buffer means to send it, rolling it back means not to send it, and in both cases the buffer is emptied.

Although transactions increase the reliability of a web application, they do not fully ensure it. There is still no way to guarantee that the network connection does not break, and that the browser processes the data as it is intended by the application. Nevertheless, transactions are useful as they allow it to report errors in a reasonable way.

Transactions are independent of the selected connector.

Processing Parameters

After the activation object has been created, it allows one to access the parameters coming with the web request. These four methods read the currently available parameters:

method arguments : (string * cgi_argument) list
method argument : string -> cgi_argument
method argument_value : ?default:string -> string -> string
method multiple_argument : string -> cgi_argument list

In particular, arguments returns all parameters as associative list, argument returns the first parameter with the passed name, and multiple_argument returns all parameters with a certain name (it is allowed to have several parameters with the same name). These three methods return cgi_argument objects containing all properties of the parameters. These properties include the name and the value of the parameter, but there may be more (see below). Especially the method value returns the current value, so the call

(actobj # argument "foo") # value

returns the string value of the argument, or raises Not_found if there is no parameter called "foo". As an abbreviation, one can alternatively write

actobj # argument_value "foo"

which is exactly the same. Furthermore, argument_value accepts a default value that is returned if the parameter is missing. The following two expressions are again equivalent:

try (actobj # argument "foo") # value with Not_found -> "0"

actobj # argument_value ~default:"0" "foo"

For simple arguments only having a name and a value this is already the whole story.

File uploads require that arguments have some extra capabilities. An HTML form containing file upload elements must look like:

<FORM ACTION="...anyURL..." METHOD="POST" ENCTYPE="multipart/form-data">
   ...
   <INPUT TYPE="FILE" NAME="...anyName...">
   ...
</FORM>

The attributes METHOD and ENCTYPE must have exactly these values, otherwise the upload will not work.

Arguments resulting from file uploads have some extra properties:

  • There is a content type such as "text/plain" or "application/octet-stream"

  • The content type may have parameters, such as charset specifying the character set of texts

  • The file has a filename, normally the original name of the file that is uploaded

  • Optionally, the file is stored as temporary file in the filesystem of the server (to avoid that large files are loaded into memory); this file must be somehow accessible

The first three of these properties can be simply retrieved by calling the right methods of the argument object, as in

let arg = actobj # argument "...anyName..."
let content_type = arg # content_type 
let charset = arg # charset
    (* or: let charset = List.assoc "charset" (arg # content_type_params) *)
let filename = arg # filename

Note that filename is None if the file upload box was not used.

The last special feature of upload arguments is the possibility to store the uploaded contents of files again in files on the server side. This must be explicitly enabled, by default the contents are kept in the main memory. To do so, you must pass the option ~processing when you create the activation object, as in

let processing arg_name arg_header =
  (* return either `Memory, `File, or `Automatic *)
  ...
in
let actobj = new std_activation ~processing ()

The function processing is called just after the header of the argument has been received, and the function must decide how to handle the argument. The function is called for all MIME arguments, not only file upload arguments![1] Possible result values are: `Memory meaning that the argument is loaded into memory, or `File meaning that the argument is stored in a file, or `Automatic meaning that arguments with filenames are stored in files, and all other are kept in memory. What you probably want is

let processing arg_name arg_header = `Automatic

What changes when you store an argument into a file? At the first glance: nothing. When you call value or argument_value, the contents of the argument are automatically loaded from the file. However, you can now call some extra methods:

let store = arg # store in
let server_filename = 
  match store with
    `File name -> name
  | `Memory -> failwith "Argument not stored in file"

The store method returns the location where the contents of the argument are stored, either `Memory or `File _; the returned filename is the absolute path of the file in the filesystem of the server. - Furthermore, you can open the file for reading:

let ch = arg # open_value_rd()

Here, ch is an input netchannel reading the contents of the file[2]. - Last but not least, it is a good idea to delete such files after usage. To do so, just call:

arg # finalize()

This deletes the file if still existing. You can also call actobj # finalize() which deletes all existing argument files of the current activation at once.

Arguments work for all connectors in the same way.

Setting the header

There are two methods that set the header of the reply:

  method set_header :
           ?status:status -> 
           ?content_type:string ->
           ?cache:cache_control ->
           ?filename:string ->
           ?language:string ->
           ?script_type:string ->
           ?style_type:string ->
           ?set_cookie:cgi_cookie list ->
           ?fields:(string * string list) list ->
           unit ->
             unit

  method set_redirection_header :
           string ->
             unit

set_header must be used for a normal reply, while set_redirection_header initiates a redirection to another URL.

set_header has a number of optional arguments. You can omit all arguments, in which case a standard header is used. We discuss here only status, content_type, filename, and set_cookie:

  • status sets the return code of the HTTP reply, by default `Ok (code 200). As already mentioned it is important to set the status to the right value, otherwise caches do not work properly. For the exact meaning of the various status code you have to look into RFC 2616. The most important error code is `Internal_server_error which should also be used for generic malfunctions of web applications. Note that you can send an error document along with the status code which is displayed by the browser[3].

  • content_type sets the content type of the replied document, by default text/html. You can change the character set of the document by appending the charset parameter, e.g. text/html; charset=utf-8. By returning different content types you can trigger various content handlers of the browser. It is a common misunderstanding that the content type application/octet-stream enables the "save as" dialog, use the filename option to do so (below). application/octet-stream only means that the server does not know the real content type; the client is allowed to recognize it by magic numbers or other techniques.

  • filename sends the "content-disposition" header that enables the "save as" dialog. The dialog is initialized with the passed string as file name. Note that the name must neither contain backslashes nor double quotes as many browsers have problems with these characters. In general, it is a good idea to omit any directories, and ideally the name consists only of alphanumeric characters (including "_") and a three-letter extension (guess why?)

  • set_cookie can be used to set or delete cookies. You can pass a list of cookie records:

    type cgi_cookie = Cgi.cookie = 
        { cookie_name : string;
          cookie_value : string;
          cookie_expires : float option;
          cookie_domain : string option;
          cookie_path : string option;
          cookie_secure : bool;
        }
    

    The cookie_name should only consist of alphanumeric characters, whereas the cookie_value can be an arbitrary string. You can pass an expiration time in seconds since the epoch; None means that the cookie expires when the browser session ends. The other three arguments determine to which web requests the browser adds the cookie; all three criterions must match. cookie_domain specifies the domain that must occur in the URL; None means that only the same server gets the cookie that set it. cookie_path specifies the path prefix that must occur in the URL; None means that the current path must occur. Finally, cookie_secure specifies whether SSL is necessary.

    Existing cookies are overwritten. You can delete cookies by using an expiration time in the past. You can retrieve existing cookies from the environment (see below).

The other header-setting method is used for redirections. There are two type of redirections, server-driven and client-driven redirections. If the redirection is performed by the web server, the browser will not notice that a redirection has happened; the browser just gets the page to which the redirection points. Of course, this works only if the new location is served by the same web server. In the case of client-driven redirections, a special reply is sent to the browser asking to go to the new location. The user normally sees this because the web address changes.

Use set_redirection_header in both cases. A server-driven redirection will be chosen if you only pass the part of the URL that contains the path, and the client-driven redirection is taken for absolute URLs:

actobj # set_redirection_header "/cgi/myapplication";   (* server-driven *)
actobj # set_redirection_header "http://myserver/cgi/myapplication"; 
                                                     (* client-driven *)

Note that the url method can be useful to create these strings when they are derived from the current URL. See below for details.

After you have set the header, it is still necessary to commit the output channel, although you need not to output anything. So don't forget to call actobj # output # commit_work()!

The environment

Up to now, we have mostly used the activation object. This object represents the knowledge about the current request/response cycle. There is a second object, the environment, that focuses more on the conditions on which the current cycle bases, and that also represents the used connection method.

In particular, you can get the following information by looking at the environment:

  • The input and output channels that can be used to transfer the contents of the request and the response, respectively

  • The input and (designated) output headers

  • The meta data about the request, i.e. the current URL, the request method etc.

  • The environment keeps the overall state of the input/output processing, e.g. whether the output header has already been sent or not

Most of these data are under control of the activation object, and it is strongly recommended not to modify any header, channel, or state as this would confuse the activation object. Of course, it is allowed to read the current header, and the meta data:

let env = actobj # environment 
let date = env # input_header_field "date" 
let ctype = env # input_header_field "content-type"
let script_name = env # cgi_property "SCRIPT_NAME"
                  (* or: env # cgi_script_name *)

Note that the names of header fields are normalized according to the conventions of HTTP even if the connection protocol uses a different style. For example, CGI transports the content type as "CONTENT_TYPE", and the user agent string as "HTTP_USER_AGENT", but the environment translates these names back to "content-type" and "user-agent" (case-insensitive), as they are used in the HTTP protocol. This does not apply to the meta data of the current request that do not occur in the HTTP header, like the URL, or the internet address of the client. So the "SCRIPT_NAME" property must be called "SCRIPT_NAME", no name translation takes place here. (Sorry for this distinction, but we have to find good connector-independent names for these fields.)

Bad style: It is not a good idea to get these values directly from the process environment (Sys.getenv). This would work only for CGI, but not for other connectors.

By the way, you can also find the current set of cookies in the environment, because cookies are transferred in the HTTP header. For convenience, there is an access method:

let cookies = env # cookies in
let my_cookie = List.assoc "my_cookie" cookies

cookies returns the cookies as associative list.

Now that we have an idea what data are represented in the environment: When is the environment object created? By default, together with the activation object, but you can do that yourself. The line

let actobj = new std_activation()

creates a default environment because no env argument is present. The default environment is

let env =
  try
    new std_environment()
  with
    Std_environment_not_found ->
      new test_environment()

where the class std_environment creates an environment that works for CGI, and test_environment is responsible for the command-line test loop.

You can specify the environment to use. For example,

let actobj = new std_activation ~env:(new std_environment()) ()

has a CGI-only environment, the test loop is disabled.

A very good reason to create your own environment is the possibility to pass the configuration record:

type cgi_config =
    { (* System: *)
      tmp_directory : string;
      tmp_prefix : string;
      (* Limits: *)
      permitted_http_methods : string list;
      permitted_input_content_types : string list;
      input_content_length_limit : int;
      workarounds : workaround list;
    }

val default_config : cgi_config

The configuration record is kept inside the environment, and can be specified when the environment object is created. The component tmp_directory is the absolute path of the directory for temporary files, and tmp_prefix is the file name prefix to use for these files. input_content_length_limit is the maximum allowed size for a web request. (The other components are documented in the mli file.)

Example: Create a customized configuration record, an environment, and an activation object in turn:

let config =
  { default_config with
      tmp_directory = "/var/spool/webapps";
      tmp_prefix = "myapplication";
      input_content_length_limit = 10485760;  (* 10 MB *)
  } in

let env =
  new std_environment ~config () in

let actobj =
  new std_activation ~env ()

This way of creating environments works only for the CGI connector, as other connectors use different process models that are incompatible with this particular way. The contents of the environment object, however, are connector-independent.

Generating links and forms

OcamlNet contains some convenience functions to generate HTML code. Most important is the function to HTML-encode a string. In HTML, the characters <, >, &, and sometimes[4] " and ' have special meanings and must be denoted in a special way:

  • &lt; encodes <

  • &gt; encodes >

  • &amp; encodes &

  • &quot; encodes "

  • &#39; encodes '[5]

HTML defines more such entities, but you need not to use them, just send the character as such. For example, it is not necessary to write a-umlaut as &auml;, just use ä. Of course, you must ensure that the right character set is mentioned in the content-type header (e.g. do actobj # set_header ~content_type:"text/html;charset=utf-8" () to select the UTF-8 encoding). Note that missing charset parameters are often added by the web server, and this may go wrong.

The function that HTML-encodes any string is:

let enc_str = Netencoding.Html.encode ~in_enc ~out_enc () unenc_str

where unenc_str is the unencoded string, in_enc is the assumed character set of unenc_str (e.g. `Enc_iso88591 or `Enc_usascii or `Enc_utf8), and out_enc is the assumed character set of enc_str. Normally, it is best to choose in_enc = out_enc because that avoids unnecessary transformations, but it is also allowed to pass `Enc_usascii for out_enc. This forces that entities are used for non-ASCII characters (e.g. &auml; instead of ä). Note that the encode function normally does not transform ', but you can enforce this by passing the additional parameter unsafe_chars:

let unsafe_chars = Netencoding.Html.unsafe_chars_html4 ^ "'" in
let enc_str = Netencoding.Html.encode ~in_enc ~out_enc ~unsafe_chars () unenc_str

You may wonder why there is the () argument. Because encode rebuilds the transformation tables for every call, it is possible to split the invocation in two parts:

let encode_utf8 = Netencoding.Html.encode 
                    ~in_enc:`enc_utf8 ~out_enc:`Enc_utf8 () in
let enc_str = encode_utf8 unenc_str

The transformation tables are already built when the partial application of the first let is executed, and this extra work is avoided when encode_utf8 is called. This technique is highly recommended when you call encode_utf8 frequently.

By the way, there is also Netencoding.Html.decode for the reverse transformation.

Now that we know how to efficiently apply the HTML encoding, let us now go a step further. An often needed feature is to generate FORM element that points to its generator (a "recursive" form). All you have to do is to put the URL into the ACTION attribute of this element. But how to do it right? This code shows a good solution:

let own_url = actobj # url() in
let enc_own_url = Netencoding.Html.encode ~in_enc ~out_enc () own_url in
actobj # output # output_string (sprintf "<FORM ACTION=\"%s\">" enc_own_url)

The url method returns the current URL of the activation object (see below how to get variations of the current URL), but without query string, if any (the part after the question mark). The URL is HTML-encoded and included as ACTION attribute. It is very unlikely that the URL contains characters that actually must be encoded, but we get it from an untrusted source, and it is better to program defensively here. Note that URLs may contain & in general, but this cannot happen here because the query string is omitted.

The following piece of code creates a complete FORM element that does not only point recursively to its generator, but also includes all web parameters. Note that we use POST because this method does not limit the length of the transferred parameters:

let enc = Netencoding.Html.encode ~in_enc ~out_enc () in
let own_url = actobj # url() in
actobj # output # output_string 
                  (sprintf "<FORM ACTION=\"%s\" METHOD=POST>\n" (enc own_url));
List.iter
  (fun (arg_name,arg) ->
    let arg_value = arg # value in
    actobj # output # output_string 
      "<INPUT TYPE=HIDDEN NAME=\"%s\" VALUE=\"%s\">\n"
      (enc arg_name)
      (enc arg_value)
  )
  (actobj # arguments);
actobj # output # output_string "</FORM>\n";

It is essential that the NAME and VALUE attributes are HTML-encoded because they can contain arbitrary characters. Note that this technique if often used, but usually not all arguments are included into the FORM. Furthermore, you may encounter problems with multi-line arguments (containing line separators), and it is a good idea to include CR and LF into the list of unsafe_chars to avoid that.

The url method has the following interface:

  method url : ?protocol:Netcgi_env.protocol ->   
                                            (* default: from environment *)
               ?with_authority:other_url_spec ->        (* default: `Env *) 
               ?with_script_name:other_url_spec ->      (* default: `Env *)
               ?with_path_info:other_url_spec ->        (* default: `Env *)
               ?with_query_string:query_string_spec ->  (* default: `None *)
               unit ->
                 string

There are a lot of optional arguments referring to URL fragments. In general, an (absolute) URL has the format

authority/script_name/path_info?query_string

where /path_info and ?query_string are optional fragments. The authority consists usually of the protocol name and host name of the denoted server (but may include more information), as in "protocol://hostname". The script name is the "directory path" of the server that is bound to the web application. Path info is the exceeding directory path, and the query string contains the parameters that are passed by URL.

The url method allows the programmer to specify which parts of the URL should be taken from the current URL, which parts are to be left out, and which parts be overriden manually with certain values. For with_authority, with_script_name, and with_path_info one can pass these values: `Env means to use the corresponding fragment of the current URL. `None means to omit the fragment. `This "string" means to include this string value as fragment.

The protocol argument is only used if with_authority=`Env, see below why one can pass it separately in this case.

with_query_string has different cases: `Initial means to use the arguments as they were passed to the current invocation of the web application (i.e. from the request). `Current means to use the current arguments (they can be modified), `None means to omit the query string, and `Args args forces url to include exactly the arguments args into the URL.

Example: Create a hyperlink that refers recursively to the web application, but sets the parameter x to the value 1 (as only parameter):

let x_arg = new simple_argument "x" "1" in
let u = actobj # url ~query_string_spec:(`Args [x_arg]) in
let enc_u = Netencoding.Html.encode ~in_enc ~out_enc () u in
actobj # output # output_string 
  (sprintf "<A HREF=\"%s\">The same again with x=1</A>")

Example: Create a hyperlink that refers recursively to the web application, but redirects the accesses to the secure server:

let u = actobj # url ~protocol:(`Http((1,1), [`Secure_https])) () in
let enc_u = Netencoding.Html.encode ~in_enc ~out_enc () u in
actobj # output # output_string 
  (sprintf "<A HREF=\"%s\">The same again but secure</A>")

FAQ

  • Help! My CGI program always displays "Internal server error"

    There are a large number of reasons, and it depends on your web server software. From my experience, the most frequent problems are:

    • Your application does not work, and produces incorrect output. You can check that by calling it on the command line, passing the necessary arguments, and looking at the output. If the output has the format of a mail message, i.e. a header and a body, your application is probably not the cause of malfunction.

    • Your web server is improperly configured. First look into the error log file, because there is usually a hint explaining what is exactly going wrong. Often, access is too restrictive, e.g. CGI support is disabled for the directory, or there are problems with symbolic links. If you have Apache, another problem might be that "suexec" is enabled (as in most Linux distributions), and "suexec" is very restrictive to avoid security holes in the server setup.

    • User permissions are often wrong. First find out which user account of the system is used to run your application. This can be the user of the web server process, or any other user if this is configured in the web server (e.g. by using "suexec"). Then check whether your application actually can run as this user.

    Links:

  • Where can I find a complete example?

    The source distribution of OcamlNet contains several examples:

    • counter: Displays a counter that is increased by one when a button is hit (very simple example)

    • add: Adds two numbers and displays the result (simple example)

    • filemanager: A directory of files can be managed over the web (realistic small application)

    • mailbox: A web page displays the contents of a mailbox; RFC822 mails can be viewed including attachments (also demonstrates how to parse mail messages)

  • How can I output debug messages?

    Of course, you cannot just print messages to stdout. Nevertheless, there are ways to get debug messages:

    • Some web servers (including Apache) append lines printed to stderr to the error log file. Other web servers do not distinguish between stdout and stderr (iPlanet), and this method is not applicable.

    • Of course, you can just append your messages to a file. Ensure that the user running your application actually is allowed to create the file, and to write to it.

    • Another idea is to create a named pipe, and to display any arriving messages (e.g. in an xterm):

      $ mkfifo /tmp/debugcgi
      $ chmod 777 /tmp/debugcgi
      $ while true; do cat /tmp/debugcgi; done
      

      At the beginning of your program, close stderr and reopen it as /tmp/debugcgi:

      Unix.close Unix.stderr;
      ignore(Unix.openfile "/tmp/debugcgi" [Unix.O_WRONLY] 0);
      

      Now stderr is again your error channel, and prerr_endline, eprintf etc. display the messages in your xterm.

  • How can I run the program in the debugger?

    If possible, run the program from the command-line, because it is difficult to debug in a CGI environment.

    If that fails, do the following:

    • Increase the time-out value of the web server. The default time-outs are quite short (5 minutes), and if your application does not generate output within the time-out, the server closes the connection, and generated an error message.

    • Make sure your program has been compiled with the -g option.

    • Manage that the environment variable CAML_DEBUG_SOCKET is set to the name of a socket that is used for the communication between the debugger and the application, e.g. /tmp/debugsocket. You can use a wrapper script to set the variable, e.g.

      #! /bin/sh
      CAML_DEBUG_SOCKET=/tmp/debugsocket
      export CAML_DEBUG_SOCKET
      exec myscript.bin "$@"
      

      Some web servers allow it to set environment variables by changing the configuration. Apache: Add to .htaccess

      SetEnv CAML_DEBUG_SOCKET /tmp/debugsocket
      

      provided that mod_env is available.

      Note that a Unix domain socket has been found to be more convenient as an Internet socket, because "Address already in use" errors can be avoided easily.

    • Ensure that your socket does not exist yet (rm -f /tmp/debugsocket)

    • Start the debugger as in the following example:

      $ (umask 0; ocamldebug add.cgi)
              Objective Caml Debugger version 3.06
      
      (ocd) set socket /tmp/debugsocket
      (ocd) set loadingmode manual
      (ocd) break @Add 67
      Loading program... 
      Waiting for connection...(the socket is /tmp/debugsocket)
      

      You can start the debugger in any directory you want. It may be necessary to enter directory directives to extend the search path for O'Caml modules.

      Note that umask 0 has the effect that the socket is created world-writeable, so the CGI process can write to it.

    • Trigger the web request such that the CGI is started. The CGI stops at the breakpoint, and the debugger displays

      done.
      Breakpoint 1 at 403728 : file Add, line 68 column 3
      (ocd) _
      

      and is ready to accept your commands. You can now step through the program. During debugging, the browser waits for incoming data.

  • The browser displays the page without calling my program again. What do I wrong?

    Most likely, this is a caching problem. The browser caches certain types of pages as explained now. Proxies count as caches, too, and the same rules apply.

    Depending on the type of the page, one of the following behaviours may happen:

    • (1) The page is not written into the cache.

    • (2) The page is written into the cache, but at most taken for display after user interaction. Think of historical pages that can be reached by the back button of the browser.

    • (3) The page is cached with a certain expiration period. Within this period, the page is taken from the cache. After this period, it is checked whether the page is still up to date (revalidation).

    • (4) The page is cached with a certain expiration time, but no revalidation is performed even after expiration, at least for some time.

    For all types of pages, there is a certain default behaviour, but you can demand a different behaviour by sending a cache control directive. This directive controls whether caching is allowed at all, how long the expiration period is, and whether revalidation is forced. Note that these directives must really be included in the response header, and it is not sufficient to encode them as HTML META tags. In Netcgi, you can use the set_header method with the cache option to change the caching behaviour.

    Revalidation is technically implemented by a variant of GET requests: GET-IF-MODIFIED-SINCE (there is an additional header field). The page is only sent if there are modifications, otherwise the "not modified" code is returned by the web server. For CGIs, the web server always calls the CGI script and returns the page unconditionally, so revalidation requests are effectively treated like normal GET requests.

    Now the behaviours for the different request/response types:

    • Responses to GET requests, default behaviour: This is (4), and expiration is controled by a certain rule of the cache configuration. Some browsers have an option to force (3) as default.

    • Responses to GET requests with explicit expiration period: This is (3). From Netcgi this mode can be demanded by setting the cache option of set_header to `Max_age n where n is the duration of the expiration period in seconds.

      A `Max_age 0 has the effect of (2).

    • Responses to GET requests with disabled cache: This is (1). From Netcgi this mode can be demanded by setting the cache option of set_header to `No_cache. Note that many browsers disable history caching, too, and "Back" shows only an error message.

    • Responses to POST requests, default behaviour: This is (2).

      Note that it does not make sense to specify an expiration period for POST.

    • Responses to POST requests with disabled cache: This is (1). From Netcgi this mode can be demanded by setting the cache option of set_header to `No_cache. Note that many browsers disable history caching, too, and "Back" shows only an error message.

    HEAD requests behave like GET, and PUT and DELETE requests behave like POST.

    As a rule of thumb: Applications that use only POST requests do not suffer from inappropriate caching behaviour. The same applies to GET requests when responses include the cache options `Max_age 0 or `No_cache.

    A final comment on the HTTP versions: The newer protocol HTTP/1.1 specifies caching behaviour much better than the older protocol HTTP/1.0. An important improvement is that clock skews between client and server are irrelevant for HTTP/1.1 but not for HTTP/1.0. Fortunately, all current browsers support HTTP/1.1 meaning that caching problems can be mastered. Older browsers may be problematic, though.

  • My application does not work under a secure server. Why?

    There are some subtle differences between secure (SSL-enabled) web serving and normal serving. First, the protocol prefix is "https" and not "http". For URLs returned by the url method Netcgi adds the correct protocol prefix, but URLs may be wrong when computed by non SSL-aware routines.

    For some browser configurations, caching is turned totally off when SSL is used. The idea is that securely transferred data should not be stored on disks that are not specially protected, at least not without knowledge of the user. Unfortunately, some browser functions do not work properly for such configurations, such as embedding client application into browser windows.

    Another known incompatibility is that authentication by client certificates works differently than basic authentication ("username/password" dialogs). The user name is, at least for some web servers, not written into the REMOTE_USER variable.

    SSL has the concept of "sessions", and the web server decides which request is part of which session. Some servers begin a new session whenever a request arrives for which a different access control rule is configured, even if this rule is weaker than the old rule (e.g. you try to get a GIF from a less-protected /images directory). The effect is that the user is frequently asked for certificates. That means: it might be necessary to rearrange directories to avoid problems with "session boundaries".

  • How do I restrict access?

    This depends on your web server software.

    You can get the authenticated username by calling the method cgi_remote_user of the environment object. As explained above, this sometimes does not work with SSL-enabled web servers. In this case, you can try to call cgi_property with the right name of the variable containing the user name.

  • Are filenames of downloads allowed to contain non-ASCII/Unicode characters?

    Try to avoid this. Although there are RFCs that specify how this could work, no browser implements them. Furthermore, filesystems often have no special support for non-ASCII characters.

  • The browser sends strange byte sequences when non-ASCII characters are entered into text boxes. How do I change this?

    Add the ACCEPT-CHARSET attribute to that FORM. The browser has simply guessed the wrong character set to return, but you can specify that.

  • What is the maximum length of URLs?

    URLs should not be longer than 256 bytes, but most software products support longer URLs. There is often a hard limit, I know that some web servers reject URLs that are longer than 32 KB.

    Netcgi does not limit the length of URLs, so URLs can be as long as strings can be.

  • What is the maximum length of cookies?

    The encoded cookie must not be larger than 2 KB.

  • How secure are applications developed with Netcgi?

    Well, I think that Netcgi implements all security-related features correctly, but it is of course possible that the author of the web application introduces security holes, e.g. improper configuration of the web server, or improper access rights of files, or stupidities like transferring passwords in cleartext.

    In general, O'Caml is a good basis for secure applications. There are no buffer overflows (unless you use the unsafe access functions for strings and arrays, so don't do that), and the exception mechanism helps that operating problems are usually detected before they can be abused. Nevertheless, some advices:

    • Don't use scripts. When the web server is improperly configured, it is possible that attackers get the source code of your application. Better compile your program. With O'Caml you can, fortunately.

    • Don't use the toploop to execute your program. In theory, it is possible that malicious code gets compiled and executed. Without toploop, this is much harder.

    • Don't use the Dynlink module. It makes it possible that bytecode is dynamically added to your program.

    • If you have the choice, prefer ocamlopt-compiled programs. Their text segment is not writeable. Furthermore, you can turn the "r" bit off in the file permissions making it impossible to download the program by accident. In general, bytecode programs are less secure than native programs, as it is possible to attack the bytecode interpreter by overwriting the memory blocks containing the bytecode, and to execute arbitrary code (although this is very difficult, normally the garbage collector causes a segfault before that happens). The bytecode interpreter needs the "r" bit to read the bytecode in.

    • Don't expose the exact exceptions in error pages (unless it is clearly a user error). This may help attackers to find problems in your configuration. Better write such exceptions into log files, and show the user only a generic text like "something has went wrong".


[1]
Arguments are encoded as MIME messages when the FORM element contains the attribute ENCTYPE="multipart/form-data". Of course, all arguments are encoded in this way, not only those dealing with file uploads.
[2]
This works even for argument that are not stored in files!
[3]
Some browsers (Internet Explorer) replace very short error messages by built-in messages. This violates Internet standards, but complaining does not help, and you have to live with such strange features.
[4]
In attribute values that use these characters as delimiters. You cannot use the character inside the attribute value that delimits the value.
[5]
In XML, you can use &apos;, but this is not officially supported in HTML.