www.tartoos.com Client/Server Issues

CGI programs are a way of extending the capability of the Web server without actually modifying the server. You cannot completely master CGI unless you have a thorough conceptual understanding of how the client and server interact. This chapter begins with a conceptual introduction to the Web client and server, starting with the simple operation of accessing a single HTML page and ending with descriptions of more complex examples such as what the server is doing when it runs a CGI program. It then introduces the Web server protocol-HTTP-which is discussed in fairly great detail. Finally, you learn various ways to extend the capabilities of both the Web client and server.

A computer's software largely performs data processing. It reads data in some form, processes that data, and then returns the results. On a single machine, all of the data input, output, and processing is performed on the same machine. A client/server model is one way of extending the software model to work on several machines. The client/server model divides up the data processing and transfers data back and forth over the network.

The server sends the data to the client (possibly processing it first) which the client then processes. The client can send the data back to the server for further processing or display the results.

The World Wide Web is based on this client/server model. The Web server is essentially a distributor of information where the information is stored on the server and sent to the client upon request. The protocol for distributing this information is called the HyperText Transfer Protocol (HTTP).

· The client must always send a request and the server must always send some sort of response. Even if you don't want the server to return any content, you must at least tell the browser to expect no content returned; you cannot just send nothing at all. Additionally, the server sends only one response to the client per request. I elaborate more on this in "HTTP: the Web Server Protocol" later in the chapter, because people are sometimes confused by this.

· The Web protocol (HTTP) follows the response/request paradigm. The client sends one request to the server, and the server sends one response back.

· HTTP is media-independent. It can send any type of information, from raw text to complex video data. By using a standard means of labeling the content, HTTP can distribute any kind of information properly.

· HTTP is stateless. There is only one response per request, and neither the Web server nor client are aware of previous or future transfers. It does not inherently remember information from state to state, although you can manipulate the Web to simulate state as you learn later.

You learn the HTTP protocol in detail in the next section. Now, I want to present a few familiar examples of the Web in action. These should help you understand how the Web works and will prevent you from making mistakes in your CGI programs based on misconceptions about the Web protocol. Additionally, these examples will give you a good idea of what roles both the client and the server play in common transactions. This knowledge will help you think of creative applications to extend the Web either by extending the client, the server, or both.

The most basic example of a request/response paradigm is accessing one HTML document and displaying it on your browser. When the server receives this request, it looks for the document. If it finds it, it sends a success message back to the browser, tells the browser the type of document, and sends the actual document. If it cannot find the document or for some other reason cannot send it, it sends an error message back to the browser instead.

What if the HTML document has an inline image? When the browser receives the HTML document, it parses the document so that it can display it properly. If it finds an <img> tag, it sends another request to the server and accesses that image separately. If it successfully receives the image from the server, it displays it inline along with the rest of the HTML document; otherwise, it displays a broken image.

A common misconception among beginning CGI programmers is that if you want to send an HTML file with inline images, you need to send both the HTML file and the images upon request. This is not the case. When the browser receives the data, it processes the information. The client parses the document and decides whether it needs to make another request to the server. When your browser accesses an HTML file with three inline images, it makes four separate requests to the server: one for the document and one for each image.

What happens if you request a particular location in an HTML file? For example, suppose index.html looks like the following:

<html><head>
<title>Hello, World!</title>
</head>

<body>
<h1>Hello, World!</h1>

<a name="chapter1">
<h2>Chapter 1</h2>

<p>Once upon a time...</p>

<a name="chapter2">
<h2>Chapter 2</h2>

<p>The end.</p>
</body></html>

If you request the URL
http://myserver.org/index.html#chapter2

the client only requests the file index.html. After it retrieves the file, it looks for the HTML tag <a name="chapter2">. If it finds it, it displays the HTML file starting at that tag.

You can see that both the client and server process the data to some extent. The server finds the data, determines the data type, and sends that information back to the client. The Web client receives the data, looks at the data, and determines whether it needs to do anything else. For the rest of this chapter, you learn different scenarios where either the Web client or server or both do additional data processing.

Submitting user input using a form requires additional processing by the client before making the document request. Suppose you have an HTML form that a user fills out and submits. The browser needs to convert the user input into something the server will understand before sending a request to the server. The first thing the browser does is CGI encode the inputBecause the browser does nothing to the data other than CGI encode it before sending it to the server, it is impossible to implement custom encoding of data from the server side alone. For example, the client would have to be modified to understand this encoding and respond in kind.

. After encoding the input, the browser sends the request and the input to the server in one of two ways, GET or POST, depending on the method specified in the form. When the request is made, the browser once again waits for any kind of response from the server.

What does the server do when it receives a request to run a CGI program? The client has no way of knowing that the user wants to access a CGI program; in fact, the client never knows the type of request the user is making. When the browser sends the request, it makes no difference to the browser whether the user wants to receive an image file or an audio file or the HTML output of a CGI program. When the browser receives a response, then it is important to know the data type of the information so that the browser can display it properly.

The server must determine whether the request is for a CGI program or not, and then take the appropriate action. Servers can determine whether a request is for a CGI program in several different ways largely dependent on the server platform. A common way is for the server to check the directory of the request. Some servers (such as the NCSA server) assume that any requests for items in the /cgi-bin directory are requests for CGI programs. Another common method is to look at the extension of the data requested: many servers (such as WebStar for the Macintosh) assume that programs that end in the extension .cgi are CGI programs.

The server cannot assume that the document requested is a CGI program even if the request is embedded in a form. For example, suppose you have the following form:

When this form is submitted, the server receives a request to post the input to the file /index.html. It cannot assume that /index.html is a CGI program (which it is not); otherwise, it will try and run /index.html and will fail. Instead, it first looks at the URL and determines whether the URL specifies a CGI program or not. In this case, the server realizes it doesn't and sends an error message back to the browser stating that you cannot post to a file.

Note

Whereas the preceding form will result in an error message, the following form will not:

When the input is submitted, the browser sends a regular GET request, the same type of request it sends if it wants to access any other type of data such as an HTML document. The server then ignores the submitted input and sends /index.html back to the browser. If the server just assumed that the file specified in the action parameter of the <form> tag was a CGI program, then this request would fail as well.

As soon as the server realizes the request is to run a CGI program, it sets up an environment for the program, passes the input to the program. The program's output is captured by the server, which parses the headers, creates some new headers, and sends the response back to the browser.

Parsing the headers is necessary because although CGI programs need to send one header only at minimum-a Content-Type header, for example-the browser expects more information than that. The browser needs to know whether the request was successful or not, and if it was, what kind of information the server is sending. The server determines this information from the output of the CGI program, and creates these headers appropriately while sending the proper headers from the CGI program, as well.

Remember that the browser always expects some kind of response from the server. If you don't want to send any information to display back to the browser, then you must tell the browser not to expect any information.

Having the server process the output of the CGI program before sending the output to the browser slows down the total transaction time. The server knows that the CGI program is NPH by once again looking at the request. If the filename begins with nph and is a CGI program, then the server assumes that it is an nph CGI program. It runs the program just as it runs any other CGI program, but instead of processing the output headers before sending the information back to the browser, it enables the CGI application to speak directly to the browser. Consequently, the program must send a complete and valid HTTP response to the client rather than a minimal CGI response.

One useful Web server feature is the capability to redirect a request to another location using the CGI header location. Although the result of a redirection is usually transparent to the user, both the client and server can behave in very different ways depending on how the request is made.

The desired result is clear; when someone accesses this CGI program, you want them to receive the contents of /index.html instead. When the browser requests /cgi-bin/goto, the server realizes the file requested is a CGI program and runs it . The program returns a Location header, which the server then parses. Because the Location header does not specify a full URL, the server knows that the new file is located somewhere on that server. It then looks for that file and sends a response as if the client had directly requested that file. In this case, the server does additional processing before sending data back to the browser.The browser receives the same response as it would had it directly requested that file.

When the server parses this header, instead of trying to send a request to the new server and parse that output, it sends this location back to the browser and lets the browser make the new request. Unless otherwise specified, the server tells the client that the file has temporarily moved (status code 302) to the location in the Location header. When the browser receives this information, it forms a new request and attempts to access the URL specified in the Location header. This time, the client is doing additional processing rather than the server.

You can also have the server tell the browser that the file has permanently, rather than temporarily, moved by specifying a new status code (301).

The server and client behave in almost identical ways as they do when the file has only temporarily moved to the new URL. The only difference is that the client stores this information for the entire session. The next time the client wants to access /cgi-bin/goto, the client remembers that /cgi-bin/goto has permanently moved to http://hcs.harvard.edu/ and changes the request to go directly to the new URL rather than go through the same redirection procedure all over again.

Retrieving access-restricted files requires additional processing by both the server and the client. Again, when the client first makes the request for the file, it has no way of knowing that a file is restricted. When the server receives the request, it determines whether it can send the file back or not. If the file is in a password-protected area, the server informs the client by sending the appropriate status code (401) and by telling the client how the area is restricted . Password-protected areas are restricted using the basic authentication type.

Upon receiving this information, the client prompts the user for a username and password . After the user enters this information, the client encodes this information and sends it back to the server. The server determines whether the information is correct or not; if it is, it sends the desired file to the client. After a successful transaction has occurred, the browser can remember to use this authentication information every time it accesses the protected space so that the user does not have to enter a username and password each time. If the authentication information is incorrect, the server informs the browser (using status code 403), in which case the browser can either prompt the user to try entering the information again, or it can display a failure message.

Knowing how HTTP works is extremely useful conceptual knowledge for Web developers. Knowing the protocol-the language the server and client use to communicate-can be useful to the CGI programmer in many ways as well:

· Several types of Web transactions need to know the proper status codes and HTTP headers to send. Knowledge of the various subtleties of HTTP enables you to create much more powerful applications.

· Understanding how the client and server communicate makes it easier to debug your CGI programs. If you know exactly what information is being sent and how, you can determine whether your programs are receiving the correct information and parsing it correctly.

I discuss the parts of HTTP I find most relevant to CGI programming in this section. For a more complete and detailed description of HTTP, you should read the draft of the HTTP specification, available at the W3 organization's Web site: URL:http://www.w3.org/.

The first half of a Web transaction is the request. The basic format of an HTTP request is

The first line of a request always consists of a command followed by the location on the server and HTTP/1.0, which describes what version of the protocol to use.

The additional headers contain information about the browser and any additional data included in the request. For example, if you are sending input to a CGI program using a POST request, the headers will specify the type and length of the information being sent. Any additional headers and data are optional and are dependent on the HTTP request. The blank line following either the headers or the command is necessary; if your request consists of only a one-line command, you must send a blank line as well so that the server knows the request is complete.

The GET request is the most basic type of request and is universally supported by all Web servers. GET tells the server you want to retrieve a document located at the specified location. For example, if you want to retrieve the document /index.html, the browser sends this request followed by a blank line:

The browser could include additional informational headers as well. For example, it could send this request once again followed by a blank line:

GET /index.html HTTP/1.0
Accept: text/html
Accept: */*
User-Agent: Browsername Browserversion

This tells the server the browser's name and version and that the browser can understand HTML files internally and can properly handle any other type of document (probably by opening an external viewer or by saving the data to the disk). These headers are optional and purely informational.

If the GET request is made after a form is submitted, the form input is appended to the end of the GET request. For example:

Caution

Requests for specific locations in an HTML file such as the following request:

http://myserver.org/index.html#chapter2

are not sent as

GET /index.html#chapter2 HTTP/1.0

The server would look for the file called index.html#chapter2 upon receiving this request, which isn't the desired result. Instead, the client remembers the marker name chapter2 and sends the following command:

GET /index.html HTTP/1.0

When it receives index.html, it looks for the marker chapter2 and displays the file beginning at that marker.

One other request header I want to mention is the If-Modified-Since header. When users revisit a page, you want the browser to determine whether the information has changed before downloading the file again. If the information has not changed, the browser can just reload the file from its cache. The way to determine whether a file has been modified since it was last loaded is to use the If-Modified-Since header. For example, the following will ask the server if the file has been modified since November 30, 1996 at 10:29 GMT:

If it has, the server resends the new data. If it hasn't, the server tells the browser the file has not been modified (status code 304) and the browser just reloads the file from its own cache.

The HEAD command is almost equivalent to the GET command except it is used only to determine whether that file exists or not. It sends the exact same headers the GET command would without the accompanying data (similar to the results of a GET command with the If-Modified-Since header if the file has not been modified). HEAD is useful for Web robots that traverse the Web to see if certain files still exist at certain locations.

The POST command is almost universally supported as well and is useful for transferring large amounts of data to the server. For example, if you had a form with several kilobytes of input, it is not feasible to append all of this information on the end of a GET request.

Not only is this ugly, it usually presents technical problems as well. Servers and browsers are not required to handle URLs larger than 1024 bytes. Additionally, because the server stores this input in an environment variable (QUERY_STRING), the length of this information is limited to how much data the environment variable can store.

The POST method provides a flexible and more efficient way of sending data to the server. The following is the proper format; type/subtype describes the format of the data and Content-Length is the length of the data:

POST location HTTP/1.0
Content-Type: type/subtype
Content-Length: n

[encoded data]

The Content-Length header is required; otherwise, the server has no way of knowing how much data to read. The Content-Type is optional; by default, Content-Type is set to the following:

This is the standard MIME type for CGI encoded data. For example, if you wanted to send the form input name=eugene&age=21 using the POST method, the request would look like the following:

POST /cgi-bin/form.cgi HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 19

name=eugene&age=21

There are a few other HTTP commands that most servers have not implemented, including PUT, which enables you to put files on the server, and DELETE. For more information on these and other HTTP commands, see the draft of the HTTP protocol.

HTTP/1.0 xxx Status message
[general headers]
[response headers]
[entity headers]

[entity body]

In this code, xxx is the status code and the headers and body are separated by a blank line. The order of the headers following the status line does not matter. The status code tells the client whether the transaction was successful or not and what to do next. Table 8.1 lists the possible status codes and their definitions. There are four classes of three-digit status codes: 2xx, 3xx, 4xx, and 5xx. Any status code beginning with a 2 means that the transaction was successful. A status code beginning with a 3 means that the client must take further action to complete the transaction (as in the case of redirections). Status codes beginning with a 4 mean that the client has probably made some sort of error such as requesting a file that does not exist. Finally, an error preceded by a 5 means that there was some sort of server error.

Status Code	Definition
200	The request was successful and a proper response has been sent.
201	If a resource or file has been created by the server, it sends a 201 status code and the location of the new resource. Of the methods GET, HEAD, and POST, only POST is capable of creating new resources (for example, file uploading).
202	The request has been accepted although it might not have been processed yet. For example, if the user requested a long database search, you could start the search, respond with a 202 message, and inform the user that the results will be e-mailed later.
204	The request was successful but there is no content to return.
301	The requested document has a new, permanent URL. The new location should be specified in the Location header.
302	The requested document is temporarily located at a different location, specified in the Location header.
304	If the client requests a conditional GET (that is, it only wants to get the file if it has been modified after a certain date) and the file has not been modified, the server responds with a 304 status code and doesn't bother resending the file.
400	The request was bad and incomprehensible. You should never receive this error if your browser was written properly.
401	The client has requested a file that requires user authentication.
403	The server understands the request but refuses to fulfill it, most likely because either the server or the client does not have permission to access that file.
404	The requested file is not found.
500	The server experienced some internal error and cannot fulfill the request. You often will see this error if your CGI program has some error or sends a bad header that the server cannot parse.
501	The command requested has not been implemented by the server.
502	While the server was acting as a proxy server or gateway, it received an invalid response from the other server.
503	The server is too busy to handle any further requests.

General headers can be used by either HTTP requests or responses and have no direct bearing on the information being transmitted. The two general headers are Date and Pragma. Date provides the date of the transaction. Pragma enables you to specify application-specific commands in headers that apply to every point that reads the data (for example, a Pragma header will be read and understood by both a proxy server and the browser accessing the data through the proxy server). For more information on the Pragma header,

The response header contains additional information about the response that the server cannot include on the status line. For example, when you are redirecting the browser to a new location, you specify the new location of the file using the response header Location rather than on the status line:

The Server header contains the name of the server, and the WWW-Authenticate header contains information about the type of authentication being used to protect a file. For example, the status code 401 (file is restricted) is not a useful code unless the browser knows exactly how the file is protected.

Entity headers describe the entity body of data being returned. Valid entity headers and their definitions are shown in Table 8.2.

Header Name	Header Purpose
Allow	Informs the client of the commands allowed by the server, such as GET and POST.
Content-Type	The MIME type of the entity body.
Content-Encoding	Enables you to describe an additional encoding on the data. For example, if the data is a gzipped HTML file, the Content-Type would be text/html and the Content-Encoding would be x-gzip.
Content-Length	The length of the entity body in bytes.
Expires	The expiration date of the data. The data should be reloaded from the server after the date listed by this header.
Last-Modified	The last date and time the file was modified.

The entity body is the raw data being returned. All data must be accompanied by at minimum a Content-Type and Content-Length field. Every response to a GET or POST request requires an entity body except for responses with status code 204 (no content) or 304 (data not modified). If you do not want to include a body (for example, with a redirection using status code 302), you must specify a Content-Length of zero. Responses to HEAD requests must not have entity bodies, although they will most likely have entity headers.

You hopefully now have a strong familiarity with how the Web client and server work. What you might now be interested in is extending the basic capability of your server and possibly your client to perform powerful tasks. CGI programming extends the capability and functionality of the Web server. Understanding the limitations of server-side computing exposes the limitations of CGI programming in general.

CGI applications are simply incapable of doing certain things, mostly because they only process from the server side. I have already mentioned how data encryption is impossible using just CGI because the client needs to encrypt the data as well. As another example, how would you send an HTML file from a CGI program to the browser so that it displays the file at a certain marker rather than from the top of the list? The answer is that you cannot, because the client, not the server, is responsible for storing information about the starting point at which to display the document.

Before the CGI protocol was developed, the only way to extend the server's functionality was to modify the server. Web servers written in interpreted languages such as the Plexus Perl Web server made it quite convenient not only to extend the functionality of the server but to modify other behavior as well. For example, you could modify the server's logging functions so that it logged additional variables such as the last modified date of a file.

Although the CGI protocol has created a powerful platform and server-independent way of extending the functionality of the server, it still has certain limitations. For example, you cannot modify the way the server logs variables or handles requests. You cannot implement another HTTP command in the server such as PUT or DELETE. You cannot change the server's behavior when handling errors or access restrictions to certain files or directories. CGI also inherits limitations from the HTTP protocol. For example, because HTTP is stateless, writing applications that truly maintain state is difficult .

Finally, there is a practical limitation to CGI. CGI is inherently inefficient. In order to run a CGI program, the server must first parse the request and store the information in environment variables, run the CGI program, send the data to this program, parse the output of the program, and finally send the information back. Although for most practical applications this overhead does not decrease the effectiveness of your applications, for busier sites that rely on resource efficiency, the extra overhead can be problematic.

One approach to solving these limitations has been to modularize the server and to write an application programming interface (API) to easily extend the server. If you understand the API of a server, you can extend the server or change its functionality in almost limitless ways by writing modules that process input in various ways. These modules are part of the server rather than separate programs, and as a result, they do not suffer from the same efficiency problems as CGI programs do. Several servers including Netscape's Netsite, Apache, and Microsoft's Internet Server support modules. Writing a module, however, is far more difficult than writing a CGI program and is less portable.

Other servers overcome some of the limitations of CGI by building some commonly desired tasks into the server. Many servers offer server-side includes , which enable you to create dynamic or conditional text by using special tags and having the server parse the document before sending the output to the browser. Some servers (such as the WN server for UNIX) have built-in search capabilities; others have built-in database capabilities. Many servers support content negotiation so that they send a certain document depending on what capabilities the browser has or what language it understands.

Even though many servers offer good alternatives to CGI, CGI programs are still widely used. CGI programs are portable, easy to write, and normally more than effective. While the Web continues to evolve and advance, it is unlikely that there will be any standard or technology that replaces CGI anytime soon.

This book focuses mostly on how to extend the server. The rest of this chapter discusses how to extend the capabilities of the client to create more powerful Web applications. You have already seen several of the things the client can do that the Web server is unable to do. Even though there are many ways to extend the functionality of the server, until recently, there were very limited ways to extend the power of the client.

Previously, users were limited by the features available to them from their Web browser. Fortunately, the recent popularity of the Web has forced programmers to write more powerful, more capable, and more reliable Web browsers. You could do things like client-side imagemaps only if your browser supported that feature. Until recently, there was no standard protocol for extending the powers of the Web client analogous to CGI for Web servers.

Some Web browsers took steps to enable the Web author to extend the capability of the browser by downloading programs from Web servers and executing them within the browser. One of the earliest browsers that could download and execute code from the server was Joseph Wang's TkWWW in 1992. Written using the John Ousterhout's Tk/Tcl scripting languages, this browser could download and execute code from Web servers, allowing Web authors to design custom Web applications. For example, although TkWWW did not support inline animations, you could write a module in Tk that would display an inline animation within the browser and distribute it from your Web server.

Most recently, a technology from Sun called Java has emerged as a standard for extending the Web client. Java is a programming language that was designed specifically for portability and security. Java programs are compiled to run on a "virtual machine." As a result, Java shares the portability advantage of scripting languages and some of the speed advantages of a compiled language.

Java has many potential applications but is most widely used as a way to increase the power of the Web client. Browsers that support Java can download Java applications (or applets, as they are called on the Web) and can run them internally. These applets can perform almost limitless tasks from multimedia to networking. With Java applets, you can have inline multimedia, real-time data updating, and more.

The "almost" limitless is important for security reasons. You can imagine what sort of problems a totally unrestricted language could cause if you were allowed to download any code snippets and execute them automatically. Malicious users could write code that would upon execution read and e-mail the contents of your hard drive to them, for example. Other, more serious possibilities exist as well. As a result, Java was written specifically with security in mind and has several restrictions that prevent people from writing malicious code.

Many people are somewhat confused as to what you can do with Java, and some people have mistakenly claimed that Java and similar technology mean the end of CGI programming. Java is a client-side technology, not a server-side one like CGI. It can accomplish many things that CGI can't, but the reverse is true as well. Java helps overcome some of the limitations by allowing you to extend the Web client.

I believe some of the most exciting potential of Java lies with the interaction of Java applets and CGI applications. I already described how if you could somehow encrypt data from the client-side, you could build your own custom data encryption system into your Web site. Another example of the interesting things you can do with both Java and CGI is to extend HTML forms. Currently, HTML forms are fairly limited in capability. There are only a few types of input fields available (with good reason; the HTML drafters did not want the specification to be too large). With Java, you can create pull-down menus, sliding bars, and any interface enhancement you want.

What if you wanted to have a field that would accept a dollar amount? With HTML forms, you would use a text field tag. However, with Java, you could create a field that would accept only positive numerical amounts, thus taking care of type-handling before the information is submitted. Other possibilities like this one exist with the Java language and other client-side technologies.

I have emphasized over and over again the different roles of both the Web client and server. I believe that many of the misconceptions about CGI programming and newer technologies such as Java is the confusion over how the Web works and the differences between server-side and client-side programming.

This chapter focused on both conceptual and more specific features of HTTP-the Web server protocol. Knowing specifically how HTTP clients and servers communicate helps you debug your software and write more powerful applications. Finally, knowing when to use the server or the client to process information enables you to extend the Web in new and exciting ways.