eXtropia: the open web technology company
Technology | Support | Tutorials | Development | About Us | Users | Contact Us
 ::   Tutorials
 ::   Presentations
Perl & CGI tutorials
 ::   Intro to Perl/CGI and HTML Forms
 ::   Intro to Windows Perl
 ::   Intro to Perl 5
 ::   Intro to Perl
 ::   Intro to Perl Taint mode
 ::   Sherlock Holmes and the Case of the Broken CGI Script
 ::   Writing COM Components in Perl

Java tutorials
 ::   Intro to Java
 ::   Cross Browser Java

Misc technical tutorials
 ::   Intro to The Web Application Development Environment
 ::   Introduction to XML
 ::   Intro to Web Design
 ::   Intro to Web Security
 ::   Databases for Web Developers
 ::   UNIX for Web Developers
 ::   Intro to Adobe Photoshop
 ::   Web Programming 101
 ::   Introduction to Microsoft DNA

Misc non-technical tutorials
 ::   Misc Technopreneurship Docs
 ::   What is a Webmaster?
 ::   What is the open source business model?
 ::   Technical writing
 ::   Small and mid-sized businesses on the Web

Offsite tutorials
 ::   ISAPI Perl Primer
 ::   Serving up web server basics
 ::   Introduction to Java (Parts 1 and 2) in Slovak


Intro to the Web Application Development Environment
Introduction to Transmission  
Previous | Next | Table of Contents

[The Communication Layer]

After the user has submitted whatever information they need to submit into whatever GUI you have designed, it is time to get that information from the web browser to the web server.

To do this, the information has to pass through the Communication Layer.

As you might imagine, there are a host of technologies which all poke their way into the Communication Layer. You'll hear terms like TCP/IP, OSI/ISO, HTTP, HTTP-S, SSL, etc. You'll also hear about technologies that modify the data in transmission such as encryption.

Whichever the case, the main idea that you should get used to when thinking about the Communication Layer is that information needs to get from one place to another and that data needs to get there in one piece, in the right order, and with no parts of that data changed or corrupted. In addition, the information could/should be transferred securely. That is, it should go from one place to another without anyone being able to read it, change it, or fake authorship of it.

When we say that information must go from one place to another, we mean specifically, from one computer to another computer and more specifically, from one software program on one computer to another software program on another computer.

Since there are so many types of computers and so many software packages that must communicate, you can imagine that it all just works out to a mass of blabber.

Well, to prevent such chaos, and to make the web work, everyone (software and hardware) is required to follow very rigorous standards of communications called protocols. Protocols guarantee that each participant of the discussion can understand the other.

For the purposes of the web, there are three types of communications protocols we care about:

  • Computer to computer communication - This is handled by the IP protocol
  • Encryption of plain text over the wires - This is handled by a motley of technologies but most commonly involve some form of public key encryption.
  • Software to software communication - Generally, TCP will move data from the hardware level to the software level. Once the data has been moved to the software level, the software packages themselves will provide their own protocols for communication. In the realm of web serves and browsers, this is usually handled by HTTP or SSL in the case of encrypted communication.

Consider the following diagram in which hardware hosts open a TCP/IP connection to exchange data between software programs. Once the TCP/IP connection is opened, the software communicates via HTTP or SSL. The actual data exchanged may or may not be encrypted.

[Communications Layers]

Let's take a deeper look into each of these basic technologies.

The Basics of TCP/IP  
TCP/IP (The Transmission Control Protocol/Internet Protocol) is the protocol suite that drives the Internet. Specifically, TCP/IP handles network communications between network nodes (computers, or nodes, connected to the net).

The suite is actually composed of several protocols including IP which handles the movement of data between host computers, TCP which manages the movement of data between applications, UDP which also manages the movement of data between applications but is less complex and reliable than TCP, and ICMP which transmits error messages and network traffic statistics.

Internet Protocol (IP)  
Internet Protocol (IP) is a connectionless protocol that gateways use to identify networks and paths to networks and hosts. In other words, IP handles the routing of data between networks and nodes on those networks.

You will be familiar with IP because in order to address web pages, you must use domain names like www.barclayscapital.com, or their numerical counterparts called IP addresses.

NOTE: Domain names are far easier to remember than IP addresses so we usually use domain names instead of IP addresses and rely upon Domain Name Servers to tie domain names to their IP address counterparts behind the scenes.

IP addresses are unique sets of four period delimited octets that represent individual hosts on specific networks. A TCP/IP octet is a number between 0 and 255. Thus, an IP Address might look something like:


which you might read as something like "computer named 26, on network 1, on host 85, of network 30."

NOTE: 255 is usually a reserved number which means broadcast to all hosts and 0 represents "this", or the current host.

Besides defining the address scheme, IP also handles the transmission of data from an originating computer to the computer specified by the IP address. It does so by breaking up large, unwieldly chunks of data into easily manageable IP packets that it can deliver across the network.

That is why we talk of IP being a connectionless protocol. In an effort to better manage network traffic, IP specifies the protocol for breaking single messages into a slew of portions. Each portion is responsible for finding its way across the network based on changing traffic congestion and the IP protocol.

Each time a message arrives at an IP router, the router decides where to send it next. There is no concept of a session with a preselected path for all traffic. Routers can send data along the path of least resistance regardless of local network traffic congestion.

If one phone line on the network breaks down, for example, traffic can still reach its destination through a roundabout path.

Transmission Control Protocol (TCP)  
Whereas IP focuses on getting data across the vast network from one computer to another, TCP assures that the data gets sucked off the network connected computer and delivered to the receiving application and that the data is delivered in the correct sequence.

Recall that since IP chops messages up into packets, we need to assure that the messages are assembled correctly at the other end of transport. And if any errors occur during transport, such as message degradation or the loss of some packets, the sender will know to resend.

TCP also tries to optimize network bandwidth by controlling the flow of information dynamically, slowing down as network traffic becomes congested.

Okay, so that is how data is sent from a software program on one computer, transferred through the network, and reassembled on the other end. TCP/IP pretty successfully handles network navigation. However, browsers and servers need to transfer more information than what is required in network navigation.

For example, browsers and servers must understand how to exchange web documents (multimedia files such as images, text, audio, etc). To do so, browsers and servers must speak their own standard protocol to help them exchange these files as well as meta information about those files.

Specifically, browsers and servers speak the HTTP protocol Where TCP/IP drives the internet, HTTP drives the web.

HTTP is a stateless, "request/response" protocol that specifies the means of transport as well as the process for maintaining the integrity of those multimedia files. It does so by using the Multipurpose Internet Mail Extensions (MIME) specification that describes the transfer and format of multimedia files.

The web consists of millions of hyperlinked files. These millions of files in turn, each may contain webs of links pointing to other files in the network. However, because the web is used to deliver files of all types (audio, video, text, etc), web browsers and servers need some way of defining what it is that is being linked to and transported.

To do so, MIME headers are used by HTTP to specify the contents of any transported file. The header will specify a file's type.

[HTTP Mime Usage]

MIME Types and subtypes examples include the following:

MIME Type MIME Subtype
Application msword
Application pdf
Audio aif
Audio mpeg
Image gif
Image jpeg
Image bmp
Multipart form-data
Multipart mixed
Text html
Text plain

The list is hardly exhaustive of course, and in fact web servers and web browsers can easily extend the MIME specification to add new MIME types. All you need to do is make sure that the server is configured to output the new type and that browsers are configured to expect the new type.

Getting browsers to understand new types is usually a function of the options or preferences settings in that browser. Consider the Netscape preferences dialog.

[Setting recognized MIME headers in Netscape]

MIME types are specified as part of the "HTTP header" which defines meta-information about the document and the session. MIME type is specified in the Content-type field.

Thus, if a web server were to send a document of MIME type...

          Content-type: application msword

the web browser would know that it should open Word and load the document.

Besides specifying the MIME type, the HTTP header also contains other important information including the following:

Header Description
Accept Specifies what types of media output the client is prepared to handle. Often, a client will specify several types of media that is can handle such as in the following case

Accept: image/gif
Accept: image/jpeg

Alternatively, a web browser might specify that it is prepared to handle "any" media using the following line:

Accept: */*

Authorization Specifies whether or not the user has permission to access a secure area.
Content-encoding Specifies that the message body is encrypted, compressed, or encoded
Content-type Indicates the media type of the information in the message body.
Content-length Specifies the number of bytes in the message body for requests with a message body
Date Indicates the date and time of a request if the request has a message body
From Provides the email address of the user using the client if it is available
If-modified-since Tells the server not to deal with the request if the document has not been modified since a given time.
Mime-version Specifies the MIME version used to generate the message body.
Pragma Contains any additional information that the client wishes to specify to the server
Referer Indicates the URL of the page from which the request was made. (This request header is misspelled in the protocol itself)
User-agent Specifies the name and version number of the web browser making the request.
Parts of an HTTP Transaction: The Request  
Besides specifying information about the file being transported, HTTP also defines the phases of a request/response interaction.

HTTP provides two primary methods to request documents: GET or POST

The foundation of HTTP/0.9 (the first implementation of the HTTP protocol) was the definition of the GET method that was used by a web browser to request a specific document.

For example, the following HTTP request would return the document "index.html" that was located in the web server's root directory called "webdocs"

          GET /webdocs/index.html CRLF

Notice that the GET request began with the GET keyword, included a document to retrieve, and ended with a carriage return and line feed combination.

If you would like, you can try making a GET request by connecting to your favorite web server and sending the GET request yourself (as if you were a web browser).

Below is a GET session I cut and pasted from a telnet window. In this case, I used telnet to contact the web server "www.extropia.com" and asked for the file "irobot.html" in the "Scripts/Columns" directory (Don't forget the two carriage returns at the end). The server responded by sending me the contents of that file (the HTML code you see).

	selena: telnet www.extropia.com 80
	Connected to www.extropia.com.
	Escape character is '^]'.
	GET /Scripts/Columns/irobot.html
	<TITLE>Hello there</TITLE>
	Hello there.  My, you are awfully good-looking to be a web 
	Connection closed by foreign host.

The beauty of web browsers of course, is that they take care of the HTTP protocol specifications so that the user only needs to enter the URL of the page they want to see. The web browser formulates the actual GET request, sends it to the web server, receives the HTML document back, and then displays the HTML document according to the HTML instructions.

Besides allowing web browsers (or you pretending to be one) to get documents from a web server, the GET method also implements a method for a web browser to send optional search parameters as well (it was used with ISINDEX HTML files originally).

Search parameters were encoded in a special way that the web server can deal with.

Encoding works like this:

The URL is differentiated from the search parameters by a question mark (?). In other words, a URL generically looks like the following:

http://www.domain.com/dir/file?search parameters

Since you may want to have multiple search parameters, the GET method specifies that parameters are differentiated by placing an ampersand sign (&) between them. Thus, the encoded URL above becomes something like the following:


Next, search parameters themselves are specified as "name/value pairs" separated by an equal sign (=) such as in the following example that sets the variable "lname" equal to "Sol" and the variable "fname" equal to "Selena":


Further, any spaces in the encoding string are replaced by plus signs (+) as in the following example:


Finally, any non-alphanumeric characters are replaced with their hexadecimal equivalents that are escaped with the percent sign (%). For example, a single quote character (') is encoded as %27 and a line break (which is a carriage return plus a line feed) is encoded as %0D%0A. Thus, we might see the following example that specifies that the variable pageName is equal to "Selena Sol's Page":


Though the GET method was very useful, a couple of serious problem remained.

First, the GET method only allowed a limited amount of data (1024 characters) to be sent as URL encoded data.

If there were too many name/value pairs, some of them would be clipped and data would get lost.

Further, since the information was sent as part of the URL, the user could see all of that data. On the one hand, that made URL's look really could see all of that data. On the one hand, that made URL's look really ugly and scary. On the other hand, it meant that the user got to see all of the inner workings of your CGI input.

This all changed with the development of HTTP/1.0.

The HTTP/1.0 protocol was developed from 1992 to 1996 in order to satisfy the need to exchange more than simple text information.

The first major change from the HTTP/0.9 specification was the use of MIME-like headers in request and response messages.

The next HTTP change was the definition of new request methods: HEAD and POST.

Let's look at both of these changes in greater depth. Under HTTP/1.0 an HTTP transaction consisted of a header followed by an empty line and then some extra data.

We have already talked about the header. The POST method of input was the other important change brought about by the introduction of HTTP/1.0.

The POST method allowed web browsers to send an unlimited amount of data to a web server by allowing them to tag it on to an HTTP request after the request headers as the message body.

Typically, the message body would be our old familiar encoded URL string after the question mark (?).

Thus, it would not be strange for a web server to get a POST request that looked something like the following:

          POST /cgi-bin/phone_book.cgi HTTP/1.0
          Referer: http://www.somedomain.com/Direcory/file.html
          User-Agent: Mozilla/1.22 (Windows: I: 32bit)
          Accept */*
          Content-type: application/x-www-form-urlencoded
          Content-length: 29


Notice that the "Content-length" request header is equal to the number of characters in the body of the request. This is important because a CGI script could easily parse through the variables in the body using the length.

Of course, as with the GET method, the user never needs to deal with the protocol itself. Instead, the browser does all the work of preparing the POST request headers and body.

So the million-dollar question is how does the browser get the name/value pairs to put into the HTTP message body?

The answer to that is HTML Forms. Remember those things from last section?

Parts of an HTTP Transaction: The Response  
HTTP also specifies a response protocol. Response codes follow the basic logic of:

Response Code Description Examples
1xx Informational Seldom used code for information to be supplied to the browser.
2xx Success The client or server successfully received the instructions and could perform the actions requested. 200 OK, 201 POST Success, 202 Request Accepted
3xx Redirection The client or server needs to do something else to finish the request. 301 Resurce has moved
4xx Client Error There has been a problem on the client side (usually syntax) 401 Unauthorized Request, 402 Payment Required for Request, 404 Resource not Found
5xx Server Error The server could not deal with the request correctly 500 Internal Server Error, 501 Method Not IMplemented

As we mentioned earlier, when a browser requests something from the server, it uses, for the most part, either the GET or POST methods of HTTP to do so.

As more and more account numbers, credit cards, and personal information begins streaming through the web, it becomes ever more important to adopt some form of data protection.

The most common form of data protection is encryption. An encrypted transmission is a transmission that contains plain text data which has been mathematically altered so as to be unreadable, but which can be transformed back to the original by a reverse mathematical algorithm.

Consider the simple encryption algorithm used in Arthur C Clarke's 2001 of letter = letter + 1 in which HAL is turned into IBM. Notice that H + 1 = I, A + 1 = B and L + 1 = M

Of course modern cryptographic algorithms are MUCH more complex and hard to solve backwards.

Public Versus Private Key Encryption  
The example above was an example of private key encryption. In this type of encryption, the sender and receiver both own a secret decoder/encoder algorithm. The sender encrypts using the algorithm and the receiver decrypts using the reverse.

There are several problems with this approach. For one, how do you exchange keys safely. If you are worried about the security of your communication, you can't just send the private key to the receiver over the network. If someone gets a hold of the private key, all communication can be decoded.

Unfortunately, using private key encryption, you must exchange the key in person.

Further, what happens when you exchange your key with someone who is less than dependable and some third party steals it?

Modern day encryption gets around these problems through the use of public key encryption.

Public key encryption is based on a certain type of mathematical algorithm that provides one way encryption/decryption. That is, a public key algorithm allows you to encrypt a message with a special key that has some very special properties....

  1. Encrypted messages can only be unencrypted with a single private key.
  2. Having both a public key and a message encoded with a public key is not realistically enough to crack the secret key.

Thus, rather than distributing your private decryption and encryption keys to trusted parties and crossing your fingers, you distribute your public key to anyone who wants it with full confidence that messages sent to you that have been encrypted with your public key will only be readable by you.

Any sender can encrypt their message with your public key and that message will only be able to be unencrypted with your private key.

[Private Key Encryption]

There are several popular of these magical algorithms for public key encryption including RSA, Diffie-Hellman, and Elliptic-Curve cryptography and several packages such as PGP to implement them within your web applications.

There is another function of private keys that should be mentioned. Private keys also have the added benefit that a bit of text which has been encrypted with the private key, can be verified through the use of the public key to have been encrypted by the holder of the private key. This is called a digital signature and can provide message authenticity because only the holder of the private key could encrypt such a message. The same method can be used to verify message integrity because a hash digest may be created by the sender representing the pre transmission file state.

Secured Transmission (SSL , HTTPS)  
SSL, an open, non-proprietary protocol designed by Netscape, is perhaps the most common way of providing encrypted transmission of data between web browsers and web servers (HTTP-S is the runner up). Built upon private key encryption technology, SSL provides data encryption, server authentication, message integrity, and client authentication for any TCP/IP connection.

For our purposes, SSL provides these services between SSL-enabled browsers and SSL-enabled servers. Fortunately, since both Netscape and Microsoft have implemented SSL, we as web developers can use SSL without much fear.

For the most part, as a developer implementing SSL will be out of your hands. Your code remains the same. All that changes is the web server that you serve your application from. When served from an SSL enabled server and directory, the browser and server will do all the work of encryption.

The browser will even let the client know that they have moved into a secure transmission mode for you. Consider the Netscape lock symbol

[Netscape in SSL Mode]

Previous | Next | Table of Contents