So now you have seen how user requests are gathered by the GUI Layer and how they are sent across the wire to a web server by the Communication Layer. So now what happens?

Well, in 99.9% of web applications on the other end of the wire from the web browser is a web server. The web server is the entry point to the Middleware Layer.

The purpose of the Middleware Layer is to accept incoming requests and process them, using the resources provided by the web server, the machine that the web server runs on, or by the network of servers and resources that the web server is connected to.

Consider the most basic function of a web server: distributing HTML files. In this case, a web browser requests a given HTML file from the web server. In response, the web server will find the given file on the local (or networked) file system and send it back to the browser.

Of course, the web server has at its disposals a whole world of resources above and beyond file systems. Web servers, in fact, are in close contact with all sorts of resources including data stores, applications, business objects, operating system resources, authentication services (server-based or directory services like LDAP/ADSI) and more.

A good Middleware Layer web application developer will have her fingers in every pie in an organization and will know what resources she can rip off from the work of other developers.

Of course, the next question is how.

You can think of a CGI script as taking the place of the HTML file in the standard web server activity. However, the key difference is dynamism. Rather than containing a pre-defined set of text (like an HTML document) which can only change when the author edits it, a CGI script can dynamically generate any information it is programmed to generate.

A simple example would be a CGI-generated clock that would always show the current time when it is loaded. To do the same thing with HTML, an HTML author would have to edit the HTML document every second with the new time.

A CGI script, on the other hand, can access the time/date resources of the operating system it runs on and independently output the current time whenever it is called without the author ever having to do anything.

If you know Perl, here is the code for that CGI script:

           #!/usr/local/bin/perl
           use Time::localtime;
           print "Content-type:text/html\n\n";
           my $time = localtime;
           print "Today is: " . 
                  ($time->year() + 1900) . "/" . 
                   $time->mon() . "/" . 
                   $time->mday();

Of course, CGI is not a programming language. CGI is an "interface". It defines a way of bridging the web server to the back end resources. It does not say anything about how that bridging will be implemented. In fact, CGI applications can be written in just about any programming language in use today.

So why exactly do most web developers choose to use the programming language Perl for their CGI applications? Could one use another language like C, C++, Apple Script or Visual Basic instead?

This is a good and extremely frequently asked question. In fact, CGI applications can be written in any programming language that is able to accept and process input, and that is able to output the results of that processing.

Fortunately however, because Perl is specialized, it does the jobs it sets out to do exceptionally well. And, better yet, the limited set of problems that Perl can solve happens to fit very well with the demands of CGI.

Perl and CGI are simply a match made in heaven. Common Gateway Interface (CGI), as its name implies, provides a "gateway" between a human user with unexpected and complex needs, and a powerful, command/logic oriented server. As a gateway, CGI must perform one task very well. It must translate.

All CGI applications must translate the needs of clients into server requests, and translate server replies into meaningful and well-presented "answers". This can be quite a chore, since computers and humans typically speak very different languages.

As such, CGI must be adept at manipulating text (be it the text input by the client or the text received from the server). A CGI application must be able to take "strings" of data and translate them from one language to another constantly and quickly.

As it so happens, Perl has a wide variety of tools designed to manipulate strings of data. It is, in fact, one of the best languages around for string manipulation.

What's more, CGI must serve as gateway not just for one client and one server but for as many types of clients and as many types of servers as possible. It must be more than a bilingual translator, it must be multilingual, providing translation services between dozens of browser types, server types and operating systems.

Again Perl shines. Fortunately, Perl is highly portable. Due to the hard work and good intentions of many net hackers, Perl has been ported to just about every operating system you would want to run a Web server on.

Finally, Perl's weaknesses are not so negative in a web environment.

Since Perl is simple in design, it is also easy to modify, maintain and customize (which is really where the cost of software comes from anyway). That is, because Perl source code is so legible, it is very easy for one to pick up a script and quickly modify it to solve similar or new problems. Perl is a cut and paste language and program logic is easily transferred and manipulated between projects.

The benefit of this, of course, is that Perl is supported by a wide body of CGI freelance programmers. Unlike the more cryptic languages like C++ or Visual Basic, Perl is accessible to anthropology majors and computer science majors alike. In fact, newsgroups like comp.lang.perl are often too prolific to frequent on a regular basis. The Perl community is thriving and thanks to the web, expanding rapidly.

What's worse, if you are using a web server into which the Perl interpreter has not already been embedded, you will need to load the Perl interpreter every time. If you begin to get thousands of requests per second, this could quickly cause your web services to grind to a halt.

One way to get around this problem is to embed the processing into the web server itself. Rather than rely on another layer, most web servers provide several ways to extend the web server itself; to add logic and processing power.

The earliest technology to take advantage of this idea was SSI (Server Side Includes). The concept of SSI is simple. An application developer codes special tags into her HTML document. Those special tags are understood by the web server and can be translated on the fly by the web server as the HTML document passes through on its way to the browser.

WARNING: NCSA notes that having the server parse documents is "a double edged sword. It can be costly for heavily loaded servers to perform parsing of files while sending them. Further, it can be considered a security risk to have average users executing commands as the server's User."

All SSI directives are formatted as SGML comments within an HTML document and thus looks something like the following:

          <!--#command tag1="value1" tag2="value2" -->

There are several possible commands that you can use including the following:

For example, to display the current date on an HTML page (if your web server us running on UNIX) you might have something like:

          <HTML>
          <HEAD>
          <TITLE>Date Test</TITLE>
          </HEAD>
          <BODY>
          The date is:
          <!--#exec cmd = "date" -->
          </BODY>
          </HTML>

The user would never see the tag because the web server would dynamically perform the work and the substitution before it went out.

They also provided a huge number of formatting options as well including complex tabular display.

For example, what happens if you need to define operations not supported by the operating system (SSI-traditional) or the custom web server (SSI- Cold Fusion style)? What happens when you need to code your own tag logic?

What developers need is a way to embed dynamically interpreted code into HTML that can be processed by the web server on demand. That way every web site can develop its own set of custom tags.

What was needed was a hybrid CGI-SSI animal.

And thus was born ASP. ASP is a server extension of the IIS web server released by Microsoft (By the way, Apache does have MOD_ASP at this stage so that you can code ASP pages on Apache web servers. Also third party vendors also provide ASP functionality for non-IIS servers.). ASP allows developers to code custom tags in JavaScript (JScript) or VBScript. These tags can be interpreted by IIS before the pages are sent out.

An ASP page at its core is simply a text file that has been named using the extension .asp and which contains HTML and scripting. Scripting, usually in VBScript provides a means to embed programmatic logic into HTML files that will be dynamically interpreted as the HTML page goes through the web server and also provides access to any server side object.

Like SSI, ASP provides a means to specify a "tag" with instructions that should be interpreted by the web server. However, unlike SSI, ASP has a robust set of objects that you can use to do serious programming. It also gives you the ability to instantiate server side resources (any COM component). Consider the following simple ASP page:

          <HTML>
          <HEAD>
          <TITLE>Test ASP</TITLE>
          </HEAD>
          <BODY>
          <% @Language = "VBScript" %>
          <% Response.Write("Hello cyberspace") %>
          </BODY>
          </HTML>

In the above example, you saw the "Response" object being used to print a message out to the web browser. ASP has a whole set of objects for the convenience of the application developer . These objects conveniently cover all key aspects of creating dynamic web pages.

The basic object heirarchy builds off the Scripting Context Object (which you will never really use itself) and looks something like the following:

	Scripting Context
		Request
		Response
		Application
		Session
		Server

Using COM, (or its web catch phrase alias, Active-X) a developer can bring the entire power of Microsoft to bear in any web application. You can instantiate IE to parse your XML, ask Excel to output dynamic graphs, or tell Outlook to send email for you. Everything in Microsoft is an object and everything can be spoken to using a standard interface from your web page.

COM works by creating objects that have a standard interface such that they can be used by any COM-aware program.

However, to understand COM, it is best to step back and look at the history of the Microsoft architecture.

Back in the 80s, the Microsoft architecture was application-centric. That is, every application in the Microsoft universe worked independently. As a result, each application saved its data in its own special format.

For example, you could not read a Word Perfect document in Microsoft Word. Each word processor program worked in its own way, and exchanging data between them was problematic if at all possible. Users had to rely on special bridge programs or klunky export and import functions that often did a mediocre job.

Consider how hard it used to be to import a spreadsheet into a word processor. You would probably have to export the spreadsheet as plain text, import it into a word processor, and then type in the tabs yourself to make the data presentable. Worse yet, not only did you lose any cool features like column addition, but if you changed any data in the spreadsheet, you would have to go through the same painful process of conversion.

Microsoft was never far behind market demand of course, and quickly modified their architecture towards data-centricity and away from application-centricity.

In a data centric universe, instead of focussing on applications and files specific to those applications, users could think of "documents". Documents could contain any type of "object" including text, sound, animation, spreadsheets, or even types that did not yet exist.

Data centricity requires several things of applications that deal with documents. Specifically, there needs to be a way for applications to:

1. display objects with structures unknown to the application.
2. load and save documents containing objects with structures unknown to the application
3. provide editing functionality for objects with structures unknown to the application.
4. execute commands that manipulate objects with structures unknown to the application.
5. support the drag and drop of objects with structures unknown to the application.

In a perfect data-centric world, users would never again need to worry about applications and application specific files. When users opened a document, the operating system would automatically run the associated application in order to present the requested object.

Any document might include several embedded objects from completely separate applications. Sounds familiar? This is the Web? In fact "Active-X" is simply Microsoft's implementation of the data-centric architectural model.

Of course, the evolution of Active-X has been a long process.

In fact, Active-X was born many years ago in the guise of Dynamic Data Exchange (DDE). DDE was a Microsoft technology that 1) allowed applications to communicate with each other (exchange data) and 2) provided a means for applicaitons to execute commands in other applications.

Unfortunately, DDE was slow, difficult to use, and pretty limited.

Fairly soon after DDE was released, Microsoft rendered it obsolete with OLE 1.0 (Object Linking and Embedding). OLE 1.0 defined the essential "compound document" and specified the standard way for an application to work with a compound document.

Any application could display a document consisting of many different types of objects and you could double click on the object in order to edit it using the native application.

Of course OLE was only a first step. real data-centric architecture was approached with OLE 2.0 that was based on COM objects.

COM objects provided:

1. A common way for apps to access and perform operations on objects
2. A mechanism for keeping track of whether an object is in use and deleting it if it is no longer needed
3. A standard error reporting mechanism and set of error codes and values
4. A mechanism for apps to exchange objects
5. A way to identify objects and associate objects with apps that understand how these objects are implemented.

There are two primary ways to do this: 1)Java Servers and 2) Java Servlets.

Implementing a server in Java is fairly trivial since the language was designed with the network in mind. Essentially you create sockets for each client and then process the data coming in through the socket.

On a very simplistic level, you might have something like:

	try {
		ServerSocket socket = new ServerSocket(1969);
		Socket s = socket.accept;
		PrintStream ps = new printStream(s.getOutputStream());
		ps.println("Hello Cyberspace");
		ps.flush;
		}
	catch (IOException e) {
		System.out.println(e);
		}

Once your server is written, you can start it up and it should wait patiently for clients to connect to it. Connection code might be as simple as the following:

	try {
		Socket s = new Socket("www.mydomain.com", 1969);
		DataInputStream dis = new DataInputStream(s.getInputStream());
		String serverResponse = dis.readLine();
		System.out.println(serverResponse);
		}
	catch (IOException e) {
		System.out.println(e);
		}

However, most people actually implement server-side Java with Servlets. Servlets allow you to embed a Java server service through the web server. You can think of them as server-side applets. The web server will load and execute them the same way a browser would an applet.

The lifecycle of a servlet execution would be something like

1. Web browser makes an HTTP request, specifying the servlet to be used to handle the request
2. The web server passes off the request to the servlet. If the servlet has not already been loaded into memory, the web server will do so (that way future requests can be handled immediately).
3. The servlet responds by sending data back to the client via the web server.

As you can see, this works much like CGI and ASP. In fact, like ASP, or perl embedded CGI, servlets work very fast because they are loaded into the web server.

Servlets (like Servers) have the extra benefit in that since they run on the server side, security and communications restrictions typical of Java applet programming do not apply. Servlets are freed of the Java sandbox.

Servlets can also maintain state, are platform independent, and extremely extensible as they are written in object oriented Java.