How could a webmaster display the current time and date on every page accessed by a client? How could she collect information about clients who were accessing her web site? How could she create a web site that was more than just an information warehouse, but a meaningful and dynamic conversation?

Certainly, HTML was great for distributing "pre-prepared" web pages on request. A client would use a web browser to contact a web server and use HTTP to ask the web server for a specific HTML document. (2) The web server would then send the requested document back to the web browser which in turn, would display the document as defined by the HTML to the client.

Pretty nifty really, and far superior to older technologies like gopher and ftp. However, the interaction between the client and the server was still extremely trivial. The server could only provide HTML documents that had been specially encoded by a webmaster, and that had been placed in certain publicly-accessible directories. The interaction between web browser and web server was pretty mind numbingly simplistic and the coolness of surfing through hyperlinks quickly became droll.

HTML fell short for anything truly "dynamic".

For example, to put the current date on every page using only HTML would require a webmaster to manually edit every file, every day. As you can imagine, this got tiring very quickly for sites with more than 5 pages. Webmasters needed a way to have HTML pages created and modified "on-the-fly," with information that could change weekly, daily, by the second, or for each and every request. And they needed those pages to be modified automatically, without their constant oversight.

As it so happens, the hardware that web server software runs on typically has quite a few resources that can be utilized to help solve these problems. Not only do servers have processing power to spare, they also have a battery of applications (such as e-mail, database manipulation, or calendaring) already installed and ripe for utilization.

• Common - CGI programs can be written with many languages. CGI can be programmed in C, C++, Java, Perl, Visual Basic, or any other language that can accept user input, process that input, and respond with output.

  Further, CGI works with many different types of systems. CGI works on Mac, NT, Windows, OS2, Amiga, UNIX, and pretty much every operating system that runs web server software.

  By the way, if you use a "platform independent" language like Perl or Java to write your CGI script, the exact same CGI script can be moved directly from one platform to another without it breaking! (4)

• Gateway - CGI's usefulness does not stem from what it can accomplish by itself, but what it can accomplish by making partnerships with other resources.

  I often see CGI as a middleman or a translator whose job it is to help more powerful resources like databases, graphics generation programs, or network applications talk to each other and work towards a common solution.

  CGI is the gateway between the lone web surfer with her trusty web browser and the vast web of computers (each with their own specific language and protocols) and computer programs (each with their own interfaces and methods of output).

  CGI translates between the language the client speaks (perhaps HTMLized English) and the multitude of languages spoken by the resources the client wants to utilize (such as SQL for relational databases). This is a crucial job, cause let me tell ya, my grandmother does not want to know how to speak SQL when she is browsing the web!

• Interface - CGI is not a language. Neither is it a program. CGI is a standard of communication, a process, an interface that provides well-defined rules for creating partnerships. The benefit is that if everyone follows the rules of the interface, then everyone can talk to each other. Thus, typically we say that we write "CGI programs" or "CGI scripts" that perform the functions of the common gateway interface.

Let's look at the CGI processes in the following chart.

Okay, so that is probably a lot of abstract stuff to take in all at once, especially if you have not worked with CGI already. So let's back up a minute and go over what CGI is by looking at it in the wild. Let's look at some examples of CGI in action.

• Processing Forms - One common use for CGI is providing a way for web surfers to enter data into HTML forms and send that data to some site administrator. Take a look at this Feedback Form Demo. Try it out! With a feedback form, the web surfer can not only read pre-prepared HTML documents, but can actually send feedback in response. Forget about doing that with HTML alone. This form processing script actually takes the data input by the web surfer and sends it as e-mail to the form administrator.

  Another common form processing application is the Guestbook that allows web surfers to leave publicly readable greetings. The CGI manages a common file on the web server that everyone can read and write to.

• Discussing Things - More than simply allowing clients to "talk back", CGI helps in creating an ongoing dialog between multiple clients. Check out this Bulletin Board System that actually lets many people collectively create an archive of related messages. And when you're done with that, check out how CGI implements Realtime Chat that allows people to chat anywhere in the world in real time (as if it were a telephone call but typed)

• Shopping - CGI has also been very useful in selling products on the web. Take a look at this Demonstration Web Store.

• Databases - Finally, CGI can be used to manage databases. Here is an example of a Database Management System for CGI. And here is a similar application used to perform Groupware Calendaring

• CGI introduces security holes. Lincoln Stein writes the following eloquent warning on the problem:

  Unfortunately, there's a lot to worry about [when running a web server with CGI]. The moment you install a Web server at your site, you've opened a window into your local network that the entire Internet can peer through. Most visitors are content to window shop, but a few will try to peek at things you don't intend for public consumption. Others, not content with looking without touching, will attempt to force the window open and crawl in.

  It's a maxim in system security circles that buggy software opens up security holes. It's a maxim in software development circles that large, complex programs contain bugs. Unfortunately, Web servers are large, complex programs that can (and in some cases have been proven to) contain security holes.

  Furthermore, the open architecture of Web servers allows arbitrary CGI scripts to be executed on the server's side of the connection in response to remote requests. Any CGI script installed at your site may contain bugs, and every such bug is a potential security hole.

  It is one thing to allow any freako on the Internet access to your web server, when the communication is controlled through the boundaries defined by HTTP and implemented by web browsers. It is another thing to allow a stranger access to an unlimited amount of applications housed on the same server through a renegade CGI script.

  In the WWW Security FAQ, Stein identifies four overlapping types of risk:

  • Private or confidential documents stored in the Web site's document tree may fall into the hands of unauthorized individuals.
  • Private or confidential information sent by the remote user to the server (such as credit card information) might be intercepted.
  • Information about the Web server's host machine might leak through, giving outsiders access to data that can potentially allow them to break into the host.
  • Bugs can allow outsiders to execute commands on the server's host machine, allowing them to modify and/or damage the system. This includes "denial of service" attacks, in which the attackers pummel the machine with so many requests that it is rendered effectively useless.

  I recommend checking out the following CGI Security sites if you are interested in getting more detailed information.

  • Writing safe CGI scripts -- an overview (Paul Phillips)
  • NCSA's tips for writing secure CGI scripts
  • Latro, a tool for identifying insecure Perl CGI installations, by Tom Christiansen
  • Are You Safe? (Keith Gardner)

• CGI is at the mercy of HTTP. It is important to note that HTTP only provides for a one-time, question/answer type of communication. After all, it was defined primarily for web browsers and web servers to exchange HTML documents. Thus, by definition, HTTP is not very dynamic.

  One-time, question/answer communication works like this: the web browser and the web server are only connected as long as it takes for the web browser to send one document request and the web server to send one requested document. If the browser wants a second document, it must recontact the server and ask again. Each request is new. The server maintains no ongoing connection or record of past exchanges.

  While this is very efficient for network traffic (because the bandwidth is only used when information needs to be exchanged), it is a big pain in the butt when it comes to CGI, because CGI is about conversations, not about one-time question/answers

  Imagine that when talking on the phone you had to hang up and redial every time you said something and received an answer. Imagine further that every time you called back you had to go over every previous exchange before you could get to the next piece. That is the way web browsers work with web servers and this makes communication tough.

  This makes communication tough for three reasons.

  First, if the client and server are to maintain information over several exchanges, the CGI must be responsible for keeping a running dictation of the conversation so that every time there is a new exchange, the web server can consult the record of the entire conversation up to that point. This is what CGI aficionados call "maintaining state". The CGI script must be able to keep track of certain information like username or the contents of a virtual shopping cart for every "instance" of a script. (6). That is, there must be a way to tie the current HTTP request to related ones that have gone on before. Maintaining state is possible with CGI using hidden variables, by encoding the URL, or by maintaining a state file on the server, it's just not easy or efficient. (7).

  Second, every set of question/answers causes the web server to execute a unique instance of the CGI script. This is pretty expensive, especially on a high volume web site that may have 100 instances of a CGI script executing at any given moment, each, perhaps, with its own Perl interpreter. (8) Every one of those CGI scripts takes a little bit of umph out of the server engine. If we were not limited to question/answer format, we would not need to execute so many instances.

  Consider the following CGI application executing....

  
      Client: Hello?
  
      Server: Welcome, what would you like (CGI script executed once)
  
      Client: I would like a list of products you are selling
  
      Server: Here is a list (another one)
  
      Client: I want to purchase this product
  
      Server: Okay.  (yep)
  
      Client: I'm done, can I check out?
  
      Server: Yes, what is your credit card number? (another script)
  
      Client: Here it is.
  
      Server: Thanks (another instance of the script that also emails the
              results to some store admin) (9)
      

  Yuck, this exchanged caused 5 instances of the store script to be executed as well as 5 Perl interpreters if the CGI script was written in Perl.

  Third, CGI is extremely slow. Every time the client does something, the CGI Script must recreate the entire dialog and execute a new request. Add a new item to a virtual shopping cart - new request. Calculate a running total - new request. Submit an order - yet another request. Each request takes time and since the CGI script must be executed again and everyone must wait for a busy Internet.

• CGI is ugly. Finally, CGI scripts produce fairly ugly user-interfaces. Basically, CGI is limited to bland HTML-based forms and whatever bells and whistles can be provided by surrounding HTML layout. Thus, no CGI application looks like your swank bootleg copy of Word.

  This may not seem like a big issue at first, but when you start competing for web hits with multi-million dollar companies, image is indeed everything. CGI simply cannot compare with web based applications that are not limited to HTML.

Well, those are some pretty damning flaws. Like I said, many systems administrators would love to see CGI fall off the face of the Earth. Unfortunately for those system administrators, the fact is that CGI has continued to be the workhorse of the web, powering 90% of the dynamic web pages out there.

The fact is that CGI, especially CGI/Perl is easy to work with and most non-technically oriented webmasters out there can get their needs filled, and filled right away. However amazingly, brand-fantasmagorically wonderful other technologies sound, they are still vaporware as far as the average web developer is concerned. Either the ISP does not provide those technologies, or the learning and development curve is too steep or expensive. And of course for small applications typical of most websites, the big guns of C or C++ are just overkill.

Client-side programming is based on the idea that the computer that the client is using to browse the web has quite a bit of CPU power sitting there doing nothing. Meanwhile, web servers are being tasked to death handling hundreds of CGI requests above and beyond their regular duties. Thus, it makes sense to share some of that burden between the client and server by taking some of the processing load off the server and giving it to the client.

As it so happens, much of what CGI does, can be handled on the client's side. Typically, the only time the server needs to be involved is when the web application needs to send email or access datafiles. Things like maintaining state, filling out forms, error checking, or performing numeric calculation, on the other hand can be handled by the client's own computer. The web browser need not check back with a CGI script every time the user wants to do something. A "script-enabled" HTML page can carry with it instructions on how to handle certain events.

In the following figure, client-side scripting has reduced server load by over 80% for every client accessing the CGI script. And of course, since most of the processing is handled locally, the application as a whole runs 5 times faster.

Obviously, this solves many of the problems posed by CGI. Client-side applications maintain security by keeping server processing to a minimum. They are not restricted by HTTP and the GUI can be as pretty and sleek as any traditional software package out there.

The two most popular languages for client-side scripting are JavaScript (Netscape Navigator) and VBScript (Microsoft's Internet Explorer). Both technologies allow web programmers to encode short program "snippets" into their HTML documents that can be executed by a web browser. JavaScript Made Easy[link has gone dead--ed.] provides several excellent examples of JavaScript in action and Reaz Hoque provides a very straight forward tutorial on JavaScript basics. On the other side of the coin, Microsoft provides a good list of samples for VBScript.

Actually, script-enabled HTML pages can be fairly dynamic and do indeed cut down on the work of the sever. Of course, in any real application, there will need to be a CGI script on the server to email results or access data, but much of the work, perhaps 75% of it, is done by the client. This can cut down server load by 80% on complex applications.

Unfortunately, script-enabled HTML pages have their problems too. The most obvious problem, of course, is that the web browser program must be able to interpret the language used for scripting. And since Netscape and Microsoft are too knuckleheaded to build upon common standards, we are left in the cold. JavaScript programs continually break when viewed using Internet Explorer and VBScripts do the same when viewed with Netscape.

Thus, client-side scripting has remained primarily useful only for limited, controlled intranets where webmasters can be sure that all users are using the same browser software to view web pages.

In syntax and execution, Java is a lot like a simplified version of C++. ("simplified" should be read in the previous sentence as "an improved"). It is a highly robust, distributed, high performance, object-oriented, multi-threaded language with all of the usual features. As such, it builds upon years of C++ development, taking the good and dispensing with the bad.

As it so happened however, Java did not make it into the consumer electronics market. Instead it wound up in our web browsers.

Java seemed to be a perfect fit for the web. The language itself was extremely small (as it was built to go inside toasters and alarm clocks with tiny amounts of memory). Thus it could quickly be transferred over the web.

Further, Java was platform independent. That is, any computer with a Java virtual machine can run a Java program. Programs can be written anywhere and be run anywhere. This is crucial because, as we saw in the case of the client-side scripting languages, if a language can not run on any machine, it cannot be used on the web that must service every machine, language, and environment imaginable.

Platform independence works because Java is an interpreted rather than a compiled language. Unlike C or C++ code, when Java is compiled, it is not compiled into platform specific machine code, but into platform independent byte code. This byte code is distributed over the web and interpreted by a virtual machine (typically built right into a web browser these days) on whichever platform it is being run. Perhaps a picture would be useful...

Thus, as a programmer, you need only concern yourself with the generic Java programming language and compile your applications into bytecode on whatever system you are using. You can then be assured that your bytecode will be executed correctly whether your clients are using Macs, Pcs, Unix boxes or anything else.

Java, of course, demands books worth of explanation and description. So, of course, we will not delve too deeply into the language here. Instead, I recommend that you browse through the resources collected at Gamelan which is the be all and end all of Java resource sites. There you can sample several Java programs yourself and see how amazing Java really is.

Of particular concern are the security restrictions built into Java such as the fact that Java programs (Java applet specifically) cannot easily write files to the local harddrive or get data from servers other than the ones they came from. While this may make the public more confident about the language (an important thing and perhaps worth the limitations it causes), it makes Java programs fairly useless for the average developer who absolutely needs such capabilities to create full featured applications.

Further, Java programs with a lot of logic take longer to download. If you went to the Gamelan site linked above and tried to run some example Java apps, you certainly found that you had to wait quite a bit for them to download.

Similarly, because the programs run on the client's machine, they do not have access to resources on the server. Thus, a Java program cannot even query simple flat-file databases located remotely without a proxy (some other program working as a helper on the server).

Finally, Java is still a new language. As such, it is plagued by all the bugs, inconsistency and incompatibilities that any new language is faced with. Though Java boasts platform independence, in reality, programs run differently from platform to platform...if they work at all. Further, though programs might be platform independent, they are not browser independent. Each browser, in fact each operating system, has its own buggy virtual machine that produces different output for the same program. Thus, when you distribute a Java program, you can never be sure exactly how it will run, or if it will be run at all.

Though the restrictions of Java are being addressed slowly, the picture looks bleak in the short term (next couple of years) for the Internet developer. Although, code signatures, and other security fixes are arriving, they will still cause complications for the average web developer with regards to centrally storing information and trusting it. Security will be a continuing thorn in our sides.

But even still, if all of the well-publicized inconveniences of Java were solved tomorrow, there would still be issues preventing the average web developer from writing all their web apps with Java. For example, not everyone has a database to program against or can afford the cost of JDBC (Java database connectivity) proxy servers. In fact, it is safe to say that "most" web developers do not have those tools to work with. Typically, Internet Service Providers do not allow customers to run servers of any kind through their account, let alone complicated database servers. Thus, in order to perform database management functions essential to many applications, the average web developer will still need to work with flat files on the server hardware...and this means CGI.

There are also issues preventing the spoiled "intranet" developer from using Java as well. For example, the JDBC standard will not necessarily help in a corporate environment in which some sort of proxy to a real database server may still be needed that can communicate across a firewall with a web server. Not only will Java be blocked by a firewall, but it cannot use standard encryption standards to provide secure, encrypted transactions.

Instead, every web developer should have at her disposal a wide array of tools to solve problems. Sometimes a server-side solution will be appropriate, other times a client-side solution will be best. Your main goal as a web developer is to develop an intuition about when to use which.

That said, I would like to suggest one combination of tools that I see as becoming extremely important for all web developers. The combination is that of CGI and Java. Consider the following Problems and Solutions...

Problem:

The average "Internet" web developer has probably already picked up Perl/CGI programming. Most have not picked up Java with the exception of being able to code GUI interfaces using various visual tools such as Symantec's Visual Cafe or Microsoft J++.

Solution:

A Java to CGI interface leverages existing Perl/CGI knowledge so that the core program logic can be located on a server while merely having to code a thin (very small and easily downloaded) GUI Java client. In addition, a developer experienced in Perl will be able to whip out 80% of their program in a short period of time using a language like Perl while leaving a mere 20% (The GUI) left to Java (A hard language for most people).

Problem:

Internet developers who do work for sites on a virtual web server or an ISP typically cannot use Sybase, Oracle, or another commercial database to store data via JDBC. Frequently, the ONLY option that these developers or consultants have is to do flat-file processing using CGI/PERL that generally has precluded the use of Java.

Solution:

A Java to CGI interface will allow applets to be created that can use flat-file databases that an average small-business can afford (free).

Problem:

Developers who have already invested a lot of time creating CGI/PERL for their site do not want to rewrite all their applications in Java.

Solution:

A Java to CGI interface will allow existing applications to be leveraged by allowing a developer to create a Java applet on top of an existing CGI script with minor modifications to make the CGI output data conducive to interpretation by the Java applet

• The Web Developer's Resource Library has an excellent HTML Primer if you need a refresher on what HTML is! But basically, HTML (Hyper Text Markup Language) is a way of describing how a web browser should display text and images. HTML codifies page layout into a series of instructions called "HTML tags". The job of Web Browser programs is to translate ugly HTML instructions into the beautiful web pages we see while surfing.

• Okay so there are quite a few jargon words in that sentence which might need quick explanations.

In this column we will use the word "client" to refer to a person who is using a "web browser" program like Netscape Navigator or Internet Explorer to display HTML documents received from a "web server".

A web server is a combination of hardware (an actual computer that stores all of the HTML files) and software (the program that listens for web browser requests and utilizes the hardware resources to fulfill those requests).

Web browsers and web servers communicate using HTTP (Hyper Text Transfer Protocol) which provides a communication standard for efficient and intelligible dialog. Essentially HTTP allows a web browser to contact a web server somewhere on the web and ask for a specific document (or resource). It also allows the server to send the requested document (or execute the resource) back to the web browser.