eXtropia: the open web technology company
Technology | Support | Tutorials | Development | About Us | Users | Contact Us
Free support
 ::   Support forums
 ::   Frequently asked questions
 ::   Functional Specifications
 ::   eXtropia Tutorials
 ::   Books by eXtropia
 ::   Other books
 ::   Offsite resources
eXtropia FAQs

CGI troubleshooting FAQ
Table of Contents
What is Unix? And what is A Unix/System Prompt? (by Matt Wright)
I know my HTML downpat, but this CGI stuff is mighty confusing. What is CGI and how can I get it to work on my pages?
What is a cgi-bin? (by Matt Wright)
Do I need a cgi-bin to run [CGI] scripts? (by Matt Wright)
How do I download one of your scripts (PLEASE READ THIS BEFORE YOU EMAIL)
What is .tar and .gz and how the heck to I download your scripts
Once I unarchive and uncompress, then what? (PLEASE READ THIS BEFORE YOU EMAIL)
I can't get one of your scripts to work on my server! Can you help? (PLEASE READ THIS BEFORE YOU EMAIL)
How do you trouble shoot? (PLEASE READ THIS BEFORE YOU EMAIL)
I'm trying to run a script of yours on my server and I get a "403 Forbidden" Error.
I keep getting the error "500 Server Error. The server encountered an internal error or misconfiguration and was unable to complete your request.
I keep getting a "Document Contains no Data Error.
When I try running the script from the web, all I receive is the text of the script
The scripts work fine on my site, but they just won't mail.
Should my clients be worried about sending their credit card numbers over the net? (by Info World)
Miscellaneous stuff people sent me regarding their experiences
How do I port your scripts to NT? (by Netscape Tech Support)
How do I use Perl on a Mac?
On my NT, the img src = "script" tag only works for small images
Some browsers break when they are trying to run your scripts and others work just fine. What can I do?
You are a pretty pitiful FAQ-maker, are there any better FAQs out there?
Using mail-lib.pl the From field does not work right

Q: What is Unix? And what is A Unix/System Prompt? (by Matt Wright)

Short Answer: Unix is an operating system that is one of the most popular among net users. As more and more people are getting on you will see more and more Macintosh and Windows/DOS Environments. Unix comes in many names, some of them: HP-UX, SunOS, OSF/1, Linux, BSD, and many more.

Long Answer: All of these scripts were written and tested on machines running OSF/1 Unix and SunOS along with HP-UX. They have been successfully installed by me on all of those platforms with both the Apache and NCSA servers. When I refer to Unix I am talking about an operating system, just like DOS, Macintosh (and arguably Windows). Unix is one of the most complex operating systems, and used to be what kept the newbies clueless the most. When I refer to a system or unix prompt, I am talking about your Korn Shell or C Shell, or the many other shells out there. From there you can execute such commands as chmod, mv, cp, vi, and many other useful tools. These scripts have also been set up on other Platforms and Operating Systems

More Information:

Unix Usenet FAQ
Everything you ever wanted to know about Unix and More! These are the frequently asked questions on the Unix Usenet Newsgroups.
The Unix Reference Desk
FAQs, Applications, Programming, Humor and more. This site has lots of information about Unix.
Unix Resources
This site also houses a lot of information about Unix, including tutorials, reference and more!
Yahoo: Unix
Over 250 Links and Resources to Unix Information! If you're still looking for more information, check out this site.

Q:I know my HTML downpat, but this CGI stuff is mighty confusing. What is CGI and how can I get it to work on my pages?

Firstly, if you don't know anything about CGI, I suggest you not start with an application like the Database Manager or the Electronic Outlet. Start with an easy one like Form Processor or Guestbook and work your way up.

I'm always amazed at the amount of people who want to start with the hardest possible project without doing any research first. They ask me "I know nothing about CGI but I want to install the BBS. How do I do it?" The problem with this question is that it is impossible to answer without writing a book. The question is far too vague. As with all technical fields, the catch 22 is that you need to understand what you are doing before you are able to ask questions which can be answered.

CGI takes research and time to master. Once you have gotten the basics down however, CGI programming, especially in Perl, is very easy. However, though incredibly short, the learning curve is very, very steep. That means you should be prepared to spend a solid month studying CGI before you will be ready to do real CGI work or ask answerable questions. After that month, which can sometimes seem like a year, you should be very comfortable. But you MUST dedicate the month of hair-pulling study.

In the "Research Library" link on my scripts archive I list several books which I recommend you buy. If you are going to be doing Perl CGI, I recommend that your first book is "Learning Perl" by O'Reilley publishing. "CGI Programming With Perl in a Week" by SAMS Publishing is also a good introductory work as is "Introduction to CGI/Perl" by M&T Press. Either of these should be your "second" purchase.

You can also spend some time reading through the scripts which I have made available. I have taken great pains to comment them so that a beginner could pick them up and understand what is going on. In fact the original goal of this site was not to distribute code, but rather, to provide a place for others to learn how to program CGI applications by example. I am NOT a guru or a professional programmer. I am learing too. But hopefully the documentation of my learning process will be of use to you.

And then of course, there are the many CGI sites around the net with much valuable information that I have linked from my "Offsite Resources" page.

Here are a few thoughts of mine to help you on your way. CGI (Common Gateway Interface) is sort've like a translator. It translates the needs (questions) of a person (who we will call a "client") into a language that the computer (which we will call a "server") can undertand. Then, it translates the server's answer into a language that the client can understand.

Thus, CGI is not a programming language as much as it is an activity. Actually, you can use most if not all programming languages to create CGI applications. The most common programming languages used for CGI these days are Perl, C and Visual Basic. I use Perl because it is 1) easy to understand, 2) quick and painlesss to modify and customize 3) highly portable from operating system to operating system, 4) easy to find people to hack scripts if you are a business on the web, 5) excellent at handling strings which is very important in CGI and 6) fast and powerful enough for most CGI needs...C is an overkill most of the time.

The problem with HTML is that it is "static". That is, an HTML page is designed once and is displayed that way every time it is accessed. That is nice if you just want to shove pre defined information down the client's throat, but what if you want the client to interact in a more active way than just clicking on one "pre" designed link to another? What if the information displayed depends on their complex interaction like searching a database or ordering products from an online wherehouse.

The answer, of course, is CGI. CGI is able to listen to the client for instructions. Then, it is intelligent enough to make some decisions about what to do with those instructions, follow through with its decision by utilizing the resources of the server, and respond to the client (hopefully with the information requested). This output could be as simple as the current time or as complex as a database row in a database of 24,000 entries with 19 fields.

Whatever the case, CGI sits in between the two worlds and must mediate between them constantly. To do this job it must 1) be able to accept information from a client, 2) decide what to do with the information, 3) use the resources of the local server to do what it decides must be done, 4) accept whatever information the server sends back and 5) present that information back to the client who asked in the first place.

Step one is done for the most part by using HTML forms or URL encoded strings. Both of these methods are ways that the standards developers of the web created to pass information to a CGI. I am not going to go into this stuff because the use of forms and URL encoded strings are covered in many, many places available to you for free and as published works. Here are two online sources of information:

Step Two involves writing a program with enough logic to figure out what to do with the information submitted. This is a CGI program. As I said above, a CGI Program can be written in any language and can be used for any operating system that has a web server.

The first thing the CGI program must do is "parse" the incoming form or URL encoded data. Parsing means that the program must split incoming information into name/value pairs. That is, the CGI program creates an "associative array" which associates an administratively defined variable name with its client defined value. So if a form had two input fields,

<INPUT TYPE = "text" NAME = "name"">
<INPUT TYPE = "text" NAME = "email"">

and I filled in Selena Sol and selena@eff.org and hit the submit button, the CGI would get name = Selena Sol and email = selena@eff.org. The CGI could then use those variables and values to do things that it was designed to do.

Step Three involves using the server's resources to do something with the client defined information. Such a resource might be email. The CGI script could take the submitted name and email and then email those values to someone who is maintaining a guest log for example.

Thus, the CGI program needs to be able to talk to the server it is on and ask the server for resources that the server provides. Other examples might be encryption, data handling (keyword searching), string manipulation (substituting one value for another), or reading or writing to files.

Step Four is similar to Step Three. The program needs to be able to get the output from the server for whatever project that the program assigned to it.

Step Five involves taking the server supplied information and returning to the client a dynamically generated HTML page with the information supplied by the server.

In returning a dynamically generated HTML page, the CGI is basically doing what the server would do when it is asked for an HTML page...it simply returns to the client's web browser a file containing HTML using the standards that the server would use to do the same. Instead of sending back a pre designed HTML page, however, the CGI writes its own HTML page and sends back that.

Thus, the CGI must know 1) how to write in HTML and 2) what are the correct protocols that are used to send an HTML document back to a web browser (like Netsc ape 2.0). Number One is done by hardcoding HTML into the scripts logic and writing other logic which can be used to generate specific HTML depending on what information has been supplied by the server.

Thus, you might design logic into a database searching application which says, take a keyword submitted by the client from the web and search a local database for a database row contsaining that keyword. When you find it, output that database row as an HTML table putting each database row field into a table cell. Thus, though we cannot tell ahead of time what the information in the table is going to be, we can make the program smart enough to just fill in the HTML table with whatever information it retrieves from the server.

Part Two is done by following basic HTTP protocols, like the first text seen by the browser must be Content-type: text/html followed by two new lines. Again, a discussion of HTTP protocols are more than I want to cover here and can be found elsewhere. Fo the most part, you just have to remember that the first line outputted by your CGI script must be:

Content-type: text/html

(Notice two blank lines)

This is about all I can think of as far as introductory stuff. As I said before, do your research. CGI is not an insurmountable skill to achieve. You just need to put in the time.

Q: What is a cgi-bin? (by Matt Wright)

Short Answer: A cgi-bin is a special directory designated in the config files of a web server to allow execution of CGI scripts in specified directories.

Long Answer: The cgi-bin is used to help keep systems secure. Keeping CGI scripts limited to trusted users is necessary since CGI opens up a lot of security risks. Therefore, most web platforms allow only files inside of a cgi-bin to be executed....

Selena adds...

In most cases, you cannot put images or HTML files within a cgi-bin directory tree because any references to those documents will be considered as directoves to execute the files as scripts. Thus, if you have scripts which take advantage of HTML and IMgaes, you will need to move those HTML and Images to an HTML area in order to load them with your CGI programs.

Q: Do I need a cgi-bin to run [CGI] scripts? (by Matt Wright)

Short Answer: Yes

Long Answer: Yes and No. A cgi-bin is needed to run these scripts unless your system administrator has turned on ExecCGI, which enables .cgi extensions to be used in any directory. To find out if you can use CGI scripts, your best bet is to ask your system administrator if you can. He/She may opt to give you a cgi-bin in your directory, turn on ExecCGI or check your script and then place it in the server cgi-bin. Any of these options [are okay for cgi] scripts purposes. Just make sure you set all variables to reflect any changes you have to make to the location of files and scripts.

Q: How do I download one of your scripts

A: The first thing you should do is find the archived/compressed version of the application you are interested in. In most cases, I will will have a link called "Download the scripts as one tar file", which when chosen will download the archived scripts directly. In other cases, I will have a link like "Read through the Scrits". Usually, one of the files under that link will be something like filename.tar or filename.tar.gz. This is the link you shuold choose. More on this in a bit.

If the file you download ends in .gz, that means that I have compressed the file using gnuzip. The reason that one compresses a file is to make it more compact for download. For large files, to save time, we use compression to make the file smaller using complex mathematical algorithms. Gnuzip is the one most UNIX CGIers use. "Unzippping" this file is the first thing you will have to do and the process is extraordinatrily easy. >From the command line type the following:

gunzip filename.gz

For example, here is what it might look like from the command line:

kronos(selena) 10>ls
kronos(selena) 11>gunzip banner.tar.gz
kronos(selena) 12>ls

Notice that when you are done unzipping, the .gz extension will be gone.

If you do not use UNIX, there are many applications to unzip gnuzip files available at Shareware.com

Once you have unzipped the file, you will still probably see a .tar extension as in the example above. Similarly, you may not have seen a .gz file in the first place and only a .tar file.

Most of the applications on my site consist of many separate files/subdirectories which work together and must be placed in specific locations relative to each other (in the "directory heirarchy"). A file's position relative to the base heirarchy is essential. For example, a shopping cart script will be written to expect to see a User_cart subdirectory with some user_cart files inside them...if the user_carts are not being created there, because you have moved that subdirectory or have not even created it, the scripts will fail.

Such relative placement is assured by creating a "tar" file. Tar is a utility that gathers many files and subdirectoies into one big file which has internal references so that when you "untar" it, it will "expand" into the original file heirarchy.

Thus, I create one file which preserves the dirctory heirarchy necessary for the scripts to run.

To untar a tar file, you type the following at the command line:

tar xvfp filename.tar

Here is an example of what you should see from the command line

kronos(selena) 12>ls
kronos(selena) 13>tar xvfp banner.tar
x Banner, 0 bytes, 0 tape blocks
x Banner/nph-banner.pl, 334 bytes, 1 tape blocks
x Banner/Images, 0 bytes, 0 tape blocks
x Banner/Images/2.gif, 6916 bytes, 14 tape blocks
x Banner/Images/3.gif, 2279 bytes, 5 tape blocks
x Banner/Images/4.gif, 8028 bytes, 16 tape blocks
x Banner/Images/5.gif, 2070 bytes, 5 tape blocks
x Banner/Images/1.gif, 20798 bytes, 41 tape blocks
x Banner/frontpage.pl, 213 bytes, 1 tape blocks
kronos(selena) 14>

Now you will have a subdirectory called Banner, with 2 files and one subdirectory called Images with 5 files

If you are not working on a UNIX system, there are equivalent programs which you can use to untar an archive available at Shareware.com

Q: What are .tar and .gz and how do I download your scripts?

A: .gz means that the files were compressed using gzip, and .tar means they were archived using tar. If you don't have these programs, you can get them at Shareware.com

If you are on a UNIX sysstem, the command to unzip is:

gunzip filename.gz

And the command to untar is:

tar xvfp filename.tar

Q: Once I unarchive and uncompress, then what?

Once you have the application expanded onto your server, you will need to configure the scripts to run on your local system. Many of my applications have a .setup file which centralizes your server specific varioables and options into one place.

For the most part, you need only change these setup variables to the specifics of your local server. If there is no setup file, it should be the case that all such options and server specific variables are defined in the first lines of the main scriopt. Check there and make appropriate changes.

This is the first difficult thing for you to overcome by yourself, because there is no way for me to tell you what your own environment is like. I recommend that you try to get the distribution scripts working as they are before you start changing options and moving subdirectories or files.

You will also have to make sure that your perl interpreter is referenced correctly on the first line. If you don't know what I am talking about, read the next section on trouble shooting. There is a long section on this topic.

Finally, you will need to make sure that the permissions are set correctly for the web server to run any applications, read any data or setup files and write to any supporting files it needs to. Read the section on permissions below if you have any trouble figuring out how to do that.

Q: I can't get one of your scripts to work on my server! Can you help?

A: Selena Sol has a full time job, full time school, develops freelance CGI applications constantly, volunteers 20 hrs/wk at the Electronic Frontier Foundation and has a life to boot. Please be understanding when you ask for help installing, customizing and trouble shooting scripts. Being understanding means

  • Provide me with as much information as possible and take some time to figure out how to state the problem as clearly as possible.
  • Read through this FAQ and try to trouble shoot it on your own first.
  • Be patient

That said, please feel free to ask when you can't solve the problem on your own. I will do my utmost to help you figure out the problem as quickly as possible.

If you email me, however, there are a few things you need to know.

  • It is hard to trouble shoot scripts.
  • It is very hard to trouble shoot scripts virtually (That is, not having access to the server to try things out myself. Like auto maintenance, scripts generally require one to get there hands inside of the machine and fiddle with things. Every server has its own special quirks that can only be discovered with intuition and hands on twiddling).
  • It is near impossible to trouble shoot virtual scripts without receiving...
    • The script in question and the associated URL.
    • The HTML which calls the script, if any.
    • The "exact" eror message you receive using a web browser.
    • The "exact" error message you receive when running the scripts from the UNIX command line.

Q: How do you trouble shoot?

A: Troubleshooting scripts is a lot like fixing a motorcycle. It is all about inductively discovering the problem with a motley of diverse "clues" which will change from project to project. As a cgi developer, you must realize that every client, browser, server software, server hardware, operating system, web server configuration, security configuration, cgi-application, and web administrator is different. Thus, very few generic Public Domian (PD) scripts will work right out of the box on your system.

That is, the fact that you get a 500 Server Error after installing a PD script is not necessarily the fault of the programmer and is not necessarily something to get flustered over. Most good PD scripts, in fact, will be designed generically, so that they will work in as many environments as possible with configuration, but on no environments without (this is especially the case with complex cgi applications like a BBS or a database manager). Thus, you will need to play with all the scripts you download to configure them for the specificities of your own environment.

For example, consider the first line of any Perl script:


The above line will be mandatory for every perl cgi but, unfortunately can vary widely from system to system. For example, though I distribute my code with the first line as shown above. I have had to change that line when I am freelanceing to such values as:


The reason that this line may be different from system to system is that the systems administrator may have installed the "Perl Interpreter" in any number of directories. Thus, the first thing you must do in customizing your scripts is to find out where your local Perl interpreter is located so that you can reference it correctly on the first line.

To find the location of the perl interpreter, you can use commands such as which, wehereis, whence or find from the UNIX prompt, but a simple email to your sysadmin will probably suffice. Ask him or her "would you please tell me the absolute path of the perl interpreter" and she/he will respond with the path.

So what does the first line actually mean? Well the #! tells the server that the file is an interpreted executable and that in order to run it, it must pass the contents of the file to the "Perl interpreter (perl) which is located in the directory /usr/local/bin.

The Perl intrpreter will then translate your CGI script, which is written in English (sort've) into machine code binary which is legible only to your computer (well some people can read machine code).

Thus, if your perl is located in a different directoy, this first command will be wrong, the server will not be able to find the perl interpreter, and you will get a 500 server error from the web.

So then what? What if you change the first line to represent the local path of perl and you still get an error? My general strategy is to fiddle with things, see what happens, and let the solution...just...emerge. I'm sure that that does not help you very much right now with your boss breathing down your neck incessantly tapping his timex. But it is the truth and you should feel comfortable telling him/her "I don't know, let me play with the scripts." As I said, customizing scripts is very much an intuitive process...so time limits are very hard to define, espeically when customization involves more than just getting it to run, but involves changing its GUI or data strcutures.

Step one, of course, is always to sit back and take three deep breaths. Take the advice of Douglas Adams, "Don't Panic".

Once you are ready to begin, there are some standard customization tools you can use though. It isn't all magic...

First of all, do not trouble shoot from the web! Web error messages are completely useless for the most part. At best, you'll get some frustrating message like:

500 Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Error: HTTPd: malformed header from script /cgi-bin/web_store.cgi

As you can see, this tells you nothing about what the problem actually is.

What you need to do is to telnet in to your server and run the scripts from the command line (If your ISP only gave yuo ftp access, I'm very sorry...I recommend another ISP, CGI is going to be a really big headache). From the command line, I usually type something like the following:

perl blah_blah.cgi
or better yet, just

Which one of the above commands works will depend on your UNIX environment. If neither works, ask your sysadmin to "please put the perl interpreter in my PATH in my shell configuration setup file".

If the command ran fine, hopefully, you will not receive an error message. That is, hopefully you will see whatever it was that you wanted the script to display. For example, look at what I did here.

%perl test.cgi
Content-type: text/html

This is a successful test

However, if there is a syntax error in the code, you may get something like the following...

%perl test.cgi

String found where operator expected at test.cgi line 3, at end of line (Missing operator before "; ?) Can't find string terminator '"' anywhere before EOF at test.cgi line 3.

The error messages you receive from the command line will be much, much more useful than what you receive from the web. For example, perl will tell you which line it thinks the mistake was on. It will also do its best to diagnose the problem. In this example, I simply removed the quote mark that was on the line print "This is a test"; thus, screwing up the syntax.

Note: in "Teach Yourself CGI Programming in a Week", Hermann has an excellent section on the most common syntax errors that you will find. It is on Page 420-422 and is an easy read. To summarize, he notes the top three bugs as 1) punctuation problems 2) assignment of equality operators, and 3) string and numeric equality operators. I recommend you read this section if you can, but for most of you, who will not be designing, but will be customizing, the main problem I see is that when you edit the HTML output, you will invariably use the @ sign in an email addres or a "" in an URL. You must remember that " and @ have meaning to perl, so if you use them, you need to "escape" them with a backslash. Thus, <A HREF = "mailto:selena@eff.org">selena@eff.org</A> becomes <A HREF = \"mailto:selena\@eff.org\">selena\@eff.org</A>.

Note: Sometimes, you need to be aware that the web server will be running with a different "environment" than you. Thus, for example, it may not have permission to edit a data file that you do. If this was the case, what worked from the command line would still not work from the web. However, you would have discovered a valueable clue. Savor these clues, and use deduction to weed out possible erros until the only explanation is the only one left.

Alternatively, you may need to change the CGI extension to .cgi instead of .pl or .sh. Every sysadmin has set a different level of security as far as how CGIs can be executed. Some systems only allow you to run scripts from a special direcotry (often called cgi-bin). Others allow you to run cgis from HTML area directories with the .cgi extension (also you may have to create a .htraccess file or the equivalent for a non-NCSA-based server). It is crucial that you find out the specifics of how your sysadmin has configured the cgi executable options for your web server because, what may work from the command line, may still fail from the web. But hey...at least you have isolated the problem down further. Now you know that it is not a syntax bug.

In order to run scripts from the command line, you may very well have to "hardcode" incoming form variables since the script will be run from the command line rather than from being called by some HTML form.

Often, I will actually insert something like...

$MYDATA{'phone_number'} = "xxx"

so that I can comment out the routine which parses the form data having faked the script into thinking it was actually getting incoming form data.

Why? Reduce the routines to nil. Deal with one routine at a time, make it work right, then build. Kevin Kelly has a great discussion of emergent copmplexity based on simple routines compounded in Out of Control and it would be worthwhile to read that as a study of trouble shooting.

Also, force the script to tell you what it is doing at every point. I have it do so, by adding print lines everywhere.

print "$MYDATA{'phone_number'} = FORMDATA PHONE NUMBER";

Thus, you can make sure that the right values are getting assigned to the right variables. A way to automate this, for example, if you are using cgi-lib.pl to parse data with a call of


is to use the following code...

@incoming_form_variables = keys (%form_data);
foreach $incoming_form_variable (@incoming_form_variables)
print "$incoming_form_variable = $form_data{$incoming_form_variable}";

Thus, you will get a print out of all the variables sent in from a form as well as the values of all those variables.

Another very useful tactic is to type the line

print "test";exit;

directly after the line

print "Content-type: text/html\n\n";

Hopefully the Content-type line should be one of the very first lines. Thus, when you run the script from the web, you should receive a lone "test" in the browser window and the script will exit. Now you know that the lines up to the print "Content-type... line are all okay. So, cut the print "test";exit; line and paste it a few lines further down. Then try again from the web browser. Repeat this process until you finally find the line which causes the problem. Then sit down and think about it awhile. What are the symptoms and what are my clues?

Note: Thus, it is always best to have the content-type... line at the very beginning, not just because you want the browser to not time out, but in order to debug when necessary. not to mention that the Content-type..." flag must be printed out BEFORE ANY other text is sent to the browser or else you will get a 500 server error.

Note: beware of "if tests". You may not hit the print "test";exit; line if it is inside an if test that returns false.

Finally, if I simply cannot isolate a problem, I'll comment every damn line in the script and decomment them one by one until I hit a wall. Of course as Nick Bicanic wrote me, "however here's one for you that I'm sure others have encountered too. Something doesn't work. No amount of fixing fixes it. So you comment everythig out - then uncomment one by one....and now everything works - however nothing has been changed. Fuck that shit pisses me off."

In response, Greg Greene writes, "An answer to your FAQ about the irritating problem of scripts that don't work till you commented evry line out then uncommented them. The problem is pesky control characters caused by hitting the control key instead of the shift key that puts an unprintable (and unspeakable) hidden control character in the script. Commenting it then erasing the comment also erases the control character and voila ! the damm think works. Sometimes an editor that can show ALL 255 ascii characters on demand can save you time !! I ran into this often back in the early Benton Harbour Basic days when everything was a port from some mainframe somewhere."

When debugging scripts, especially those with associated files (libraries, data files, setup files...) you want to be very careful about setting permissions. Permissions errors are THE MOST COMMON error I see. Make sure to read the discussion of permissions below.

Yet another method for debugging is to use the perl debugger. This is a very simple process and the information that the debugger will tell you will be pretty good. Simply type the following at the command line...

perl -c programname.cgi

This will check the syntax of the program without actually executing the code...this is sometimes good when you are worried about affecting datafiles during debugging.

You can also test the scripts using telnet by "pretending" that you are a web browser. This is covered in "Teach Yourself CGI Programming with Perl in a Week" by Hermann on Page 425 and 428 or in CGI Programming on the WWW by Gundavaram on page 373-374.

You can also use the server's error log which will have yet more information about problems. This is also covered in "Teach Yourself CGI Programming with Perl in a Week" by Hermann on Page 431-432.

You can also use a debugging tool like CGI Lint as discussed in CGI Programming on the WWW by Gundavaram on page on pages 375-380.

I recommend that before you do anything else, you send the following letter to your sysadmin (the answsers to these questions will be essential for you in customizing and invaluable to those you ask for help, in fact, it is pretty frustrating for me to get help requests from people who do not provide me with this basic information):

"I would like to run CGI scripts and I have a few questions to ask you.

  • What is the absolute path of the perl interpreter which I should use? Please include both the location of Perl 4.x and 5.x if you have both.
  • What is the absolute path of sendmail...or do you have a sendmail interface that I can use?
  • What web server software do you use?
  • What Operating System do you use?
  • What are the specific security barriers you have set up regarding CGI usage?
    • Which directories may I execute CGI applications from? May I use the central cgi-bin directory, or must I create my own cgi directory within my own html area?
    • Must I use .cgi extension or another such extension for my cgi's?
    • Are there any special access files like .htaccess that I need to create in my local cgi-bin area which will allow me to run CGIs.
    • Do you use a CGIWRAPPER? If so, what is the URL that I should use to reference a cgi script and what is the "real", "absolute" path that I may use when referring to supporting files within a CGI application since the web server will not be inherritting my personal environment.
  • Is there any other information you can think of that might be crucial for me in trying to customize public domain CGI's for my web site?

Thank you very much for your time. I know you are extremely busy, but if you answer these questions, perhaps it will save you time in the long run.

Q: I'm trying to run a script of yours on my server and I get a "403 Forbidden" Error.

A:Permissions can often be a real drag. Many systems have funky automatic settings that you need to override if you are to allow the world (web browsers) to utilize your scripts. Permissions are UNIX's way of allowing or disallowing various people access to files on them. Permissions are set by using the chmod, chown and chgrp command.

To find out what permissions your files are set to, type "ls -l -g" at the UNIX command line.

You'll probably receive something like (if not, type man ls and figure out how your setup works)...

  eff.org:~/Fortune$ ls -l -g
  total 237
  -rw-rw-r--  1 selena   doc           607 Oct  9 17:14 README.txt
  drwxrwxr-x  2 selena   doc           512 Apr  6 10:05 Test/
  -rw-rw-r--  1 selena   doc          2065 Jan 29 07:37 eff_quotes
  -rw-rw-r--  1 selena   doc        104434 Oct  9 17:14 fortunes
  -rw-rw-r--  1 selena   doc        106720 Oct  9 17:14 fortunes.dat

The permissions can be seen in the part of the line which looks like -rw-rw-r-- The part is divided into four sections. directory or file?, User permissions, group permissions and world permissions. In this case you can see that Test is a directory "d" it can be read, written to and executed (that is, we can read files within that directory) by user and group "rwxrwx" and can be read and executed by world "r-x". fortunes.dat is a file "-" and can be read and written to by user and group "rw-rw" and can only be read by world "r--".

My general rule for web permissions is to begin by setting "all" files and directories to chmod 777. This is the most insecure, but if your scripts work at this level, but do not work otherwise, you have isolated the problem to a permissions problem. Then, one by one, increase the security. I usually jump to chmods 775, then 755, etc. for directories and scripts and 666, then 664, then 644, etc. for files. The idea is to narrow things down until you have the most secure setting. But, make sure that you do not leave things at 777 unless absolutely necessary (temp, user carts, and session directories, for example, will probably need to be left as 777 because the web server must write to them, but your main application filename.cgi should probably not be left as 777...actually, you may want to make it 755)!!!!!!!

Also, in Teach Yourself CGI Programming with Perl, Eric Herrmann writes "Most servers run CGI SCripts as user NOBODY. A file that "you" can access is not necessarily accessible by that user." Very true...beware. Something that runs okay from the command line may not work from the web. To avoid this, you might try telnetting to port 80 and doing GET, becomong your own browser and getting more info. Anyways, enough of that, I recommend looking at a UNIX manual for a real explanation. It is not all that hard actually.

Also, Matt Wright found this great list of links to the UNIX MAN pages on the web at Yahoo I recommend that you take some time to read these pages as permissions is the number one reason that scripts don't work on your server right out of the box.

Finally, in some cases, you most also change the permissions of the root directory itself becasue the main scripts must write to the same directory in which it is located. Make sure you go up a directory and chmod the root directory of the script you are trying to run.

Q:I keep getting the error "500 Server Error. The server encountered an internal error or misconfiguration and was unable to complete your request.

A: You need to check the sucker out from the command line, there is probably a syntax error somewhere, cause the script is not executing.

Possible errors include...

  • Your Perl path on the first line of the script is wrong.
  • You have syntax errors. Have you included illegal characters in a print statement? Common ones are " and @. If you've changed some HTML output to be personalized for your site, I'd bet that is what happened. BTW, in many of my scripts, I use the print "<< end_of_html"; method to prevent the need to escape those chracters...but that is only good for long blocks of HTML really. You still need to know about escaping bad characters.
  • Are you sure that you gave the server the right script name in your form tag?
  • Does the script have the right permissions?

Here is a installation report from Scott Barkey which details errors which may occur when transferring scripts via ftp...

"This is the description of the problem I was having while trying to install the db_manager.cgi script. I hope this can help another.

First of all I downloaded the db_manager_scripts form your site then I made the changes in the db_manager.setup file and ftp'd the files to my server from my computer. I then ran telnet and ran chmod +x db_manager.cgi and then ran the file in my browser, after getting a very general 500 server error from netscape, I ran the script from telnet using perl db_manager.cgi and got the error...

% perl db_manager.cgi
Can't find string terminator " end_of_html" anywhere before EOF at db_manager.cgi line 132.

Well the moral of the story is that after many hours of torment and a couple of emails to selena trying to figure out what I did wrong, I discovered that when I ftp'd the files to my server the ftp program was in binary transfer mode instead of ascii format. After changing this everything was well with the world."

Here is another one from Chris Edmunds
I have run into this problem before (when printing a text file and having it bomb at the printer). You can use something like (cat -v file.txt >newfile.txt) to display non-printable characters - and then do a (diff file.txt newfile.txt) to see if that was the problem.

You can also search for control characters in an editor - for example using vi in command mode to remove all control C's you would type: :g/^C/s///g (in order to get ^C you would have to type control v, then control C

Here is another one from Jeff Wilkinson

BTW, you might mention in your FAQ that a common problem is using WinZip or a similar program to unzip and having the "Tar file Smart CR/LF Conversion" ON. This converts to PC-style CR/LF's. If you then upld that file and try to execute it on a unix box, it will not work. Took me a while to find/fix that one.

Q: I keep getting a "Document Contains No Data" Error.

A: My guess is that somewhere along the way you have misnamed a variable and the typo is causing the script not to find the file it was asked to find. Maybe it is a file you are opening or one you are requiring. It could also be a permissions problem. The main idea you should realize is that there is not a syntax problem...it is that the file is not outputting anything to the web...so somewhere along the way, your print statements are coming up empty. What you should do is go into the script and isolate the output lines and figure out why they are not getting anything to output.

Drea Leed writes, "The most common reason that I get the no data box is because the directory I'm dealing with isn't writeable by the httpd daemon. Three different scripts I had this problem with; the program writes to a tmp file, the tmp file is changed to the working file, and if the tmp file can't be created than the whole thing goes spla."

This is great advice. I would say that 60% of the file contains no data help requests I get are due to this. Not only do you have to worry about permissions in the root directory of the application you are trying to install, but you need to worry about the permissions of the root directory itself.

The other 40% of these occurances are due to the fact that you are probably using a virtual server which has created an alias for your cgi-bin with some program like cgiwarp in order to maintain greater security. Thus, the "real" present working directory that you see is not the same as the present working directory that the web servre sees. Thus, when you tell a script that it's data file is located in "./Data" and your environment is set to read as /selena/cgi-bin, when you run the script from the commandline, it will see the correct path of "/selena/cgi-bin/Data/" and all will be fine.

However, the webserver may be running with a different environment (probably one at a more "root" level. What it sees could be something like "/usr/local/www/home/selena/" thus, when it sees reference to a ".Data", it may look for "/usr/local/www/home/selena/Data" and not "/usr/local/www/home/selena/cgi-bin/Data/".

What you need to do is find out what the web server is seeing as its present working directory. That way, you can refernce all files absolutely. I do not yet know how tyo do this internally to Perl, so I can only give you a UNIX solution (although some service providers might disable this option too).

Try putting the following script in the same directory as the file you are trying to customize:

print "Content-type: text/html\n\n";
$present_working_directory = `pwd`;
print "$present_working_directory";

Then run that from the web. Hopefully, you will see the directory that your web server thinks it is in. Then you can modify all of tyou server specific variables in the setup file or wherever they are located and go on your way.

This in from Advanced Business Systems

Document contains no data:

There may be serveral causes of this error.

  1. This is the MAJOR reason. The server has run out of threads, and simply terminates the connection with no warning message. The client then sees this as no data returned. Because the sendmail is called so many times (over and over again) the server simply runs out of threads. Providers are limiting the total threads one can have as a way to get around the "unlimimited hits/download" thing.
  2. The client has cached the document, and is empty. It therefore sees the cached document, and the server does not get asked for the document again. You need to remove the empty document from the cache (this step is client dependent).
  3. The client is going through a proxy/gateway server, which is refusing the requests, or malfunctioning. This causes the client to see an empty document. Use a different proxy server, or don't use a proxy server if you have direct access to the server. Something for you to try:
I have found it was my provider. They were not allowing enough threads. Try this from telnet. Remember to use a phony *.list. I have set this to Private.list. I found I can run this from telnet with no problem. I think it is because access is different that allows me to do it this way. If I run the same script with my browser, I will get the "document contains no data" error message.



# where's sendmail located?
$sendmail = "/usr/lib/sendmail -t";

$Email_From = "you\@domain.com";

# Now set up the database you wish to send from
$database = "./Databases/Private.list";

# Let's try it out

    open (DATABASE, "$database") || &Exit ("I am sorry, but I was not
        able to open the data file in the Do a Mass Mailing routine.  The
        value I  have is $database.  Would you check the path and

    @lines = ;
    chop (@lines);
    close (DATABASE);

    @emails = @lines;

$counting = 1;
  foreach (@emails) {
      ($recipient_email, $recipient_name) = split (/\|/, $_);

if ($recipient_name ne "") {
    $to = "Hello $recipient_name";
  } else {
    $to = "Hello <$recipient_email>";

open(MAIL,"|$sendmail") || &Exit("Could not execute \"$sendmail\"\n");

print MAIL <<"TAG";
From: $Email_From
To: $recipient_email
Subject: Testing
Errors-To: $Email_From


This is just a test!

------------------- End of Document -------------------

print "$counting) Sent To: $recipient_name $recipient_email\n";
$counting ++;


sub Exit {
    local($errorheader) = shift(@_);
    print "It looks like the mailer died...";

Q: When I try running the script from the web, all I receive is the text of the script

A: Many servers are configured to execute perl scripts only from the root cgi-bin directory. Usually, however, they have allowed you to run a script if it has a .cgi suffix and if you have some access file set like .htaccess for NCSA. You'll need to check with your sysadmin about how your particular system is setup, but you can at least try changing the script from .pl to .cgi. You might also make sure that you have set the script to be executable...chmod 775. Here are a couple of sample .htaccess lines to give you an example of the kind of things you might be looking for.

AddType: application/x-httpd-cgi .cgi
Options ExecCGI.

The .htaccess file is defined in the access.conf file in the conf directory on NCSA and will vary from server to server.

Q: The scripts work fine on my site, they just won't mail

A:The most likely reason for this is that you are not calling the mail program correctly. If you are using mail-lib.pl, the most probable reason is that you did not change the variable which defines the location of the sendmail program for your local server. On line 42 is the following command:

$mail_program = "/usr/lib/sendmail -t -n";

You need to change the path to the correct value for your local server.

Q: Miscellaneous things people wrote me that I thought was useful

Regarding the Groupware Calendar 2.0, Jonathan K. Cohen wrote,

"I had to replace all your instances of die; at the end of routines (as opposed to filetests) with exit(0);. die; caused an instantaneous and inelegant abort in all cases."

Selena notes:

I have since stopped using die, but this was a nice thing to learn.

Jonathan also wrote, "Paths should be absolute where possible. Relative paths (./) led to problems."

Q: How do I port your scripts to NT? (By Netscape Tech Support)

A:If you are having trouble getting Perl scripts to work as CGI on Windows get a simple CGI batch file working on your system first. It is wise to be sure that a simple batch file works first before you go through the extra steps required to set up a Perl script.

Also be sure to download the NT version of the application you want!

And read about tar files and Windows/Macintosh.

To run Perl scripts on Windows NT, you will need to download and install a Perl interpreter program, since Perl itself does not come with either the Windows NT operating system or the Netscape server software. You can find an Intel version of a Windows NT Perl interpreter at "ftp://ftp.intergraph.com/pub/win32/perl", and an Alpha version of it at "ftp://ftp.garply.com/pub/pc/nt/alpha/ntperla.zip". (Try using a web search utility to see if there are newer versions of these programs available.)

To run a Perl script as CGI, set up a batch file which calls it. In other words, if "MYCGI.PL" is your Perl CGI script, then create "MYCGI.BAT" which contains only these two lines (or the equivalent):


and then call it from your page page like this:

<A HREF = "/cgi-bin/mycgi.bat">My Perl Script</A>

If you want to run this as a NPH script (non-parsed headers), then name your batch file "NPH-MYCGI.BAT" so that the web server knows it's calling a NPH script.

and stdin properly, so you can use it as a form handler for a POSTed form, but arguments from the URL will not be passed to it, so you can not use it as a form handler whose METHOD is "GET". It is very dangerous to allow URL's to pass arguments directly (or indirectly) to arbitrary commands on your system.

anywhere else where it could be accessed and run from a URL, because that would let people pass arbitrary parameters to it which could wreak havoc with your system.

Be warned that due to the way the Netscape web server passes batch files to the command shell on Windows NT, there is a security hole whereby a knowledgeable user will be able to use it to execute arbitrary commands on your system. (This security hole also occurs with other web servers, such as O'Reilly's "Website" server.) Because of this, you should NOT use batch scripts as CGI on your web server unless you're testing something temporarily (such as getting Perl to work in the first place, as above), or you're certain that your web server will be safe from malicious users (this is helped by being behind a firewall or on a small, restricted network). To get around the security hole entirely, you should make all of your CGI programs be compiled EXE programs instead, since EXE files are not subject to security hole (EXE files do not need to be passed to a command shell in order to be run). You can have a very simple EXE program call your Perl script safely.

Q: On my NT, the img src = "script" tag only works for small images

A: Marlin sent the following help. Recently I ran into a problem with an NT/Perl script that others might encounter. The goal was to have Perl place a graphic on an HTML page by responding to an <img border=2 ></a>tag. The tag for the Perl call looks like this:

<img src="...cgi-bin/placepic.pl" border=2 >

The code (Perl) that grabs a image and sends it to the browser looks like this on a Unix machine:

print "Content-type: image/gif";
print "\n\n";
open (IMAGE, $adpic) || print "can't open image file";
read (IMAGE, $buffer, -s $adpic);
print $buffer;
close (IMAGE);

When this is used on the NT nothing gets returned that the browser can recognize; you see a standard Netscape "missing picture." The missing ingredient for NT is "binmode." Both the read in and the send out must be specified "binary." Do this by adding the lines shown below:

print "Content-type: image/gif";
print "\n\n";
open (IMAGE, $adpic) || print "can't open image file";
binmode (IMAGE); #ADDED FOR NT
read (IMAGE, $buffer, -s $adpic);
print $buffer;
close (IMAGE);

Q: Some browsers bvreak when they are trtying to run your scripts and others work just fine. What can I do?

A: As it turns out, the latest version of Internet Excplorer has some serious bugs when it comes to handling CGI. Apparently, for example, it does not deal well with the ampersand sign (&) which is an important CGI tool for passing URL encoded data. I have heard that Microsoft is aware of the problem and is going to fix the bug in the next version of IE. In the meantime, there is not much CGI programmers can do. The following is information provided by Jeff Gordon, about netscape 3.0beta and Web Explorer...maybe this will help you...

I've just run around on the Net, with -both- OS/2's WebExplorer and the new beta OS/2 Netscape browsers. The two exhibit -very- different behaviors that might be important to your scripts.

First: trying to write a reply to a previously posted BBS message, I can crash the Netscape beta, every time without fail. It reports it's unable to load in text from the earlier message (though WebExplorer has no trouble).

Second, though: WebExplorer has -failed- to recognize a link at one location, that Netscape sees. At the front page of the Webstore/Outlet that's installed at www.heartnsoul.com, there's a simple link to take folks back to the home page of the site. Netscape sees it and can activate it; WebExplorer doesn't see that link as anything other than a call to the web_store.cgi script, and can't activate the link, though the rest of the page seems to work correctly.

Third: WebExplorer doesn't wrap the elaborate shopping cart items display, so that the items appear in columns under their headings; everything stays on one line, extending to the right past the edge of the viewing window. This has the secondary effect of causing the "change quantity" entry fields to appear tiny, only one character wide.

Fourth and finally: WebExplorer is drawing much larger entry fields than Netscape is doing. The field for Logon name in the BBS, for example, extends essentially all the way across the screen in WebEx, but is maybe one-sixth of the screen in Netscape.

Q: You are a pretty pitiful FAQ-maker, are there any better FAQs out there?

A: Yes, please try Submit FAQ questions and answers.

Q: When using mail-lib.pl the From field does not work right

A: From Joshua Gerth

...In particular I have gone through and replace most of my calls to blat.exe (sendmail for NT) with calls to your mail-lib.pl sendmail libraries. However, my only difficulty was that ( if I remember ) the real_send_mail from your script did not produce a "To:" or "From:" in the body of the sendmail text.

I did a pretty basic hack to your script which looks for any comments inside the to and from:

Strips out the comment for the actual
Mail From:
Then, just after "DATA" is printed and just before the body of the message is printed prints:
print S "DATA\n";
$buf = read_sock(S, 6);
print S "From: $fromname\n";
print S "To: $touser\n";
print S $messagebody . "\n";
This way when the message is received it has a friendly "from" and "to". If you are interested in seeing the hack I would be more then happy to send it to you, otherwise you can completely ignore this message.