The web has entered another stage of growth. The next stage is bringing dynamic content to the wide audience of smaller web developers. Until recently, adding any dynamic content to a web site required a programmer and, probably administrator access to the web server. A number of options have become available to open dynamic content to a much wider range of web sites. These tools combine server side functions with a much simpler development user interface.

This trend has been accentuated by a number of common shifts on webs sites: the search of 'free page' sites for enhancements, by small web site creators to copy features of larger sites and by corporate sites to provide more utility for their customers.. Sites like GeoSites and Tripod have started to add features to allow individual web page creators to post more than just text pages. As the large sites all provide for chat and discussion areas, small web developers look for way to provide these features. Likewise, corporate web masters are being given the requirement that their sites produce business justification for their existence, this can not be done with just static web pages.

This paper will explore techniques for non-programmers to produce dynamic web content. A number of approaches will be explored and a case study will be presented for the i-Depth system. i-Depth has been running for two years and serves over 2 million dynamic functions per month. It is a fully automated system that allows web masters with an HTML skill level to add dynamic content to their web sites.

This paper will also propose a standard to be used to simplify authoring and installing CGI scripts, independent of the i-Depth system and to be issued under a GPL.

By Richard Roth, CEO
On-the-Net
Box 927, Northampton, MA 01061
[413-586-9668] ... [rich@on-the-net.com] ... [http://www.on-the-net.com/rr] ... [http://www.i-depth.com]

Short Bio: Richard Roth is a software architect and has been developing software for over 30 years and software products for 20, including the well-known MacLink and X2c. He registered his first Internet domain in 1985, for email using Unix and UUCP, and put up his first web site in 1994. He is chief architect for On-the-Net, and the i-Depth system, is advanced technical script support for Compuserve's Business Web and is chief server architect for the AlphaGraphics printshop chain. He has been writing, speaking and teaching about computers and databases since 1977, the Internet since 1994, spoke at ApacheCon (98) and helped with WWW4 (95) and WWW6 (97). He is heavily involved in Linux, including as an early advisor to Redhat, and ringmaster for the Apache Group Gui development project.

Introduction

1. Desirable Functions
2. Requirements of the mass developer audience
3. Methods of creating Dynamic Content
4. Implementations

Introduction

  First came the web itself, then Mosaic's GUI interface drove the first burst of web growth. Web sites were originally made of static web pages written in HTML.

Next came dynamic web pages, created first from custom programming and then tied to database engines or live data feeds. Dynamic content was limited to those with the technical skill it took to design, install and maintain a web site with programming. This limited dynamic content to larger web sites or the few with programming skill, technical ability and administrative authority on their web server.

Just as Mosaic and Netscape allowed a non-technical user to access the web, so for dynamic content to be widely implemented, much simpler methods must be available to web developers. This is specifically important when considering the offerings of public Internet providers and on-line services like Aol, Compuserve, and Prodigy. These allow a typical online user to put on their own web pages and number in the hundred of thousands of web sites. Geocites, Tripod and Compuserve each have over 100,000 mini-web sites.

Desirable Functions

• Status: counters, time, date
  The most common active elements are counters. Although these are really fairly useless, they are a personal measure of popularity of a web site. Other simple information items are the date and time, as well as "days left" to holidays (like Christmas), events and Y2K.

• Form forwarding: mailing, templates
  The first really useful active function on a web site is form handling. Although the results from a form do not have to modify the content of a web page, it still requires a program somewhere to process the page into useful data. In its simplest form, this consists of reformatting the http form field encoding to a readable form and emailing it to an administrator's email address. More advanced form functions can use an output template to map the form fields to a readable format and can produce email, data files or faxes.

• Interpersonal Discussions: Forums, Chats, Guestbooks
  There a number of styles of interpersonal interactions that can be placed on a web site. Chat groups are direct, real-time interaction while forums are message boards with "threading" of messages by subject. A guestbook or graffiti wall is often a vanity use like counters, but can be an easy way to allow open comments on a web site.

• Business product Ordering: order forms, shopping carts
  Once web site crosses into commercial use, no matter how small, there are a whole range of business uses, the most essential of which are ordering. Even a simple order form can be made more effective if it can handle price total calculation and shipping cost determination. Advanced ordering situations allow selecting products for purchase from a series of web pages as one browses through a web site, and collecting them into a 'shopping cart' with the final step being to place the order.

• Database functions: Page searching, data lookups
  Databases behind a web site become important once a site contains any significant amount of information. Being able to find the right web page requires a search engine having an index of all the pages on the site. Any collection of data, beyond a few hundred entries, requires an active function to search it.

• Shared information: Schedules and Calendars
  The whole use of the web begs for collaboration and the most basic form is tracking schedules and appointments. The web provides a whole new level of operation for mobile workers and is the ideal place for coordinating their activities, and making that information available to others.

• Content control and negotiation: protected and browser-driven pages
  Once shared information becomes useful on a web page, practical considerations become important. Some pages need to kept restricted to a selected audience. This is usually based on a user name and password. Different browsers do not all functions the same. Pages need to be served differently based on the the browser the visitor has or the language they are using.

Requirements of the mass developer audience

Allowing a wider range of web developers to have these desirable active functions requires that the functions be easy to use and yet reliable and secure. These goals are often in conflict yet both must be in balance for the solution to be accepted.

The only base requirement that might be assumed is a knowledge of HTML, but even this might be questioned, since newer WYSIWYG HTML editors do not even require the web site developer to be working directly with HTML.

• Interface Requirements
  The user interface, to these functions, has to meet the same goals that has made web browsers a successful front end for the web.
  • Simple user interface
    Backend functions are usually designed more like programmer API (application programming interfaces), in other words, from a technical perspective. Any technical components need to be hidden or automatically handled for the low end developer.

  • Flexible functionality
    The functions need to be robust and flexible. Even for a simple function like a counter, it needs to meet the requirement of the user: counters need to be settable and provide reporting. Developers want control over something even as simple as a counter.

  • Clear functionality
    The terms and description used for the functions and their features need to be written to the terms a web developer will understand -- not assuming a technical or programming background.

  • Transparent interface with server
    The most complex part of dynamic functions are coordination with the server. This interaction has to be buried in the interface in as transparent a way as possible. The interface needs to automatically create the HTML codes used and, perhaps even place them in the user's HTML code.
    
    

• Administrative Requirements
  The backend of a dynamic functions server must be able to support a highly loaded operation. This is significantly beyond the simple server functions that are generally provided.
  • Reliable, high performance operation
    Any dynamic content system that supports multiple accounts with multiple users has a higher loading and more structural requirements than a typical backend system.

  • Security of operation
    Both the security requirements of a open access web site plus the requirements of a client specific system need to be supported. Client accounting information needs to be separated from operating information, while the operational information also needed to be secure from corruption.

  • Control and accounting for resources
    Because such a dynamic system can be expected to support many web sites, with various levels of charges and security access, any such system needs to be designed with this in mind.

Methods of creating Dynamic Content

• Server Side
  These methods describe the major approaches to creating dynamic content, they are not mutually exclusive, nor are the complete solutions.
  • SSI/Template processors
    Uses server processed output pages, with a text file template controlling the output formatting.

  • CGI
    CGI is the most common method of interfacing a program behind a web server.

  • Server API
    Server API is a more efficient but complex technique way of running a program behind the server.

  • FASTCGI
    FASTCGI is a method that allows simpler and standard programming for persistent applications running behind a web server. Unlike CGI, it does require coding to handle multiple threads in the code itself.

  • Web-bots
    Web-bots is a method developed by MicroSoft to allow supporting dynamic content behind servers. It is a proprietary protocol that is supported using a client wizard and a server side extension.

  • Server Modules:
    Some servers, such as Apache, allow creating custom modules that are directly integrated with the server kernel. This allows high performance with relatively simple coding. It can present a number of risks to server reliability and increase resource requirements.
    
    
  • Server Special Language applets:
    By using a number of the above services, Java, perl, Tcl, or PHP applets can run directly on the server.
    
    

  CGI, Server APIs, FASTCGI and Server Java applets are all technologies that can be used to implement a dynamic content solution, not systems providing such a solution to developers.

• Client Side
  Many dynamic content functions can be implemented with a significant part of the processing on the client side. These are methods that can be used for that purpose.
  • Java applets
    Java is a programming language specifically implemented to allow downloading secure small applications that execute on the browser and communicate with the web server. Using an applet, offloads processing from the server to the browser, at the cost of a one time download time lag.

  • JavaScript
    Javascript is a simplified Java implemented as a interpreter with the program code being included as text within a web page. It can not directly communicate with the server but can be used to enhance the user experience or simplify the interface.

  • VBScript
    Similar to Javascript, VBScript is a MS script based on Basic and works only with MS Internet Explorer.
    
    

• Server Support Requirements
  These functions must be provided on the server to implement dynamic content functions. Unlike static HTML pages, this information varies between each web site supported on a server and thus provides a challenge for mirroring and load balancing.
  • State storage:
    Many dynamic content functions require state information to be maintained on the web server. Such functions as counters require the hit count be stored and updated. Shopping systems need to track the return visits by a shopper and relate it to the proper shopping cart.

  • Text and Document files
    Most basic dynamic content functions start with raw empty pages and substitute information or add information to a pages. This information may be formatted based on a page template.

  • Databases & Textbases:
    Many dynamic content functions require a database or textbase running behind a web server. Whether a database or textbase is appropriate is a function of the type of data being served. Databases are used when the data is of an accounting or tabular format, while a textbase is more appropriate to mass collections of text articles such as a magazine archive. The search methodologies of the two are very different.

  • Reporting
    The server activity must be tracked and reports must be available based on accounts and traffic. If the service is fee-based, accounting reports may be used to create billing records and perhaps even credit card transactions,

Implementations

• cgiemail: Template System
  cgiemail is a simple but powerful server side program that allow processing form results through templates to mail or save files. cgiemail uses the CGI interface to accept input from the web client browser. The template must be transferred via ftp to the web server host. This is a good example of flexible solution with no developer interface beyond a classic UNIX man page. Installing it requires a server administrator, but once done a reasonably competent HTML coder can create the templates it requires.

  Cgiemail - (http://web.mit.edu/wwwdev/cgiemail/)

• Chat Touring: Java Application
  This is an example of a class of applications that is growing rapidly using client side technologies like Java. Char Touring is a client chat system, with the client in Java and the server provided by a small number of web sites working with the original developer. Some are based on open standards, like IRC for a chat system, others are totally cutting new ground, which may or may not be nonproprietary. Because of Java's built-in delivery system, the developer interface is greatly simplified, the tradeoff being the requirement and load of Java on each browser client. Most of these systems do not have full blown support and accounting systems, although a few commercial ones, like i-Chat do.

  Chat Touring - (http://www.cs.princeton.edu/~burchard/www/interactive/chat/index.html)
  iChat - (http://www.ichat.com) (This link requires a 4.0 version browser and Shockwave)

• Digits, Lpage, LinkExchange: Dedicated Server
  The most obvious approach is to create a service based on a server dedicated to one function. Digits (counters) , Lpage, (guestbooks) and LinkExchange (banners) all do this. Since they are one function, they are moderately simple to use. What they don't provide is a robust range of functions and are limited in that only one source is available for each. LinkExchange has started to step into other functions. And in-line with the expansion directions mentioned at the start of this paper, Tripod bought Lpage after it was bought by Lycos.

  Digits - (http://www.digits.com)
  Lpage - (http://lpage.com/wguestbk/) -- Now at (http://saturn.guestworld.tripod.lycos.com/)
  LinkExchange - (http://www.linkexchange.com/)
  
  

• MicroSoft Frontpage: Proprietary Client/Server Web-bots
  MicroSoft has attacked the problem with a robust but typically proprietary solution. Frontpage has a built-in wizard technology that can directly communicate with a web server running special extensions. MS does provide a wizard tool kit to create additional wizards and web-bots, so the system is somewhat open. The system as provided is intended for intranet use and so has some areas of potential security risk and no accounting. From the perspective of an web developer who is using all MS tools, there is nothing like Frontpage web-bots. The downside is to take advantage of the technology, the web developer has to use a complete MS solution. A number of ISPs support the Frontpage server side extensions, but there a number of downsides for ISP for reasons just stated. In addition, the FP server extensions are much more reliable under NT, rather than the UNIX or Linux that many ISPs use.

  Frontpage - (http://www.microsoft.com/frontpage)
  Frontpage Server-side extensions - (http://microsoft.saltmine.com/frontpage/wpp/list/)
  
  

• i-Depth: Generalized Server Side Framework
  i-Depth is a server side system designed to work with all web developer tools, while still providing as simple and interface and all the other aspects of mass audience web developer that have been discussed here.

  The first release in Sept. 1996 was written strictly using standard CGI. It provides a web-based developer interface that guides the developer through use of each function, finally providing the developer with pre-made HTML code that only needs to be cut and pasted into their HMTL page. These are called 'Gurus', being server side wizards.
  
  The system has an internal template driven framework allowing easily supporting each function. The Guru features allow writing Gurus that guide the developer through the use of each function. The framework then provides the web developer interface, user information to the operational code, and accounting functions. The system has complete user accounting and secure file management. The system itself can be mirrored at multiple ISP's or, even, online services. A custom SDK is planned that will allow corporate intranet developers to add specialized corporate applications.
  
  User reaction has been very strong. With a minimum of word of mouth advertising, and competing with a number of free counter services, i-Depth has been growing at a steady rate and has attracted attention from some strong web developers. Many early users have remained with the system since its inception, and when users leave, it is usually because they have learned web programming to the point where they can program their own CGI functions.
  
  i-Depth - (http://www.i-depth.com)
  

Summary of Technology

Case Study of Practical Experiences and a proposed Script Specification

This section will describe the i-Depth remote hosting service, completely written in Perl, and over 24 months of live experience with the system. The service has been online since Sept., 1996 and is now getting a steady 2.5 million page views a month. The system provides a number of web functions to non-programmer webmasters. It includes a complete web based template driven authoring assist to webmaster users. It provides administrators with super user access functions, and script authors with a framework for plugging scripts into the system. This section will also propose a standard to be used to simplify authoring and installing CGI scripts, independent of the i-Depth system and to be issued under a GPL.

i-Depth was designed as web based service providing remote hosted CGI for webmaster for whom CGI is a term of mystery. It came on line in Sept., 1996, before the term "remote hosted" existed and this lead to confusion when trying to described what i-Depth provided. Since then, we have had automated trial sign-ups of 5-10 a day, with a few support emails a week. Since i-Depth is not free, and is a paid service, we can provide a guarantee of service and have had a better than 99.8% uptime. The fact that we have gathered a steady group of regular users, while free services have come and gone, and have gotten a steady stream of compliments from webmasters attest to the strength of the system.

i-Depth Frameworks

The i-Depth system provides a number of functional frameworks, to support web functions within an account based file structure. Each component plugs into a section of the framework. The framework definition allows adding new functions complete with their support gurus and related accounting and maintenance utilities.

The i-Depth framework supports 3 main components:

1. User functions
2. Web Authoring Gurus
3. Super User Accounting and Maintenance Utilities.

User Functions:

There are two groups of User functions: Standard and Gold. There are currently eleven functions in total.

The standard add-on functions are those desired by small site webmasters: counter, forum, guestbook, and the like. These functions are defined with a one or two steps but have a minimum of configuration options.

The advanced functions (called Gold) are built using a template system. Each template is primarily HTML with custom tags, similar to XML. The tags provide variable substitution, conditional logic and an extensible <TPL> tag hook to define function specific procedures. The TPL processing logic allows expanding the <TPL></TPL> contents, repeating as needed for the specific function.

Each Gold function consists of a few configuration parameter files, usually consisting of Perl assignments, a couple of scripts (4-6) and a dozen templates. When a new function is desired, a standard set of initial components is 'cloned' under a new name.

GOLD functions provided are: postcards, two types of ordering systems (calculating and shopping cart), chat and classified ads and login access control.

Web Authoring Gurus

To provide a non-programming interface for webmasters, each function has a matching Guru that provides a web interface to configure the respective function. The standard gurus are simple one page descriptions, outline oriented step-by-step, with an entry outline giving the process.

The Gold gurus are a series of buttons, each being a step leading to a second page. The second level Gold pages can be simple text editor windows (using TEXTAREA), specialized editors (eg, tables for databases), common functions (such as for generating HTML code) or special functions (such as for rebuilding the shopping cart parts database). Included in both gurus are a standard method for providing help screens, testing and general maintenance functions, for update or deleting functions.

Accounting and Maintenance Utilities.

The basic account system provides an automated sign-up function, so users can create their own new accounts, assign a unique password and enter the "member's area" with no manual assistance. There they need to complete a customer contact profile. This profile includes billing information if the i-Depth server is being run as a paid service.

i-Depth provides a "super user" back-end to allow accessing all accounts and logs. It also has a number internal functions for the i-Depth administrator to manage customer accounts based on a customer profile. This provides automated email of announcements to registered users and general monitor activity on the system. The administration area has its own documentation area for defining standard procedures. A consultant account feature allows advanced web developers that manage multiple web sites to access all their sites from one screen.

Considerations and Experience

The i-Depth system went live Sept., 1996. With a total of 14 functions, it is now serving over two million functions/page views a month to 4000 accounts. The main public server runs on a small pentium (P60, 48 meg RAM) server, running linux and apache via standard CGI and perl, that serves a total of 10 million hits a month. There are a number of virtual sites on the same machine, with i-Depth being about 35% of the load on the server. This good performance on such a small server is due to careful coding techniques and use of a very small version of perl4.

Overall Experience:

Webmasters using the system have been extremely pleased. The majority, accept the need for a small monthly fee to provide reliable service, have used the system for an extended period, some since the system went online.

Improvements considered:

The only common complaint is the request for additional functions. It takes about a week or two for a capable programmer to adapt a well written CGI script to the i-Depth script framework. It is a harder job to properly extend a script to be flexible for a broad audience without requiring code changes. Another area of improvement is providing a standard mechanism for custom scripts on a per account basis, now all scripts are in a common area, standard or gold. The guru presentation works fairly well, once the webmaster understands the basic structure of the system. A more graphic display of the flow of functions with active hot spots for the steps is being explored.

Security:

We has taken basic measures to insure that the i-Depth system was designed with careful considerations of security, but not extreme measures. We have been pleased to find that the only serious breaches occurred by some web crackers using automated tools to spam forums of groups with life styles they disagreed with. We suffered no outages and no corruption beyond the requirement of cleaning the affected forums. Improvements are being considered to alleviate the maintenance burden in such situations, and usually the solution is rolling back the specific account and function effected to the last of regular weekly backups (The backup operation backs up only modified information each run).

We do plan on making use of the perl 'opcode' controls to allow controlled execution of perl code in the template processes. This is allowed in administration templates but restricted in user templates.

Automated Support and System Enhancement Procedures:

There is no live support for the i-Depth service - online, email support only. By enhancing the function guru and the indexed technical FAQ as problems reoccur, we have held support to a couple of email messages a week, usually from beginning users, or very experienced users pushing the system's features.

It is clear the system is well used, because we get immediate messages if there are any service problems, whether due to the net, our servers (usually a Sprint or MCI problem) or update errors - often within minutes from all corners of the world.

We have developed an mirroring method for system development. We exercise improvements on a test server and then run an update script that uploads the changed scripts to the live server. The method provides for configuration differences in the systems. There is one public server and a number of special servers for custom applications.

Server Load:

Many of the functions have been kept running using an older Perl4 interpreter because it is smaller and faster. Experimentation has been done using apache's mod_perl and the system will be switched to a mix of perl5/mod_perl and C as part of the next major upgrade.

Except for accounting runs and chat, the server rarely exceeds a load factor of 8, and is usually lower. The problem with accounting runs is less load on the server than the time-out of a web browser to long tasks. A method was developed to fire off background tasks, with an email sent to the account on completion. The only major load problem occurred when another large database driven site was also on the public server and when a banner net used i-Depth to get started.

Proposal for a standard CGI script parameter specification

A constant problem for webmaster and script authors alike is the configuration of scripts to a specific web server and user account. There are a number of common parameters that must be defined and each script author develops their own scheme to address these values. This leaves webmasters to have to scrounge around the script documentation for what to setup for each script and redefine their system multiple times. A web hosting firm must create their own scheme to install each user account, by hand on the small firms and custom automated systems for larger firms.

A specification to define needed values for web scripts provides for a common set of variables, file handle, functions, objects and other information. This allows one definition of a virtual server to be sufficient for all scripts that will be installed. Experience with the i-Depth system and other scripts has allowed us to define a standard set of parameters needed by the vast majority of scripts and structure a data format that makes the required information to perl, C and other language scripts.

Specification contents:

General Information:

1. Naming conventions: Variable and URL formation rules
2. Common routines: logging, file locks.
3. Installation and Maintenance procedures.
4. Script Parameters
5. Logging and Error handling specification

Parameter types (with example values):

1. Host server information: host name, host master email, location and needed parameters for common programs (sendmail, date, mail)
2. Virtual Server common information: domain name, server port, web master email, location of server logs, security blocking information, common files and logs filenames and URLs, file locking information.
3. Script specific information: File locations and URLs, script identifier codes