The web has entered another stage of growth. The next stage is bringing dynamic content to the wide audience of smaller web developers. Until recently, adding any dynamic content to a web site required a programmer and, probably administrator access to the web server. A number of options have become available to open dynamic content to a much wider range of web sites. These tools combine server side functions with a much simpler development user interface.
This trend has been accentuated by a number of common shifts on webs sites: the search of 'free page' sites for enhancements, by small web site creators to copy features of larger sites and by corporate sites to provide more utility for their customers.. Sites like GeoSites and Tripod have started to add features to allow individual web page creators to post more than just text pages. As the large sites all provide for chat and discussion areas, small web developers look for way to provide these features. Likewise, corporate web masters are being given the requirement that their sites produce business justification for their existence, this can not be done with just static web pages.
This paper will explore techniques for non-programmers to produce dynamic web content. A number of approaches will be explored and a case study will be presented for the i-Depth system. i-Depth has been running for two years and serves over 2 million dynamic functions per month. It is a fully automated system that allows web masters with an HTML skill level to add dynamic content to their web sites.
This paper will also propose a standard to be used to simplify authoring and installing CGI scripts, independent of the i-Depth system and to be issued under a GPL.
By Richard Roth, CEO
Box 927, Northampton, MA 01061
[413-586-9668] ... [email@example.com] ... [http://www.on-the-net.com/rr] ... [http://www.i-depth.com]
Short Bio: Richard Roth is a software architect and has been developing software for over 30 years and software products for 20, including the well-known MacLink and X2c. He registered his first Internet domain in 1985, for email using Unix and UUCP, and put up his first web site in 1994. He is chief architect for On-the-Net, and the i-Depth system, is advanced technical script support for Compuserve's Business Web and is chief server architect for the AlphaGraphics printshop chain. He has been writing, speaking and teaching about computers and databases since 1977, the Internet since 1994, spoke at ApacheCon (98) and helped with WWW4 (95) and WWW6 (97). He is heavily involved in Linux, including as an early advisor to Redhat, and ringmaster for the Apache Group Gui development project.
First came the web itself, then Mosaic's GUI interface drove the first burst of web growth. Web sites were originally made of static web pages written in HTML.
Next came dynamic web pages, created first from custom programming and then tied to database engines or live data feeds. Dynamic content was limited to those with the technical skill it took to design, install and maintain a web site with programming. This limited dynamic content to larger web sites or the few with programming skill, technical ability and administrative authority on their web server.
Just as Mosaic and Netscape allowed a non-technical user to access the web, so for dynamic content to be widely implemented, much simpler methods must be available to web developers. This is specifically important when considering the offerings of public Internet providers and on-line services like Aol, Compuserve, and Prodigy. These allow a typical online user to put on their own web pages and number in the hundred of thousands of web sites. Geocites, Tripod and Compuserve each have over 100,000 mini-web sites.
Allowing a wider range of web developers to have these desirable active functions requires that the functions be easy to use and yet reliable and secure. These goals are often in conflict yet both must be in balance for the solution to be accepted.
The only base requirement that might be assumed is a knowledge of HTML, but even this might be questioned, since newer WYSIWYG HTML editors do not even require the web site developer to be working directly with HTML.
CGI, Server APIs, FASTCGI and Server Java applets are all technologies that can be used to implement a dynamic content solution, not systems providing such a solution to developers.
The first release in Sept. 1996 was written strictly using standard CGI.
It provides a web-based developer interface that guides the developer through
use of each function, finally providing the developer with pre-made HTML
code that only needs to be cut and pasted into their HMTL page. These are
called 'Gurus', being server side wizards.
The system has an internal template driven framework allowing easily supporting each function. The Guru features allow writing Gurus that guide the developer through the use of each function. The framework then provides the web developer interface, user information to the operational code, and accounting functions. The system has complete user accounting and secure file management. The system itself can be mirrored at multiple ISP's or, even, online services. A custom SDK is planned that will allow corporate intranet developers to add specialized corporate applications.
User reaction has been very strong. With a minimum of word of mouth advertising, and competing with a number of free counter services, i-Depth has been growing at a steady rate and has attracted attention from some strong web developers. Many early users have remained with the system since its inception, and when users leave, it is usually because they have learned web programming to the point where they can program their own CGI functions.
i-Depth - (http://www.i-depth.com)
Case Study of Practical Experiences and a proposed Script Specification
This section will describe the i-Depth remote hosting service, completely
written in Perl, and over 24 months of live experience with the system. The
service has been online since Sept., 1996 and is now getting a steady 2.5 million
page views a month. The system provides a number of web functions to non-programmer
webmasters. It includes a complete web based template driven authoring assist
to webmaster users. It provides administrators with super user access functions,
and script authors with a framework for plugging scripts into the system. This
section will also propose a standard to be used to simplify authoring and installing
CGI scripts, independent of the i-Depth system and to be issued under a GPL.
i-Depth was designed as web based service providing remote hosted CGI for
webmaster for whom CGI is a term of mystery. It came on line in Sept., 1996,
before the term "remote hosted" existed and this lead to confusion
when trying to described what i-Depth provided. Since then, we have had automated
trial sign-ups of 5-10 a day, with a few support emails a week. Since i-Depth
is not free, and is a paid service, we can provide a guarantee of service and
have had a better than 99.8% uptime. The fact that we have gathered a steady
group of regular users, while free services have come and gone, and have gotten
a steady stream of compliments from webmasters attest to the strength of the
The i-Depth system provides a number of functional frameworks, to support web functions within an account based file structure. Each component plugs into a section of the framework. The framework definition allows adding new functions complete with their support gurus and related accounting and maintenance utilities.
The i-Depth framework supports 3 main components:
There are two groups of User functions: Standard and Gold. There are currently eleven functions in total.
The standard add-on functions are those desired by small site webmasters:
counter, forum, guestbook, and the like. These functions are defined with a
one or two steps but have a minimum of configuration options.
The advanced functions (called Gold) are built using a template system. Each template is primarily HTML with custom tags, similar to XML. The tags provide variable substitution, conditional logic and an extensible <TPL> tag hook to define function specific procedures. The TPL processing logic allows expanding the <TPL></TPL> contents, repeating as needed for the specific function.
Each Gold function consists of a few configuration parameter files, usually consisting of Perl assignments, a couple of scripts (4-6) and a dozen templates. When a new function is desired, a standard set of initial components is 'cloned' under a new name.
GOLD functions provided are: postcards, two types of ordering systems (calculating and shopping cart), chat and classified ads and login access control.
To provide a non-programming interface for webmasters, each function has a matching Guru that provides a web interface to configure the respective function. The standard gurus are simple one page descriptions, outline oriented step-by-step, with an entry outline giving the process.
The Gold gurus are a series of buttons, each being a step leading to a second page. The second level Gold pages can be simple text editor windows (using TEXTAREA), specialized editors (eg, tables for databases), common functions (such as for generating HTML code) or special functions (such as for rebuilding the shopping cart parts database). Included in both gurus are a standard method for providing help screens, testing and general maintenance functions, for update or deleting functions.
The basic account system provides an automated sign-up function, so users can create their own new accounts, assign a unique password and enter the "member's area" with no manual assistance. There they need to complete a customer contact profile. This profile includes billing information if the i-Depth server is being run as a paid service.
i-Depth provides a "super user" back-end to allow accessing all accounts and logs. It also has a number internal functions for the i-Depth administrator to manage customer accounts based on a customer profile. This provides automated email of announcements to registered users and general monitor activity on the system. The administration area has its own documentation area for defining standard procedures. A consultant account feature allows advanced web developers that manage multiple web sites to access all their sites from one screen.
The i-Depth system went live Sept., 1996. With a total of 14 functions, it is now serving over two million functions/page views a month to 4000 accounts. The main public server runs on a small pentium (P60, 48 meg RAM) server, running linux and apache via standard CGI and perl, that serves a total of 10 million hits a month. There are a number of virtual sites on the same machine, with i-Depth being about 35% of the load on the server. This good performance on such a small server is due to careful coding techniques and use of a very small version of perl4.
Webmasters using the system have been extremely pleased. The majority, accept the need for a small monthly fee to provide reliable service, have used the system for an extended period, some since the system went online.
The only common complaint is the request for additional functions. It takes
about a week or two for a capable programmer to adapt a well written CGI script
to the i-Depth script framework. It is a harder job to properly extend a script
to be flexible for a broad audience without requiring code changes. Another
area of improvement is providing a standard mechanism for custom scripts on
a per account basis, now all scripts are in a common area, standard or gold.
The guru presentation works fairly well, once the webmaster understands the
basic structure of the system. A more graphic display of the flow of functions
with active hot spots for the steps is being explored.
We has taken basic measures to insure that the i-Depth system was designed with careful considerations of security, but not extreme measures. We have been pleased to find that the only serious breaches occurred by some web crackers using automated tools to spam forums of groups with life styles they disagreed with. We suffered no outages and no corruption beyond the requirement of cleaning the affected forums. Improvements are being considered to alleviate the maintenance burden in such situations, and usually the solution is rolling back the specific account and function effected to the last of regular weekly backups (The backup operation backs up only modified information each run).
We do plan on making use of the perl 'opcode' controls to allow controlled
execution of perl code in the template processes. This is allowed in administration
templates but restricted in user templates.
There is no live support for the i-Depth service - online, email support only. By enhancing the function guru and the indexed technical FAQ as problems reoccur, we have held support to a couple of email messages a week, usually from beginning users, or very experienced users pushing the system's features.
It is clear the system is well used, because we get immediate messages if there are any service problems, whether due to the net, our servers (usually a Sprint or MCI problem) or update errors - often within minutes from all corners of the world.
We have developed an mirroring method for system development. We exercise improvements on a test server and then run an update script that uploads the changed scripts to the live server. The method provides for configuration differences in the systems. There is one public server and a number of special servers for custom applications.
Many of the functions have been kept running using an older Perl4 interpreter because it is smaller and faster. Experimentation has been done using apache's mod_perl and the system will be switched to a mix of perl5/mod_perl and C as part of the next major upgrade.
Except for accounting runs and chat, the server rarely exceeds a load factor of 8, and is usually lower. The problem with accounting runs is less load on the server than the time-out of a web browser to long tasks. A method was developed to fire off background tasks, with an email sent to the account on completion. The only major load problem occurred when another large database driven site was also on the public server and when a banner net used i-Depth to get started.
A constant problem for webmaster and script authors alike is the configuration of scripts to a specific web server and user account. There are a number of common parameters that must be defined and each script author develops their own scheme to address these values. This leaves webmasters to have to scrounge around the script documentation for what to setup for each script and redefine their system multiple times. A web hosting firm must create their own scheme to install each user account, by hand on the small firms and custom automated systems for larger firms.
A specification to define needed values for web scripts provides for a common set of variables, file handle, functions, objects and other information. This allows one definition of a virtual server to be sufficient for all scripts that will be installed. Experience with the i-Depth system and other scripts has allowed us to define a standard set of parameters needed by the vast majority of scripts and structure a data format that makes the required information to perl, C and other language scripts.
Parameter types (with example values):