Measure Progress with Log Analysis 

Basic Vocabulary
Log analysis sounds like something an engineer might do, but on the Internet it's a tool for measuring how well a website is doing. Here are the basic concepts you need to understand:

• Log files: Record all activity on a given website. Types of logs include:
  
  
  1. Access logs record website visits. Most web server logs are kept in common log file format or can be converted to this format. This format makes it possible for statistics programs to analyze web site activity. Typically, common log entries include:
     
     
     1. Remote host name or IP number
     2. User_logname - often not implemented and replaced by "-"
     3. Authenticated_user - replaced by "-" if not an authenticated request
     4. Date and time
     5. Request from client
     6. HTTP status code returned to client
     7. Number of bytes sent
        
        
     Sample log entries:
     
     Note: This request generated four "hits" but the only significant one was the one for the HTML file: states.html. This user's browser was capable of displaying graphics so three graphics files were automatically sent: rstate.gif (the headline graphic), new.gif (an attention-getter for new entries), and redbk.gif (a red and black line). The access log used "- -" in place of the user's logon and to indicate that authentication was not required. Each requested item was successfully transmitted so the log shows a HTTP status code of "200."
     
     
  2. Error logs identify file problems such as missing files.
     
     
  3. Referrer logs list the site that a user came from before accessing a particular page.
     
     
  4. Agent logs record the type of browser or client software used to access web pages on a particular host.
     
     
• Visit: Each time a specific user accesses a webpage is considered a visit. Multiple visits by a single user reflect a high degree of interest in the site content. Frequently updated and content-rich sites generate multiple visits.
  
  
• Session: A session includes all the activities of a user during a single website visit. Website management can benefit from knowing time, length of a session and the path that a user follows within a web site.
  
  
• Hit: The access log records each attempt to retrieve a page or file from a website. These "hits" can add up to impressive numbers but are misleading. Hit statistics include not just webpages but also graphics. Watch the visitor count and ignore the hits!
  
  
• Cookie: Cookies are small data fragments left on a user's computer by a website. These "crumbs" can be used to track a complete visitor web session. For some mysterious reason, many folks are absolutely paranoid about "cookies" and see them as an alarming threat to their privacy. With the increased reliance on all forms of electronic commerce, cookies should be the least of their worries.
  
  

Web hosting should include either: 1) access to log files for your site and/or 2) a log analysis utility that generates daily, weekly, or monthly reports based on the log files. If you plan to do your own analysis, you will need to obtain log analysis software. Important considerations include:

• What platform does the software run on?
• How easy is it to install, maintain and use?
• Is it intended for system administrators or webmasters?
• Can it be customized easily?
• What does it cost? Is there an annual fee?

The most outstanding freeware log analysis program is Analog, which runs on UNIX, MAC, DOS and Windows systems. Unfortunately, it requires a bit more technical knowledge and patience than most folks can muster.

Commercial programs offer ease of use and a little bit of handholding. Although not all commercial log analysis programs offer all of the following features, they are the most common and the most desirable. Commercial programs:

Focus on the needs of management, marketing staff and webmasters—not system administrators. Group and even cross-tabulate files, visitors, etc. Offer report query engines for webmasters and others. Provide sleek graphics and tables. Compress and archive log data. Incorporate past report data or compressed logs. Translate domains, hosts, filenames, etc. into easy to understand plain English. Track how visitors got to your site by using referrer and agent logs. Produce general or targeted reports based on user-selected files or directories. Generate reports in various formats: plaintext, HTML, word processor, spreadsheet or database.

At this point, most commercial software packages offer more sizzle than steak—rarely offering more substance than the freeware log analysis programs.

However, some programs offer significant advantages in terms of time required to produce reports by automatically generating visually stunning graphics and tables that show patterns of web site usage.

To learn more about log analysis programs, visit Yahoo: Log Analysis Tools.

Counters are so popular that Yahoo devotes a whole page to them. Why are they popular? Mainly because lots of web publishers don't have access to decent log analysis programs so they use counters. To meet this need Web-Counter offers free and easy to install counters and regular usage reports.

Before installing a counter on your site's homepage, consider the following:

• Counters are mostly used on vanity websites designed by amateurs.
• A counter showing a low number could be embarrassing. It's a bit like bragging about placing last in a popularity poll.
• Counters generated by other websites can make your pages load more slowly and irritate visitors who already complain about the great World Wide Wait.

Note: Some of this material was taken from the Piper Resources article:Users and Usage.

Extreme-Tracker offers a similar service but fewer statistics. The graphic it uses is small and unobtrusive. Traffic reports are public with the free version but a non-public tracker is available starting at $6 a month.