| AWStats 7.8 log file parser documentation |
- I. Configuration: Installation and Configuration
- II. Process logs: creating/updating a statistics database
- III. Run Reports: Create and read reports
I. Setup: Installation and configuration with awstats_configure.pl
A) Configuration for an Apache or compatible web server (on Unix/Linux, Windows, MacOS...)* Step 1:
(If you're using a package that comes with a Linux distribution or a Windows installer, step 1 may already be done. If you don't know, you can do this step again.)
After downloading and extracting the AWStats package, you need to run the awstats_configure.pl script to perform various configuration actions. You can find it on AWStatsToolDirectory (when using the Windows installer, the script starts automatically):
perl awstats_configure.pl
- This is what the script does/asks (you can do all these steps manually instead of running awstats_configure.pl if you prefer):
A) awstats_configure.pl tries to determine your current log format from the Apache web server configuration file httpd.conf (it will ask for the path if it doesn't find it).togetherlog, awstats_configure.pl suggests changing it toCombined NCSA/XLF/ELF(You can use your own custom log format, but this predefined log format is usually the best choice and makes setup easier.)
If you answer yes, awstats_configure.pl will change yourshttpd.conf, which changes the following statement:
von
CustomLog /yourlogpath/yourlogfile comum
for
CustomLog /yourlogpath/yourlogfile combined
For more information about this directive, see the Apache manual (may be installed on your server at www.mysite.com/manual).
B) awstats_configure.pl adds the following directives to your Apache configuration file if not already present (note that the path "/usr/local/awstats/wwwroot" may differ depending on the distribution or operating system:
#
# Directives to add to the Apache conf file to allow AWStats to be used as a CGI.
# Note that the path "/usr/local/awstats/" must reflect your AWStats installation path.
#
Alias /awstatsclasses „/usr/local/awstats/wwwroot/classes/“
Aliase /awstatscss "/usr/local/awstats/wwwroot/css/"
Aliases /awstatsicons "/usr/local/awstats/wwwroot/icon/"
ScriptAlias/awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"
#
# This is to allow URL access to scripts/files in the AWStats directory.
#
Options None
AllowOverride None
Order allowed, denied
allow by everyone
C) If the changes are made as specified in parts A and B, awstats_configure.pl restarts Apache to apply the changes. To ensure that the log format change takes effect, go to your home page. This is an example of the record type that should be inserted into your new log file after restarting Apache:
62.161.78.75 - - [dd/mmm/aaaa:hh:mm:ss +0000] "GET / HTTP/1.1" 200 1234"http://www.from.com/from.html" "Mozilla/4.0 (kompatibel ;MSIE 5.01;Windows NT 5.0)“
D) awstats_configure.pl asks for a name for the configuration profile file. Enter a suitable name, e.g. B. Your web server or the virtual domain to be analyzed, i. H.my page.
awstats_configure.pl creates a new file namedawstats.my page.confCopy the template fileawstats.model.conf.The new file location is:
- For Linux/BSD/Unix users: /etc/awstats.
- For macOS, Windows and other operating systems: same directory as awstats.pl(cgi-bin).
E) awstats_configure.pl ended.
Once a configuration file has been created (byawstats_configure.pl, by your package installer, or simply a manual copy of awstats.model.conf), it is important to verify that the "MAIN PARAMETERS" match your needs. Open awstats.my page.conf in your favorite text editor (e.g. notepad.exe, vi, gedit, etc.) - do not use a word processor - and make the necessary changes.
Particular attention should be paid to these parameters:
- Check thelog fileValue. This must be the full path of the server's log file (you can also use a relative path from the awstats.pl directory, but a full path will prevent errors).
- Check theLogTypeValue. Must be "W" to parse web log files.
- Check ifLogFormatis set to 1 (for NCSA combined Apache/ELF/XLF log format) or use a custom log format if not using the combined log format.
- Put thatSiteDomainParameter for the primary domain name or intranet server name used to access the site for analysis (example: www.mysite.com). If you have multiple possible names for the same website, use the main domain name and add the others to the listHostAliasParameter.
- You can also change other parameters if needed. The complete list is described inSettings/Policy Optionsbook page.
Installation and configuration are now complete. You can jumpProcess logs: create/update a statistics databaseSection.
B) Configuration for Microsoft's IIS server
* Step 1:Configure IIS to create logs"Extended W3C Log Format"(You can still use your own custom log format, but setup is easier if you use the extended default format.) To do this, launch the IIS Management Console snap-in, select the appropriate website, and open yoursCharacteristics. Choose "Extended W3C log format", ThenCharacteristics, then the tab "Advanced Properties" and uncheck everything under "ExtendedProperties". Once all are cleared, just check the following fields:
Data
Tempo
c-ip
cs username
cs-Methode
cs-uri-stem
cs-uri-abfrage
sc-status
sc-Bytes
cs-Version
cs(user agent)
cs(Referrer)
To ensure that the log format change takes effect, you should stop IIS, back up (if desired) and remove any old log files, restart IIS, and go to its home page. This is an example of the type of records you should find in the new log file:
2000-07-19 14:14:14 62.161.78.73 - GET / 200 1234 HTTP/1.1Mozilla/4.0+(kompatibel;+MSIE+5.01;+Windows+NT+5.0)http://www.from.com/ de.htm
* Step 2:
Copy the contents of the AWStats-provided cgi-bin folder from which the AWStats package placed it on your local hard drive to your server's cgi-bin directory (includingawstats.pl,awstats.model.conf,it's atLanguage,libePluginssubdirectories).
* Level 3:Moving company AWStatsicon subdirectoriesand its contents in a directory readable by your web server, for example C:\yourwwwroot\icon.
* Step 4:Create a configuration file by copying itawstats.model.confinto a new file calledawstats.my page.confWo "my page" is a value of your choice, but is usually the domain or virtual hostname. This new file must be saved in the same directory as awstats.pl (i.e. cgi-bin).
* Step 5:Edit your newawstats.my page.confFile appropriate to your specific environment:
- Change thatlog fileValue to the full path of the web server log file (you can also use a relative path from the awstats.pl (cgi-bin) directory).
- Change thatLogTypeSet the value to "W" to analyze web log files.
- Change thatLogFormatto2 if you use that"Extended W3C Log Format"described in step 1; In case of custom format, list the logged IIS fields, for example:
LogFormat="data hora c-ip cs-username cs-method cs-uri-stemcs-uri-query sc-status sc-bytes cs-version cs(User-Agent) cs(Referer)"
- Change thatDirIconsParameter specifying the relative path of the symbol directory.
- Put thatSiteDomainParameter for the main domain name or the name of the intranet web server used to access the analyzed website (example: www.mydomain.com).
- Put thatAllowToUpdateStatsFromBrowserParameter to 1 if you don't have command line access and only have cgiaccess.
- Check and change other parameters if necessary.
Installation and configuration are now complete. You can jumpProcess logs: create/update a statistics databaseSection.
C) Configuration for other web servers
The configuration process is similar to configuring Apache or IIS.
UseLogFormatto a value of 3 if you have the native WebStar log format. Use a customLogFormatif your log format is different.
D) Configuration for other internet servers, e.g. B. FTP, mail, streaming media
The setup process for other file formats is described in the relevant FAQ topics:FAQ-COM090: FTPFAQ-COM100: MaileFAQ-COM110: Streaming media.
II. Process logs: creating/updating the statistics database
* Command line update (recommended):
The initial log analysis should be done manually from the command line as the process can be lengthy and troubleshooting is easier when you can see the output of the command (if you don't have access to the command line, skip to step 2). The AWStats command to create (and update) a statistics database is:
perl awstats.pl -config=mysite -update
Womy pageshould be replaced with the virtual host/domain name you chose earlier during AWStats setup.
AWStats reads the configuration file awstats.mysite.conf (or awstats.conf if not found) and creates/updates its database with all summary information output from the analyzed log file.
AWStats statistics database files are stored in the directory defined byDirDataparameter configuration file.
When the creation/update is complete, you should see output similar to this on your screen:
Update for configuration "/etc/awstats/awstats.mysite.conf"
With data in the log file "/pathtoyourlog/yourlog.log"...
Phase 1: First, ignore old records and look for new records...
Searching for new records from the beginning of the log file...
Phase 2: Now process new records (save history to disk after 20,000 hosts)...
Lines skipped in file: 0
Lines analyzed in the file: 225730
122 lost records found,
87 corrupt records found,
0 old records found,
225,521 new qualifying records found.
lost recordsare discarded records because they were not "HTTP user requests" or were requests that matched AWStats filters (seeSkipHosts,SkipUserAgents,ignore files,OnlyHosts,OnlyUserAgentseOnlyFilesParameter). If you want to see which rows have been discarded, you can add those-displayedoption on the command line.
corrupt recordsare records that do not match the log format defined by the LogFormat parameter in the AWStats configuration file. Normally all web servers have some corrupted records (<5%) even if everything is working properly. This can have several reasons: 1) internal web server errors, 2) erroneous requests from faulty browsers, 3) an unclean shutdown of the web server, e.g. B. disconnecting the server...
If all your lines are damaged and theLogFormatIf the parameter in the AWStats configuration file is correct, there may be a configuration issue with your web server's log format.LogFormatThe parameter in the AWStats configuration file MUST match the format of the log file you are analyzing. If you want to see which rows are corrupt you can add the following- is shown as corruptedoption on the command line.
old recordingsThey are just records that have already been processed from a previous update session. While it is not necessary to clear your log file after each upgrade operation, it is strongly recommended that you do so as often as possible.
new recordsare records in your log file that were successfully used to create/update the statistics database.
Note: A log analysis process can be slow (one second for every 4500 lines of your log file on an Athlon 1GHz, plus the DNS resolution time for each different IP address in your log file if).DNS-Sucheis set to 1 and not ready in your log file). see theBenchmarks pagefor more detailed information.
* Updating a browser:
AWStats statistics can also be updated via a browser and provides real-time statistics by clicking the "Update Now" link that appears when AWStats is used as a CGI (the URL is described in the next section).Run Reports: Create and read reports').
note!!
To enable this link, the configuration file parameter must be usedAllowToUpdateStatsFromBrowsermust be set to 1 (the link is not enabled by default).
Using online update does not prevent you from running the update process automatically and on a schedule (the command is the same as for the first update process above).
You have two options for this:
- Paste the update command into your fileLogloaderProcedure. To seeFAQ-COM120for details.
- Or add instructions in yourcrontab(Unix/Linux) or yourtask scheduler(Windows) to start the Awstats update process regularly. To seeFAQ-COM130for details.
See the AWStatsBenchmarks pagefor recommended log update/rotation frequency.
III. Run Reports: Create and read reports
Depending on your needs, you have several options for viewing the analysis resultssecurity policy.
Note: You must have created a statistical database for the analysis period by processing your log files before attempting to generate reports. See the previous section.
1. The first option is to create the main reports in a static HTML page from the command line with the following syntax (proceed with the second option if you only have CGI access):
perl awstats.pl -config=mysite -output -staticlinks> awstats.mysite.html
Womy pageshould be replaced with the virtual host/domain name you chose earlier during AWStats setup.
To create specific one-off reports, specify the report name on the command line like this¹:
perl awstats.pl -config=mysite -output=alldomains-staticlinks > awstats.mysite.alldomains.html
perl awstats.pl -config=mysite -output=allhosts-staticlinks > awstats.mysite.allhosts.html
perl awstats.pl -config=mysite -output=lasthosts-staticlinks > awstats.mysite.lasthosts.html
perl awstats.pl -config=mysite -output=unknownip-staticlinks > awstats.mysite.unknownip.html
perl awstats.pl -config=mysite -output=alllogins-staticlinks > awstats.mysite.alllogins.html
perl awstats.pl -config=mysite -output=lastlogins-staticlinks > awstats.mysite.lastlogins.html
perl awstats.pl -config=mysite -output=allrobots-staticlinks > awstats.mysite.allrobots.html
perl awstats.pl -config=mysite -output=lastrobots-staticlinks > awstats.mysite.lastrobots.html
perl awstats.pl -config=mysite -output=urldetail-staticlinks > awstats.mysite.urldetail.html
perl awstats.pl -config=mysite -output=urlentry-staticlinks > awstats.mysite.urlentry.html
perl awstats.pl -config=mysite -output=urlexit-staticlinks > awstats.mysite.urlexit.html
perl awstats.pl -config=mysite -output=browserdetail-staticlinks > awstats.mysite.browserdetail.html
perl awstats.pl -config=mysite -output=osdetail-staticlinks > awstats.mysite.osdetail.html
perl awstats.pl -config=mysite -output=unknownbrowser-staticlinks > awstats.mysite.unknownbrowser.html
perl awstats.pl -config=mysite -output=unknownos-staticlinks > awstats.mysite.unknownos.html
perl awstats.pl -config=mysite -output=refererse-staticlinks > awstats.mysite.refererse.html
perl awstats.pl -config=mysite -output=refererpages-staticlinks > awstats.mysite.refererpages.html
perl awstats.pl -config=mysite -output=keyphrases-staticlinks > awstats.mysite.keyphrases.html
perl awstats.pl -config=mysite -output=keywords-staticlinks > awstats.mysite.keywords.html
perl awstats.pl -config=mysite -output=errors404-staticlinks > awstats.mysite.errors404.html
¹You can use the if you wantawstats_buildstaticpagesTool to create all these pages in one command or generate PDF files.
Nuts:
a) You can also add aFilterin the following reports:urldetail, urlentry, urlexit, allhosts, refererpages. ÖFiltercan be a regular expression (regexp) with the full key that you want AWStats to report, appended to the output parameter separated by a ":".
For example, to create the URL Details report that only includes pages whose URL contains /news, you can use the following command line:
perl awstats.pl -config=mysite -output=urldetail:/news-staticlinks > awstats.mysite.urldetailwithfilter.html
b) If you want to create a report for a specific month, add the options-Month=MM -Year=YYYYHere MM is the two-digit month, i.e. 03, and year is the four-digit year. To create a report for a full year, add the options-month=all -year=yyyy(Warning: This is often resource intensive and can consume large amounts of memory and CPU. Unix/Linux-like operating systems may benefit from using the "nice" command.)
2) The second option is to display your statistics dynamically in the browser. To do this, use the URL:
http://www.myserver.mydomain/awstats/awstats.pl?config=mysite
Womy pageSpecifies the configuration file to use (AWStats uses the awstats.my page.conf).
All output command line options (except -staticlinks and -logfile) are also available when using AWStats with a browser. Just use them as URL parameters: change "-option" to "&option" i.e. H.http://www.myserver.mydomain/awstats/awstats.pl?month=MM&year=YYYY&output=unknownos
Reports are generated in real-time from the statistics database. If this is slow or is putting too much load on your server, consider generating static reports.
Se oAllowToUpdateStatsFromBrowserIf the parameter is set to 1 in the AWStats configuration file, you can also run the update process in your browser. Just click the "Update Now" link.