Table of Content:
Introduction:
I started writing the first installement of aila back here on Connect in the winter of 2009. I had both Linux and Windows build for a while but when I implemented complex data structures the windows build broke and I switch to support Windows via Cygwin.
This made it difficult to install and run on servers, so I built-up a web-service to receive and handle IIS log files and show the results in a web-page using google charts. This worked well and I had a growing number of users. However interesting biulding and maintaining the web-service was a recent customer need showed me that it was time to offer a similar service in premise.
And this is why I have writen aila2, a c#, build from scratch IIS Log analyzer.
Data processing:
aila2 starts by resolving the provided input (log file or stdin) to a schema. The IIS schemas are defined using commented line (#) and starting with the "Fields: " (case insensitive) string. Once a schema is found the tool will pasrse of the non commented line and update statistics according to the line content. This will continue until we reach the end of file or until we found another schema
aila2 currently collected the following data during its execution:
File based information:
- file name
- file md5 hash
- linecount
Log entry based statistics:
- Hourly hits count (Columns: Hit #, PostEvent, GetClientPolicy, GetPackageInfo
- 0
- 1
- 2
- ...
- 23
- Mime type hit count (Columns: Hit #, Sum(time-taken), Max(time-taken), Avg(time-taken))
- htm
- html
- asp
- aspx
- asmx
- ascx
- axh
- ashx
- xml
- css
- js
- gif
- png
- jpg
- other
- Web applications (Columns: Hit# count, Sum(time-taken), Max(time-taken), Avg(time-taken))
- /Altiris/NS/Agent/
- /Altiris/TaskManagement/
- /Altiris/InventoryRuleManagement/
- /Altiris/NS/NSCap/
- /Altiris/PackageShare/
- /Altiris/SoftwarePortal/
- /Altiris/ClientTaskAgent/
- /Altiris/ClientTaskServer/
- /Altiris/Console/
- /Altiris/ActivityCenter/
- /Altiris/Resource/
- /Altiris/Workflow/
- /Altiris
- Other
- Agent interface hit counts (Columns: Hit# count, Sum(time-taken), Max(time-taken), Avg(time-taken))
- PostEvent.asp
- PostEvent.aspx
- GetPackageInfo.aspx
- GetClientPolicies.aspx
- GetPackageSnapshot.aspx
- CreateResource.aspx
- GetLicense.asmxGetLicenseDetails.aspxOther
The output generated is a java script object with the following outline:
{
"file" : "<file name>",
"hash" : "<file md5 hash>",
"linecount" : <count>,
"stats" : {
"hourly" : [
"Hour",
"Total hit #",
"Post Event",
"Get Client Policy",
"Get Package Info"
]
"mime_type" : [
"Mime type",
"Hit #"
]
"web_application" : [
"Web-application",
"Hit #",
"Sum(time-taken)",
"Max(time-taken)",
"Avg(time-taken)"
]
"agent_interface" : [
"Agent interface",
"Hit #", "Sum(time-taken)",
"Max(time-taken)",
"Avg(time-taken)"
]
}
}
Usage:
Usage : aila2 [Parameter] [Option(s)]
Parameters:
-h, --help Show this help message
-v, --version Output program version only
-f, --file <file path> The IIS log file to parse
--stdin The log file data will come from the console input
instead of a file.
Options:
-l, --log-level <lvl> Output log data <= to to stdout:
--log-level 1 -> error
--log-level 2 -> warning
--log-level 4 -> information
--log-level 8 -> verbose
--log-level 16 -> debug
-o, --out-path <path> The location where the result file will be created.
Samples:
aila2 -f iis.log
aila2 --file iis.log -l 4
aila2 -f iis.log -o c:\inetpub\wwwroot\aila2\
{CWoc} info: http://www.symantec.com/connect/search/apachesolr_...Usage notes:
If no command line arguments are provided the program will expect to receive data from the console (stdin). If no output path is provided the output data will be writen to the console.
When processing a file aila2 prints out a counter of currently processed line, and upon completion a processed line count and time taken (in millisecond).
Output:
Here is the output from running the following command on a test server (so the data will not be very imp[rpessive indeed):
{
"file" : "u_ex131126.log",
"hash" : "1cadf0dbcb17988b5c88626e7b1cdf2f",
"linecount" : 12095,
"stats" : {
"hourly" : [
["Hour", "Total hit #", "Post Event", "Get Client Policy", "Get Package Info"],
["0", 388, 0, 0, 1],
["1", 338, 0, 0, 1],
["2", 460, 2, 0, 1],
["3", 457, 1, 0, 1],
["4", 385, 0, 0, 1],
["5", 359, 0, 0, 1],
["6", 363, 0, 0, 1],
["7", 366, 1, 0, 1],
["8", 435, 0, 0, 1],
["9", 458, 0, 0, 1],
["10", 455, 0, 0, 0],
["11", 449, 1, 0, 1],
["12", 456, 0, 0, 1],
["13", 493, 0, 0, 1],
["14", 2202, 0, 0, 1],
["15", 656, 1, 0, 1],
["16", 630, 0, 0, 1],
["17", 643, 0, 0, 1],
["18", 509, 0, 0, 1],
["19", 564, 1, 0, 1],
["20", 461, 0, 0, 1],
["21", 199, 0, 0, 1],
["22", 184, 0, 0, 1],
["23", 185, 1, 0, 1]
],
"mime_type" : [
["Mime type", "Hit #"],
["htm", 0],
["html", 22],
["asp", 0],
["aspx", 5425],
["asmx", 5065],
["ascx", 0],
["axd", 416],
["ashx", 0],
["xml", 0],
["css", 60],
["js", 154],
["gif", 691],
["png", 230],
["jpg", 8],
["Other", 24]
],
"web_application" : [
["Web-application", "Hit #", "Sum(time-taken)", "Max(time-taken)", "Avg(time-taken)"],
["/altiris/ns/agent/", 31, 363308, 44343, 11719.61],
["/altiris/taskmanagement/", 81, 11378, 4250, 140.4691],
["/altiris/inventoryrulemanagement/", 0, 0, 0, 0],
["/altiris/ns/nscap/", 0, 0, 0, 0],
["/altiris/ns/", 1112, 924678, 193530, 831.545],
["/altiris/packageshare/", 0, 0, 0, 0],
["/altiris/swportal/", 0, 0, 0, 0],
["/altiris/clienttaskagent/", 0, 0, 0, 0],
["/altiris/clienttaskserver/", 48, 167013, 21968, 3479.438],
["/altiris/console/", 5633, 674149, 32289, 119.6785],
["/altiris/activitycenter/", 0, 0, 0, 0],
["/altiris/resource/", 0, 0, 0, 0],
["/altiris/workflow/", 1446, 165549, 47281, 114.4875],
["/altiris", 833, 120750, 16443, 144.958],
["other", 2911, 93745, 28406, 32.20371]
],
"agent_interface" : [
["Agent interface", "Hit #", "Sum(time-taken)", "Max(time-taken)", "Avg(time-taken)"],
["postevent.asp", 0, 0, 0, 0],
["postevent.aspx", 8, 11372, 2531, 1421.5],
["getpackageinfo.aspx", 0, 0, 0, 0],
["getclientpolicies.aspx", 23, 351936, 44343, 15301.57],
["getpackagesnapshot.aspx", 0, 0, 0, 0],
["createresource.aspx", 0, 0, 0, 0],
["getlicense.asmx", 0, 0, 0, 0],
["getlicensedetails.aspx", 0, 0, 0, 0],
["other", 0, 0, 0, 0]
]
}
}
Conclusion:
You can now parse individual log file in their row format or filtered using aila2-filter [1]. Next we will see how to view the JSON data in a graphical format (similar to what was available on aila-web), how to create output file automatically for all IIS logs found in a given directory, and how to generate a calendar view webinterface for easy consumption of the generated data.
So this is quite a program we have here for Connect :D.
References:
[1]aila2-filter Connect download
Document history:
Version 1
Releasing version 1 of the tool (based on Github tag Version-1.2).
File details:
- File name: aila2-version1.zip
- File sha256 sum: d53034db3d0ed59ee4c5126d31b88685ca748da848fe16017e1f91bd0b5a8717
File content:
| File name | Description | Documentation |
|---|---|---|
| aila2.exe | The main aila2 program with the usage explained here. | This download page. |
| aila2.html | Part of the aila2-web Calendar user interface. It displays individual result file data usgni the Google Chart WAPI | "aila2-web Calendar" - upcoming |
| aila2.js | A javascript file with functions used in both aila2-web interfaces. | n/a |
| index.html | Part of the aila2-web Calendar user interface. This is the landing page that shows all or part of the result files in a reverse calendar (from most recent to oldest). | "aila2-web Calendar" - upcoming |
| quickview.html | A standlone viewer to use for quick checks on filtered IIS log files or to get up and running quickly with aila2 capabilities. | "aila2-web Quickview" - upcoming |
| style.css | A cascading style sheet containing styles used in aila2-web interfaces. | n/a |