FinalAnalytics Blog

A dark theme for the HttpLogBrowser

Thu, 18 Apr 2024 18:36:37 GMT

The version 4.63 of the HttpLogBrowser is available with the following new feature.

Dark theme mode

With the professional edition, the application can now be displayed with a dark theme. By default the dark theme is automatically selected if Windows is configured to display applications in dark mode.

And if needed the dark theme can be enabled/disabled in the preferences.

Bug fixes in this version are listed in the change log.

HttpLogBrowser 4.62 released

Mon, 19 Dec 2022 21:05:00 GMT

A new small release of the HttpLogBrowser is available with the following new feature.

Download this version

Ability to change the date time format in data grids

If you need to export the data grids in .csv files in order to import the data in another application you may prefer to have the date and time in ISO format rather than the default date and time format from the system regional settings. You can now customize that in the preferences. For example specify yyyy-MM-dd HH:mm:ss if you want to use the ISO date/time format.

Bug fixes are listed in the change log.

HttpLogBrowser 4.61 released

Mon, 03 Jan 2022 18:11:00 GMT

A new small release of the HttpLogBrowser is available with the following new features included in the free edition.

Download this version

Ignore IP addresses or user agents

A new page was added in the log settings in order to exclude web requests with specific client IP addresses or user agent strings.

This is useful when you have some monitoring tools that are constantly doing web requests on your web site and fill up the logs with useless information. Excluding these web requests will allow loading the data faster, use less memory and filter faster.

Use UTC time

With this new option you can display web request times in UTC instead of your local time. After changing this setting you need to reload the data and if you use the cache or the database mode you first need to clear the cache or the database.

Bug fixes in this version are listed in the change log.

HttpLogBrowser 4.6 is available!

Thu, 09 Sep 2021 18:12:00 GMT

A new version of the HttpLogBrowser is available and the professional edition gets new features.

Download this version

You can find hereafter the new features.

Professional edition

Root log folder mode

In previous versions there already was the multiple log folder mode to analyze several websites simultaneously but the goal was rather to analyze simultaneously several instances off a load balanced web site. It was not easy to analyze all web sites of a single server if you had many web sites.

Now with the root folder mode you can select the parent log folder (e.g. C:\inetpub\logs\LogFiles) to analyze all web sites of a server simultaneously.

And if the application is running on the IIS server you just need to select All sites in Local IIS sites to do the same.

With this mode several fields are added:

The LogName field with the name of the log folder in the root folder (e.g. W3SVC1)
The SiteName field with the name of the web site retrieved from the IIS metabase (With the All Sites mode only).
The new LogicalPath field that is the concatenation of one of the previous field with the UrlPath field.

Export Tree statistics

In previous versions when you enabled the Tree mode for the UrlPath field it was not possible to export the tree statistics in a MS Excel or CSV file from the detailed statistics. This is now fixed. By default only one level below a collapsed node is exported. So, if you want to export the whole tree. You need to right click on the root node and click on Expand All before launching the export.
You can see hereafter an example of export.

Bug fixes in this version are listed in the change log.

This new version is free for all customers of the professional edition. So don’t hesitate to download and install it.

HttpLogBrowser 4.5 released !

Wed, 02 Dec 2020 23:40:00 GMT

The beta period of version 4.5 of the HttpLogBrowser is now over and the final release is available. Here is a quick reminder of what’s new:

Free edition

New time measure units to improve time statistics histograms zooming
New time selection modes to easily select periods of the calendar as time window
Display the average week or day activity shape with a better time resolution with the new time span fields TimeOfWeek and TimeOfDay

Professional edition

Dynamic log file loading to only load log files concerned by the selected time window in order to display log rows faster.
Database favorite filters to share saved filters with other users with a read only database profile.
Lately added in the beta 2 the Web request report allows to generate a printable report of the filtered web requests.

For a full list and full explanation of new features you can read what's new in version 4.5.

If you are already using the beta version you can check the change log to see the improvements and fixes in the final release.

The professional edition of version 4.5 is free for all existing customers with a professional license of an older version even if their maintenance has expired. So don’t hesitate to upgrade to this new version!

Download the version 4.5

Thanks to users that tested the beta version and sent their feedback.

HttpLogBrowser 4.5 is coming out !

Wed, 09 Sep 2020 20:51:58 GMT

A new version of the HttpLogBrowser is available in beta. Both free and professional editions get new features.

Download the beta version

You can find hereafter a short list of the new features and for more information please read What’s new in version 4.5.

Free edition

New available Time measure units (6 hours, 15 minutes, 5 minutes, …) for a better zooming experience in time evolution charts.
New time selection modes. Easily select whole weeks, months, quarter and years
New time span fields TimeOfWeek, TimeOfDay with better time resolution than the previous DayOfWeek and HourOfDay fields.
Support of log files generated by the IIS advanced logging module

Professional edition

When possible only needed log files are loaded according to the time selection window. This will speed up dramatically log files loading when you choose a whole log folder but select only a short time frame to be displayed.
Ability to store favorite filters in database profiles with the ability to share them with other users

You can send any feedback or problem to support\@finalanalytics.com or by adding a comment to this blog post. If a previous version is already installed, there is no need to uninstall it, just launch the setup and an upgrade will automatically take place.

The early adopters operation allowing existing customers with a professional license of an older version to use the latest version even if their maintenance has expired is still in place. So don’t hesitate to try this new version out.

The final release of HttpLogBrowser 4.0 is available!

Tue, 10 Dec 2019 20:48:00 GMT

The beta period of version 4 of the HttpLogBrowser is now over and the final release is available. If you are already using the beta version you can check the change log to see the improvements and fixes. If you are still using the version 3 you can read what's new in version 4 to see the new major features in this version.

The professional edition of version 4 is free for all existing customers with a professional license of an older version even if their maintenance has expired. So don’t hesitate to upgrade to this new version!

Download the version 4.0

Thanks to users that tested the beta version and sent their feedback.

HttpLogBrowser 4.0 is available in beta!

Wed, 23 Oct 2019 20:23:00 GMT

After more than one year of development the new version of the HttpLogBrowser is available as a beta version. Both free and professional editions get some new features.

Download the beta version

For exhaustive information on this new version you can read What’s new in version 4.0. You’ll also find a short list of the new features hereafter:

Free edition

Newly available log rows are automatically loaded during a refresh
Improved IIS FTP logs analysis
Improved quick access to local IIS sites logs
Fields can be organized by categories
New welcome screen

Professional edition

Multiple log folders mode for load balanced web sites
Filter cache for the database mode
Real time mode
Ability to browse remote log folders to synchronize
Ability to schedule the log synchronization and insertion from the UI

Download the beta version

Don’t hesitate to send any feedback or problem to support@finalanalytics.com or by adding a comment to this blog post. If a previous version is already installed, there is no need to uninstall it, just launch the setup and an upgrade will automatically take place. In the case you want to downgrade later you just need to uninstall the beta version and reinstall the previous version. All settings from before the upgrade will be restored.

As a sign of thanks to early adopters, the professional edition of version 4 is free for all existing customers with a professional license of an older version even if their maintenance has expired. So don’t hesitate to try this new version out!

For fixes and improvements published during the beta period you can consult the Change Log.

The WordPress user enumeration attack

Mon, 26 Nov 2018 18:41:00 GMT

Recently I was contacted by a company that was developing a WordPress web site for a real estate agency but after making the web site public it was immediately hacked. The webmaster did not want to put the website back online until he new why it was compromised. So I was asked to determine how the attacker had gained control over the web site and then to secure it.

The investigation

I downloaded the Apache access log files and loaded them in the HttpLogBrowser and I found several suspicious activities:

Suspicious activity

First, a plugin (ubh) was uploaded from a foreign IP address (Spain) (The real estate agency was located in France). You can see that in the following screenshot (Chronological order is upwards).

Secondly, two themes (maxbusiness and fuence) were uploaded from a Ukrainian IP address

A botnet involved

After the first theme (maxbusiness) was uploaded a few hours later there was much activity on a PHP file added by this theme and the activity came from many IP addresses. You can see that in the following screenshot. 900 IP addresses requested the PHP file /wp-content/themes/maxbusiness/fonts/qfsqjiul.php installed by the uploaded theme with a rate of 50 requests per hour. Among these IP addresses only a few requested the file several times.

When I was doing the investigation the rogue files were already removed so I could not see the code of this PHP file. However according to what the webmaster told me, the web hosting provider cut the hosting because the web site was sending too many E-mails. So it was most probable that the mails were sent by this PHP file controlled by a botnet.

The initial attack

It was also clear that the whole purpose of the attack was to use the web site to send spam E-Mail. However at this stage of the investigation I still didn’t know how the attacker could gain admin access to WordPress in order to upload themes. So I continued to analyze the traffic that came from outside the country and more particularly from Ukraine and I found the following suspicious activity from a second Ukrainian IP address just one hour before the malicious theme was uploaded.

I did not immediately understand the attack but there were requests to the XML-RPC module that is known to be used to brute force passwords. There were also requests to wp-login.php, the login page of WordPress. And there were also some other web requests that were cryptic to me because I was a newbie in WordPress security at that time.

I asked then to the webmaster to put the web site back online and when I ran a security audit with Hacker Target

I saw this in the result: Warning! User Enumeration is possible

That’s something I didn’t know. It’s possible to enumerate users on a default installation of WordPress.

If you take a look again at the screenshot of the attack (see below) you see several requests with author=N (with N=1 to 7) as query parameter. You also see that for author=1 the request is redirected to a URL that contains the name of the admin account (wpadmin). So it’s easy to guess the account used to administer WordPress. The other requests with author > 1 lead to a 404 (not found) error because there was only one account configured.

Weak password

What was worse is that when I got the password of the admin account I saw that it was identical to the account name. The Webmaster did this by thinking that the admin account name could not be guessed by someone outside and this was the mistake.

Trying the account name as password was probably the first thing the attacker did. A brute force attack to guess the password wasn’t even needed!

Securing the web site

Strong password

After understanding that, I immediately changed the admin password with a complex password generated by WordPress itself.

The lesson to learn from this is to never use a password that can be guessed from the account name because the account name is not a secret. Today it’s easy enough to use a complex password stored in a password manager.

Avoid user enumeration

However if needed there is a way to disable the user enumeration in WordPress and this is explained in the following article: Stop User Enumeration in WordPress

One of the two methods proposed by this article is to add the following lines in the .htaccess file located at the root of the WordPress site (If hosted by an Apache web server).

# Block User ID Phishing Requests  
<IfModule mod_rewrite.c>  
RewriteCond %{QUERY_STRING} ^author=([0-9]*)  
RewriteRule .* http://example.com/? [L,R=302]  
</IfModule>

Replace http://example.com with the base URL of the WordPress site. The effect will be to automatically redirect any request with “author=?” as query string to the root of the web site instead of the author page. We can see the effect by checking the Apache logs after the modification was done. Take a look at the following screenshot:

But this may not be enough because there is now a new way since WordPress 4.7 to enumerate users through the json API. The json API is only enabled if the Permalinks are not set to Plain in the settings. You can check if you are concerned by requesting the following URL on your WordPress site:
http://your-wordpress-site.com/wp-json/wp/v2/users/
If you are concerned you will get a json file with all authors with published content on the web site as in the following screenshot.

So here is an improved version of the rewrite rule that handles both ways to enumerate users and respond with a 403 status (forbidden). These lines need to be added before the line # BEGIN WordPress where the WordPress directives start in the .htaccess.

<IfModule mod_rewrite.c>
RewriteCond %{REQUEST_URI} ^/wp-json/wp/v2/users [OR]
RewriteCond %{QUERY_STRING} ^author=([0-9]*)
RewriteRule ^ - [L,R=403]
</IfModule>

Now you get this when you try to get users through the json API:

So the user enumeration can be easily stopped but what is more difficult is that user names may also be displayed by the WordPress theme at different places on the web site and if that’s the case you will need to change the code of the theme possibly at several locations in order to completely hide all account names.

Conclusion

We saw how a WordPress web site could be easily hacked by an attacker because the webmaster used the account name as password. We saw how it was possible to investigate the attack with the Apache access logs and the HttpLogBrowser to analyze them. Then we finally saw how to secure the web site against such an attack.

As last advice, it’s always important to understand how a web site was hacked. Otherwise you cannot learn from your mistakes. The webmaster was maybe young and unexperienced but he had the good reflex to not put the web site back online until the problem was understood and fixed.

HttpLogBrowser 3.0 final release available!

Tue, 31 Jul 2018 09:28:00 GMT

The beta period for the version 3 of the HttpLogBrowser is now over and the final release is available. If you are already using the beta version you can check the change log to see the improvements and fixes. If you are still using the version 2 you can read what's new in version 3 to see the new major features in this version..

Download the version 3.0

Thanks to users that tested the beta version and sent their feedback.

HttpLogBrowser 3.0 beta has been released!

Tue, 29 May 2018 21:14:00 GMT

After several months of development the new version of the HttpLogBrowser is available as a beta version. The free edition and the professional edition get new features.

Don’t hesitate to send any feedback or problem to support@finalanalytics.com or by adding a comment to this blog post. If a previous version is installed there is no need to uninstall it. Just launch the setup and an upgrade will automatically take place. In the case you want to downgrade later you just need to uninstall the beta version and reinstall the version 2.0. All settings from before the upgrade will be restored.

To see all new features in this new version you can read What's new in version 3.0 You'll also find a short list below.

Download the beta version

Short list of new features:

Quickly reload recent logs
Load local IIS sites logs from the file menu
An icon allows you to quickly see the status of a web request
The day of week and the hour of day are extracted in new fields
The column order is persistent
Log synchronization (Professional edition)
Database mode (Professional edition)
Fast reverse DNS to determine clients host names (Professional edition)
Command line mode (Professional edition)

Download the beta version

Configure Apache access logs on Ubuntu server

Wed, 02 May 2018 19:04:00 GMT

All my previous articles were about how to configure HTTP logs in IIS. It’s now time to see how to monitor the web site activity on an Apache web server running on Linux. The article is based on an Ubuntu installation on which a web site is running. We’ll see in this article how to configure the Apache access logs to facilitate archiving and synchronization of the log files. We’ll then see how to add useful additional fields and how to share the log files to be easily downloadable and synchronized from a remote workstation. Finally we’ll see how we can analyze these logs on a Windows workstation.

Presentation of the environment

You’ll find hereafter how the Ubuntu server was prepared for this article. If you have a configuration a little different it should not be a problem.

Ubuntu 16.04 LTE server
LAMP (Linux Apache / MySQL / PHP) has been installed
SSH has been enabled
A WordPress site has been installed
The WordPress site is accessible from a URL like www.wptest.com and wptest.com with redirection
HTTPS has been activated for the WordPress site
If the web site is accessed in HTTP a redirection occurs to HTTPS

This is a very basic WordPress site.

Log rotation

In the default configuration the web activity on the Apache server is stored in the file /var/log/apache2/access.log

So if you don’t do anything this file will keep growing and will be every time more difficult to consult. So the first thing to do is to configure the log rotation to store the log lines in a new file every day. This will allow archiving old log files and facilitate consulting the recent web activity.

The access log configuration is stored either in the Apache configuration file /etc/apache2/apache2.conf or in the virtual site configuration. By default you’ll have the following two virtual sites configuration files:

/etc/apache2/sites-available/000-default.conf (default HTTP site)
/etc/apache2/sites-available/default-ssl.conf (default HTTPS site)

In both files you have by default the following log directive

CustomLog /var/log/apache2/access.log combined

You’ll find more information on how to configure Apache log files and the CustomLog directive in the Apache log files documentation.

In order to put a log rotation in place we will pipe the log rows to the rotatelogs program (See the rotatelogs documentation) and as argument we will tell to rotatelogs in which folder to store the log files, how to name them and after how much time a new file needs to be created.

The following new CustomLog directive will rotate the log file everyday:

CustomLog "||/usr/bin/rotatelogs /var/log/apache2/site000/access/access-%Y-%m-%d.log 86400" combined

Before applying the new configuration we need to create the folder to store the log files of our web site

sudo mkdir /var/log/apache2/site000

And a folder to specifically store access logs for the web activity in case later we want to store error logs in a separate folder.

sudo mkdir /var/log/apache2/site000/access

Then we can change the virtual site configuration files:

sudo nano /etc/apache2/sites-available/000-default.conf

Replace the CustomLog directive with the new one mentioned before. Then save and exit (CTRL+O and CTRL+X)

Do the same with the https virtual site

sudo nano /etc/apache2/sites-available/default-ssl.conf

Now to apply the new configuration we restart Apache gracefully

sudo apache2ctl graceful

Then after some activity happened on the web server we can take a look at the just created log folder

ls /var/log/apache2/site000/access

We see that a file with today’s date has been created.

We can display the content of the file with the following command (replace the date in the command)

cat /var/log/apache2/site000/access/access-2018-04-26.log|tail

And we see this

Add additional fields

The logged fields are defined with the LogFormat directive in the /etc/apache2/apache2.conf configuration file. You’ll find hereafter the default LogFormat directives.

LogFormat "%v:%p %h %l %u %t \"%r\" %\>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined  
LogFormat "%h %l %u %t \"%r\" %\>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined  
LogFormat "%h %l %u %t \"%r\" %\>s %O" common  
LogFormat "%{Referer}i -\> %U" referrer  
LogFormat "%{User-agent}i" agent

In our CustomLog directive we use the combined format. This is a standard format that contains the client IP address, the used account if any, the date and time, the request (Method + path + query), the HTTP status, the number of bytes sent, the referrer and the user agent.

The variables used in the LogFormat directive are explained in the Apache logging module documentation.

A few important fields are missing from the combined log format.

You don’t have the time taken by the request to be served. This is important if you want monitor the performance of your web site.
You don’t have the port used by the request. This is important if you want to differentiate HTTP (port 80) from HTTPS (port 443) requests. Of course you could put both in different log files instead as HTTPS is configured on a different virtual site. However if you want to follow SEO best practice to avoid duplicate content you will make any HTTP request on your web site automatically redirect to HTTPS. If both original HTTP requests and redirected requests are in the same file it will be easier to follow.
You don’t have the hostname entered in the browser. This is also important if you want to avoid duplicate content. For example you may choose to have your main URL with the www but also want that a request without the www is automatically redirected to the main URL. So if you have the hostname in the log file you will be able to monitor if these redirections are working as expected.

So now that we have selected 3 missing fields important for us we will create a new LogFormat directive. We start from the combined log format and add the new fields at the end. It’s important to add them at the end so any log tool that needs to parse the log file will still be able to extract the standard fields from the log line by just ignoring the end of the log lines.

We will add the following variables:

%{ms}T Time taken by the request in ms
%p Port used by the request on the server
%{Host}i Host name provided by the browser

So this leads us to the following new directive that creates a new log format named extended.

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %{ms}T %p %{Host}i" extended

We edit the Apache configuration file

sudo nano /etc/apache2/apache2.conf

We add the new LogFormat directive and then we save and exit.

Then we need to change the CustomLog directives in the virtual site configuration files to use this new extended format.

CustomLog "||/usr/bin/rotatelogs /var/log/apache2/site000/access/access-%Y-%m-%d.log 86400" extended

First we edit the HTTP virtual site configuration file

sudo nano /etc/apache2/sites-available/000-default.conf

In the CustomLog directive replace combined with extended. Then save and exit.

Then we edit the HTTPS virtual site configuration file

sudo nano /etc/apache2/sites-available/default-ssl.conf

In the CustomLog directive replace combined with extended. Then save and exit.

Now to apply the new configuration we restart Apache gracefully

sudo apache2ctl graceful

Then after some activity took place we can display the last lines of the log file (replace the date in the following command)

cat /var/log/apache2/site000/access/access-2018-04-26.log|tail

And we can see the new fields

Access logs remotely

Analyzing the logs directly on the server is a little difficult so we will see now how we can retrieve them remotely.

Access logs remotely with the admin account

As SSH is enabled it’s very easy to access the log folder through the SFTP protocol by using FileZilla for example.

In FileZilla we create a new site in the Site manager.

Specify the web site hostname as host
Select SFTP as protocol
Select Normal as Logon type
And specify the credentials of your admin account

Once connected you can browse the whole server folder tree and you can retrieve more specifically the content of the log folder.

However only an admin account (member of the adm group) can retrieve the logs this way. If the user is not member of the adm group he gets an access denied when trying to access to /var/log/apache2.

Share the log folder for a less privileged account

You may want to allow an auditor from outside your organization to access the logs. So you cannot give him the credentials of an account with access to the whole folder tree and with shell access.

So to work around that we will create a specific account only allowed to read the content of the log folder.

It’s possible with SSH to share a specific folder without access to anything else. However there is a condition for this folder. Root must be the owner of the folder and of all parent folders. Unfortunately that’s not the case for folders inside /var/log/.

We don’t want to alter permissions in /var/log/ to avoid any problems with the system logging. So we will create instead a symbolic link to access the log folder from the folder /var/www/ were the web site root folder is already located. This folder has the required permissions by default.

To do that we execute the following command:

sudo ln -s /var/log/apache2/site000 /var/www/logs

We can check that the link is working

ls /var/www/logs/access

And we can see the content of the web site log folder from the new location

Now that the log folder is ready to be shared we create a user named log-guy

sudo adduser log-guy

Once the user is created we edit the SSH configuration

sudo nano /etc/ssh/sshd_config

We add in the configuration file the following lines that will only allow the log-guy account to access the log folder.

Match User log-guy  
ForceCommand internal-sftp  
PasswordAuthentication yes
ChrootDirectory /var/www/logs  
PermitTunnel no
AllowAgentForwarding no  
AllowTcpForwarding no  
X11Forwarding no

We can then save and quit and restart the SSH service to apply the new configuration

sudo systemctl restart sshd

We are now ready to do a test with FileZilla

We create a new site with the same settings but with the log-guy account credentials

And when we connect we see that we land directly in the log folder without any way to go to a parent folder. And we can see our log files in the access subfolder.

Synchronize logs

Now that you can access Apache logs remotely you may not want to regularly retrieve new log files manually. To do that you can use some free synchronization tools that support the SFTP protocol.

For example FreeFileSync will do the job.

If your workstation is on Windows you can also use the LogSync command line tool presented in a previous article.

The main advantage of this tool is that only the increased part of the log file will be retrieved. So if you need to get the latest web requests several times in a day it will avoid downloading every time the whole file saving bandwidth and speeding up the synchronization.

For the occasion the support of the SFTP protocol has been added to the tool. Please refer to the article for a full explanation on this tool.

Download LogSync

First run the following command to store the password

LogSync encrypt -n LogGuyPassword

You are then prompted to enter the password that will be encrypted and saved under the name LogGuyPassword.

Then launch the following command to start the synchronization. The destination folder (e.g. d:\Logs\wptest) needs to be created first.

LogSync sftpsync -h www.wptest.com -l d:\Logs\wptest -r /access -u log-guy -p LogGuyPassword

We now have our log files on Windows and it is very easy to keep them synchronized by launching a batch.

Analyze the logs

Now that we have all logs on our workstation we can start analyzing them. If you are on a Windows workstation you can use the free edition of the HttpLogBrowser.

Download it from this link. Make sure you have the just released version 2.0.0.11 (or more) that fixes some issues with Apache logs and adds the support of the three fields we added previously.

Once installed and launched go in the Files menu, select Folder and browse for the local log folder.

Once the log files have been processed you can see the log rows in the application. In particular you clearly see that the port and host fields have been correctly parsed and you see that requests to the port 80 or host wptest.com are redirected to the canonical URL https://www.wptest.com.

To continue further the analysis you could for example click on the value 80 in the Port statistics to check if all http requests are correctly redirected.

You can also see that the TimeTaken field was also retrieved. In the field statistics panel you can see a histogram like hereafter that will allow you to verify the response speed of your web site:

You can easily filter on the slowest requests by clicking at a specific position in the histogram to move the cursor and then click on the link TimeTaken > X below the chart.

You see that the HttpLogBrowser makes it very easy to filter on specific field values to only display the web requests you are interested in. I’ll let you discover by yourself all the features of this tool.

Conclusion

You learned in this article how to configure Apache access logs on an Ubuntu server and more specifically how to configure the log rotation, how to add custom fields and how to share the log folder to retrieve the log files from a remote workstation. Then you also learned how to easily synchronize these log files and how to analyze them from a Windows machine with the free edition of the HttpLogBrowser.

References:

Identify and forbid weak TLS usage in IIS

Mon, 01 Jan 2018 18:02:00 GMT

In this article we will see which web encryption protocols are considered as weak and how it's possible to identify web traffic encrypted with these weak protocols in IIS and finally we’ll see how to disable them to keep the web traffic more secure.

Weak encryption protocols

Lately there have been several attacks on encryption protocols used to encrypt communications between web browsers and web servers (https). Most of these attacks use flaws in older protocols that are still active on web servers in a Man In The Middle scenario. For example the POODLE attack forces the server to fall back to the flawed SSL3 protocol even that the latest TLS protocol is available. Some attacks are directly against TLS but for now only some implementations of TLS are concerned. In particularly TLS 1.0 has some weaknesses that facilitate these attacks and could lead soon to successful attacks on the whole protocol and not only on specific implementations. So TLS 1.1 and 1.2 should be used instead and fallback to older unsecure protocols should not be possible.

For this reason TLS 1.0 needs to be disabled as soon as possible as well as any older protocols (SSL). After 30^th June 2018 these encryption protocols will no longer be allowed for organization that need to be compliant with PCI DSS (Payment Card Industry Data Security Standard) and for US government servers and clients only TLS 1.1 and 1.2 are now compliant with the NIST Special Publication 800-52.

However you can’t disable these deprecated protocols before being sure that you will not break anything. You need to identify in which cases those old protocols are still used and upgrade all required systems to remove all dependencies on them and then only you can disable them.

That’s what we will see in this this article in the case of an IIS web server.

Configure cryptographic fields in IIS HTTP logs

On September 17th 2017 Microsoft announced in the article New IIS functionality to help identify weak TLS usage that the ability to log some new fields allowing to know which encryption algorithms are used for every web request has been added.

This feature was added to Windows 2012 R2 and Windows 2016 in the July 2017 Monthly rollup. So before trying to implement this, make sure your web servers are up to date.

This feature is based on the IIS custom logging fields feature introduced with Windows 2012 R2. You can see how to use this feature in my blog post Configure IIS HTTP logging. However we will do it differently in this article by editing directly the IIS configuration.

The custom log fields configuration is stored in the IIS configuration file ApplicationHost.config located in the folder C:\Windows\System32\inetsrv\config\

We will use the notepad to edit this file. If your account needs privilege elevation to have full administrative rights directly start the notepad as administrator so you will be able to save the modifications later without any trouble.

Then load the IIS configuration file in the notepad.

In the XML file locate the following section configuration/system.applicationHost/sites. At this location, you will have one section for each site running on the server. Choose the site you want to audit and locate the section logfile/customFields for this site.

Then in this section after the <clear/> line, just paste the following custom field configuration copied from the Microsoft article.

 <add logFieldName="crypt-protocol" sourceName="CRYPT_PROTOCOL" sourceType="ServerVariable" />
 <add logFieldName="crypt-cipher" sourceName="CRYPT_CIPHER_ALG_ID" sourceType="ServerVariable" />
 <add logFieldName="crypt-hash" sourceName="CRYPT_HASH_ALG_ID" sourceType="ServerVariable" />
 <add logFieldName="crypt-keyexchange" sourceName="CRYPT_KEYEXCHANGE_ALG_ID" sourceType="ServerVariable" />

If the section logfile/customFields doesn't exists you need to create it like in the screenshot below.

Then save the file.

If you open the logging configuration of the web site in the IIS manager you should see the just added custom fields like in the following screenshot.

Then after some activity occurred on the concerned web site you can check the latest log file located for example in the folder C:\inetpub\logs\LogFiles\W3SVC1 for the default web site.

If you open this log file in the notepad you will see at the end of each line 4 numeric fields like in the following screenshot.

The meaning of these numeric values can be found in the MSDN either on the page secure protocol version or on the page cipher suite algorithm but it’s a little cumbersome to get the meaning for every value found in the logs. This is where the FinalAnalytics tool HttpLogBrowser comes into play.

Audit weak TLS usage with the HttpLogBrowser

The HttpLogBrowser is an IIS HTTP log viewer and analyzer. The free edition will be enough to do what is needed in this article except if you need to export or print the data. In the just released version 2 of the application, a new feature named Translate cryptographic fields has been added to display meaningful values for the new log fields that we just enabled in the previous section.

You can download the HttpLogBrowser from this link. For the purpose of this article I will install the tool on the web server but in a real life scenario it’s recommended to install it on a workstation and download the logs to the workstation because log analysis can be CPU and memory consuming. I have two blog articles that explain how to do this: How to retrieve IIS HTTP logs remotely and Synchronize IIS HTTP logs

Once the HttpLogBrowser is installed and launched you first need to enable the required setting in the Default log settings accessible in the tools menu.

Click next in the wizard until you see the following page.

Then select Translate cryptographic fields and click Finish.

Once done you can load the log file from the Files menu

Browse for the log file and click OK

You get then a result like this:

In the cryptographic fields, the numeric values have been replaced by names. You see in the previous screenshot web requests done by IE9 on Windows 7, IE11 on Windows server 8.1, FireFox 57 and Chrome 63. You can see that IE 9 still uses TLS 1.0 when the other browsers are using TLS 1.2.

You can also easily display statistics on the CryptProtocol field thanks to the field statistics panel like in the following screenshot.

You can then easily determine how much traffic goes through the web site with the TLS 1.0 protocol. You can click on SP_PROT_TLS1_SERVER in the statistics to automatically only display web requests encrypted with TLS 1.0 and then you can go over all client IP addresses or client web browsers to find out who or what is generating this traffic. You can also use the reverse DNS feature of the HttpLogBrowser to get the involved client hostnames if needed.

Now you can start finding problematic web clients and start upgrading what needs to be upgraded. This doesn’t mean just upgrading all web browsers accessing the web server. Custom client applications may consume a web service running on the server. Those applications may be based for example on an old version of the .NET framework. So such applications may need to be rebuilt against a more recent version of the .NET framework by the developer to be able to use a recent version of TLS.

Once you no longer get traffic for weak cryptographic protocols (TLS 1.0, SSL 3.0, …) after a period of time long enough you can consider disabling them but before some additional verifications are needed.

Verify if other applications use weak protocols

At this step we are sure that the IIS web server no longer relies on weak protocols. However other applications may rely on them.

For example you may have an FTP server configured in IIS. Unfortunately there is no way to audit which encryption protocols are used in FTP sessions. It’s even not possible to identify which FTP clients are used because there is no such as a user agent string like for the HTTP protocol in the FTP logs. The only way, is to inform all users of the FTP server to use an up to date FTP client supporting TLS 1.2.

The RDP protocol for remote desktop sessions may also use TLS encryption depending on how the encryption is configured in RDP settings and negotiated with the client. So you should also make sure that all RDP clients accessing the web server are up to date.

This article only concerns Windows Server 2012 R2 and Windows 2016 but as an illustration if you enable TLS 1.2 on Windows Server 2008 R2 (disabled by default) the uploads will stop working in encrypted FTP sessions due to a bug in the TLS 1.2 implementation and if you disable TLS 1.0 the Remote Desktop may fail if RDP is configured to only use TLS and the proper patch is not installed.
On Windows Server 2012 R2 and Windows 2016 you should not have these problems but this illustrates the implications when you move from old encryption protocols and also illustrates the need of full regressions tests.

Disable weak protocols with IIS Crypto

Now that you are sure that all weak cryptographic protocols are no longer used you can disable them. The official Microsoft documentation explains how to do this through the registry but this may be cumbersome and prone to mistakes. Instead you can use the free tool IIS Crypto from Nartac Software. You can download it from this page. Select the version with the GUI. Once downloaded execute the file directly. No installation is required.

This is how this tool looks like when started for the first time on the web server. All check boxes are grayed meaning default Windows settings are effective.

You could unselect manually all weak protocols and algorithms but instead you can switch to the Templates view and select for example the predefined template PCI 3.1 that will unselect for you all the weak protocols.

You can then see the result in the following screenshot. Only TLS 1.1 and 1.2 are allowed.

Now you just need to click on Apply and restart the server. If you did all your preparation work all should work perfectly after the reboot.

Conclusion

You have learned in this article how to enable cryptographic fields in the IIS HTTP logging of Windows server 2012 R2/2016 and then how to use the free edition of the HttpLogBrowser to identify weak TLS usage in order to later disable these deprecated protocols with another free tool IIS Crypto without breaking anything on your web server. If you need more information on the subject you will find hereafter a few links that may be of interest to you.

Known SSL/TLS Attacks: POODLE, DROWN, FREAK, Logjam, BEAST
NIST: NIST Revises Guide to Use of Transport Layer Security (TLS) in Networks
PCI DSS: Are You Ready for 30 June 2018? Saying Goodbye to SSL/early TLS, Migrating from SSL and Early TLS
Microsoft: Solving the TLS 1.0 Problem

HttpLogBrowser 2.0 final release available!

Thu, 23 Nov 2017 11:37:00 GMT

The final release of the HttpLogBrowser 2.0 is now available. Thanks to everybody that sent me feedback to help me improve this version during the beta period. If you are still using version 1.0 of the application, you can read what’s new in this major version on the following page:

What’s new in version 2?

This version also includes a professional edition that will be on sale soon. Until then you are able to use features of the professional edition thanks to the 30 days evaluation period.

Additional features included in the Professional edition

If you already installed the Release Candidate you’ll find hereafter the minor changes added in the final release:

Fixed: Windows were not always correctly restored to their last position
Added: Ability to display/hide time statistics from the web statistics report by selecting/unselecting the EvenTime field.
Updated: Final touches to the help file. The two following use cases have been added:

Download HttpLogBrowser 2.0

HttpLogBrowser 2.0 Release Candidate released

Mon, 13 Nov 2017 14:53:00 GMT

We get closer to the final release of the version 2.0 of the HttpLogBrowser. The Release Candidate was just released. If you do not know yet what’s new in version 2 you can read my post announcing the beta of the version 2.
You can see hereafter what’s specifically new in this release:

Added: Translation of cryptographic fields to help detecting weak TLS usage in IIS. These cryptographic fields need to be enabled in IIS as explained in the following article:
New IIS functionality to help identify weak TLS usage
Fixed: The URL exclusion was not working.
Fixed: Customized column widths were lost after an application restart
Fixed: Applying the error 500 filter after clicking on the error 500 notification was making the user interface hang.
Changed: The free evaluation of the professional edition is now automatically started for 30 days once you enable it. Existing users of the professional edition will automatically get 30 additional days of evaluation with this release.
Changed: The license dialog box and the evaluation expired dialog box to propose to get an extended evaluation key once the standard evaluation period has expired.

Download the Release Candidate

HttpLogBrowser 2.0 beta 4 released

Wed, 11 Oct 2017 14:12:00 GMT

The beta 4 of HttpLogBrowser 2.0 was released. You can find hereafter what’s new in this version:

Fixed: The tracking field SinceDays had incorrect values when using the cache mode. The problem was introduced in the beta 2. Users of the beta 2 and beta 3 will need to reset the cache to correct the problem in existing data.
Improved: The user is automatically prompted to specify limits when about to load too much data.
Changed: The ability to specify a period in days to load also applies now when loading specific files and not only a whole folder.
Changed: The free evaluation of the professional edition was extended until 15th November

Download HttpLogBrowser 2.0 beta 4

Main new features of version 2.0

HttpLogBrowser 2.0 beta 3 released

Mon, 02 Oct 2017 14:16:53 GMT

The beta 3 of HttpLogBrowser 2.0 was just released. If you do not know yet what’s new in version 2 you can read my post announcing the beta of the version 2.
You can see hereafter what’s new in the beta 3:

Fixed: The cache mode of the professional edition was creating duplicate entries. The problem was introduced in the beta 2. Users of the beta 2 will need to reset the cache file of all profiles to remove existing duplicate entries.
Fixed: It was possible to enable the cache mode even that log settings were not saved in a profile.

Download the beta 3 version

HttpLogBrowser 2.0 beta 2 has been released

Fri, 22 Sep 2017 16:20:00 GMT

The beta 2 of the HttpLogBrowser 2.0 was just released. If you do not know yet what’s new in version 2 you can read my previous post annoucing the beta of this new major version.
You can read hereafter what’s new in the beta 2:

Improved: Statistics are built against case insensitive string values to be consistent with the filters that are also case insensitive.
Improved: Added the ability to set a maximum number of log rows to load to avoid consuming too much memory. The default value is 1 million log rows.
Fixed: Crash when exporting field statistics
Fixed: The time unit was not correct in detailed field statistics after changing the time filter
Improved: The time selection controls were replaced with new ones more consistent with the UI skin.
Fixed: The W3C field s-computername did not have a display name.
Added: A draft of the documentation of version 2 new features
Changed: The free evaluation of the professional edition was extended until end October

Download the beta 2 version

HttpLogBrowser 2.0 beta is available!

Thu, 31 Aug 2017 09:54:00 GMT

After several months of development the new version of the HttpLogBrowser is here. There is a bunch of new features and improvements for this free application but with this new version also comes a professional edition that will be on sale as soon as the beta period is over. Don’t hesitate to send any feedback or problem at support@finalanalytics.com or by adding a comment to this blog post. If a previous version is installed there is no need to uninstall it, just launch the setup and an upgrade will automatically take place.

Download the beta version

Now, I let you discover what’s new in this version.

New features

Good news if you are running web sites on Apache. You can now analyze Apache access logs. The supported formats are the common log format and the combined log format.
The program can now also determine in which country client IP addresses are located. If the option is selected you will see a new column Country in the log rows view.
More, the program can resolve client IP addresses to their host name. Two fields are then added. ClientHostName (full host name) and ClientHostDomain (with only the domain of the internet access provider).
If you need more information on a field you can now display a detailed field statistics window with the complete list of all values for the field and with time activity information (first/last activity time and activity duration) for each value.
In the detailed field statistics window you will also be able to display field evolution statistics in terms of number of different values or average/sum for numeric fields (e.g. in the previous screenshot the number of different Client IP addresses per month).
Improved analysis of numeric fields: In the new version you will also see the ranges of numeric values in the statistics data and will be able to easily display all web requests in a specific range or with a value greater than a specific value (e.g. requests taking more than 10000 ms). The histogram is also more interactive and allows you to display the percentage of requests with a field value less than the value at the cursor position.
With this new version you are also allowed to specify a tracking field (e.g. client IP address or a specific extracted cookie) and copy specific fields value of the first request of a visitor to all successive requests (e.g. in the following screenshot the fields RefererSite and UrlPath are copied under the new names Source.RefereSite and Source.UrlPath on all request from the same visitor).

In the result you will also see a column Source.EvenTime with the time of the landing request of this visitor, a field Source.SinceDays with the number of days since the landing request and Source.RequestNumber with the number of requests done by the visitor.
With such a feature you will be able to quickly know from where a visitor doing a specific request (e.g. downloading a file) came from, when and where he landed and how much activity he did since the landing.

Improvements

Latest filtered views are kept in cache so if you go back in the filter history the data of an already viewed filter is displayed faster.
Ability to directly filter exclusively on a field value instead of adding the value filter to the existing filter.
New more comprehensive field names. Example the W3C field cs-uri-stem is now UrlPath and the field s-bytes is now BytesReceived. In the case you prefer W3C field names you can still switch back to the original names.
Filter expressions are no longer generated with brackets arround the field name
Improved Time Statistics chart by allowing to change the time unit. In the previous version the time unit was determined automatically and you could not be changed.
Ability to filter fields to hide unused fields. With all the extracted fields there are very many fields for a web request and you may take some time to find the field you are interested in in the field statistics. So you can now specify in the field filter parts of field names to display all fields including one of the part in their name.
New field Request Status including a textual description of the HTTP status and sub-status and the meaning of the Win32 error code. This is very useful if you are not familiar with all the HTTP status codes.
The position of the vertical and horizontal splitters in the main window is kept

New features of the Professional edition

The professional edition of the HttpLogBrowser comes with the following additional features:

Favorites filters: You will be able to save your most used filters inside the application in order to reuse them quickly later from the Favorites menu
Log files cache: The application can keep parsed logs data in a binary file in order to reload the data later 5 times faster. If new log rows are available only these new rows will need to be parsed.
Server errors warning: If the application sees HTTP errors 5xx in a log file being loaded a notification is displayed to warn you and by clicking on the message you can filter the view to see these errors.
Load more than 2 million log rows: If you are running the application on a 64 bits edition of Windows you will be able to load an unlimited number of log rows. The only limit will come from the available RAM and from your patience for waiting the log rows to be loaded.
Export filtered log rows: You can export filtered log rows in .csv/.xls/.xlsx files.
Export statistics: You can export field statistics in .csv/.xls/.xlsx files
Generate statistics reports: You can generate printable field statistics reports. You can then export them in PDF to send them to other people. See an example of report exported in PDF.

The professional edition is free during the beta test period so don't hesitate to give it a try.

Download the beta version

HttpLogBrowser 1.01 released!

Mon, 10 Jul 2017 11:20:30 GMT

The version 1.01 of the HttpLogBrowser is available. This release includes the following bug fixes:

Fixed: The count for the last bar of the time statistics histogram was wrong
Fixed: When there were more than 50 different values for a field, the remaining values count was wrong in the pie chart leading to wrong percentages.
Fixed: The first entry of the filter history was missing when loading selected log files.
Fixed: In some cases fields were not correctly extracted from the cookie field

If version 1.0 is installed there is no need to uninstall it, just launch the setup and an upgrade will automatically take place.

Download the lastest version of the HttpLogBrowser