Synchronize IIS HTTP logs

In a previous article I explained how to share IIS HTTP logs either through the Microsoft file and printer sharing when possible or through the FTP server in order to be able to retrieve those logs locally. Retrieving them manually every time a new file is available is cumbersome. So in this article I will give you solutions on how to automate this.

Free folder synchronization tools

My first intend by writing this article was to find a free folder synchronization tool and demonstrate how to use it to synchronize IIS HTTP log files.

So I listed some known folder synchronization tools:

SyncToy

This tool is provided by Microsoft and is very easy to use. Unfortunately it doesn’t support the FTP protocol. So you can only use it if your IIS server is on your local network and log files are available on a Windows share.

https://www.microsoft.com/en-us/download/details.aspx?id=15155

FreeFileSync

Update! FreeFileSync now supports the FTP protocol so the following section is no longer true. 

This tool has more options and can use more protocols like SFTP to synchronize folders. Unfortunately the FTP protocol is not supported and Windows doesn’t provide a SFTP server out of the box. Installing a third party SFTP server on the web server would be complicate even impossible for an Azure web app.

https://www.freefilesync.org/

FullSync

This program supports folder synchronization through the FTP protocol. Unfortunately it doesn’t support encryption over FTP (FTPS). So using it is not very secure.

https://fullsync.sourceforge.io/

FTPBox

On the paper this one should do the job. However I was unable to use it. The program was hanging and unusable.

http://ftpbox.org/

LogSync command line tool

With the lack of free folder synchronization tools supporting the FTPS protocol I decided to develop myself a simple command line tool to do the job. With this program, you can for example make a shortcut that will allow you to launch the synchronization before analyzing your logs.

The tool can be downloaded at the following URL and is free to use:

https://www.finalanalytics.com/downloads/LogSync.zip

Features

Optimized data transfer

The most interesting feature of LogSync is that when it detects that a log file has increased in size, only the additional data will be transferred. Of course, this is only working with text log files for which successive writes are added at the end of the file. That’s why by default only files with a .log extension will be synchronized by the program. You can add other extensions if needed with the –e switch (e.g. –e log,txt)

The benefit of this feature is that it is minimizing the amount of data transferred during the synchronization. For example if you are synchronizing several times a day log files from an IIS web site and the log rotation is the day, this will avoid downloading the whole current log file again every time. Just remember that on Windows 2008 R2 to 2012 R2 you need to install a patch to be able to retrieve the log file currently being written. See the solution in my previous article on how to share IIS HTTP log files.

Supported remote log folder share technologies

• The tool was tested with the IIS FTP server and the FTP site of Azure web apps but should work with any kind of FTP server.
• Windows shares are supported through the file system synchronization command

Prerequisites

• Runs on Windows Vista/2008 and more
• The .NET Framework 4.0 or more is required.

Known limitations

• For the FTP synchronization, encryption is required (FTPS) and also authentication.
• The program is not optimized to do massive file transfer. So if you already have a big number of log files in the remote folder, I advise to first transfer all existing files with FileZilla for example and then only, do regular synchronizations with a LogSync command line.

How to use the LogSync

The command line tool has three commands. Below is explained how to use them.

Encrypt the password

If you want to synchronize logs from a FTP server, the first thing to do is to encrypt and store the password of the FTP account. You do that with the following command line:

LogSync encrypt -n %PasswordName%

You need to specify a name for the password to encrypt. Later you will need to specify that same name in order to use the encrypted password in a synchronization command.

Once the command is launched the tool will prompt you to enter the password in clear

And once the password has been validated it will display a confirmation message that the password was encrypted and stored.

The password is encrypted with the Windows DPAPI (Data Protection API) and can only be decrypted by the account that encrypted it. Take this into account if you want to schedule the synchronization with a scheduled task running under another account. In this case you need to run the encryption command with this specific account.

Run a FTP synchronization

Once the password has been encrypted and stored you can run FTP synchronizations with the following command line:

LogSync ftpsync -h %FtpServerAddress% -l %LocalFolderPath% -r %RemoteFolderPath% -u %FtpUser% -p %PasswordName%

You can see an example in the following screenshot.

You see in the screenshot that for some files it says updating meaning that only additional data is downloaded and for other it says downloading meaning that the file is new so the whole file was downloaded.

Run a file system synchronization

With the LogSync program you can also do synchronizations through the file system for example when the remote logs are on a Windows share. You do that with the following command line:

LogSync filesystemsync -l %LocalFolderPath% -r %RemoteFolderPath%

You will find an example in the screenshot hereafter.

Conclusion

So you are now able to easily create either a shortcut or a scheduled task that will synchronize your IIS HTTP logs locally. In a next version of the LogSync tool the SFTP protocol will be added in order to also synchronize logs from Apache web sites running on Linux servers. At the same time the version 2.0 of the HttpLogBrowser will be able to read and analyze Apache web access logs.

1 Comment

• jnh said 5 years ago

  Edited! Added a warning about FreeFileSync now supporting the FTP protocol

You must log on to comment.