How to configure Log rotation in Apache on Centos and backup on AWS S3

Logrotate is a unix utility which helps in administration of systems that generate a large number of log files or large sized log files. One of the most common use cases of logrotate is in the management of web server logs, for example error logs in Apache.

There are some great articles available on internet which talks about how to setup logrotate. Here is the link to a great article on Digital Ocean - how-to-configure-logging-and-log-rotation-in-apache-on-an-ubuntu. I recommend reading this article if you are new to logrotate utility.

For a light weight server, error logs can be as small as few hundred KBs for weeks. On the other hand, log size can be as large as a few GBs per hour for a system with high traffic. In this blog post, I will focus on systems that generate large volume of logs. When the volume of logs is in the order of GBs per hour, system will soon be under space crunch and the mailbox will be flooded with nagios alerts. There is great value in retaining these logs so that they can be consumed later in time. For example, to debug a downtime in the service.

Amazon S3 is the storage service provided by Amazon Web Services. Any amount of data can be stored on S3 and can be retrieved from any where on the web. We will use S3 to backup our logs generated by logrotate. AWS provides a CLI to interact with their services. Installation directions for AWS CLI are here.

Logrotate provides an option to set the frequency of log rotation. On Centos, it was set to 4 weeks. This means that a new log file will be generated every 4 weeks. For the production systems I am working on, we have set the log rotation frequency to one hour. Our requirement is to upload the log files to S3 every hour. When the log files are successfully uploaded to S3, log files should be deleted from the server to make space for next hour's logs.

Here is what one of the entries in logrotate looks like :

/var/log/httpd/error_log {

    firstaction

        find /var/log/httpd -name "error_log-*" -ctime +45 -exec rm -f {} \; -print

    endscript

    copytruncate

    rotate 45

#   compress

    dateext

    missingok

    ifempty

    postrotate

       /sbin/service httpd reload > /dev/null 2>/dev/null || true

    endscript

    lastaction

        export HOME=/root

        time=`date +%Y%m%d-%H-%M`

        date=`date +%Y%m%d`

        hour=`date +%H`

        path=`echo $1 | tr -d ' '`

        server=`hostname | cut -d "." -f1`

        logtype=`basename $path`

        final_top_dir="/var/log/httpd/hourly/error_log/"

        final_hourly_dir="$final_top_dir""dt=$date/hour=$hour/"

        final_filename=$server-$logtype-$time

        final_filepath="$final_top_dir""dt=$date/hour=$hour/$final_filename"

        temp_path="/var/log/httpd/$logtype-$time"

        echo "temp_path: $temp_path"

        mkdir -p `dirname $final_filepath`

        mv $path-$date $temp_path

        echo -e "Logrotation of $logtype @ $time"

        echo '--'

        echo "Splitting log file"

        startts_split=`date +%s`

        q="split -b 1G $temp_path $final_filepath-"; echo $q; $q;

        endts_split=`date +%s`

        echo "Time to split :"$(($endts_split-$startts_split))" seconds"

        echo "Time now:" `date +%Y%m%d-%H-%M-%S`

        rm -f $temp_path

        s3_top_dir=s3://mycompanylogs/error_log/

        s3_hourly_dir="$s3_top_dir""$date/hour=$hour/"

        echo '--'

        echo "Syncing logs"

        startts_sync=`date +%s`

        nMaxAttempts=3

        attemptCount=0

        while [ "$attemptCount" -lt "$nMaxAttempts" ]; do

            echo "Attempt Number: $attemptCount";

            q="/usr/bin/trickle -s -u10000 /usr/bin/aws s3 sync $final_hourly_dir $s3_hourly_dir"; echo $q; $q

            if [ "$?" == "0" ]; then

                echo "Successfully synced."

                break;

            else

                echo "Failed. Retry needed. [$?]"

            fi

            attemptCount=$(($attemptCount + 1))

        done

        endts_sync=`date +%s`

        echo "Time to sync :"$(($endts_sync-$startts_sync))" seconds"

        echo "Time now:" `date +%Y%m%d-%H-%M-%S`

    endscript

}

The above configuration is overwhelming but allow me to break it down. The code between "firstaction" and "endscript" is the first piece of code that is executed. We are instructing to delete any log file which is more than 45 days older.

The code between "lastaction" and "endscript" contains the crux of the code. This part of the code executes after the log rotation has already taken place. the name of the rotated file is "$path-$date" where $path and $date are as mentioned in the logrotate config.

Splitting the file
As you can notice that we are splitting the file. On AWS S3, there are various restrictions on files which are larger than 5 GB. We have decided that size of one log file should not be larger than 1 GB. So we split the file into chunks of 1GB each.

Bandwidth Limiting 
We do not want log rotation to take unproportionate amount bandwidth in uploading logs as it might affect a service running on this machine. We use trickle to cap the bandwidth.

Aws CLI
aws s3 sync is used to sync the logs of a particular hour to S3.

Retries
As apparent from the 'while' loop above that we are trying to sync same set of files 3 times. This is to make sure that if sync fails for the first time, there is a chance that it will work the second time. In general, retries is a common practise to make reliable systems over unreliable network.