DevTech101

DevTech101
1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 1.00 out of 5)
Loading...

Configuring Nginx Logging - Headers, Body, And Millisecond Timing, Using Nginx LUA Module.

Nginx is one of the popular choices used as as a Web Server, Proxy Server and Load Balancer. Therefore it’s impotent to configure properly to log and capture as much log data as possible.

One would think such a simple task would be simple to to configure. For the most part the answer is YES.

However, to rely get the most benefit and value of the log data, a number of  configuration changes are necessary, making the configuration not so simple anymore. But don’t worry, I will go over the full configuration and changes necessary below. making it as smooth, simple and hopefully an easy process.

Some of the benefits and missing pieces with the default configuration are listed below.

  1. Nginx logging with timestamp is not captured time in milliseconds, a very helpful feature, especially in web sites with lots of traffic. 
  2. Capturing all headers – split in separate fields of request-header and response-header.
  3. Capturing full body is not possible out of the box, and is very helpful in many cases.
  4. Full body capture can be overwhelming, so limiting the body capture by the number of characters.
  5. Splitting request-body and response-body captures in separate fields.
  6. Filter out / ignore AWS Load Blanacer health requests.

In order to address all these missing pieces. We will be using the Nginx LUA Module (for the most part).

Note: The Nginx LUA module is not included in the Ubuntu official repository.

Below I will be showing you, step by step on how to install and configure the Nginx LUA module with your Nginx configuration.

First things first, lets make sure we are using an Nginx version that supports the LUA module.

You do so by running the below

Check your Nginx version by running nginx -v  and make sure we are using version 1.23.2. don’t worry I will be showing you below how to do so.

# Default Ubuntu 22.04 version
$ nginx -v
nginx version: nginx/1.18

# After the update version
$ nginx -v
nginx version: nginx/1.23.2

Lets add the Nginx ppa repository which will be required to updated the Nginx version and LUA Modules

Create a file /etc/apt/sources.list.d/ondrej-ubuntu-nginx-mainline-jammy.list  – contents should be like the one below 

sudo cat /etc/apt/sources.list.d/ondrej-ubuntu-nginx-mainline-jammy.list
deb https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu/ jammy main
# deb-src https://ppa.launchpadcontent.net/ondrej/nginx-mainline/ubuntu/ jammy main

# Refresh the apt packge list
apt update

Note: A word of caution before continuing to install/update your Nginx version. Make sure to create a BACKUP of your Nginx configuration, as the new version might below avway your configuration. 

To create a backup you can simply run something like the below.

zip -rqq /etc/nginx-backup.zip /etc/nginx

Once a backup was created, we will be installing the below 3 packages.

  1. nginx
  2. libnginx-mod-http-lua
  3. lua-cjson
  4. jq

You can do so by running the below.

# At time of this writing the nginx version was - nginx 1.23  
apt install nginx libnginx-mod-http-lua lua-cjson

# optional Install jq to make easier JSON parsing
apt  install jq

Once installation is completed – before we modify any configuration, lets make sure the  current nginx is not broken. you can do so by running the below.

# At time of this writing the nginx version was - nginx 1.23  
$ nginx -v
nginx version: nginx/1.23.2

# Also check for configuration errors (always a good practice)
nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Next we are going to included the LUA cjson  module in the nginx configuration by adding the below line to the top of /etc/nginx/nginx.conf file.

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;   # <<<-- Make sure to add this line prioir the cjson blocks

Do you need help with this configuration ? Just let us know.

Use the Contact Form to get in touch with us for a free Consultation.

We are now ready to update the rest of the configuration.

Below I will try to break-out and explain each part of the configuration.

Note: The below configuration(s) goes in the /etc/nginx/nginx.conf file, under the http { … block

The section below adds – maiking sure to ignore any loggs originating from AWS health checks – as this might clutter your access log (this is only necessary if you are using an AWS Load Balancers in front of your Nginx instances).

http {

    ##
    # Basic Settings
    ##
.... 

    ##
    # Logging Settings
    ##
    map $http_user_agent $ignore_ua {
        default                 1;
        "ELB-HealthChecker/2.0" 0;
    }

The next part adds the capability of capturing upstream status, time, etc..

    map $upstream_status $upstream_status_rt {
      default $upstream_status;
      ""      0;
    }
    map $upstream_response_time $upstream_response_time_rt {
      default $upstream_response_time;
      ""      0;
    }
    map $upstream_connect_time $upstream_connect_time_rt {
      default $upstream_connect_time;
      ""      0;
    }
    map $upstream_header_time $upstream_header_time_rt {
      default $upstream_header_time;
      ""      0;
    }

The next part adds the capability to capture log headers.

    #######################
    # Log full headers using lua
    #######################

    map $req_headers $req_headers {
      default $req_headers;
      ""      0;
    }
    map $resp_headers $resp_headers {
      default $resp_headers;
      ""      0;
    }

    header_filter_by_lua_block {
      local cjson = require "cjson"

      local h = ngx.req.get_headers()
      ngx.var.req_headers = cjson.encode(h)

      local rh = ngx.resp.get_headers()
      ngx.var.resp_headers = cjson.encode(rh)
    }

The next part adds the capability to capture and  log the full body (and limit the number of characters captured if necessary).

    #######################
    # Log request_body using lua
    #######################

    lua_need_request_body on;

    map $resp_body $resp_body {
      default $resp_body;
      ""      0;
    }

    body_filter_by_lua '
        local resp_body = string.sub(ngx.arg[1], 1, 1000)
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
            ngx.var.resp_body = ngx.ctx.buffered
        end
    ';

The below adds a field to capture logs in milliseconds instead of just seconds.

    map "$time_local:$msec" $time_local_ms { ~(^\S+)(\s+\S+):\d+\.(\d+)$ $1.$3$2; }

Finally, we create a log capture with all new fields  formats as an output stream.

Note: We are capturing all logs in a JSON format seen below by adding the escape=json this will helps for systems like Elasticsearch or Splunk for easy log ingestion.

    map "$time_local:$msec" $time_local_ms { ~(^\S+)(\s+\S+):\d+\.(\d+)$ $1.$3$2; }
    log_format main escape=json
        '{'
           '"time":"$time_local_ms",'
           '"client":"$http_x_forwarded_for",'
           '"method":"$request_method",'
           '"request":"$request",'
           '"request_length":$request_length,'
           '"status":$status,'
           '"bytes_sent":$bytes_sent,'
           '"body_bytes_sent":$body_bytes_sent,'
           '"referer":"$http_referer",'
           '"user_agent":"$http_user_agent",'
           '"upstream_addr":"$upstream_addr",'
           '"upstream_status":$upstream_status_rt,'
           '"request_time":$request_time,'
           '"upstream_response_time":$upstream_response_time_rt,'
           '"upstream_connect_time":$upstream_connect_time_rt,'
           '"upstream_header_time":$upstream_header_time_rt,'
           '"request_body":"$request_body",'
           '"resp_body":"$resp_body",'
           '"req_headers":"$req_headers",'
           '"resp_headers":"$resp_headers"'
        '}';

    access_log /var/log/nginx/access.log main;
    error_log /var/log/nginx/error.log;

Note: In order to get the remote client IP Address if using a cloud providers such as AWS ELB I configured the client field to use the $http_x_forwarded_for,  this can be changed back to the default $remote_addr if necessary.

Putting it all together, would look similar to  something like the one below.

    ##
    # Logging Settings
    ##
    
    map $http_user_agent $ignore_ua {
        default                 1;
        "ELB-HealthChecker/2.0" 0;
    }

    # Fix for empty (upstream_status, upstream_response_time, upstream_connect_time, upstream_header_time)
    map $upstream_status $upstream_status_rt {
      default $upstream_status;
      ""      0;
    }
    map $upstream_response_time $upstream_response_time_rt {
      default $upstream_response_time;
      ""      0;
    }
    map $upstream_connect_time $upstream_connect_time_rt {
      default $upstream_connect_time;
      ""      0;
    }
    map $upstream_header_time $upstream_header_time_rt {
      default $upstream_header_time;
      ""      0;
    }

    #######################
    # Log full headers using lua
    #######################

    map $req_headers $req_headers {
      default $req_headers;
      ""      0;
    }
    map $resp_headers $resp_headers {
      default $resp_headers;
      ""      0;
    }

    header_filter_by_lua_block {
      local cjson = require "cjson"

      local h = ngx.req.get_headers()
      ngx.var.req_headers = cjson.encode(h)

      local rh = ngx.resp.get_headers()
      ngx.var.resp_headers = cjson.encode(rh)
    }

    #######################
    # Log request_body using lua
    #######################

    lua_need_request_body on;

    map $resp_body $resp_body {
      default $resp_body;
      ""      0;
    }

    body_filter_by_lua '
        local resp_body = string.sub(ngx.arg[1], 1, 1000)
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
            ngx.var.resp_body = ngx.ctx.buffered
        end
    ';

    # MS format
    map "$time_local:$msec" $time_local_ms { ~(^\S+)(\s+\S+):\d+\.(\d+)$ $1.$3$2; }
    
    log_format main escape=json
        '{'
           '"time":"$time_local_ms",'
           '"client":"$http_x_forwarded_for",'
           '"method":"$request_method",'
           '"request":"$request",'
           '"request_length":$request_length,'
           '"status":$status,'
           '"bytes_sent":$bytes_sent,'
           '"body_bytes_sent":$body_bytes_sent,'
           '"referer":"$http_referer",'
           '"user_agent":"$http_user_agent",'
           '"upstream_addr":"$upstream_addr",'
           '"upstream_status":$upstream_status_rt,'
           '"request_time":$request_time,'
           '"upstream_response_time":$upstream_response_time_rt,'
           '"upstream_connect_time":$upstream_connect_time_rt,'
           '"upstream_header_time":$upstream_header_time_rt,'
           '"request_body":"$request_body",'
           '"resp_body":"$resp_body",'
           '"req_headers":"$req_headers",'
           '"resp_headers":"$resp_headers"'
        '}';

    access_log /var/log/nginx/access.log main;
    error_log /var/log/nginx/error.log;

We are almost done. Lets just make sure there are no errors in the configuration, by running the below.

nginx -t 

If there are no errors, we are now redye to start using the new configuration. 

Lets restart the service for the new changes to take effect.

you do so by running the below.

systemcctl restart nginx

The default location of Nginx logs are in /var/log/nginx/access..

Logs should look similar to the ones below.

Note: I am using jq for nicer formatting.

{
  "time": "02/Jan/2023:19:55:18.169 +0000",
  "client": "108.29.90.240",
  "method": "GET",
  "request": "GET /linux-bond-and-vlans?share=pinterest HTTP/1.1",
  "request_length": 473,
  "status": 302,
  "bytes_sent": 964,
  "body_bytes_sent": 31,
  "referer": "https://www.devtech101.com/2016/06/08/linux-bond-and-vlans",
  "user_agent": "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)",
  "upstream_addr": "unix:/run/php/php7.4-fpm.sock",
  "upstream_status": 302,
  "request_time": 0.389,
  "upstream_response_time": 0.388,
  "upstream_connect_time": 0,
  "upstream_header_time": 0.388,
  "request_body": "",
  "resp_body": "\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0003\u0003\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
  "req_headers": "{\"host\":\"www.devtech101.com\",\"accept-language\":\"en\",\"connection\":\"Close\",\"accept\":\"text\\/html,application\\/xhtml+xml,application\\/xml;q=0.9,*\\/*;q=0.8\",\"accept-encoding\":\"gzip, deflate\",\"referer\":\"https:\\/\\/www.devtech101.com\\/2016\\/06\\/08\\/linux-bond-and-vlans\",\"user-agent\":\"Mozilla\\/5.0 (Linux; Android 7.0;) AppleWebKit\\/537.36 (KHTML, like Gecko) Mobile Safari\\/537.36 (compatible; PetalBot;+https:\\/\\/webmaster.petalsearch.com\\/site\\/petalbot)\"}",
  "resp_headers": "{\"x-redirect-by\":\"WordPress\",\"content-type\":\"text\\/html; charset=UTF-8\",\"connection\":\"close\",\"cache-control\":\"max-age=3600, public\",\"pragma\":\"public\",\"last-modified\":\"Mon, 02 Jan 2023 19:55:18 GMT\",\"vary\":\"Accept-Encoding\",\"set-cookie\":[\"ultp_view_1338=1; expires=Tue, 03-Jan-2023 19:55:18 GMT; Max-Age=86400; path=\\/\",\"pvc_visits[0]=1672775718b1338; expires=Tue, 03-Jan-2023 19:55:18 GMT; Max-Age=86400; path=\\/; secure; HttpOnly; SameSite=LAX\"],\"content-encoding\":\"gzip\",\"x-robots-tag\":\"noindex, nofollow\",\"location\":\"https:\\/\\/www.pinterest.com\\/pin\\/create\\/button\\/?url=https%3A%2F%2Fwww.devtech101.com%2Flinux-bond-and-vlans%2F&media=https%3A%2F%2Fi0.wp.com%2Fwww.devtech101.com%2Fwp-content%2Fuploads%2F2022%2F12%2Fcropped-devtech101-shape-black.png%3Ffit%3D96%252C96%26ssl%3D1&description=Creating%20a%20Linux%20Bond%20With%20vLan%20Tags\"}"
}

Do you need help with this configuration ? Just let us know.

Like this article. Please provide feedback, or let us know in the comments below.

Use the Contact Form to get in touch with us for a free Consultation.

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x
%d bloggers like this: