How to Enable Structured Logging

Good news! There is nothing to do for a brand new install. The log4j-default.xml file that comes with Kinetic Request CE located in the WEB-INF/config directory is already configured to start writing to the structured log files.

Upgrading Kinetic Request CE

If you have customized the log4j.xml or log4j-default.xml file, you'll need to append the following text in your log4j.xml file if you wish to enable the new structured logs. These lines can be added anywhere inside the >log4j:configuration ...< XML element in your customized log4j xml file, but we suggest adding it before the >!-- LOGGERS --< line. For example of an example of where to place the additional lines, take a look at the new WEB-INF/config/log4j-default.xml file that comes with the 2.0.2 Kinetic Request CE war file.

    <!-- Structured Access File Appender -->
    <appender name="structured-access-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.access.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} - %X{access.requestMethod} %X{access.requestPath}%X{access.requestQuery} %X{access.responseStatus} %X{access.responseTime}ms%n"/> 
        </layout> 
    </appender>
    
    <!-- Structured Authentication File Appender -->
    <appender name="structured-authentication-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.authentication.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} - %X{auth.event} %X{auth.strategy} PRINCIPAL:%X{auth.principal} %m%n"/>
        </layout> 
    </appender>
    
    <!-- Structured Application File Appender -->
    <appender name="structured-application-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.application.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} %5p [%t] - %m%n"/> 
        </layout>
        <filter class="com.kineticdata.commons.logging.ContextFilter">
            <param name="KeyToMatch" value="app.requestId" />
            <param name="PatternToMatch" value=".+" />
        </filter>
    </appender>
    
    <!-- Structured System File Appender -->
    <appender name="structured-system-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.system.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %5p [%t] - %m%n"/> 
        </layout>
        <filter class="com.kineticdata.commons.logging.ContextFilter">
            <param name="KeyToMatch" value="app.requestId" />
            <param name="PatternToMatch" value="^$" />
        </filter>
    </appender>

Finally, restart Tomcat or the Kinetic Request CE web application.

CE Structured Logging Variables

Overview

The structured logging variables are variables that are available to use to format a log message in a format you want. The sub-section 'Upgrading Kinetic Request CE' above shows a sample of how these variables, which are highlighted, are used in the log4j-default.xml and log4j.xml files to configure the output format in the structured.access.log, structured.application.log, structured.auth.log, and structured.system.log files.

Variable Definitions

Variable	Definition
access.addressChain	Either the IP addresses listed in the X-Forwarded-For HTTP header or the IP address establishing the request to Kinetic Core if no X-Forwarded-For HTTP header is present. Examples for the IP address establishing the request include a load balancer's IP address, or the client's IP address if no load balancer or reverse proxy is in front of Kinetic Request.
access.requestMethod	The HTTP request method used for the access request
access.requestPath	Portion of the request uri after the host and before the query string, ie `/kinetic/app/api/v1/version	`
access.request	Protocol Protocol used for the request - Example: HTTP/1.1
access.request	Query Provides the request query string, starting with “?”. The request query string is the part of a URL proceeding the request path and the start of the request query is indicated by the question mark character.
access.response	Status The HTTP response code returned from Kinetic Request.
access.response	Time The amount of time taken to process the request on the server side in milliseconds. This does not include any network latency.
app.correlationId	This is an id that is generated for each end user HTTP request, passed between kinetic components to associate internal calls
app.instanceId	A consistent key based upon the server’s network interfaces and webapp location, can be used when aggregating logging to determine the source of the log message
app.requestId	A unique ID generated for each received HTTP request
app.sessionId	A user's session id or NO_SESSION for sessionless interaction; such as HTTP Basic API calls
app.space	The space slug for which the logged event occurred within
app.user	The username of the user associated to the logged event
auth.event	This is the type of the authentication event that occurred. Valid options are LOGIN, LOGIN_FAILURE, LOGOUT, CHECK, EXPIRE.

Because the application always allows local authentication (default CE form based authentication) as a fallback, there will only be LOGIN_FAILURE messages when all of the configured security strategies fail AND local authentication fails (this means pre-authentication strategies like x509 will never have LOGIN_FAILURE messages). CHECK represents an unsuccessful authentication check for a security strategy which indicates the next security strategy will be tried
auth.principal The authentication principal is the unique identifier from an authentication strategy for a user, before they're mapped to a local CE username.
auth.strategy The authentication strategy used for this logged event. Some example strategies are: ldap, ldap/basic, local, local/basic, x509, saml