How to Enable Structured Logging

Good news! There is nothing to do for a brand new install. The log4j-default.xml file that comes with Kinetic Request CE located in the WEB-INF/config directory is already configured to start writing to the structured log files.

Upgrading Kinetic Request CE

If you have customized the log4j.xml or log4j-default.xml file, you'll need to append the following text in your log4j.xml file if you wish to enable the new structured logs. These lines can be added anywhere inside the >log4j:configuration ...< XML element in your customized log4j xml file, but we suggest adding it before the >!-- LOGGERS --< line. For example of an example of where to place the additional lines, take a look at the new WEB-INF/config/log4j-default.xml file that comes with the 2.0.2 Kinetic Request CE war file.

    <!-- Structured Access File Appender -->
    <appender name="structured-access-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.access.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} - %X{access.requestMethod} %X{access.requestPath}%X{access.requestQuery} %X{access.responseStatus} %X{access.responseTime}ms%n"/> 
        </layout> 
    </appender>
    
    <!-- Structured Authentication File Appender -->
    <appender name="structured-authentication-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.authentication.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} - %X{auth.event} %X{auth.strategy} PRINCIPAL:%X{auth.principal} %m%n"/>
        </layout> 
    </appender>
    
    <!-- Structured Application File Appender -->
    <appender name="structured-application-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.application.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} %5p [%t] - %m%n"/> 
        </layout>
        <filter class="com.kineticdata.commons.logging.ContextFilter">
            <param name="KeyToMatch" value="app.requestId" />
            <param name="PatternToMatch" value=".+" />
        </filter>
    </appender>
    
    <!-- Structured System File Appender -->
    <appender name="structured-system-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.system.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %5p [%t] - %m%n"/> 
        </layout>
        <filter class="com.kineticdata.commons.logging.ContextFilter">
            <param name="KeyToMatch" value="app.requestId" />
            <param name="PatternToMatch" value="^$" />
        </filter>
    </appender>

Finally, restart Tomcat or the Kinetic Request CE web application.

CE Structured Logging Variables

Overview

The structured logging variables are variables that are available to use to format a log message in a format you want. The sub-section 'Upgrading Kinetic Request CE' above shows a sample of how these variables, which are highlighted, are used in the log4j-default.xml and log4j.xml files to configure the output format in the structured.access.log, structured.application.log, structured.auth.log, and structured.system.log files.

Variable Definitions

VariableDefinition
access.addressChainEither the IP addresses listed in the X-Forwarded-For HTTP header or the IP address establishing the request to Kinetic Core if no X-Forwarded-For HTTP header is present. Examples for the IP address establishing the request include a load balancer's IP address, or the client's IP address if no load balancer or reverse proxy is in front of Kinetic Request.
access.requestMethodThe HTTP request method used for the access request
access.requestPathPortion of the request uri after the host and before the query string, ie `/kinetic/app/api/v1/version`
access.requestProtocol Protocol used for the request - Example: HTTP/1.1
access.requestQuery Provides the request query string, starting with “?”. The request query string is the part of a URL proceeding the request path and the start of the request query is indicated by the question mark character.
access.responseStatus The HTTP response code returned from Kinetic Request.
access.responseTime The amount of time taken to process the request on the server side in milliseconds. This does not include any network latency.
app.correlationIdThis is an id that is generated for each end user HTTP request, passed between kinetic components to associate internal calls
app.instanceIdA consistent key based upon the server’s network interfaces and webapp location, can be used when aggregating logging to determine the source of the log message
app.requestIdA unique ID generated for each received HTTP request
app.sessionIdA user's session id or NO_SESSION for sessionless interaction; such as HTTP Basic API calls
app.spaceThe space slug for which the logged event occurred within
app.userThe username of the user associated to the logged event
auth.eventThis is the type of the authentication event that occurred. Valid options are LOGIN, LOGIN_FAILURE, LOGOUT, CHECK, EXPIRE.

Because the application always allows local authentication (default CE form based authentication) as a fallback, there will only be LOGIN_FAILURE messages when all of the configured security strategies fail AND local authentication fails (this means pre-authentication strategies like x509 will never have LOGIN_FAILURE messages). CHECK represents an unsuccessful authentication check for a security strategy which indicates the next security strategy will be tried
auth.principal The authentication principal is the unique identifier from an authentication strategy for a user, before they're mapped to a local CE username.
auth.strategy The authentication strategy used for this logged event. Some example strategies are: ldap, ldap/basic, local, local/basic, x509, saml