Class LogFilePatternReceiver

All Implemented Interfaces:
Plugin, Component, org.apache.log4j.spi.OptionHandler, Thresholdable
Direct Known Subclasses:
VFSLogFilePatternReceiver

public class LogFilePatternReceiver extends Receiver
LogFilePatternReceiver can parse and tail log files, converting entries into LoggingEvents. If the file doesn't exist when the receiver is initialized, the receiver will look for the file once every 10 seconds.

This receiver relies on java.util.regex features to perform the parsing of text in the log file, however the only regular expression field explicitly supported is a glob-style wildcard used to ignore fields in the log file if needed. All other fields are parsed by using the supplied keywords.

Features:
- specify the URL of the log file to be processed
- specify the timestamp format in the file (if one exists, using patterns from SimpleDateFormat)
- specify the pattern (logFormat) used in the log file using keywords, a wildcard character (*) and fixed text
- 'tail' the file (allows the contents of the file to be continually read and new events processed)
- supports the parsing of multi-line messages and exceptions - 'hostname' property set to URL host (or 'file' if not available) - 'application' property set to URL path (or value of fileURL if not available) - 'group' property can be set to associate multiple log file receivers

Keywords:
TIMESTAMP
LOGGER
LEVEL
THREAD
CLASS
FILE
LINE
METHOD
RELATIVETIME
MESSAGE
NDC
PROP(key)
(NL)

(NL) represents a new line embedded in the log format, supporting log formats whose fields span multiple lines

Use a * to ignore portions of the log format that should be ignored

Example:
If your file's patternlayout is this:
%d %-5p [%t] %C{2} (%F:%L) - %m%n

specify this as the log format:
TIMESTAMP LEVEL [THREAD] CLASS (FILE:LINE) - MESSAGE

To define a PROPERTY field, use PROP(key)

Example:
If you used the RELATIVETIME pattern layout character in the file, you can use PROP(RELATIVETIME) in the logFormat definition to assign the RELATIVETIME field as a property on the event.

If your file's patternlayout is this:
%r [%t] %-5p %c %x - %m%n

specify this as the log format:
PROP(RELATIVETIME) [THREAD] LEVEL LOGGER * - MESSAGE

Note the * - it can be used to ignore a single word or sequence of words in the log file (in order for the wildcard to ignore a sequence of words, the text being ignored must be followed by some delimiter, like '-' or '[') - ndc is being ignored in the following example.

Assign a filterExpression in order to only process events which match a filter. If a filterExpression is not assigned, all events are processed.

Limitations:
- no support for the single-line version of throwable supported by patternlayout
(this version of throwable will be included as the last line of the message)
- the relativetime patternLayout character must be set as a property: PROP(RELATIVETIME)
- messages should appear as the last field of the logFormat because the variability in message content
- exceptions are converted if the exception stack trace (other than the first line of the exception)
is stored in the log file with a tab followed by the word 'at' as the first characters in the line
- tailing may fail if the file rolls over.

Example receiver configuration settings (add these as params, specifying a LogFilePatternReceiver 'plugin'):
param: "timestampFormat" value="yyyy-MM-d HH:mm:ss,SSS"
param: "logFormat" value="PROP(RELATIVETIME) [THREAD] LEVEL LOGGER * - MESSAGE"
param: "fileURL" value="file:///c:/events.log"
param: "tailing" value="true"

This configuration will be able to process these sample events:
710 [ Thread-0] DEBUG first.logger first - <test> <test2>something here</test2> <test3 blah=something/> <test4> <test5>something else</test5> </test4></test>
880 [ Thread-2] DEBUG first.logger third - <test> <test2>something here</test2> <test3 blah=something/> <test4> <test5>something else</test5> </test4></test>
880 [ Thread-0] INFO first.logger first - infomsg-0
java.lang.Exception: someexception-first
at Generator2.run(Generator2.java:102)

  • Field Details

  • Constructor Details

    • LogFilePatternReceiver

      public LogFilePatternReceiver()
  • Method Details

    • getFileURL

      public String getFileURL()
      Accessor
      Returns:
      file URL
    • setFileURL

      public void setFileURL(String fileURL)
      Mutator
      Parameters:
      fileURL -
    • setCustomLevelDefinitions

      public void setCustomLevelDefinitions(String customLevelDefinitions)
      If the log file contains non-log4j level strings, they can be mapped to log4j levels using the format (android example): V=TRACE,D=DEBUG,I=INFO,W=WARN,E=ERROR,F=FATAL,S=OFF
      Parameters:
      customLevelDefinitions - the level definition string
    • getCustomLevelDefinitions

      public String getCustomLevelDefinitions()
    • isAppendNonMatches

      public boolean isAppendNonMatches()
      Accessor
      Returns:
      append non matches
    • setAppendNonMatches

      public void setAppendNonMatches(boolean appendNonMatches)
      Mutator
      Parameters:
      appendNonMatches -
    • getFilterExpression

      public String getFilterExpression()
      Accessor
      Returns:
      filter expression
    • setFilterExpression

      public void setFilterExpression(String filterExpression)
      Mutator
      Parameters:
      filterExpression -
    • isTailing

      public boolean isTailing()
      Accessor
      Returns:
      tailing
    • setTailing

      public void setTailing(boolean tailing)
      Mutator
      Parameters:
      tailing -
    • isUseCurrentThread

      public final boolean isUseCurrentThread()
      When true, this property uses the current Thread to perform the import, otherwise when false (the default), a new Thread is created and started to manage the import.
      Returns:
    • setUseCurrentThread

      public final void setUseCurrentThread(boolean useCurrentThread)
      Sets whether the current Thread or a new Thread is created to perform the import, the default being false (new Thread created).
      Parameters:
      useCurrentThread -
    • getLogFormat

      public String getLogFormat()
      Accessor
      Returns:
      log format
    • setLogFormat

      public void setLogFormat(String logFormat)
      Mutator
      Parameters:
      logFormat - the format
    • setGroup

      public void setGroup(String group)
      Mutator
    • getGroup

      public String getGroup()
      Accessor
      Returns:
      group
    • setTimestampFormat

      public void setTimestampFormat(String timestampFormat)
      Mutator. Specify a pattern from SimpleDateFormat
      Parameters:
      timestampFormat -
    • getTimestampFormat

      public String getTimestampFormat()
      Accessor
      Returns:
      timestamp format
    • getWaitMillis

      public long getWaitMillis()
      Accessor
      Returns:
      millis between retrieves of content
    • setWaitMillis

      public void setWaitMillis(long waitMillis)
      Mutator
      Parameters:
      waitMillis -
    • getExceptionLine

      private int getExceptionLine()
      Walk the additionalLines list, looking for the EXCEPTION_PATTERN.

      Return the index of the first matched line (the match may be the 1st line of an exception)

      Assumptions:
      - the additionalLines list may contain both message and exception lines
      - message lines are added to the additionalLines list and then exception lines (all message lines occur in the list prior to all exception lines)

      Returns:
      -1 if no exception line exists, line number otherwise
    • buildMessage

      private String buildMessage(String firstMessageLine, int exceptionLine)
      Combine all message lines occuring in the additionalLines list, adding a newline character between each line

      the event will already have a message - combine this message with the message lines in the additionalLines list (all entries prior to the exceptionLine index)

      Parameters:
      firstMessageLine - primary message line
      exceptionLine - index of first exception line
      Returns:
      message
    • buildException

      private String[] buildException(int exceptionLine)
      Combine all exception lines occuring in the additionalLines list into a String array

      (all entries equal to or greater than the exceptionLine index)

      Parameters:
      exceptionLine - index of first exception line
      Returns:
      exception
    • buildEvent

      private org.apache.log4j.spi.LoggingEvent buildEvent()
      Construct a logging event from currentMap and additionalLines (additionalLines contains multiple message lines and any exception lines)

      CurrentMap and additionalLines are cleared in the process

      Returns:
      event
    • process

      protected void process(BufferedReader bufferedReader) throws IOException
      Read, parse and optionally tail the log file, converting entries into logging events.

      A runtimeException is thrown if the logFormat pattern is malformed.

      Parameters:
      bufferedReader -
      Throws:
      IOException
    • createPattern

      protected void createPattern()
    • passesExpression

      private boolean passesExpression(org.apache.log4j.spi.LoggingEvent event)
      Helper method that supports the evaluation of the expression
      Parameters:
      event -
      Returns:
      true if expression isn't set, or the result of the evaluation otherwise
    • processEvent

      private Map processEvent(MatchResult result)
      Convert the match into a map.

      Relies on the fact that the matchingKeywords list is in the same order as the groups in the regular expression

      Parameters:
      result -
      Returns:
      map
    • convertTimestamp

      private String convertTimestamp()
      Helper method that will convert timestamp format to a pattern
      Returns:
      string
    • setHost

      protected void setHost(String host)
    • setPath

      protected void setPath(String path)
    • getPath

      public String getPath()
    • initialize

      protected void initialize()
      Build the regular expression needed to parse log entries
    • updateCustomLevelDefinitionMap

      private void updateCustomLevelDefinitionMap()
    • isInteger

      private boolean isInteger(String value)
    • quoteTimeStampChars

      private String quoteTimeStampChars(String input)
    • singleReplace

      private String singleReplace(String inputString, String oldString, String newString)
    • getShortPropertyName

      private String getShortPropertyName(String longPropertyName)
    • replaceMetaChars

      private String replaceMetaChars(String input)
      Some perl5 characters may occur in the log file format. Escape these characters to prevent parsing errors.
      Parameters:
      input -
      Returns:
      string
    • convertToEvent

      private org.apache.log4j.spi.LoggingEvent convertToEvent(Map fieldMap, String[] exception)
      Convert a keyword-to-values map to a LoggingEvent
      Parameters:
      fieldMap -
      exception -
      Returns:
      logging event
    • shutdown

      public void shutdown()
      Close the reader.
    • activateOptions

      public void activateOptions()
      Read and process the log file.