Chapter 6: Filters

Have lots of ideas and throw away the bad ones. You aren't going to have good ideas unless you have lots of ideas and some sort of principle of selection.

—LINUS PAULING

As we have seen, logback has several built-in ways for filtering log requests, including the context-wide filter, logger-level selection rule and appender filters. These provide high performance filtering for the most commonly encountered cases. These filters are largely inspired from Linux ipchains or iptables as they are called in more recent Linux kernels. Logback filters are based on ternary logic allowing them to be assembled or chained together to compose an arbitrarily complex filtering policy.

There are two main types of filters, namely Filter and TurboFilter.

Logback Classic

Filter objects all extend the Filter abstract class. The decide(Object event) method is passed a newly created ILoggingEvent instance.

Filter chains

This abstract class assumes that filters are organized in a linear chain. Its member field next points to the next filter in the chain, or null if there are no further filters in the chain. Figure 6.1 depicts a sample filter chain consisting of three filters.

A sample filter chain

Filters are based on ternary logic. The decide(Object event) method of each filter is called in sequence. This method returns one of the FilterReply enumeration values, i.e. one of FilterReply.DENY, FilterReply.NEUTRAL or FilterReply.ACCEPT. If the returned value is FilterReply.DENY, then the log event is dropped immediately without consulting the remaining filters. If the value returned is FilterReply.NEUTRAL, then the next filter in the chain is consulted. If there are no further filters to consult, then the logging event is processed normally. If the returned value is FilterReply.ACCEPT, then the logging event is processed immediately skipping the remaining filters.

In logback-classic Filter objects can only be added to Appender instances. By adding filters to an appender you can filter events by various criteria, such as the contents of the log message, the contents of the MDC, the time of day or any other part of the logging event.

Implementing your own Filter

Creating your own filter is not difficult. All you have to do is extend the Filter abstract class. The only method that you will have to implement is the decide() method, allowing you to concentrate only on the behaviour of your filter.

The next class is all it takes to implement one's own filter. All it does is accept logging events who's message contains the String sample. The filter will give a neutral response to any logging event whose message does not contain this String.

Example 6.1: Basic custom filter (logback-examples/src/main/java/chapter6/SampleFilter.java)
package chapter6;

import ch.qos.logback.classic.spi.LoggingEvent;
import ch.qos.logback.core.filter.Filter;
import ch.qos.logback.core.spi.FilterReply;

public class SampleFilter extends Filter {

  @Override
  public FilterReply decide(Object eventObject) {
    LoggingEvent event = (LoggingEvent)eventObject;
    
    if (event.getMessage().contains("sample")) {
      return FilterReply.ACCEPT;
    } else {
      return FilterReply.NEUTRAL;
    }
  }
}

What is shown above might be the simplest filter. Like any filter, it can be attached to any appender using the <Filter> element, as shown below:

Example 6.2: SampleFilter configuration (logback-examples/src/main/java/chapter6/SampleFilterConfig.xml)
<configuration>
  <appender name="STDOUT"
    class="ch.qos.logback.core.ConsoleAppender">
    <Filter class="chapter6.SampleFilter" />

    <layout class="ch.qos.logback.classic.PatternLayout">
      <pattern>
        %-4relative [%thread] %-5level %logger - %msg%n
      </pattern>
    </layout>
  </appender>
	
  <root>
    <appender-ref ref="STDOUT" />
  </root>
</configuration>

Thanks to Joran, logback's powerful configuration framework, adding an option to such a filter is very easy. Just add the corresponding getter and setter methods in the class, and you can specify the option's value in an xml element, nested within the filter element.

In case you want to implement a filter that provides different behaviour depending on the result of its test (say, a filter that would accept or deny an event depending on the content of its message), you can extend the AbstractMatcherFilter class. It will provide your filter with two properties: OnMatch and OnMismatch, both of which can be configured like any other property.

Logback Filters

At the moment, there are two filters that ship with logback. LevelFilter provides event filtering based on a Level value. If the event's level is equal to the configured level, the filter accepts or denies the event, depending on its configuration. It allows you to choose the behaviour of logback for a precise given level. Here is a sample configuration that uses LevelFilter.

Example 6.3: Sample LevelFilter configuration (logback-examples/src/main/java/chapter6/levelFilterConfig.xml)
<configuration>
  <appender name="CONSOLE"
    class="ch.qos.logback.core.ConsoleAppender">
    <filter class="ch.qos.logback.classic.filter.LevelFilter">
      <level>INFO</level>
      <onMatch>ACCEPT</onMatch>
      <onMismatch>DENY</onMismatch>
    </filter>
    <layout class="ch.qos.logback.classic.PatternLayout">
      <pattern>
        %-4relative [%thread] %-5level %logger{30} - %msg%n
      </pattern>
    </layout>
  </appender>
  <root level="DEBUG">
    <appender-ref ref="CONSOLE" />
  </root>
</configuration>

The second filter that ships with logback is ThresholdFilter. It is also based on level value, but acts as a threshold to deny any request whose level is not equal or greater to the configured level. A sample use of the ThresholdFilter is shown below.

Example 6.4: Sample ThresholdFilter configuration (logback-examples/src/main/java/chapter6/thresholdFilterConfig.xml)
<configuration>
  <appender name="CONSOLE"
    class="ch.qos.logback.core.ConsoleAppender">
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
      <level>INFO</level>
    </filter>
    <layout class="ch.qos.logback.classic.PatternLayout">
      <pattern>
        %-4relative [%thread] %-5level %logger{30} - %msg%n
      </pattern>
    </layout>
  </appender>
  <root level="DEBUG">
    <appender-ref ref="CONSOLE" />
  </root>
</configuration>

Evaluator Filters taking Java Expressions

A special category of filters is implemented by the EvaluatorFilter class. These filters use an EventEvaluator object to decide whether to accept or deny a request. This allows unprecedented flexibility in the way that you can affect filtering of logging events.

As a user, you do not need to worry about the actual plumbing. All you need to do is to give a name to the evaluator and to specify an evaluation expression, that is a boolean expression in regular Java syntax. These evaluation expressions are compiled on-the-fly during the interpretation of the configuration file. It is the users reponsibility to ensure that the expression is boolean, that it evaluates to true or false. In evaluation expressions, logback implicitly exposes various fields of a logging event as variables. The list of these implicit variables is given below. The scope of evaluation expressions is limited to the logging event.

Name Type Description
event LoggingEvent The raw logging event associated with the logging request. All of the following variables are also available from the event. For example, event.getMessage() returns the same String value as the message variable described next.
message String The message of the logging request.
logger LoggerRemoteView This object is a proxy for the logger object where the logging request was issued. It can be viewed as a regular logger object. However, it has some nice performance properties, especially with respect to serialization.
level int The int value corresponding to the level. To help create easily expressions involving levels, the default value DEBUG, INFO, WARN and ERROR are also available. Thus, using level > INFO is a correct expression.
timeStamp long The timestamp corresponding to the logging event's creation.
marker Marker The Marker object associated with the logging request.
mdc Map A map containing all the MDC values at the time of the creation of the logging event. A value can be accessed by using the following expression: mdc.get("myKey").
throwable Throwable The exception associated with the logging event

The behaviour of the EvaluatorFilter is also affected by its OnMatch and OnMismatch options taking values of type FilterReply, i.e. DENY, ACCEPT, NEUTRAL.

Here is a concrete example.

Example 6.5: Basic event evaluator usage (logback-examples/src/main/java/chapter6/basicEventEvaluator.xml)
<configuration>

  <appender name="STDOUT"
    class="ch.qos.logback.core.ConsoleAppender">
    <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
      <evaluator name="myEval">
        <expression>message.contains("billing")</expression>
      </evaluator>
      <OnMismatch>NEUTRAL</OnMismatch>
      <OnMatch>DENY</OnMatch>
    </filter>
    <layout>
      <pattern>
        %-4relative [%thread] %-5level %logger - %msg%n
      </pattern>
    </layout>
  </appender>

  <root level="INFO">
    <appender-ref ref="STDOUT" />
  </root>
</configuration>

The bold part in the previous configuration adds an EvaluatorFilter to a ConsoleAppender. An EventEvaluator is then injected into the EvaluatorFilter. The expression element corresponds to the evaluation expression described previously. The expression message.contains("billing") returns a boolean value. Notice that the message variable is defined implicitly.

This evalutor filter will drop all logging events whose message contains the string "billing".

The FilterEvents application issues ten logging requests, numbered 0 to 9. Let us first run FilterEvents class without any filters:

java chapter6.FilterEvents src/main/java/chapter6/basicConfiguration.xml

All requests will be displayed, as shown below:

0    [main] INFO  chapter6.FilterEvents - logging statement 0
0    [main] INFO  chapter6.FilterEvents - logging statement 1
0    [main] INFO  chapter6.FilterEvents - logging statement 2
0    [main] DEBUG chapter6.FilterEvents - logging statement 3
0    [main] INFO  chapter6.FilterEvents - logging statement 4
0    [main] INFO  chapter6.FilterEvents - logging statement 5
0    [main] ERROR chapter6.FilterEvents - billing statement 6
0    [main] INFO  chapter6.FilterEvents - logging statement 7
0    [main] INFO  chapter6.FilterEvents - logging statement 8
0    [main] INFO  chapter6.FilterEvents - logging statement 9

Suppose that we want to get rid of the billing information. The basicEventEvaluator.xml configuration file just described, does exactly what we want.

Running with filters:

java chapter6.FilterEvents src/main/java/chapter6/basicEventEvaluator.xml

we obtain:

0 [main] INFO chapter6.FilterEvents - logging statement 0 0 [main] INFO chapter6.FilterEvents - logging statement 1 0 [main] INFO chapter6.FilterEvents - logging statement 2 0 [main] DEBUG chapter6.FilterEvents - logging statement 3 0 [main] INFO chapter6.FilterEvents - logging statement 4 0 [main] INFO chapter6.FilterEvents - logging statement 5 0 [main] INFO chapter6.FilterEvents - logging statement 7 0 [main] INFO chapter6.FilterEvents - logging statement 8 0 [main] INFO chapter6.FilterEvents - logging statement 9

TurboFilters

TurboFilter objects all extend the TurboFilter abstract class. Like the regular filters, they use ternary logic to return their evaluation of the logging event.

Overall, they work much like the previously mentioned filters. However, there are two main differences between Filter and TurboFilter objects.

TurboFilter objects are tied to the logging context. Hence, they are called not only when a given appender is used, but each and every time a logging request is issued. Their scope is wider than appender-attached filters.

More importantly, they are called before the LoggingEvent object creation. TurboFilter objects do not require the instantiation of a logging event to filter a logging request. As such, turbo filters are intended for high performance filtering of logging events, even before they are created.

Implementing your own TurboFilter

To create your own TurboFilter component, just extend the TurboFilter abstract class. As previously, when implementing a customized filter object, developing a custom TurboFilter only asks that one implement the decide() method. In the next example, we create a slightly more complex filter:

Example 6.6: Basic custom TurboFilter (logback-examples/src/main/java/chapter6/SampleTurboFilter.java)
package chapter6;

import org.slf4j.Marker;
import org.slf4j.MarkerFactory;

import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.turbo.TurboFilter;
import ch.qos.logback.core.spi.FilterReply;

public class SampleTurboFilter extends TurboFilter {

  String marker;
  Marker markerToAccept;

  @Override
  public FilterReply decide(Marker marker, Logger logger, Level level,
      String format, Object[] params, Throwable t) {

    if (!isStarted()) {
      return FilterReply.NEUTRAL;
    }

    if ((markerToAccept.equals(marker))) {
      return FilterReply.ACCEPT;
    } else {
      return FilterReply.NEUTRAL;
    }
  }

  public String getMarker() {
    return marker;
  }

  public void setMarker(String markerStr) {
    this.marker = markerStr;
  }

  @Override
  public void start() {
    if (marker != null && marker.trim().length() > 0) {
      markerToAccept = MarkerFactory.getMarker(marker);
      super.start(); 
    }
  }
}

The TurboFilter above accepts events that contain a specific marker. If said marker is not found, then the filter passes the responsability to the next filter in the chain.

To allow more flexibility, the marker that will be tested can be specified in the configuration file. Hence the getter and setter methods. We also implemented the start() method, to check that the option has been specified during the configuration process.

Here is a sample configuration that makes use of our newly created TurboFilter.

Example 6.7: Basic custom TurboFilter configuration (logback-examples/src/main/java/chapter6/sampleTurboFilterConfig.xml)
<configuration>
  <turboFilter class="chapter6.SampleTurboFilter">
    <Marker>sample</Marker>
  </turboFilter>

  <appender name="STDOUT"
    class="ch.qos.logback.core.ConsoleAppender">
    <layout class="ch.qos.logback.classic.PatternLayout">
      <pattern>
        %-4relative [%thread] %-5level %logger - %msg%n
      </pattern>
    </layout>
  </appender>

  <root>
    <appender-ref ref="STDOUT" />
  </root>
</configuration>

Logback classic ships with several TurboFilter classes ready for use. The MDCFilter checks the presence of a given value in the MDC whereas DynamicThresholdFilter allows filtering based on MDC key/level threshold associations. On the other hand, MarkerFilter checks for the presence of a specific marker associated with the logging request.

Here is a sample configuration, using both MDCFilter and MarkerFilter.

Example 6.8: MDCFilter and MarkerFilter configuration (logback-examples/src/main/java/chapter6/turboFilters.xml)
<configuration>

  <turboFilter class="ch.qos.logback.classic.turbo.MDCFilter">
    <MDCKey>username</MDCKey>
    <Value>sebastien</Value>
    <OnMatch>ACCEPT</OnMatch>
  </turboFilter>
	
  <turboFilter class="ch.qos.logback.classic.turbo.MarkerFilter">
    <Marker>billing</Marker>
    <OnMatch>DENY</OnMatch>
  </turboFilter>

  <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
    <layout class="ch.qos.logback.classic.PatternLayout">
      <Pattern>%date [%thread] %-5level %logger - %msg%n</Pattern>
  </layout>
  </appender>

  <root level="info">
    <appender-ref ref="console" />
  </root>  
</configuration>

You can see this configuration in action by issuing the following command:

java chapter6.FilterEvents src/main/java/chapter6/turboFilters.xml

As we've seen previously, the FilterEvents application issues 10 logging requests, numbered 0 to 9. Except for requests 3 and 6, all of the requests are of level INFO, the same level as the one assigned to the root logger. The 3rd request, is issued at the the DEBUG level, which is below the effective level. However, since the MDC key "username" is set to "sebastien" just before the 3rd request and removed just afterwards, the MDCFilter specifically accepts the request (and only that request). The 6th request, issued at the ERROR level, is marked as "billing". As such, it is denied by the MarkerFilter (the second turbo filter in the configuration).

Thus, the output of FilterEvents application configured with turboFilters.xml file shown above is:

2006-12-04 15:17:22,859 [main] INFO chapter6.FilterEvents - logging statement 0 2006-12-04 15:17:22,875 [main] INFO chapter6.FilterEvents - logging statement 1 2006-12-04 15:17:22,875 [main] INFO chapter6.FilterEvents - logging statement 2 2006-12-04 15:17:22,875 [main] DEBUG chapter6.FilterEvents - logging statement 3 2006-12-04 15:17:22,875 [main] INFO chapter6.FilterEvents - logging statement 4 2006-12-04 15:17:22,875 [main] INFO chapter6.FilterEvents - logging statement 5 2006-12-04 15:17:22,875 [main] INFO chapter6.FilterEvents - logging statement 7 2006-12-04 15:17:22,875 [main] INFO chapter6.FilterEvents - logging statement 8 2006-12-04 15:17:22,875 [main] INFO chapter6.FilterEvents - logging statement 9

One can see that the 3rd request, which should not be displayed if we only followed the overall INFO level, appears anyway, because it matched the first TurboFilter requirements and was accepted.

On the other hand, the 6th request, that is an ERROR level request should have been displayed. But it satisfied the second TurboFilter whose OnMatch option is set to DENY. Thus, the 6th request was not displayed.

DuplicateMessageFilter

The DuplicateMessageFilter merits a separate presentation. This filter detects duplicate messages, and beyond a certain number of repetitions, drops repeated messages.

To detect repetition, this filter uses simple String equality between messages. It does not detect messages which are very similar, varying only by few characters. For example, if you write:

logger.debug("Hello "+name0);
logger.debug("Hello "+name1);

Assuming name0 and name1 have different values, the two "Hello" messages will be considered as unrelated. Depending on user demand, future releases may check for string similarity, eliminating repetitions of similar but not identical messages.

Note that in case of parameterized logging, only the raw message is taken into consideration. For example, in the next two requests, the raw messages, i.e. "Hello {}.", are identical, and thus considered as repetitions.

logger.debug("Hello {}.", name0);
logger.debug("Hello {}.", name1);

The number of allowed repetitions can be specified by the AllowedRepetitions property. For example, if the said property is set to 1, then the 2nd and subsequent occurrences of the same message will be dropped. Similarly, if the said property is set to 2, then the 3rd and subsequent occurrences of the same message will be dropped. By default, the AllowedRepetitions property is set to 5.

In order to detect repetitions, this filter needs to keep references to old messages in an internal cache. The size of this cache is determined by the CacheSize property. By the default, this is set to 100.

Example: DuplicateMessageFilter configuration (logback-examples/src/main/java/chapter6/duplicateMessage.xml)
<configuration>

  <turboFilter class="ch.qos.logback.classic.turbo.DuplicateMessageFilter"/>

  <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
    <layout class="ch.qos.logback.classic.PatternLayout">
      <Pattern>%date [%thread] %-5level %logger - %msg%n</Pattern>
  </layout>
  </appender>

  <root level="info">
    <appender-ref ref="console" />
  </root>  
</configuration>

Thus, the output for FilterEvents application configured with duplicateMessage.xml is:

2008-12-19 15:04:26,156 [main] INFO chapter6.FilterEvents - logging statement 0 2008-12-19 15:04:26,156 [main] INFO chapter6.FilterEvents - logging statement 1 2008-12-19 15:04:26,156 [main] INFO chapter6.FilterEvents - logging statement 2 2008-12-19 15:04:26,156 [main] INFO chapter6.FilterEvents - logging statement 4 2008-12-19 15:04:26,156 [main] INFO chapter6.FilterEvents - logging statement 5 2008-12-19 15:04:26,171 [main] ERROR chapter6.FilterEvents - billing statement 6

"logging statement 0" is the first occurrence of the message "logging statement {}". "logging statement 1" is the first repetition, "logging statement 2" is the second repetition. Interestingly enough, "logging statement 3" of level DEBUG, is the third repetition, even though it is later dropped by virtue of the basic selection rule. This can be explained by the fact that turbo filters are invoked before other types of filters, including the basic selection rule. Thus, DuplicateMessageFilter considers "logging statement 3" as a repetition, oblivious to the fact that it will be dropped further down in the processing chain. "logging statement 4" is the fourth repetition and "logging statement 5" the fifth. Statements 6 and beyond are dropped because only 5 repetitions are allowed by default.

Logback-access

Logback-access offers most of the features available with logback-classic. Filter objects are available and work in the same way as their logback-classic counterparts. They handle access' implementation of logging events: AccessEvent. Thus, a customized filter for logback access follows strictly the same rules as those for logback-classic, except for the event type received as parameter. On the other hand, TurboFilter objects are supported by logback-access.

Filters

EvaluatorFilter objects with java expressions supplied in in evaluator configuration elements are supported by logback-access. However, list implicit variables available for constructing an expression are different. Only the AccessEvent object can be used by inserting the event variable in the expression. Nevertheless the access evaluator is just as powerfull. All the request and response components are reachable from the event variable.

Here is a sample configuration that will ensure that any 404 error will be logged:

Example 6.9: Access Evaluator (logback-examples/src/main/java/chapter6/accessEventEvaluator.xml)
<configuration>

  <appender name="STDOUT"
    class="ch.qos.logback.core.ConsoleAppender">
    <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
      <evaluator name="myEval">
        <expression>event.getStatusCode() == 404</expression>
      </evaluator>
      <OnMismatch>NEUTRAL</OnMismatch>
      <OnMatch>ACCEPT</OnMatch>
    </filter>
    <layout class="ch.qos.logback.access.PatternLayout">
      <pattern>
        %h %l %u %t %r %s %b
      </pattern>
    </layout>
  </appender>

  <appender-ref ref="STDOUT" />
</configuration>

We might imagine a slightly more complex use of filters to ensure logging of 404 errors, except those returned on access to CSS resources. Here is what such a configuration would look like:

Example 6.10: Access Evaluator (logback-examples/src/main/java/chapter6/accessEventEvaluator2.xml)
<configuration>

  <appender name="STDOUT"
    class="ch.qos.logback.core.ConsoleAppender">
    <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
      <evaluator name="Eval404">
        <expression>event.getStatusCode() == 404</expression>
      </evaluator>
      <OnMismatch>NEUTRAL</OnMismatch>
      <OnMatch>ACCEPT</OnMatch>
    </filter>
    <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
      <evaluator name="EvalCSS">
        <expression>event.getRequestURI().contains("css")</expression>
      </evaluator>
      <OnMismatch>NEUTRAL</OnMismatch>
      <OnMatch>DENY</OnMatch>
    </filter>
    <layout class="ch.qos.logback.access.PatternLayout">
      <pattern>
        %h %l %u %t %r %s %b
      </pattern>
    </layout>
  </appender>

  <appender-ref ref="STDOUT" />
</configuration>