Skip to end of metadata
Go to start of metadata

Introduction

FEATURE AVAILABILITY STARTING FROM RELEASE 2.4.0

Log files include data which is considered sensitive by a number of users. This includes:

  • Hostnames, IP Addresses, Ports
  • Accounts
  • Database Names

There are situations where log files should be anonymized before forwarding to a 3rd party, to the SOS Ticketing System or to a public forum to receive community support.

JS7 allows logs to be anonymized by replacing sensitive data with placeholders:

Example for Log Output holding Host names and Ports
2022-05-14T11:27:27,026 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-seconda
2022-05-14T11:27:27,829 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-secondary:4443)
2022-05-14T11:27:28,526 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'standalone' (https://controller-2-0-standalone:4443)
2022-05-14T11:27:28,527 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'standalone' (https://controller-2-0-standalone:4443)
2022-05-14T11:27:31,343 INFO  JControllerProxy-42  c.s.j.c.p.ProxyContext                       - 'standalone' (https://controller-2-0-standalone:4443): ProxyCoupled(1652478862797000)
2022-05-14T11:27:32,908 INFO  JControllerProxy-41  c.s.j.c.p.ProxyContext                       - 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-secondary:4443): ProxyCoupled(1652520420689258)
Example for anonymized Log Output
2022-05-14T11:27:27,026 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'testsuite' cluster (https://<host>:<port>)
2022-05-14T11:27:27,829 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'testsuite' cluster (https://<host>:<port>)
2022-05-14T11:27:28,526 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'standalone' (https://<host>:<port>)
2022-05-14T11:27:28,527 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'standalone' (https://<host>:<port>)
2022-05-14T11:27:31,343 INFO  JControllerProxy-42  c.s.j.c.p.ProxyContext                       - 'standalone' (https://<host>:<port>): ProxyCoupled(1652478862797000)
2022-05-14T11:27:32,908 INFO  JControllerProxy-41  c.s.j.c.p.ProxyContext                       - 'testsuite' cluster (https://<host>:<port>): ProxyCoupled(1652520420689258)

Log Anonymizer Script

Location

Anonymization is available from a Java class and is invoked by a shell script available at the following default locations:

Location of Log Anonymizer Script for Unix
# JOC Cockpit
/opt/sos-berlin.com/js7/joc/jetty/bin/anonymize-logs.sh

# Controller
/opt/sos-berlin.com/js7/controller/bin/anonymize-logs.sh

# Agent
/opt/sos-berlin.com/js7/agent/bin/anonymize-logs.sh
Location of Log Anonymizer Script for Windows
@rem JOC Cockpit
C:\Program Files\sos-berlin.com\js7\joc\jetty\bin\anonymize-logs.cmd

@rem Controller
C:\Program Files\sos-berlin.com\js7\controller\bin\anonymize-logs.cmd

@rem Agent
C:\Program Files\sos-berlin.com\js7\agent\bin\anonymize-logs.cmd

Usage

Invocation of Log Anonymizer Script
Usage: log-anonymizer.sh [Options]

  Options:
    -l | --log-file=<log-file>       | optional: location of log files to be anonymized; a single file, directory or wildcards can be specified;
                                                 the argument can occur any number of times
    -o | --output-dir=<directory>    | optional: output directory for anonymized log files
    -r | --rules-file=<rules-file>   | optional: path to a YAML file holding rules for anonymization; by default built-in rules will be applied
    -e | --export-rules=<rules-file> | optional: path to a YAML file to which built-in rules will be exported

Explanation:

  • Options
    • -l | --log-file=<log-file>: Specifies the location of the log file(s) to be anonymized.
      • This option can be specified repeatedly for a number of files. Wildcards can be specified and directories can be specified if all included files are to be anonymized.
      • Plain text log files with the .log file extension and compressed log files with the .gz file extension are considered - see the JS7 - Log Rotation article.
      • Anonymized log file names are prefixed with the string: anonymized- 
      • By default anonymized log files are stored in the directory in which original log files are found.
    • -o | --output-dir=<directory>: Optionally specifies the output directory in which anonymized log files are to be stored. If this argument is omitted then anonymized log files are stored in their original directory.
    • -r | --rules-file=<rules-file>: Optionally specifies the location of a file in YAML format that holds the rules to be applied for anonymization.
    • -e | --export-rules=<rules-file>: Optionally specifies the location of a file in YAML format to which the built-in rules for anonymization will be exported.

Rules

Rules include specifying regular expressions for searching and related placeholders as replacements. The built-in rules cover typical configuration items such as URLs, IP addresses, Host names etc. 

  • Users can export the built-in rules to a file to verify available rules and expressions.
  • Users can add individual rules to a file that is used when invoking the Log Anonymizer Script. It is recommended that individual files are validated as being YAML compliant.

Example for built-in Rules
rules:
- item: url-component
  search: ://(.*):(\d{2,5})
  replace:
  - <host>
  - <port>
- item: ip-address
  search: (([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5]))
  replace:
  - <ip-address>
- item: host-install
  search: host[\s]*=[\s]*(.*)
  replace:
  - <host>

Explanation:

  • Each item specifies an expression to search for and to replace.
  • The search expression makes use of capturing groups specified by ( ... ).
  • The replace placeholder specifies a number of strings that replace the content of related capturing groups.

Examples

The following examples illustrate typical use cases.

Example for Invocation of Log Anonymizer Script
log-anonymizer.sh --log-file=/var/sos-berlin.com/js7/controller/logs/controller.log

Explanation:

  • Creates an anonymized version of the log file in the same directory with the name: anonymized-controller.log


Example for Invocation of Log Anonymizer Script for a number of Log Files
log-anonymizer.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc.log  \
                  --log-file=/var/sos-berlin.com/js7/joc/logs/joc-debug.log \
                  --output-dir=/tmp/logs

Explanation:

  • Creates anonymized versions of the indicated log files in the given output directory:
    • /tmp/logs/anonymized-joc.log
    • /tmp/logs/anonymized-joc-debug.log


Example for Invocation of Log Anonymizer Script with Wildcards for Log Files
log-anonymizer.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc*  \
                  --output-dir=/tmp/logs

Explanation:

  • Creates anonymized versions of the log files indicated from the given directory. The wildcard specifies any log files carrying a name that starts with joc will be used.
  • The wildcard includes consideration of plain text log files with the .log file extension and also consideration of compressed log files with a .gz file extension.
  • Anonymized log files are stored in the output directory.


Example for Export of Rules to a File
log-anonymizer.sh --export-rules=/tmp/rules.yaml

Explanation:

  • The build-in rules are exported to a file in YAML format.
  • This file can be used to adjust rules and can be applied for anonymization later on.


Example for Invocation of Log Anonymizer Script with individual Rules from a File
log-anonymizer.sh --log-file=/var/sos-berlin.com/js7/agent_4445/logs/*.log \
                  --output-dir=/tmp/logs \
                  --rules-file=/tmp/rules.yaml

Explanation:

  • Creates anonymized versions of all log files in the indicated directory and stores them in the output directory.
  • A YAML file with individual rules is applied.

Running Log Anonymizer outside of JS7

Users who wish to run the Log Anonymizer outside of a JS7 installation can:

  • copy a number of Java related files as available from the Controller or Agent installation,
  • create a shell script to invoke the Log Anonymizer.

Java related Files required by Log Anonymizer

The following files can be found in a Controller or Agent installation for Unix or Windows.

  • Note that version numbers of binary files can change over time. The examples below assume Java related files as provided with Release 2.4.0.
  • Users can copy the following Java related files from the ./lib/3rd-party, ./lib/sos, ./lib/stdout directories to the location where they want to use Log Anonymizer:

Java binary files required by Log Anonymizer
./lib/sos/sos-commons-util-2.4.0.jar
./lib/stdout/log4j2.xml
./lib/3rd-party/org.apache.logging.log4j.log4j-api-2.17.2.jar
./lib/3rd-party/org.apache.logging.log4j.log4j-core-2.17.2.jar
./lib/3rd-party/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar
./lib/3rd-party/org.slf4j.slf4j-api-1.7.36.jar
./lib/3rd-party/snakeyaml-1.30.jar

Shell Script to run Log Anonymizer

An individual script to invoke Log Anonymizer can look like this:

Example for Invocation of Log Anonymizer from an individual Script on Unix
#!/bin/sh

JAVA_BIN="${JAVA_HOME}/bin/java"
test -x "${JAVA_BIN}" || JAVA_BIN="java"

script_dir="$(echo $(dirname "$0") | cd > /dev/null && pwd)"
cp="-classpath "${script_dir}/lib/sos/*:${script_dir}/lib/3rd-party/*:${script_dir}/lib/stdout""

"${JAVA_BIN}" ${JAVA_OPTIONS} ${cp} com.sos.commons.util.loganonymizer.SOSLogAnonymizer $*
Example for Invocation of Log Anonymizer from an individual Script on Windows
@echo off

set JAVA_BIN=java.exe
if exist "%JAVA_HOME%\bin\java.exe" set JAVA_BIN=%JAVA_HOME%\bin\java.exe

set script_dir=%~dp0
set cp=-classpath "%script_dir%lib\sos\*;%script_dir%lib\3rd-party\*;%script_dir%lib\stdout"

"%JAVA_BIN%" %JAVA_OPTIONS% %cp% com.sos.commons.util.loganonymizer.SOSLogAnonymizer %*


Explanation:

The directory hierarchy is assumed like this:

  • log-anonymizer.sh | .cmd
    • lib
      • sos
        • sos-commons-util-2.4.0.jar
      • stdout
        • log4j2.xml
      • 3rd-party
        • org.apache.logging.log4j.log4j-api-2.17.2.jar
        • org.apache.logging.log4j.log4j-core-2.17.2.jar
        • org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar
        • org.slf4j.slf4j-api-1.7.36.jar
        • snakeyaml-1.30.jar



  • No labels
Write a comment…