JS7 JobScheduler

Space shortcuts

Page tree

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Next »

Introduction

Log files include data that are considered sensitive by a number of users. This includes

  • Hostnames, IP Addresses, Ports
  • Accounts
  • Database Names

There are situations when log files should be anonymized before forwarding to a 3rd party, to the SOS Ticketing System or to a public forum to receive community support.

JS7 offers to anonymize logs by replacing sensitive data with placeholders:

Example for Log Output holding Host names and Ports
2022-05-14T11:27:27,026 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-seconda
2022-05-14T11:27:27,829 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-secondary:4443)
2022-05-14T11:27:28,526 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'standalone' (https://controller-2-0-standalone:4443)
2022-05-14T11:27:28,527 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'standalone' (https://controller-2-0-standalone:4443)
2022-05-14T11:27:31,343 INFO  JControllerProxy-42  c.s.j.c.p.ProxyContext                       - 'standalone' (https://controller-2-0-standalone:4443): ProxyCoupled(1652478862797000)
2022-05-14T11:27:32,908 INFO  JControllerProxy-41  c.s.j.c.p.ProxyContext                       - 'testsuite' cluster (https://controller-2-0-primary:4443, https://controller-2-0-secondary:4443): ProxyCoupled(1652520420689258)
Example for anonymized Log Output
2022-05-14T11:27:27,026 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'testsuite' cluster (https://<host>:<port>)
2022-05-14T11:27:27,829 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'testsuite' cluster (https://<host>:<port>)
2022-05-14T11:27:28,526 INFO  main                 c.s.j.c.p.ControllerApiContext               - connect ControllerApi of 'standalone' (https://<host>:<port>)
2022-05-14T11:27:28,527 INFO  main                 c.s.j.c.p.ProxyContext                       - start Proxy of 'standalone' (https://<host>:<port>)
2022-05-14T11:27:31,343 INFO  JControllerProxy-42  c.s.j.c.p.ProxyContext                       - 'standalone' (https://<host>:<port>): ProxyCoupled(1652478862797000)
2022-05-14T11:27:32,908 INFO  JControllerProxy-41  c.s.j.c.p.ProxyContext                       - 'testsuite' cluster (https://<host>:<port>): ProxyCoupled(1652520420689258)

Log Anonymizer Script

Location

Anonymization is available from a Java class and is invoked from the a shell script available with the following default locations:

Location of Log Anonymizer Script for Unix
# JOC Cockpit
/opt/sos-berlin.com/js7/joc/jetty/bin/anonymize-logs.sh

# Controller
/opt/sos-berlin.com/js7/controller/bin/anonymize-logs.sh

# Agent
/opt/sos-berlin.com/js7/agent/bin/anonymize-logs.sh
Location of Log Anonymizer Script for Windows
@rem JOC Cockpit
C:\Program Files\sos-berlin.com\js7\joc\jetty\bin\anonymize-logs.cmd

@rem Controller
C:\Program Files\sos-berlin.com\js7\controller\bin\anonymize-logs.cmd

@rem Agent
C:\Program Files\sos-berlin.com\js7\agent\bin\anonymize-logs.cmd

Usage

Invocation of Log Anonymizer Script
Usage: log-anonymizer.sh [Options]

  Options:
    -l | --log-file=<log-file>       | optional: location of a log file that should be anonymized; files, directories and wildcards can be specified
    -r | --rules-file=<rules-file>   | optional: the path to a file holding rules for anonymization
    -e | --export-rules=<rules-file> | optional: the path to a file to which built-in rules are exported

Explanation:

  • Options
    • -l | --log-file=<log-file>: Specifies the location of a log file to be anonymized. This option can be specified repeatedly for a number of files. Wildcards can be specified and directories can be specified should all included files be anonymized.
    • -r | --rules-file=<rules-file>: Optionally specifies the location of a file in YAML format that holds the expressions that should be applied for anonymization.
    • -e | --export-rules=<rules-file>: Optionally specifies the location of a file to which default expressions for anonymization are exported.

Rules

Rules include to specify regular expressions to search for and related placeholders as replacements. The built-in rules cover typical configuration items such as URLs, IP addresses, Host names etc. 

  • Users can export the built-in rules to a file to verify available rules and expressions.
  • Users can add individual rules to a file that is used when invoking the Log Anonymizer Script. It is recommended to validate individual files to be YAML compliant.

Example for built-in Rules
rules:
- item: url-component
  search: ://(.*):(\d{2,5})
  replace:
  - <host>
  - <port>
- item: ip-address
  search: (([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5]))
  replace:
  - <ip-address>
- item: host-install
  search: host[\s]*=[\s]*(.*)
  replace:
  - <host>

Explanation:

  • Each item specifies an expression to search for and to replace.
  • The search expression makes use of capturing groups specified by ( ... ).
  • The replace placeholder specifies a number of strings that replace the content of related capturing groups.

Examples

The following examples illustrate typical use cases.

Example for Invocation of Log Anonymizer Script
log-anonymizer.sh --log-file=/var/sos-berlin.com/js7/controller/var/logs/controller.log
Example for Invocation of Log Anonymizer Script for a number of Log Files
log-anonymizer.sh --log-file=/var/sos-berlin.com/js7/joc/logs/joc.log  \
                  --log-file=/var/sos-berlin.com/js7/joc/logs/joc-debug.log
Example for Export of Rules to a File
log-anonymizer.sh --export-rules=/tmp/rules.yaml
Example for Invocation of Log Anonymizer Script with individual Rules from a File
log-anonymizer.sh --log-file=/var/sos-berlin.com/js7/agent/var_4445/logs/*.log \
                  --rules-file=/tmp/rules.yaml



  • No labels