Introduction

JS7 - Encryption and Decryption are integrated with jobs in a number of ways. Basic requirements include that

  • Jobs can be subject to a number of scenarios:
    • Jobs might want to read secrets from specific sources such as arguments, configuration files etc.
    • Jobs might want to create secrets on-the-fly that should be processed by successor jobs.
  • Jobs should consider not to expose secrets to logging, database persistence etc.

Integration

Use with Workflow Variables

Encrypting Secrets to Workflow Variables


Explanation:

  • The job encrypts a secret using the target Agent's certificate.
  • A new variable is added to the workflow that holds the encrypted secret.
  • Examples:

    • For details see JS7 - How to encrypt and decrypt using Unix Shell

      Example for Encryption using Unix Shell
      # encrypt secret
      result=$($JS7_AGENT_HOME/bin/js7_encrypt.sh --cert=$JS7_AGENT_CONFIG_DIR/foobar.crt --in="12345678")
      
      # forward "new_var" workflow variable holding the encrypted result
      echo "new_var=$result" >> $JS7_RETURN_VALUES
    • For details see JS7 - How to encrypt and decrypt using Windows Shell

      Example for Encryption using Windows Shell
      @rem encrypt secret and return result with JS7_ENCRYPT_VALUE environment variable
      call "%JS7_AGENT_HOME%\bin\js7_encrypt.cmd" "--cert=%JS7_AGENT_CONFIG_DIR%\foobar.crt" "--in=12345678"
      
      @rem forward "new_var" workflow variable holding the encrypted result
      echo new_var=%JS7_ENCRYPT_VALUE% >> %JS7_RETURN_VALUES%
    • For details see JS7 - How to encrypt and decrypt using PowerShell

      Example for Encryption using PowerShell
      # encrypt secret and return result with JS7_ENCRYPT_VALUE environment variable
      $result = Invoke-JS7Encrypt -CertificatePath $env:JS7_AGENT_CONFIG_DIR/foobar.crt -Value '12345678' -JavaLib $env:JS7_AGENT_HOME/lib
      
      # forward "new_var" workflow variable holding the encrypted result
      "new_var=$result" | Out-File $env:JS7_RETURN_VALUES -Append

Decrypting Secrets from Workflow Variables


Explanation:

  • Workflow variables are provided from environment variables for shell jobs, see JS7 - Job Instruction.
  • The job decrypts a secret using the current Agent's Private Key.
  • Examples:

    • Example for Decryption using Unix Shell
      # encrypted result is assumed being available from NEW_VAR environment variable
      secret=$($JS7_AGENT_HOME/bin/js7_decrypt.sh \
          --key=$JS7_AGENT_CONFIG_DIR/private/foobar.key \
          --in="$NEW_VAR")
      echo $secret
    • Example for Decryption using Windows Shell
      @rem encrypted result is assumed being available from NEW_VAR environment variable
      
      call "%JS7_AGENT_HOME%\bin\js7_decrypt.cmd" ^
          "--key=%JS7_AGENT_CONFIG_DIR%\private\foobar.key" ^
          "--in=%NEW_VAR%"
      @echo %JS7_DECRYPT_VALUE%
    • Example for Decryption using PowerShell
      # encrypted result is assumed being available from NEW_VAR environment variable
      $secret = Invoke-JS7Decrypt -Value $env:NEW_VAR -KeyPath $env:JS7_AGENT_CONFIG_DIR/private/foobar.key -JavaLib $env:JS7_AGENT_HOME/lib
      Write-Output $secret

Use with Job Resources

Encrypting Secrets to Job Resources


Explanation:

  • The job encrypts a secret using the target Agent's certificate and stores the encrypted result to a Job Resource variable.
  • Examples:

    • For details see  JS7 - How to update a Job Resource using Unix Shell

      Example for Encryption using Unix Shell
      $JS7_AGENT_HOME/bin/js7_set_job_resource.sh \
          --url=http://joc-2-0-primary:7446 \
          --controller-id=controller \
          --user=root \
          --password=root \
          --job-resource=/ProductDemo/Variables/pdBusinessSecret \
          --key=businessSecret \
          --value='12345678' \
          --env-var=BUSINESS_SECRET \
          --encrypt-cert=$JS7_AGENT_CONFIG_DIR/foobar.crt
    • For details see JS7 - How to update a Job Resource using PowerShell

      Example for Encryption using PowerShell
      Set-JS7JobResource `
          -Path /ProductDemo/Variables/pdBusinessSecret `
          -Key 'businessSecret' `
          -Value '12345678' `
          -EnvVar 'BUSINESS_SECRET' `
          -EncryptCertificatePath $env:JS7_AGENT_CONFIG_DIR/foobar.crt `
          -JavaLib $env:JS7_AGENT_HOME/lib

Decrypting Secrets from Job Resources

Variables from Job Resources are available from environment variables similar to workflow variables.

The Job Resource is assigned the workflow to make environment variables available to all jobs or is assigned individual jobs. The environment variables specified with the Job Resource are automatically available for shell jobs. 

Decryption of secrets is the same as for Decrypting Secrets from Workflow Variables.

Encrypting Configuration Files to Job Resources


Explanation:

  • An external application encrypts a configuration file using the target Agent's certificate. The encrypted configuration file is added to a Job Resource.
  • When the Job Resource is assigned a workflow or job then JS7 takes care to transfer the Job Resource to all Agents operating related jobs.
  • Examples:

    • For details see JS7 - How to update a Job Resource using Unix Shell

      Example for Encryption using Unix Shell
      ./js7_set_job_resource.sh \
          --url=http://joc-2-0-primary:7446 \
          --controller-id=controller \
          --user=root \
          --password=root \
          --job-resource=/ProductDemo/Variables/pdConfigurationData \
          --key=configurationData \
          --file=application.conf \
          --env-var=CONFIGURATION_DATA \
          --encrypt-cert=foobar.crt
    • For details see JS7 - How to update a Job Resource using PowerShell

      Example for Encryption using PowerShell
      Set-JS7JobResource `
          -Path /ProductDemo/Variables/pdConfigurationData `
          -Key 'configurationData' `
          -File application.conf `
          -EnvVar 'CONFIGURATION_DATA' `
          -EncryptCertificatePath foobar.crt `
          -JavaLib /js7/js7.encryption/lib

Decrypting Configuration Files from Job Resources


Explanation;:

  • JS7 takes care to transfer the Job Resource to all Agents operating workflows or jobs which are assigned the Job Resource. The encrypted configuration file included with the Job Resource variable is stored to a temporary file per instance of a job (task, process) and is automatically removed on termination of the job instance.
  • Environment variables from the Job Resource denote the location of the temporary file and the encryption result:
    • <env-var>: location of temporary file
    • <env-var>_KEY:  encryption result holding the encrypted symmetric key and intialization vector
  • The job makes use of the js7_decrypt.sh | .cmd scripts to decrypt the encrypted configuration file by use of its Private Key. Similarly the Invoke-JS7Decrypt PowerShell cmdlet can be used.
  • Examples:

    • Example for Decryption using Unix Shell
      # previous encryption is assumed to create the CONFIGURATION_DATA and CONFIGURATION_DATA_KEY environment variables in the Job Resource:
      # ./js7_set_job_resource.sh --job-resource=/ProductDemo/Variables/pdConfigurationData --key=configurationData --file=application.conf --env-var=CONFIGURATION_DATA ...
      
      # CONFIGURATION_DATA environment variable specifies the path to a temporary file provided by JS7 that holds the encrypted configuration data
      echo "$CONFIGURATION_DATA"
      # CONFIGURATION_DATA_KEY environment variable holds the encrypted symmetric key and initialization vector
      echo "$CONFIGURATION_DATA_KEY"
      
      # decrypt configuration file and store to a temporary file
      $JS7_AGENT_HOME/bin/js7_decrypt.sh \
          --key=$JS7_AGENT_CONFIG_DIR/private/foobar.key \
          --in="$CONFIGURATION_DATA_KEY" \
          --infile="$CONFIGURATION_DATA" \
          --outfile=application-$$.conf
      
      # on job termination JS7 automatically performs cleanup of the temporary file denoted by CONFIGURATION_DATA
      # user's job implementation has to clean up temporary files created by the job
      rm -f application-$$.conf
    • Example for Decryption using Windows Shell
      @rem previous encryption is assumed to create the CONFIGURATION_DATA and CONFIGURATION_DATA_KEY environment variables in the Job Resource:
      
      @rem CONFIGURATION_DATA environment variable specifies the path to a temporary file provided by JS7 that holds the encrypted configuration data
      @echo %CONFIGURATION_DATA%
      @rem CONFIGURATION_DATA_KEY environment variable holds the encrypted symmetric key and initialization vector
      @echo %CONFIGURATION_DATA_KEY%
      
      set "TempFile=%TEMP%\application-%DATE%-%TIME::=.%.%RANDOM%.conf"
      call "%JS7_AGENT_HOME%\bin\js7_decrypt.cmd" ^
          "--key=%JS7_AGENT_CONFIG_DIR%\private\foobar.key" ^
          "--in=%CONFIGURATION_DATA_KEY%" ^
          "--infile=%CONFIGURATION_DATA%" ^
          "--outfile=%TempFile%"
      
      # on job termination JS7 automatically performs cleanup of the temporary file denoted by CONFIGURATION_DATA
      # user's job implementation has to clean up temporary files created by the job
      del /F %TempFile%
    • Example for Decryption using PowerShell
      # previous encryption is assumed to create the CONFIGURATION_DATA and CONFIGURATION_DATA_KEY environment variables in the Job Resource:
      # Set-JS7JobResource -Path /ProductDemo/Variables/pdConfigurationData -Key 'configurationData' -File application.conf -EnvVar 'CONFIGURATION_DATA' ...
      
      # CONFIGURATION_DATA environment variable specifies the path to a temporary file provided by JS7 that holds the encrypted configuration data
      Write-Output $env:CONFIGURATION_DATA
      # CONFIGURATION_DATA_KEY environment variable holds the encrypted symmetric key and initialization vector
      Write-Output $env:CONFIGURATION_DATA_KEY
      
      # decrypt configuration file
      $tempFile = New-TemporaryFile
      Invoke-JS7Decrypt `
          -Value $env:CONFIGURATION_DATA_KEY `
          -File $env:CONFIGURATION_DATA `
          -OutFile $tempFile `
          -KeyPath $env:JS7_AGENT_CONFIG_DIR/private/foobar.key `
          -JavaLib $env:JS7_AGENT_HOME/lib
      
      # on job termination JS7 automatically performs cleanup of the temporary file denoted by CONFIGURATION_DATA
      # user's job implementation has to clean up temporary files created by the job
      Remove-Item -Path $tempFile -Force

Use with Configuration Items from Files

Encrypting Secrets to Configuration Files


Explanation;:

  • A secret should be stored to a configuration file, for example a password for access to a database,
  • An external application creates/updates the secret, for example triggered by password rotation.
  • The application makes use of the js7_encrypt.sh | .cmd scripts to encrypt the secret and stores the encrypted secret to the configuration file. Similarly the Invoke-JS7Encrypt PowerShell cmdlet can be used.
  • Find details from JS7 - How to encrypt and decrypt Database Credentials.
  • Examples:

    • Example for Encryption using Unix Shell
      # encrypt secret and return result
      result=$(./js7_encrypt.sh --cert=foobar.crt --in="12345678")
      
      # update <password> placeholder in application.conf file
      sed -i'' -e "s@<password>@${result}@g" application.conf
    • Example for Encryption using Windows Shell
      @rem encrypt secret and return result with JS7_ENCRYPT_VALUE environment variable
      call .\js7_encrypt.cmd "--cert=foobar.crt" "--in=12345678"
      
      @rem update <password> placeholder in application.conf file
      powershell.exe -Command "((Get-Content application.conf) -replace '<password>', $env:JS7_ENCRYPT_VALUE) | Set-Content -Path application.conf"
    • Example for Encryption using PowerShell
      # encrypt secret and return result
      $result = Invoke-JS7Encrypt -CertificatePath foobar.crt -Value '12345678' -JavaLib /js7/js7.encryption/lib
      
      # update <password> placeholder in application.conf file
      ((Get-Content application.conf) -replace '<password>', $result) | Set-Content -Path application.conf

Decrypting Secrets from Configuration Files


Explanation;:

  • A job reads the encrypted secret from the configuration file.
  • The job makes use of the js7_decrypt.sh | .cmd scripts to decrypt the secret by use of its Private Key. Similarly the Invoke-JS7Decrypt PowerShell cmdlet can be used.
  • Examples:

    • Example for Decryption using Unix Shell
      # read secret from application.conf file that is assumed to look like this:
      #     username=foo
      #     password=gE4NPEbrrUFFdjbnvtX9ZWF9owz1ghwbjl2WLAP19k6ps1tXX+fug== KeU1ZYgNfF1kNfYQaIp/Wg== osXNvG2evsSnf3WUE8I8TQ==
      
      result=$(grep -E 'password=.*' application.conf | cut -d '=' -f 2-)
      
      secret=$($JS7_AGENT_HOME/bin/js7_decrypt.sh \
          --key=$JS7_AGENT_CONFIG_DIR/private/foobar.key \
          --in="$result")
      echo $secret
    • Example for Decryption using Windows Shell
      # read secret from application.conf file that is assumed to look like this:
      #     username=foo
      #     password=gE4NPEbrrUFFdjbnvtX9ZWF9owz1ghwbjl2WLAP19k6ps1tXX+fug== KeU1ZYgNfF1kNfYQaIp/Wg== osXNvG2evsSnf3WUE8I8TQ==
      
      @set "TempFile=%TEMP%\application-%DATE%-%RANDOM%.tmp"
      @findstr /c:"password=" application.conf >> %TempFile%
      for /f "tokens=1,* delims='='" %%i in (%TempFile%) do set RESULT=%%j
      @del /F %TempFile%
      
      call "%JS7_AGENT_HOME%\bin\js7_decrypt.cmd" ^
          "--key=%JS7_AGENT_CONFIG_DIR%\private\foobar.key" ^
          "--in=%RESULT%"
      @echo %JS7_DECRYPT_VALUE%
    • Example for Decryption using PowerShell
      # read secret from application.conf file that is assumed to look like this:
      #     username=foo
      #     password=gE4NPEbrrUFFdjbnvtX9ZWF9owz1ghwbjl2WLAP19k6ps1tXX+fug== KeU1ZYgNfF1kNfYQaIp/Wg== osXNvG2evsSnf3WUE8I8TQ==
      
      $matches = ( Get-Content application.conf | Select-String "password=(.*)" ).Matches
      $result = if ( $matches.Groups.count -gt 1 ) { $matches.Groups[1].Value }
      
      $secret = Invoke-JS7Decrypt -Value $result -KeyPath $env:JS7_AGENT_CONFIG_DIR/private/foobar.key -JavaLib $env:JS7_AGENT_HOME/lib
      Write-Output $secret

Use with Configuration Files

Encrypting Configuration Files


Explanation;:

  • Instead of encrypting individual items in a configuration file, the file will be encrypted.
  • An external application updates and encryptes the configuration file in case of changes.
  • The application makes use of the js7_encrypt.sh | .cmd scripts to encrypt the configuration file. Similarly the Invoke-JS7Encrypt PowerShell cmdlet can be used.
  • Examples:

    • Example for Encryption using Unix Shell
      # encrypt configuration file and return result
      result=$(./js7_encrypt.sh --cert=foobar.crt --infile=application.conf --outfile=application.conf.enc)
      echo "$result"
    • Example for Encryption using Windows Shell
      @rem encrypt configuration file and return result with JS7_ENCRYPT_VALUE environment variable
      call .\js7_encrypt.cmd "--cert=foobar.crt" "--infile=application.conf" "--outfile=application.conf.enc"
      echo %JS7_ENCRYPT_VALUE%
    • Example for Encryption using PowerShell
      # encrypt configurationn file and return result
      $result = Invoke-JS7Encrypt -CertificatePath foobar.crt -File application.conf -OutFile application.conf.enc -JavaLib /js7/js7.encryption/lib
      
      # update <password> placeholder in application.conf file
      ((Get-Content application.conf) -replace '<password>', $result) | Set-Content -Path application.conf

Decrypting Configuration Files


Explanation;:

  • The job makes use of the js7_decrypt.sh | .cmd scripts to decrypt the encrypted configuration file by use of its Private Key. Similarly the Invoke-JS7Decrypt PowerShell cmdlet can be used.
  • On termination the job will drop the decrypted configuration file.
  • Examples:

    • Example for Decryption using Unix Shell
      # previous encryption assumed to create the output file like this:
      # result=$(./js7_encrypt.sh --cert=foobar.crt --infile=application.conf --outfile=application.conf.enc)
      
      # decrypt configuration file
      $JS7_AGENT_HOME/bin/js7_decrypt.sh \
          --key=$JS7_AGENT_CONFIG_DIR/private/foobar.key \
          --in="$result" \
          --infile=application.conf.enc \
          --outfile=application.conf
    • Example for Decryption using Windows Shell
      @rem previous encryption assumed to create the output file like this:
      @rem call .\js7_encrypt.cmd "--cert=foobar.crt" "--infile=application.conf" "--outfile=application.conf.enc"
      @echo %JS7_DECRYPT_VALUE%
      
      call "%JS7_AGENT_HOME%\bin\js7_decrypt.cmd" ^
          "--key=%JS7_AGENT_CONFIG_DIR%\private\foobar.key" ^
          "--in=%JS7_DECRYPT_VALUE%" ^
          "--infile=application.conf.enc" ^
          "--outfile=application.conf"
    • Example for Decryption using PowerShell
      # prevous encryption assumed to create the output file like this:
      # $result = Invoke-JS7Encrypt -CertificatePath foobar.crt -File application.conf -OutFile application.conf.enc -JavaLib /js7/js7.encryption/lib
      
      # decrypt configuration file
      Invoke-JS7Decrypt -Value $result -File application.conf.enc -OutFile application.conf -KeyPath $env:JS7_AGENT_CONFIG_DIR/private/foobar.key -JavaLib $env:JS7_AGENT_HOME/lib

Key Distribution

Keys can be distributed in a number of ways. Frequently used scenarios include that

  • Users create individual Private Keys and Certificates for encryption/decryption of secrets per Agent.
  • Users create Private Keys and Certificates that are shared amongst a number of Agents.
    • This applies to use of an Agent Cluster that allows to execute jobs on any Agent in the cluster.

It is possible that jobs access an Agent's Private Key and SSL Certificate that are used to secure HTTPS connections, see JS7 - Agent HTTPS Connections. This requires the Agent's SSL certificate to be created with the dataEncipherment key usage option. Many users consider it more secure to use separate keys for HTTPS connections and for encryption/decryption of secrets.

Further Resources