PowerShell CLI 1.2 - Use Cases - Task Log

Starting Situation

User might be interested to automatically forward task logs to some external location.
- This allows to use log analysis tools such as Splunk, Elasticsearch etc.
- You are strongly advised not to access task logs from their original location with the JobScheduler Master's ./logs directory for analysis with 3rd party tools.
  - Log files are constantly updated, approx. every 5s, therefore you cannot decide when the log is completed.
  - Log files are stored to the database on task completion and will be overwritten on disk with the next task starting from the same job.
The forwarding of logs can be scheduled e.g. on a daily basis or more frequently to allow immediate log analysis.

Use Cases

Forward Task Logs from a job

The PowerShell CLI can be used by jobs to forward logs. This requires prior installation of

the JobScheduler PowerShell Module

The Get-JobSchedulerTaskHistory cmdlet is used to retrieve task history items and to forward them to the Get-JobSchedulerTaskLog cmdlet within a job. Two flavors of the job are available for Windows and Linux. The difference is not about the handling of cmdlets or parameters but due to the fact that PowerShell is invoked differently on Windows and Linux. For Windows environments usually PowerShell is available with the OS, for Linux the job has to call pwsh to invoke the PowerShell.

Please consider that below jobs are examples that have to be adjusted for your environment.

Windows Version

Download: forward_task_logs_windows.job.xml

Forward Task Logs (Windows version)

<?xml version="1.0" encoding="UTF-8" ?>
<job title="Forward Task Logs" process_class="agent_windows">
  <params>
    <param name="history_results_directory" value="/users/js/history"/>
  </params>
  <script language="powershell"><![CDATA[
Import-Module $env:SCHEDULER_DATA/config/powershell/Modules/JobScheduler;
Connect-JS -Url $JOCCockpitUrl -Credential $JOCCockpitCredential | Out-Null;

mkdir $env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY

# retrieve last history results if available
if ( Test-Path -Path "$($env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY)/task.history" -ErrorAction continue )
{
    $lastHistory = Import-Clixml -Path "$($env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY)/task.history";
} else {
    $lastHistory = Get-JobSchedulerTaskHistory -RelativeDateFrom -8h | Sort-Object -Property startTime;
}
 
# Copy log files to target directory
Get-JSTaskHistory -DateFrom $lastHistory[0].startTime | Tee-Object -Variable lastHistory | Get-JobSchedulerTaskLog | Select-Object @{name='path'; expression={ "$env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY/$(Get-Date $_.startTime -f 'yyyyMMdd-hhmmss')-$([io.path]::GetFileNameWithoutExtension($_.job)).log"}}, @{name='value'; expression={ $_.log }} | Set-Content;
 
# store last history results to a file for later retrieval
$lastHistory | Export-Clixml -Path "$env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY/task.history";
                 
Write-Output ".. logs forwarded to: $env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY";
]]></script>
  <run_time/>
</job>

Explanations

Line 2-3: The job is executed with a Windows Agent that is assigned by a process class. The job is of type "powershell" and will use the Powershell version provided with the server.
Line 4: The JobScheduler PowerShell module is imported. They could be installed with any location in the file system
Line 7: The Connect-JS cmdlet is used to authenticate with the JOC Cockpit REST Web Service. The required URL and credentials are specified in a PowerShell profile, see PowerShell CLI 1.2 - Use Cases - Credentials Management
Line 10: The Get-JSTaskHistory cmdlet is called
- with the parameter -Timezone to specify to which timezone date values in the report should be converted. The parameter value -Timezone (Get-Timezone) specifies that the timezone of the Agent's server is used. Otherwise specify the desired timezone e.g. like this: -Timezone (Get-Timezone -Id 'GMT Standard Time'). Without using this parameter any date values are stored as UTC dates to the report.
- optionally with additional parameters, e.g. to specify the date range for which the report is created A value -DateFrom (Get-Date -Hour 0 -Minute 0 -Second 0).AddDays(-7).ToUniversalTime() specifies that the report should cover the last 7 days (from midnight). Keep in mind that dates have to be specified for the UTC timezone. Without this parameter the report will be created for the last day.
- see the Get-JSTaskHistory cmdlet for a full parameter reference.
Line 11-19: From the output of the Get-JSTaskHistory cmdlet a number of properties are selected and and are specified for the sequence in which they should occur in the report.
- To add more speaking column headers the property names are mapped to a more readable textual representation.
- Consider the handling of date formats in lines 15, 16. Use of the Get-Date cmdlet converts the output format of dates (not the timezone) to the default format that is in place on the Agent's server. Without using the Get-Date cmdlet any date values will be stored to the report in ISO format, e.g. 2020-12-31 10:11:12+02:00 for a date in the European central timezone that is UTC+1 in winter time and UTC+2 in summer time.
- Lines 17 introduces a new property, a calculated duration. From the start time and end time values of a past start the difference in seconds is calculated and is forwarded to the report.
Line 20: The list of properties per task history item is piped to the Export-Excel cmdlet that is available with the ImportExcel PowerShell Module. The report file name is specified and optionally the worksheet. For a full list of parameters see the ImportExcel PowerShell Module.

Linux Version

Download: forward_task_logs_linux.job.xml

Forward Task Logs (Linux version)

<?xml version="1.0" encoding="UTF-8" ?>
<job title="Report Task History" process_class="agent_linux">
  <params>
    <param name="history_results_directory" value="/home/js/history"/>
  </params>
  <script language="shell"><![CDATA[
pwsh -NoLogo -NonInteractive -Command '& {
    . $env:SCHEDULER_DATA/config/powershell/JobScheduler.PowerShell_profile.ps1;
    Import-Module $env:SCHEDULER_DATA/config/powershell/Modules/JobScheduler;
    Connect-JS -Url $JOCCockpitUrl -Credential $JOCCockpitCredential | Out-Null;
 
    mkdir -p /tmp/history $env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY
 
    # retrieve last history results if available
    if ( Test-Path -Path "$($env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY)/task.history" -ErrorAction continue )
    {
        $lastHistory = Import-Clixml -Path "$($env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY)/task.history";
   } else {
        $lastHistory = Get-JobSchedulerTaskHistory -RelativeDateFrom -8h | Sort-Object -Property startTime;
    }
 
    # Copy log files to target directory
    Get-JSTaskHistory -DateFrom $lastHistory[0].startTime | Tee-Object -Variable lastHistory | Get-JobSchedulerTaskLog | Select-Object @{name="path"; expression={ "$env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY/$(Get-Date $_.startTime -f 'yyyyMMdd-hhmmss')-$([io.path]::GetFileNameWithoutExtension($_.job)).log"}}, @{name="value"; expression={ $_.log }} | Set-Content;
 
    # store last history results to a file for later retrieval
    $lastHistory | Export-Clixml -Path "$env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY/task.history";
                 
    Write-Output ".. logs forwarded to: $env:SCHEDULER_PARAM_HISTORY_RESULTS_DIRECTORY";
}'
]]></script>
  <run_time/>
</job>
</job>