Introduction

• For JS7 - Rollout of Scheduling Objects, the JOC Cockpit can be configured used to interface with Git repositories in order to store its configuration and rollout scheduling objects such as workflows and jobs.
  • JS7 supports the use of GitLab serversGit compliant servers to manage remote repositories such as GitLab, GitHub.
  • JS7 makes use of the GitLab REST API.a Git CLI Client that has to be provided by the user.
• Scheduling Configuration objects can be versioned (tagged) with Git and can be rolled out to different JS7 instances environments via Git. This allows to create and to test the creation and testing of workflows in a development environment and to perform a rollout and  rollouts to a production environment by use of to be performed using a Git repository.
• A connection to a Git repository is considered an additional option to manage for managing versions and rollouts of JS7 configuration scheduling objects.
• For operations available from the JOC Cockpit GUI see JS7 - Inventory Git Integration. For automation see JS7 - How to rollout from test to prod environments in a CI CD pipeline.
• The Git Repository Interface is made available with the following JS7 releases:

  • display-

  display-

  • feature-availability
    StartingFromRelease 2.

  2

  • 3.0

Object Mappings

Configuration Objects

Environment-independent Objects

JS7 configuration objects that should be rolled out across environments include

• Workflows
• Locks
• File Order Sources
• Notice Boards

Such objects can be managed with JS7 in a way that allows to re-use the objects - without any changes - across rollout environments such as dev, test, prod.

Environment-related Objects

The following configuration objects typically hold values that can be specific for an individual rollout environment and therefore optionally are added to a separate branch. Such objects should not be rolled out across environments.

• Job Resources
• Calendars
• Schedules

Configuration Mappings

The JS7 inventory is organized by folders.

• Folders hold configuration objects such as workflows etc. and any level of sub-folders.
• Folders are the basic unit for JS7 - Permissions, for example, users can be granted access to specific folders only.
• Folders allow to perform bulk operations such as to deploy, to release, to remove and to rename configuration objects.

Folders can be mapped to Git repositories managed by the same or by different Git servers like this:

USER1 [label="   User: 1   ",fillcolor="lightskyblue"]
USER2 [label="   User: 2   ",fillcolor="lightskyblue"]
USER3 [label="   User: 3   ",fillcolor="lightskyblue"]

FOLDER1 [label="   Folder: 1   ",fillcolor="white"]
FOLDER1a [label="   Sub-folder: 1.1   ",fillcolor="white"]
FOLDER1b [label="   Sub-folder: 1.2   ",fillcolor="white"]
FOLDER2 [label="   Folder: 2   ",fillcolor="white"]
FOLDER3 [label="   Folder: 3   ",fillcolor="white"]

REPOSITORY1 [label="   Git Repository: 1   ",fillcolor="orange"]
REPOSITORY2 [label="   Git Repository: 2   ",fillcolor="orange"]
REPOSITORY3 [label="   Git Repository: 3   ",fillcolor="orange"]

USER1 -> FOLDER1 [label="  can access  "]
USER2 -> FOLDER1 [label="  can access  "]
USER2 -> FOLDER2 [label="  can access  "]
USER3 -> FOLDER3 [label="  can access  "]

FOLDER1 -> FOLDER1a [label="  includes  "]
FOLDER1 -> FOLDER1b [label="  includes  "]
FOLDER1a -> REPOSITORY1 [label="  maps to  "]
FOLDER1b -> REPOSITORY1 [label="  maps to  "]

FOLDER2 -> REPOSITORY2 [label="  maps to  "]
FOLDER3 -> REPOSITORY3 [label="  maps to  "]

The mapping of folder objects to repositories implies

• user access:
  • users with JS7 access permissions to manage folders and to access the Git repository can manage configuration objects in JOC Cockpit and can commit, push, pull configuration objects to/from Git,
  • users with JS7 access permissions to manage folders but without access to the Git repository can manage configuration objects in JOC Cockpit,
  • users with JS7 access permissions to view folders do not require Git access.
• folder usage:
  • if a single Git repository should be used then all folders have to be available from a single root folder,
  • for use with a number of Git repositories each folder maps to a different repository,
  • folders can remain without Git integration for local management and deployment.

JOC Cockpit maps folders to Git repositories and stores the respective Git repository URL for each folder in its inventory.

Git Access

User Access

Each user holds an individual set of credentials to connect to a Git server.

The following settings are stored with a user's profile:

• git_account
  • account that is used to connect to a Git server
• git_key
  • the user's private key for Git access
• git_access_token
  • the access token for Git access that works as an alternative to use of a private key

If repositories are used from a number of Git servers that require different sets of user credentials then separate user accounts in JOC Cockpit have to be used.

Repository Access

• The JS7 - REST Web Service API executes a set of Git operations by use of the JGit implementation.
• Any operations to add configuration objects to a repository, to commit, push, pull and merge objects are available from the JS7 REST Web Service API.
• The JOC Cockpit offers the user interface for respective Git operations.
• Users are free to use additional tools such as Git Extensions to manage repositories and branches.

Git Operations

Management

JS7 configuration objects are managed in a Git repository via the JS7 REST Web Service API.

To this purpose JOC Cockpit holds a local repository per user in the JETTY_BASE/resources/joc/repo folder. For each user with Git access a sub-directory is created from the name of the user account that holds the local repository.

• The local user repository is created when a user performs the checkout/clone operations for a Git repository via JOC Cockpit.
• The local user repository is persistent and is updated from the JOC Cockpit database (commit operation) and from the Git repository (pull operation).
• The Git repository is updated from the local user repository by the push operation.
• The JS7 - Cleanup Service removes any local user repositories if the user profile no longer exists, i.e. if a user is dropped from JOC Cockpit.

The following operations are available:

• git-checkout
  • checks out a specific branch or tag from the Git repository to the local user repository
• git-clone
  • initially clones a Git repository to a local user repository
• git-add
  • adds (new) configuration objects from the JOC Cockpit database to the local user repository
• git-delete
  • removes an already added configuration object from the local user repository
• git-commit
  • commits newly added, updated or removed configuration objects to a local user repository
• git-push
  • pushes changes from the local user repository to the remote Git repository

Versioning

Versioning of JS7 configuration objects is performed by use of the tagging capabilities of Git. 

The JS7 REST Web Service API manages the Git tagging process.

The following operations are available:

• git-tag
  • tags the current commit with a version label
• git-push
  • see above

Rollout

Rollout between different JS7 instances can be performed by use of the Git functionality.

The JS7 REST Web Service API manages the processes for checkout and cloning.

The following operations are available:

• git-checkout
  • checks out a specific branch or tag from the Git repository to the local user repository
• git-clone
  • initially clones a Git repository to a local user repository
• git-pull
  • pulls recent changes from the Git repository to the local user repository

Force Overwrite

Forcefully overwrites a target configuration with the source configuration in use.

The JS7 REST Web Service API manages to forcefully overwrite configuration objects in the Git repository.

The following operations are available depending on the force direction:

• JS7 JOC Cockpit instance => Git repository
  • git-add
    • adds (new) configuration objects from the JOC Cockpit database to the local user repository
  • git-delete
    • removes an already added configuration object from the local user repository
  • git-commit
    • commits newly added, changed or removed configuration objects to a local user repository
  • git-push
    • pushes changes from the local user repository to the Git repository
• Git repository => JS7 JOC Cockpit instance
  • git-checkout
    • checks out a specific branch or tag from the Git repository to the local user repository
  • git-clone
    • initially clones a Git repository to a local user repository
  • git-pull
    • pulls recent changes from the Git repository to the local user repository

Use Cases

Single Client - Single Repository

Scenario

• Architecture
  • A single JOC Cockpit instance is used with a single Controller or optionally with separate Controllers and Agents for rollout environments such as dev, test, prod.
  • The Git repository is used per folder to hold a copy of JS7 configuration objects and to create branches of JS7 configuration objects such as dev,, test, prod in a single repository.
• Users
  • All users act in the same JOC Cockpit instance that is a single client to the Git server.
  • Changes to the Git repository are performed by the users' individual accounts and are added to branches in a single directory.
• Rollout
  • Deployments are performed from a single source (JOC Cockpit) to a number of Controllers and Agents (dev, test, prod).
  • The Git repository can be used to fallback (pull) to previous versions of configuration objects, however, only one version is available with JOC Cockpit at a given point in time.

Branches marked as "env" are specific for an environment and are excluded from rollout. Branches marked as "rollout" are forwarded across scheduling environments.

USER1a [label="   User: 1   ",fillcolor="lightskyblue"]
USER1b [label="   User: 2   ",fillcolor="lightskyblue"]
USER1c [label="   User: 3   ",fillcolor="lightskyblue"]
FOLDER1 [label="   Folder: 1   ",fillcolor="white"]
JOC1 [label="   JOC Cockpit   ",fillcolor="lightskyblue"]
DATABASE [shape="ellipse",label="     Database     ",fillcolor="lightskyblue"]
GIT [label="    Git Server    ",fillcolor="orange"]
REPO [label="    Repository    ",fillcolor="orange"]
BRANCH1R [shape="ellipse",label="Branch: env dev",fillcolor="darkolivegreen1"]
BRANCH1I [shape="ellipse",label="Branch: rollout dev",fillcolor="darkolivegreen1"]
BRANCH2R [shape="ellipse",label="Branch: env test",fillcolor="darkolivegreen2"]
BRANCH2I [shape="ellipse",label="Branch: rollout test",fillcolor="darkolivegreen2"]
BRANCH3R [shape="ellipse",label="Branch: env prod",fillcolor="darkolivegreen3"]
BRANCH3I [shape="ellipse",label="Branch: rollout prod",fillcolor="darkolivegreen3"]

JOC1 -> FOLDER1
JOC1 -> DATABASE

FOLDER1 -> BRANCH1R [label="Commit/Push\nPull"]
FOLDER1 -> BRANCH1I [label="Commit/Push\nPull"]
FOLDER1 -> BRANCH2R [label="Commit/Push\nPull"]
FOLDER1 -> BRANCH3R [label="Commit/Push\nPull"]

BRANCH1R -> REPO
BRANCH2R -> REPO
BRANCH3R -> REPO
BRANCH1I -> BRANCH2I -> BRANCH3I -> REPO

REPO -> GIT

USER1a -> JOC1
USER1b -> JOC1
USER1c -> JOC1

Repository Operations

• management
• versioning
• rollout

Requirements

• The JOC Cockpit database is the leading system that holds the configuration objects.

Single Client - Multiple Repositories (dev, test, prod)

Scenario

• Architecture
  • A dedicated JOC Cockpit instance is used with Controller and Agents for each rollout environment such as dev, test, prod.
  • The same Git repository is used per folder by all JOC Cockpit instances, however, each instance is acting on a specific branch per rollout environment.
  • The Git repository is used to hold a copy of JS7 configuration objects, to create versions of JS7 configuration objects and to merge branches such as dev => test, test => prod.
• Users
  • All users act in the same JOC Cockpit instance of the respective rollout environment that is a single client to the Git server.
  • Changes to the Git repository are performed by the users' individual accounts and are added to each branch such as dev, test, prod.
• Rollout
  • Deployments are performed from a single source (JOC Cockpit) to each Controller and Agents per rollout environment (dev, test, prod).
  • The Git repository is used to merge branches and to rollout JS7 configuration objects to different Controllers and Agents, such as dev => test => prod.
  • The Git repository can be used to fallback (pull) to previous versions of configuration objects, however, only one version is available with a given JOC Cockpit instance at a given point in time.

Branches marked as "env" are specific for an environment and are excluded from rollout. Branches marked as "rollout" are forwarded across scheduling environments.

USER1a [label="   User: 1a   ",fillcolor="lightskyblue"]
USER1b [label="   User: 1b   ",fillcolor="lightskyblue"]
USER2 [label="   User: 2   ",fillcolor="lightskyblue"]
USER3 [label="   User: 3   ",fillcolor="lightskyblue"]

JOC1 [label="   JOC Cockpit: dev   ",fillcolor="lightskyblue"]
JOC2 [label="   JOC Cockpit: test   ",fillcolor="lightskyblue"]
JOC3 [label="   JOC Cockpit: prod   ",fillcolor="lightskyblue"]

FOLDER1 [label="   Folder: 1 (dev)   ",fillcolor="white"]
FOLDER2 [label="   Folder: 1 (test)  ",fillcolor="white"]
FOLDER3 [label="   Folder: 1 (prod)   ",fillcolor="white"]

DATABASE1 [shape="ellipse",label="     Database: dev     ",fillcolor="lightskyblue"]
DATABASE2 [shape="ellipse",label="     Database: test     ",fillcolor="lightskyblue"]
DATABASE3 [shape="ellipse",label="     Database: prod     ",fillcolor="lightskyblue"]

GIT [label="Git Server",fillcolor="orange"]
REPO1 [label="Repository: dev",fillcolor="darkolivegreen1"]
REPO2 [label="Repository: test",fillcolor="darkolivegreen2"]
REPO3 [label="Repository: prod",fillcolor="darkolivegreen3"]

BRANCHES1R [shape="ellipse",label="Branches: env dev",fillcolor="darkolivegreen1"]
BRANCHES1I [shape="ellipse",label="Branches: rollout dev",fillcolor="darkolivegreen1"]
BRANCHES2R [shape="ellipse",label="Branches: env test",fillcolor="darkolivegreen2"]
BRANCHES2I [shape="ellipse",label="Branches: rollout test",fillcolor="darkolivegreen2"]
BRANCHES3R [shape="ellipse",label="Branches: env prod",fillcolor="darkolivegreen3"]
BRANCHES3I [shape="ellipse",label="Branches: rollout prod",fillcolor="darkolivegreen3"]

JOC1 -> DATABASE1
JOC2 -> DATABASE2
JOC3 -> DATABASE3

JOC1 -> FOLDER1 
FOLDER1-> BRANCHES1R [label="Commit/Push\nPull"]
FOLDER1-> BRANCHES1I [label="Commit/Push\nPull"]
BRANCHES1R -> REPO1
BRANCHES1I -> REPO1

JOC2 -> FOLDER2
FOLDER2 -> BRANCHES2R [label="Commit/Push\nPull"]
FOLDER2 -> BRANCHES2I [label="Commit/Push\nPull"]
BRANCHES2R -> REPO2
BRANCHES2I -> REPO2

JOC3 -> FOLDER3 
FOLDER3-> BRANCHES3R [label="Commit/Push\nPull"]
FOLDER3-> BRANCHES3I [label="Commit/Push\nPull"]
BRANCHES3R -> REPO3
BRANCHES3I -> REPO3

BRANCHES1I -> BRANCHES2I [label="checkout/clone"]
BRANCHES2I -> BRANCHES3I [label="checkout/clone"]

REPO1 -> GIT
REPO2 -> GIT
REPO3 -> GIT

USER1a -> JOC1
USER1b -> JOC1
USER2 -> JOC2
USER3 -> JOC3

Repository Operations

• management
• versioning
• rollout
• force overwrite

Requirements

• The JOC Cockpit database holds the configuration objects:
  • each JOC Cockpit instance is connected to its own database,
  • each Git branch is mapped to an individual JOC Cockpit instance,
  • the database can be updated from a Git branch.
• each Git branch makes use of its own version of configuration objects.

Multiple Clients - Multiple Branches (dev, test, prod)

Scenario

• Architecture
  • A dedicated JOC Cockpit instance is used with each Controller and Agents for each user in the dev rollout environment.
  • A dedicated JOC Cockpit instance is used with each Controller and Agents for all users in the test and prod rollout environments.
  • The same Git repository is used per folder by all JOC Cockpit instances, however, each instance is acting on a specific branch per rollout environment.
  • The Git repository is used to hold a copy of JS7 configuration objects, to create versions of JS7 configuration objects and to merge branches such as dev => test, test => prod.
• Users
  • Users act in individual JOC Cockpit instances of the dev rollout environment as multiple clients to the Git server.
  • Users act in the same JOC Cockpit instance of the test and prod rollout environment as a single client to the Git server.
  • Changes to the Git repository are performed by the users' individual accounts to a each branch.
• Rollout
  • Deployments are performed from a single source (JOC Cockpit) to the connected Controller and Agents per rollout environment (dev, test, prod).
  • The Git repository is used to merge branches and to rollout JS7 configuration objects to different Controllers and Agents, such as dev => test => prod.
  • The Git repository can be used to fallback (pull) to previous versions of configuration objects, however, only one version is available with a given JOC Cockpit instance at a given point in time.

Branches marked as "env" are specific for an environment and are excluded from rollout. Branches marked as "rollout" are forwarded across scheduling environments.

USER1a [label="   User: 1a   ",fillcolor="lightskyblue"]
USER1b [label="   User: 1b   ",fillcolor="lightskyblue"]
USER2a [label="   User: 2a   ",fillcolor="lightskyblue"]
USER2b [label="   User: 2b   ",fillcolor="lightskyblue"]
USER3 [label="   User: 3   ",fillcolor="lightskyblue"]

JOC1a [label="   JOC Cockpit: dev 1a   ",fillcolor="lightskyblue"]
JOC1b [label="   JOC Cockpit: dev 1b   ",fillcolor="lightskyblue"]
JOC2  [label="   JOC Cockpit: test   ",fillcolor="lightskyblue"]
JOC3  [label="   JOC Cockpit: prod   ",fillcolor="lightskyblue"]

DATABASE1a [shape="ellipse",label="   Database: dev 1a   ",fillcolor="lightskyblue"]
DATABASE1b [shape="ellipse",label="   Database: dev 1b   ",fillcolor="lightskyblue"]
DATABASE2  [shape="ellipse",label="   Database: test   ",fillcolor="lightskyblue"]
DATABASE3  [shape="ellipse",label="   Database: prod   ",fillcolor="lightskyblue"]

GIT [label="Git Server",fillcolor="orange"]
REPO1 [label="Repository: dev",fillcolor="darkolivegreen1"]
REPO2 [label="Repository: test",fillcolor="darkolivegreen2"]
REPO3 [label="Repository: prod",fillcolor="darkolivegreen3"]

BRANCHES1R [shape="ellipse",label="Branches: env dev",fillcolor="darkolivegreen1"]
BRANCHES1I [shape="ellipse",label="Branches: rollout dev",fillcolor="darkolivegreen1"]
BRANCHES2R [shape="ellipse",label="Branches: env test",fillcolor="darkolivegreen2"]
BRANCHES2I [shape="ellipse",label="Branches: rollout test",fillcolor="darkolivegreen2"]
BRANCHES3R [shape="ellipse",label="Branches: env prod",fillcolor="darkolivegreen3"]
BRANCHES3I [shape="ellipse",label="Branches: rollout prod",fillcolor="darkolivegreen3"]

JOC1a -> DATABASE1a
JOC1b -> DATABASE1b
JOC2 -> DATABASE2
JOC3 -> DATABASE3

JOC1a -> BRANCHES1R [label="Commit/Push\nPull"]
JOC1a -> BRANCHES1I [label="Commit/Push\nPull"]
JOC1b -> BRANCHES1R [label="Commit/Push\nPull"]
JOC1b -> BRANCHES1I [label="Commit/Push\nPull"]
JOC2 -> BRANCHES2R [label="Commit/Push\nPull"]
JOC2 -> BRANCHES2I [label="Commit/Push\nPull"]
JOC3 -> BRANCHES3R [label="Commit/Push\nPull"]
JOC3 -> BRANCHES3I [label="Commit/Push\nPull"]

BRANCHES1R -> REPO1
BRANCHES1I -> REPO1

BRANCHES2R -> REPO2
BRANCHES2I -> REPO2

BRANCHES3R -> REPO3
BRANCHES3I -> REPO3

BRANCHES1I -> BRANCHES2I [label="checkout/clone"]
BRANCHES2I -> BRANCHES3I [label="checkout/clone"]

REPO1 -> GIT
REPO2 -> GIT
REPO3 -> GIT

USER3 -> JOC3
USER2a -> JOC2
USER2b -> JOC2
USER1a -> JOC1a
USER1b -> JOC1b

Repository Operations

• management
• versioning
• rollout
• force overwrite

Requirements

• •  
  • Release 2.3.0: Make JS7 objects available from the file system to allow use of external Git tools.

    • Jira
      server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JOC-1144

  • Release 2.4.0: Enable JOC Cockpit to manage JS7 objects with Git.

    • Jira
      server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JOC-1213

    • Jira
      server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JOC-1214

    • Jira
      server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JOC-1215

Object Mappings

Anchor
configuration_object_mappings configuration_object_mappings

Scheduling Objects

JS7 handles the separation of Rollout Objects and Local Objects as described below. JOC Cockpit allows configuration of the category applied for individual object types.

• Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JOC-1216

• Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JOC-1217

Rollout Objects

JS7 scheduling object types that are independent of a specific environment (dev, test, prod) and that can be rolled out across environments include:

• Workflows
• Resource Locks
• File Order Sources
• Notice Boards
• Script Includes

Such objects can be managed with JS7 in a way that allows re-use of the objects - without any changes - across rollout environments such as dev, test, prod.

Local Objects

The following scheduling object types typically hold values that are specific for an individual environment (dev, test, prod) and are therefore usually added to a separate repository. Such objects are not rolled out across environments.

• Job Resources
• Calendars
• Schedules

Anchor
folder_mappings folder_mappings

Folder Mappings

The JS7 inventory is organized with folders.

• Folders hold scheduling objects such as workflows etc. and any level of sub-folders.
• Folders are the basic unit for JS7 - Default Roles and Permissions. For example, users can be granted access only to specific folders.
• Folders allow bulk operations such as deploying, releasing, removing and the renaming of scheduling objects to be performed.

Folders can be mapped to Git repositories managed by the same or by different Git Servers like this:

USER1 [label="User: 1",fillcolor="lightskyblue",fontname="Arial",fontsize="10pt"]
USER2 [label="User: 2",fillcolor="lightskyblue",fontname="Arial",fontsize="10pt"]
USER3 [label="User: 3",fillcolor="lightskyblue",fontname="Arial",fontsize="10pt"]

FOLDER1 [label="Folder: 1",fillcolor="white",fontname="Arial",fontsize="10pt"]
FOLDER1a [label="Sub-folder: 1.1",fillcolor="white",fontname="Arial",fontsize="10pt"]
FOLDER1b [label="Sub-folder: 1.2",fillcolor="white",fontname="Arial",fontsize="10pt"]
FOLDER2 [label="Folder: 2",fillcolor="white",fontname="Arial",fontsize="10pt"]
FOLDER3 [label="Folder: 3",fillcolor="white",fontname="Arial",fontsize="10pt"]

REPOSITORY1L [shape="ellipse",label="Repository: 1-local\njob resources etc.",fillcolor="darkolivegreen1",fontname="Arial",fontsize="10pt"]
REPOSITORY1R [shape="ellipse",label="Repository: 1-rollout\nworkflows etc.",fillcolor="darkolivegreen1",fontname="Arial",fontsize="10pt"]
REPOSITORY2L [shape="ellipse",label="Repository: 2-local\njob resources etc.",fillcolor="darkolivegreen1",fontname="Arial",fontsize="10pt"]
REPOSITORY2R [shape="ellipse",label="Repository: 2-rollout\nworkflows etc.",fillcolor="darkolivegreen1",fontname="Arial",fontsize="10pt"]
REPOSITORY3L [shape="ellipse",label="Repository: 3-local\njob resources etc.",fillcolor="darkolivegreen1",fontname="Arial",fontsize="10pt"]
REPOSITORY3R [shape="ellipse",label="Repository: 3-rollout\nworkflows etc.",fillcolor="darkolivegreen1",fontname="Arial",fontsize="10pt"]

USER1 -> FOLDER1 [label="  can access  ",fontname="Arial",fontsize="10pt"]
USER2 -> FOLDER1 [label="  can access  ",fontname="Arial",fontsize="10pt"]
USER2 -> FOLDER2 [label="  can access  ",fontname="Arial",fontsize="10pt"]
USER3 -> FOLDER3 [label="  can access  ",fontname="Arial",fontsize="10pt"]

FOLDER1 -> FOLDER1a [label="  includes  ",fontname="Arial",fontsize="10pt"]
FOLDER1 -> FOLDER1b [label="  includes  ",fontname="Arial",fontsize="10pt"]
FOLDER1a -> REPOSITORY1L [label="  maps to  ",fontname="Arial",fontsize="10pt"]
FOLDER1a -> REPOSITORY1R [label="  maps to  ",fontname="Arial",fontsize="10pt"]
FOLDER1b -> REPOSITORY1L [label="  maps to  ",fontname="Arial",fontsize="10pt"]
FOLDER1b -> REPOSITORY1R [label="  maps to  ",fontname="Arial",fontsize="10pt"]

FOLDER2 -> REPOSITORY2L [label="  maps to  ",fontname="Arial",fontsize="10pt"]
FOLDER2 -> REPOSITORY2R [label="  maps to  ",fontname="Arial",fontsize="10pt"]
FOLDER3 -> REPOSITORY3L [label="  maps to  ",fontname="Arial",fontsize="10pt"]
FOLDER3 -> REPOSITORY3R [label="  maps to  ",fontname="Arial",fontsize="10pt"]

The mapping of folder objects to repositories implies:

• user access:
  • users with JS7 access permissions to manage folders and to access the Git repository can manage scheduling objects in the JOC Cockpit and can commit, push, pull scheduling objects to/from Git,
  • users with JS7 access permissions to manage folders but without access to the Git repository can manage scheduling objects in the JOC Cockpit,
  • users with JS7 access permissions to view folders do not require Git access.
• folder usage:
  • if a single Git repository should be used then all folders have to be available from a single top-level folder,
  • for use with a number of Git repositories each top-level folder maps to a different repository,
  • folders can remain without Git integration for local management and deployment.

JOC Cockpit maps top-level folders to Git repositories and stores the relevant Git repository URL for each folder in its inventory.

Git Access

User Access

User access is available in line with JOC Cockpit security levels used for deployment of workflows, see JS7 - Security Architecture.

• Security level LOW maps to use of a single Git account used by all users. Credentials are stored with the root account's profile in the JS7 database.
• Security level MEDIUM maps to use of an individual Git account per user. Credentials are stored with each user account's profile in the JS7 database.
• Security level HIGH does not allow storage of Git credentials with JOC Cockpit. Operations on the remote Git repository have to be performed with tools external to JOC Cockpit such as Git Extensions.

The following credentials are managed depending on the Git Server's authentication method:

• Git Server
  • host(:port) of the remote Git Server
• Git Account
  • account that is used to connect to a Git server
• User name
  • name used as author name e.g. for commits
• E-Mail
  • mail address used as author mail address, for example for commits
• password
  • the user's password for HTTP(S) Git access
• private key
  • the path to the user's private key file for SSH Git access
• Access Token
  • the access token for HTTP(S) Git access

If repositories are used from a number of Git Servers, that require different sets of user credentials, then this can be managed within one JOC Cockpit account, Only one set of credentials per Git Server is used.

If repositories are used from the same Git Servers that require different sets of user credentials then separate user accounts in JOC Cockpit have to be used.

• Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JOC-1216

• Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JOC-1217

Repository Access

• The JS7 - REST Web Service API executes a set of Git operations by using the Git Command Line Client.
• All operations to add scheduling objects to a repository, to stage, commit, push, pull and to merge objects are available from the JS7 REST Web Service API.
• The JOC Cockpit provides a user interface for Git operations.
• Users are free to use additional tools such as Git Extensions to manage repositories.

Git Operations

Management

JS7 scheduling objects are transparently managed in a Git repository via the JS7 REST Web Service API. In addition, tools such as Git Extensions and Git command line clients can be used for repository operations.

JOC Cockpit holds a number of local repositories inside the JETTY_BASE/resources/joc/repositories folder. Each top-level folder in the JS7 Configuration view can be mapped to a Git repository. 

• The local repository for a JS7 top-level folder is created when a user performs the clone operation for the Git repository via the JOC Cockpit.
• The local repository is persistent and is updated from the JOC Cockpit database (store operation) and from the remote Git repository (pull operation).
• The remote Git repository is updated from the local repository by use of the push operation.
• The JS7 - Cleanup Service removes any local repositories if the JS7 top-level folder no longer exists, i.e. if a top-level folder has been dropped from the JOC Cockpit.

The following REST Web Service API operations are available:

• /inventory/repository/git/checkout
  • checks out a specific branch or tag from the Git repository to the local repository
• /inventory/repository/git/clone
  • initially clones a Git repository to a local repository
• /inventory/repository/git/add
  • adds or updates (stages) all new or modified scheduling object on the file system to the local repository
• /inventory/repository/git/commit
  • commits all staged scheduling objects to a local repository
• /inventory/repository/git/push
  • pushes changes from the local repository to the remote Git repository

Versioning

Versioning of JS7 scheduling objects is performed by using the tagging capabilities of Git. 

The JS7 REST Web Service API manages the Git tagging process.

The following operations are available:

• git-tag
  • JS7 REST Web Service API not implemented yet
  • tags a commit with a version label
• /inventory/repository/git/push
  
  • see above

Rollout

Rollout between JS7 instances is performed using the Git functionality. The JS7 REST Web Service API manages the processes for checkout and cloning.

The following operations are available:

• /inventory/repository/git/checkout
  
  • checks out a specific branch or tag on the local repository
• /inventory/repository/git/clone
  • initially clones a remote repository to a local repository
• /inventory/repository/git/pull
  • pulls recent changes from the remote repository to the local repository

JOC Cockpit Operations

The synchronization of scheduling objects between the JOC Cockpit database and the local repositories is managed by the JS7 REST Web Service API and is available from the JOC Cockpit GUI.

The following JS7 REST Web Service operations are available:

• /inventory/repository/read
  • reads the file tree of the local repository file system folder and updates the JOC Cockpit database
• /inventory/repository/store
  • stores a set of scheduling objects from the JOC Cockpit database to the file system folder of the local repository
• /inventory/repository/delete
  
  • deletes a set of scheduling objects from the file system folder of the local repository
• /inventory/repository/update
  
  • adds or updates a set of scheduling objects in the JOC Cockpit database with scheduling objects from the local repository's file system folder

Scenarios

Single Client - Multiple Repositories

Scenario

• Architecture
  • A dedicated JOC Cockpit instance is used with Controller and Agents for each rollout environment such as dev, test, prod.
  • A set of two Git repositories is used per folder and per rollout environment:
    • one repository holding local objects such as job resources that are specific for an environment,
    • one repository holding rollout objects such as workflows that are rolled out across environments.

  • The JOC Cockpit database holds the JS7 scheduling objects:

    • each JOC Cockpit instance is connected to its own database,
    • each repository is mapped to an individual JOC Cockpit instance,
    • the database can be updated from a local repository.
• Users
  • All users act in the same JOC Cockpit instance of the relevant rollout environment that is a single client to the Git server.
  • Changes to the Git repositories are performed on behalf of the users' individual accounts and are added to the remote repositories.
• Rollout
  • Deployments are performed from a single source (JOC Cockpit) to each Controller and Agents per rollout environment (dev, test, prod).
  • Rollouts are performed by copying rollout objects from a local Git repository to a remote Git repository of the target rollout environment (dev => test => prod).

Repositories holding local objects are excluded from rollout. Repositories holding rollout objects are copied across repositories and environments.

USER1a [label="  User: 1a  ",fillcolor="lightskyblue1",fontname="Arial",fontsize="10pt"]
USER1b [label="  User: 1b  ",fillcolor="lightskyblue1",fontname="Arial",fontsize="10pt"]
USER2 [label="  User: 2  ",fillcolor="lightskyblue2",fontname="Arial",fontsize="10pt"]
USER3 [label="  User: 3  ",fillcolor="lightskyblue3",fontname="Arial",fontsize="10pt"]

JOC1 [label="  JOC Cockpit: dev  ",fillcolor="lightskyblue1",fontname="Arial",fontsize="10pt"]
JOC2 [label="  JOC Cockpit: test  ",fillcolor="lightskyblue2",fontname="Arial",fontsize="10pt"]
JOC3 [label="  JOC Cockpit: prod  ",fillcolor="lightskyblue3",fontname="Arial",fontsize="10pt"]

FOLDER1 [label="  Folder: 1 (dev)  ",fillcolor="white",fontname="Arial",fontsize="10pt"]
FOLDER2 [label="  Folder: 1 (test)  ",fillcolor="white",fontname="Arial",fontsize="10pt"]
FOLDER3 [label="  Folder: 1 (prod)  ",fillcolor="white",fontname="Arial",fontsize="10pt"]

DATABASE1 [shape="ellipse",label="  Database: dev  ",fillcolor="lightskyblue1",fontname="Arial",fontsize="10pt"]
DATABASE2 [shape="ellipse",label="  Database: test  ",fillcolor="lightskyblue2",fontname="Arial",fontsize="10pt"]
DATABASE3 [shape="ellipse",label="  Database: prod  ",fillcolor="lightskyblue3",fontname="Arial",fontsize="10pt"]

REPO1L [shape="ellipse",label="Repository: 1-dev-local\njob resources etc.",fillcolor="darkolivegreen1",fontname="Arial",fontsize="10pt"]
REPO1R [shape="ellipse",label="Repository: 1-dev-rollout\nworkflows etc.",fillcolor="darkolivegreen1",fontname="Arial",fontsize="10pt"]
REPO2L [shape="ellipse",label="Repository: 1-test-local\njob resources etc.",fillcolor="darkolivegreen2",fontname="Arial",fontsize="10pt"]
REPO2R [shape="ellipse",label="Repository: 1-test-rollout\nworkflows etc.",fillcolor="darkolivegreen2",fontname="Arial",fontsize="10pt"]
REPO3L [shape="ellipse",label="Repository: 1-prod-local\njob resources etc.",fillcolor="darkolivegreen3",fontname="Arial",fontsize="10pt"]
REPO3R [shape="ellipse",label="Repository: 1-prod-rollout\nworkflows etc.",fillcolor="darkolivegreen3",fontname="Arial",fontsize="10pt"]

{ rank = same; JOC1 -> DATABASE1 }
{ rank = same; JOC2 -> DATABASE2 }
{ rank = same; JOC3 -> DATABASE3; }

JOC1 -> FOLDER1 
FOLDER1 -> REPO1L [label="  commit\n  push/pull  ",fontname="Arial",fontsize="10pt"]
FOLDER1 -> REPO1R [label="  commit\n  push/pull  ",fontname="Arial",fontsize="10pt"]

JOC2 -> FOLDER2
FOLDER2 -> REPO2L [label="  commit\n  push/pull  ",fontname="Arial",fontsize="10pt"]
FOLDER2 -> REPO2R [label="  commit\n  push/pull  ",fontname="Arial",fontsize="10pt"]

JOC3 -> FOLDER3 
FOLDER3 -> REPO3L  [label="  commit\n  push/pull  ",fontname="Arial",fontsize="10pt"]
FOLDER3 -> REPO3R  [label="  commit\n  push/pull  ",fontname="Arial",fontsize="10pt"]

REPO1R -> REPO2R [label="  copy  ",fontname="Arial",fontsize="10pt"]
REPO2R -> REPO3R [label="  copy  ",fontname="Arial",fontsize="10pt"]

USER1a -> JOC1;
USER1b -> JOC1;
USER2 -> JOC2;
USER3 -> JOC3;

JOC Cockpit Operations

The following operations are provided from the JS7 REST Web Service API and from the JOC Cockpit GUI: 

• Job Management
  • Manage changes to scheduling objects such as workflows and jobs.
  • Deploy scheduling objects to Controller and Agents.
• Repository Management
  • Store and commit changes to a set of local repositories, push changes to a set of remote repositories.
    • one repository holding local objects such as job resources that are specific for an environment
    • one repository holding rollout objects such as workflows that are rolled out across environments.
  • Pull changes from a remote repository to a local repository and update JOC Cockpit from the local repository.
  • Tag changes (commits) for versioning purposes in a remote repository.

Git Operations

The following operations can be performed with a Git Client outside of JOC Cockpit:

• Operations similarly available from JOC Cockpit:
  • Push changes from a local repository to a remote repository.
  • Pull changes from a remote repository to a local repository.
  • Tag changes (commits) for versioning purposes in a remote repository.
• Operations not available from JOC Cockpit:
  • Manage branches in a remote repository.
  • Checkout branches to a local repository.

Multiple Clients - Multiple Repositories

Scenario

• Architecture
  • A dedicated JOC Cockpit instance is used with Controller and Agents
    • for each user in the dev environment,
    • for all users in the test and prod environments.
  • A set of two Git repositories are used per folder and per rollout environment (dev, test, prod):
    • one repository holding local objects such as job resources that are specific for an environment,
    • one repository holding rollout objects such as workflows that are rolled out across environments.
• Users
  • Users act in individual JOC Cockpit instances of the dev environment as multiple clients to the Git server.
  • Users act in the same JOC Cockpit instance of the test and prod environments as a single client to the Git server.
  • Changes to the Git repositories are performed on behalf of the users' individual accounts.
• Rollout
  • Deployments are performed from multiple sources (JOC Cockpit) to the connected Controller and Agents per rollout environment.
  • Rollouts are performed by copying rollout objects from a local Git repository to a remote Git repository of the target rollout environment (dev => test => prod).

Repositories holding local objects are excluded from rollout. Repositories holding rollout objects are copied across repositories and environments.

USER1a [label="  User: 1a  ",fillcolor="lightskyblue1",fontname="Arial",fontsize="10pt"]
USER1b [label="  User: 1b  ",fillcolor="lightskyblue1",fontname="Arial",fontsize="10pt"]
USER2a [label="  User: 2a  ",fillcolor="lightskyblue2",fontname="Arial",fontsize="10pt"]
USER2b [label="  User: 2b  ",fillcolor="lightskyblue2",fontname="Arial",fontsize="10pt"]
USER3 [label="  User: 3  ",fillcolor="lightskyblue3",fontname="Arial",fontsize="10pt"]

JOC1a [label="JOC Cockpit: dev 1a",fillcolor="lightskyblue1",fontname="Arial",fontsize="10pt"]
JOC1b [label="JOC Cockpit: dev 1b",fillcolor="lightskyblue1",fontname="Arial",fontsize="10pt"]
JOC2  [label="JOC Cockpit: test",fillcolor="lightskyblue2",fontname="Arial",fontsize="10pt"]
JOC3  [label="JOC Cockpit: prod",fillcolor="lightskyblue3",fontname="Arial",fontsize="10pt"]

#DATABASE1a [shape="ellipse",label="Database: dev 1a",fillcolor="lightskyblue1",fontname="Arial",fontsize="10pt"]
#DATABASE1b [shape="ellipse",label="Database: dev 1b",fillcolor="lightskyblue1",fontname="Arial",fontsize="10pt"]
#DATABASE2  [shape="ellipse",label="Database: test",fillcolor="lightskyblue2",fontname="Arial",fontsize="10pt"]
#DATABASE3  [shape="ellipse",label="Database: prod",fillcolor="lightskyblue3",fontname="Arial",fontsize="10pt"]

REPO1L1 [shape="ellipse",label="Repository: dev-local-1a\njob resources etc.",fillcolor="darkolivegreen1",fontname="Arial",fontsize="10pt"]
REPO1L2 [shape="ellipse",label="Repository: dev-local-1b\njob resources etc.",fillcolor="darkolivegreen1",fontname="Arial",fontsize="10pt"]
REPO1R  [shape="ellipse",label="Repository: dev-rollout\nworkflows etc.",fillcolor="darkolivegreen1",fontname="Arial",fontsize="10pt"]
REPO2L  [shape="ellipse",label="Repository: test-local\njob resources etc.",fillcolor="darkolivegreen2",fontname="Arial",fontsize="10pt"]
REPO2R  [shape="ellipse",label="Repository: test-rollout\nworkflows etc.",fillcolor="darkolivegreen2",fontname="Arial",fontsize="10pt"]
REPO3L  [shape="ellipse",label="Repository: prod-local\njob resources etc.",fillcolor="darkolivegreen3",fontname="Arial",fontsize="10pt"]
REPO3R  [shape="ellipse",label="Repository: prod-rollout\nworkflows eetc.",fillcolor="darkolivegreen3",fontname="Arial",fontsize="10pt"]

#{ rank = same; JOC1a -> DATABASE1a }
#{ rank = same; JOC1b -> DATABASE1b }
#{ rank = same; JOC2 -> DATABASE2 }
#{ rank = same; JOC3 -> DATABASE3; }

JOC1a -> REPO1L1 [label="  commit/push/pull  ",fontname="Arial",fontsize="10pt"]
JOC1a -> REPO1R  [label="  commit/push/pull  ",fontname="Arial",fontsize="10pt"]
JOC1b -> REPO1L2 [label="  commit/push/pull  ",fontname="Arial",fontsize="10pt"]
JOC1b -> REPO1R  [label="  commit/push/pull  ",fontname="Arial",fontsize="10pt"]
JOC2  -> REPO2L  [label="  commit/push/pull  ",fontname="Arial",fontsize="10pt"]
JOC2  -> REPO2R  [label="  commit/push/pull  ",fontname="Arial",fontsize="10pt"]
JOC3  -> REPO3L  [label="  commit/push/pull  ",fontname="Arial",fontsize="10pt"]
JOC3  -> REPO3R  [label="  commit/push/pull  ",fontname="Arial",fontsize="10pt"]

REPO1R -> REPO2R [label="  copy  ",fontname="Arial",fontsize="10pt"]
REPO2R -> REPO3R [label="  copy  ",fontname="Arial",fontsize="10pt"]

USER1b -> JOC1b
USER1a -> JOC1a
USER2a -> JOC2
USER2b -> JOC2
USER3 -> JOC3

Operations

JOC Cockpit operations and Git operations are the same as for the above scenario: Single Client - Multiple Repositories.

Use Cases

Lifecycle Support for Workflow Changes

Use Case

Each environment dev, test, prod makes use of an individual JS7 instance using JOC Cockpit and database, Controller and Agents. Any operations are performed by the JS7 REST Web Service API and are offered by the JOC Cockpit GUI.

• Job Development in a dev environment 
  
  • A number of scheduling objects, e.g. workflows, are created or changed by a job developer.
  • On completion of the change the job developer deploys the workflows to Controller and Agents to perform functional tests in the dev environment.
  • With functional tests being successful the developer:
    • stores changes to workflows and related rollout objects to the local dev-rollout repository, commits changes and pushes changes to the remote dev-rollout repository.
    • optionally stores changes of local objects such as job resources to the local dev-local repository, commits changes and pushes changes to the remote dev-local repository.
• Integration Testing in a test environment
  • A tester with access to the job developer's remote dev-rollout repository adopts the current change:
    • The tester performs a checkout operation of the remote dev-rollout repository to a local dev-rollout repository.
    • The tester is offered the list of recent commits to the dev-rollout repository. From this list the tester filters the commits related to the change by date, author or message. Typically the change is assigned an identifier (Issue Key or similar) that is available with the messages of any commits related to this change.
    • From the selected commits the tester receives the list of changed files and launches the operation to copy the files in this list from the local dev-rollout repository to the local test-rollout repository.
    • The tester updates the JOC Cockpit instance from the local test-rollout repository.
  • The tester adjusts local objects such as job resources as required by the change.
  • The tester deploys changed workflows and related local objects to Controller and Agents to perform integration testing.
  • On completion of integration tests with changes being approved the tester:
    • optionally tags the latest commit in the remote test-rollout repository with a label, e.g. v2.3-approved.
      • Should the tester modify workflows then the tester commits changes to the local test-rollout repository and pushes changes to the remote test-rollout repository before tagging.
      • Some users might deny object modifications being performed in a test environment and instead prefer to apply modifications in dev environments only.
    • optionally stores changes of local objects to the local test-local repository, commits changes and pushes changes to the remote test-local repository.
• Production Rollout to a prod environment
  • An operator with access to the tester's test-rollout repository adopts the current change:
    • The operator performs a checkout operation of the remote test-rollout repository to a local test-rollout repository. Optionally the checkout is performed for the tag v2.3-approved should the tester have tagged the change accordingly.
    • The operator is offered the list of recent commits to the test-rollout repository. From this list the operator filters the commits related to the change by date, author or message. Typically the change is assigned an identifier (Issue Key or similar) that is available with the messages of any commits related to this change.
    • From the selected commits the operator receives the list of changed files and launches the operation to copy the files in this list from the local test-rollout repository to the local prod-rollout repository.
    • The operator updates the JOC Cockpit instance from the local prod-rollout repository.
  • The operator adjusts local objects such as job resources as required by the change. 
  • The operator deploys changed workflows and local objects to the Controller and Agents for production use.
  • On completion of deployment the operator:
    • stores changes to the local prod-rollout repository, commits changes and pushes changes to the remote prod-rollout repository.
    • optionally tags changes in the remote prod-rollout repository with a label, e.g. v2.3.
    • optionally stores changes to local objects to the local prod-local repository, commits changes and pushes changes to the remote prod-local repository.

References

• Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JOC-1213

• Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JOC-1214

Rollout by a CI/CD Pipeline

Use Case

Each environment dev, test, prod makes use of an individual JS7 instance using JOC Cockpit and database, Controller and Agents. Any operations are performed by the JS7 REST Web Service API.

• Job Development and Integration Testing are explained in the use case described above: Lifecycle Support for Workflow Changes.
• Production Rollout
  • Checkout remote test-rollout repository to local test-rollout repository, optionally based on use of a tag such as v2.3-approved.
  • Filter commits relevant to a change by an identifier available from the commit message and create the list of files affected by resulting commits.
  • Copy files in this list from the local test-rollout repository to the local prod-rollout repository.
  • Update the JOC Cockpit instance from the local prod-rollout repository.
  • Deploy changed workflows to Controller and Agents for production use.
  • Store changes to the local prod-rollout repository, commit changes and push changes to the remote prod-rollout repository.
  • Optionally tag changes in the remote prod-rollout repository with a label, e.g. v2.3.

Automation

Automation of the CI/CD pipeline can be achieved by scripting using the JS7 REST Web Service API:

• from the JS7 - PowerShell Module, see JS7 - How to rollout from test to prod environments in a CI CD pipeline.
• from any scripting language that supports a REST Client, executing curl from a shell script might be sufficient.
• Operations can be performed on any platform, no direct access to the JS7 production environment is required, however, credentials have to be provided to connect to the JS7 REST Web Service API.

References

• Jira
  server SOS JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 6dc67751-9d67-34cd-985b-194a8cdc9602 key JOC-1215

• The JOC Cockpit database holds the configuration objects:
  • each JOC Cockpit instance is connected to its own database.
  • each Git branch is connected to one or more JOC Cockpit instances.
• Each branch makes use of an individual version.
• Each user (client) makes use of an individual JS7 instance using JOC Cockpit and database, Controller and Agents