Introduction

Amazon AWS® CloudWatch is a service of Amazon Web Services that monitors the customer's applications and resources running on the AWS® infrastructure in real-time. It is used to track and collect metrics, variables you can measure for your applications and resources. AWS® CloudWatch monitors resources like Amazon Elastic Compute Cloud (EC2®) instances, Elastic Load Balancing (ELB®), Amazon Elastic Block Store (EBS®) volumes, and Amazon Relational Database Service (RDS®) instances.

JS7 can be integrated with the AWS® CloudWatch. This integration allows users to check the logs and current statistics in a graphical UI. Users can also detect and shutdown unused or underused EC2® instances.

For JS7 JobScheduler as a Service, find the following suggested architecture.

Set up CloudWatch Agent

Create IAM Role

• Login to AWS® Management Console and open the IAM console. Select Roles from the menu and then click the Create role button.
• Select the service that will use this Role for the AWS® services (EC2, Lambda, and others). Select EC2 and click the Next: Permission button.
• To select the policy for your new Role, search for the CloudWatchAgentServerPolicy, check the checkbox and create the Role.

Attach the IAM Role

To allow an EC2 instance to connect with AWS® CloudWatch, the user must attach the IAM role to the EC2 Instance. It can be done through the AWS console or via the AWS Command Line Interface (CLI).

• Using the AWS Console
  To attach the IAM Role, navigate to the EC2 Dashboard, select Instance from the menu and check the checkbox of the EC2 Instance. Click on the Actions dropdown and choose Security>Modify IAM Role>Choose IAM Role and update IAM Role.
  
  
• Using the AWS CLI
  For AWS CLI Version 1.11.46, use the below command to attach the IAM Role:
  
  

$ aws ec2 associate-iam-instance-profile --instance-id YourInstanceId --iam-instance-profile Name=CloudWatchAgentServerRole

Install CloudWatch Agent

Run the following commands to connect to the EC2 instance. First, you need to install CloudWatch Agent from S3. Use the below command to download it for AMD64 Ubuntu.

$ wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

For Windows, follow the below command to connect to the EC2 instance.

Invoke-WebRequest -Uri "https://s3.amazonaws.com/amazoncloudwatch-agent/windows/amd64/latest/amazon-cloudwatch-agent.msi" -OutFile "C:\amazon-cloudwatch-agent.msi" 

Use the following command to install the Agent on Ubuntu:

$ sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

To install the Agent on Windows user need to execute the below command from the path where the MSI file is installed:

msiexec /i amazon-cloudwatch-agent.msi

After installing the CloudWatch Agent, the user needs to be configured before it starts. It can be configured in two ways: by manually creating a config file or using the wizard (In which the user needs to answer a series of questions). It generates a config file.

1. Manually create config.json

The log agent uses the config file. Users need to create it at the below path:

/opt/aws/amazon-cloudwatch-agent/bin/config.json

From the above path, create or edit the config file with the following content:

{
     "agent": {
         "run_as_user": "root"
     },
     "logs": {
         "logs_collected": {
             "files": {
                 "collect_list": [
                     {
                         "file_path": "{path to your log file}",
                         "log_group_name": "scheduler-error-log",
                         "log_stream_name": "{instance_id}"
                     }
                 ]
             }
         }
     }
 }

In the above config file, "file_path" is the path to the log file from which the user wants to collect data from e.g., controller.log, agent.log. For the naming of Log Group and Log Streams, log_group_name and log_stream_name can be used respectively in the CloudWatch.

2. Wizard

Follow the below command to start the wizard run:

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

In the wizard run, there are a number of questions as the Log Agent used to collect system-level metrics, so users need to answer these questions and can ignore the questions that are not related to collecting logs. While using the wizard, users can always take the generated config file and then manually add it to any additional instances. 

Start the Agent

Run the below command to run the Agent.

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

View Logs

From the CloudWatch Overview, the user can select Logs from the menu. The user can find the label for the Log Group created in the configuration of Logs. Select the Log Group name to see the Log Streams. Click on the Log Stream uses the EC2 instance ID, so the user knows in which EC2 instance the data is logged.

The user needs to search in the filter text box to filter the required logs. All the searched logs will appear.