ORACLE IO exception - Connection reset or Timeout reached - 30s - for process

Error Reported

If you face the error

 Exception in thread "main" java.lang.Exception: connect to database failed: Io exception: Connection reset

or

 java.lang.RuntimeException: Timeout reached (30s) for process:

then you probably are hit by a problem with your entropy pool or network settings. This problem can occur with any JDBC database connection and is not limited to use of the Oracle® DBMS. This problem is not related to JobScheduler and here is explained why this happens and what you can do about this.

Entropy Pool Issues

The JDBC interface requires random numbers to encrypt the connection string. Normally the /dev/random file is used for a high quality of randomness. But when the entropy pool is is falling below the number of 64 units then /dev/random will block while reading random numbers.

The JDBC interface might be configured to read from the file /dev/random to get random numbers. The difference with the /dev/urandom file is, that /dev/urandom does not block if no random numbers are available.

Check Entropy Pool Issues

Check Entropy Pool Configuration

You can check available entropy pool units with the command:

cat /proc/sys/kernel/random/entropy_avail

The /dev/random file will deliver the next random number when the pool has reached more than 64 entropy units and otherwise blocks any application accessing the entropy pool. Such blocks can delay e.g. a JDBC connection to a database and may result in timeouts being exceeded.

Check the entropy pool size (normally 4096) with the command:

cat /proc/sys/kernel/random/poolsize

If the "entropy_avail" result is too small (JDBC needs 40 bytes of secure random numbers) then you have to increase the pool by producing some environmental noise. This could be a hurdle, when you operate a headless server (no console) as the noise is produced by keyboard, mouse etc.

Check Temporary Resolution

To verify the entropy pool being the root cause of this issue try this (requires root permission):

Make /dev/random symlink to /dev/urandom

rm /dev/random
ln -s /dev/urandom /dev/random

If this solves your problem then the JDBC interface was not able to get random numbers from the OS in good time. Please note that the effect of the two given commands is reverted on reboot.

Monitor Entropy Pool Use

You can check use of random numbers by running the following commands in two separate console windows:

Monitor use of random numbers with Unix

while true
do
    cat /proc/sys/kernel/random/entropy_avail
    sleep 1
done

Run test for random numbers with Unix

# initial test
dd if=/dev/random of=/dev/null bs=1024 count=1 iflag=fullblock 

# full test (should rngtest be available)
rngtest -c 100 </dev/random

Resolve Entropy Pool Issues

There are two alternative solutions to modify Java security settings or to modify JobScheduler settings.

Modify Java Security Configuration

Java holds the security configuration with the ./jre/lib/security/java.security file. You can modify this file to point to /dev/urandom instead of /dev/random like this:

Modification to java.security file

# original configuration
# securerandom.source=file:/dev/random

# updated configuration
securerandom.source=file:/dev/urandom

Modify JobScheduler Configuration

Installation

Should the entropy issue have occurred during installation then create or update the JAVA_OPTIONS environment variable like this:

Set environment variable JAVA_OPTIONS with Unix

export JAVA_OPTIONS="-Djava.security.egd=file:///dev/urandom"

Set environment variable JAVA_OPTIONS with Windows

set JAVA_OPTIONS="-Djava.security.egd=file:///dev/urandom"

Job Scheduler Operation

For permanent operation of JobScheduler add the following setting to your JobScheduler's ./config/sos.ini file:

Set Java options with ./config/sos.ini

[java]
options                = -Djava.security.egd=file:///dev/urandom

Job Execution

For execution of jobs with a JobScheduler Master add to your JobScheduler's ./config/sos.ini file:

Set Java job options with ./config/sos.ini

[java]
job-options            = -Djava.security.egd=file:///dev/urandom

When running jobs with Agents then you can use the following environment variables in your Agent's instance start script, e.g. in ./bin/jobscheduler_agent_4445.sh | .cmd file, like this:

Set environment variables JAVA_OPTIONS and SCHEDULER_JOB_JAVA_OPTIONS with Unix

JAVA_OPTIONS="-Djava.security.egd=file:///dev/urandom"
SCHEDULER_JOB_JAVA_OPTIONS="-Djava.security.egd=file:///dev/urandom"

Set environment variables JAVA_OPTIONS and SCHEDULER_JOB_JAVA_OPTIONS with Windows

set JAVA_OPTIONS="-Djava.security.egd=file:///dev/urandom"
set SCHEDULER_JOB_JAVA_OPTIONS="-Djava.security.egd=file:///dev/urandom"

Alternatively to the above add the similar setting for jobs in the "Java Options" section of the job definition like this:

Set Java options with job definition

<job java_options="-Djava.security.egd=file:///dev/urandom">
    <script  language="java" java_class_path="" java_class="..."/>
    <run_time />
</job>

Network Issues

A wrong network configuration can cause delays when executing Java and when accessing a database, e.g. if host name resolution takes too long.

For Unix check the /etc/resolve.conf configuration file if entries for name servers and host name resolution are correct.

Other Root Causes

Another possible reason for delays could be a huge number of files in /tmp as the JDBC interface tries to list files in the /tmp directory when SecureRandom.nextBytes(byte[]) is invoked.

Space shortcuts

Page tree