Problem

If you face the error

 Exception in thread "main" java.lang.Exception: connect to database failed: Io exception: Connection reset

or

 java.lang.RuntimeException: Timeout reached (30s) for process: 

then you are probably being hit by a problem with your entropy pool or network settings. This problem can occur with any JDBC database connection and with any operating system. It is not related to the JOC Cockpit or the DBMS.

The article explains why this happens and what you can do about it.

Entropy Pool Issues

The JDBC interface requires random numbers to encrypt the connection. Java releases before 1.12 use the /dev/random file for a high randomness quality. However, when the entropy pool falls below the number of 64 units then /dev/random will block while reading random numbers.

The JDBC interface might be configured to read from the file /dev/urandom to get random numbers. The difference between the two files is that /dev/urandom does not block if random numbers are not immediately available.

Check Entropy Pool Issues (Unix)

Check Entropy Pool Configuration

You can check available entropy pool units with the command:

cat /proc/sys/kernel/random/entropy_avail

If the "entropy_avail" result is too small (JDBC needs 40 bytes of secure random numbers) then you have to increase the pool by producing some environmental noise. This could be a hurdle when you operate a headless server (no console) as the noise is produced by the keyboard, mouse, login etc.

Check the entropy pool size (normally 4096) with the command:

cat /proc/sys/kernel/random/poolsize

The /dev/random file will deliver the next random number when the pool has reached more than 64 entropy units and otherwise blocks applications from accessing the entropy pool. Such blocks can delay, for example, a JDBC connection to a database and may result in timeouts being exceeded.

Check Temporary Resolution

To verify the entropy pool being the root cause of this issue try this (requires root permission):

rm /dev/random
ln -s /dev/urandom /dev/random

If this solves your problem then the JDBC interface was not able to get random numbers from the OS in good time. Please note that the effect of the above commands is reverted on reboot.

Monitor Entropy Pool Use

You can check consumption of random numbers by running the following commands in two separate console windows:

while true
do
    cat /proc/sys/kernel/random/entropy_avail
    sleep 1
done

You can consume random numbers from a script like this - be careful as this affects parallel applications that consume random numbers:

for ((i=1; i<=100; i++)); do
   head -100 /dev/random | cksum | cut -f1 -d ' '
   sleep 0.1
done

You can consume random numbers from a similar script like this:

# initial test
dd if=/dev/random of=/dev/null bs=1024 count=1 iflag=fullblock 

# full test (should rngtest be available)
rngtest -c 100 </dev/random

Resolve Entropy Pool Issues

There are two alternative solutions: modify the Java security settings or modify the JOC Cockpit settings.

Both solutions apply to Unix and Windows operating systems.

Modify Java Security Configuration

Java holds the security configuration, for example, in the ./jre/lib/security/java.security or ./conf/security/java.security files. You can modify this file to point to /dev/urandom instead of /dev/random like this:

# original configuration
# securerandom.source=file:/dev/random

# updated configuration
securerandom.source=file:/dev/urandom

Modify JOC Cockpit Configuration

Installation

Should the entropy issue have occurred during installation then create or update the JAVA_OPTIONS environment variable like this:

export JAVA_OPTIONS="-Djava.security.egd=file:///dev/urandom"

set JAVA_OPTIONS="-Djava.security.egd=file:///dev/urandom"

Operation

For the permanent operation of the JOC Cockpit on Unix, add the following setting to your /home/<user-account>/.jocrc file:

export JAVA_OPTIONS="-Djava.security.egd=file:///dev/urandom"

For the permanent operation of the JOC Cockpit on Windows, modify or add the following setting to your /home/<user-account>/.jocrc file:

When operating the JOC Cockpit as a Windows Service run the following from the .\service directory of the installation:

• for JOC Cockpit: js7_jocw.exe
  
  • Example: C:\Program Files\sos-berlin.com\js7\joc\service\js7_jocw.exe
• This brings up a utility which allows Java options to be specified:
  
  
  
  

For further information see the JS7 - How To - Apply Java Options article.

Network Issues

A wrong network configuration can cause delays when executing Java and when accessing a database - for example, if host name resolution takes too long. 

For Unix check whether entries for name servers and host name resolution are correct in the /etc/resolv.conf configuration file.

Oracle® DBMS Issues

The Oracle® FAN functionality provides enhanced high availability by allowing very fast detection of failures. However, if the FAN functionality is not set up correctly, this can result in a denial of connection although the Oracle listener is accessible, for example, by tnsping.

To check this root cause the FAN can be disabled by using a Java option like this:

export JAVA_OPTIONS="-Doracle.jdbc.fanEnabled=false"

Other Root Causes

Another possible reason for delays could be a huge number of files in /tmp as the JDBC interface tries to list files in the /tmp directory when SecureRandom.nextBytes(byte[]) is invoked.