Multiple ORDS instances in WebLogic Server or Apache Tomcat

ORDS 22.1.0 introduced a substantial set of changes: Java 11 as minimum, new command line interface and a new configuration directory structure. In fact it is a release that introduces a whole new deployment paradigm.

Deployment Options

What we knew as Standalone Mode in previous ORDS releases still exists and is now initiated through a serve command. The embedded Eclipse Jetty web server and servlet container engine is still at the heart of this convenient way to have ORDS listen HTTP/HTTPS traffic. The significant change is how the configuration directory is determined at startup. There is no longer a requirement to modify the distributed ords.war to set the configuration directory location.

Deployment on a supported servlet container, such as Oracle WebLogic Server or Apache Tomcat, is similarly impacted by this principle that the distributed ords.war should not be modified. A new mechanism for specifying the configuration directory when the application is deployed is required. The ORDS documentation covers the standard approach for a typical setup, use a Java System Property ( -Dconfig.url=/ords_config/ ), startup your server(s) and deploy the web application.

Single Server

For some developers that may not have a full production environment scaled out for high availability, or may be restricted in the Apache Tomcat or WebLogic Server deployment options, the approach to set different servers with their own ords.war and configuration is not always feasible. With previous ORDS releases they may have had the practice of deploying multiple ORDS web applications with different contexts and configurations. At first it would appear that the new direction in ORDS 22.1.0 makes that impossible but it is not. All that is required are a few extra steps.

Bake your own

For deploying to Apache Tomcat or WebLogic Server you can make your own, separate web application with the config.url baked in. Moreover, you can call it whatever makes sense in your context.

Heres a Python script to do that for you: create_deploy_war.py

This script produces a new web application archive file with the jars from the distributed ords.war and web.xml deployment descriptor has the configuration directory path set. That way, wherever it is deployed, it will refer to the specified configuration directory.

Usage: create_deploy_war <source ords.war> <destination war filename> <configuration directory>

For example, if I want to have two ORDS instances deployed called tom and jerry with two separate configuration directories and I have extracted the ORDS 22.1.0 to /opt/oracle/ords-22.1.0.105.1723/. The jerry web application will be configured with a default pool pointing to an Oracle 19c database. The tom web application will be configured with a default pool pointing to an Oracle 21c database. We will show both web applications have these configurations by using the REST Enabled SQL service in ORDS to return database version information. Before that, the web application files must first be created…

./create_deploy_war.py /opt/oracle/ords-22.1.0.105.1723/ords.war /scratch/ords_webapps/tom.jar /scratch/ords_configs/tom/
./create_deploy_war.py /opt/oracle/ords-22.1.0.105.1723/ords.war /scratch/ords_webapps/jerry.jar /scratch/ords_configs/jerry/

That will produce two web application war files in /scratch/ords_webapps/ and I can then deploy them…

Oracle WebLogic Server

Two ORDS web applications deployed to a single WebLogic Server
curl -u hr:hr http://localhost:7001/jerry/hr/_/sql

{
   "database_major_version" : 19,
   "database_minor_version" : 0,
   "database_product_name" : "Oracle",
   "database_product_version" : "Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production\nVersion 19.3.0.0.0",
   "env" : {
      "defaultTimeZone" : "GMT",
      "ordsVersion" : "22.1.0.r1051723"
   }
}
curl -u hr:hr http://localhost:7001/tom/hr/_/sql

{
   "database_major_version" : 21,
   "database_minor_version" : 0,
   "database_product_name" : "Oracle",
   "database_product_version" : "Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\nVersion 21.3.0.0.0",
   "env" : {
      "defaultTimeZone" : "GMT",
      "ordsVersion" : "22.1.0.r1051723"
   }
}

As you can see from the above, the different web applications have different context paths and are configured to use two different databases.

Apache Tomcat

Two ORDS web applications deployed to Apache Tomcat
curl -u hr:hr http://localhost:8080/jerry/hr/_/sql

{
   "database_major_version" : 19,
   "database_minor_version" : 0,
   "database_product_name" : "Oracle",
   "database_product_version" : "Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production\nVersion 19.3.0.0.0",
   "env" : {
      "defaultTimeZone" : "GMT",
      "ordsVersion" : "22.1.0.r1051723"
   }
}
curl -u hr:hr http://localhost:8080/tom/hr/_/sql

{
   "database_major_version" : 21,
   "database_minor_version" : 0,
   "database_product_name" : "Oracle",
   "database_product_version" : "Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production\nVersion 21.3.0.0.0",
   "env" : {
      "defaultTimeZone" : "GMT",
      "ordsVersion" : "22.1.0.r1051723"
   }
}

Similar behaviour with Apache Tomcat with two different web applications with their own, separate, configuration.

Summary

What I have demonstrated is a script which produces a new web application for deployment in Apache Tomcat or WebLogic Server. The script is written for Python 2 because that’s the version distributed with Oracle WebLogic Server. It creates a new web application based on the distributed ords.war but without the META-INF content. That content includes a signature digest of the web.xml which will be invalid when the new deployment descriptor is written to the new WAR file.

This is what makes it possible to have multiple ORDS instances deployed to the same server all with different configuration directories.

HR Web Application – Tomcat & UCP

Many years ago a sample web application for using JDBC was published as part of the oracle-db-examples GitHub repository. The HR Web Application example was the starting point for some to build their first simple web interface to their database. Who knows how many simple, in-house applications have this as their inspiration?

The example had a particular focus on Apache Tomcat and the steps to getting it built and deployed where simple for the standard Apache Tomcat setup at the time.

-- Get the code
git clone https://github.com/oracle/oracle-db-examples.git
cd oracle-db-examples/java/HRWebApp
-- Copy the tomcat-users.xml and start tomcat
cp tomcat-users.xml $CATALINA_HOME/conf
catalina.sh start
-- Build the war file and deploy it
mvn package
cp target/JdbcWebSamples.war $CATALINA_HOME/webapps

By copying the JdbcWebSamples.war into the $CATALINA_HOME/webapps directory, the web application is automatically deployed by Apache Tomcat. The context path is based on the file name so the URL is http://localhost:8080/JdbcWebSamples/

Use your browser to access the web application.

The tomcat-users.xml defined two new users: hradmin and hrstaff. Both have welcome as their password. Login and click on the List All menu item to see the list of HR.EMPLOYEES records. Of course, that’s if the database connection details are correct.

Your JdbcBeanImpl.java will have to be changed to point to the correct database in the getConnection() method.

 public static Connection getConnection() throws SQLException {
    DriverManager.registerDriver(new oracle.jdbc.OracleDriver());
    Connection connection = DriverManager.getConnection(
         "jdbc:oracle:thin:@//mydatabaseserver:1521/orclpdb1", 
         "hr", 
         "hr");

    return connection;
  }

Make the above change for your database, run mvn package and copy the JdbcWebSamples.war to $CATALINA_HOME/webapps to get that running with the new connection details.

Take a look through JdbcBeanImpl methods in detail and you’ll notice that getConnection() method is called using the try-with-resource syntax so the connection is automatically closed after every operation. It’s a good practice because it does not leave INACTIVE sessions on the database. In fact, if you check your database v$session you shouldn’t find any records.

SELECT
    *
FROM
    v$session
WHERE
    program = 'JDBC Thin Client'
    AND schemaname = 'HR';

Creating a database connection for every request does have an overhead and if that is a remote database the network latency could be sufficient to cause problems for the users of the web application. To see the impact of creating a connection every time, let’s add some basic logging on the elapsed time to that getConnection() method.

 public static Connection getConnection() throws SQLException {
    final long start = System.currentTimeMillis();
    DriverManager.registerDriver(new oracle.jdbc.OracleDriver());
    Connection connection = DriverManager.getConnection(
         "jdbc:oracle:thin:@//mydatabaseserver:1521/orclpdb1", 
         "hr", 
         "hr");
    final long end = System.currentTimeMillis();
    logger.log(Level.INFO, 
         "Creating a connection duration(ms): " + (end - start));

    return connection;
  }

Like before, make the above change for your database, run mvn package and copy the JdbcWebSamples.war to $CATALINA_HOME/webapps. Every time the list of employees is retrieved, or any action that involves the connection, the cataline.out will show a log message.

Creating a connection to my remote database takes over 2 seconds!

One way to address this is by using Oracle’s Universal Connection Pool. This could be used through a data source defined in the Tomcat configuration but it can also be done used programmatically. Let’s do that.

The first thing is to upgrade to a more recent version of JDBC/UCP. Not the latest 21.1 version, more on that later, but we’ll use 19.9.0.0 for now. Note that the group id has changed from com.oracle.jdbc to com.oracle.database.jdbc and that’s what we’ll use in the pom.xml

Change this

    <dependency>
      <groupId>com.oracle.jdbc</groupId>
      <artifactId>ojdbc8</artifactId>
      <version>12.2.0.1</version>
    </dependency>

to this

    <dependency>
      <groupId>com.oracle.database.jdbc</groupId>
      <artifactId>ojdbc8</artifactId>
      <version>19.9.0.0</version>
    </dependency>
    <dependency>
      <groupId>com.oracle.database.jdbc</groupId>
      <artifactId>ucp</artifactId>
      <version>19.9.0.0</version>
    </dependency>

Now add a class to create and configure a Universal Connection Pool PoolDataSource. Let’s call it JdbcSource in the com.oracle.jdbc.samples.bean package.

package com.oracle.jdbc.samples.bean;

import java.sql.Connection;
import java.sql.SQLException;

import java.util.logging.Level;
import java.util.logging.Logger;

import oracle.ucp.jdbc.PoolDataSource;
import oracle.ucp.jdbc.PoolDataSourceFactory;

public class JdbcSource {
  JdbcSource() {

    //Create pool-enabled data source instance.
    this.pds = PoolDataSourceFactory.getPoolDataSource();

    //set the connection properties on the data source.
    try {
      pds.setConnectionPoolName(POOL_NAME);
      pds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
      pds.setURL("jdbc:oracle:thin:@//mydatabaseserver:1521/orclpdb1");
      pds.setUser("hr");
      pds.setPassword("hr");

      //Override any pool properties.
      pds.setInitialPoolSize(2);

    } catch (SQLException ex) {
      logger.log(Level.SEVERE, null, ex);
      ex.printStackTrace();
    }
  }

  public Connection connection() throws SQLException {
      return this.pds.getConnection();
  }

  private final PoolDataSource pds;
  private static final String POOL_NAME = "JdbcWebSamples_pool";

  // Singleton data source. Not a great pattern but simple for demonstrations.
  public static JdbcSource INSTANCE = new JdbcSource();
  static final Logger logger = Logger.getLogger("com.oracle.jdbc.samples.bean.JdbcSource");
}

The above will create a pool of database connections that can be reused every time connection() method is called. The pool is initialised with 2 connections. That means there will be two sessions on the database that will be INACTIVE most of the time. The web application may deal with concurrent requests so having an extra connection ready will help with that additional load.

Let’s revisit that JdbcBeanImpl getConnection() method and change it to use the new pooled connection. It’s not all that complicated…

 public static Connection getConnection() throws SQLException {
    final long start = System.currentTimeMillis();
    DriverManager.registerDriver(new oracle.jdbc.OracleDriver());
    Connection connection = JdbcSource.INSTANCE.connection();
    final long end = System.currentTimeMillis();
    logger.log(Level.INFO, 
         "Creating a connection duration(ms): " + (end - start));

    return connection;
  }

Like before, make the above for your code, run mvn package and copy the JdbcWebSamples.war to $CATALINA_HOME/webapps. Everytime the list of employees is retrieved, or any action that involves the connection, the cataline.out will show a log message for the first connection taking a long time, but every subsequent call takes milliseconds.

There’s an overhead for the 1st connection but subsequent requests get a connection in milliseconds

The query on v$sessions will show 2 INACTIVE sessions at least. As more concurrent requests are received, the pool size will grow automatically so more sessions could be created. In fact, hit that List All menu item repeatedly 20 or 30 times and you’ll see the number of v$sessions for HR schema grow. There’s more to explore here on setting UCP properties for optimising pool behaviour.

Upgrade to JDBC / UCP 21.1

Let’s leave that as a bonus point exercise for you and discuss upgrading to 21.1.0.0. This version has support for defining data sources in JBoss and Spring. That enhancement does break these example web applications which include the UCP jars in them.

Update the pom.xml to use the new 21.1.0.0 JDBC and UCP jars.

    <dependency>
      <groupId>com.oracle.database.jdbc</groupId>
      <artifactId>ojdbc8</artifactId>
      <version>21.1.0.0</version>
    </dependency>
    <dependency>
      <groupId>com.oracle.database.jdbc</groupId>
      <artifactId>ucp</artifactId>
      <version>21.1.0.0</version>
    </dependency>

Rebuild and deploy to see the following SEVERE error messages.

SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal Context [/JdbcWebSamples] startup failed due to previous errors

The localhost log will have more details.

SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.listenerStart Error configuring application listener of class [oracle.ucp.jdbc.UCPServletContextListener]
	java.lang.NoSuchMethodException: oracle.ucp.jdbc.UCPServletContextListener.<init>()
		at java.base/java.lang.Class.getConstructor0(Class.java:3427)
		at java.base/java.lang.Class.getConstructor(Class.java:2165)
		at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:151)
		at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4607)
		at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5146)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
		at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
		at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
		at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
		at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978)
		at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1849)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
		at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
		at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:773)
		at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427)
		at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1620)
		at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:305)
		at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
		at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1151)
		at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
		at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
		at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
		at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.base/java.lang.Thread.run(Thread.java:832)
SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.listenerStart Skipped installing application listeners due to previous error(s)

Put simply there is an annotated class that Tomcat discovers in the web application classpath and tries to initialise it, and it fails. It’s not a great work around, but a quick fix is to use the web.xml to tell Tomcat the names of components it should attempt to initialise. In this example web application there are two servlets defined: GetRole and WebController. These are in the com.oracle.jdbc.samples.web package. We’ll mention their names explicitly in the web.xml. Add this absolute-ordering entry just after the login-config.

  </login-config>
  <absolute-ordering>
    <name>GetRole</name>
    <name>WebController</name>
  </absolute-ordering>
</web-app>

It may not be a feasible work around for all, and it sort of defeats the purpose of having annotated servlets, but explicitly mentioning their names in absolute-ordering makes upgrading the JDBC/UCP jars possible.

Tomcat ~ override that web.xml

For many, their first experience with Java Servlets and Java Server Pages was on Apache Tomcat. For decades Tomcat has been a viable production platform for Java web applications. It’s Open-Source, free, configurable and stable.

Like many web applications, the Oracle REST Data Services application can be automatically deployed in a running Tomcat container just by copying the ords.war to the host’s appBase ( $CATALINA_BASE/webapps by default). There is a prerequisite step though. The ords.war must be updated with the location of the configuration directory. That location is written to the web.xml as a <context-param> called config.dir.

Since ORDS a standard Java web application, Tomcat can be configured to use a different web.xml than what is in the WAR file. There are not many valid cases where one might wish to do this though and there are often better alternatives. For example, if you want the ords.war to be unchanged but use a different database, just update the files in the configuration directory. That’s what they are there for.

Perhaps there are additional settings you want to define for your environment that are only applicable to the ORDS web application. Those entries probably don’t belong in the central server.xml or web.xml. So what can one do? Once an application is deployed, the web.xml can be edited in $CATALINA_BASE/webapps but those changes are transient and may not survive a server restart. One approach is to use a Context configuration file for the deployment. Here’s an example, using a Tomcat 9.0.29 server with the default configuration.

In my case, I want to have a different display name and point the config.dir to a different location. This allows me to have the exact same ords.war used in two different Tomcat servers. Let’s say that my ords.war config.dir has the correct location for where to find the configuration files on the production server, but I want to use the exact same war, unmodified on a developer server for testing.

Note that the approach outlined here is not specific to ORDS and applies to any web application in Tomcat.

So I define my custom web.xml for ORDS and call it ords_tomcat_test_web.xml because that sounds descriptive enough. I’m changing the display name to Oracle REST Data Services on Apache Tomcat and the config.dir to /scratch/ords_conf. Let’s say the previous values were Oracle REST Data Services and /u01/oracle/ords/config/ respectively.

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/j2ee" id="WebApp_ID" version="2.4" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">
	<display-name>Oracle REST Data Services on Apache Tomcat</display-name>

	<context-param>
		<param-name>config.dir</param-name>
		<!-- Enter the location where configuration settings should be stored -->
		<param-value>/scratch/ords_conf</param-value>
	</context-param>
	<context-param>
		<param-name>version</param-name>
		<param-value>20.3.0.301.1819</param-value>
	</context-param>

	<listener>
		<listener-class>oracle.dbtools.entrypoint.WebApplicationEntryPoint</listener-class>
	</listener>

	<servlet>
		<description>
		</description>
		<display-name>HttpEndPoint</display-name>
		<servlet-name>HttpEndPoint</servlet-name>
		<servlet-class>oracle.dbtools.entrypoint.WebApplicationRequestEntryPoint</servlet-class>
	</servlet>

	<servlet-mapping>
		<servlet-name>HttpEndPoint</servlet-name>
		<url-pattern>/*</url-pattern>
	</servlet-mapping>

	<servlet>
		<description>
		</description>
		<display-name>Forbidden</display-name>
		<servlet-name>Forbidden</servlet-name>
		<servlet-class>oracle.dbtools.entrypoint.Forbidden</servlet-class>
	</servlet>

	<servlet-mapping>
		<servlet-name>Forbidden</servlet-name>
		<url-pattern>/oracle/dbtools/jarcl</url-pattern>
	</servlet-mapping>

	<welcome-file-list>
		<welcome-file>index.html</welcome-file>
		<welcome-file>index.htm</welcome-file>
		<welcome-file>index.jsp</welcome-file>
		<welcome-file>default.html</welcome-file>
		<welcome-file>default.htm</welcome-file>
		<welcome-file>default.jsp</welcome-file>
	</welcome-file-list>

</web-app>

To tell Apache Tomcat to use this ords_tomcat_test_web.xml instead of the web.xml in the ords.war I define a Context configuration xml file in $CATALINA_BASE/conf/[enginename]/[hostname]/. In my case $CATALINA_BASE/conf/Catalina/localhost/ords.xml and the entry defines a altDDName attribute. The value is the absolute path to the alternative deployment descriptor for this context. This overrides the default deployment descriptor located at /WEB-INF/web.xml.

<?xml version="1.0" encoding="UTF-8"?>
<Context altDDName="/scratch/ords_tomcat_test_web.xml">
</Context>

The link between the Context configuration and the ords.war is that, ignoring suffix, they have the same filename. ords.xml is the Context configuration for ords.war. At startup, Tomcat will expect to find an ords web application in the webapps directory. If ords.war is not there, Tomcat will still attempt to deploy a web application with the path /ords. An error will show in the logs, and since there’s no real application to run, it will not be in a started state but will be deployed.

2020-12-16T12:45:05.831Z INFO        Deploying deployment descriptor [/scratch/apache-tomcat-9.0.29/conf/Catalina/localhost/ords.xml]
2020-12-16T12:45:05.838Z SEVERE      Error deploying deployment descriptor [/scratch/apache-tomcat-9.0.29/conf/Catalina/localhost/ords.xml]
java.lang.IllegalStateException: Error starting child
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:720)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)

... removed for brevity ...

2020-12-16T12:45:05.839Z INFO        Deployment of deployment descriptor [/scratch/apache-tomcat-9.0.29/conf/Catalina/localhost/ords.xml] has finished in [7] ms

Just drop in the ords.war as you would with any other web application for auto deployment and Tomcat picks up the change…

2020-12-16T12:46:25.877Z INFO        Reloading context [/ords]
2020-12-16T12:46:34.238Z INFO        Configuration properties for: |apex|pu|
database.api.enabled=true
db.connectionType=basic
db.hostname=localhost
db.port=1521
db.servicename=mydevdb

... removed for brevity ...

2020-12-16T12:46:46.402Z INFO        Oracle REST Data Services initialized
Oracle REST Data Services version : 20.3.0.r3011819
Oracle REST Data Services server info: Apache Tomcat/9.0.29

The configuration properties are being picked up from /scratch/ords_conf/ords/defaults.xml which point to mydevdb. Also the display name is coming from the alternative deployment descriptor.

Display name from customer deployment descriptor

Auto deployment continues to work and the Context configuration survives restarts. Keep in mind that undeploying the application will remove the Context configuration ords.xml as well as the application files from webapps.

Note that this is not the prescribed way to use ORDS in Tomcat and providing your own web.xml to override the package one may result in unspecified behaviour with future versions of ORDS or Tomcat.