Oracle APM agent generates NullPointerException

April 2024 Update !
ORDS 24.1.0 no longer has this Java Agent / Classloader issue.
Use ORDS 24.1.0 or a later version.

When using Oracle APM Java Agent with ORDS, it crashes with a NullPointerException at startup. Not just when running in standalone mode, if your JVM is configured to use the Oracle APM Java Agent, you could encounter this issue with any command.

~/Downloads/ords-23.2.3.242.1937/bin/ords config list  
Oracle APM Agent: Starting APM Agent [premain]
Oracle APM Agent: Wrapper: version, hybrid
Oracle APM Agent: [DirectoryLocation] initialized on classloader [null]
Oracle APM - temp log directory is /var/folders/hq/6cg5drc54c3f371r9v65c8qm0000gn/T//
Oracle APM Agent: Parsing instrumentation directives
Oracle APM Agent: Loading directives from [built-in.directives]
Oracle APM Agent: Loading directives from [/~/work/ora_apm/oracle-apm-agent/config/1.8.3326/DirectivesConfig.acml]
Oracle APM Agent: Parsed a total of [116] directives
Oracle APM Agent: Initializing AgentInstance
Oracle APM Agent: Initialized AgentInstance
Oracle APM Agent: Started [premain] Agent
null
java.lang.NullPointerException
	at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
	at java.base/java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1541)
	at java.base/java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:667)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:591)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:575)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	at oracle.dbtools.launcher.executable.jar.ExecutableJarEntrypoint.invoke(ExecutableJarEntrypoint.java:42)
	at oracle.dbtools.launcher.executable.jar.ExecutableJarEntrypoint.main(ExecutableJarEntrypoint.java:64)

This was one of the reasons that a previous article on using Oracle APM with ORDS focused on using Apache Tomcat rather then ORDS standalone.

When running standalone, ORDS uses a built-in embedded jetty server for HTTP(s) service handling. The ords.war file contains a META-INF/MANIFEST.MF which has the following 2 lines used in standalone mode:

Main-Class: oracle.dbtools.launcher.executable.jar.ExecutableJarEntrypoint 
Executable-Jar-Main-Class: oracle.dbtools.cmdline.CommandLine

The first line is used by by the JVM to launch the main method in class ExecutableJarEntrypoint. Inside this main method ORDS is trying to read the second line. This is done via a mechanism like what is shown below:

ClassLoader cl = Thread.currentThread().getContextClassLoader();
URL url = cl.getResource("META-INF/MANIFEST.MF");

This works fine when running standalone since there is only 1 single war/jar file and therefore only a single manifest. However, when running with the APM Agent there are now 2 war/jar files and also 2 manifests in the classpath. In this case, the getResource call returns the MANIFEST.MF from the agent jar file instead of the MANIFEST.MF within ords.war file. This manifest from the APM agent does not have any Executable-Jar-Main-Class which makes ORDS throw the NullPointerException!

Workaround

Until this is addressed in ORDS java code, a simple workaround to allow the Oracle APM java agent monitor the standalone ORDS env is to simply edit the MANIFEST.MF file inside the agent to add a line like this:

Executable-Jar-Main-Class: oracle.dbtools.cmdline.CommandLine

This is an approach provided to me by the Oracle APM team.

Since a runtime environment may not have the jar command to unpack the agent jar, they were kind enough to write a python script to edit the agent jar for this and repackage everything properly afterwards. 

import zipfile
import io
import shutil
# Specify the path to the APM Agent jar file
jar_filename = '/opt/oracle/product/oracle-apm-agent/bootstrap/ApmAgent.jar'
file_to_edit = 'META-INF/MANIFEST.MF'
new_line = 'Executable-Jar-Main-Class: oracle.dbtools.cmdline.CommandLine\n'
# Create a temporary file to hold the modified JAR content
temp_jar_filename = 'temp_modified.jar'
# Open the original JAR file in read mode
with zipfile.ZipFile(jar_filename, 'r') as original_jar:
    # Create a new JAR file in write mode
    with zipfile.ZipFile(temp_jar_filename, 'w') as temp_jar:
        for item in original_jar.infolist():
            # Copy all files from the original JAR to the new JAR
            if item.filename != file_to_edit:
                temp_jar.writestr(item, original_jar.read(item))
        # Read the contents of the file to be edited
        with original_jar.open(file_to_edit) as file:
            content = file.read().decode('utf-8')
        # Append the new line to the content
        modified_content = content + new_line
        # Write the modified content to the new JAR
        temp_jar.writestr(file_to_edit, modified_content.encode('utf-8'))
# Replace the original JAR with the modified JAR
shutil.move(temp_jar_filename, jar_filename)
print('New line appended and old file removed successfully.')

Change the jar_filename variable to the location of your ApmAgent.jar and save the above as ords_apm.py. Then run it..

>python3 ords_apm.py

New line appended and old file removed successfully.

Now you can use the Oracle APM Java Agent with your ORDS command line.

> ~/Downloads/ords-23.2.3.242.1937/bin/ords config list  
Oracle APM Agent: Starting APM Agent [premain]
Oracle APM Agent: Wrapper: version, hybrid
Oracle APM Agent: [DirectoryLocation] initialized on classloader [null]
Oracle APM - temp log directory is /var/folders/hq/6cg5drc54c3f371r9v65c8qm0000gn/T//
Oracle APM Agent: Parsing instrumentation directives
Oracle APM Agent: Loading directives from [built-in.directives]
Oracle APM Agent: Loading directives from [~/work/ora_apm/oracle-apm-agent/config/1.8.3326/DirectivesConfig.acml]
Oracle APM Agent: Parsed a total of [116] directives
Oracle APM Agent: Initializing AgentInstance
Oracle APM Agent: Initialized AgentInstance
Oracle APM Agent: Started [premain] Agent

ORDS: Release 23.2 Production on Thu Oct 05 19:47:01 2023

Copyright (c) 2010, 2023, Oracle.

Configuration:
  /path/to/config/

Database pool: default

Setting                  Value                                          Source     
----------------------   --------------------------------------------   -----------
database.api.enabled     true                                           Global     
db.hostname              localhost                                      Pool       
db.password              ******                                         Pool Wallet
db.port                  2193                                           Pool       
db.servicename           DB193P                                         Pool       
db.username              ORDS_PUBLIC_USER                               Pool       
feature.sdw              false                                          Pool       
jdbc.MaxLimit            100                                            Pool       
plsql.gateway.mode       disabled                                       Pool       
restEnabledSql.active    true                                           Pool

Conclusion

It really is as simple as that. Note that a similar workaround might work with other java agents which encounter a NullPointerException.

8080 is already in use

When trying to start Oracle REST Data Services (ORDS), you may encounter the frustrating error message: “could not start standalone mode because the listen port: 8080 is already in use by another process. check if another instance of ords is already running“. This error indicates that another application or process is already using port 8080, preventing ORDS from starting successfully. In this article, we will guide you through the steps to identify the application using port 8080 on Windows, macOS, and Linux, allowing you to resolve this issue and get ORDS up and running.

Note that this can happen if you are trying to run multiple ORDS instances using HTTPS traffic. See Multiple ORDS Instances on the Same Machine for more information on that.

Find the Application Using Port 8080:

Windows:

  1. Open the Command Prompt: Press Win + R, type “cmd,” and hit Enter.
  2. In the Command Prompt, enter the following command: netstat -ano | findstr :8080.
  3. Identify the line with the local address 0.0.0.0:8080 or 127.0.0.1:8080 in the output.
  4. Take note of the PID (Process Identifier) associated with that line.
  5. Open the Task Manager: Press Ctrl + Shift + Esc.
  6. Go to the “Details” tab in the Task Manager.
  7. Locate the process with the corresponding PID from step 4 to identify the application using port 8080.

macOS:

  1. Open Terminal: Navigate to the Applications/Utilities folder or use Spotlight Search to find it.
  2. In the Terminal, enter the following command: lsof -i :8080.
  3. Look for the line with the local address *:8080 in the output.
  4. Note the PID associated with that line.
  5. Enter the command: ps -p [PID] -o comm= (replace [PID] with the PID noted in step 4).
  6. The output of the above command will display the name of the application using port 8080.
On my mac the output of the lsof command looks like this

Linux:

  1. Open a Terminal: Use a terminal application such as GNOME Terminal, Konsole, or xterm.
  2. In the terminal, enter the following command: sudo lsof -i :8080.
  3. Look for the line with the local address *:8080 in the output.
  4. Note the PID associated with that line.
  5. Enter the command: ps -p [PID] -o comm= (replace [PID] with the PID noted in step 4).
  6. The output of the above command will display the name of the application using port 8080.
On a linux machine this is the output of “lsof -i :9193” which shows docker containers using that port

Resolving the Issue:

Once you have identified the application or process using port 8080, you have a few options to resolve the issue:

  1. Terminate the Conflicting Application: If the application using port 8080 is not essential or can be temporarily shut down, you can terminate it from the Task Manager (Windows) or using the appropriate command for macOS or Linux.
  2. Change the ORDS Port: If terminating the conflicting application is not feasible, you can have ORDS use a different port that is not in use such as port 8081. This can be achieved in one of two ways:
    • On the command line when starting standalone: ords serve --port 8081
    • In the configuration: ords config set standalone.http.port 8081
  3. Configure the Conflicting Application: If the conflicting application is critical and cannot be terminated, you may need to adjust its configuration to use a different port, freeing up port 8080 for ORDS. Consult the documentation or support resources for the specific application to learn how to modify its port configuration.
Just one more thing…

Of course you can change port 8080 for any other port that you happen to want to use. Using the standard ports 80 and 443 can be problematic in that there might be a service already using that port and the above approaches to not always list these background services. For example, if you are using a container management system such as Rancher and have Traefik enabled, it will not be shown as listing on the port, but it will have your traffic redirected to it, and not your ORDS instance.

Conclusion

Hitting the “8080 is already in use by another process” error when starting Oracle REST Data Services (ORDS) can be frustrating, but by following the steps outlined in this article, you can identify the application using that port and take appropriate action to resolve the issue. Whether it involves terminating the conflicting application, changing the ORDS port, or configuring the conflicting application differently, you’ll soon have ORDS running smoothly.

ORDS, Database Vault and ORA-20031

Picture this…you’ve been working happily with ORDS for ages and realise that it’s time to upgrade but in your upgrade test environment you start seeing issues with ORDS not being able to create connections at runtime and nothing works!

No matter what you try to do, it’s ORA-20031: Management of Schema enablement has been restricted to ORDS_ADMINISTRATOR_ROLE privilege error messages everywhere!

It is not immediately obvious but the root cause is likely to be the Oracle Database Vault feature that has been doing such a good job in protecting data in the database. Database Vault is a security feature designed to provide additional layers of protection for sensitive data, and can enhance the security of data-driven applications. The sort of data-driven applications that one would use with ORDS. Users can encounter this hurdle while working with this combination of ORDS and Database Vault when upgrading ORDS. This happens especially when ORDS was initially installed first and then Database Vault enabled afterwards.

SQL> BEGIN

 ORDS.ENABLE_SCHEMA(p_enabled => TRUE,
    p_schema => ‘AVSYS’,
    p_url_mapping_type => ‘BASE_PATH’,
    p_url_mapping_pattern => ‘auditvault’,
    p_auto_rest_auth => FALSE);

 commit;
end;
/ 2  3  4  5  6  7  8  9  10  11
’BEGIN
*
ERROR at line 1:
ORA-20031: Management of Schema enablement has been restricted to
ORDS_ADMINISTRATOR_ROLE privilege.
ORA-06512: at “ORDS_METADATA.ORDS”, line 183
ORA-06512: at “ORDS_METADATA.ORDS_INTERNAL”, line 1048
ORA-01031: insufficient privileges
ORA-06512: at “ORDS_METADATA.ORDS_INTERNAL”, line 456
ORA-06512: at “ORDS_METADATA.ORDS_INTERNAL”, line 468
ORA-06512: at “ORDS_METADATA.ORDS_INTERNAL”, line 468
ORA-06512: at “ORDS_METADATA.ORDS_INTERNAL”, line 1034
ORA-06512: at “ORDS_METADATA.ORDS_INTERNAL”, line 922
ORA-06512: at “ORDS_METADATA.ORDS_INTERNAL”, line 1063
ORA-06512: at “ORDS_METADATA.ORDS”, line 167
ORA-06512: at line 3

When you see this error it’s because the database user is protected by Database Vault. The ORDS documentation does have a section about Authorizing Oracle REST Data Services to Access Protected Users but it does not specifically call out the error message you get when you don’t perform this additional authorisation step using DBMS_MACADM.AUTHORIZE_PROXY_USER

DBMS_MACADM.AUTHORIZE_PROXY_USER(
  proxy_user   IN VARCHAR2,
  user_name    IN VARCHAR2);

Conclusion

The upshot is that for every database user that you REST Enable, when using Oracle Database Vault, there’s an extra step:

BEGIN
  ORDS_ADMIN.ENABLE_SCHEMA(
    p_schema => 'tickets'
  );
  DBMS_MACADM.AUTHORIZE_PROXY_USER('ORDS_PUBLIC_USER','TICKETS');
END;
/

Note: In the majority of cases ORDS_PUBLIC_USER is the configured ORDS Runtime User specified by the db.username configuration setting.

Beyond the Common Name: Secure multiple domains with a single self-signed certificate

Oracle REST Data Services (ORDS) will generate a self-signed automatically when starting for the first time in standalone mode. That is, if it has not already been configured with an existing cert and key. The Common Name used for the self-signed certificate is taken from the standalone.https.host configuration setting. If not set, the value is localhost. Traditional certificate practices often rely solely on the Common Name (CN) field. That’s the approach taken by ORDS by default when generating the self-signed certificate. The upshot is that you are limited to one certificate for one domain. In this article, we explore the untapped potential of self-signed certificates with Subject Alternative Name (SAN) extension, enabling you to secure multiple domains efficiently

Understanding Self-Signed Certificates

Self-signed certificates are created and signed by the same entity without involving a trusted third-party Certificate Authority (CA). They provide encryption and authentication capabilities, although they are not initially recognised by most browsers and operating systems. Self-signed certificates can be manually trusted or imported, making them a viable option for specific use cases.

Subject Alternative Name (SAN)

SAN is an extension within the X.509 certificate standard that allows multiple domain names to be associated with a single certificate. Unlike the limited scope of the Common Name field, SAN enables you to secure multiple domains or subdomains with one certificate, simplifying management and reducing costs.

Benefits of Self-Signed Certificates with SAN

  • Enhanced Flexibility: Self-signed certificates with SAN offer the flexibility to secure numerous domains or subdomains, accommodating complex web architectures and diverse configurations.
  • Cost-Effectiveness: By leveraging self-signed certificates, you can save costs associated with purchasing individual certificates for each domain.
  • Internal Environments: Self-signed certificates with SAN are especially valuable for securing internal networks, intranets, or testing environments that don’t require public trust.
  • Rapid Deployment: Generating and deploying self-signed certificates with SAN is a swift and straightforward process, making it ideal for time-sensitive projects.

Considerations and Best Practices:

  • Trust Management: Users must manually trust self-signed certificates, so clear instructions or documentation on certificate importation are essential.
  • Certificate Lifecycle: Although self-signed certificates have no defined expiration period, regular rotation is recommended to uphold security best practices.
  • Public-Facing Websites: For public-facing websites, obtaining trusted certificates from recognised CAs is crucial to establish trust with visitors and avoid browser warnings.
  • Hybrid Approach: Consider combining self-signed certificates with SAN for internal domains and trusted CA certificates for public-facing domains to strike a balance between cost-efficiency and public trust.
  • Load Balancers: For public-facing websites, use a load balancer in front of your ORDS instance(s). That way you can aim for high availability but also configure single Common Name certs for public-facing services.

Generate your self-signed SAN cert…

Try saying that fast three times 😀

There’s no command in ORDS to generate self-signed certificate with a Subject Alternative Name ( SAN ) field. To achieve this we’ll use the openssl utility which is most likely already available on your machine. The example openssl command below can be executed where the ORDS configuration directory is the working directory. The output is two files with the same name as what ORDS uses as the default standalone.https.cert.key and standalone.https.cert configuration settings respectively. 

openssl req -x509 -nodes -days 365 \
       -newkey rsa:2048 \
       -keyout global/standalone/self-signed.key \
       -out global/standalone/self-signed.pem \
       -subj "/CN=localhost" \
       -addext "subjectAltName = DNS:vanity.example.com, DNS:myhost.example

In the above example the self-signed certificate is generated with localhost in the Common Name ( CN ) field. This is the same as the certificate that would normally be generated by ORDS. However, that certificate also has a subjectAltName field listing two DNS hostnames: vanity.example.com and myhost.example. Start up ORDS and your traffic to the server will be secured for both domains. Note that you will have to look after your DNS network routing so that the domain names you choose route to the machine you are running ORDS on.

That certificate and key could also be used with your load balancer to accept HTTPS traffic too. As mentioned earlier, this is really only suitable for internal environments where you can have your users manually trust the certificate.

Conclusion

Ditching the limitations of the Common Name field, self-signed certificates with Subject Alternative Name (SAN) extension enable you to secure multiple domains with ease. While self-signed certificates may not be suitable for public-facing websites, these certificates provide numerous benefits for internal environments or specific use cases. By using self-signed certificates with SAN in your ORDS instance, you can fortify your infrastructure security while efficiently managing multiple domains, all without breaking the bank.