Application Process Monitoring – Peter's Random Topic Library

April 2024 Update !
ORDS 24.1.0 no longer has this Java Agent / Classloader issue.
Use ORDS 24.1.0 or a later version.

When using Oracle APM Java Agent with ORDS, it crashes with a NullPointerException at startup. Not just when running in standalone mode, if your JVM is configured to use the Oracle APM Java Agent, you could encounter this issue with any command.

~/Downloads/ords-23.2.3.242.1937/bin/ords config list  
Oracle APM Agent: Starting APM Agent [premain]
Oracle APM Agent: Wrapper: version, hybrid
Oracle APM Agent: [DirectoryLocation] initialized on classloader [null]
Oracle APM - temp log directory is /var/folders/hq/6cg5drc54c3f371r9v65c8qm0000gn/T//
Oracle APM Agent: Parsing instrumentation directives
Oracle APM Agent: Loading directives from [built-in.directives]
Oracle APM Agent: Loading directives from [/~/work/ora_apm/oracle-apm-agent/config/1.8.3326/DirectivesConfig.acml]
Oracle APM Agent: Parsed a total of [116] directives
Oracle APM Agent: Initializing AgentInstance
Oracle APM Agent: Initialized AgentInstance
Oracle APM Agent: Started [premain] Agent
null
java.lang.NullPointerException
	at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
	at java.base/java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1541)
	at java.base/java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:667)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:591)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:575)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	at oracle.dbtools.launcher.executable.jar.ExecutableJarEntrypoint.invoke(ExecutableJarEntrypoint.java:42)
	at oracle.dbtools.launcher.executable.jar.ExecutableJarEntrypoint.main(ExecutableJarEntrypoint.java:64)

This was one of the reasons that a previous article on using Oracle APM with ORDS focused on using Apache Tomcat rather then ORDS standalone.

When running standalone, ORDS uses a built-in embedded jetty server for HTTP(s) service handling. The ords.war file contains a META-INF/MANIFEST.MF which has the following 2 lines used in standalone mode:

Main-Class: oracle.dbtools.launcher.executable.jar.ExecutableJarEntrypoint 
Executable-Jar-Main-Class: oracle.dbtools.cmdline.CommandLine

The first line is used by by the JVM to launch the main method in class ExecutableJarEntrypoint. Inside this main method ORDS is trying to read the second line. This is done via a mechanism like what is shown below:

ClassLoader cl = Thread.currentThread().getContextClassLoader();
URL url = cl.getResource("META-INF/MANIFEST.MF");

This works fine when running standalone since there is only 1 single war/jar file and therefore only a single manifest. However, when running with the APM Agent there are now 2 war/jar files and also 2 manifests in the classpath. In this case, the getResource call returns the MANIFEST.MF from the agent jar file instead of the MANIFEST.MF within ords.war file. This manifest from the APM agent does not have any Executable-Jar-Main-Class which makes ORDS throw the NullPointerException!

Workaround

Until this is addressed in ORDS java code, a simple workaround to allow the Oracle APM java agent monitor the standalone ORDS env is to simply edit the MANIFEST.MF file inside the agent to add a line like this:

Executable-Jar-Main-Class: oracle.dbtools.cmdline.CommandLine

This is an approach provided to me by the Oracle APM team.

Since a runtime environment may not have the jar command to unpack the agent jar, they were kind enough to write a python script to edit the agent jar for this and repackage everything properly afterwards.

import zipfile
import io
import shutil
# Specify the path to the APM Agent jar file
jar_filename = '/opt/oracle/product/oracle-apm-agent/bootstrap/ApmAgent.jar'
file_to_edit = 'META-INF/MANIFEST.MF'
new_line = 'Executable-Jar-Main-Class: oracle.dbtools.cmdline.CommandLine\n'
# Create a temporary file to hold the modified JAR content
temp_jar_filename = 'temp_modified.jar'
# Open the original JAR file in read mode
with zipfile.ZipFile(jar_filename, 'r') as original_jar:
    # Create a new JAR file in write mode
    with zipfile.ZipFile(temp_jar_filename, 'w') as temp_jar:
        for item in original_jar.infolist():
            # Copy all files from the original JAR to the new JAR
            if item.filename != file_to_edit:
                temp_jar.writestr(item, original_jar.read(item))
        # Read the contents of the file to be edited
        with original_jar.open(file_to_edit) as file:
            content = file.read().decode('utf-8')
        # Append the new line to the content
        modified_content = content + new_line
        # Write the modified content to the new JAR
        temp_jar.writestr(file_to_edit, modified_content.encode('utf-8'))
# Replace the original JAR with the modified JAR
shutil.move(temp_jar_filename, jar_filename)
print('New line appended and old file removed successfully.')

Change the jar_filename variable to the location of your ApmAgent.jar and save the above as ords_apm.py. Then run it..

>python3 ords_apm.py

New line appended and old file removed successfully.

Now you can use the Oracle APM Java Agent with your ORDS command line.

> ~/Downloads/ords-23.2.3.242.1937/bin/ords config list  
Oracle APM Agent: Starting APM Agent [premain]
Oracle APM Agent: Wrapper: version, hybrid
Oracle APM Agent: [DirectoryLocation] initialized on classloader [null]
Oracle APM - temp log directory is /var/folders/hq/6cg5drc54c3f371r9v65c8qm0000gn/T//
Oracle APM Agent: Parsing instrumentation directives
Oracle APM Agent: Loading directives from [built-in.directives]
Oracle APM Agent: Loading directives from [~/work/ora_apm/oracle-apm-agent/config/1.8.3326/DirectivesConfig.acml]
Oracle APM Agent: Parsed a total of [116] directives
Oracle APM Agent: Initializing AgentInstance
Oracle APM Agent: Initialized AgentInstance
Oracle APM Agent: Started [premain] Agent

ORDS: Release 23.2 Production on Thu Oct 05 19:47:01 2023

Copyright (c) 2010, 2023, Oracle.

Configuration:
  /path/to/config/

Database pool: default

Setting                  Value                                          Source     
----------------------   --------------------------------------------   -----------
database.api.enabled     true                                           Global     
db.hostname              localhost                                      Pool       
db.password              ******                                         Pool Wallet
db.port                  2193                                           Pool       
db.servicename           DB193P                                         Pool       
db.username              ORDS_PUBLIC_USER                               Pool       
feature.sdw              false                                          Pool       
jdbc.MaxLimit            100                                            Pool       
plsql.gateway.mode       disabled                                       Pool       
restEnabledSql.active    true                                           Pool

Conclusion

It really is as simple as that. Note that a similar workaround might work with other java agents which encounter a NullPointerException.

Tag: Application Process Monitoring

Oracle APM agent generates NullPointerException

Workaround

Conclusion