Invalid SNI – What is it and how to fix it

Everyone was happy. The web application uptime and response rates have been well within the Service Level Agreements for months. Uptake of the published REST Services was on the rise. Both internally and externally, more and more systems were consuming the services and the three ORDS instances behind an NGINX load balancer took it all in their stride. No long running queries or inefficient PL/SQL code locking DB resources. Everything humming along like a finely tuned engine. Then it came time to upgrade…

The ORDS product team have been operating to the usual cadence of one major release every 3 months. Although not ancient, our system is a few releases behind and not only are there interesting new features that are worth exploring but there’s always the security fixes to recent critical 3rd party Common Vulnerabilities and Exposures ( CVE ) that justifies being at the most recent stable release.

Upgrade time comes around so the test environment is setup to run through the ORDS upgrade process before applying the same to production. Then this happens for all the requests: HTTP ERROR 400 Invalid SNI

But this was working before I upgraded !

What is my SNI and how did it get invalid?

The same web server could be hosting multiple sites. In the case of ORDS, the same ORDS instance could have multiple domain name entries that route to it. Server Name Indication ( SNI ) is an extension to the Transport Layer Security (TLS) networking protocol for a client ( such as your browser ) to indicate to the server ( ORDS running standalone mode for example ) which site it wishes to talk to. To put it simply, SNI is used at the start of the secure connection handshake between client and server. The client sends a Host header in the request to indicate what site it is requesting and the server goes looking for its certificate for that server name. See https://datatracker.ietf.org/doc/html/rfc4366#section-3.1 for more details on the role of Server Name Indication in keeping communication secure.

You’ll notice from the stack trace and the error page that the Invalid SNI response does not look like a standard ORDS error page. It is in fact generated by the Eclipse Jetty embedded in ORDS when running in standalone mode.

org.eclipse.jetty.http.BadMessageException: 400: Invalid SNI
A typical ORDS generated error page. It looks a lot different from the Invalid SNI error page!

It is a feature, not a bug – Jetty 10

When running ORDS in standalone mode an embedded Eclipse Jetty server is used. Before ORDS 22.3.0 that was Jetty 9 but since that release ORDS has been using Jetty 10 and for very good reason: more secure. As mentioned in the second paragraph at the start of this article there a fixes and optimisation that are worth making the upgrade for. Jetty 10 addressed some issues in the TLS handshake. Not least Better handling for wrong SNI #5379. Beforehand it didn’t really matter what the client sent in the Host header, Jetty would return any certificate it had.

What is in that certificate anyway?

The certificate, self-signed or otherwise, holds some important information about the server and is essential in establishing trust. To see the contents of the certificate, use the keytool utility in your Java distribution.

keytool -printcert -file self-signed.pem 
Owner: CN=localhost
Issuer: CN=localhost
Serial number: 5e90f747912dd350
Valid from: Thu Feb 08 17:55:19 GMT 2024 until: Fri Feb 07 17:55:19 GMT 2025
Certificate fingerprints:
	 SHA1: FB:F2:E7:30:B5:3F:D1:8B:AC:D0:8E:C3:49:15:3B:B2:75:F1:6E:AD
	 SHA256: 54:B1:4E:6E:92:DC:7F:88:E8:66:6B:69:91:C9:E1:01:CB:69:97:4A:B7:24:BA:F9:A0:52:AC:F3:C3:15:AB:39
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 3072-bit RSA key
Version: 3

The above output shows that the Owner of this certificate has a Common Name ( CN ) value localhost which is the default standalone.https.host configuration setting value.

What to do about it

Now that you know the root cause how do you go about resolving it? It is as simple as ensuring the Common Name ( CN ) or Subject Alternative Name ( SAN ) field in the certificate matches what the client is sending the in Host header of the request. For more information on having more than one hostname in a single certificate see a previous article: Beyond the Common Name: Secure multiple domains with a single self-signed certificate

ORDS will generate a self-signed certificate if necessary when it is started in standalone mode and configured for HTTPS. In other words, the standalone.https.port is specified in the global configuration settings or --secure option is used in the ords serve command. If no certificate exists at the configured standalone.https.cert location, ORDS will generate the self-signed certificate and key file.

The Common Name used for the self-signed certificate is taken from the standalone.https.host configuration setting. If not set, the value is localhost. Traditional certificate practices often rely solely on the Common Name (CN) field. That’s the approach taken by ORDS by default when generating the self-signed certificate.

This can sometimes catch people out when they start up ORDS in secure standalone mode the first time. The self-signed certificate is generated but they may not have specified the standalone.https.host configuration setting and when they do later, they still get HTTP 400 Invalid SNI responses. That’s because the self-signed certificate is already generated so no matter what the standalone.https.host configuration setting says, the certificate will not be generated again.

Simple steps

The simplest course of action is to rename the self-signed certificate and key files and restart ORDS. It will not find the files and therefore will attempt to generate new self-signed certificates.

In summary, make sure the standalone.https.host configuration setting matches the Host header value that clients will send for requests being routed to your ORDS standalone server.

Be careful with OAuth2 client roles

There’s a performance section in the ORDS Best Practice document that encourages the use of OAuth2 clients rather than basic authentication for REST Services. It is excellent advice. The overhead of verifying a database username and password can add hundreds of milliseconds to the total response time. That can be avoided by using an OAuth2 client with ORDS.

A client can be granted ORDS roles: standard roles and custom roles. One of the standard ORDS roles that can be granted is SQL Developer role. This is a very useful, and powerful, role which when it is granted, should be granted with caution. Having said that, any grant of any role should be carefully considered and the implications evaluated.

Here’s an example of creating a Client Credentials type OAuth2 client and granting it SQL Developer role:

BEGIN
    OAUTH.CREATE_CLIENT(
        P_NAME => 'sql_dev_client',
        P_GRANT_TYPE => 'client_credentials',
        P_OWNER => 'HR',
        P_DESCRIPTION => 'OAuth Client With SQL Developer role',
        P_ORIGINS_ALLOWED => '',
        P_REDIRECT_URI => NULL,
        P_SUPPORT_EMAIL => 'test@example.com',
        P_SUPPORT_URI => 'https://example.com',
        P_PRIVILEGE_NAMES => ''
    );
    OAUTH.GRANT_CLIENT_ROLE(
        P_CLIENT_NAME => 'sql_dev_client',
        P_ROLE_NAME => 'SQL Developer'
    );
    COMMIT;
END;

With that client created one can get the client_id and secret for obtaining an access token:

SELECT name, client_id, client_secret FROM user_ords_clients;

NAME             CLIENT_ID  CLIENT_SECRET 
---------------- ---------- --------------
sql_dev_client   3WjIAi..   myb-nW..

Using curl one can request an access token. This is what is referred to as a Two-Legged process where the client_id and client_secret is used to get an access token and that token is then used for subsequent service calls. Note that <schema alias> is the alias of the REST Enabled user that has created the client:

curl \
--user 3WjIAi.:myb-nWh.. \
--data 'grant_type=client_credentials' \
https://my-database.adb.eu-frankfurt-1.oraclecloudapps.com/ords/<schema alias>/oauth/token

{
 "access_token":"9EVGMlgDLQ8N5clLKVLj0Q",
 "token_type":"bearer",
 "expires_in":3600
}

That access token is time based. It will only be valid for an hour. After which the above oauth/token request would have to be submitted again. Now that we have an access token, let’s use it to invoke a service which ships with ORDS but requires the SQL Developer role. In this example we’ll get a list of Data Pump jobs and our schema alias is hr:

curl 'https://my-database.adb.eu-frankfurt-1.oraclecloudapps.com/ords/hr/_/db-api/stable/database/datapump/jobs/' \
  -H 'Authorization: bearer 9EVGMlgDLQ8N5clLKVLj0Q'

Which gives the below response:

{
   "count" : 1,
   "hasMore" : false,
   "items" : [
      {
         "attached_sessions" : 0,
         "datapump_sessions" : 0,
         "degree" : 0,
         "job_mode" : "TABLE                         ",
         "job_name" : "EXP_SD_123",
         "links" : [
            {
               "href" : "https://my-database.adb.eu-frankfurt-1.oraclecloudapps.com/ords/hr/_/db-api/stable/database/datapump/jobs/HR,EXP_SD_123/",
               "rel" : "self"
            }
         ],
         "operation" : "EXPORT                        ",
         "state" : "NOT RUNNING"
      }
   ],
   "limit" : 25,
   "links" : [
      {
         "href" : "https://my-database.adb.eu-frankfurt-1.oraclecloudapps.com/ords/hr/_/db-api/stable/database/datapump/jobs/",
         "rel" : "self"
      },
      {
         "href" : "https://my-database.adb.eu-frankfurt-1.oraclecloudapps.com/ords/hr/_/db-api/stable/database/datapump/jobs/",
         "rel" : "edit"
      },
      {
         "href" : "https://my-database.adb.eu-frankfurt-1.oraclecloudapps.com/ords/hr/_/db-api/stable/metadata-catalog/",
         "rel" : "describedby"
      },
      {
         "href" : "https://my-database.adb.eu-frankfurt-1.oraclecloudapps.com/ords/hr/_/db-api/stable/database/datapump/jobs/",
         "rel" : "first"
      }
   ],
   "offset" : 0
}

An interesting snippet of information. Of course one can do a lot more with these Database API services. Those Database API services that can be access by clients with SQL Developer role are not limited to just getting information.

Anything your database account can do

The Database API services perform a specific set of well defined operations. While access to the services require an ORDS role, such as SQL Developer or SQL Administrator, the database account used to execute the corresponding SQL is the REST Enabled schema.

There are other services provided by ORDS which require the SQL Developer role to access and one of note is the REST Enabled SQL Service. With the access token for a client with SQL Developer role one can submit any SQL script or statement.

For example, show the corresponding database account username:

curl 'https://my-database.adb.eu-frankfurt-1.oraclecloudapps.com/ords/hr/_/sql' \
  -H 'Authorization: bearer 9EVGMlgDLQ8N5clLKVLj0Q' \
  -H 'Content-Type: application/json' \
  --data-raw '{"statementText":"select user from dual","offset":0,"limit":256}'

Which confirms that the database user in this case is the HR schema…

{
   "env" : {
      "defaultTimeZone" : "UTC"
   },
   "items" : [
      {
         "response" : [],
         "result" : 0,
         "resultSet" : {
            "count" : 1,
            "hasMore" : false,
            "items" : [
               {
                  "user" : "HR"
               }
            ],
            "limit" : 256,
            "metadata" : [
               {
                  "columnClassName" : "java.lang.String",
                  "columnName" : "USER",
                  "columnTypeName" : "VARCHAR2",
                  "isNullable" : 1,
                  "jsonColumnName" : "user",
                  "precision" : 128,
                  "scale" : 0
               }
            ],
            "offset" : 0
         },
         "statementId" : 1,
         "statementPos" : {
            "endLine" : 2,
            "startLine" : 1
         },
         "statementText" : "select user from dual",
         "statementType" : "query"
      }
   ]
}

Conclusion

For those familiar with ORDS this is not a great revelation and there are plenty of legitimate cases where an ORDS OAuth2 client would have the SQL Developer role. In fact a quick search online for ORDS OAuth2 examples will show some. My intent is to highlight that the SQL Developer role provides access to a range of powerful ORDS services and the use of the role should be carefully considered.

Should we ban anonymity on the Internet?

In an Information Security article a few months back, Bruce Schneier (author of Schneier on Security) and Marcus Ranum put some points forward for and against internet anonymity. I have to admit that I agree with Schneier and find Ranum’s argument quite weak. He appears to suggest that the main reason to enforce identity is to avoid spam. The tools aren’t great, but there are already mechanisms in place to address this. Criminals are always getting better at finding ways to exploit weaknesses in internet technologies increasingly at the heart of the way we shop, interact, work, entertain and inform ourselves. We just have to keep up with the pace in the cat and mouse game. Sacrificing anonymity, and the right to privacy, is too great a cost for just avoiding emails about Viagra ™ and Nigerian generals with a stash of cash to move out of the country.

What is the great danger of not being anonymous? Well it’s all the inferring that goes on about facts that get gathered around the things you search for, shop for, chat about, view and listen to. These are then used to categorise you for advertising, inclusion or exclusion from groups or activities. NetFlix provided a great example of this last year. Just weeks after the contest began, two University of Texas researchers showed that with the NetFlix data one could identify users and in some cases their political leanings and sexual orientation.

Getting back to Schneier’s point, trying to implement a robust identification system, which criminals can not outwit or take advantage of, is not possible…

Mandating universal identity and attribution is the wrong goal. Accept that there will always be anonymous speech on the Internet. Accept that you’ll never truly know where a packet came from. Work on the problems you can solve: software that’s secure in the face of whatever packet it receives, identification systems that are secure enough in the face of the risks. We can do far better at these things than we’re doing, and they’ll do more to improve security than trying to fix insoluble problems.

Your hosted application is a virus vector

We are all used to seeing SaaS as an acronym for Software as a Service. It also doubles as another acronym for the non-functional features of a SaaS architecture. When you’re providing a hosted service there are a number of non-functional features that need to be considered such as Scalability, Availability, Agility and Security. I’ll flesh out these concepts at a later date as they are all quite broad and encompass a lot of features. Security, for example, includes, but is not limited to ensuring the integrity of the system and the systems that connect to it.

One threat to system integrity is a computer virus. If your hosted application becomes a vector for the dissemination of viruses to your clients system it becomes a threat to your business, not just your systems. So, how do you test you hosted application to make sure infected files are handled properly?

At the beginning of this year I started working on Oracle CRM Sales Library OnDemand, which permits the uploading of documents to be shared with others in your organisation. It provides a rich set of social networking features that enables users to share, rate, review, and tag PowerPoint presentations an MS Word documents so that everyone in your organization can leverage the most effective sales materials. To share a document, or it’s contents, it must be uploaded to Sales Library and of course, the upload process involves virus scanning. In fact, virus scanning is the first step. Irrespective of the file type, before any logical validation takes place, such as supported formats, the file is interrogated by an anti-virus system. When a file fails the virus scan, the user has to be informed with meaningful message. So, while your hosted application does not have to be coded with anti-virus logic, it does need to be able to handle and report effectively, when a virus is found.

Once you have this logic in place, you have to be able to test that the behaviour of your application is correct. Now, you don’t want your development or testing environment to be using files with viruses to test with. In fact, your organisations anti-malware system would probably prevent you from storing the files anyway. Thankfully, the European Institute for Computer Antivirus Research has a solution. For testing the anti-virus error handling logic in our application we attempt to upload a file with just the EICAR test string as content.

X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*

The above is not a virus, but is a test string for anti-malware systems. In unit tests, this can be used as the only content of an InputStream. You should provide a dummy implementation of your virus scan system when unit testing too. Remember, you’re not trying to test the anti-virus software. You want to test the behaviour of your application when the anti-virus software reports a virus is found. When testing the deployed Sales Library application, the text file can be saved with a .ppt or .doc extension and then uploaded. The only content of the file should be the test string.

I have yet to find an anti-virus system that does not comply with this EICAR anti-virus test string. A simple test in your own environment will reveal if your anti-virus software does or not. I suspect that it will, and you can proceed with confidence that your hosted application is handling contaminated files properly for the protection of yours and your customers business.

Social CRM Goes Live!

Announced at last year’s OpenWorld, and previewed at last month’s Enterprise 2.0 Conference, Oracles’ Social CRM goes live today at http://sales.oracle.com. This adds another social network, although quite a unique one, to the Oracle social grid.

So, how does Social CRM compare to the Oracle social networks and what makes it so special? Let’s look at what is in the Oracle Community already. There are blogs, forums and wikis that bring together people who are interested in Oracle products and technology. They provide the communication aspect of collobaration. And, more importantly, it is collobaration around what Oracle sells.

Social CRM introduces tools to support the doing aspect of colloboration. However, it is collobaration around the products and services you sell. The heavy weight, process driven, data entry intensive tools of which we are so familiar with, and still serves a valuable purpose in the enterprise, does not cater for every aspect of how business is done. The human element in selling, in particular, is not easily modelled. Social CRM is a small step in the right direction with tools, such as Sales Prospector, to increase sales and decrease data entry.

My own involvement in Social CRM started in January of this year, looking after the Security and User Provisioning side of the platform. Of course we hadn’t even reached code chill before we started work on the next version, so we are still very busy. Watch out for more (1, 2) later in the year. For me, the most rewarding part of the work is, coincidentally, the collaboration. You see, Social CRM is made possible through Agile software development practices. It is collaborative from the inside out.