Thomas Hampel

 14 December 2017
A colleague approached me with a problem where a Domino server did not start up the HTTP task after upgrading to 9.0.1 FP9

After upgrading a Domino server on Windows from 9.0.1 FP8 to FP9 the http task no longer starts up.
HTTP Server: Cannot load resource file. Exiting

For a different server the same FP9 installation worked fine.
Although the problem is not new, it seems nobody investigated the root cause yet...

Oliver Busse blogged about this problem in combination with FP3 install on Windows and IBM Technote 1298729 indicates that this problem is caused by a missing file: "Similar behavior can be seen in the Windows environment if the nstrings.dll file located in the Domino folder is damaged or missing."

In our case this file did exist and was not the root cause of the problem.

We wanted to avoid reinstalling the server from scratch as done by Oliver.
So we compared the program directory of the server with another server where FP9 has been installed succesfully and found the file "nrshttp.dll" is missing.

The solution is rather simple: just copy the file from a working server...

then start up the HTTP task and get yourself a coffee.

References

• Oliver Busse
• IBM Technote 1298729 - Domino console messages appear as numbers instead of text in UNIX or Windows

Thomas Hampel

 9 November 2015
There is a seucrity issue with Domino which allows anybody to gain access without authentication.
Jesper Kiaer wrote about this problem before in his blog post ( Part1 and Part2 ) and also created a video showing the problem.

If the Notes.ini variable HTTPEnableConnectorHeaders is set to 1, an attacker just needs to pass the user name he wants to be within a request header to get unauthorized access to Domino servers.
This notes.ini variable is referenced in the product documentation as well as in this technote for configuring Domino servers behind an IIS reverse proxy.

So there is a good chance that some people have enable this variable in production.
None of the Domino servers I have checked was affected, however I was able to reproduce the findings and can confirm it is working as described even with Domino 9.0.1 with latest fixes installed.

Steps to reproduce

• Add the Notes.ini variable "HTTPEnableConnectorHeaders=1" to the Notes.ini of the Domino server
  Remark: This will make the server insecure.
• Restart the HTTP task
• Use Firefox and install this plugin => https://addons.mozilla.org/en-US/firefox/addon/modify-headers/
• Restart Firefox for the plugin to be initialized
• In Firefox, open the configuration of the new plugin
  
• Add a new header called $WSRU with the desired username / shortname as available in the target environment
  
  Save + Enable the configuration
• Start the Plugin
  
• Navigate to an existing Domino server resource, e.g. https://your-domino-server.your-domain.com/mail/username.nsf

Surprise, surprise... you now have access rights of the user name you have specified in the request header, in my case thats PaulSmith.
Just imagine what can be done when using the name of an administrator...

How to fix it?
Well, as simple as removing the Notes.ini variable in question, using the following two commands at the Domino server console:

Of course you would use a configuration document in production to keep your Notes.ini under control.

References:

• Nevermind.dk - http://nevermind.dk/
• Sean Cull - Apache Proxy for Domino and HTTPEnableConnectorHeaders
• Darren Duke  - If you get page errors after disabling HTTPEnableConnectorHeaders in Domino, try this
• Jesse Gallagher - Domino's Server-Side User Security

Thomas Hampel

 22 September 2014
Following up on David's post, here is a WIMP's Guide to get a GUI version of an iKeyman which is supproting the CMS format that is used by the IBM HTTP Server.
iKeyman is actually part of your Notes Client, it is available in "/jvm/bin/ikeyman.exe" but this version does not provide support for CMS keystores unfortunately.

So what does it take to add support for the CMS format
There is a (much) longer method to get the same done by downloading a specific version of ikeyman which includes CMS support... but this I'll explain at the end of this post.

Here is the short version:

What you need:

• Notes Client (which you should already have installed)
• Text editor of your choice

Actions:

• Edit the file /jvm/lib/security/java.security
• Find the list of security providers, e.g. by searching for "security.provider", which should look like this:
  security.provider.1=com.ibm.jsse2.IBMJSSEProvider2
  security.provider.2=com.ibm.crypto.provider.IBMJCE
  security.provider.3=com.ibm.security.jgss.IBMJGSSProvider
  security.provider.4=com.ibm.security.cert.IBMCertPath
  security.provider.5=com.ibm.security.sasl.IBMSASL
  security.provider.6=com.ibm.xml.crypto.IBMXMLCryptoProvider
  security.provider.7=com.ibm.xml.enc.IBMXMLEncProvider
  security.provider.8=org.apache.harmony.security.provider.PolicyProvider
  security.provider.9=com.ibm.security.jgss.mech.spnego.IBMSPNEGO
• Append one new line at the end of this list, where [X] is the next integer value available
  security.provider.[X]=com.ibm.security.cmskeystore.CMSProvider
• Save

So it should look like this:
{...}
#
# List of providers and their preference orders (see above):
#
security.provider.1=com.ibm.jsse2.IBMJSSEProvider2
security.provider.2=com.ibm.crypto.provider.IBMJCE
security.provider.3=com.ibm.security.jgss.IBMJGSSProvider
security.provider.4=com.ibm.security.cert.IBMCertPath
security.provider.5=com.ibm.security.sasl.IBMSASL
security.provider.6=com.ibm.xml.crypto.IBMXMLCryptoProvider
security.provider.7=com.ibm.xml.enc.IBMXMLEncProvider
security.provider.8=org.apache.harmony.security.provider.PolicyProvider
security.provider.9=com.ibm.security.jgss.mech.spnego.IBMSPNEGO
security.provider.10=com.ibm.security.cmskeystore.CMSProvider
{...}

Result:
Voila: CMS support is ready



This blog post could end here but I'd like to share what someone would have to do without using the method above:

The (very) long route:

1. Try downloading IBM HTTP Server from www.ibm.com/software/webservers/httpservers/download or here
2. Recognize this website only offers version:  7.0.0.0 dated from 15 May 2009
3. Try anyway and download and install v7.0.0
4. Notice iKeyman in this version does not Subject Alternate Support
   
5. Read Technote 1444027
   Notice it clearly describes "later versions of IBM HTTP Server (IHS), after v7.0, do not require these special steps to enable SAN functionality."
6. Conclude the Technote 1444027 is wrong or needs update
7. Try downloading t he IBM HTTP Server trial 8.0 or 8.5
8. Notice the web site does not offer a download link and only shows a blank page (why?)
9. Get IBM HTTP Server in the latest version, (which is part of Websphere Application Server 8.5.5 Supplements disks, so its just 3 Gbyte to download)
10. Install the IBM Installation Manager v1.8  and add the WAS Supplements folder location as a new repository
11. Install IBM HTTP Server
12. Launch iKeyman with CMS support and Subject Alternative Support
    
13. Notice that iKeyman is actually part of the Notes Client anyway and the same could have been done without all those actions before: priceless

For reference:
Creating a certificate request incl. Subject Alternate Names can be done by using the GSKTool command line version
/opt/IBM/HTTPServer/bin/gskcapicmd -certreq -create -db /opt/IBM/HTTPServer/ssl/keystore.kdb -pw passw0rd -label foobar -dn "cn=www.foobar.ibm.com" -size 2048 -file /tmp/foobar.csr -san_dnsname "www.foobar.ibm.com" -san_emailaddr "admin@us.ibm.com" -san_ipaddr "192.168.1.221"

Further reading:

• Wikipedia Article subjectAltName
• GSKCapiCmd User's Guide (for GSKit version 8)
• Blog post IBM HTTP Server / IBM Global Security Toolkit - Commanding the line by David Hay

Thomas Hampel

 20 September 2014
All of a sudden (as usual) clients started to receive SSL expiration warnings when accessing the customer's Connections environment.

A quick look into the log file  /opt/IBM/HTTPServer/logs/error.log confirmed the SSL certificate has expired.:
[Sat Sep 20 22:21:05 2014] [error] [client 10.175.198.62] [8222a80] [30015] SSL0221E: SSL Handshake Failed, Either the certificate has expired or the system clock is incorrect. [10.175.198.62:40028 -> 10.175.220.11:443] [22:31:05.000019743]

Opening the *.kdb file with the gsktool showed the default certificate had expired.

Ok, nothing easier than that... so lets create a new signing request and get this signed by the certificate authority.
Once that is completed we can import the new certificate incl. any trusted roots quickly.

However when you try to import/receive your signed certificate keep the following in mind:

• You can only import a signed certificate into >exactly< the same *.kdb file which was used to create the certificate request.

Within the iKeyman utility, switch from "Personal certificates" to "Personal Certificate Requests" (sorry, only got screenshots in German available and hope the translation is correct)


normally it would look like this...


but if it looks like the following screenshot, then bad luck... you can not import your signed certificate anymore.


Instead you'll see "The certificate request created for the certificate is not in the key database"  / "Die für das Zertifikat erstellte Zertifikatsanforderung ist nicht in der Schlüsseldatenbank vorhanden."


Now your options to solve this are:
a) find the original key ring file (*.kdb) which was used to create the certificate request
b) create a new certificate request, but this time make sure to keep the *.kdb file
c) set up a self signed certificate - although this is an option, it should not be considered
d) update the SSL directives on your IBM HTTP Server and set SSLClientAuth to "noverify". This will not fix the problem but will at least allow the server to be up and running with an expired certificate until you have obtained a new one.
(...if there are other options, please let me know)

Lessons learned:

• Keep the key ring file backed up
• Track certificate expiration time
• When expired, take action well in advance
• Even when delegating simple work, supply detailled instructions on least 250 pages

Off topic:
Why do I need to troubleshoot such a very very basic problem myself on a Saturday night?

Further reading:

• Guide to properly setting up SSL within the IBM HTTP Server
• MustGather : CMS key database (.kdb) and certificate problems