I’m blogging this to help other people who might experience the same problem. On the other hand there are some loose ends where someone out there might contribute to.
So a project required SOAP/HTTP(S) requests over two-way SSL between Oracle Service Bus and a remote service running on Microsoft IIS 6.0.
Weblogic & OSB setup:
All certicates are stored in keystores and used by the nodemgr + admin + ms.
Weblogic has a PKICredentialMapper and in the SBconsole we have our ServiceKeyProvider configured. We build our proxy and business services where:
- BS is confgured with SSL and Authentication ClientCert (http transport Tab)
- PS is configured with the ServiceKeyProvider on the Security tab
Pretty default stuff, working with other backends. The callout however fails and the IIS guys claim they never receive a client-cert.
Callout from OSB:
clientIPhere, -, 1/11/2011, 15:51:18, serviceNameHere, serverNameHere, serverIPhere, 449859, 1160, 0, 403, 64, POST, /SomeService.asmx, -,
Callout from SOAPui from client workstation:
clientIPhere, -, 1/11/2011, 16:01:21, serviceNameHere, serverNameHere, serverIPhere, 187, 918, 916, 200, 0, POST, /SomeService.asmx, -,
According to this page the 403 is a forbidden error and the 0 in front of it means 0 bytes have been send from the server to the client.
So we enable Weblogic SSL debugging with the arguments: -Dssl.debug=true -Dweblogic.StdoutDebugEnabled=true
After the next callout we see this in the logging:
</soap:Body>> *<-------- this is the last $body logging in the ProxyService before it calls the business* <Debug> <SecuritySSL> <BEA-000000> <SSLSetup: loading trusted CA certificates><Debug> <SecuritySSL> <BEA-000000> <clientInfo has new style certificate and key><Debug> <SecuritySSL> <BEA-000000> <Filtering JSSE SSLSocket><Debug> <SecuritySSL> <BEA-000000> <SSLIOContextTable.addContext(ctx): 270694655><Debug> <SecuritySSL> <BEA-000000> <SSLSocket will be Muxing><Debug> <SecuritySSL> <BEA-000000> <write SSL_20_RECORD><Debug> <SecuritySSL> <BEA-000000> <isMuxerActivated: false><Debug> <SecuritySSL> <BEA-000000> <270694400 SSL3/TLS MAC><Debug> <SecuritySSL> <BEA-000000> <270694400 received HANDSHAKE><Debug> <SecuritySSL> <BEA-000000> <HANDSHAKEMESSAGE: ServerHello><Debug> <SecuritySSL> <BEA-000000> <HANDSHAKEMESSAGE: Certificate><Debug> <SecuritySSL> <BEA-000000> <Validating certificate 0 in the chain: Serial number: 130583870796631153730390131696172958331Issuer:C=xxxxxxxxxxNot Valid Before:Wed Oct 20 02:00:00 CEST 2010Not Valid After:Sun Oct 20 01:59:59 CEST 2013Signature Algorithm:SHA1withRSA><Debug> <SecuritySSL> <BEA-000000> <Validating certificate 1 in the chain: Serial number: 50682576685400420811231509872710703774Issuer:C=xxxxxxxNot Valid Before:Tue Jun 07 10:09:10 CEST 2005Not Valid After:Sat May 30 12:48:38 CEST 2020Signature Algorithm:SHA1withRSA><Debug> <SecuritySSL> <BEA-000000> <validationCallback: validateErr = 0><Debug> <SecuritySSL> <BEA-000000> < cert[0] = Serial number: 130583870796631153730390131696172958331Issuer:C=xxxxxxxxxxSubject:C=xxxxxxxNot Valid Before:Wed Oct 20 02:00:00 CEST 2010Not Valid After:Sun Oct 20 01:59:59 CEST 2013Signature Algorithm:SHA1withRSA><Debug> <SecuritySSL> <BEA-000000> < cert[1] = Serial number: 50682576685400420811231509872710703774Issuer:CxxxxxxxxNot Valid Before:Tue Jun 07 10:09:10 CEST 2005Not Valid After:Sat May 30 12:48:38 CEST 2020Signature Algorithm:SHA1withRSA><Debug> <SecuritySSL> <BEA-000000> <weblogic user specified trustmanager validation status 0><Debug> <SecuritySSL> <BEA-000000> <SSLTrustValidator returns: 0><Debug> <SecuritySSL> <BEA-000000> <Trust status (0): NONE><Debug> <SecuritySSL> <BEA-000000> <Performing hostname validation checks: www.nahetvo.nl><Debug> <SecuritySSL> <BEA-000000> <HANDSHAKEMESSAGE: ServerHelloDone><Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacMD5><Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacMD5><Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacSHA1><Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacSHA1><Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm MD5><Debug> <SecuritySSL> <BEA-000000> <Using JCE Cipher: SunJCE version 1.6 for algorithm RC4><Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacMD5><Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacMD5><Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacSHA1><Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacSHA1><Debug> <SecuritySSL> <BEA-000000> <Using JCE Cipher: SunJCE version 1.6 for algorithm RSA><Debug> <SecuritySSL> <BEA-000000> <write HANDSHAKE, offset = 0, length = 134><Debug> <SecuritySSL> <BEA-000000> <write CHANGE_CIPHER_SPEC, offset = 0, length = 1><Debug> <SecuritySSL> <BEA-000000> <Using JCE Cipher: SunJCE version 1.6 for algorithm RC4><Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HMACMD5><Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HMACMD5><Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacMD5><Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacMD5><Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacSHA1><Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacSHA1><Debug> <SecuritySSL> <BEA-000000> <write HANDSHAKE, offset = 0, length = 16><Debug> <SecuritySSL> <BEA-000000> <isMuxerActivated: false><Debug> <SecuritySSL> <BEA-000000> <270694400 SSL3/TLS MAC><Debug> <SecuritySSL> <BEA-000000> <270694400 received CHANGE_CIPHER_SPEC><Debug> <SecuritySSL> <BEA-000000> <Using JCE Cipher: SunJCE version 1.6 for algorithm RC4><Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HMACMD5><Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HMACMD5><Debug> <SecuritySSL> <BEA-000000> <isMuxerActivated: false><Debug> <SecuritySSL> <BEA-000000> <270694400 SSL3/TLS MAC><Debug> <SecuritySSL> <BEA-000000> <270694400 received HANDSHAKE><Debug> <SecuritySSL> <BEA-000000> <HANDSHAKEMESSAGE: Finished><Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacMD5><Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacMD5><Debug> <SecuritySSL> <BEA-000000> <Ignoring not supported JCE Mac: SunJCE version 1.6 for algorithm HmacSHA1><Debug> <SecuritySSL> <BEA-000000> <Will use default Mac for algorithm HmacSHA1><Debug> <SecuritySSL> <BEA-000000> <write APPLICATION_DATA, offset = 0, length = 321><Debug> <SecuritySSL> <BEA-000000> <write APPLICATION_DATA, offset = 0, length = 839><Debug> <SecuritySSL> <BEA-000000> <SSLIOContextTable.findContext(sock): 270693819><Debug> <SecuritySSL> <BEA-000000> <SSLIOContextTable.findContext(sock): 270693819><Debug> <SecuritySSL> <BEA-000000> <activateNoRegister()><Debug> <SecuritySSL> <BEA-000000> <SSLFilterImpl.activate(): activated: 270693844 270909954><Debug> <SecuritySSL> <BEA-000000> <270694391 read(offset=0, length=4080)><Debug> <SecuritySSL> <BEA-000000> <isMuxerActivated: true><Debug> <SecuritySSL> <BEA-000000> <hasSSLRecord()><Debug> <SecuritySSL> <BEA-000000> <hasSSLRecord returns true><Debug> <SecuritySSL> <BEA-000000> <270694400 SSL3/TLS MAC><Debug> <SecuritySSL> <BEA-000000> <270694400 received HANDSHAKE><Debug> <SecuritySSL> <BEA-000000> <NEW ALERT with Severity: WARNING, Type: 100java.lang.Exception: New alert stack at com.certicom.tls.record.alert.Alert.<init>(Unknown Source) at com.certicom.tls.record.handshake.HandshakeHandler.handleHandshakeMessages(Unknown Source) at com.certicom.tls.record.MessageInterpreter.interpretContent(Unknown Source) at com.certicom.tls.record.MessageInterpreter.decryptMessage(Unknown Source) at com.certicom.tls.record.ReadHandler.processRecord(Unknown Source) at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source) at com.certicom.tls.record.ReadHandler.read(Unknown Source) at com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source) at weblogic.socket.SSLFilterImpl.isMessageComplete(SSLFilterImpl.java:202) at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:896) at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:840) at weblogic.socket.EPollSocketMuxer.dataReceived(EPollSocketMuxer.java:215) at weblogic.socket.EPollSocketMuxer.processSockets(EPollSocketMuxer.java:177) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)><Debug> <SecuritySSL> <BEA-000000> <write ALERT, offset = 0, length = 2><Debug> <SecuritySSL> <BEA-000000> <isMuxerActivated: true><Debug> <SecuritySSL> <BEA-000000> <hasSSLRecord()><Debug> <SecuritySSL> <BEA-000000> <hasSSLRecord returns false 1><Debug> <SecuritySSL> <BEA-000000> <270694391 Rethrowing InterruptedIOException><Debug> <SecuritySSL> <BEA-000000> <270694391 read(offset=0, length=8192)><Debug> <SecuritySSL> <BEA-000000> <isMuxerActivated: false><Debug> <SecuritySSL> <BEA-000000> <264112838 read(offset=0, length=4080)><Debug> <SecuritySSL> <BEA-000000> <isMuxerActivated: true><Debug> <SecuritySSL> <BEA-000000> <hasSSLRecord()><Debug> <SecuritySSL> <BEA-000000> <hasSSLRecord returns true><Debug> <SecuritySSL> <BEA-000000> <264114219 SSL3/TLS MAC><Debug> <SecuritySSL> <BEA-000000> <264114219 received APPLICATION_DATA: databufferLen 0, contentLength 29><Debug> <SecuritySSL> <BEA-000000> <264112838 read databufferLen 29><Debug> <SecuritySSL> <BEA-000000> <264112838 read A returns 29><Debug> <SecuritySSL> <BEA-000000> <264112838 read(offset=29, length=4051)><Debug> <SecuritySSL> <BEA-000000> <isMuxerActivated: true><Debug> <SecuritySSL> <BEA-000000> <hasSSLRecord()><Debug> <SecuritySSL> <BEA-000000> <hasSSLRecord returns false 1><Debug> <SecuritySSL> <BEA-000000> <264112838 Rethrowing InterruptedIOException><Debug> <SecuritySSL> <BEA-000000> <write APPLICATION_DATA, offset = 0, length = 29>
Just before the dump we see a: Warning, Type 100. The TLS protocol states informs us that this alert 100 would mean NO_RENEGOTIATION:
A.3. Alert messages
enum { warning(1), fatal(2), (255) } AlertLevel;
enum { close_notify(0), unexpected_message(10), bad_record_mac(20), decryption_failed(21), record_overflow(22), decompression_failure(30), handshake_failure(40), bad_certificate(42), unsupported_certificate(43), certificate_revoked(44), certificate_expired(45), certificate_unknown(46), illegal_parameter(47), unknown_ca(48), access_denied(49), decode_error(50), decrypt_error(51), export_restriction(60), protocol_version(70), insufficient_security(71), internal_error(80), user_canceled(90), no_renegotiation(100), (255) } AlertDescription;
struct { AlertLevel level; AlertDescription description; } Alert;
So we use Wireshark which is one of the coolest free software tools out there. And even better, did I mention it’s free? With a Wireshark capture you can get a lot of information about the connection.
![]()
Essentially, what is going on here can be summarized as:
- Step 4: The client makes a hello request
- Step 8: The server responds with its certificate
- Step 10: The client key is send to server
- Step 11+12: Clients send a change of cipher spec and handshake message
- Step 14: Server send a change of cipher spec and handshake message
- Step 19: We see another server triggered handshake message followed by
- Step 20: Encrypted Alert
Since we are looking for something interesting, this Encrypted Alert is it. The problem here is that you can see an Alert but cant see the exact alert type.
I was hoping for a CERT_CHAIN_UNTRUSTED or BAD_CERTIFICATE since that would give me a good direction. But it looks like Weblogic logging shows an alert 100 (no_renegotiation) and we had to go with that.
no_renegotiation:
Sent by the client in response to a hello request or by the server in response to a client hello after initial handshaking. Either of these would normally lead to renegotiation; when that is not appropriate, the recipient should respond with this alert; at that point, the original requester can decide whether to proceed with the connection. One case where this would be appropriate would be where a server has spawned a process to satisfy a request; the process might receive security parameters
(key length, authentication, etc.) at startup and it might be difficult to communicate changes to these parameters after that point. This message is always a warning.
I’m still not fully convinced of a no_renegotiation. Because for the client to trigger a renegotiation, it should send a new Client Hello message (in the encrypted channel, like any other handshaking message) which then the server responds with a Server Hello, and negotiation goes exactly as above. Other case when the server initiates a renegotiation it will send the client a Hello Request message on which he client simply sends a new Client Hello, exactly as above, and the process goes as usual. In the wireshark captures this behavious is not visible.
<update.2011-01-27>
I found this Wireshark capture in a document called “Renegotiating TLS” by Ray and Dispensa which shows the process of renegotiation.
![]()
</update.2011-01-27>
So as a fix we tried switching over from JRockit to Sun JDK, tried some arguments but in the end rolled everything back since 1 simple options seem to solve the issue:
With older version of Weblogic you can use these arguments for the jvm:
-Djava.protocol.handler.pkgs=com.sun.net.ssl.internal.www.protocol
-Dssl.SocketFactory.provider=com.sun.net.ssl.internal.SSLSocketFactoryImpl
-DUseSunHttpHandler=true
-Dweblogic.wsee.client.ssl.usejdk=true (for webservice clients)
With the latest versions of Weblogic there is a “Use JSEE SSL” setting. The setting is located on the SSL (advanced) tab of the managed server.
![]()
So now our Wireshark capture looks like this:
![]()
The difference looks to occur after the server sends an encrypted handshake (ERR-capture step 19 & OK-capture step 42). In the OK-capture you can see the client sending an encrypted handshake after that (step 43), however the ERR-capture results in a Alert (step 20).
Focusing on the Weblogic logging and the NO_RENEGOTIATION message a guess would be that the different client implementations handle these renegotiations differently.
The Oracle Support Knowledge base mentions something like this (ID 1078957.1). The trace mentioned there isn’t identical as our error, however the solution seems to point in the same direction.
ERROR = HANDSHAKE_FAILURE
CAUSE = This may happen when the WebLogic server connects a remote Microsoft server.
SOLUTION =
This happened before the client got the ServerHello message. The Microsoft Server was refusing the handshake because the cipher suites given to the remote server were not 128 bits – the remote server wasn’t allowing anything lower. If the client license is changed on the WebLogic Server side, it will then work.
Our issue however is not identical (i think), because I can see that the client (weblogic) sends 19 cipher specs (among them TLS_RSA_WITH_RC4_128_MD5) and that the server agrees with this specific 00 44
TLS_RSA_WITH_NULL_MD5 = { 0x00,0x01 }TLS_RSA_WITH_NULL_SHA = { 0x00,0x02 }TLS_RSA_EXPORT_WITH_RC4_40_MD5 = { 0x00,0x03 }TLS_RSA_WITH_RC4_128_MD5 = { 0x00,0x04 }TLS_RSA_WITH_RC4_128_SHA = { 0x00,0x05 }TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 = { 0x00,0x06 }TLS_RSA_WITH_IDEA_CBC_SHA = { 0x00,0x07 }TLS_RSA_EXPORT_WITH_DES40_CBC_SHA = { 0x00,0x08 }TLS_RSA_WITH_DES_CBC_SHA = { 0x00,0x09 }TLS_RSA_WITH_3DES_EDE_CBC_SHA = { 0x00,0x0A }TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA = { 0x00,0x62 }TLS_RSA_EXPORT1024_WITH_RC4_56_SHA = { 0x00,0x64 }TLS_DH_DSS_EXPORT_WITH_DES40_CBC_SHA = { 0x00,0x0B }TLS_DH_DSS_WITH_DES_CBC_SHA = { 0x00,0x0C }TLS_DH_DSS_WITH_3DES_EDE_CBC_SHA = { 0x00,0x0D }TLS_DH_RSA_EXPORT_WITH_DES40_CBC_SHA = { 0x00,0x0E }TLS_DH_RSA_WITH_DES_CBC_SHA = { 0x00,0x0F }TLS_DH_RSA_WITH_3DES_EDE_CBC_SHA = { 0x00,0x10 }TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA = { 0x00,0x11 }TLS_DHE_DSS_WITH_DES_CBC_SHA = { 0x00,0x12 }TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA = { 0x00,0x13 }TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA = { 0x00,0x14 }TLS_DHE_RSA_WITH_DES_CBC_SHA = { 0x00,0x15 }TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA = { 0x00,0x16 }TLS_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA= { 0x00,0x63 }TLS_DHE_DSS_EXPORT1024_WITH_RC4_56_SHA = { 0x00,0x65 }TLS_DHE_DSS_WITH_RC4_128_SHA = { 0x00,0x66 }TLS_DHE_DSS_WITH_NULL_SHA = { 0x00,0x67 }TLS_DH_anon_EXPORT_WITH_RC4_40_MD5 = { 0x00,0x17 }TLS_DH_anon_WITH_RC4_128_MD5 = { 0x00,0x18 }TLS_DH_anon_EXPORT_WITH_DES40_CBC_SHA = { 0x00,0x19 }TLS_DH_anon_WITH_DES_CBC_SHA = { 0x00,0x1A }TLS_DH_anon_WITH_3DES_EDE_CBC_SHA = { 0x00,0x1B }
So our problems seems to be solved by using the Sun SSL implementation instead of the default Certicom one.
However that still leaves questions unanswered for me as:
- how come that other 2-way SSL trusts succeed with the default, but in this case with IIS6.0 as a service provider fails
- where does the 2-way SSL trust exactly fail, and how can I pinpoint this with tools like Wireshark ?
- what is the exact difference between Certicom and Sun’s implementation off TLS 1.0 ?