The aftermath of the heart bleed bug crisis

Written by Peter Rietveld and Bram van Pelt.

A preliminary Post-Mortem of Heartbleed. A lesson in Crisis Management? Last week was Heart Bleed week. It was what experts call the biggest security breach in the last years. Both the regular and specialized press have spent a lot of time explaining what this is and how it could be fixed.

You would expect that everything on the subject is said by now; the fix is known and already applied in most cases. It was just a bothersome bug but we can get back to business as usual. Right?

No. The Heart Bleed bug reveals how we deal with this kind of crisis. With all the efforts put in structuring security over the last years the process should have been established by now. We will be exploring the handling of the crisis in this Post-mortem to see what went right and where we need to improve.

The recovery operation applied for the Heart Bleed has been extremely pragmatic, not to say, it looks a lot like chaotically applied emergency bandages. If this is not going to improve, next time we will be in the same situation again. And furthermore; this crisis is far from over.

Observe

The news of the Heart Bleed bug came at us through the mainstream press, the security press and through Twitter from all sides. That was very nice. Why did it happen like this? Because it is a critical gap in a product used on a massive scale (two-thirds of all Internet-enabled devices is said to use OpenSSL. Similar gaps also occur in less commonly used software. Indeed, there are almost no product with a use as wide as OpenSSL. If your organization depends on the mainstream media for security alerts you will miss nearly all critical defects in any software less important. Theguardian.com or Slate magazine normally pay no attention to software bugs.  It is very likely that your organization has more than the most common used software out there. So if there is a super critical bug in it now; would you know of it? The answer is almost certainly negative.

It is somewhat of a coincidence that this bug has drawn so much attention. The very critical Pass The Hash attack discovered in 2013 affected almost certainly a larger part of the world than Heart Bleed – since it affects all networks with Microsoft products on it. But the press did not run the story – and if your business is depending on information drawn from the mainstream press, your organization missed it too.

Orient

How quick did we know the full impact of Heart Bleed when the news arrived? Have we established the full impact already?

How could you figure out on what machines the faulty software is running? OpenSSL has been tucked away in many products deep under the hood, even in products where you least expect it. OpenSSL is sometimes even embedded in hardware. Now, there is the well-known test tool but this test only tests websites. OpenSSL is not only embedded in websites, but in many other products such as VPNs, directories, mail servers, and many more. So how do you know if you’re vulnerable to Heart Bleed bug or even more important how would you know when the crisis subdued?

Of course you can wait until the vendor reports something. A survey of some major suppliers makes it clear that you must be following the staff blog to pick up onthis. At other companies you need to check the security advisories, while some vendors have nothing to offer or ask patience to find a solution. There are no standards for this type of news, so it is not easy to make a proper inventory. Moreover, you should know exactly what you have in-house and which suppliers you have to look for to fix a certain bug.

Another approach would be to start testing yourself. The test tool mentioned above is only suitable for websites, but that does not mean that it is difficult to test other systems. Indeed, it is possible to test other systems.

The Heart Bleed bug is an error in a specific SSL function. This function, the heartbeat function, has been proposed and written by Dr. Seggelmann and is intended to maintain an SSL session when the underlying protocol cannot handle sessions. Instead of ending and restarting an SSL session, the client sends out a heartbeat call. This call keeps the SSL session open so that no new ‘expensive’ SSL session should be started.

The bug which Heart Bleed abuses is in the implementation of the heartbeat response. In order to carry out the heartbeat call, a user must send a series of bytes, and specify the length of the series of bytes. OpenSSL stores this information, then sends the bytes back and the session continues. In an attack scenario it goes as follows. Instead of indicating how many bytes it sends, it claims a larger number of bytes than was actually send. In that case, OpenSSL retrieves more bytes from the memory than was send and returns this potentially sensitive information. That’s all there is to it.

Because the problem is in the software that creates and maintains SSL session, protocols that operate on top of SSL such as FTPS, SSH, SSL-POP3, SCEP, etc, are all compromised. SSL is a protocol that acts on the layer below HTTP, and must therefore be tested on that layer. Mail servers, for example, which only handle POP3 will probably not be tested right now, but may very well be equally vulnerable.

It is not so difficult to adapt the Heart bleed test to other types of servers. You can follow the next steps to test machines other than http machines:

  1. Download the following project https://github.com/musalbas/heartbleed-masstest
  2. Add the following at the command line options (somewhere between lines 35-46):
  3. options.add_option (‘- port’, ‘-p’, dest = “port”, default = 443, help = “ check port, useful for scanning non https servers)
  4. Next, search the document for the text ‘443 ‘. There are two hits, the first is in the line that has just been added, the second a bit further in the script
  5. Replace the 2nd hit: s.connect((host, 443)) to s.connect((host, opts.port))

From that point you can use the -p option to test a non-http port. This way you can single-handedlydetermine if a deviceis vulnerable.

Which brings us to the last question.

Decide and Act

Are you responding properly? Are all vulnerable components found and patched, all potentially sensitive material replaced and all users and customers sufficiently informed? According to most suppliers, you need to replace all certificates, passwords and the like. The American and the Dutch government agree with the suppliers; replace everything.

In practice, very few organizations replace their certificates like Netstat indicated April the 11th (http://news.netcraft.com/archives/2014/04/11/heartbleed-certificate-revocation-tsunami-yet-to-arrive.html). That means most of the organizations have not implemented the advice to replace all (yet). A good reason may be that the certificate revocation system of CRL and OCSP is not able to handle such massive numbers of revocations.

What ‘everything’ means is also far from clear. For example, if you look at token suppliers, you can see that many strong authentication products may have been affected by the products they cooperate with, such as the well-known login tokens used by banks and software tokens installed on phones. A good explanation of how they work and what the solution might look like for the OTP can be read here.

It is exactly as the suppliers of hardware tokens report, whether  or not they do so prominently, the tokens need new keying material. This means a huge logistical operation is required to re-initialize or replace hardware tokens. Every customer should therefore request new hardware tokens from the banks. Ouch.

Would they really do this?

In practice it is really difficult to replace these tokens. It is understandable that organizations doubt about what to do, given the cost and impact. After all, we do not know whether any information has been leaked, and we will never know. It is such a large and costly operation to replace everything that may have been compromised. With such a large operation ​​big mistakes can also be made, as Akamai found out. Let’s chalk that up to act before completely thinking it through. There is much appeal in ignoring this entire situation as much as possible.

The disadvantage of blissful ignorance is that you will not learn from Heart Bleed. Carrying on and learning is vital because Heart Bleed is not over yet. Even worse, it is just a taste of crises that are certain to come.

Written by Peter Rietveld and Bram van Pelt.