January 20, 2015

Running SCAP Scans

OpenSCAP can be run from the command line, but there are easier ways to do it.

OpenSCAP support has been integrated into Red Hat Satellite and into the Spacewalk open source management platform.

Red Hat Satellite has the ability to push SCAP content to managed systems and to run the SCAP audit scans. Red Hat Satellite has the ability to schedule SCAP audit scans and to retrieve the reports and access them through the Red Hat Satellite Audit tab.

If you are going to be using SCAP in production, especially on large numbers of systems, you should really be using a management framework like Red Hat Satellite or Spacewalk.

For development, testing, tuning SCAP benchmarks, and small scale use, the SCAP Workbench is a friendly and flexible tool. We will cover this in more detail in the next post.

January 12, 2015

Security Tests – SCAP Content

While the SCAP technologies are interesting, they have limited value without security content – the actual set of security tests run by SCAP. Fortunately there is a good set of content available that can be used as a starting point.

The US Government has released a set of SCAP content that covers the baseline security required – the United States Government Configuration Baseline (USGCB), which contains the security configuration baselines for a variety of computer products which are widely deployed across federal agencies. USGCB content covers Internet Explorer, Windows, and Red Hat Enterprise Linux Desktop 5.

Also from the US Government is the Department of Defense STIG or Security Technical Implementation Guides. A specific example of this would be downloadable SCAP Content for RHEL 6, the Red Hat 6 STIG Benchmark, Version 1, Release 4.

A number of vendors include SCAP content in their products. This is often a sample or an example – it is enough to get you started, but does not provide a comprehensive security scan.

While the available SCAP content is a good start, most organizations will have additional needs. This can be addressed in two ways: by tailoring existing SCAP content and by writing new SCAP content.

Tailoring SCAP content involves choosing which SCAP rules will be evaluated and changing parameters.

An example of changing parameters is minimum password length. The default value might be 12 characters. You can change this in a tailoring file, perhaps to 8 characters or to 16 characters for a highly secure environment.

A common way to use SCAP is to have a large SCAP benchmark (content) which is used on all systems, and to select which rules will be used for each scan. This can be changed for each system and each run. You do this by providing the SCAP benchmark, an SCAP Tailoring file, and running the SCAP scanner.

Writing new SCAP content can be a daunting task. SCAP is a rich enterprise framework – in other words, it is complex and convoluted… If you are going to be writing SCAP content (and you really should), I suggest starting with Security Automation Essentials, getting very familiar with the various websites we’ve mentioned, studying the existing SCAP content, and being prepared for a significant learning curve.

Update on Red Hat Enterprise Linux 6 and FIPS 140 validations

Red Hat achieved its latest successful FIPS 140 validation back in April 2013. Since then, a lot has happened. There have been well publicized attacks on cryptographic protocols, weaknesses in implementations, and changing government requirements. With all of these issues in play, we want to explain what we are doing about it.

One of the big changes was that we enabled support of Elliptic Curve Cryptography (ECC) and Elliptic Curve Diffie Hellman (ECDH) in Red Hat Enterprise Linux to meet the National Institute of Standards and Technology’s (NIST’s) “Suite B” requirements taking effect this year. Because we added new ciphers, we knew we needed to re-certify. Re-certification brings many advantages to our government customers, who not only benefit from the re-certification, but they also maintain coverage from our last FIPS 140 validation effort. One advantage of re-certification is that we have picked up fixes for BEAST, Lucky 13, Heartbleed, Poodle, and some lesser known vulnerabilities around certificate validation. It should be noted that these attacks are against higher level protocols that are not part of any crypto primitives covered by a FIPS validation. But, knowing the fixes are in the packages under evaluation should give customers additional peace of mind.

The Red Hat Enterprise Linux 6 re-certification is now under way. It includes reworked packages to meet all the updated requirements that NIST has put forth taking effect Jan. 1, 2014, such as a new Deterministic Random Bit Generator (DRGB) as specified in SP 800-90A (PDF); an updated RSA key generation technique as specified in FIPS 186-4 (PDF); and updated key sizes and algorithms as specified in SP 800-131A (PDF).

Progress on the certification is moving along – we’ve completed review and preliminary testing and are now applying for Cryptographic Algorithm Validation System (CAVS) certificates. After that, we’ll submit validation paperwork to NIST. All modules being re-certified are currently listed on NIST’s Modules in Process page, except Volume Encryption (dm-crypt). Its re-certification is taking a different route because the change is so minor thus not needing CAVS testing. We are expecting the certifications to be completed early this year.

January 06, 2015

What’s worse?
<noscript>Take Our Poll</noscript><script type="text/javascript"> (function(d,c,j){if(!d.getElementById(j)){var pd=d.createElement(c),s;pd.id=j;pd.src='https://s1.wp.com/wp-content/mu-plugins/shortcodes/js/polldaddy-shortcode.js';s=d.getElementsByTagName(c)[0];s.parentNode.insertBefore(pd,s);} else if(typeof jQuery !=='undefined')jQuery(d.body).trigger('pd-script-load');}(document,'script','pd-polldaddy-loader')); </script>
Securing Secure Shell

I was passed an interesting article, this morning, regarding hardening secure shell (SSH) against poor crypto that can be a victim of cracking by the NSA and other entities.  The article is well written and discusses why the changes are necessary in light of recent Snowden file releases.

January 05, 2015

SCAP Component Technologies

We’re going to dig into SCAP in a fair amount of detail. So, let’s start by covering the various technologies that make up SCAP:

  • XCCDF – the Extensible Configuration Checklist Description Format. An XML based language for creating machine parsable security checklists.
  • OVAL – the Open Vulnerability and Assessment Language. Standardizes how to assess and report on the machine state of computer systems.
  • OCIL – the Open Checklist Interactive Language. Ask users questions. For example, “do you know who to report security breaches to?” and allowing the user to respond with yes or no – or perhaps the name and contact information of where to report security breaches.
  • CCE – Common Configuration and Enumeration. Uniquely identify configuration characteristics. For example, how do you identify minimum password length across Windows, Unix, Linux and Mac?
  • CPE – Common Platform Enumeration. A structured naming scheme for IT systems, software and packaging.
  • CVE – Common Vulnerability Enumeration. A standard way to uniquely identify computer vulnerabilities, for example HeartBleed – CVE-2014-0160.
  • CEE – Common Event Expression. A common way to record events – i.e. a standard logging format.
  • CRE – Common Remediation Enumeration. Describes how to remediate or mitigate security vulnerabilities.
  • CVSS – Common Vulnerability Scoring System. A consistent methodology for measuring and quantifying the impact and risk of vulnerabilities identified through CVE.

Some of these are widely used. For example, the CVE Database maintained by Mitre is the common resource used for sharing information on security vulnerabilities. It has been used by security professionals around the world for over 15 years.

Others are new, such as the use of XCCDF and OVAL to create standardized security content that can be shared across organizations and industries and be used by automated scanners.

December 18, 2014

Before you initiate a “docker pull”

In addition to the general challenges that are inherent to isolating containers, Docker brings with it an entirely new attack surface in the form of its automated fetching and installation mechanism, “docker pull”. It may be counter-intuitive, but “docker pull” both fetches and unpacks a container image in one step. There is no verification step and, surprisingly, malformed packages can compromise a system even if the container itself is never run. Many of the CVE’s issues against Docker have been related to packaging that can lead to install-time compromise and/or issues with the Docker registry.

One, now resolved, way such malicious issues could compromise a system was by a simple path traversal during the unpack step. By simply using a tarball’s capacity to unpack to paths such as “../../../” malicious images were able to override any part of a host file system they desired.

Thus, one of the most important ways you can protect yourself when using Docker images is to make sure you only use content from a source you trust and to separate the download and unpack/install steps. The easiest way to do this is simply to not use “docker pull” command. Instead, download your Docker images over a secure channel from a trusted source and then use the “docker load” command. Most image providers also serve images directly over a secure, or at least verifiable, connection. For example, Red Hat provides a SSL-accessible “Container Images”.  Fedora also provides Docker images with each release as well.

While Fedora does not provide SSL with all mirrors, it does provide a signed checksum of the Docker image that can be used to verify it before you use “docker load”.

Since “docker pull” automatically unpacks images and this unpacking process itself is often compromised, it is possible that typos can lead to system compromises (e.g. a malicious “rel” image downloaded and unpacked when you intended “rhel”). This typo problem can also occur in Dockerfiles. One way to protect yourself is to prevent accidental access to index.docker.io at the firewall-level or by adding the following /etc/hosts entry: index.docker.io

This will cause such mistakes to timeout instead of potentially downloading unwanted images. You can still use “docker pull” for private repositories by explicitly providing the registry:

docker pull registry.somewhere.com/image

And you can use a similar syntax in Dockerfiles:

from registry.somewhere.com/image

Providing a wider ecosystem of trusted images is exactly why Red Hat began its certification program for container applications. Docker is an amazing technology, but it is neither a security nor interoperability panacea. Images still need to come from sources that certify their security, level-of-support, and compatibility.

December 17, 2014

Container Security: Isolation Heaven or Dependency Hell

Docker is the public face of Linux containers and two of Linux’s unsung heroes: control groups (cgroups) and namespaces. Like virtualization, containers are appealing because they help solve two of the oldest problems to plague developers: “dependency hell” and “environmental hell.”

Closely related, dependency and environmental hell can best be thought of as the chief cause of “works for me” situations. Dependency hell simply describes the complexity inherent in modern application’s tangled graph of external libraries and programs they need to function. Environmental hell is the name for the operating system portion of that same problem (i.e. what wrinkles, in particular which bash implementation,on which that quick script you wrote unknowingly relies).

Namespaces provide the solution in much the same way as virtual memory simplified writing code on a multi-tenant machine: by providing the illusion that an application suite has the computer all to itself. In other words,”via isolation”. When a process or process group is isolated via these new namespace features, we say they are “contained.” In this way, virtualization and containers are conceptually related, but containers isolate in a completely different way and conflating the two is just the first of a series of misconceptions that must be cleared up in order to understand how to use containers as securely as possible. Virtualization involves fully isolating programs to the point that one can use Linux, for example, while another uses BSD. Containers are not so isolated. Here are a few of the ways that “containers do not contain:”

  1. Containers all share the same kernel. If a contained application is hijacked with a privilege escalation vulnerability, all running containers *and* the host are compromised. Similarly, it isn’t possible for two containers to use different versions of the same kernel module.
  2. Several resources are *not* namespaced. Examples include normal ulimit systems still being needed to control resources such as filehandlers. The kernel keyring is another example of a resource that is not namespaced. Many beginning users of containers find it counter-intuitive that socket handlers can be exhausted or that kerberos credentials are shared between containers when they believe they have exclusive system access. A badly behaving process in one container could use up all the filehandles on a system and starve the other containers. Diagnosing the shared resource usage is not feasible from within
  3. By default, containers inherit many system-level kernel capabilities. While Docker has many useful options for restricting kernel capabilities, you need a deeper understanding of an application’s needs to run it inside containers than you would if running it in a VM. The containers and the application within them will be dependent on the capabilities of the kernel on which they reside.
  4. Containers are not “write once, run anywhere”. Since they use the host kernel, applications must be compatible with said kernel. Just because many applications don’t depend on particular kernel features doesn’t mean that no applications do.

For these and other reasons, Docker images should be designed and used with consideration for the host system on which they are running. By only consuming images from trusted sources, you reduce the risk of deploying containerized applications that exhaust system resources or otherwise create a denial of service attack on shared resources. Docker images should be considered as powerful as RPMs and should only be installed from sources you trust. You wouldn’t expect your system to remain secured if you were to randomly install untrusted RPMs nor should you if you “docker pull” random Docker images.

In the future we will discuss the topic of untrusted images.

December 12, 2014

How to really screw up TLS

I’ve noticed a few of my favorite websites failing with some odd error from Firefox.

Firefox's Unable to connect securely error messageThe Firefox error message is a bit misleading.  It actually has nothing to do with the website supporting SSL 3.0 but the advanced info is spot on.  The error “ssl_error_no_cypher_overlap” means that the client didn’t offer any ciphers that the server also supports.  Generally when I see this I assume that the server has been setup poorly and only supports unsafe ciphers.  In this case the website only supports the RC4 cipher.  I wondered why I was starting to see a reversal of removing RC4 from so many websites recently (especially since RC4 is very weak and is on the way out).  Apparently these websites all use the F5 load balancer that had a bad implementation of the TLS 1.0 standard causing a POODLE-like vulnerability.

Stepping back for a moment, back in October the POODLE vulnerability hit the streets and a mass exodus from SSL 3.0 happened around the world.  I was happy to see so many people running away from the broken cryptographic protocol and very happy to see the big push to implementing the latest version of TLS, TLS 1.2.  So with SSL 3.0 out of the way and the POODLE vulnerability being squelched why are we seeing problems in TLS 1.0 now?

Well, simply put, F5 load balancers don’t implement TLS 1.0 correctly.  The problem with SSL 3.0 is that the padding format isn’t checked.  Apparently in the F5 devices it’s still a problem in TLS 1.0.  And while the company did offer up patches to fix the issue, some really bad advice has been circulating the Internetz telling people to only support RC4, again.  Sigh.

When RC4 finally dies a fiery death I’ll likely throw a party.  I’m sure I won’t be the only one…

December 11, 2014

Rolekit (or “How I learned to stop thinking in terms of packages”)

What’s the problem?

Let’s start with a simplification and discuss the lifecycle of software at a high-level:

  1. Research and Development – In this phase, the software is designed, coded and (hopefully) tested.
  2. Packaging – Here, we take the compiled, tested bits of the software and bundle it up into some sort of package that can be used to deliver it to a user.
  3. Deployment – An end-user takes the package and does something interesting with it (for the purists out there, I’m lumping the test, staging and production environments into the “deployment” category).

Despite the brevity of the list above, there are a lot of moving parts here. I’m going to use the Fedora process to illustrate how this all works in a pre-rolekit world and then talk a little bit about the limitations, some of the alternatives and finally how rolekit addresses the issue. First, though, I’ll answer the question I posited in the header: “What’s the problem?”

The problem to be solved is how to get useful software up and running in an end-user’s environment with the least amount of difficulty for the user. The first and most important rule in software is this: software is a means to an end, not an end unto itself. People install a piece of software in order to achieve a goal. This goal could be something relatively simple, such as “I want to listen to this MP3 I bought” or as complex as “I run the IT department for a multinational manufacturing company and I want to keep track of all my products, the rate of their sales and margins as well as what my competitors are doing”. The job of software is to enable the user to get to that desired state. To that end, I would argue this: it is far more important to help the user get started than it is to offer them every possible feature.

Some of you may interject: “But if you don’t have the feature they need, won’t they go to someone who does?”. Sure, sometimes that will happen. But you will probably discover that people will make a different tradeoff than you might think: “I can get 90% of what I need and get it set up in a few weeks” is a far more compelling statement to make to a financial decision-maker than “This product provides everything we need, but I’ll need two more full-time people to get it running next year”.

What are we doing today?

Open source development is fairly unique compared to traditional software development. One of its major advantages for development can also become its biggest challenge to deployment. Because of the breadth of open source projects out there, there is almost always someone who has done at least a piece of what you want to do already. These other projects, such as coding libraries, web application frameworks, video game engines, etc. all provide the building blocks to start your work. The great thing here is that you can pick up the pieces that you need from somewhere else and then focus your attention only on the parts that make your project unique or exciting.

However, the challenge starts happening when you get to the packaging phase. Now that you have something you want to share with the world, you need to package it in a manner that allows them to use it. There are generally two schools of thought on how to do this, each with their own strengths and weaknesses.

  1. Grab the source code (or pre-built binaries) for everything that you depend on for your project to work and package them all together in a single deliverable.
  2. Package all of your dependencies separately in their own deliverables

I’m not going to go into the details of why, but the Fedora Project has policies that require the second option. (If you’re interested in the reasoning, I strongly recommend reading the Fedora Packaging Guidelines page on the subject). Fedora then provides a dependency-resolution mechanism that simplifies this case by ensuring that when you attempt to retrieve the package you want, it also automatically installs all of the packages that it depends on (and so on, recursively until they are all satisfied).

How do we deploy it now?

There are two schools of thought on this subject, which I will refer to as the “Fedora Approach” and the “Debian Approach”, since those two Linux distributions best represent them. (Note: my understanding of the Debian Approach is second-hand, so if I get any of the subtleties incorrect, please feel free to leave a comment and I’ll correct it).

The Debian Approach

In Debian and its derivatives (such as Ubuntu, Mint, etc.), when the package resolution is completed and the packages are downloaded, the user is required to indicate at that time their explicit decision on how the package must behave. Through a system called “debconf”, package installation is directly tied to deployment; the package installation cannot conclude without it being explicitly configured at that time. If the installation is non-interactive (such as if the installation was initiated by another service, rather than the user), the configuration must either be specified by an “answer file” (a configuration file passed to debconf stating the answers in advance) or else the package must provide a sensible set of defaults to automatically deploy it.

 The Fedora Approach

In Fedora and its derivatives (such as Red Hat Enterprise Linux, CentOS, Scientific Linux, etc.), when the package resolution is completed and the packages are downloaded, that’s it. In the vast majority of cases, the software is now on the system, but it is not configured to do anything at all. (There are a few specific exceptions which have been granted by the Fedora Engineering Steering Committee for things like the firewall). On these systems, nothing will happen until the user takes an explicit action to configure and start the services.

“That sounds like the Debian Approach is better!” you may say. However, there are concerns to be had here. For one, the above explanation I made about dependency-resolution comes into play; you as a user may not be fully aware of what packages are going to be pulled in by your dependencies (even accidentally). Furthermore, just because you installed a web-server package, it doesn’t mean that you necessarily want it running immediately. So, Fedora forces you to make these decisions explicitly, rather than implicitly. So when you’re ready, you configure the software and then start it up.

Where does this fall down?

The real problem is that the concept of “packages” derives very much from the engineering side of things. A package is a logical bundling of software for the developers. Not all problems can be solved with a single package, though. For example, the FreeIPA identity-management solution requires many top-level packages including an LDAP directory server, a certificate authority server, a DNS server and others. In this, the concept of a “package” gets more than a little fuzzy. In this particular case (as has been common historically), the solution was “Let’s make another package that glues them together!”. So the FreeIPA package just adds those other packages to its dependency chain.

But just adding more packages doesn’t necessarily solve the end-user concern: How do I easily deploy this?

Enter rolekit

Rolekit was designed to be specifically for handling the deployment situation and shield end-users from the concept of project-level packages. Instead, complete solutions will be “packaged” as Server Roles. Users will come to rolekit and declare a machine to be e.g. a Domain Controller, providing the minimum information necessary to set it up (today, that’s just an admin password in the Domain Controller example). Rolekit will handle all of the other necessary work under the hood, which involves downloading the appropriate packages, installing them on the system, setting up the configuration, starting the appropriate services and carefully opening up the firewall to allow access to it.

There are a lot of moving parts involved in deploying a role, but the user doesn’t really need to know what they are. If they can be shielded from much of the noise and churn inherent in package installation, configuration, service management and firewall settings, then they get back much of their time for solving the problems unique to their environments.

Fedora and Server Roles

As of Fedora 21, we have implemented the first release of the rolekit framework as well as a single representative Role: the Domain Controller. For Fedora 22, we’re working with the Cockpit project to produce a simple and powerful graphical interface to deploy the Domain Controller Role as well as building a new Database Server Role. As the project progresses, we very much hope that others will come forward to help us build more solutions. A few that I’d love to see (but don’t have time to start on yet):

  • A fileserver role that manages Samba and NFS file-shares (maybe [s]ftp as well).
  • A mail and/or groupware server role built atop something like Kolab
  • A backup server

Welcome to the post-package world, my friends!

December 10, 2014

Analysis of the CVE-2013-6435 Flaw in RPM

The RPM Package Manager (RPM) is a powerful command-line driven package management system capable of installing, uninstalling, verifying, querying, and updating software packages. RPM was originally written in 1997 by Erik Troan and Marc Ewing. Since then RPM has been successfully used in all versions of Red Hat Linux and currently in Red Hat Enterprise Linux.

RPM offers considerable advantages over traditional open-source software install methodology of building from source via tar balls, especially when it comes to software distribution and management. This has led to other Linux distributions to accept RPM as either the default package management system or offer it as an alternative to the ones which are default in those distributions.

Like any big, widely used software, over time several features are added to it and also several security flaws are found. On several occasions Red Hat has found and fixed security issues with RPM.

Florian Weimer of Red Hat Product Security discovered an interesting flaw in RPM, which was assigned CVE-2013-6435. Firstly, let’s take a brief look at the structure of an RPM file. It consists of two main parts: the RPM header and the payload. The payload is a compressed CPIO archive of binary files that are installed by the RPM utility. The RPM header, among other things, contains a cryptographic checksum of all the installed files in the CPIO archive. The header also contains a provision for a cryptographic signature. The signature works by performing a mathematical function on the header and archive section of the file. The mathematical function can be an encryption process, such as PGP (Pretty Good Privacy), or a message digest in the MD5 format.

If the RPM is signed, one can use the corresponding public key to verify the integrity and even the authenticity of the package. However, RPM only checked the header and not the payload during the installation.

When an RPM is installed, it writes the contents of the package to its target directory and then verifies its checksum against the value in the header. If the checksum does not match, that means something is wrong with the package (possibly someone has tampered with it) and the file is removed. At this point RPM refuses to install that particular package.

Though this may seem like the correct way to handle things, it has a bad consequence. Let’s assume RPM installs a file in the /etc/cron.d directory and then verifies its checksum. This offers a small race-window, in which crond can run before the checksum is found to be incorrect and the file is removed. There are several ways to prolong this window as well. So in the end we achieve arbitrary code execution as root, even though the system administrator assumes that the RPM package was never installed.

The approach Red Hat used to solve the problem is:

  • Require the size in the header to match with the size of the file in the payload. This prevents anyone from tampering with the payload, because the header is cryptographically verified. (This fix is already present in the upstream version of RPM)
  • Set restrictive permissions while a file is being unpacked from an RPM package. This will only allow root to access those file. Also, several programs, including cron, perform a check for permission sanity before running those files.

Another approach to mitigate this issue is the use of the O_TMPFILE flag. Linux kernel 3.11 and above introduced this flag, which can be passed to open(2), to simplify the creation of secure temporary files. Files opened with the O_TMPFILE flag are created, but they are not visible in the file system. As soon as they are closed, they are deleted. There are two uses for these files: race-free temporary files and creation of initially unreachable files. These unreachable files can be written to or changed same as regular files. RPM could use this approach to create a temporary, unreachable file, run a checksum on it, and either delete it or atomically link it to set the file up, without being vulnerable to the attack described above. However, as mentioned above, this feature is only available in Linux kernel 3.11 and above, was added to glibc 2.19, and is slowly making its way into GNU/Linux distributions.

The risk mentioned above is greatly reduced if the following precautions are followed:

  • Always check signatures of RPM packages before installing them. Red Hat RPMs are signed with cryptographic keys provided at https://access.redhat.com/security/team/key. When installing RPMs from Red Hat or Fedora repositories, Yum will automatically validate RPM packages via the respective public keys, unless explicitly told not to (via the “nogpgcheck” option and configuration directive).
  • Package downloads via Red Hat software repositories are protected via TLS/SSL so it is extremely difficult to tamper with them in transit. Fedora uses a whole-file hash chain rooted in a hash downloaded over TLS/SSL from a Fedora-run central server.

The above issue (CVE-2013-6435) has been fixed along with another issue (CVE-2014-8118), which is a potentially exploitable crash in the CPIO parser.

Red Hat customers should update to the latest versions of RPM via the following security advisories:

December 03, 2014

Disabling SSLv3 on the client and server

Recently, some Internet search engines announced that they would prefer websites secured with encryption over those that were not.  Of course there are other reasons why securing your website with encryption is beneficial.  Protecting authentication credentials, mitigating the use of cookies as a means of tracking and allowing access, providing privacy of your users, and authenticating your own server thus protecting the information you are trying to convey to your users.  And while setting up and using encryption on a webserver can be trivial, doing it properly might take a few additional minutes.

Red Hat strives to ship sane defaults that allow both security and availability.  Depending on your clients a more stringent or lax configuration may be desirable.  Red Hat Support provides both written documentation as well as a friendly person that can help make sense of it all.  Inevitably, it is the responsibility of the system owner to secure the systems they host.

Good cryptographic protocols

Protocols are the basis for all cryptography and provide the instructions for implementing ciphers and using certificates.  In the asymmetric, or public key, encryption world the protocols are all based off of the Secure Sockets Layer, or SSL, protocol.  SSL has come along way since its initial release in 1995.  Development has moved relatively quickly and the latest version, Transport Layer Security version 1.2 (TLS 1.2), is now the standard that all new software should be supporting.

Unfortunately some of the software found on the Internet still supports or even requires older versions of the SSL protocol.  These older protocols are showing their age and are starting to fail.  The most recent example is the POODLE vulnerability which showed how weak SSL 3.0 really is.

In response to the weakened protocol Red Hat has provided advice to disable SSL 3.0 from its products, and help its customers implement the best available cryptography.  This is seen in products from Apache httpd to Mozilla Firefox.  Because SSL 3.0 is quickly approaching its twentieth birthday it’s probably best to move on to newer and better options.

Of course the protocol can’t fix everything if you’re using bad ciphers.

Good cryptographic ciphers

Cryptographic ciphers are just as important to protect your information.  Weak ciphers, like RC4, are still used on the Internet today even though better and more efficient ciphers are available.  Unfortunately the recommendations change frequently.  What was suggested just a few months ago may no longer be good choices today.  As more work goes into researching the available ciphers weaknesses are discovered.

Fortunately there are resources available to help you stay up to date.  Mozilla provides recommended cipher choices that are updated regularly.  Broken down into three categories, system owners can determine which configuration best meets their needs.

Of course the cipher can’t fix everything if your certificate are not secure.


Certificates are what authenticate your server to your users.  If an attacker can spoof your certificate they can intercept all traffic going between your server and users.  It’s important to protect your keys and certificates once they have been generated.  Using a hardware security module (HSM) to store your certificates is a great idea.  Using a reputable certificate authority is equally important.


Most clients that support SSL/TLS encryption automatically try to negotiate the latest version.  We found with the POODLE attack that http clients, such as Firefox, could be downgraded to a weak protocol like SSL 3.0.  Because of this many server owners went ahead and disabled SSL 3.0 to prevent the downgrade attack from affecting their users.  Mozilla has, with their latest version of Firefox, disabled SSL 3.0 by default (although it can be re-enabled for legacy support).  Now users are protected even though server owners may be lax in their security (although they are still at the mercy of the server’s cipher and protocol choices).

Much of the work has already been done behind the scenes and in the development of the software that is used to serve up websites as well as consume the data that comes from these servers.  The final step is for system owners to implement the technology that is available.  While a healthy understanding of cryptography and public key infrastructure is good, it is not necessary to properly implement good cryptographic solutions.  What is important is protecting your data and that of your users.  Trust is built during every interaction and your website it usually a large part of that interaction.

December 02, 2014

Security Audit Automation Made Easy with SCAP

Security automation can be defined as the use of standardized specifications and protocols to perform specific common security functions.

Which leads us to SCAP – the Security Content Automation Protocol, an industry and government initiative to automate security audits and compliance.

The basic concept of SCAP is that security guides should be executable content, not paper documents. You should be able to define your security requirements (or security content) in a form that can be run on a computer with no human intervention, and which produces an audit report that can be understood by both computers and people. You should be able to run these reports – effectively, to perform a complete security audit on a system – as frequently as you want.

Further, these security guides should be dynamic, extensible, customizable, and actionable.

  • Dynamic – as new security threats are discovered, the threat and how to respond to the threat should be added to the security guide.
  • Extensible – you should be able to get security content from multiple sources, as well as create your own specialized security content.
  • Customizable – you should be able to choose which security rules apply to which systems. For example, a web server in a DMZ, a database server and a development system will all have different security requirements.
  • Actionable – the security guide should not only identify security issues, it should also give you assistance in resolving these security issues. Specifically, it should help you understand what the issue is, what the risk is, and what the exposure is, as well as what steps can be taken to resolve or mitigate it.

And, of course, consistent. You may recall the discussion of password rules from a few posts back. You need to apply the same security rules across Windows, Solaris, AIX, HP-UX, Linux, Mac, and all other computers you have.

For people who want to jump ahead, good resources for SCAP include:

MySQL On Fedora 20 Setup

I’ve set up MySQL enough times figuring things out from docs that I decided I need to take notes.

This is a destructive re-install. Don’t do this if you value your data. In fact, just don’t do this.

Cleanup after old installs.

sudo systemctl stop mysqld.service
sudo yum erase mysql mariadb-libs
#remove files that have the vestiges of old installs
sudo rm -rf  /var/lib/mysql/
sudo rm -rf /etc/mysql/conf.d/
sudo rm -rf /etc/my.cnf.d/
sudo rm -rf /etc/my.cnf
sudo yum install mysql-server
# run the server
sudo systemctl start mysqld.service
#create a db
sudo mysqladmin create keystone

Connect to the database as root to do the basics. Yes, this could be scripted from the command line:

To Create a user for ayoung

$ sudo mysql keystone
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 5.5.39-MariaDB-wsrep MariaDB Server, wsrep_25.10.r4014

Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [keystone]> CREATE USER 'ayoung'@'localhost' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [keystone]> GRANT ALL PRIVILEGES ON * . * TO 'ayoung'@'localhost';
Query OK, 0 rows affected (0.00 sec)

MariaDB [keystone]> 

Log in as ayoung

$ mysql keystone --password
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 14
Server version: 5.5.39-MariaDB-wsrep MariaDB Server, wsrep_25.10.r4014

Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [keystone]> 

November 24, 2014

Security Checklists and the US National Checklist Program

If you are going to perform a security audit you need a checklist.

Let’s spend a minute on this. If you want a predictable outcome, you need a standard process – a standard set of steps to go through to reach that outcome. Basic stuff. But here is the tricky part: people are bad about remembering things and doing things the same way every time. If the results are important, you need a checklist.

Rather than spending a lot of time here, I’m going to hand out a reading assignment: The Checklist Manifesto: How to Get Things Right by Atul Gawande. This is one of the books I strongly recommend everyone should read. Go ahead, I’ll wait until you come back.

OK, welcome back.

Let’s take a look at applying checklists to security. The first suggestion I will make is don’t write checklists from scratch. Find one that is close to what you need and modify it. It takes several iterations and considerable experience to develop a solid process that works – the more you can build on other peoples experience, the less work you have to do. And the better your chances of getting it right!

A good resource for checklists on computer security is the US National Checklist Program. This is a repository of publicly available security checklists to provide detailed guidance on setting the security configuration of operating systems and applications.

Let’s start out with a written checklist – how about the HPLaserJet 4345 MFP Security Checklist. This is a 49 page document detailing how to secure a printer. Yes, a printer. Modern printers are actually servers with a print engine hanging off the side. They can be a major security risk. They have an internal disk drive that stores the documents being printed. Did you securely remove classified documents from the last printer you got rid of?

The document covers threat models, network security, printer settings, and ramifications of the various settings. It includes many screenshots of how to use the Web-based management interface to access and change the many settings.

The good news is that this security guide exists. The bad news is that it is a time consuming manual process to apply it. Speaking of which – who configured your printer when it was installed six years ago? Did they do it right? What has happened in the intervening time? Did someone disable security on the printer so that they could get their job done?

It looks like it is time to print out the security guide and start pointing your browser at all the printers in your organization!

There has got to be a better way to do this. And no, ignoring security until you show up on the front page of the newspaper or in front of a congressional committee isn’t a better way!…

November 19, 2014

Availability of OpenLMI in Various Linux Distributions

A quick update on the availability of OpenLMI:

I have tested Fedora, RHEL, CentOS, and OEL servers using the LMI CLI running on a Fedora system – the cross platform access works.


Fedora is the primary development platform for OpenLMI. OpenLMI support has been included in Fedora starting with Fedora 18. We strongly recommend using Fedora 20 or the upcoming Fedora 21 release when using Fedora with OpenLMI, as these include the latest versions of OpenLMI. Fedora includes all OpenLMI capabilities: the CIMOM, all Providers, the client tools and all client scripts.

Red Hat Enterprise Linux

RHEL 7 includes the OpenLMI CIMOM and Providers. RHEL 7 includes the client side infrastructure (LMIShell and the LMI CLI). Many of the client scripts are available through the EPEL repository.


CentOS 7 includes the OpenLMI CIMOM and Providers. CentOS 7 includes the client side infrastructure (LMIShell and the LMI CLI). Many of the client scripts are available through the EPEL repository.

Oracle Enterprise Linux

OEL 7 includes the OpenLMI CIMOM and Providers.


SLES 12 includes a subset of the OpenLMI Providers. SuSE uses the sfcb CIMOM instead of the OpenPegasus CIMOM used by default in the other distributions (both sfcb and OpenPegasus ship it all of these Linux distributions).

SLES 12 includes the following OpenLMI Providers:

  • openlmi-fan
  • openlmi-hardware
  • openlmi-journald
  • openlmi-logicalfile
  • openlmi-pcp
  • openlmi-powermanagement
  • openlmi-python-base
  • openlmi-python-providers
  • openlmi-realmd
  • openlmi-service
  • openlmi-software

The SLES 12 documentation notes that “Only reading of management information is supported for the ‘openlmi’ providers.”.

SLES 12 does not include the OpenLMI storage or network Providers; thus, you can not use OpenLMI to query or configure storage or networks on a SLES 12 system.


OpenLMI support is not currently available in Debian.


OpenLMI support is not currently available in Ubuntu.

November 18, 2014

LISA’14 – Are We Making Linux Too Easy?

LISA’14, the Large Installation System Administration conference, was held in Seattle last week. I had the opportunity to give a talk on Server Management – if you are interested, the slides are available here.

One of the questions caught me completely off guard: “Aren’t you afraid that you are making system management too simple and that people won’t learn how to really manage Linux? They will just learn a few simple commands and not go any further. Today, they have to learn how Linux works and how to solve problems. OpenLMI will leave them unprepared.”

Wow… Where to start?

Thinking about this further, it could happen. In fact it will happen! Many people are looking for the quickest fix to a problem – a common way of working is to Google what you need, find something that looks like it should work, try a quick cut and paste, and move on.

OpenLMI is designed to support this. The LMI CLI is task oriented, simple, and easy to use. All you really need to use the LMI CLI is “LMI help”. The LMIShell scripts are designed to do useful work, to be easy to read, and to be modified for specific tasks.

If someone is simply looking for a way to perform a specific task, use it, and move on the the next problem, OpenLMI is a good way to go. You can use OpenLMI at a shallow level, even use it to avoid having to learn how Linux really works.

On the other hand, OpenLMI can also be used to ease into a deep knowledge of Linux: Start with the LMI CLI and use it to perform tasks. Move into LMIShell and start using and developing scripts. From there it is straightforward to develop custom automation tools. You have several ways to dive deeper into Linux administration, perhaps even developing custom OpenLMI Providers.

I would suggest that OpenLMI makes Linux more approachable. Some people will only use OpenLMI, and will never go deeper – if they can do what they need to do, this seems like a reasonable approach. Some people will use OpenLMI as a tool and and entry point to mastering Linux administration; this is great.

I don’t believe everyone needs to master Linux to use it. Consider the car analogy: All some people want to do is drive a car – automatic transmissions are perfect for them. Some people want to be able to do light repairs such as oil changes. Some want to rebuild engines and repair major subsystems of the car. And some people want to design the eight speed computer controlled automatic transmissions that are part of the integrated drive train of modern cars!

What do you think? Do we face a real risk of making Linux “too easy”, or should we try to make Linux more approachable?

Automation – a Security Imperative

So far we have established:

  • Security Guides are a good idea and exist in almost all organizations.
  • Security audits are good and widely used.
  • Security guides are often poorly written, subject to interpretation, and difficult to apply.
  • Security audits are expensive and not performed as often as they should be.

Hmmm…. Well, computers are good at following rules and measuring things. And if security guide rules are precise enough to be implemented and measured, they are very close to what you need to create a computer program.

The obvious next step is to create computer programs to implement security rules and perform computer audits!

In fact, this is what has been done for years. Numerous programs have been written for security, many security capabilities are built into operating systems, and scripts to configure systems are widely used.

However: security at the enterprise level is a big, complex undertaking.

You need a large investment in tracking threats as they emerge. It would be terribly convenient if there were a standard way to talk about threats – for example, the first 6 people who identify a new computer virus are going to call it different things, unless something is done to create a standard definition.

The vast majority of computer security issues are quickly fixed after they are identified. Decades of experience show that most computer intrusions can be prevented by applying existing patches. The question is what patches need to be applied to each specific system? This is a more complex question than it appears to be – few organizations automatically apply all patches to all systems. Instead, they test patches and carefully apply specific patches to specific systems.

The challenge is knowing which patches have been applied, which patches are available, and which patches are needed for each system. What is the risk addressed by each patch, what is the impact, and how relevant is the exposure?

Creating a useful set of security rules is a huge undertaking. If each organization is 90% common with other organizations and 10% unique, it is incredibly wasteful for each organization to build 100% of the security rules themselves.

And enterprise systems are complex. You need a workflow and extensible frameworks to be able to effectively secure, manage and monitor them.

All of these things call out for an industry wide initiative to build a standard foundation for automating security.

November 17, 2014

Dynamic Policy in Keystone

Ever get that feeling that an epiphany is right around the corner? I spent a good portion of the OpenStack summit with that feeling. I knew that it would not be earth shattering, or lead me to want to rewrite Keystone, but rather a clarification of how a bunch of things should fall together. The “click” happened on the second to last day, and it can be summarized in a few key points.

When discussing the OAUTH1.0 extension to Keystone, several people commented on how it was similar to trusts, and that we should have a unified mechanism between them for delegation. During a discussion with David Chadwick, he mentioned that the role assignments themselves were a form of delegation, and lamented that we were losing the chain of delagtion by how we delegate roles. So the first point was this:

Keystone should have a single, unified mechanism for delegation.

One key feature that feeds into that is the ability to break a big role into a small one. I had posted a spec for hierarchical roles prior to the summit, but wasn’t clear for how to implement it; I could see how it coule be implemented on the token side, but all people I talked to insisted it made more sense on the enforcement side. That is the second big point.

Role inheritance should be expanded by policy enforcement.

Policy is almost all static. Each OpenStack project had it’s own policy file in its own it repo. Extending it to cover is user requests for things like project specific policy or more granular roles has not been possible.

UPDATE: I’ve been asked to make clearer what problems this addresses.

  1. Determine what roles a user can assign to another user
  2. Allow a user to determine what roles they need to perform some action
  3. Allow some user interface to determine what a user is capable of doing based on their roles
  4. Establish an iterative process solve the long-standing bug that a user with admin on any scope has admin on all scoped.
  5. Allow a user to delegate a subset of their capabilites to a remote service.

What we have now is a simple set of specs that build on each other that will, in the end, provide a much more powerful, flexible, and consistant delegation mechanism for Keystone. Here are the General steps:

  1. Graduate oslo policy to a library
  2. Add to the policy library the essential code to enforce policy based on a keystone token.  I’ve looked at both the Keystone and Nova pieces that do this, and they are similar enough that we should not have too much problem making this happen.
  3. Add in the ability to fetch the policy.json file from Keystone.
  4. Add a rule to the Keystone policy API to return the default policy file if no policy file is specified for an endpoint.
  5. Merge the current default policy files from all of the projects into a single policy file, with namespaces that keep the rules from conflicting across services.  Reduce the duplication of rules like “admin_or_owner”  so that we have a consistent catalog of capabilities across OpenStack.  Make this merged file the default that is served out of Keystone when an endpoint asks for a policy file and Keystone does not have an endpoint specific file to give it.
  6. Make a database schema to hold the rules from the policy file.  Use this to generate the policy files served by Keystone.  There should be no functional difference between the global file and the one produced in the above merge.
  7. Use the hierarchical role definitions to generate the rules for the file above.  For example, rules that essentially say “grant access to a user with any role on this project”  will now say  “grant access to any user with the member role, or with any role that inherits the member role.  The member role will be the lowest form of access.  Admin will inherit member, as will all other defined roles.
  8. Break member up into smaller roles.  For example,  we could distinguish between actions that can only read state from those that can change it:  “Observer”  and “Editor”  Member would inherit editor, and editor would inherit observer.
  9. Change the rules for specific API policy enforcement points to know about the new roles.  For example, the API to create a new image in glance might now require the editor role instead of the member role.  But, since member inherits editor, all current users will be able to perform the same set of operations.
  10. Change the role assignment mechanism so that a user can only assign a role that they themselves have on the designated scope.  In order to assign Member, the user must have the member role, or a role that inherits Member,such as admin.  Role assignment, trusts, oauth, and any other mechanism out there will follow this limitation.  We will have to perform additional limitations, such as determining what happens to a delegated role when the person that does the delegation has that role removed;  perhaps one will need a specific role in order to perform “sticky” role assignments that last past your employment, or perhaps we will allow a user to pass some/all their delegations on to another user.


This is still in the planning stage.  One nice thing about a plan like this is that each stage shows value on its own, so that if we only get as far as, say stage 3, we still have a better system than we do today.  Many of the details are still hiding in the weeds, and will require more design.  But I think the above approach makes sense, and will make Keystone do what a lot of people need it to do.

Minimal Token Size

OpenStack Keystone tokens can become too big to fit in the headers between mod_wsgi and the WSGI applications. Compression mitigates the problem somewhat, but if token sizes continue to grow, eventually they outpace the benefits of compression. How can we keep them to a minimal size?

There are two variables to the size of the tokens: the packaging, and the data inside. The packaging for a PKIZ token has a lower bound based on the the signing algorithm. An empty CMS document of compressed data is going to be no less than 650 bytes. An unscoped token with proper compression comes in at 930 bytes. This are for headers, but it means that we have to keep additional data inside the token body as small as possible.


Lets shift gears back to the encoding. A recent proposal suggested using symmetric encryption instead of asymmetric. The idea is that a subset of data would be encrypted by Keystone, and the data would have to be sent back to Keystone to validate. What would this save us?

Lets assume for a moment that we don’t want to pay any of the overhead of the CMS message format. Instead, keystone will encrypt just the JSON and base64 the data. How much does that save us? Depends on the encryption algorithm. An empty token will be tiny: 33 bytes when encrypted like this:

openssl bf -salt -a -in cms/empty.json -out cms/empty.bf

Which, according to the openssl man page, is blowfish encrypted and base64 encoded. What about a non-trivial token? Turns out, our unscoped token is quite a bit bigger: 780 bytes for the comparable call:

openssl bf -d -k key.data -in cms/auth_token_unscoped.json -out cms/auth_token_unscoped.bf

Compared with the PKIZ format at 929 bytes, the benefit does not seem all that great.

What about for a scoped token with role data embedded in it, but no service catalog? It turns out the compression actually makes the PKIZ format more effecient: PKIZ is 917 bytes versus 1008 for the bf.


What data is in the token?

Identification. This is what you would see in an unsigned token: user id and name, domain id and possibly name.

Scope: domain and project info Roles: specific to the scope. service catalog. The sets of services and endpoints that implement those services.

It is the service catalog that is so problematic. While we have stated that you can make tokens without a service catalog, doing so is rally not going to allow the endpoints to make any sort of decisions about where to get resources.

There is a lot of redundant data in the catalog. We’ve discussed doing ID only service catalogs. That implies that each endpoint is expandable on the endpoint size: the endpoints need to be able to fetch the service catalog and then look up the endpoints by ID.

But let us think in terms of scale. If there is a service catalog with, say, 512 endpoints, we are still going to be sending tokens that are 512 * length(endpoint_id)

Can we do better? According to Jon Bently in Programming Pearls, yes we can. We can use a bitmap. No, not the image format. Here a bitmap is an array of bits, each of which, when set, indicates inclusion of the member in the set.

We need two things. One, a cached version of the service catalog on the endpoints. But now we need to put a slightly stricter constraint on it: the token must match up exactly to a version of the service catalog, and the service catalog must contain that version number. I’d take the git approach, do a sha256 hash of the service catalog document, and include that version in the token.

Second, we need to enforce ordering on the service catalog. Each endpoint must be in a repeatable location in the list. I need to be able to refer to the endpoints, not by ID, but by sequence number.

Now, what the token would contain? Two things:

The hash of the service catalog. A bitmap of the included services.

Here’s a minimal service catalog

Index | Service name | endpoint ID
 0 | Nova | N1
 1 | Glance | G1
 2 | Neutron | T1
 3 | Cinder | C1

A service catlog that had all of the endpoints would be (b for binary) b1111 or, in Hex, 0xF

A service catalog with only Nova would be b0001 or 0×1.

Just cinder would be b1000 or 0×8

A service catalog with 512 endpoints would be 512 bits in length. That would be 64 characters long, the length of a string comparable to a sha256. A comparable list of uuids would take 16384 characters, not including the JSON overhead of commas and quotes.

I’ve done a couple tests with token data in both the minimized and the endpoing_id only formats. With 30 endpoint ids, the compressed token size is 1969 bytes. Adding one ID to that increases the size to 1989. The minimized format is 1117 when built with the following data:

"minimizedServiceCatalog": { 
    "catalog_sha256": "7c7b67a0b88c271384c94ed7d93423b79584da24a712c2ece0f57c9dd2060924",
    "entrymap": "Ox2a9d590bdb724e6d888db96f846c9fd8" },

The ID only one would scale up at rougly 20 bytes per entry point, the minimized one would stay fairly fixed in length.

Are there other options? If a token without a catalog assumed that all endpoints were valid, and auth_token middleware set the environment for the request appropriately, then there is no reason to even send a catalog on over.

Project filtering of endpoints could allow for definitions of the service catalog that is a subset of the overall catalog. These subordinate service catalogs could have their own ids, and be sent over in the token. This would minimize the size of data in the token at the expense of the server; a huge number of projects, each with their own service catalog would lead to a large synchronization effort between the endpoints and the keystone server.

If a token is only allowed to work with a limited subset of the endpoints assigned to the project, then maintaining strictly small service catalogs in their current format would be acceptable. However, this would require a significant number of changes on how users and service request tokens from Keystone.

November 12, 2014

Enterprise Linux 6.5 to 6.6 risk report

Red Hat Enterprise Linux 6.6 was released the 14th of October, 2014, eleven months since the release of 6.5 in November 2013. So lets use this opportunity to take a quick look back over the vulnerabilities and security updates made in that time, specifically for Red Hat Enterprise Linux 6 Server.

Red Hat Enterprise Linux 6 is in its fourth year since release, and will receive security updates until November 30th 2020.

Errata count

The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 6 Server if you had installed 6.5, up to and including the 6.6 release, broken down by severity. It’s split into two columns, one for the packages you’d get if you did a default install, and the other if you installed every single package.

During installation there actually isn’t an option to install every package, you’d have to manually select them all, and it’s not a likely scenario. For a given installation, the number of package updates and vulnerabilities that affected you will depend on exactly what you selected during installation and which packages you have subsequently installed or removed.

Security errata 6.5 to 6.6 Red Hat Enterprise Linux 6 ServerFor a default install, from release of 6.5 up to and including 6.6, we shipped 47 advisories to address 219 vulnerabilities. 2 advisories were rated critical, 25 were important, and the remaining 20 were moderate and low.

Or, for all packages, from release of 6.5 up to and including 6.6, we shipped 116 advisories to address 399 vulnerabilities. 13 advisories were rated critical, 53 were important, and the remaining 50 were moderate and low.

You can cut down the number of security issues you need to deal with by carefully choosing the right Red Hat Enterprise Linux variant and package set when deploying a new system, and ensuring you install the latest available Update release.


Critical vulnerabilities

Vulnerabilities rated critical severity are the ones that can pose the most risk to an organisation. By definition, a critical vulnerability is one that could be exploited remotely and automatically by a worm. However we also stretch that definition to include those flaws that affect web browsers or plug-ins where a user only needs to visit a malicious (or compromised) website in order to be exploited. Most of the critical vulnerabilities we fix fall into that latter category.

The 13 critical advisories addressed 42 critical vulnerabilities across six different projects:

  • An update to php RHSA-2013:1813 (December 2013).  A memory corruption flaw was found in the way the openssl_x509_parse() function of the PHP openssl extension parsed X.509 certificates. A remote attacker could use this flaw to provide a malicious self-signed certificate or a certificate signed by a trusted authority to a PHP application using the aforementioned function, causing the application to crash or, possibly, allow the attacker to execute arbitrary code with the privileges of the
    user running the PHP interpreter.
  • An update to JavaOpenJDK
    • RHSA-2014:0026 (January 2014).  Multiple improper permission check issues were discovered in the Serviceability, Security, CORBA, JAAS, JAXP, and Networking components in OpenJDK. An untrusted Java application or applet could use these flaws to bypass certain Java sandbox restrictions.
    • RHSA-2014:0406 (April 2014).  An input validation flaw was discovered in the medialib library in the 2D component. A specially crafted image could trigger Java Virtual Machine memory corruption when processed. A remote attacker, or an untrusted Java application or applet, could possibly use this flaw to execute arbitrary code with the privileges of the user running the Java Virtual Machine.
    • RHSA-2014:0889 (July 2014).  It was discovered that the Hotspot component in OpenJDK did not properly verify bytecode from the class files. An untrusted Java application or applet could possibly use these flaws to bypass Java sandbox restrictions.
  • An update to ruby RHSA-2013:1764 (November 2014).  A buffer overflow flaw was found in the way Ruby parsed floating point numbers from their text representation. If an application using Ruby accepted untrusted input strings and converted them to floating point numbers, an attacker able to provide such input could cause the application to crash or, possibly, execute arbitrary code with the privileges of the
  • An update to nss and nspr RHSA-2014:0917 (July 2014).  A race condition was found in the way NSS verified certain certificates.  A remote attacker could use this flaw to crash an application using NSS or, possibly, execute arbitrary code with the privileges of the user running that application.
  • An update to bash (Shellshock) RHSA-2014:1293 (September 2014).  A flaw was found in the way Bash evaluated certain specially crafted environment variables. An attacker could use this flaw to override or bypass environment restrictions to execute shell commands. Certain services and applications allow remote unauthenticated attackers to provide environment variables, allowing them to exploit this issue.
  • An update to Firefox:
    • RHSA-2013:1812 (December 2013).   Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to terminate unexpectedly or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:0132 (February 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:0310 (March 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:0448 (April 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:0741 (June 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:0919 (July 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:1144 (September 2014). Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:1635 (October 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
      A flaw was found in the Alarm API, which allows applications to schedule
      actions to be run in the future. A malicious web application could use this
      flaw to bypass cross-origin restrictions.

97% of updates to correct 42 critical vulnerabilities were available via Red Hat Network either the same day or the next calendar day after the issues were public.

Previous update releases

We generally measure risk in terms of the number of vulnerabilities, but the actual effort in maintaining a Red Hat Enterprise Linux system is more related to the number of advisories we released: a single Firefox advisory may fix ten different issues of critical severity, but takes far less total effort to manage than ten separate advisories each fixing one critical PHP vulnerability.

To compare these statistics with previous update releases we need to take into account that the time between each update release is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:

Security Errata per month Red Hat Enterprise Linux 6 Server Default InstallThis data is interesting to get a feel for the risk of running Enterprise Linux 6 Server, but isn’t really useful for comparisons with other major versions, distributions, or operating systems — for example, a default install of Red Hat Enterprise Linux 6 Server does not include Firefox, but Red Hat Enterprise Linux 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

See also: 6.5, 6.4, 6.3, 6.2, and 6.1 risk reports.

November 10, 2014

System Audits – There Has to be a Better Way!

We’re now at the point where we can discuss a system audit. We have defined what an audit is, what security requirements are, and what a security guide is.

At the most basic level, a system audit involves examining a system to verify that it conforms to specifications. This includes operational specifications for the role the system is performing, verifying the integrity and configuration of the system, and compliance against the company security guide.

In many cases system audits are manual processes. A team of people, either internally or from an external company hired to do the audit, go though a written set of checklists and manually verify system settings and configuration.

These audits are time consuming, tedious, and expensive. They are also error prone…

As a result, companies may only audit a system every six months, once a year, or even every two years.

There has to be a better way!

November 06, 2014

High Level Requirements for a Security Guide

Let’s lay out some basic requirements for a security guide:

  • The security guide must exist. It must be available, updated, and maintained.
  • The security guide must incorporate relevant government and industry requirements.
  • The security guide must be actionable. If it can’t be implemented it is useless.
  • The security guide should be pro-active, describing what should be done, not what is forbidden. And, where applicable, how to do it.
  • It should be possible to verify compliance with the security requirements through a system audit.
  • The security guide should support the company mission.

This last item may strike you as a bit odd… Recall that the reason we have computer systems is to generate business value. The security guide should balance security risk against generation of business value. If a computer system can’t be used to generate business value, you might as well get rid of it. And, of course, the most secure system is one that doesn’t exist! (Just for the record, this is a joke…)

Ideally, the security guide is a tool to improve the operation of an organization, balancing protection needs against business needs, ease of use, and the threat profile a company actually faces.

October 30, 2014

Ability to remove TLS 1.0 from httpd in CentOS 6

Due to a bug in mod_ssl, the ability to remove TLS 1.0 (and only support TLS 1.1 and/or TLS 1.2) has not been available.  The fix has now made it to CentOS 6 and you can now fine-tune your cryptographic protocols with ease.

Before the fix my /etc/httpd/conf.d/ssl.conf file had this line:

SSLProtocol all -SSLv2 -SSLv3

This allows all SSL protocols except SSLv2 and SSLv3 to be used with httpd.  This isn’t a bad solution but there are a couple of sites that I’d prefer to further lock down by removing TLS 1.0 and TLS 1.2 1.1.  With the fix now in mod_ssl my settings can now look like this:

SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1

…and I’ll only support TLS 1.2 and beyond.  Of course doing this will significantly reduce the number of clients that can connect to my server.  According to SSLLabs I’m blocking all IE users before IE 11, Android before 4.4.2, Java 7, and Firefox 24.2.0 ESR.  But luckily I really don’t have a problem with any of these browsers for a couple of things I do so I’ll likely tighten up security there and leave my more public sites alone.

Update (2014-12-12)

NSS and mod_nss for httpd wasn’t discussed because it’s not in use on my systems.  it should be noted that mod_nss can be similarly configured as mod_ssl however mod_nss does not support TLS 1.2 and you’ll max out at TLS 1.1.

October 22, 2014

Configuring FreeBSD as a FreeIPA client

A recent thread on the freeipa-users mailing list highlighted one user’s experience with setting up FreeBSD as a FreeIPA client, complete with SSSD and Sudo integration. GNU+Linux systems have ipa-client-install, but the lack of an equivalent on FreeBSD means that much of the configuration must be done manually. There is a lot of room for error, and this user encountered several "gotchas" and caveats.

Services that require manual configuration include PAM, NSS, Kerberos and SSSD. Certain features may require even more services to be configured, such as sshd, for known_hosts management. Most of the steps have been outlined in a post on the FreeBSD forums.

But before one can even begin configuring all these services, SSSD, Sudo and related software and dependencies must be installed. Unfortunately, as also outlined in the forum post, non-default port options and a certain make.conf variable must be set in order to build the software such that the system can be used as a FreeIPA client. Similarly, the official binary package repositories do not provide the packages in a suitable configuration.

This post details how I built a custom binary package repository for FreeBSD and how administrators can use it to install exactly the right packages needed to operate as a FreeIPA client. Not all FreeBSD administrators will want to take this path, but those who do will not have to worry about getting the ports built correctly, and will save some time since the packages come pre-built.

Custom package repository

poudriere is a tool for creating binary package repositories compatible with FreeBSD’s next-generation pkg(8) package manager (also known as "pkgng".) The official package repositories are built using poudriere, but anyone can use it to build their own package repositories. Repositories are built in isolated jails (an OS-level virtualisation technology similar to LXC or Docker) and can build packages from a list of ports (or the entire ports tree) with customised options. A customised make.conf file can also be supplied for each jail.

Providing a custom repository with FreeIPA-compatible packages is a practical way to help people wanting to use FreeBSD with FreeIPA. It means fewer steps in preparing a system as a FreeIPA client (fewer opportunities to make mistakes), and also saves a substantial amount of time since the administrator doesn’t need to build any ports. The BSD Now podcast has a detailed poudriere tutorial; all the detail on how to use poudriere is included there, so I will just list the FreeIPA-specific configuration for the FreeIPA repository:

  • security/sudo is built with the SSSD option set
  • WANT_OPENLDAP_SASL=yes appears in the jail’s make.conf

The repository is currently being built for FreeBSD 10.0 (both amd64 and i386.) 10.1 is not far away; once it is released I will build it for 10.1 instead. If anyone out there would like it built for FreeBSD 9.3 I can do that too – just let me know!

Assuming the custom repository is available for the release and architecture of the FreeBSD system, the following script will enable the repository and install the required packages.

pkg install -y ca_root_nss
ln -s /usr/local/share/certs/ca-root-nss.crt /etc/ssl/cert.pem
mkdir -p /usr/local/etc/pkg/repos
cat >/usr/local/etc/pkg/repos/FreeIPA.conf <<"EOF"
FreeIPA: {
  url: "https://frase.id.au/pkg/${ABI}_FreeIPA",
  signature_type: "pubkey",
  pubkey: "/usr/share/keys/pkg/FreeIPA.pem",
  enabled: yes
cat >/usr/share/keys/pkg/FreeIPA.pem <<EOF
-----END PUBLIC KEY-----
pkg update
pkg install -r FreeIPA -y cyrus-sasl-gssapi sssd sudo

Once the packages are installed from the custom repository, configuration can continue as indicated in the forum post.

Future efforts

This post was concerned with package installation. This is an important but relatively small part of setting up a FreeBSD client. There is more that can be done to make it easier to integrate FreeBSD (and other non-GNU+Linux systems) with FreeIPA. I will conclude this post with some ideas along this trajectory.

Recent versions of FreeIPA include the ipa-advise tool, which explains how various legacy systems can be configured to some extent as FreeIPA clients. ipa-advise config-freebsd-nss-pam-ldapd shows advice on how to configure a FreeBSD system, but the information is out of date in many respects – it references the old binary package tools (which have now been completely removed) and has no information about SSSD. This information should be updated. I have had this task on a sticky-note for a little while now, but if someone else beats me to it, that would be no bad thing.

The latest major version of SSSD is 1.12, but the FreeBSD port is back at 1.9. The 1.9 release is a long-term maintenance (LTM) release, but any efforts to bring 1.12 to FreeBSD alongside 1.9 would undoubtedly be appreciated by the port maintainer and users.

A longer term goal should be a port of (or an equivalent to) ipa-client-install for FreeBSD. Most of the software needed for FreeIPA integration on FreeBSD is similar or identical to that used on GNU+Linux, but there are some differences. It would be a time consuming task – lots of trial runs and testing – but probably not particularly difficult.

In regards to the package repository, work is underway to add support for package flavours to the FreeBSD packaging infrastructure. When this feature is ready, a small effort should be undertaken to add a FreeIPA flavour to the ports tree, and ensure that the resultant packages are made available in the official package repository. Once this is achieved, neither manual port builds nor the custom package repository will be required –
everything needed to configure FreeBSD as a FreeIPA client will be available to all FreeBSD users by default.

October 20, 2014

Can SSL 3.0 be fixed? An analysis of the POODLE attack.

SSL and TLS are cryptographic protocols which allow users to securely communicate over the Internet. Their development history is no different from other standards on the Internet. Security flaws were found with older versions and other improvements were required as technology progressed (for example elliptic curve cryptography or ECC), which led to the creation of newer versions of the protocol.

It is easier to write newer standards, and maybe even implement them in code, than to adapt existing ones while maintaining backward compatibility. The widespread use of SSL/TLS to secure traffic on the Internet makes a uniform update difficult. This is especially true for hardware and embedded devices such as routers and consumer electronics which may receive infrequent updates from their vendors.

The fact that legacy systems and protocols need to be supported, even though more secure options are available, has lead to the inclusion of a version negotiation mechanism in SSL/TLS protocols. This mechanism allows a client and a server to communicate even if the highest SSL/TLS version they support is not identical. The client indicates the highest version it supports in its ClientHello handshake message, then the server picks the highest version supported by both the client and the server, then communicates this version back to the client in its ServerHello handshake message. The SSL/TLS protocols implement protections to prevent a man-in-the-middle (MITM) attacker from being able to tamper with handshake messages that force the use of a protocol version lower than the highest version supported by both the client and the server.

Most popular browsers implement a different out-of-band mechanism for fallback to earlier protocol versions. Some SSL/TLS implementations do not correctly handle cases when a connecting client supports a newer TLS protocol version than supported by the server, or when certain TLS extensions are used. Instead of negotiating the highest TLS version supported by the server the connection attempt may fail. As a workaround, the web browser may attempt to re-connect with certain protocol versions disabled. For example, the browser may initially connect claiming TLS 1.2 as the highest supported version, and subsequently reconnect claiming only TLS 1.1, TLS 1.0, or eventually SSL 3.0 as the highest supported version until the connection attempt succeeds. This can trivially allow a MITM attacker to cause a protocol downgrade and make the client/server use SSL 3.0. This fallback behavior is not seen in non HTTPS clients.

The issue related to the POODLE flaw is an attack against the “authenticate-then-encrypt” constructions used by block ciphers in their cipher block chaining (CBC) mode, as used in SSL and TLS. By using SSL 3.0, at most 256 connections are required to reliably decrypt one byte of ciphertext. Known flaws already affect RC4 and non block-ciphers and their use is discouraged.

Several cryptographic library vendors have issued patches which introduce the TLS Fallback Signaling Cipher Suite Value (TLS_FALLBACK_SCSV) support to their libraries. This is essentially a fallback mechanism in which clients indicate to the server that they can speak a newer SSL/TLS versions than the one they are proposing. If TLS_FALLBACK_SCSV was included in the ClientHello and the highest protocol version supported by the server is higher than the version indicated by the client, the server aborts the connection, because it means that the client is trying to fallback to a older version even though it can speak the newer version.

Before applying this fix, there are several things that need to be understood:

  • As discussed before, only web browsers perform an out-of-band protocol fallback. Not all web browsers currently support TLS_FALLBACK_SCSV in their released version. Even if the patch is applied on the server, the connection may still be unsafe if the browser is able to negotiate SSL 3.0
  • Clients which do not implement out-of-protocol TLS version downgrades (generally anything which does not speak HTTPS) do not need to be changed. Adding TLS_FALLBACK_SCSV is unnecessary (and even impossible) if there is no downgrade logic in the client application.
  • Thunderbird shares a lot of its code with the Firefox web browser, including the connection setup code for IMAPS and SMTPS. This means that Thunderbird will perform an insecure protocol downgrade, just like Firefox. However, the plaintext recovery attack described in the POODLE paper does not apply to IMAPS or SMTPS, and the web browser in Thunderbird has Javascript disabled, and is usually not used to access sites which require authentication, so the impact on Thunderbird is very limited.
  • The TLS/SSL server needs to be patched to support the SCSV extension – though, as opposed to the client, the server does not have to be rebuilt with source changes applied. Just installing an upgrade TLS library is sufficient. Due to the current lack of browser support, this server-side change does not have any positive security impact as of this writing. It only prepares for a future where a significant share of browsers implement TLS_FALLBACK_SCSV.
  • If both the server and the client are patched and one of them only supports SSL 3.0, SSL 3.0 will be used directly, which results in a connection with reduced security (compared to currently recommended practices). However, the alternative is a total connection failure or, in some situations, an unencrypted connection which does nothing to protect from an MITM attack. SSL 3.0 is still better than an unencrypted connection.
  • As a stop-gap measure against attacks based on SSL 3.0, disabling support for this aging protocol can be performed on the server and the client. Advice on disabling SSL 3.0 in various Red Hat products and components is available on the Knowledge Base.

Information about (the lack of ongoing) attacks may help with a decision. Protocol downgrades are not covert attacks, in particular in this case. It is possible to log SSL/TLS protocol versions negotiated with clients and compare these versions with expected version numbers (as derived from user profiles or the HTTP user agent header). Even after a forced downgrade to SSL 3.0, HTTPS protects against tampering. The plaintext recovery attack described in the POODLE paper (Bodo Möller, Thai Duong, Krzysztof Kotowicz, This POODLE Bites: Exploiting The SSL 3.0 Fallback, September 2014) can be detected by the server and just the number of requests generated by it could be noticeable.

Red Hat has done additional research regarding the downgrade attack in question. We have not found any clients that can be forcibly downgraded by an attacker other than clients that speak HTTPS. Due to this fact, disabling SSL 3.0 on services which are not used by HTTPS clients does not affect the level of security offered. A client that supports a higher protocol version and cannot be downgraded is not at issue as it will always use the higher protocol version.

SSL 3.0 cannot be repaired at this point because what constitutes the SSL 3.0 protocol is set in stone by its specification. However, starting in 1999, successor protocols to SSL 3.0 were developed called TLS 1.0, 1.1, and 1.2 (which is currently the most recent version). Because of the built-in protocol upgrade mechanisms, these successor protocols will be used whenever possible. In this sense, SSL 3.0 has indeed been fixed – an update to SSL 3.0 should be seen as being TLS 1.0, 1.1, and 1.2. Implementing TLS_FALLBACK_SCSV handling in servers makes sure that attackers cannot circumvent the fixes in later protocol versions.

October 15, 2014

POODLE – An SSL 3.0 Vulnerability (CVE-2014-3566)

Red Hat Product Security has been made aware of a vulnerability in the SSL 3.0 protocol, which has been assigned CVE-2014-3566. All implementations of SSL 3.0 are affected. This vulnerability allows a man-in-the-middle attacker to decrypt ciphertext using a padding oracle side-channel attack.

To mitigate this vulnerability, it is recommended that you explicitly disable SSL 3.0 in favor of TLS 1.1 or later in all affected packages.

A brief history

Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL), are cryptographic protocols designed to provide communication security over networks. The SSL protocol was originally developed by Netscape.  Version 1.0 and was never publicly released; version 2.0 was released in February 1995 but contained a number of security flaws which ultimately led to the design of SSL 3.0. Over the years, several flaws were found in the design of SSL 3.0 as well. This ultimately lead to the development and widespread use of the TLS protocol.

Most TLS implementations remain backward compatible with SSL 3.0 to incorporate legacy systems and provide a smoother user experience. Many SSL clients implement a protocol downgrade “dance” to work around the server side interoperability issues. Once the connection is downgraded to SSL 3.0, RC4 or a block cipher with CBC mode is used; this is where the problem starts!

What is POODLE?

The POODLE vulnerability has two aspects. The first aspect is a weakness in the SSL 3.0 protocol, a padding oracle. An attacker can exploit this vulnerability to recover small amounts of plaintext from an encrypted SSL 3.0 connection, by issuing crafted HTTPS requests created by client-side Javascript code, for example. Multiple HTTPS requests are required for each recovered plaintext byte, and the vulnerability allows attackers to confirm if a particular byte was guessed correctly. This vulnerability is inherent to SSL 3.0 and unavoidable in this protocol version. The fix is to upgrade to newer versions, up to TLS 1.2 if possible.

Normally, a client and a server automatically negotiate the most recent supported protocol version of SSL/TLS. The second aspect of the POODLE vulnerability concerns this negotiation mechanism. For the protocol negotiation mechanism to work, servers must gracefully deal with a more recent protocol version offered by clients. (The connection would just use the older, server-supported version in such a scenario, not benefiting from future protocol enhancements.) However, when newer TLS versions were deployed, it was discovered that some servers just terminated the connection at the TCP layer or responded with a fatal handshake error, preventing a secure connection from being established. Clearly, this server behavior is a violation of the TLS protocol, but there were concerns that this behavior would make it impossible to deploy upgraded clients and widespread interoperability failures were feared. Consequently, browsers first try a recent TLS version, and if that fails, they attempt again with older protocol versions, until they end up at SSL 3.0, which suffers from the padding-related vulnerability described above. This behavior is sometimes called the compatibility dance. It is not part of TLS implementations such as OpenSSL, NSS, or GNUTLS; it is implemented by application code in client applications such as Firefox and Thunderbird.

Both aspects of POODLE require a man in the middle attack at the network layer. The first aspect of this flaw, the SSL 3.0 vulnerability, requires that an attacker can observe the network traffic between a client and a server and somehow trigger crafted network traffic from the client. This does not strictly require active manipulation of the network transmission, passive eavesdropping is sufficient. However, the second aspect, the forced protocol downgrade, requires active manipulation of network traffic.  As described in the POODLE paper, both aspects require the attacker to be able to observe and manipulate network traffic while it is in transit.

How are modern browsers affected by the POODLE security flaw?

Browsers are particularly vulnerable because session cookies are short and an ideal target for plain text recovery, and the way HTTPS works allows an attacker to generate many guesses quickly (either through Javascript or by downloading images). Browsers are also most likely to implement the compatibility fallback.
By default, Firefox supports SSL 3.0, and performs the compatibility fallback as described above. SSL 3.0 support can be switched off, but the compatibility fallback cannot be configured separately.

Is this issue fixed?

The first aspect of POODLE, the SSL 3.0 protocol vulnerability, has already been fixed through iterative protocol improvements, leading to the current TLS version, 1.2. It is simply not possible to address this in the context of the SSL 3.0 protocol, a protocol upgrade to one of the successors is needed. Note that TLS versions before 1.1 had similar padding-related protocol issues, which is why we recommend to switch to TLS 1.1, if possible. These issues, while present, are currently not known to be exploitable in a way such as POODLE. The first aspect of the POODLE vulnerability is not exploitable with TLS 1.0. (SSL and TLS are still quite similar as protocols, the name change has non-technical reasons.)

The second aspect, caused by browsers which implement the compatibility fallback in an insecure way, has yet to be addressed. Strictly speaking, this is a security vulnerability in browsers due to the way they misuse the TLS protocol. One way to fix this issue would be to remove the compatibility dance, focusing instead on making servers compatible with clients implementing the most recent TLS implementation (as explained, the protocol supports a version negotiation mechanism, but some servers refuse to implement it).

However, there is an industry-wide effort under way to enable browsers to downgrade in a secure fashion, using a new Signaling Cipher Suite Value (SCSV). This will require updates in browsers (such as Firefox) and TLS libraries (such as OpenSSL, NSS and GNUTLS). However, we do not envision changes in TLS client applications which currently do not implement the fallback logic, and neither in TLS server applications as long as they use one of the system TLS libraries. TLS-aware packet filters, firewalls, load balancers, and other infrastructure may need upgrades as well.

Is there going to be another SSL 3.0 issue in the near future? Is there a long term solution?

Disabling SSL 3.0 will obviously prevent exposure to future SSL 3.0-specific issues. The new SCSV-based downgrade mechanism should reliably prevent the use of SSL 3.0 if both parties support a newer protocol version. Once these software updates are widely deployed, the need to disable SSL 3.0 to address this and future vulnerabilities will hopefully be greatly reduced.

SSL 3.0 is typically used in conjunction with the RC4 stream cipher. (The only other secure option in a strict, SSL 3.0-only implementation is Triple DES, which is quite slow even on modern CPUs.) RC4 is already considered very weak, and SSL 3.0 does not even apply some of the recommended countermeasures which prolonged the lifetime of RC4 in other contexts. This is another reason to deploy support for more recent TLS versions.

I have patched my SSL implementation against BEAST and LUCKY-13, am I still vulnerable?

This depends on the type of mitigation you have implemented. If you disabled protocol versions earlier than TLS 1.1 (which includes SSL 3.0), then the POODLE issue does not affect your installation. If you forced clients to use RC4, the first aspect of POODLE does not apply, but you and your users are vulnerable to all of the weaknesses in RC4. If you implemented the n/n-1 split through a software update, or if you deployed TLS 1.1 support without enforcing it, but made no other configuration changes, you are still vulnerable to the POODLE issue.

Is it possible to monitor for exploit attempts?

The protocol downgrade is visible on the server side. Usually, servers can log TLS protocol versions. This information can then be compared with user agents or other information from the profile of a logged-in user, and mismatches could indicate attack attempts.

Attempts to abuse the SSL 3.0 padding oracle part of POODLE, as described in the paper, are visible to the server as well. They result in a fair number of HTTPS requests which follow a pattern not expected during the normal course of execution of a web application. However, it cannot be ruled out that a more sophisticated adaptive chosen plain text attack avoids confirmation of guesses from the server, and this more advanced attack would not be visible to the server, only to the client and the network up to the point at which the attacker performs their traffic manipulation.

What happens when i disable SSL 3.0 on my web server?

Some old browsers may not be able to support a secure connection to your site. Estimates of the number of such browsers in active use vary and depend on the target audience of a web site. SSL protocol version logging (see above) can be used to estimate the impact of disabling SSL 3.0 because it will be used only if no TLS version is available in the client.

Major browser vendors including Mozilla and Google have announced that they are to deactivate the SSL 3.0 in their upcoming versions.

How do I secure my Red Hat-supported software?

Red Hat has put together several articles regarding the removal of SSL 3.0 from its products.  Customers should review the recommendations and test changes before making them live in production systems.  As always, Red Hat Support is available to answer any questions you may have.

October 14, 2014

Who can sign for what?

In my last post, I discussed how to extract the signing information out of a token. But just because the signature on a document is valid does not mean that the user who signed it was authorized to do so. How can we got from a signature to validating a token? Can we use that same mechanism to sign other OpenStack messages?

The following is a proposed extension to Keystone client based on existing mechanisms.


  1. Extract signer data out of the certificates
  2. Fetch the compete list of certificate from Keystone using the OS-SIMPLE-CERT extension
  3. Match the signer to the cert to validate the signature and extract the domain data for the token
  4. Fetch the mapping info from the Federation extension
  5. Use the mapping info to convert from the signing cert to a keystone user and groups
  6. Fetch the effective roles from Keystone for the user/groups for that domain
  7. Fetch policy from Keystone
  8. Execute the policy check to validate that the signer could sign for the data.

We need a method to go from the certificate used to sign the document to a valid Keystone user. Implied in there is that everything signed in an OpenStack system is going to be signed by a Keystone user. This is an expansion on how things were done in the past, but there is a pretty solid basis for this approach: in Kerberos, everything is a Principal, whether user or system.

From Tokens to Certs

The Token has the CMS Signer Info.  We can extract that information as previously shown.

The OS-SIMPLE-CERT extension has an API for fetching all of the signing certs as once:

This might not scale greatly, it is sufficient for supporting a proof-of-concept.  It reduces the problem of “how to find the cert for this token”  down to a match between the signing info and the attributes of the certificates.

To extract the data from the certificates, We can Popen the OpenSSL command to validate a certificate.  This is proper security practice anyway, as, while we trust the authoritative Keystone, we should verify whenever possible.  It will be expensive, but this result can be cached and reused, so it should not have to happen very often.

From Certs to Users

To translate from a certificate to a user, we need to first parse the data out of the certificate. This is possible doing a call to OpenSSL. We can be especially efficient by using that call to validate the certificate itself, and then converting the response to a dictionary. Keystone already has a tool to convert a dictionary to the Identity objects (user and groups): the mapping mechanism in the Federation backend. Since a mapping is in a file we can fetch, we do not need to be inside the Keystone server to process the mapping, we just need to use the same mechanism.


The OS-FEDERATION extension has an API to List all mappings.

And another to get each mapping.

Again, this will be expensive, but it can be cached now, and optimized in the future.

The same process that uses the mappings to translate the env-vars for an X509 certificate  to a user inside the Keystone server can be performed externally. This means extracting code from the Federation plugin of the Keystone server to python-keystoneclient.

From User to Roles

Once we have the users and groups, we need to get the Role data appropriate to the token. This means validating the token, and extracting out the domain for the project. Then we will use the Identity API to list effective role assignments

We’ll probably have to call this once for the user ID and then once for each of the groups from the mapping in order to get the full set of roles.

From Roles to Permission

Now, how do we determine if the user was capable of signing for the specified token? We need a policy file. Which one? The one abstraction we currently have is that a policy file can be associated with an endpoint. Since keystone is responsible for controlling the signing of tokens, the logical endpoint is the authoritative keystone server where we are getting the certificates etc:

We get the effective policy associated with the keystone endpoint using the policy API.

And now we can run the users RBAC through the policy engine to see if they can sign for the given token.  The policy engine is part of oslo common.  There is some “flattening” code from the Keystone server we will want to pull over.  But of these will again land in python-keystoneclient.


This is a lot of communication with Keystone, but it should not have to be done very often: once each of these API calls have been made, the response can be cached for a reasonable amount of time. For example, a caching rule could say that all data is current for a minimum of 5 minutes. After that time, if a newly submitted token has an unknown signer info, the client could refetch the certificates. The notification mechanism from Keystone could also be extended to invalidate the cache of remote clients that register for such notifications.

For validating tokens in remote endpoints, the process will be split between python-keystoneclient and keystonemiddleware.  The Middleware piece will be responsible for cache management and maintaining the state of the validation between calls.  Keystone Client will expose each of the steps with a parameter that allows the cached state to be passed in, as well as already exposing the remote API wrapping functions.

At this state, it looks like no changes will have to be made to the Keystone server itself.  This shows the power of the existing abstractions.  In the future, some of the calls may need optimization.   Of example, the fetch for certificates may need to be broken down into a call that fetches an individual certificate by its signing info.

October 10, 2014

Who Signed that Token?

The specification For multiple signers requires a mechanism to determine who signed the token and then determine I’d the signer had the authority to issue a token for the scope of the token.  These are the steps he he necessary to perform that validation.

The CMS document is signed by a certificate, but, due to size constraints, the certificate has been stripped out of the token.  All that remains in the token is the ‘signer info’ section of the CMS document, as defined here http://tools.ietf.org/html/rfc5652#page-13 as

SignerIdentifier ::= CHOICE {
issuerAndSerialNumber IssuerAndSerialNumber,
subjectKeyIdentifier [0] SubjectKeyIdentifier }

IssuerAndSerialNumber is defined as:

The IssuerAndSerialNumber type identifies a certificate, and thereby
an entity and a public key, by the distinguished name of the
certificate issuer and an issuer-specific certificate serial number.

The definition of Name is taken from X.501 [X.501-88], and the
definition of CertificateSerialNumber is taken from X.509 [X.509-97].

IssuerAndSerialNumber ::= SEQUENCE {
issuer Name,
serialNumber CertificateSerialNumber }

CertificateSerialNumber ::= INTEGER


SubjectKeyIdentifier is a more general approach which allows the user to identify if a specific certificate matches (it has the Subject Key Identifier in it as well) but does not provide a way to identify who signed it, or where to fetch the corresponding certificate.

With the Keystone server acting as the system of reference, we know where to fetch the certificate. All signing certificates can be fetched using the OS-SIMPLE-CERT extension. So, regardless of which form the signing info takes, we could determine which certificate to use in order to verify the token.

To date, only a single certificate is used to sign tokens. Identifying the signer of the token is the first step to expanding that in the future.

What does the current OpenSSL binary call produce:  I took the DER form of one of the sample tokens and passed it through /usr/lib64/nss/unsupported-tools/derdump  (one of the most useful of tools)  and saw:

C-Sequence  (816)
   Object Identifier  (9)
      1 2 840 113549 1 7 2 (PKCS #7 Signed Data)
   C-[0]  (801)
      C-Sequence  (797)
         Integer  (1)
         C-Set  (13)
            C-Sequence  (11)
               Object Identifier  (9)
                  2 16 840 1 101 3 4 2 1 (SHA-256)
         C-Sequence  (309)
            Object Identifier  (9)
               1 2 840 113549 1 7 1 (PKCS #7 Data)
            C-[0]  (294)
               Octet String  (290)
                  7b 22 61 63 63 65 73 73 22 3a 20 7b 22 74 6f 6b 65 6e 
                  22 3a 20 7b 22 65 78 70 69 72 65 73 22 3a 20 22 32 31 
                  31 32 2d 30 38 2d 31 37 54 31 35 3a 33 35 3a 33 34 5a 
                  22 2c 20 22 69 64 22 3a 20 22 30 31 65 30 33 32 63 39 
                  39 36 65 66 34 34 30 36 62 31 34 34 33 33 35 39 31 35 
                  61 34 31 65 37 39 22 7d 2c 20 22 73 65 72 76 69 63 65 
                  43 61 74 61 6c 6f 67 22 3a 20 7b 7d 2c 20 22 75 73 65 
                  72 22 3a 20 7b 22 75 73 65 72 6e 61 6d 65 22 3a 20 22 
                  75 73 65 72 5f 6e 61 6d 65 31 22 2c 20 22 72 6f 6c 65 
                  73 5f 6c 69 6e 6b 73 22 3a 20 5b 5d 2c 20 22 69 64 22 
                  3a 20 22 63 39 63 38 39 65 33 62 65 33 65 65 34 35 33 
                  66 62 66 30 30 63 37 39 36 36 66 36 64 33 66 62 64 22 
                  2c 20 22 72 6f 6c 65 73 22 3a 20 5b 7b 22 6e 61 6d 65 
                  22 3a 20 22 72 6f 6c 65 31 22 7d 2c 20 7b 22 6e 61 6d 
                  65 22 3a 20 22 72 6f 6c 65 32 22 7d 5d 2c 20 22 6e 61 
                  6d 65 22 3a 20 22 75 73 65 72 5f 6e 61 6d 65 31 22 7d 
                  7d 7d 
         C-Set  (462)
            C-Sequence  (458)
               Integer  (1)
               C-Sequence  (164)
                  C-Sequence  (158)
                     C-Set  (10)
                        C-Sequence  (8)
                           Object Identifier  (3)
                              2 5 4 5 (X520 Serial Number)
                           Printable String  (1)
                     C-Set  (11)
                        C-Sequence  (9)
                           Object Identifier  (3)
                              2 5 4 6 (X520 Country Name)
                           Printable String  (2)
                     C-Set  (11)
                        C-Sequence  (9)
                           Object Identifier  (3)
                              2 5 4 8 (X520 State Or Province Name)
                           Printable String  (2)
                     C-Set  (18)
                        C-Sequence  (16)
                           Object Identifier  (3)
                              2 5 4 7 (X520 Locality Name)
                           Printable String  (9)
                     C-Set  (18)
                        C-Sequence  (16)
                           Object Identifier  (3)
                              2 5 4 10 (X520 Organization Name)
                           Printable String  (9)
                     C-Set  (17)
                        C-Sequence  (15)
                           Object Identifier  (3)
                              2 5 4 11 (X520 Organizational Unit Name)
                           Printable String  (8)
                     C-Set  (37)
                        C-Sequence  (35)
                           Object Identifier  (9)
                              1 2 840 113549 1 9 1 (PKCS #9 Email Address)
                           IA5 String  (22)
                     C-Set  (20)
                        C-Sequence  (18)
                           Object Identifier  (3)
                              2 5 4 3 (X520 Common Name)
                           Printable String  (11)
                              "Self Signed"
                  Integer  (1)
               C-Sequence  (11)
                  Object Identifier  (9)
                     2 16 840 1 101 3 4 2 1 (SHA-256)
               C-Sequence  (13)
                  Object Identifier  (9)
                     1 2 840 113549 1 1 1 (PKCS #1 RSA Encryption)
                  NULL  (0)
               Octet String  (256)
                  6e 93 08 58 52 dd 52 db 65 b9 aa 9b f5 87 37 bc 56 f2 
                  b5 25 05 a5 9b 37 68 cc 9e 2e f3 80 49 e2 58 d8 70 01 
                  35 0a 7b 66 c7 15 2c 65 6b b3 15 31 e6 8b 8e 27 eb 12 
                  d5 70 cd 71 b1 ae 68 fe b6 cf a6 b5 d7 a3 a6 84 d9 0d 
                  52 d9 e6 cd 38 fa b9 7e c5 09 63 76 99 14 3a f6 5b 71 
                  9c b7 90 9b 36 64 b5 f3 77 e6 5e ca e1 06 d1 bb a9 fc 
                  39 4c a5 e4 b2 0b 86 ae 46 d7 40 67 9d 82 38 3c 4e 69 
                  ee 00 d0 0d a8 d4 38 f9 8d a3 96 36 4d ed 18 6a 2f c4 
                  09 9f 13 9e 71 b4 31 5a f9 24 57 80 52 a2 dc 69 6e e4 
                  76 96 1b ef ae 2a cb 2d f7 fe 6d d9 6e db 3d e2 03 d1 
                  00 00 8d 8e 2c 13 49 bf 0a 10 09 74 c0 d9 25 2f 7d 1e 
                  8e f2 f0 ff 79 a4 ce 45 a0 4d a5 8d 4c c5 18 44 66 8a 
                  90 a5 55 c8 6d a9 53 1c a6 d0 47 c1 26 40 45 9f 05 91 
                  41 00 ad e0 03 6a 15 8b fc d4 7c c4 d1 26 34 0d a2 9b 
                  a7 e6 95 c7 

Th field X520 Serial Number Shows it is serial number 5 from the CA. The CA is then identified by its X500 Attributes such as

X520 Common Name

Which is “Self Signed”. The following Python code will dump the signing info for the PEM version of a token. Its basically a stripped down version of this code.

import argparse
import base64
import errno
import hashlib
import logging
import zlib

from pyasn1_modules import rfc2315, rfc2459, pem
from pyasn1.codec.der import encoder, decoder

from pyasn1.type import univ, char
from pyasn1.codec.der import decoder as der_decoder

parser = argparse.ArgumentParser()
parser.add_argument('infile', metavar='infile', help='token file to decode')
args = parser.parse_args()

in_file = open(args.infile, 'r')

idx, substrate = pem.readPemBlocksFromFile(
    in_file, ('-----BEGIN CMS-----', '-----END CMS-----'))

contentInfo, rest = der_decoder.decode(substrate, 

contentType = contentInfo.getComponentByName('contentType')
contentInfoMap = {
    (1, 2, 840, 113549, 1, 7, 1): rfc2315.Data(),
    (1, 2, 840, 113549, 1, 7, 2): rfc2315.SignedData(),
    (1, 2, 840, 113549, 1, 7, 3): rfc2315.EnvelopedData(),
    (1, 2, 840, 113549, 1, 7, 4): rfc2315.SignedAndEnvelopedData(),
    (1, 2, 840, 113549, 1, 7, 5): rfc2315.DigestedData(),
    (1, 2, 840, 113549, 1, 7, 6): rfc2315.EncryptedData()

content, _ = decoder.decode(
signerInfos = content.getComponentByName('signerInfos')[0]
issuer_and_sn = signerInfos.getComponentByName('issuerAndSerialNumber')
issuer =  issuer_and_sn.getComponentByName('issuer')
for n in issuer[0]:
    print n[0][0]
    print n[0][1]

It does something wrong with the text fields, I suspect because they are some form of Not–quite-text that needs to be parsed. I don’t think I need to take this farther, however. INstead, I plan on following an example from the OCSP generating sample code that hashes the issuer and compares the hash.

UPDATE: Thanks to gsilvis who pointed out a spurious import in the code sample.

October 09, 2014

Automated configuration analysis for Mozilla’s TLS guidelines

My friend Hubert has been doing a lot of work to make better the world a little safer.  Glad he’s getting some recognition.  Here’s a great article on testing your server for proper SSL/TLS configurations.

Ansible Hostgroups from FreeIPA

Ansible provides management for a large array of servers using ssh as the access mechanism. This is a good match for  FreeIPA.  However, by default Ansible uses a flat file to store groups of hosts.  How can we get that info from FreeIPA?


If you want to run the `uptime` command on all web servers, you would define a fragment of /etc/ansible/hosts  like this:



And then run

ansible webservers -a uptime

In order to get ansible to use a different scheme, use a dynamic inventory.  I wrote a proof of concept one  that uses the hostgroup definitions from my IPA server to populate a json file.  The format of the file is specified in this tutorial:

My Sample ignores the command line parameters, and just returns the whole set of hostgroups.

#Apache License...


import json
from ipalib import api
inventory = {}
result =api.Command.hostgroup_find()['result']
for hostgroup in result:
    inventory[hostgroup['cn'][0]] = { 'hosts': [host for host in hostgroup['member_host']]}
    for host in hostgroup['member_host']:
        hostvars[host] = {}
inventory['_meta'] = {'hostvars': hostvars}
inv_string = json.dumps( inventory)
print inv_string

I copied it to /etc/ansible/freeipa.py and ran:


$ ansible -i /etc/ansible/freeipa.py packstacked -a uptime
ayoungf20packstack.cloudlab.freeipa.org | success | rc=0 >>
20:42:33 up 141 days, 20:43, 2 users, load average: 0.22, 0.15, 0.14

multidom.cloudlab.freeipa.org | success | rc=0 >>
20:42:34 up 52 days, 3:17, 1 user, load average: 0.01, 0.03, 0.05

horizon.cloudlab.freeipa.org | success | rc=0 >>
20:42:35 up 51 days, 6:07, 2 users, load average: 0.00, 0.03, 0.05

As I said, this was a proof of concept. It does not do everything that you might want to have an inventory do. I plan on fleshing it out and submitting to the Ansible plugin repo. Meanwhile, you can look at the other examples.

If you are curious, here is the output from when I run my plugin:

$ python freeipa.py | python -mjson.tool
    "_meta": {
        "hostvars": {
            "ayoungf20packstack.cloudlab.freeipa.org": {},
            "horizon.cloudlab.freeipa.org": {},
            "ipa.cloudlab.freeipa.org": {},
            "jboss.cloudlab.freeipa.org": {},
            "multidom.cloudlab.freeipa.org": {}
    "keystone-ha-cluster": {
        "hosts": [
    "packstacked": {
        "hosts": [

October 08, 2014

The Source of Vulnerabilities, How Red Hat finds out about vulnerabilities.

Red Hat Product Security track lots of data about every vulnerability affecting every Red Hat product. We make all this data available on our Measurement page and from time to time write various blog posts and reports about interesting metrics or trends.

One metric we’ve not written about since 2009 is the source of the vulnerabilities we fix. We want to answer the question of how did Red Hat Product Security first hear about each vulnerability?

Every vulnerability that affects a Red Hat product is given a master tracking bug in Red Hat bugzilla. This bug contains a whiteboard field with a comma separated list of metadata including the dates we found out about the issue, and the source. You can get a file containing all this information already gathered for every CVE. A few months ago we updated our ‘daysofrisk’ command line tool to parse the source information allowing anyone to quickly create reports like this one.

So let’s take a look at some example views of recent data: every vulnerability fixed in every Red Hat product in the 12 months up to 30th August 2014 (a total of 1012 vulnerabilities).

Firstly a chart just giving the breakdown of how we first found out about each issue: Sources of issues

  • CERT: Issues reported to us from a national cert like CERT/CC or CPNI, generally in advance of public disclosure
  • Individual: Issues reported to Red Hat Product Security directly by a customer or researcher, generally in advance of public disclosure
  • Red Hat: Issues found by Red Hat employees
  • Relationship: Issues reported to us by upstream projects, generally in advance of public disclosure
  • Peer vendors: Issues reported to us by other OS distributions, through relationships
    or a shared private forum
  • Internet: For issues not disclosed in advance we monitor a number of mailing lists and security web pages of upstream projects
  • CVE: If we’ve not found out about an issue any other way, we can catch it from the list of public assigned CVE names from Mitre

Next a breakdown of if we knew about the issue in advance. For the purposes of our reports we count knowing the same day of an issue as not knowing in advance, even though we might have had a few hours notice: Known in advanceThere are few interesting observations from this data:

  • Red Hat employees find a lot of vulnerabilities. We don’t just sit back and wait for others to find flaws for us to fix, we actively look for issues ourselves and these are found by engineering, quality assurance, as well as our security teams. 17% of all the issues we fixed in the year were found by Red Hat employees. The issues we find are shared back in advance where possible to upstream and other peer vendors (generally via the ‘distros’ shared private forum).
  • Relationships matter. When you are fixing vulnerabilities in third party software, having a relationship with the upstream makes a big difference. But
    it’s really important to note here that this should never be a one-way street, if an upstream is willing to give Red Hat information about flaws in advance,
    then we need to be willing to add value to that notification by sanity checking the draft advisory, checking the patches, and feeding back the
    results from our quality testing. A recent good example of this is the OpenSSL CCS Injection flaw; our relationship with OpenSSL gave us advance
    notice of the issue and we found a mistake in the advisory as well as a mistake in the patch which otherwise would have caused OpenSSL to have to have
    done a secondary fix after release. Only two of the dozens of companies prenotified about those OpenSSL issues actually added value back to OpenSSL.
  • Red Hat can influence the way this metric looks; without a dedicated security team a vendor could just watch what another vendor does and copy them,
    or rely on public feeds such as the list of assigned CVE names from Mitre. We can make the choice to invest to find more issues and build upstream relationships.

October 07, 2014

Yellow Sticky of Doom in the Cloud

The password managers we discussed in the last post are a good start. If you only use one system a local password database is all you need.

Most people have multiple “devices” – a PC, a laptop, a smartphone, a tablet, and the number keeps growing. It would be terribly convenient to have access to your passwords on all of your devices, and to have everything automatically updated when you add or change a password.

This is where network – or today CLOUD BASED (highlighted for dramatic emphasis…) – password managers come into play. These networked password managers share, distribute, backup, and replicate your passwords.

Putting your passwords IN THE CLOUD should make you nervous. It is important to do your homework before choosing one – don’t just choose the first one that comes up on a search!

There are several places to look. Wikipedia has a List of Password Managers. Information Week has an article on 10 Top Password Managers. Network World published Best tools for protecting passwords. Mac World produced Mac password managers. At a minimum make sure that the password managers you are considering have at least some public review and feedback. You should also do web searches looking for user experience and any issues with the various password managers.

For cloud based password managers, one of the most important things is to make sure that you retain control of the passwords. This is done by encrypting the password data locally, on your system, and only sending encrypted data to the cloud. Done properly, the master encryption password for the password database never leaves your system – no one, including the company hosting your password manager, can decrypt your password. Of course this also means that if you lose your password manager password you are out of luck; no one can recover it.

As an anecdote, not a recommendation, a thoroughly paranoid colleague who works in the security space and whose opinions I respect recommends LastPass.  I prefer open source password managers that can be audited, like KeePassX, but there don’t seem to be any with good Cloud integration.

October 01, 2014

Wherein our hero attempts to build his own OpenStack Keystone RPMs

I have a Devstack setup. I’ve hacked the Keystone repo to add some cool feature.  I want to test it out with an RDO deployment.  How do I make my own RPM for the RDO system?

This is not a how to. This is more like a police log.

sudo mkdir fedora
sudo chown ayoung:ayoung fedora
cd fedora/
git clone http://pkgs.fedoraproject.org/cgit/openstack-keystone.git

What did that get us?

$ ls openstack-keystone

When I build, I do so in /home/ayoung/rpmbuild. How:

$ cat ~/.rpmmacros
%_topdir /home/ayoung/rpmbuild
%packager Adam Young
%dist .f20_ayoung

So, first move these files into %_topdir/SOURCES

 cp * ~/rpmbuild/SOURCES/

Technically you don’t need to move the .spec file there, but it hurts nothing to do so.

Ok, lets make a first attempt at building:

$ rpmbuild -bp openstack-keystone.spec 
error: File /home/ayoung/rpmbuild/SOURCES/keystone-2014.2.b3.tar.gz

So it is looking for a tarball named keystone-2014.2.b3.tar.gz. Where does it get that name:

In openstack-keystone.spec:

%global release_name juno
%global milestone 3

%global with_doc %{!?_without_doc:1}%{?_without_doc:0}

Name:           openstack-keystone
Version:        2014.2
Release:        0.4.b%{milestone}%{?dist}
Summary:        OpenStack Identity Service
Source0:        http://launchpad.net/keystone/%{release_name}/%{release_name}-%{milestone}/+download/keystone-%{version}.b%{milestone}.tar.gz

Now, lets assume that I already have that version installed with RDO, and I want to test my change on top of it. If I make a mile > 3 it will update. I could do this by changing the spec file.

-%global milestone 3
+%global milestone 3a
would give me :
error: File /home/ayoung/rpmbuild/SOURCES/keystone-2014.2.b3a.tar.gz: No such file or directory

But can I do that from the command line? Sure. Use -D, and put hta macro in quotes:

$ rpmbuild -D "milestone 3b"  -bp openstack-keystone.spec 
error: File /home/ayoung/rpmbuild/SOURCES/keystone-2014.2.b3.tar.gz: No such file or directory

Ok, so now I need a tarball that looks like the file name I will use in the rpmbuild process. Use git-archive to produce it.

cd /opt/stack/keystone
git archive -o /home/ayoung/rpmbuild/SOURCES/keystone-2014.2.b3.tar.gz  HEAD
cd /opt/fedora/openstack-keystone/
 rpmbuild -bp openstack-keystone.spec
error: Failed build dependencies:
	python-pbr is needed by openstack-keystone-2014.2-0.4.b3.f20_ayoung.noarch

Alright! Closer. There is a Yum utility to help out with this last problem. TO get the utilities:

sudo yum install yum-utils

then to get the build dependencies:

sudo yum-builddep openstack-keystone.spec

The next time running an rpmbuild gets the error:

+ cd keystone-2014.2.b3
/var/tmp/rpm-tmp.NSt5Oy: line 37: cd: keystone-2014.2.b3: No such file or directory

Due to the fact that the archive command above is not putting the code in a subdir. There is a flag for that: –prefix

cd /opt/stack/keystone
git archive  --prefix=keystone-2014.2.b3/   -o /home/ayoung/rpmbuild/SOURCES/keystone-2014.2.b3.tar.gz  HEAD
cd /opt/fedora/openstack-keystone
rpmbuild -bp openstack-keystone.spec
+ echo 'Patch #1 (0001-remove-runtime-dep-on-python-pbr.patch):'
Patch #1 (0001-remove-runtime-dep-on-python-pbr.patch):
+ /usr/bin/cat /home/ayoung/rpmbuild/SOURCES/0001-remove-runtime-dep-on-python-pbr.patch
+ /usr/bin/patch -p1 --fuzz=0
patching file bin/keystone-all

Now I get an error on a patch application: this is better, as it implies I am packing the tarball correctly. That change is basically removing pbr from the keystone-all startup file. PBR does versioning type stuff that competes with RPM.

Let’s ignore that patch for now: Keystone will run just fine with PBR in. In the spec file I comment out:

%patch0001 -p1

Code in the spec file is doing something comparable, and fails out due to me skipping first patch.

+ sed -i s/REDHATKEYSTONEVERSION/2014.2/ bin/keystone-all keystone/cli.py
+ sed -i s/2014.2.b3/2014.2/ PKG-INFO

Skip that code for now, too.

--- a/openstack-keystone.spec
+++ b/openstack-keystone.spec
@@ -117,7 +117,7 @@ This package contains documentation for Keystone.
 %setup -q -n keystone-%{version}.b%{milestone}
-%patch0001 -p1
+#%patch0001 -p1
 %patch0002 -p1
 find . \( -name .gitignore -o -name .placeholder \) -delete
@@ -127,9 +127,9 @@ rm -rf keystone.egg-info
 # Let RPM handle the dependencies
 rm -f test-requirements.txt requirements.txt
 # Remove dependency on pbr and set version as per rpm
-sed -i s/REDHATKEYSTONEVERSION/%{version}/ bin/keystone-all keystone/cli.py
+#sed -i s/REDHATKEYSTONEVERSION/%{version}/ bin/keystone-all keystone/cli.py
-sed -i 's/%{version}.b%{milestone}/%{version}/' PKG-INFO
+#sed -i 's/%{version}.b%{milestone}/%{version}/' PKG-INFO

And the build works. Thus far, I’ve been running -bp which just preps the source tree in the repo. Lets see what the full rpmbuild process returns:

+ cp etc/keystone.conf.sample etc/keystone.conf
+ /usr/bin/python setup.py build
error in setup command: Error parsing /home/ayoung/rpmbuild/BUILD/keystone-2014.2.b3/setup.cfg:Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. Are you sure that git is installed?

There is that python build reasonableness change. OK, what is that doing? Lets start with the last line we commented out in the spec file:

+#sed -i ‘s/%{version}.b%{milestone}/%{version}/’ PKG-INF

This is putting the RPM version of the python library into the Python EGG info. In my repo I have



Version: 2014.2.dev154.g1af2428

I wonder if I can make the RPM version look like that? It won’t be an rpm -U to go from my RDO deployment if I do, but I can work around that with –oldpackage.

What is dev154.g1af2428? I suspect some git magic for the last part. Git log shows:

commit 1af24284bdc093dae4f027ade2ddb29656b676f0

So g1af2428? is g + githash[0:6] but what about dev154? It turns out it is the number of commits since the last tag.

At this point, I resorted to IRC, and then reading the code. The short of it is that I am submitting a patch to the keystone spec file to remove all of the PBR related changes. Instead, when python setup.py is called, we’ll use the environment variable to tell PBR the version number. For example

PBR_VERSION=%{version}.%{milestone}  %{__python} setup.py build

September 30, 2014

LDAP persistent searches with ldapjdk

As part of the LDAP-based profiles feature I’ve been working on for the Dogtag, it was necessary to implement a feature where the database is monitored for changes to the LDAP profiles. For example, when a profile is updated on a clone, that change is replicated to other clones, and those other clones have to detect that change and each instance must update its view of the profiles accordingly. This post details how the LDAP persistent search feature was used to implement this behaviour.

A naïve approach to solving this problem would have been to unconditionally refresh all profiles at a certain interval. Slightly better would be to check all profiles at a certain interval and update those that have changed. Both of these methods involve some non-trivial delay between changes being replicated to the local database, and the profile subsystem reflecting those changes.

A different approach was to use the LDAP persistent search capability. With this feature, once the search is running, the client receives immediate notification of changes. This advantage commended it over the polling approach as a more appropriate basis for a solution.

ldapjdk persistent search API

A big part of the motivation for this post was the paucity of the ldapjdk documentation with respect to persistent searches. The necessary information is all there – but it is scattered across several classes, all of which play some important part in a working implementation, but none of which tells the full story.

Hopefully some people will benefit from this information being brought together in one place and explained step by step. Let’s look at the classes involved one by one as we build up the solution.


This is the server control that activates the persistent search behaviour. It also provides static flags for specifying what kinds of updates to listen for. Its constructor takes a union of these flags and three boolean values:


Whether to return existing entries that match the search criteria. For our use case, we are only interested in changes.


Whether to return entry change controls with each search result. These controls are required if you need to know what kind of change occured (add, modify, delete or modified DN).


Whether this control is critical to the search operation.

The LDAPPersistSearchControl object used for our persistent search is constructed in the following way:

int op = LDAPPersistSearchControl.ADD
    | LDAPPersistSearchControl.MODIFY
    | LDAPPersistSearchControl.DELETE
    | LDAPPersistSearchControl.MODDN;
LDAPPersistSearchControl persistCtrl =
    new LDAPPersistSearchControl(op, true, true, true);


The LDAPSearchConstraints object sets various controls and parameters for an LDAP search, persistent or otherwise. In our case, we need to attach the LDAPPersistSearchControl to the constraints, as well as disable the timeout of the search, and set the results batch size to 1 so that no buffering of results will occur at the server:

LDAPSearchConstraints cons = conn.getSearchConstraints();
cons.setServerTimeLimit(0 /* seconds */);


Executing the search method of an LDAPConnection (here named conn), yields an LDAPSearchResults object. This is the same whether or the search was a persistent search according to the LDAPSearchConstraints. The different between persistent and non-persistent searches is in how results are retrieved from the results object: if the search is persistent, the hasMoreElement method will block until the next result is received from the server (or the search times out, the connection dies, et cetera).

Let’s see what it looks like to actually execute the persistent search and process its results:

LDAPConnection conn = ... /* an open LDAPConnection */

LDAPSearchResults results = conn.search(
    "ou=certificateProfiles,ou=ca," + basedn, /* search DN */
    LDAPConnection.SCOPE_ONE, /* search at one level below DN */
    "(objectclass=*)",        /* search filter */
    null,   /* list of attributes we care about */
    false,  /* whether to only include specified attributes */
    cons    /* LDAPSearchConstraints defined above */
while (results.hasMoreElements()) /* blocks */ {
    LDAPEntry entry = results.next();
    /* ... process result ... */

We see that apart from the use of the LDAPSearchConstraints to specify a persistent search and the blocking behaviour of LDAPSearchResults.hasMoreElements, performing a persistent search is the same as performing a regular search.

Let us next examine what happens inside that while loop.


Do you recall the returnControls parameter for LDAPPersistSearchControl? If true, it ensures that each entry returned by the persistent search is accompanied by a control that indicates the type of change that affected the entry. We need to know this information so that we can update the profile subsystem in the appropriate way (was this profile added, updated, or deleted?)

Let’s look at how we do this. We are inside the while loop from above, starting exactly where we left off:

LDAPEntry entry = results.next();
LDAPEntryChangeControl changeControl = null;
for (LDAPControl control : results.getResponseControls()) {
    if (control instanceof LDAPEntryChangeControl) {
        changeControl = (LDAPEntryChangeControl) control;
if (changeControl != null) {
    int changeType = changeControl.getChangeType();
    switch (changeType) {
    case LDAPPersistSearchControl.ADD:
    case LDAPPersistSearchControl.DELETE:
    case LDAPPersistSearchControl.MODIFY:
    case LDAPPersistSearchControl.MODDN:
        /* shouldn't happen; log a warning and continue */
        CMS.debug("Profile change monitor: MODDN shouldn't happen; ignoring.");
        /* shouldn't happen; log a warning and continue */
        CMS.debug("Profile change monitor: unknown change type: " + changeType);
} else {
    /* shouldn't happen; log a warning and continue */
    CMS.debug("Profile change monitor: no LDAPEntryChangeControl in result.");

The first thing that has to be done is to retrieve from the LDAPSearchResults object the LDAPEntryChangeControl for the most recent search result. To do this we call results.getResponseControls(), which returns an LDAPControl[]. Each search result can arrive with multiple change controls, but we are specifically interested in the LDAPEntryChangeControl so we iterate over the LDAPControl[] until we find what we want, then break.

Next we ensure that we did in fact find the LDAPEntryChangeControl. This should always hold in our implementation but the code should handle the failure case anyway – we just log a warning and move on.

Finally, we call changeControl.getChangeType() and dispatch to the appropriate behaviour according to its value.

Interaction with the profile subsystem

Up to this point, we have seen how to use the ldapjdk API to execute a persistent LDAP search and process its results. Of course, this is just part of the story – the search somehow needs to be run in a way that doesn’t impede the regular operation of the Dogtag PKI, and needs to safely interact with the profile subsystem. Because the persistent search involves blocking calls, the procedure needs to run in its own thread.

Because this persistent search only concerns the ProfileSubsystem class, it was possible to completely encapsulate it within this class such that no changes to its API (including constructors) were necessary. An inner class Monitor, which extends Thread, actually runs the search. In this way, the code we saw above is neatly segregated from the rest of the ProfileSubsystem class, and there are no visibility issues when calling the readProfile and forgetProfile methods of the other class.

The following simplified code conveys the essence of the complete implementation:

public class ProfileSubsystem implements IProfileSubsystem {
    public void init(...) {
        // Read profiles from LDAP into the subsystem.
        // Calls readProfile for each existing LDAPEntry.

        monitor = new Monitor(this, dn, dbFactory);

    public synchronized IProfile createProfile(...) {
        // Create the profile

    public void readProfile(LDAPEntry entry) {
        // Read some LDAP attributes into local vars

    private void forgetProfile(LDAPEntry entry) {
        profileId = /* read from entry */

    private void forgetProfile(String profileId) {
        // Forget about this profile.

    private class Monitor extends Thread {
        public Monitor(...) {
            // constructor

        public void run() {
            // Execute the persistent search as above.
            // Calls readProfile and forgetProfile depending
            // on changes that occur.

So, what’s going on here? First of all, it must be emphasised that this example is simplified. For example, I have omitted details of how the monitor thread is stopped when the subsystem is shut down or reinitialised.

The monitor thread is started by the init method, once the existing profiles have been read into the profile subsystem. Executing the persistent search and handling results is the one job this the monitor has to do, so it can block without affecting any other part of the system. When it receives results, it calls the readProfile and forgetProfiles methods of the outer class – the ProfileSubsystem – to keep it up to date with the contents of the database.

Other parts of the system access the ProfileSubsystem as well, so consideration had to be given to synchronisation and making sure that changes to the contents of the ProfileSubsystem are done safely. In the end, the only method that was made synchronized was createProfile, which is also called by the REST interface. The behaviour of the handful of other methods that could be called simultaneously should be fine by virtue of the fact that the internal data structures used are themselves synchronised and idempotent. Hopefully I have not overlooked something important!


LDAP persistent searches can be used to receive immediate notification of changes that occur in an LDAP database. They support all the parameters of regular LDAP searches. ldapjdk’s API provides persistent search capabilities including the ability to discern what kind of change occurred for each result.

The ldapjdk LDAPSearchResults.hasMoreElements() method blocks each time it is called until a result has been received from the server. Because of this, it will usually be necessary to execute persistent searches asynchronously. Java threads can be employed to do this, but the usual "gotchas" of threading apply – threads must be stopped safely and the safety of methods that could be called from multiple places at the same time must be assessed. The synchronized keyword can be used to ensure serialisation of calls to methods that would otherwise be unsafe under these conditions.

September 29, 2014

Multiple Signers

You have a cloud, I have a cloud.

Neither of use are willing to surrender control of our OpenStack deployments, but we need to inter-operate.

We both have Keystone servers. Those servers are the system of record for user authorization through out our respective deployments. We each wish to maintain control of our assignments. How can we make a set of resources that can be shared?  It can’t be done today.  Here is why not, and how to make it possible.

Simple example:  Nova in OpenStack Deployment A (OSDA) creates a VM using an image from Glance in OpenStackDeployment B. Under the rules of policy, the VM and the image must both reside in the same project. However, the endpoints are managed by two different Keystone servers.

At the Keystone to Keystone level, we need to provide a shared view of the project. Since every project is owned by a domain, the correct level of abstraction is for the two Keystone servers to have a common view of the domain.  In our example we create a Domain in OSDA named “SharedDom” and assume it assigns “54321CBE” as the domain id. Keystone for OSDB needs to have an identical domain. Lets loosen the rules on domain creation to allow that: create domain takes both name and accepts “54321CBE” as the domain’s identifier.

Once we have the common domain definition, we can create a project under “SharedDom” in OSDA and, again, put a copy into OSDB.

Let us also assume that the service catalog has been synchronized across the two Keystone images. While you might be tempted to just make each a region in one Keystone server, that surrenders the control of the cloud to the remote Keystone admin, and so will not be organizationally acceptable. So, while we should tag each service catalog as a region, and keep the region names generally unique, they still are managed by distinct peer Keystone servers.

The API call that creates a VM in one endpoint by fetching an image from another accesses both endpoints with a single token. It is impossible to do today. Which keystone is then used to allocate the token? If it is OSDA, then Glance would reject it. if OSDB, Nova would reject it.

For PKI tokens, we can extract the “Signing Data” from the token body and determine which certificate signed it. From the certificate we can determine which Keystone server signed the token. Keystone would then have to provide rules for which Keystone server was allowed to sign for a domain. By default, it would be only the authoritative Keystone server associated with the same deployment. However, Keystone from OSDA would allow Keystone from OSDB to sign for Tokens under “SharedDom” only.

If both keystone servers are using UUID tokens we could arrange for the synchronization of all tokens for the shared domain across keystone instances. Now each endpoint would still authenticate tokens against its own Keystone server. The rules would be the same as far as determining which Keystone could sign for which tokens, it would just be enforced at the Keystone server level. My biggest concern with this approach is that there are synchronization problems; it has a tendency toward race conditions. But it could be made to work as well.

The agreement can be set one direction.  If OSDA  is managing the domain, and OSDB accepts the agreement, then OSDA can sign for tokens for the domain that are used across both deployments endpoints.   Keystone for OSDA must explicitly add a rule that says that Keystone for OSDB can sign for tokens as well.

This setup works  on a gentleman’s agreement genteel understanding that only one of the two Keystone servers  actively manages the domain.  One Keystone server is delegated the responsibility of managing the domain.  The other one performs that delegation, and agrees to stay out of it.  If the shared domain is proving to be too much trouble, either Keystone server can disable the domain without the agreement of the other Keystone server.

As Mick Jagger sang “Hey, you!  Get off of my cloud!”

September 26, 2014

A follow up to the Bash Exploit and SELinux.
One of the advantages of a remote exploit is to be able to setup and launch attacks on other machines.

I wondered if it would be possible to setup a bot net attack using the remote attach on an apache server with the bash exploit.

Looking at my rawhide machine's policy

sesearch -A -s httpd_sys_script_t -p name_connect -C | grep -v ^D
Found 24 semantic av rules:
   allow nsswitch_domain dns_port_t : tcp_socket { recv_msg send_msg name_connect } ;
   allow nsswitch_domain dnssec_port_t : tcp_socket name_connect ;
ET allow nsswitch_domain ldap_port_t : tcp_socket { recv_msg send_msg name_connect } ; [ authlogin_nsswitch_use_ldap ]

The apache script would only be allowed to connect/attack a dns server and an LDAP server.  It would not be allowed to become a spam bot (No connection to mail ports) or even attack other web service.

Could an attacker leave a back door to be later connected to even after the bash exploit is fixed?

# sesearch -A -s httpd_sys_script_t -p name_bind -C | grep -v ^D

Nope!  On my box the httpd_sys_script_t process is not allowed to listen on any network ports.

I guess the crackers will just have to find a machine with SELinux disabled.

September 25, 2014

What does SELinux do to contain the the bash exploit?
Do you have SELinux enabled on your Web Server?

Lots of people are asking me about SELinux and the Bash Exploit.

I did a quick analysis on one reported remote Apache exploit:


Shows an example of the bash exploit on an apache server.  It even shows that SELinux was enforcing when the exploit happened.

SELinux does not block the exploit but it would prevent escallation of confined domains.
Why didn't SELinux block it?

SELinux controls processes based on their types, if the process is doing what it was designed to do then SELinux will not block it.

In the defined exploit the apache server is running as httpd_t and it is executing a cgi script which would be labeled httpd_sys_script_exec_t.  

When httpd_t executes a script labeled httpd_sys_script_exec_t SELinux will transition the new process to httpd_sys_script_t.

SELinux policy allowd processes running as httpd_sys_script_t is to write to /tmp, so it was successfull in creating /tmp/aa.

If you did this and looked at the content in /tmp it would be labeled httpd_tmp_t


Lets look at which files httpd_sys_script_t is allowed to write to on my Rawhide box.

# sesearch -A -s httpd_sys_script_t -c file -p write -C | grep open | grep -v ^D
   allow httpd_sys_script_t httpd_sys_rw_content_t : file { ioctl read write create getattr setattr lock append unlink link rename open } ; 
   allow httpd_sys_script_t anon_inodefs_t : file { ioctl read write getattr lock append open } ; 
   allow httpd_sys_script_t httpd_sys_script_t : file { ioctl read write getattr lock append open } ; 
   allow httpd_sys_script_t httpd_tmp_t : file { ioctl read write create getattr setattr lock append unlink link rename open } ; 

httpd_sys_script_t is a process label which only applies to content in /proc.  This means processes running as httpd_sys_script_t can write to there process data.

anon_inodefs_t is an in memory label, most likely not on your disk.

The only on disk places it can write files labeled httpd_sys_rw_content_t and /tmp.

grep httpd_sys_rw_content_t /etc/selinux/targeted/contexts/files/file_contexts

or on my box

# find /etc -context "*:httpd_sys_rw_content_t:*"

With SELinux disabled, this hacked process would be allowed to write any content that is world writable on your system as well as any content owned by the apache user or group.

Lets look at what it can read.

sesearch -A -s httpd_sys_script_t -c file -p read -C | grep open | grep -v ^D | grep -v exec_t
   allow domain locale_t : file { ioctl read getattr lock open } ; 
   allow httpd_sys_script_t iso9660_t : file { ioctl read getattr lock open } ; 
   allow httpd_sys_script_t httpd_sys_ra_content_t : file { ioctl read create getattr lock append open } ; 
   allow httpd_sys_script_t httpd_sys_rw_content_t : file { ioctl read write create getattr setattr lock append unlink link rename open } ; 
   allow httpd_sys_script_t squirrelmail_spool_t : file { ioctl read getattr lock open } ; 
   allow domain ld_so_t : file { ioctl read getattr execute open } ; 
   allow httpd_sys_script_t anon_inodefs_t : file { ioctl read write getattr lock append open } ; 
   allow httpd_sys_script_t sysctl_kernel_t : file { ioctl read getattr lock open } ; 
   allow domain base_ro_file_type : file { ioctl read getattr lock open } ; 
   allow httpd_sys_script_t httpd_sys_script_t : file { ioctl read write getattr lock append open } ; 
   allow nsswitch_domain cert_t : file { ioctl read getattr lock open } ; 
   allow httpd_script_type etc_runtime_t : file { ioctl read getattr lock open } ; 
   allow httpd_script_type fonts_cache_t : file { ioctl read getattr lock open } ; 
   allow domain mandb_cache_t : file { ioctl read getattr lock open } ; 
   allow domain abrt_t : file { ioctl read getattr lock open } ; 
   allow domain lib_t : file { ioctl read getattr lock execute open } ; 
   allow domain man_t : file { ioctl read getattr lock open } ; 
   allow httpd_sys_script_t cifs_t : file { ioctl read getattr lock execute execute_no_trans entrypoint open } ; 
   allow domain sysctl_vm_overcommit_t : file { ioctl read getattr lock open } ; 
   allow httpd_sys_script_t nfs_t : file { ioctl read getattr lock execute execute_no_trans entrypoint open } ; 
   allow kernel_system_state_reader proc_t : file { ioctl read getattr lock open } ; 
   allow nsswitch_domain passwd_file_t : file { ioctl read getattr lock open } ; 
   allow nsswitch_domain sssd_public_t : file { ioctl read getattr lock open } ; 
   allow domain cpu_online_t : file { ioctl read getattr lock open } ; 
   allow httpd_script_type public_content_rw_t : file { ioctl read getattr lock open } ; 
   allow nsswitch_domain etc_runtime_t : file { ioctl read getattr lock open } ; 
   allow nsswitch_domain hostname_etc_t : file { ioctl read getattr lock open } ; 
   allow domain ld_so_cache_t : file { ioctl read getattr lock open } ; 
   allow nsswitch_domain sssd_var_lib_t : file { ioctl read getattr lock open } ; 
   allow httpd_script_type public_content_t : file { ioctl read getattr lock open } ; 
   allow nsswitch_domain krb5_conf_t : file { ioctl read getattr lock open } ; 
   allow domain abrt_var_run_t : file { ioctl read getattr lock open } ; 
   allow domain textrel_shlib_t : file { ioctl read getattr execute execmod open } ; 
   allow httpd_sys_script_t httpd_tmp_t : file { ioctl read write create getattr setattr lock append unlink link rename open } ; 
   allow domain machineid_t : file { ioctl read getattr lock open } ; 
   allow httpd_sys_script_t mysqld_etc_t : file { ioctl read getattr lock open } ; 
   allow domain rpm_script_tmp_t : file { ioctl read getattr lock open } ; 
   allow nsswitch_domain samba_var_t : file { ioctl read getattr lock open } ; 
   allow domain sysctl_crypto_t : file { ioctl read getattr lock open } ; 
   allow nsswitch_domain net_conf_t : file { ioctl read getattr lock open } ; 
   allow httpd_script_type etc_t : file { ioctl read getattr execute execute_no_trans open } ; 
   allow httpd_script_type fonts_t : file { ioctl read getattr lock open } ; 
   allow httpd_script_type ld_so_t : file { ioctl read getattr execute execute_no_trans open } ; 
   allow nsswitch_domain file_context_t : file { ioctl read getattr lock open } ; 
   allow httpd_sys_script_t httpd_squirrelmail_t : file { ioctl read getattr lock append open } ; 
   allow httpd_script_type base_ro_file_type : file { ioctl read getattr lock execute execute_no_trans open } ; 
   allow httpd_sys_script_t snmpd_var_lib_t : file { ioctl read getattr lock open } ; 
   allow nsswitch_domain samba_etc_t : file { ioctl read getattr lock open } ; 
   allow domain man_cache_t : file { ioctl read getattr lock open } ; 
   allow httpd_script_type bin_t : file { ioctl read getattr lock execute execute_no_trans open } ; 
   allow httpd_script_type lib_t : file { ioctl read getattr lock execute execute_no_trans open } ; 
   allow httpd_sys_script_t httpd_sys_content_t : file { ioctl read getattr lock execute execute_no_trans entrypoint open } ; 
   allow nsswitch_domain etc_t : file { ioctl read getattr lock open } ; 
ET allow nsswitch_domain cert_t : file { ioctl read getattr lock open } ; [ authlogin_nsswitch_use_ldap ]
ET allow nsswitch_domain slapd_cert_t : file { ioctl read getattr lock open } ; [ authlogin_nsswitch_use_ldap ]
ET allow nsswitch_domain net_conf_t : file { ioctl read getattr lock open } ; [ authlogin_nsswitch_use_ldap ]
ET allow domain sysctl_kernel_t : file { ioctl read getattr lock open } ; [ fips_mode ]

Looks like a lot of types, but most of these are System Types bin_t, lib_t ld_so_t, man_t fonts_t,  most stuff under /usr etc.

It would be allowed to read /etc/passwd (passwd_t) and most content in /etc.  

It can read apache static content, like web page data.

Well what can't it read?

user_home_t - This is where I keep my credit card data
usr_tmp_t where an admin might have left something
Other content in /var
*db_t - No database data.
It can not read most of apache runtime data like apache content in /var/lib or /var/log or /etc/httpd

With SELinux disabled, this process would be allowed to read any content that is world readable on your system as well as any content owned by the apache user our group.

We also need to look at what domains httpd_sys_script_t can transition to?

# sesearch -T -s httpd_sys_script_t -c process -C | grep -v ^D
Found 9 semantic te rules:
   type_transition httpd_sys_script_t httpd_rotatelogs_exec_t : process httpd_rotatelogs_t; 
   type_transition httpd_sys_script_t abrt_helper_exec_t : process abrt_helper_t; 
   type_transition httpd_sys_script_t antivirus_exec_t : process antivirus_t; 
   type_transition httpd_sys_script_t sepgsql_ranged_proc_exec_t : process sepgsql_ranged_proc_t; 
   type_transition httpd_sys_script_t sepgsql_trusted_proc_exec_t : process sepgsql_trusted_proc_t; 

SELinux would also block the process executing a setuid process to raise its capabilities.

Now this is a horrible exploit but as you can see SELinux would probably have protected a lot/most of your valuable data on your machine.  It would buy you time for you to patch your system.

Did you setenforce 1?

September 24, 2014

Bash specially-crafted environment variables code injection attack

Update 2014-09-30 19:30 UTC

Questions have arisen around whether Red Hat products are vulnerable to CVE-2014-6277 and CVE-2014-6278.  We have determined that RHSA-2014:1306, RHSA-2014:1311, and RHSA-2014:1312 successfully mitigate the vulnerability and no additional actions need to be taken.


Update 2014-09-26 12:00 UTC

We have written a FAQ to address some of the more common questions seen regarding the recent bash issues.

Frequently Asked Questions about the Shellshock Bash flaws

Update 2014-09-26 02:20 UTC

Red Hat has released patched versions of Bash that fix CVE-2014-7169.  Information regarding these updates can be found in the errata.  All customers are strongly encouraged to apply the update as this flaw is being actively attacked in the wild.
Fedora has also released a patched version of Bash that fixes CVE-2014-7169.  Additional information can be found on Fedora Magazine.

Update 2014-09-25 16:00 UTC

Red Hat is aware that the patch for CVE-2014-6271 is incomplete. An attacker can provide specially-crafted environment variables containing arbitrary commands that will be executed on vulnerable systems under certain conditions. The new issue has been  assigned CVE-2014-7169.
We are working on patches in conjunction with the upstream developers as a critical priority. For details on a workaround, please see the knowledgebase article.
Red Hat advises customers to upgrade to the version of Bash which contains the fix for CVE-2014-6271 and not wait for the patch which fixes CVE-2014-7169. CVE-2014-7169 is a less severe issue and patches for it are being worked on.

Bash or the Bourne again shell, is a UNIX like shell, which is perhaps one of the most installed utilities on any Linux system. From its creation in 1980, Bash has evolved from a simple terminal based command interpreter to many other fancy uses.

In Linux, environment variables provide a way to influence the behavior of software on the system. They typically consists of a name which has a value assigned to it. The same is true of the Bash shell. It is common for a lot of programs to run Bash shell in the background. It is often used to provide a shell to a remote user (via ssh, telnet, for example), provide a parser for CGI scripts (Apache, etc) or even provide limited command execution support (git, etc)

Coming back to the topic, the vulnerability arises from the fact that you can create environment variables with specially-crafted values before calling the Bash shell. These variables can contain code, which gets executed as soon as the shell is invoked. The name of these crafted variables does not matter, only their contents. As a result, this vulnerability is exposed in many contexts, for example:

  • ForceCommand is used in sshd configs to provide limited command execution capabilities for remote users. This flaw can be used to bypass that and provide arbitrary command execution. Some Git and Subversion deployments use such restricted shells. Regular use of OpenSSH is not affected because users already have shell access.
  • Apache server using mod_cgi or mod_cgid are affected if CGI scripts are either written in Bash, or spawn subshells. Such subshells are implicitly used by system/popen in C, by os.system/os.popen in Python, system/exec in PHP (when run in CGI mode), and open/system in Perl if a shell is used (which depends on the command string).
  • PHP scripts executed with mod_php are not affected even if they spawn subshells.
  • DHCP clients invoke shell scripts to configure the system, with values taken from a potentially malicious server. This would allow arbitrary commands to be run, typically as root, on the DHCP client machine.
  • Various daemons and SUID/privileged programs may execute shell scripts with environment variable values set / influenced by the user, which would allow for arbitrary commands to be run.
  • Any other application which is hooked onto a shell or runs a shell script as using Bash as the interpreter. Shell scripts which do not export variables are not vulnerable to this issue, even if they process untrusted content and store it in (unexported) shell variables and open subshells.

Like “real” programming languages, Bash has functions, though in a somewhat limited implementation, and it is possible to put these Bash functions into environment variables. This flaw is triggered when extra code is added to the end of these function definitions (inside the enivronment variable). Something like:

$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
 this is a test

The patch used to fix this flaw, ensures that no code is allowed after the end of a Bash function. So if you run the above example with the patched version of Bash, you should get an output similar to:

 $ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
 bash: warning: x: ignoring function definition attempt
 bash: error importing function definition for `x'
 this is a test

We believe this should not affect any backward compatibility. This would, of course, affect any scripts which try to use environment variables created in the way as described above, but doing so should be considered a bad programming practice.

Red Hat has issued security advisories that fixes this issue for Red Hat Enterprise Linux. Fedora has also shipped packages that fixes this issue.

We have additional information regarding specific Red Hat products affected by this issue that can be found at https://access.redhat.com/site/solutions/1207723

Information on CentOS can be found at http://lists.centos.org/pipermail/centos/2014-September/146099.html.

September 18, 2014

Enterprise Linux 5.10 to 5.11 risk report

Red Hat Enterprise Linux 5.11 was released this month (September 2014), eleven months since the release of 5.10 in October 2013. So, as usual, let’s use this opportunity to take a look back over the vulnerabilities and security updates made in that time, specifically for Red Hat Enterprise Linux 5 Server.

Red Hat Enterprise Linux 5 is in Production 3 phase, being over seven years since general availability in March 2007, and will receive security updates until March 31st 2017.

Errata count

The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 5 Server if you had installed 5.10, up to and including the 5.11 release, broken down by severity. It’s split into two columns, one for the packages you’d get if you did a default install, and the other if you installed every single package.

Note that during installation there actually isn’t an option to install every package, you’d have to manually select them all, and it’s not a likely scenario. For a given installation, the number of package updates and vulnerabilities that affected your systems will depend on exactly what you selected during installation and which packages you have subsequently installed or removed.

Security errata 5.10 to 5.11 Red Hat Enterprise Linux 5 ServerFor a default install, from release of 5.10 up to and including 5.11, we shipped 41 advisories to address 129 vulnerabilities. 8 advisories were rated critical, 11 were important, and the remaining 22 were moderate and low.

For all packages, from release of 5.10 up to and including 5.11, we shipped 82 advisories to address 298 vulnerabilities. 12 advisories were rated critical, 29 were important, and the remaining 41 were moderate and low.

You can cut down the number of security issues you need to deal with by carefully choosing the right Red Hat Enterprise Linux variant and package set when deploying a new system, and ensuring you install the latest available Update release.

Critical vulnerabilities

Vulnerabilities rated critical severity are the ones that can pose the most risk to an organisation. By definition, a critical vulnerability is one that could be exploited remotely and automatically by a worm. However we also stretch that definition to include those flaws that affect web browsers or plug-ins where a user only needs to visit a malicious (or compromised) website in order to be exploited. Most of the critical vulnerabilities we fix fall into that latter category.

The 12 critical advisories addressed 33 critical vulnerabilities across just three different projects:

  • An update to NSS/NSPR: RHSA-2014:0916(July 2014). A race condition was found in the way NSS verified certain certificates which could lead to arbitrary code execution with the privileges of the user running that application.
  • Updates to PHP, PHP53: RHSA-2013:1813, RHSA-2013:1814
    (December 2013). A flaw in the parsing of X.509 certificates could allow scripts using the affected function to potentially execute arbitrary code. An update to PHP: RHSA-2014:0311
    (March 2014). A flaw in the conversion of strings to numbers could allow scripts using the affected function to potentially execute arbitrary code.
  • Updates to Firefox, RHSA-2013:1268 (September 2013), RHSA-2013:1476 (October 2013), RHSA-2013:1812 (December 2013), RHSA-2014:0132 (February 2014), RHSA-2014:0310 (March 2014), RHSA-2014:0448 (Apr 2014), RHSA-2014:0741 (June 2014), RHSA-2014:0919 (July 2014) where a malicious web site could potentially run arbitrary code as the user running Firefox.

Updates to correct 32 of the 33 critical vulnerabilities were available via Red Hat Network either the same day or the next calendar day after the issues were public.

Overall, for Red Hat Enterprise Linux 5 since release until 5.11, 98% of critical vulnerabilities have had an update available to address them available from the Red Hat Network either the same day or the next calendar day after the issue was public.

Other significant vulnerabilities

Although not in the definition of critical severity, also of interest are other remote flaws and local privilege escalation flaws:

  • A flaw in glibc, CVE-2014-5119, fixed by RHSA-2014:1110 (August 2014). A local user could use this flaw to escalate their privileges. A public exploit is available which targets the polkit application on 32-bit systems although polkit is not shipped in Red Hat Enterprise Linux 5. It may be possible to create an exploit for Red Hat Enterprise Linux 5 by targeting a different application.
  • Two flaws in squid, CVE-2014-4115, and CVE-2014-3609, fixed by RHSA-2014:1148 (September 2014). A remote attacker could cause Squid to crash.
  • A flaw in procmail, CVE-2014-3618, fixed by RHSA-2014:1172 (September 2014). A remote attacker could send an email with specially crafted headers that, when processed by formail, could cause procmail to crash or, possibly, execute arbitrary code as the user running formail.
  • A flaw in Apache Struts, CVE-2014-0114, fixed by RHSA-2014:0474 (April 2014). A remote attacker could use this flaw to manipulate the ClassLoader used by an application server running Stuts 1 potentially leading to arbitrary code execution under some conditions.
  • A flaw where yum-updatesd did not properly perform RPM signature checks, CVE-2014-0022, fixed by RHSA-2014:1004 (Jan 2014). Where yum-updatesd was configured to automatically install updates, a remote attacker could use this flaw to install a malicious update on the target system using an unsigned RPM or an RPM signed with an untrusted key.
  • A flaw in the kernel floppy driver, CVE-2014-1737, fixed by RHSA-2014:0740 (June 2014). A local user who has write access to /dev/fdX on a system with floppy drive could use this flaw to escalate their privileges. A public exploit is available for this issue. Note that access to /dev/fdX is by default restricted only to members of the floppy group.
  • A flaw in libXfont, CVE-2013-6462, fixed by RHSA-2014:0018 (Jan 2014). A local user could potentially use this flaw to escalate their privileges to root.
  • A flaw in xorg-x11-server, CVE-2013-6424, fixed by RHSA-2013:1868 (Dec 2013). An authorized client could potentially use this flaw to escalate their privileges to root.
  • A flaw in the kernel QETH network device driver, CVE-2013-6381, fixed by RHSA-2014:0285 (March 2014). A local, unprivileged user could potentially use this flaw to escalate their privileges. Note this device is only found on s390x architecture systems.

Note that Red Hat Enterprise Linux 5 was not affected by the OpenSSL issue, CVE-2014-0160, “Heartbleed”.

Previous update releases

We generally measure risk in terms of the number of vulnerabilities, but the actual effort in maintaining a Red Hat Enterprise Linux system is more related to the number of advisories we released: a single Firefox advisory may fix ten different issues of critical severity, but takes far less total effort to manage than ten separate advisories each fixing one critical PHP vulnerability.

To compare these statistics with previous update releases we need to take into account that the time between each update release is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:

Security Errata per month to 5.10This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn’t really useful for comparisons with other major versions, distributions, or operating systems — for example, a default install of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, time scales, and severity range of interest.

See also:
5.10, 5.9, 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, and 5.1 risk reports.

September 15, 2014

Confusion with sesearch.
I just saw an email where a user was asking why sesearch is showing access but the access is still getting denied.

I'm running CentOS 6. I've httpd running which accesses a file but it results in access denied with the following --

type=AVC msg=audit(1410680693.979:40): avc:  denied  { read } for pid=987 comm="httpd" name="README.txt" dev=dm-0 ino=12573 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file


sesearch -A | grep 'allow httpd_t' | grep ': file' | grep user_home_t
   allow httpd_t user_home_t : file { ioctl read getattr lock open } ;
   allow httpd_t user_home_t : file { ioctl read getattr lock open } ;


sesearch is a great tool that we use all the time.  It allows you to analyze and look the the SELInux policy.  It is part of the setools-console package.  It uses the "Apol" libraries to examine policy, the same libraries we have used to build the new tool set sepolicy.

The problem was that he was using sesearch incorrectly.  sesearch -A shows you all possible, allow rules not just the allow rules that are currently in effect.

The user needs to add a -C option to the sesearch.  The -C options shows you the booleans required for that access.  It also shows a capital E or D indicating whether or not the boolean is enabled or disabled in policy at the beginning of the line.

On my machine, I will use a more complicated command, this command says show the allow rules for a source type of httpd_t, and a target type of user_home_t, permission=read on a class=file.

sesearch -A -C -s httpd_t -t user_home_t -p read -c file
Found 1 semantic av rules:
DT allow httpd_t user_home_type : file { ioctl read getattr lock open } ; [ httpd_read_user_content ]

As you can see on my machine the boolean is disabled, so Apache is not allowed to read general content in my homedir, which I assume was true for the user.   If  the user wants to allow httpd_t to read all general content in the users homedir you can turn on the httpd_read_user_content boolean.

If you want to allow it to read just a certain directories/files, recommended,  you should change the label on the directory.  BTW ~/public_html and ~/www already have the correct labeling.

matchpathcon ~/public_html ~/www
/home/dwalsh/public_html    staff_u:object_r:httpd_user_content_t:s0
/home/dwalsh/www    staff_u:object_r:httpd_user_content_t:s0

I would not want to let the apache process read general content in my homedir, since I might be storing critical stuff like credit card data, passwords, and unflattering pictures of me in there. :^)

September 10, 2014

What is this new unconfined_service_t type I see on Fedora 21 and RHEL7?
Everyone that has ever used SELinux knows that the unconfined_t domain is a process label that is not confined.  But this is not the only unconfined domain on a SELinux system.  It is actually the default domain of a user that logs onto a system.  In a lot of ways we should have used the type unconfined_user_t rather then unconfined_t.

By default in an SELinux Targeted system there are lots of other unconfined domains.  We have these so that users can run programs/services without SELinux interfering if SELinux does not know about them. You can list the unconfined domains on your system using the following command.

seinfo -aunconfined_domain_type -x

In RHEL6 and older versions of Fedora, we used to run system services as initrc_t by default.  Unless someone has written a policy for them.  initrc_t is an unconfined domain by default, unless you disabled the unconfined.pp module. Running unknown serivices as initrc_t allows administrators to run an application service, even if no policy has never been written for it.

In RHEL6 we have these rules:

init_t @initrc_exec_t -> initrc_t
init_t @bin_t -> initrc_t

If an administrator added an executable service to /usr/sbin or /usr/bin, the init system would run the service as initrc_t.

We found this to be problematic, though. 

The problem was that we have lots of transition rules out of initrc_t.  If a program we did not know about was running as initrc_t and executed a program like rsync to copy data between servers, SELinux would transition the program to rsync_t and it would blow up.  SELinux mistakenly would think that rsync was set up in server mode, not client mode.  Other transition rules could also cause breakage. 

We decided we needed a new unconfined domain to run services with, that would have no transition rules.  We introduced the unconfined_service_t domain.  Now we have:

init_t @bin_t -> unconfined_service_t

A process running as unconfined_service_t is allowed to execute any confined program, but stays in the unconfined_service_t domain.  SELinux will not block any access. This means by default, if you install a service that does not have policy written for it, it should work without SELinux getting in the way.

Sometimes applications are installed in fairly random directories under /usr or /opt (Or in oracle's case /u01), which end up with the label of usr_t, therefore we added these transition rules to policy.

# sesearch -T -s init_t  | grep unconfined_service_t
type_transition init_t bin_t : process unconfined_service_t;
type_transition init_t usr_t : process unconfined_service_t;
You can see it in Fedora21.

Bottom Line

Hopefully unconfined_service_t will make leaving SELinux enabled easier on systems that have to run third party services, and protect the other services that run on your system.

Thanks to Simon Sekidde and Miroslav Grepl for helping to write this blog.
TLS landscape

Transport Layer Security (TLS) or, as it was known in the beginnings of the Internet, Secure Sockets Layer (SSL) is the technology responsible for securing communications between different devices. It is used everyday by nearly everyone using the globe-spanning network.

Let’s take a closer look at how TLS is used by servers that underpin the World Wide Web and how the promise of security is actually executed.


Hyper Text Transfer Protocol (HTTP) in versions 1.1 and older make encryption (thus use of TLS) optional. Given that the upcoming HTTP 2.0 will require use of TLS and that Google now uses the HTTPS in its ranking algorithm, it is expected that many sites will become TLS-enabled.

Surveying the Alexa top 1 million sites, most domains still don’t provide secure communication channel for their users.

Just under 40% of HTTP servers support TLS or SSL and present valid certificates.

Just under 40% of HTTP servers support TLS or SSL and present valid certificates.

Additionally, if we look at the version of the protocol supported by the servers most don’t support the newest (and most secure) version of the protocol TLSv1.2.  Of more concern is the number of sites that support the completely insecure SSLv2 protocol.

Only half of HTTPS servers support TLS 1.2

Only half of HTTPS servers support TLS 1.2

(There are no results for SSLv2 for first 3 months because of error in software that was collecting data.)

One of the newest and most secure ciphers available in TLS is Advanced Encryption Standard (AES) in Galois/Counter Mode (AES-GCM). Those ciphers provide good security, resiliency against known attacks (BEAST and Lucky13), and very good performance for machines with hardware accelerators for them (modern Intel and AMD CPUs, upcoming ARM).

Unfortunately, it is growing a bit slower than TLS adoption in general, which means that some of the newly deployed servers aren’t using new cryptographic libraries or are configured to not use all of their functions.

Only 40% of TLS web servers support AES-GCM ciphersuites.

Only 40% of TLS web servers support AES-GCM ciphersuites.

Bad recommendations

A few years back, a weakness in TLS 1.0 and SSL 3 was shown to be exploitable in the BEAST attack. The recommended workaround for it was to use RC4-based ciphers. Unfortunately, we later learned that the RC4 cipher is much weaker than it was previously estimated. As the vulnerability that allowed BEAST was fixed in TLSv1.1, using RC4 ciphers with new protocol versions was always unnecessary. Additionally, now all major clients have implemented workarounds for this attack, which currently makes using RC4 a bad idea.

Unfortunately, many servers prefer RC4 and some (~1%) actually support only RC4.  This makes it impossible to disable this weak cipher on client side to force the rest of servers (nearly 19%) to use different cipher suite.

RC4 is still used with more than 18% of HTTPS servers.

RC4 is still used with more than 18% of HTTPS servers.

The other common issue, is that many certificates are still signed using the obsolete SHA-1. This is mostly caused by backwards compatibility with clients like Windows XP pre SP2 and old phones.

SHA-256 certificates only recently started growing in numbers

SHA-256 certificates only recently started growing in numbers

The sudden increase in the SHA-256 between April and May was caused by re-issuance of certificates in the wake of Heartbleed.

Bad configuration

Many servers also support insecure cipher suites. In the latest scan over 3.5% of servers support some cipher suites that uses AECDH key exchange, which is completely insecure against man in the middle attacks. Many servers also support single DES (around 15%) and export grade cipher suites (around 15%). In total, around 20% of servers support some kind of broken cipher suite.

While correctly implemented SSLv3 and later shouldn’t allow negotiation of those weak ciphers if stronger ones are supported by both client and server, at least one commonly used implementation had a vulnerability that did allow for changing the cipher suite to arbitrary one commonly supported by both client and server. That’s why it is important to occasionally clean up list of supported ciphers, both on server and client side.

Forward secrecy

Forward secrecy, also known as perfect forward secrecy (PFS), is a property of a cipher suite that makes it impossible to decrypt communication between client and server when the attacker knows the server’s private key. It also protects old communication in case the private key is leaked or stolen. That’s why it is such a desirable property.

The good news is that most servers (over 60%) not only support, but will actually negotiate cipher suites that provide forward secrecy with clients that support it. The used types are split essentially between 1024 bit DHE and 256 bit ECDHE, scoring respectively 29% and 33% of all servers in latest scan. The amount of servers that do negotiate PFS enabled cipher suites is also steadily growing.

PFS support among TLS-enabled HTTP servers

PFS support among TLS-enabled HTTP servers


Most Internet facing servers are badly configured, sometimes it is caused by lack of functionality in software, like in case of old Apache 2.2.x releases that don’t support ECDHE key exchange, and sometimes because of side effects of using new software with old configuration (many configuration tutorials suggested using !ADH in cipher string to disable anonymous cipher suites, that unfortunately doesn’t disable anonymous Elliptic Curve version of DH – AECDH, for that, use of !aNULL is necessary).

Thankfully, the situation seems to be improving, unfortunately rather slowly.

If you’re an administrator of a server, consider enabling TLS.  Performance issues when encryption was slow and taxing on servers are long gone. If you already use TLS, double check your configuration preferably using the Mozilla guide to server configuration as it is regularly updated. Make sure you enable PFS cipher suites and put them above non-PFS ciphers and that you as well as the Certificate Authority you’ve chosen, use modern crypto (SHA-2) and large key sizes (at least 2048 bit RSA).

If you’re a user of a server and you’ve noticed that the server doesn’t use correct configuration, try contacting the administrator – he may have just forgotten about it.

September 05, 2014

Think before you just blindly audit2allow -M mydomain
Don't Allow Domains to write Base SELinux Types

A few years ago I wrote a blog and paper on the four causes of SELinux errors.

The first two most common causes were labeling issues and SELinux needs to know.

Easiest way to explain this is a daemon wants to write to a certain file and SELinux blocks
the application from writing.  In SELinux terms the Process DOMAIN (httpd_t) wants to write to the file type (var_lib_t)
and it is blocked.  Users have potentially three ways of fixing this.

  1. Change the type of the file being written.

    • The object might be mislabeled and restorecon of the object fixes the issue

    • Change the label to httpd_var_lib_t using semanage and restorecon
        semanage fcontext -a -t httpd_var_lib_t '/var/lib/foobar(/.*)?'
        restorecon -R -v /var/lib/foobar

  2. There might be a boolean available to allow the Process Domain to write to the file type
      setsebool -P HTTP_BOOLEAN 1

  3. Modify policy using audit2allow
      grep httpd_t /var/log/audit/audit.log | audit2allow -M myhttp
      semodule -i myhttpd.pp

Sadly the third option is the least recommended and the most often used. 

The problem is it requires no thought and gets SELinux to just shut up.

In RHEL7 and latest Fedoras, the audit2allow tools will suggest a boolean when you run the AVC's through it.  And setroubleshoot has been doing this for years. setroubleshoot even will suggest potential types that you could change the destination object to use.

The thing we really want to stop is domains writing to BASE types.  If I allow a confined domain to write to a BASE type like etc_t or usr_t, then a hacked system can attack other domains, since almost all other domains need to read some etc_t or usr_t content.


One other feature we have added in RHEL7 and Fedora is a list of base types.  SELinux has a mechanism for grouping types based on an attribute.
We have to new attributes base_ro_file_type and base_file_type.  You can see the objects associated with these attributes using the seinfo command.

seinfo -abase_ro_file_type -x

$ seinfo -abase_file_type -x

If you use audit2allow to add a rule to allow a domain to write to one of the base types:

Most likely you are WRONG

If you have a domain that is attempting to write to one of these base types, then you most likely need to change the type of the destination object using the semanage/restorecon commands mentioned above.
The difficult thing for the users to figure out; "What type should I change the object to?"

We have added new man pages that show you the types that you program is allowed to write

man httpd_selinux

Look for writable types?

If your domain httpd_t is attempting to write to var_lib_t then look for httpd_var_lib_t. "sepolicy gui" is a new gui tool to help you understand the types also.

Call to arms:
If an enterprising hacker wanted to write some code, it would be nice to build this knowledge into audit2allow.  Masters Thesis anyone???

September 03, 2014

Is your software fixed?

A common query seen at Red Hat is “our auditor says our Red Hat machines are vulnerable to CVE-2015-1234, is this true?” or “Why hasn’t Red Hat updated software package foo to version 1.2.3?” In other words, our customers (and their auditors) are not sure whether or not we have fixed a security vulnerability, or if a given package is up to date with respect to security issues. In an effort to help our security-conscious customers, Red Hat make this information available in an easy to consume format.

What’s the deal with CVEs?

Red Hat is committed to the CVE process. To quote our CVE compatibility page:

We believe that giving our users accurate and complete information about security issues is extremely important. By including CVE names when we discuss security issues in our services and products, we can help users cross-reference vulnerabilities so they spend less time investigating and categorizing security events.

Red Hat has a representative on the CVE Editorial Board and declared CVE compatibility in April 2002.

To put it simply: if it’s a security issue and we fix it in an RHSA it gets a CVE. In fact we usually assign CVEs as soon as we determine a security issue exists (additional information on determining what constitutes a security issue can be found on our blog.).

How to tell if you software is fixed?

A CVE can be queried at our public CVE page.  Details concerning the vulnerability, the CVSS v2 metrics, and security errata are easily accessible from here.

To verify you system is secure, simply check which version of the package you have installed and if the NVR of your installed package is equal to or higher than the NVR of the package in the RHSA then you’re safe.

What’s an NVR?

The NVR is the Name-Version-Release of the package. The Heartbleed RHSA lists packages such as: openssl-1.0.1e-16.el6_5.7.x86_64.rpm. So from this we see a package name of “openssl” (a hyphen), a version of 1.0.1e (a hyphen) and the release is 16.el6_5.7. Assuming you are running RHEL 6, x86_64, if you have openssl version 1.0.1e release 16.el6_5.7 or later you’re protected from the Heartbleed issue.

Please note, there is an additional field called “epoch”, this field actually supersedes the version number (and release), most packages do not have an epoch number, however a larger epoch number means that a package can override a package with a lower epoch. This can be useful, for example, if you need a custom modified version of a package that also exists in RPM repos you are already using.  By assigning an epoch number to your package RPM you can override the same version package RPMs from another repo even if they have a higher version number. So be aware, using packages that have the same name and a higher epoch number you will not get security updates unless you specifically create new RPM’s with the epoch number and the security update.

But what if there is no CVE page?

As part of our process the CVE pages are automatically created if public entries exist in Bugzilla.  CVE information may not be available if the details of the vulnerability have not been released or the issue is still embargoed.  We do encourage responsible handling of vulnerabilities and sometimes delay CVE information from being made public.

Also, CVE information will not be created if the software we shipped wasn’t vulnerable.

How to tell if your system is vulnerable?

If you have a specific CVE or set of CVEs that you are worried about you can use the yum command to see if your system is vulnerable. Start by installing yum-plugin-security:

sudo yum install yum-plugin-security

Then query the CVE you are interested in, for example on a RHEL 7 system without the OpenSSL update:

[root@localhost ~]# yum updateinfo info --cve CVE-2014-0224
 Important: openssl security update
 Update ID : RHSA-2014:0679
 Release : 
 Type : security
 Status : final
 Issued : 2014-06-10 00:00:00
 Bugs : 1087195 - CVE-2010-5298 openssl: freelist misuse causing 
        a possible use-after-free
 : 1093837 - CVE-2014-0198 openssl: SSL_MODE_RELEASE_BUFFERS NULL
   pointer dereference in do_ssl3_write()
 : 1103586 - CVE-2014-0224 openssl: SSL/TLS MITM vulnerability
 : 1103593 - CVE-2014-0221 openssl: DoS when sending invalid DTLS
 : 1103598 - CVE-2014-0195 openssl: Buffer overflow via DTLS 
   invalid fragment
 : 1103600 - CVE-2014-3470 openssl: client-side denial of service 
   when using anonymous ECDH
 CVEs : CVE-2014-0224
 : CVE-2014-0221
 : CVE-2014-0198
 : CVE-2014-0195
 : CVE-2010-5298
 : CVE-2014-3470
Description : OpenSSL is a toolkit that implements the Secure 
Sockets Layer

If your system is up to date or the CVE doesn’t affect the platform you’re on then no information will be returned.


Red Hat Product Security makes available as much information as we can regarding vulnerabilities affecting our customers.  This information is available on our customer portal as well as within the software repositories. As you can see it is both easy and quick to determine if your system is up to date on security patches with the provided information and tools.

The following checklist can be used to check if systems or packages are affected by specific security issues:

1) Check if the issue you’re concerned about has a CVE and check the Red Hat CVE page:


2) Check to see if your system is up to date for that issue:

sudo yum install yum-plugin-security 
yum updateinfo info --cve CVE-2014-0224

3) Alternatively you can check the package NVR in the RHSA errata listed in the CVE page (in #1) and compare it to the packages on your system to see if they are the same or greater.
4) If you still have questions please contact Red Hat Support!

Three Types of Keystone Users

Keystone supports multiple backend for Identity.  While SQL is the default, LDAP is one of the most used.  With Federation protocols, the user data won’t even be stored in the identity backend at all.  All three of these approaches have different use cases, and all work together.  The way that that I’ve come to think of them is as  three types of Keystone users:  employees, partners, and customers.  Take the following as a metaphor, not literal  truth.

LDAP is the enterprise database of choice for employees  (technically, a Directory server, accessed by LDAP, but we call the Database LDAP to distinguish from relation database which we call SQL due to the query language, to should satisfy the pedants.)  LDAP entries are managed by HR (with help from IT) and is considered read-only data by most applications in the enterprise.  The Keystone users in LDAP are people that we pay money to use OpenStack.

Partners are people from other companies that we work with.  They have their own LDAP servers, but we don’t have access to them via LDAP.  Instead, we get a SAML document, which is basically a snapshot of an LDAP query, signed with a private key.  Partners are people we work with to make money.  We neither pay them nor do they pay us.

Customers are people who pay us to use our OpenStack deployment.  There are lots of them.  They are quickly added to our Keystone store.  Since LDAP is read only, we stick them in SQL.  They belong to different companies, and people from different companies shouldn’t know about each other, so each of these companies are in their own domain.  Only SQL allows multiple domains to be added dynamically.

Obviously, this is greatly over simplified.  It does not account for service users.  SAML might be used for employees.  Another approach is that  everyone might be in LDAP, and LDAP is set to be read-write;  CERN does this.  But the three classes of users listed above each represents a different usage pattern for Keystone’s  identity store, and all three can and should  be supportable in the same Keystone deployment.

August 21, 2014


Kerberos was slow when talking to my demo machine. As part of debugging it, I was making DNS changes, so I pointed my machine directly to the DNS server. It was at my hosting provider, and authoritative for my domain.

As I tend to do, I idly checked Facebook. Its a bad habit, like biting nails. Sometimes I’m not even aware that I am doing it. This time, however, a browser warning brought me up short:

“Security Error: Domain Name Mismatch”

The certificate reported that it was valid for a domain that ended in the same domain name as the nameserver I was pointing at.

Someone just like me had the ability to push up whatever they wanted to the DNS server. This is usually fine: only the Authoritative DNS server for a site is allowed to replicate changes. It did mean, however, that anyone that was looking at this particular DNS server would be directed to something they were hosting themselves. I’m guessing it was a Phishing attempt as I did not actually go to their site to check.

Most of us run laptops set up to DNS from the DHCP server we connect to. Which means that if we are at a Coffee Shop, the local library, or the Gym, we are running against an unknown DNS server. The less trusted the location, the less reason to trust the DHCP server.

This is a nasty problem to work around. There are things you can do to mitigate, such as whitelisting DNS servers. The onus, however, should not be up to the end users. DNSSec attempts to address the issues. Until we have that, however, use HTTPS where ever possible. And check the certificates.

August 18, 2014

Musings on identity management

This post is an edited version of an email I sent to the Red Hat Identity Management (IdM) team mailing list that outlines the main take-aways from my first few months working on the FreeIPA identity management solution.

I’m over three months into my new gig on the identity management team at Red Hat now, so I would like to share a few thoughts about what I’ve learned about identity management.

I was excited to come into this role because of my innate interest in security and cryptography. I had little practical experience with PKI and security protocols beyond basic X.509/TLS and OpenPGP, so I have been relishing the opportunity to broaden my knowledge and experience and solve problems in this domain.

What I did not understand, when I joined, was just how much an effective IdM strategy and infrastructure can benefit businesses and large communities in the form of improved security and reduced risk (two sides of the same coin, one could argue) and of course, greater efficiency. The diversity of use cases and the versatility of our software to address these use cases also amazed me.

This added perspective motivates me to seek opportunities to talk to people and find out about their IdM needs and how existing offerings (ours or others) are falling short, and work out what we as a team can do to better meet and even anticipate their needs. It has also given me a foundation to explain to non-technical people what FreeIPA and related projects are all about, and help them understand how our solutions can help their business or community.

I say "community" above because I have begun to see that free software communities represent valuable proving grounds for FreeIPA. For example, a couple of weeks ago during PyCon Australia I was chatting to Nick Coghlan and learned that the Python community is currently struggling with a proliferation of identity silos – developer accounts, PSF memberships and roles, the main website, PyPI, and so on. Yet noone has put their hand up to address this. I didn’t quite commit to writing a PEP to fix all that (yet) but we agreed that this represents a great opportunity to employ FreeIPA to benefit an important project and community – important for our team and for Red Hat as well as for the software industry in general. How many other communities to whom we have links or on whom we rely could benefit from FreeIPA in a similar way? And how much will our solutions be improved, and new innovations discovered, by what we might learn in working with these communities to improve their identity management?

So, that’s most of what I wanted to say, but I want to thank you all for your assistance and encouragement during my first few months. It has been quite a shift adapting to working with a global team, but I am really enjoying working with you on Red Hat IdM and am excited for our future.

August 13, 2014

Fedora Security Team

Vulnerabilities in software happen.  When they get fixed it’s up to the packager to make those fixes available to the systems using the software.  Duplicating much of the response efforts that Red Hat Product Security performs for Red Hat products, the Fedora Security Team (FST) has recently been created to assist packagers get vulnerability fixes downstream in a timely manner.

At the beginning of July, there were over 500 vulnerability tickets open* against Fedora and EPEL.  Many of these vulnerabilities already had patches or releases available to remedy the problems but not all.  The Team has already found several examples of upstream not knowing that the vulnerability exists and was able to fix the issue quickly.  This is one of the reasons having a dedicated team to work these issues is so important.

In the few short weeks since the Team was created, we’ve already closed 14 vulnerability tickets and are working another 150.  We hope to be able to work in a more real-time environment once the backlog decreases.  Staying in front of the vulnerabilities will not be easy, however.  During the week of August 3rd, 27 new tickets were opened for packages in Fedora and EPEL.  While we haven’t figured out a way to get ahead of the problem, we are trying to deal with the aftermath and get fixes pushed to the users as quickly as possible.

Additional information on the mission and the Team can be found on our wiki page.  If you’d like to get involved please join us for one of our meetings and subscribe to our listserv.


* A separate vulnerability ticket is sometimes opened for different versions of Fedora and EPEL resulting in multiple tickets for a single vulnerability.  This makes informing the packager easier but also inflates the numbers significantly.