March 26, 2015

OpenStack keeps resetting my hostname

No matter what I changed, something kept setting the hostname on my vm to federate.cloudlab.freeipa.org.novalocal. Even forcing the /etc/hostname file to be uneditable did not prevent this change. Hunting this down took far too long, and here is the result of my journey.

Old Approach

A few releases ago, I had a shell script for spinning up new virtual machines that dealt with dhclient resetting values by putting overrides into /etc/dhclient.conf.  Find this file was a moving target.  First it moved into

/etc/dhcp/dhclient.conf.

Then to a file inside

/etc/dhcp/dhclient.d

And so on.  The change I wanted to make was to do two things:

  1.  set the hostname explicitly and keep it that way
  2. Use my own dnsserver, not the dhcp managed one

Recently, I started working on a RHEL 7.1 system running on our local cloud.  No matter what I did, I could not fix the host name.  Here are some  of the things I tried:

  1. Setting the value in /etc/hostname
  2. running hostnamectl set-hostname federate.cloudlab.freeipa.org
  3. Using nmcli to set the properties for the connections ipv4 configuration
  4. Explicitly Setting it in /etc/sysconfig/network-scripts/ifcfg-eth0
  5. Setting the value in /etc/hostname and making hostname immutable with chattr +i /etc/hostname

Finally, Dan Williams (dcbw) suggested I look in the journal to see what was going on with the host name.  I ran journalctl -b and did a grep for hostname.  Everything looked right until…

Mar 26 14:01:10 federate.cloudlab.freeipa.org cloud-init[1914]: [CLOUDINIT] stages.py[DEBUG]: Running module set_hostname (<module 'cloudinit.config.cc_set_hostname' from '/usr/lib/python2.7/site-packages/cloudinit...

cloud-init?

But…I thought that was only supposed to be run when the VM was first created? So, regardless of the intention, it was no longer helping me.

yum erase cloud-init

And now the hostname that I set in /etc/hostname survives a reboot. I’ll post more when I figure out why cloud-init is still running after initialization.

For discussion: Orphaned package in Fedora

The Fedora Security Team (FST) has uncovered an interesting problem.  Many packages in Fedora aren’t being actively maintained meaning they are unofficially orphaned.  This is likely not a problem since at least some of these packages will happily sit there and be well behaved.  The ones we worry about are the ones that pick up CVEs along the way, warning of unscrupulous behaviour.

The FST has been plugging away at trying to help maintainers update their packages when security flaws are known to exist.  So far we’ve almost hit the 250 bug level.  Unfortunately we forced a policy that still isn’t perfect.  What do you do with a package that is no longer is supported and has a known vulnerability in it?  Unless you can recruit someone to adopt the package the only responsible choice you have is to retire the package and remove it from the repositories.

This, of course, leads to other problems, specifically that someone has that package installed and they know not that the package is no longer supported nor do they know it contains a security vulnerability.  This morning, during the FST meeting, we discussed the problem a bit and I had an idea that I’ll share here in hopes of starting a discussion.

The Idea

Create a file containing all the packages that have been retired from a repository and perhaps a short reason for why this package has been retired.  Then have yum/dnf consume this information regularly and notify the user/admin when a package that is installed is added to this list.  This allows the system admin to become aware of the unsupported nature of the package and allows them to make a decision as to whether or not to keep the package on the system.

Okay, discuss…


A change in thinking…

When I entered the information security world in late 2001 I received training on communications technologies that included a significant interest in confidentiality.  Obviously the rest of the trifecta, integrity and availability, were also important but maintaining communications security was king.

Now, almost fifteen years later, I’m still focused on the trifecta with confidentiality coming out with a strong lead.  But my goals have changed.  While confidentiality is an important piece of the puzzle, for privacy and other reasons, I feel it should no longer be king with my work and writing.

Over the coming weeks I plan to focus on the availability of data.  And not just whether or not a file is on a server somewhere but diving into the heart of the availability problem.  File format standards, flexibility of the data to be used with accessibility tools, ability to translate the words into other languages to ease sharing, and the ability to move the information to other forms of media to improve access are all topics I want to cover.

I’m largely writing this as a reminder of ideas I want to research and discuss but I hope this gets other people thinking about their own works.  If you have a great idea don’t you want to make it easier for other people to consume your thoughts and be able to build on them?  Unfortunately the solution isn’t simple and I suspect much will be written over time about the topic.  Hopefully we’ll have a solution soon before that StarWriter file you have stored on a 5.25″ floppy drive is no longer readable.


March 25, 2015

Troubleshooting Keystone in a New Install

Recently heard complaints:

I’ve done a deployment , and every time I try to log in to the dashboard, I get “An error occurred authenticating. Please try again later.” Somewhat surprisingly, the only log that I’m noticing showing anything of note is the Apache error log, which reports ‘Login failed for user “admin”‘. I’ve bumped keystone — where I’d assume the error is happening — to DEBUG, but it’s showing exactly zero activity. How do I go about debugging this?’

Trying to enable LDAP with OpenStack/keystone in Juno release. All the horizon users return error “You are not authorized for any projects.” Similarly, all the OpenStack services are reported not to be authorized.’
What is supposed to happen:

  1. You Login to Horizon using admin and the correct password
  2. Horizon passes that to Keystone in a token request
  3. Keystone uses that information to create a token. If the user has a default project set, the token is scoped to the default proejct
  4. token is returned to Horizon

Let’s take a deeper look at step 3.
In order to perform an operation on a resource in a project, a user needs to be assigned a role in a project. So the failure could happen at a couple steps.

  1. The user does not exist in the identity backend
  2. The user has the wrong password
  3. The user has no role assignments
  4. The user has a default project assigned, but does not have a role assignment for that project

The Keystone configuration file

Most deployments run with Keystone reading its configuration values from /etc/keystone/keystone.conf. It is an ini file, with section headers.
In Juno and Icehouse, the storage is split into two pieces: Identity and Assignment. Identity holds users and groups. Assignment holds roles, role assignments, projects and domains. Let’s start with the simplest scenario.
Identity in SQL, Assignments in SQL:
This is what you get from devstack if you make no customizations. To confirm that you are running this way, look in your Keystone.conf file for the sections that starts with
[identity]
and
[assignment]
and look for the value driver. In a Devstack deployment that I just ran, I have

[identity]
driver = keystone.identity.backends.sql.Identity

Which confirms I am running witht he SQL driver for identity, and

[assignment]
driver = keystone.assignment.backends.sql.Assignment

Which confirms I am running with the SQL driver for Assignment
First steps
For Devstack, I get my environment variables set using

. openrc
and this will set:
$OS_AUTH_URL $OS_NO_CACHE $OS_TENANT_NAME
$OS_CACERT $OS_PASSWORD $OS_USERNAME
$OS_IDENTITY_API_VERSION $OS_REGION_NAME $OS_VOLUME_API_VERSION
echo $OS_USERNAME
demo

To change to the admin user:

$ export OS_USERNAME=admin
$ export OS_PASSWORD=FreeIPA4All

While we are trying to get people to move to the common CLI, older deployments may only have the keystone CLI to work with. I’m going to start with that.

$ keystone --debug token-get
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://192.168.1.58:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.1.58
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3783
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2015-03-25T16:03:25Z |
| id | ec7c2d1f07c5414499c3cbaf7c59d4be |
| tenant_id | 69ff732083a64a1a8e34fc4d2ea178dd |
| user_id | 042b50edf70f484dab1f14e893a73ea8 |
+-----------+----------------------------------+

OK, what happens when I do keystone token-get? The CLI uses the information I provide to try and get a token;

$ echo $OS_AUTH_URL

http://192.168.1.58:5000/v2.0

OK…It is going to go to a V2 specific URL. And, to confirm:

$ echo $OS_IDENTITY_API_VERSION

2.0

We are using Version 2.0
The username, password and tenant used are

$ echo $OS_USERNAME
admin
$ echo $OS_PASSWORD
FreeIPA4All
$ echo $OS_TENANT_NAME
demo

Let’s assume that running keystone token-get fails for you. Let’s try to isolate the issue to the role assignments by getting an unscoped token:

$ unset OS_TENANT_NAME
$ echo $OS_TENANT_NAME

That should return a blank line. Now:

$ keystone token-get
+----------+----------------------------------+
| Property | Value |
+----------+----------------------------------+
| expires | 2015-03-25T16:14:28Z |
| id | 2a3ce489422342f2b6616016cb43ebc2 |
| user_id | 042b50edf70f484dab1f14e893a73ea8 |
+----------+----------------------------------+

If this fails, it could be one of a few things:

  1. User does not exist
  2. Password is wrong
  3. User has a default tenant that is invalid

How can we check:

Using Admin Token

Bootstrapping the Keystone install requires putting users in the database before there are any users defined. Most installers take advantage of an alternate mechanism called the ADMIN_TOKEN or SERVICE_TOKEN. To see the value for this, look in keystone.conf section:
[DEFAULT]
for a value like this:
#admin_token = ADMIN
Note that devstack follows the best practice of disabling the admin token by commenting it out. This password is very powerful and should be disabled in common usage, but is very powerful for fixing broken systems. To enable it, uncomment the value, and restart Keystone.

Using the Common CLI

The keystone command line has been deprecated with an eye toward using the openstack client. Since you might be deploying an old version of Openstack that has different library dependencies, you might not be able to install the latest version on your server, but you can (and should) run an updated version on your workstation which will then be capable of talking to older versions of keystone.
To perform operations using the common cli you need to pass the endpoint and admin_token as command line parameters.

The os-url needs to be the publicly routed URL to the admin interface. The firewall port for that URL needs to be Open.

$ openstack --os-token ADMIN --os-url http://192.168.1.58:35357/v2.0/ user list
+----------------------------------+----------+
| ID | Name |
+----------------------------------+----------+
| 042b50edf70f484dab1f14e893a73ea8 | admin |
| eb0d4dc081f442dd85573740cfbecfae | demo |
+----------------------------------+----------+
$ openstack --os-token ADMIN --os-url http://127.0.0.1:35357/v2.0/ role list
+----------------------------------+-----------------+
| ID | Name |
+----------------------------------+-----------------+
| 1f069342be2348ed894ea686706446f2 | admin |
| 2bf27e756ff34024a5a9bae269410f44 | service |
| dc4e9608b6e64ee1a918030f23397ae1 | Member |
+----------------------------------+-----------------+
$ openstack --os-token ADMIN --os-url http://192.168.1.58:35357/v2.0/ project list
+----------------------------------+--------------------+
| ID | Name |
+----------------------------------+--------------------+
| 69ff732083a64a1a8e34fc4d2ea178dd | demo |
| 7030f12f6cb4443cbab8f0d040ff023b | admin |
+----------------------------------+--------------------+

Now, to check to see if the admin user has a role on the admin project:

$ openstack --os-token ADMIN --os-url http://192.168.1.58:35357/v2.0/ user role list --project admin admin

+----------------------------------+-------+---------+-------+
| ID | Name | Project | User |
+----------------------------------+-------+---------+-------+
| 1f069342be2348ed894ea686706446f2 | admin | admin | admin |
+----------------------------------+-------+---------+-------+

If this returns nothing, you probably have found the root of your problem. Add the assignment with
$ openstack --os-token ADMIN --os-url http://192.168.1.58:35357/v2.0/ role add --project admin --user admin admin
+-------+----------------------------------+
| Field | Value |
+-------+----------------------------------+
| id | 1f069342be2348ed894ea686706446f2 |
| name | admin |
+-------+----------------------------------+
Not using IPv6? Are you sure?
World IPv6 Launch logo

CC-BY World IPv6 Launch

Internet Protocol version 6 (IPv6) has been around for many years and was first supported in Red Hat Enterprise Linux 6 in 2010.  Designed to provide, among other things, additional address space on the ever-growing Internet, IPv6 has only recently become a priority for ISPs and businesses.

On February 3, 2011, ICANN announced that the available pool of unallocated IPv4 addresses had been completely emptied and urged network operators and server owners to implement IPv6 if they had not already done so.  Unfortunately, many networks still do not support IPv6 and many system and network administrators don’t understand the security risks associated with not having some sort of IPv6 control within their networks setup even if IPv6 is not supported.  The common thought of not having to worry about IPv6 since it’s not supported on a network is a false one.

The Threat

On many operating systems, Red Hat Enterprise Linux and Fedora included, IPv6 is preferred over IPv4.  A DNS lookup will search first for an IPv6 address and then an IPv4 address.  A system requesting a DHCP allocation will, by default, attempt to obtain both addresses as well.  When a network does not support IPv6 it leaves open the possibility of rouge IPv6 DHCP and DNS servers coming online to redirect traffic either around current network restrictions or through a specific choke point where traffic can be inspected or both.  Basically, if you aren’t offering up IPv6 within your network someone else could.

Just like on an IPv4 network, monitoring IPv6 on the internal network is crucial for security, especially if you don’t have IPv6 rolled out.  Without proper monitoring, an attacker, or poorly configured server, could start providing a path way out of your network, bypassing all established safety mechanisms to keep your data under control.

Implementing IPv6

There are several methods for protecting systems and networks from attacks revolving around IPv6.  The simplest, and most preferred method, is to simply start using IPv6.  It becomes much more difficult for rouge DNS and DHCP servers to be implemented on a functioning IPv6 network.  Implementing IPv6 isn’t particularly difficult either.

Unfortunately IPv6 isn’t all the simple to implement either.  As UNC‘s Dr. Joni Julian spoke about in her SouthEast LinuxFest presentation on IPv6 Security, many of the tools administrators use to manage network connections have been rewritten, and thus renamed, to support IPv6.  This adds to the confusion when other tools, such as iptables, require different rules to be written to support IPv6.  Carnegie Mellon University’s CERT addresses many different facets of implementing IPv6 including ip6tables rules.  There are many resources available to help system and network administrators setup IPv6 on their systems and networks and by doing so networks will automatically be available to IPv6-only networks of the future present.

Blocking and Disabling IPv6

If setting up IPv6 isn’t possible the next best thing is disabling, blocking, and monitoring for IPv6 on the network.  This means disabling IPv6 in the network stack and blocking IPv6 in ip6tables.

# Set DROP as default policy to INPUT, OUTPUT, and FORWARD chains.
ip6tables -P INPUT DROP
ip6tables -P OUTPUT DROP
ip6tables -P FORWARD DROP

# Set DROP as a rule to INPUT and OUTPUT chains.
ip6tables -I INPUT -p all -j DROP
ip6tables -I OUTPUT -p all -j DROP

Because it can never known that every system on a network will be properly locked down, monitoring for IPv6 packets on the network is important.  Many IDSs can be configured to alert on such activity but configuration is key.

A few final words

IPv6 doesn’t have to be scary but if you want to maintain a secure network a certain amount of respect is required.  With proper monitoring IPv6 can be an easily manageable “threat”.  Of course the best way to mitigate the risks is to embrace IPv6.  Rolling it out and using it prevents many of the risks already discussed and it could already be an availability issue if serving up information over the Internet is important.

March 23, 2015

Firewalld rule for Minecraft Server

My sons play Minecraft.  I recently decided to let them play head to head on the same server.  Aside from the financial aspect (I had to buy a second account) it was fairly straightforward running the server.  The one thing that tripped me up was a firewall rule that prevented a remote client machine from connecting to the server.  Fix was pretty simple.

When running the server, the log showed:

[23:58:50] [Server thread/INFO]: Starting Minecraft server on *:25565

And so I knew that my firewalld configuration would block it. Killing firewalld and flushing the iptables rules confirmed it:

sudo systemctl stop firewalld.service
sudo iptables -F

But I don’t want to run without a firewall.

I want to open port 25565. To do so, I need to figure out what zone holds the firewall rule blocking it, and add a rule that opens this port.

$ firewall-cmd --get-active-zones
public
  interfaces: em1 tun0 virbr0 virbr1 virbr1-nic

Simple enough; I only have one zone (Fedora 21 default setup)

I want to only open this port when I fired up the game, I would probably be better off with a sudo rule that I embedded into the game startup script that opens the port dynamically, but I can do this by hand.

sudo firewall-cmd  --zone=public --add-port=25565/tcp

and then closes it upon shutdown:

sudo firewall-cmd  --zone=public --remove-port==25565/tcp

If I was setting up a machine to be a dedicated server, I would want this port to always be opened.

$ sudo firewall-cmd --permanent --zone=public --add-port=25565/tcp
success

Did that work?

$ sudo firewall-cmd --zone=public --query-port=25565/tcp
no

Not yet. So far I’ve only said that the port should be written down to be opened in general. I want this to be persisted.

$ sudo firewall-cmd --reload 
success
$ sudo firewall-cmd --zone=public --query-port=25565/tcp
yes

Now it is open and will be kept open. How: it gets written in to the firewalld config file. If you run

 sudo less /etc/firewalld/zones/public.xml

In there you should see a line that contains:

  port protocol="tcp" port="25565"

If you decide to disable the server and want to close the port:

$ sudo firewall-cmd --permanent --zone=public --remove-port=25565/tcp
success
$ sudo firewall-cmd --reload 
success
$ sudo firewall-cmd --zone=public --query-port=25565/tcp
no

What if we want to name this port? We know that the client must look for port 25565 Even if it isn’t in /etc/services. We can name this port “minecraft-server” at least for firewalld purposes. Create this file:

sudo vi /etc/firewalld/services/minecraft.xml
<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>minecraft</short>
  <description>Port used to allow remote connections to a Minecraft server running on this machine.</description>
  <port protocol="tcp" port="25565"/>
</service>

Now, instead of the above commands:

To query:

 sudo firewall-cmd --zone=public --query-service=minecraft

To Enable:

sudo firewall-cmd --zone=public --add-service=minecraft

And to Disable

sudo firewall-cmd  --zone=public --remove-service=minecraft

And use the –permanant flags and –reload if you want to make these changes survive a reboot.

March 18, 2015

CWE Vulnerability Assessment Report 2014

Last year is almost three months over and we have been busy completing the CWE statistics of our vulnerabilities. The biggest change from the year before is the scale of the data – CWE report for 2013 was based on 37 classified vulnerabilities, whereas last year we classified 617 vulnerabilities in our bugzilla. Out of them 61 were closed with resolution NOTABUG, which means they were either not a security issues, or did not affect Red Hat products. These still include vulnerabilities which affect Fedora or EPEL packages only – narrowing this down to vulnerabilities affecting at least one supported Red Hat product we end up with 479.

The graph below shows the Top 10 weaknesses in Red Hat software. Note the total sum is bigger than overall number of vulnerabilities, as one vulnerability may be result of multiple weaknesses. The most common case is CWE-190 Integer Overflow or Wraparound causing out-of-bounds buffer access problems.

The top spot is taken by cross site scripting with 36 vulnerabilities last year. However, closer examination reveals that despite the count, it was not very common. In fact, two packages had much more XSS flaws that the average: phpMyAdmin with 13 and Openstack (Horizon and Swift) with 7. The standard recommendation to the developers would immediately be to use one of the modern web frameworks, whether it be Ruby on Rails, Django or others.

The second place is occupied by Out-of-bound Read. Again, the distribution of vulnerabilities is not flat among packages, with xorg-x11-server having 9 and chromium-browser 5 vulnerabilities of this type last year. All of the xorg-x11-server come from a single security advisory released on 2014-12-09, which fixed flaws reported by Ilja van Sprundel. The results of his security research of X, which lead to discovery of the flaws, were presented on CCC in 2013. His presentation is a great intoduction into X security problems and is still available.

From the above we could hypothesize that the statistics are dominated by a smaller set of very vulnerable packages, or that certain packages are be prone to certain kinds of weaknesses. The graph below shows a number of vulnerabilities that affected each of the packages – vulnerabilities which did not affected versions of packages we ship are excluded.

Median value of vulnerabilities per package is 1, however, not all packages are equal. Looking at the top 20, all of the packages contain large codebases, some of which are a separate product of an upstream vendor. We should not make a mistake of misinterpreting this graph as Top 20 most vulnerable projects, as it would be more fair to compare apples with apples e.g. kernel (package) with Openstack (which we ship as a product). More honest interpretation would be to see it as a list of packages that increase the attack surface of the system the most when installed.

If we look at statistics per-product, Red Hat Enterprise Linux dominates just by including vast number of packages. The distribution of weaknesses is therefore very close to the overall one show on the first graph above. However, if we look at the top 5 weaknesses in RHEL 5, 6 and 7, we can see a statistically significant drop in number of use after free types of vulnerabilities.

The root cause of this has been traced back to our source code analysis group and the mass scans performed on the Fedora versions prior to RHEL 7 rebase. These scans were performed using a couple of source code analysis tools including Coverity and cppcheck and the warnings were addressed as normal bugs. This explanation is also supported by the decreasing number of found use-after-frees in Fedora from versions 17 to 19, which served as basis for RHEL 7. Interestingly, other weaknesses like buffer access problems and overflows are unaffected, which is probably combination of a) inherent difficulty of their detection via code analysis and b) large number of false positives, making the developers less inclined to address these types of warnings.

The two most common weaknesses in Openshift Enterprise are Information Exposure and Cross Site Scripting. A closer look tells a different story – 5 out of 6 information exposure vulnerabilities were found in Jenkins, shipped as part of the Openshift product. In fact, surprising 21 out of 60 vulnerabilities that affected Openshift product were present in Jenkins. On the other hand, just 9 vulnerabilities were found in core Openshift components.

Interestingly, the distribution of vulnerable components in Openstack is more flat with no component standing out. CWE-400 Uncontrolled Resource Consumption (‘Resource Exhaustion’) is the most common weakness and all of the vulnerabilities affect core Openstack components. Number of vulnerabilities in Keystone related to session expiration (4) is also surprising, as we haven`t seen many vulnerabilities of that type in other packages last year.

Other products and components also tend to have their specific weaknesses: external entity expansion for Java/JBoss based products, out of bounds reads in Freetype, use after free in Mozilla etc. Overall the depth of the data is much bigger and provides new possibilities for the proactive research. Having more precise data for the feedback loop allowing us to both evaluate past measures and propose future ones is next step towards more efficient proactive security. Unfortunately, the time it takes for any countermeasures to make a dent in statistics is measured in releases, so this data will become much more interesting as they change in time.

March 12, 2015

Postfix Encryption

I’ve been tinkering with the encryption options in Postfix for a while.  Encryption between clients and their SMTP server and between SMTP servers is necessary to protect the to, from, and subject fields, along with the rest of the header, of an email.  The body of the message is also protected but it’s always better to utilize PGP or S/MIME cryptography to provide end-to-end protection; encryption between clients and SMTP servers doesn’t provide this.

As rolled out now, encryption between SMTP servers is opportunistic encryption and is generally not required.  While doing a review of my mail log I seem to be receiving most personal mail via some encrypted circuit while much of the mail coming out of listservs, like Yahoo! Groups, is not negotiating encryption on connect.  I’ve also noticed that some email providers actually run their incoming email through an external service, I suspect for spam control, before accepting the message into their servers.  Some of these spam services don’t support encryption making it difficult to protect mail in transit.

Postfix documentation is pretty decent.  The project seems to document most settings but sometimes they don’t actually put the entire picture together.  Encryption is one of those things where a complete picture is difficult to put together just by looking at a single page of documentation.

Postfix’s documentation on TLS is fairly complete.  What they miss on that page, forward security, must be found else where.  Until last night, I had missed that last page and now have fixed my configuration to include, what I consider, acceptable settings.

Here’s what I’ve got:

main.cf

### TLS
# enable opportunistic TLS support in the SMTP server
smtpd_tls_security_level = may
smtpd_tls_eecdh_grade = ultra
tls_eecdh_strong_curve = prime256v1
tls_eecdh_ultra_curve = secp384r1
smtpd_tls_loglevel = 1
smtpd_tls_cert_file = /etc/pki/tls/certs/mail.crt
smtpd_tls_key_file = /etc/pki/tls/private/mail.key
smtpd_tls_CAfile = /etc/pki/tls/certs/mail-bundle.crt
smtpd_tls_session_cache_timeout = 3600s
smtpd_tls_session_cache_database = btree:${queue_directory}/smtpd_scache
smtpd_tls_received_header = yes
smtpd_tls_ask_ccert = yes
smtpd_tls_received_header = yes
tls_random_source = dev:/dev/urandom
#TLS Client
smtp_tls_security_level = may
smtp_tls_eecdh_grade = ultra
smtp_tls_loglevel = 1
smtp_tls_cert_file = /etc/pki/tls/certs/mail.crt
smtp_tls_key_file = /etc/pki/tls/private/mail.key
smtp_tls_CAfile = /etc/pki/tls/certs/mail-bundle.crt

master.cf

submission inet n       –       –       –       –       smtpd
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_type=dovecot
-o smtpd_sasl_path=private/auth
-o smtpd_sasl_security_options=noanonymous

Those familiar with setting up TLS in Apache will notice a few differences here.  We haven’t defined ciphers or SSL protocols.  This is because this is opportunistic encryption.  We’re just happy if encryption happens, even using EXPORT ciphers, since the alternate is plaintext.  In a more controlled setting you could define the ciphers and protocols and enforce their use.  Until encryption becomes the norm on the Internet (and why shouldn’t it be?) I’ll have to stick with just begging for encrypted connections.

It should also be noted that client-to-SMTP server connections are forced to be encrypted in master.cf as seen in the submission portion.  This was a quick and dirty way of forcing encryption on the client side while allowing opportunistic encryption on the public (port 25) side.

It should be noted that ECC keys can be used with Postfix, which forces good ciphers and protocols, but most email servers have RSA keys established so problems could arise from that.  Dual keys can always be used to take advantage of both ECC and RSA.

As SSLLabs is for testing your web server’s encryption settings, so is CheckTLS for checking your SMTP encryption settings.  These tools are free and should be part of your regular security check of your infrastructure.


USB Killer (or maybe it’s a killer via USB?)

A co-worker passed this along to me and I felt this was worthy of further dissemination.

http://kukuruku.co/hub/diy/usb-killer

And this, my friends, is why you shouldn’t just plug in random, unknown USB devices.


Emilio’s Craftsmanship

My Saxophone is back from the workshop of Emilio Lyons.  It is a pleasure to play on it.  I would say “like new” but for two things.  First, the horn was twenty years old when I got it, so I never played it new.  Second, Emilio has customized the feel of the horn enough that o suspect it never played like this.  What did he do?

Back when I dropped off my tenor, I pointed out issues with both of the thumbs.  The lower thumb rest was a hard plastic hook that was too sharp curve for my finger.  He showed me then the replacement. He would use. Here it is installed.

image

It is much smoother.  I love the feel of the brass and the wider contact platform.  This is one benefit of working with someone with such long experience; he has older replacement parts hoarded,  some of which are quite unique.  He kept my old thumb piece in case someone else needed it in the future.

The other thumb rests very near the access hole to one of the rods.  I always found it irritating to my thumb.  Here in what he added to make it comfortable.

image

The black disk was the original.  He added the triangular shaped piece to the left. Note how he still made the screw for the rod accessible.

Right hand finger adjustment.

image

Left hand key adjustments.  Compared to a stock setup, this makes the notes much easier and faster to play.

image
All new pads.  Coverage is fantastic.  Low notes come out as cleanly as I’ve ever got them to sound.

image

Here are the replacements for the corks that prevent metal on metal during the key movements.  He replaced all the corks with rubber.  Emilio is very materials focused, and was looking for an improvement to cork for a long time.  He found in the little nubs that come on the bottom side of a carpet.  They are almost silent when tapped on the horn body.

image

Neck cork.  Very precise for my mouthpiece.

image

Tightened the neck.  You can’t see from the pictures,  but I can feel it.

image

Serial number for recorded posterity.

image

He replaced pretty much everything designed to be replaced except for the springs,  those are still the original.

The horn.

image

Emilio does not play saxophone himself.  To test it out, he had another customer, a very good professional tenor player, try it out “He liked that horn a lot.”

 

Trade marks.

image

 

 

March 11, 2015

CWE update

In the past Red Hat Product Security assigned weakness IDs only to vulnerabilities that meet certain criteria, more precisely, only vulnerabilities with CVSS score higher than 7. Since the number of incoming vulnerabilities was high, this filtering allowed us to focus on vulnerabilities that matter most. However, it also makes statistics incomplete, missing low and moderate vulnerabilities.

In the previous year we started assigning weakness IDs to almost all vulnerabilities, greatly increasing the quantity of data used to generate statistics. This was a big commitment time-wise, but resulted in 13 times more vulnerabilities with assigned weakness IDs in 2014 than the year before. There are a few exceptions – for some vulnerabilities there are not enough information available to decide the types of weaknesses. These almost always come from big upstream vendors. For this reason bugs in mysql or OpenJDK do not have weaknesses assigned and are excluded from the CWE statistics. With the exceptions mentioned, there are always at least references to commits that fix the vulnerability available, so it is possible to assign correct weakness data to vulnerabilities in any open source project.

Part of using Common Weakness Enumeration (CWE) at Red Hat is CWE Coverage – a subset of weaknesses that we use to classify vulnerabilities. As everyone can notice after scrolling through the CWE list there are a lot of weaknesses that are very similar or describe the same issue in varying level of detail. This means different people can assign different weaknesses to the same vulnerability, a very undesirable outcome. Furthermore, this may skew resulting statistics, as vulnerabilities of the same nature may be described by different weaknesses. To counter these effects, Red Hat keeps CWE coverage, a subset of weaknesses we use, to prevent both. The coverage should contain weaknesses with similar level of detail (Weakness Base) and should not contain multiple overlapping weaknesses. However there is a possibility that a vulnerability would not fit into any of the weaknesses in our coverage and for this reason the coverage is regularly updated.

Maintenance of CWE coverage has been tied with the release of new CWE revisions by MITRE in past. Since we started assigning weakness IDs to much larger number of vulnerabilities we also gathered weaknesses missing in the coverage more quickly. Therefore the coverage has been updated and the changes are now included in the statistics. Current revision of Red Hat`s CWE Coverage can be found on the Customer Portal.

Apart from adding missing weaknesses we also removed a number of unused or unsuitable weaknesses. The first version of coverage was based on CWE Cross-Section maintained as view by MITRE. The CWE Cross-Section represents a subset of weaknesses at the abstraction level most useful for general audiences. While this was a good starting point, it quickly became evident that the Cross-Section has numerous deficiencies. Some of the most common weaknesses are not included, for example CWE-611 Improper Restriction of XML External Entity Reference (‘XXE’), which ranked as 10th most common weakness in our statistics for 2014. On the other hand, we have not included considerable number of weaknesses that were not relevant in open source, for example CWE-546 Suspicious Comment. After these changes current revision of the coverage has little in common with CWE Cross-Section, but represents structure of weaknesses usually specific to open source projects well.

Last but not least, all CWE related data are kept public and statistics (even for our internal use) are generated only from publicly available data.The weakness ID is stored in whiteboard of a vulnerability in bugzilla. This is rather cryptic format and requires tooling to get the statistics into a format that can be processed. Therefore, we are currently investigating the best way how to make the statistics available online for wider audience.

March 08, 2015

CERN cares about information security… what about you?

As a security engineer it’s usually difficult for me to endure many of dumb things companies do.  It’s quite sad when a company that prides itself on creating solutions for building internal solutions to protect customer data actually starts pushing its own data out to Google and other “solution” providers.  It’s as if they don’t actually believe in their own products and actually think that a contract will actually protect their data.

So it’s quite refreshing when you run across a group that actually gets information security.  Recently, I ran across the information security bulletins at CERN (particle physics is another interest of mine) and was excited to find a group that actually gets it.  They provide internal, secure solutions for getting their work done without using outside solutions such as Google, Apple, Microsoft, Amazon, and Dropbox cloud solutions (I wish more of the internal solutions were FOSS but…).  In fact, CERN feels externally-hosted solutions are a bad idea for both business and personal uses.  I concur.

Here is a sample of their infosec bulletins:

What about you?  Do you care about the security of your information?


March 07, 2015

Keystone Federation via mod_lookup_identity redux

Last year I wrote a proof-of-concept for Federation via mod_lookup_identity. Some of the details have changed since then, and I wanted to do a formal one based on the code that will ship for Kilo. This was based on a devstack deployment.

UPDATE: Looks like I fooled myself: this only maps the first group. There is a patch outstanding that allows for lists of groups, and that is required to really make this work right.


The Configuration of SSSD and mod_lookup_identity stayed the same.
although the sssd-dbus RPM is already installed in F21.

Here is my devstack /opt/stack/devstack/local.conf

[[local|localrc]]
ADMIN_PASSWORD=<not gonna="gonna" tell="tell" you="you">
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=password
USE_SSL=True

ENABLED_SERVICES="$ENABLED_SERVICES,-tempest,-h-api,-h-eng,-h-api-cfn,-h-api-cw"

Tempest didn’t like SSL. That is a a recurring problem, and something we need to fix by making SSL the default.
I disabled Heat, too. Nothing against Heat, but I needed to speed up the install, and that was the easiest to leave off.

I’m getting a token with a request that looks like this:

#!/usr/bin/bash

OS_AUTH_URL=https://devstack.ayoung530.younglogic.net/keystone/sss


curl -v  \
-k \
-H "Content-Type:application/json" \
--negotiate -u : \
--cacert ca.crt  \
-d  '{ "auth": { "identity": { "methods": ["kerberos"], "kerberos":{"identity_provider":"sssd", "protocol":"sssd_kerberos"}}, "scope": { "unscoped": { } } } }' \
-X POST $OS_AUTH_URL/v3/auth/tokens

This is due to using the following in the keystone.conf:

[auth]
methods = external,password,token,kerberos

kerberos =  keystone.auth.plugins.mapped.Mapped

This implies that we will want to be able to put Federation data into the kerberos auth plugin for the client.

The trickiest part was getting the mapping right.    I’ve added the mapping to the bottom of this email.

To set up the call, I used the openstack client. After sourcing openrc:

export OS_AUTH_URL=https://192.168.122.182:5000/v3
export OS_USERNAME=admin

openstack --os-identity-api-version=3 group create admins
openstack --os-identity-api-version=3 group create ipausers
openstack --os-identity-api-version=3    identity provider create sssd
openstack --os-identity-api-version=3   mapping create  --rules /home/ayoung/kerberos_mapping_edited.json  kerberos_mapping
openstack --os-identity-api-version=3 federation protocol create --identity-provider sssd --mapping kerberos_mapping sssd_kerberos
       [
            {
                "local": [
                    {
                        "user": {
                            "name": "{0}",
                            "id": "{0}"
                        }
                    }
                ],
                "remote": [
                    {
                        "type": "REMOTE_USER"
                    }
                ]
            },

            {
                "local": [
                    {
                        "group": {
                            "name": "{0}",
                            "domain": {"name": "Default"}
                        }
                    }
                ],
                "remote": [
                    {
                        "type": "REMOTE_USER_GROUPS"
                    }
                ]
            }

        ]

My config for HTTPD Keystone looks like this:

LoadModule lookup_identity_module modules/mod_lookup_identity.so

WSGIDaemonProcess keystone-sss processes=5 threads=1 user=ayoung display-name=%{GROUP}
WSGIProcessGroup keystone-sss
WSGIScriptAlias /keystone/sss  /var/www/keystone/admin


WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
ErrorLogFormat "%{cu}t %M"
ErrorLog /var/log/httpd/keystone.log
CustomLog /var/log/httpd/keystone_access.log combined
SSLEngine On
SSLCertificateFile /opt/stack/data/CA/int-ca/devstack-cert.crt
SSLCertificateKeyFile /opt/stack/data/CA/int-ca/private/devstack-cert.key


<location /keystone/sss>
  AuthType Kerberos
  AuthName "Kerberos Login"
  KrbMethodNegotiate on
  KrbMethodK5Passwd off
  KrbServiceName HTTP
  KrbAuthRealms AYOUNG530.YOUNGLOGIC.NET
  Krb5KeyTab /etc/httpd/conf/openstack.keytab
  KrbSaveCredentials on
  KrbLocalUserMapping on
  Require valid-user
  SSLRequireSSL
  LookupUserAttr mail REMOTE_USER_EMAIL " "
  LookupUserGroups REMOTE_USER_GROUPS ";"
</location>

I had to pre-create all groups from the mapping due to https://bugs.launchpad.net/keystone/+bug/1429334

March 04, 2015

Convince Nova to Use the V3 version of the API

In a recent post I showed how to set up the LDAP in a domain other than default. It turns out that the Nova configuration does accept these tokens; by default, Nova uses the V2 version of the Keystone API only. This is easy to fix.

The first indication that something was wrong was that Horizon threw up a warning

Cannot Fetch Usage Information.

It turns out that all Operations against Nova were failing.
The Default for Auth token should be to perform discovery to see what version of the Keystone API is supported. However, Nova seems to have a configuration override that defaults the value to the V2.0 API. Looking in /etc/nova/nova.conf

I saw:

#auth_version=V2.0

Setting this to

auth_version=

And restarting all of the services fixed the problem.

Factoring RSA export keys – FREAK (CVE-2015-0204)

This week’s issue with OpenSSL export ciphersuites has been discussed in the press as “Freak” and “Smack”. These are addressed by CVE-2015-0204, and updates for affected Red Hat products were released in January.

Historically, the United States and several other countries tried to control the export or use of strong cryptographic primitives. For example, any company that exported cryptographic products from the United States needed to comply with certain key size limits. For RSA encryption, the maximum allowed key size was 512 bits and for symmetric encryption (DES at that time) it was 40 bits.

The U.S. government eventually lifted this policy and allowed cryptographic primitives with bigger key sizes to be exported. However, these export ciphersuites did not really go away and remained in a lot of codebases (including OpenSSL), probably for backward compatibility purposes.

It was considered safe to keep these export ciphersuites lying around for multiple purposes.

  1. Even if your webserver supports export ciphersuites, most modern browsers will not offer that as a part of initial handshake because they want to establish a session with strong cryptography.
  2. Even if you use export cipher suites, you still need to factor the 512 bit RSA key or brute-force the 40-bit DES key. Though doable in today’s cloud/GPU infrastructure, it is pointless to do this for a single session.

However, this results in a security flaw, which affects various cryptographic libraries, including OpenSSL. OpenSSL clients would accept RSA export-grade keys even when the client did not ask for export-grade RSA. This could further lead to an active man-in-the-middle attack, allowing decryption and alteration of the TLS session in the following way:

  • An OpenSSL client contacts a TLS server and asks for a standard RSA key (non-export).
  • A MITM intercepts this requests and asks the server for an export-grade RSA key.
  • Once the server replies, the MITM attacker forwards this export-grade RSA key to the client. The client has a bug (as described above) that allows the export-grade key to be accepted.
  • In the meantime, the MITM attacker factors this key and is able to decrypt all possible data exchange between the server and the client.

This issue was reported to OpenSSL in October 2014, fixed in public in OpenSSL in January 2015, and shipped in Red Hat Enterprise Linux 6 and 7 two week later via RHSA-2015-0066. This issue has also been addressed in Fedora 20 and Fedora 21.

Red Hat Product Security initially classified this as having low security impact, but after more details about the issue and the possible attack scenarios have become clear, we re-classified it as a moderate-impact security issue.

Additional information on mitigating this vulnerability can be found on the Red Hat Customer Portal.

[Updated 17th March 2014: the original article stated this issue was fixed in OpenSSL in October 2014, however the fix was not public until January 2015.  We have updated the article to clarify this].

February 26, 2015

Three Types of Tokens

One of the most annoying administrative issues in Keystone is The MySQL backend to the token database filling up. While we have a flush scrit, it needs to be scheduled via cron. Here is a short over view of the types of tokens, why the backend is necessary, and what is being done to mitigate the problem.

DRAMATIS PERSONAE:

Amanda: The companies OpenStack system Admin

Manny: The IT manager.

ACT 1 SCENE1: Small conference room. Manny has called a meeting with Amanda.

Manny: Hey Amanda, What are these keystone tokens and why are they causing so many problems?

Amanda: Keystone tokens are an opaque blob used to allow caching of an authentication event and some subset of the authorization data associated with the user.

Manny: OK…backup. what does that mean?

Amanda: Authentication means that you prove that you are who you claim to be. For the most of OpenStack’s history, this has meant handing over a symmetric secret.

Manny: And a symmetric secret is …?

Amanda: A password.

Manny:Ok Got it. I hand in my password to prove that I am me. What is the authorization data?

Amanda: In OpenStack, it is the username and the user’s roles.

Manny: All their roles?

Amanda: No. only for the scope of the token. A token can be scoped to a project. Also to a domain, but in our set up, only I ever need a domain scoped token.

Manny: The domain is how I select between the customer list and our employees out of our LDAP server, right?

Amanda: Yep. There is another domain just for admin tasks, too. It has the service users for Nova and so on.

Manny: OK, so I get a token, and I can see all this stuff?

Amanda: Sort of. For most of the operation we do, you use the “openstack” command. That is the common command line, and it hides the fact that it is getting a token for most operations. But you can actually use a web tool called curl to go direct to the keystone server and request a token. I do that for debugging sometimes. If you do that, you see the body of the token data in the response. But that is different from being able to read the token itself. The token is actually only 32 characters long. It is what is known as a UUID.

Manny (slowly): UUID? Universally Unique Identifier. Right?

Amanda: Right. Its based on a long random number generated by the operating system. UUIDs are how most of OpenStack generates remote identifiers for VMs, images, volumes and so on.

Manny: Then the token doesn’t really hold all that data?

Amanda: It doesn’t. The token is just a…well, a token.

Manny: Like we used to have for the toll machines on route 93. Till we all got Easy pass!

Amanda: Yeah. Those tokens showed that you had paid for the trip. For OpenStack, a token is a remote reference to a subset of your user data. If you pass a token to Nova, it still has to go back to Keystone to validate the token. When it validates the token, it gets the data. However, our OpenStack deployment is so small, Nova and Keystone are on the same machine. Going back to Keystone does not require a “real” network round trip.

Manny: So when now that we are planning on going to the multi host set up, validating a token will require a network round trip?

Amanda: Actually, when we move to the multi-site, we are going to switch over to a different form of token that does not require a network round trip. And that is where the pain starts.

Manny: These are the PKI tokens you were talking about in the meeting?

Amanda: Yeah.

Manny: OK, I remember the term PKI was Public Key…something.

Amanda: The I is for infrastructure, but you remembered the important part.

Manny: Two keys, Public versus private: you encode with one and decode with the other.

Amanda: Yes. In this case, it is the token data that is encoded with private key, and decode with the public key.

Manny: I thought that made it huge. Do you really encode all the data?

Amanda: No, just a signature of the data. A Hash. This is called message signing, and it is used in a lot of places, basically to validate that the message is both unchanged and that it comes from the person you think it comes from.

Manny: OK, so…what is the pain.

Amanda: Two things. One, the tokens are bigger, much bigger, than a UUID. They have all of the validation data in them. To include the service catalog. And our service catalog is growing on the multi-site deployment, so we’ve been warned that the tokens might get so big that it causes problems.

Manny: Let’s come back to that. What is the other problem?

Amanda: OK…since a token is remotely validated, there is the possibility that something hass changed on Keystone, and the token is no longer valid. With our current system, Keystoen knows this immediately, and just dumps the token. So When Nova comes to validate it, its no longer valid and the user has to get another token. With remove validation, Nova has to periodically request a list of revoked tokens.

Manny: So either way Keystone needs to store data. What is the problem?

Amanda: Well, today we store our tokens in Memcached. Its a simple Key value store, its local to the Keystone instance, and it just dumps old data that hasn’t been used in a while. With revocations, if you dump old data, you might lose the fact that a token was revoked.

Manny: Effectively un-revoking that token?

Amanda: Yep.

Manny: OK…so how do we deal with this?

Amanda: We have to move from storing token in Memcached to MySQL. According to the docs and upstream discussions, this can work, but you have to be careful to schedule a job to clean up the old tokens, or you can fill up the token database. Some of the larger sites have to run this job very frequently.

Manny: Its a major source of pain?

Amanda: It can be. We don’t think we’ll be at that scale at the multisite launch, but it might happen as we grow.

Manny: OK, back to the token size thing, then. How do we deal with that?

Amanda: OK, when we go multi-site, we are going to have one of everything at each site: Keystone, Nova, Neutron, Glance. We have some jobs to synchronize the most essential things like the glance images and the customer database, but the rest is going to kept fairly separate. Each will be tagged as a region.

Manny: So the service catalog is going to be galactic, but will be sharded out by Keystone server?

Amanda: Sort of. We are going to actually make it possible to have the complete service catalog in each keystone server, but there is an option in Keystone to specify a subset of the catalog for a given project. So when you get a token, the service catalog will be scoped down to the project in question. We’ve done some estimates of size and we’ll be able to squeak by.

Manny: So, what about the multi-site contracts? Where a company can send there VMs to either a local or remote Nova?

Amanda: for now they will be separate projects. But for the future plans where we are going to need to be able to put them in the same project, we are stuck.

Manny: Ugh. We can’t be the only people with this problem.

Amanda: Some people are moving back to UUID tokens, but there are issues both with replication of the token database and also with cross site network traffic. But there is some upstream work that sounds promising to mitigate that.

Manny: The lightweight thing?

Amanda: Yeah, lightweight tokens. Its backing off the remotely validated aspect of Keystone tokens, but doesn’t need to store the tokens themselves. They use a scheme called Authorized Encryption which puts a minimal amount of info into the token to be able to recreate the whole authorization data. But only the Keystone server can expand that data. Then, all that needs to be persisted is the revocations.

Manny: Still?

Amanda: Yeah, and there are all the same issues there with flushing of data, but the scale of the data is much smaller. Password changes and removing roles from users are the ones we expect to see the most. We still need a cron job to flush those.

Manny: No silver bullet, eh? Still how will that work for multisite?

Amanda: Since the token is validated by cryptography, the different sites will need to synchronize the keys. There was a project called Kite that was part of Keystone, and then it wasn’t, and then it was again, but it is actually designed to solve this problem. So all of the Keystone servers will share their keys to validate tokens locally.

Manny: We’ll still need to synchronize the revocation data?

Amanda: No silver bullet.

Manny: Do we really need the revocation data? What if we just … didn’t revoke. Made the tokens short lived.

Amanda: Its been proposed. The problem is that a lot of the workflows were being built around the idea of long lived tokens. The Tokens went from 24 hours valid to 1 hour valid by default, and that broke some things. Some people have had to crank the time back up again. We think we might be able to get away with shorter tokens, but we need to test and see what it breaks.

Manny: Yeah, I could see HA having a problem with that…wait, 24hours…how does heat do what it needs to. It can restart a machine a mong afterwards. DO we just hand over the passwords to HEAT?

Amanda: Heh..used to,. But Heat uses a delegation mechanism called trusts. A user creates a trust, and that effectively says that Heat can do something on the users behalf, but Heat has to get its own token first. It first proves that it is Heat, and then it uses the trust to get a token on the users behalf.

Manny: So…trusts should be used everywhere?

Amanda: Something like trusts, but more lightweight. Trusts are deliberate delegation mechanisms, and a re set op on a per user bases. TO really scale, it would have to be something where the admin set up the delegation agreement as a template. If that were the case, then these long lived work flows would not need to use the same token.

Manny: And we could get rid of the revocation events. OK, that is time, and I have a customer meeting. Thanks.

Amanda: No problem.

EXIT

The Sax Doctor
Distinctive ring that holds the bell of a Selmer Paris Saxophone to the main body of the horn.

Ring of a Selmer Paris Saxophone

Dropped my Sax off at Emilio Lyon’s house and workshop. My folks bought it for me from him at Rayburn Music in Boston back when I was a High School Freshman. I still remember him pointing to the sticker on it that indicated “This is my work.”

As someone who loves both the saxophone and working with my hands, I have to admit I was looking forward to meeting him. I was even a little nervous. He has a great reputation. Was he going to chastise me for the state of my horn? It hadn’t been serviced in…way too long. I was a little worried that the lack of changing the oil on the rods would have worn down some of the metal connections.

I spent the best forty minutes in his workshop as he explained what the horn needed; an overhaul, which meant pulling all the pads off and replacing them. I expected this.

I showed him how the bottom thumb rest didn’t fit my hand right…it was the stock Selmer piece, and it had always cut into my thumb a little. He had another from a different, older horn, that was brass. He shaped it with a hammer and .. it felt good. Very good. He gave me that piece and kept mine, in case it would work for someone else.

There was another minor issue with the left thumb catching a rough spot near the thumb rest and he covered it with some epoxy. Not magic, but a magic touch none-the-less.

To say he told me it needed an overhaul doesn’t do it justice. He explained how he would do it, step by step, especially the task of setting the pads. I know from elsewhere that this is real artistry, and takes years of experience to get right. He talked about taking the springs off and leaving the pads resting on the cups. I asked why he took the springs off?

Its about touch. You shouldn’t work hard to cover the holes of the saxophone, it should be light. Emilio understands how the top saxophonists in the world interact with their horns. He talked about advising Sonny Rollins to use a light touch “how he would like playing the horn better. Sonny tried for a week and then called back to apologize: he just couldn’t play that way. We talked about how other people played..Joe Lovano and George Garzone, heavy. David Sanborne, very light.

The corks and felts all need to be replaced. He has a new technique, I had heard about, using little black rubber nubs. He showed me, and how quiet they were. “I never liked the base.” I think he meant the base set of padding that came with the Selmers.

He assured me that the metal was fine. This is a good horn. A good number.

He quoted me the price for the overhaul. I told was less than other people had quoted me.

He didn’t take any contact info, told me to contact him. I get my horn back in two weeks. I’ll make due with playing my beat on student horn alto and EWI.

I’m really looking forward to getting back my Selmer Mark VI with the overhaul from Emilio. Will it play like new? I don’t know, the horn was at least 6 years old by the time I was born, and 20 when I first played it.

But I suspect it will play better than new.

February 25, 2015

Common Criteria

What is Common Criteria?

Common Criteria (CC) is an international standard (ISO/IEC 15408) for certifying computer security software. Using Protection Profiles, computer systems can be secured to certain levels that meet requirements laid out by the Common Criteria. Established by governments, the Common Criteria treaty agreement has been signed by 17 26 countries, and each country recognizes the other’s certifications.

In the U.S., Common Criteria is handled by the National Information Assurance Partnership (NIAP). Other countries have their own CC authorities. Each authority certifies CC labs, which do the actual work of evaluating products. Once certified by the authority, based on the evidence from the lab and the vendor, that certification is recognized globally.

Your certification is given a particular assurance level which, roughly speaking, represents the strength of the certification. Confidence is higher at a level EAL4 than at EAL2 for a certification. Attention is usually given to the assurance level, instead of what, specifically, you’re being assured of, which is the protection profiles.
CC certification represents a very specific set of software and hardware configurations. Software versions and hardware model and version is important as differences will break the certification.

How does the Common Criteria work?

The Common Criteria authority in each country creates a set of expectations for particular kinds of software: operating systems, firewalls, and so on. Those expections are called Protection Profiles. Vendors, like Red Hat, then work with a third-party lab to document how we meet the Protection Profile. A Target of Evaluation (TOE) is created which is all the specific hardware and software that’s being evaluated. Months are then spent in the lab getting the package ready for submission. This state is known as “in evaluation”.
Once the package is complete, it is submitted to the relevant authority. Once the authority reviews and approves the package the product becomes “Common Criteria certified” for that target.

Why does Red Hat need or want Common Criteria?

Common Criteria is mandatory for software used within the US Government and other countries’ government systems. Other industries in the United States may also require Common Criteria. Because it is important to our customers, Red Hat spends the time and energy to meet these standards.

What Red Hat products are Common Criteria certified?

Currently, Red Hat Enterprise Linux (RHEL) 5.x and 6.x meet Common Criteria in various versions. Also, Red Hat’s JBoss Enterprise Application Platform 5 is certified in various versions. It should be noted that while Fedora and CentOS operating systems are related to RHEL, they are not CC certified. The Common Criteria Portal provides information on what specific versions of a product are certified and to what level. Red Hat also provides a listing of all certifications and accreditation of our products.

Are minor releases of RHEL certified?

When a minor release, or a bug fix, or a security issue arises, most customers will want to patch their systems to remain secure against the latest threats. Technically, this means falling out of compliance. For most systems, the agency’s Certifying Authority (CA) requires these updates as a matter of basic security measures. It is already understood that this breaks CC.

Connecting Common Criteria evaluation to a specific minor versions is difficult, at best, for a couple of reasons:

First, the certifications will never line up with a particular minor version exactly. A RHEL minor version is, after all, just a convenient waypoint for what is actually a constant stream of updates. The CC target, for example, began with RHEL 6.2, but the evaluated configuration will inevitably end up having packages updated from their 6.2 versions. In the case of FIPS, the certifications aren’t tied to a RHEL version at all, but to the version of the certified package. So OpenSSH server version 5.3p1-70.el6 is certified, no matter which minor RHEL version you happen to be using.

This leads to the second reason. Customers have, in the past, forced programs to stay on hopelessly outdated and unpatched systems only because they want to see /etc/redhat-release match the CC documentation exactly. Policies like this ignore the possibility that a certified package could exist in RHEL 6.2, 6.3, 6.4, etc., and the likelihood that subsequent security patches may have been made to the certified package. So we’re discouraging customers from saying “you must use version X.” After all, that’s not how CC was designed to work. We think CC should be treated as a starting point or baseline on which a program can conduct a sensible patching and errata strategy.

Can I use a product if it’s “in evaluation”?

Under NSTISSP #11, government customers must prefer products that have been certified using a US-approved protection profile. Failing that, you can use something certified under another profile. Failing that, you must ensure that the product is in evaluation.

Red Hat has successfully completed many Common Criteria processes so “in evaluation” is less uncertain than it might sound. When a product is “in evaluation”, the confidence level is high that certification will be awarded. We work with our customers and their CAs and DAAs to help provide information they need to make a decision on C&A packages that are up for review.

I’m worried about the timing of the certification. I need to deploy today!

Red Hat makes it as easy as possible for customers to use the version of Red Hat Enterprise Linux that they’re comfortable with. A subscription lets you use any version of the product as long as you have a current subscription. So you can buy a subscription today, deploy a currently certified version, and move to a more recent version once it’s certified–at no additional cost.

Why can’t I find your certification on the NIAP website?

Red Hat Enterprise Linux 6 was certified by BSI under OS Protection Profile at EAL4+. This is equivalent to certifying under NIAP under the Common Criteria mutual recognition treaties. More information on mutual recognition can be found on the CCRA web site. That site includes a list of the member countries that recognize one another’s evaluations.

How can I keep my CC-configured system patched?

A security plugin for the yum update tool allows customers to only install patches that are security fixes. This allows a system to be updated for security issues while not allowing bug fixes or enhancements to be installed. This makes for a more stable system that also meets security update requirements.

To install the security plugin, from a root-authenticated prompt:

# yum install yum-plugin-security
# yum updateinfo
# yum update --security

Once security updates have been added to the system, the CC-evaluated configuration has changed and the system is no longer certified.  This is the recommended way of building a system: starting with CC and then patching in accordance with DISA regulations. Consulting the CA and DAA during the system’s C&A process will help establish guidelines and expectations.

You didn’t answer all my questions. Where do I go for more help?

Red Hat Support is available anytime a customer, or potential customer, has a question about a Red Hat product.

Additional Reading

February 24, 2015

What Can We Do About Superfish?

Perhaps the greatest question about Superfish is what can we do about it. The first response is to throw technology at it.

The challenge here is that the technology used by Superfish has legitimate uses:

  • The core Superfish application is interesting – using image analysis to deconstruct a product image and search for similar products is actually quite ingenious! I have no reservations about this if it is an application a user consciously selects and installs and deliberately uses.
  • Changing the html data returned by a web site has many uses – for example, ad blocking and script blocking tools change the web site. Even deleting tracking cookies can be considered changing the web site! Having said that, changing the contents of a web site is a very slippery slope. And I have real problems with inserting ads in a web site or changing the content of the web site without making it extremely clear this is occurring.
  • Reading the data being exchanged with other sites is needed for firewalls and other security products.
  • Creating your own certificates is a part of many applications. However, I can’t think of many cases where it is appropriate to install a root certificate – this is powerful and dangerous.
  • Even decrypting and re-encrypting web traffic has its place in proxies, especially in corporate environments.

The real problem with Superfish is how the combination of things comes together and is used. And quality of implementation – many reports indicate poor implementation practices, such as a single insecure password for the entire root certificate infrastructure. It doesn’t matter what encryption algorithm you are using if your master password is the name of your company!

Attempting a straight technology fix will lead to “throwing the baby out with the bath water” for several valuable technologies. And a technical fix for this specific case won’t stop the next one.

The underlying issue is how these technologies are implemented and used. Attempting to fix this through technology is doomed to failure and will likely make things worse.

Yes, there is a place for technology improvements. We should be using dnssec to make sure dns information is valid. Stronger ways of validating certificate authenticity would be valuable – someone suggested DANE in one of the comments. DANE involves including the SSL certificate in the dns records for a domain. In combination with dnssec it gives you higher confidence that you are talking to the site you think you are, using the right SSL certificate. The issue here is that it requires companies to include this information in their dns records.

The underlying questions involve trust and law as well as technology. To function, you need to be able to trust people – in this case Lenovo – to do the right thing. It is clear that many people feel that Lenovo has violated their trust. It is appropriate to hold Lenovo responsible for this.

The other avenue is legal. We have laws regulate behavior and to hold people and companies responsible for their actions. Violating these regulations, regardless of the technology used, can and should be addressed through the legal system.

At the end of the day, the key issues are trust, transparency, choice, and following the law. When someone violates these they should expect to be held accountable and to pay a price in the market.


February 23, 2015

Samba vulnerability (CVE-2015-0240)

Samba is the most commonly used Windows interoperability suite of programs, used by Linux and Unix systems. It uses the SMB/CIFS protocol to provide a secure, stable, and fast file and print services. It can also seamlessly integrate with Active Directory environments and can function as a domain controller as well as a domain member (legacy NT4-style domain controller is supported, but the Active Directory domain controller feature of Samba 4 is not supported yet).

CVE-2015-0240 is a security flaw in the smbd file server daemon. It can be exploited by a malicious Samba client, by sending specially-crafted packets to the Samba server. No authentication is required to exploit this flaw. It can result in remotely controlled execution of arbitrary code as root.

We believe code execution is possible but we’ve not yet seen any working reproducers that would allow this.

This flaw arises because of an uninitialized pointer is passed to the TALLOC_FREE() funtion. It can be exploited by calling the ServerPasswordSet RPC api on the NetLogon endpoint, by using a NULL session over IPC.
Note: The code snippets shown below are from samba-3.6 shipped with Red Hat Enterprise Linux 6. (All versions of samba >= 3.5 are affected by this flaw)
In the _netr_ServerPasswordSet() function, cred is defined as a pointer to a structure. It is not initialized.

1203 NTSTATUS _netr_ServerPasswordSet(struct pipes_struct *p,   
1204                          struct netr_ServerPasswordSet *r)   
1205 {   
1206         NTSTATUS status = NT_STATUS_OK;   
1207         int i;  
1208         struct netlogon_creds_CredentialState *creds;

Later netr_creds_server_step_check() function is called with cred at:

1213  status = netr_creds_server_step_check(p, p->mem_ctx,   
1214                               r->in.computer_name,   
1215                               r->in.credential,   
1216                               r->out.return_authenticator,   
1217                               &creds);

If netr_creds_server_step_check function fails, it returns and cred is still not initialized. Later in the _netr_ServerPasswordSet() function, cred is freed using the TALLOC_FREE() function which results in an uninitialized pointer free flaw.
It may be possible to control the value of creds, by sending a number of specially-crafted packets. Later we can use the destructor pointer called by TALLOC_FREE() to execute arbitrary code.

As mentioned above, this flaw can only be triggered if netr_creds_server_step_check() fails. This is dependent on the version of Samba used.

In Samba 4.1 and above, this crash can only be triggered after setting “server schannel = yes” in the server configuration. This is due to the
adbe6cba005a2060b0f641e91b500574f4637a36
commit, which introduces NULL initialization into the most common code path. It is still possible to trigger an early return with a memory allocation failure, but that is less likely to occur. Therefore this issue is more difficult to exploit. Red Hat Product Security team has rated this flaw as having important impact on Red Hat Enterprise Linux 7.

In older versions of Samba (samba-3.6 as shipped with Red Hat Enterprise Linux 5 and 6. samba-4.0 as shipped with Red Hat Enterprise Linux 6) the above mentioned commit does not exist. An attacker could call _netr_ServerPasswordSet() function with a NULLED buffer, which could trigger this flaw. Red Hat Product Security has rated this flaw as having critical impact on all other versions of samba package shipped by Red Hat.

Lastly the version of Samba 4.0 shipped with Red Hat Enterprise Linux 6.2 EUS is based on an alpha release of Samba 4, which lacked the password change functionality and thus the vulnerability. The same is true for the version of Samba 3.0 shipped with Red Hat Enterprise Linux 4 and 5.

Red Hat has issued security advisories to fix this flaw and instructions for applying the fix are available on the knowledgebase.  This flaw is also fixed in Fedora 20 and Fedora 21.

February 21, 2015

Superfish – Man-in-the-Middle Adware

Superfish has been getting a lot of attention – the Forbes article is one of the better overviews.

Instead of jumping in and covering the details of Superfish, let’s look at how it might work in the real world.

Let’s say that you are looking for a watch and you visit Fred’s Fine Watches. Every time you want to look at a watch, someone grabs the key to the cabinet from Fred, uses a magic key creator to create a new key, opens the cabinet, grabs the watch from Fred, studies the watch, looks for “similar” watches, and jams advertising fliers for these other watches in your face – right in the middle of Fred’s Fine Watches! Even worse, they leave the key in the lock, raising the possibility that others could use it. Further, if you decide to buy a watch from Fred, they grab your credit card, read it, and then hand it to Fred.

After leaving Fred’s Fine Watches you visit your bank. You stop by your doctor’s office. You visit the DMV for a drivers license renewal. And, since this article is written in February, you visit your accountant about taxes. Someone now has all this information. They claim they aren’t doing anything with it, but there is no particular reason to trust them.

How does all this work? Superfish is a man-in-the-middle attack that destroys the protection offered by SSL (Secure Sockets Layer). It consists of three basic components: the Superfish adware program, a new SSL Root Certificate inserted into the Windows Certificate Store, and a Certificate Authority program that can issue new certificates.

SSL serves two purposes: encryption and authentication. SSL works by using a certificate that includes a public encryption key that is used to negotiate a unique encryption key for each session. This encryption key is then use to uniquely encrypt all traffic for that session. There are two types of SSL certificates: public and private.

Public certificates are signed. This means that they can be verified by your browser as having been created from another certificate – you have at least some assurance of where the certificate came from. That certificate can then be verified as having been created from another certificate. This can continue indefinitely until you reach the top of the certificate tree, where you have a master or root certificate. These root certificates can’t be directly verified and must be trusted.

Root certificates are connected to Internet domains. For example, Google has the google.com root certificate, and is the only one who can create a signed certificate for mail.google.com, maps.google.com, etc.

Bills Browser Certificates, Inc., can only create signed certificates for billsbrowsercertificates.com. The details are a bit more complex, but this is the general idea – signed certificates can be traced back to a root certificate. If the owner of that root certificate is cautious, you can have a reasonable level of trust that the certificate is what it claims to be.

Your browser or OS comes with a (relatively small) list of root certificates that are considered trusted. Considerable effort goes into managing these root certificates and ensuring that they are good. Creation of new signed certificates based on these root certificates is tightly controlled by whoever owns the root certificate.

Certificate signing is a rather advanced topic. Let’s summarize it by saying that the mathematics behind certificate signing is sound, that implementations may be strong or weak, and that there are ways of over-riding the implementations.

Private certificates are unsigned. They are the same as public certificates, work in exactly the same way, but can’t be verified like public certificates can. Private certificates are widely used and are a vital part of communications infrastructure.

According to reports, Lenovo added a new Superfish root certificate to the Microsoft Certificate Store on certain systems. This means that Superfish is trusted by the system. Since Superfish created this certificate, they had all the information that they needed to create new signed certificates. Which they did by including a certificate authority program which creates new certificates signed by the Superfish root certificate – on your system while you are browsing. These certificates are completely normal, and there is nothing unusual about them – except the way they were created.

Again, according to reports, Superfish hijacked web sessions. Marc Rogers shows an example where Superfish has created a new SSL certificate for Bank of America. The way it works is that Superfish uses this certificate to communicate with the browser and the user. The user sees an https connection to Bank of America, with no warnings – there is, in fact, a secure encrypted session in place. Unfortunately, this connection is to Superfish. Superfish then uses the real Bank of America SSL certificate to communicate with Bank of America. This is a perfectly normal session, and BOA has no idea that anything is going on.

To recap, the user enters their bank id and password to login to the BOA site. This information is encrypted – and sent to Superfish. Superfish decrypts the information and then re-encypts it to send to BOA using the real BOA SSL certificate. Going the other way, Superfish receives information from BOA, decrypts it, reads it, re-encrypts it with the Superfish BOA certificate, and sends it back to you.

Superfish apparently creates a new SSL certificate for each site you visit. The only reason that all this works is that they were able to add a new root certificate to the certificate store – without this master certificate in the trusted certificate store they would not be able to create new trusted certificates.

Superfish can also change the web page you receive – this is the real purpose of of Superfish. In normal operation Superfish will modify the web page coming back from the web site you are visiting by inserting new ads. Think about it – you have no idea of what the original web site sent, only what Superfish has decided to show you!

Superfish is sitting in the middle of all your web sessions. It reads everything you send, sends arbitrary information to an external server (necessary for the image analysis it claims to perform, but can be used for anything), forges encryption, and changes the results you get back.

The real threat of Superfish is that it contains multiple attack vectors and, by virtue of the root certificate, has been granted high privileges. Further, the private key Superfish is using for their root certificate has been discovered, meaning that other third parties can create new signed certificates using the Superfish root certificate. There is no way to do secure browsing on a system with Superfish installed. And no way to trust the results of any browsing you do, secure or not.


February 19, 2015

RC4 prohibited

Originally posted on securitypitfalls:

After nearly half a year of work, the Internet Engineering Task Force (IETF) Request for Comments (RFC) 7465 is published.

What it does in a nutshell is disallows use of any kind of RC4 ciphersuites. In effect making all servers or clients that use it non standard compliant.

View original


February 17, 2015

SCAP Workbench

SCAP Workbench allows you to select SCAP benchmarks (content) to use, tailor an SCAP scan, run an SCAP scan on a local or remote system, and to view the results of a scan. The SCAP Workbench page notes:

The main goal of this application is to lower the initial barrier of using SCAP. Therefore, the scope of very narrow – scap-workbench only scans a single machine and only with XCCDF/SDS (no direct OVAL evaluation). The assumption is that this is enough for users who want to scan a few machines and users with huge amount of machines to scan will just use scap-workbench to test or hand-tune their content before deploying it with more advanced (and harder to use) tools like ​spacewalk.

SCAP Workbench is designed to hide the complexity of the SCAP tools and CLI. I can vouch for the ease of use of SCAP Workbench – I’ve been using it to run SCAP and find it the easiest and most flexible way to perform SCAP scans.

SCAP Workbench is an excellent tool for tailoring SCAP benchmarks. SCAP Workbench allows you to select which Benchmark to use, and then displays a list of all the rules in the Benchmark, allowing you to select which rules to evaluate.

SCAP Workbench Tailoring
In addition, SCAP Workbench allows you to modify values in the Benchmark. In the screenshot above you see list of rules. The Set Password Expiration Parameters rule is selected and has been expanded so that we can see the various components of this rule. We have selected the minimum password length rule, and can see the details of this rule on the right side of the window.

We see the title of this rule, the unique identifier for the rule, and the type of this rule. Since this as an xccdf:Value rule, it has an explicit value that will be checked. Since this rule is checking the minimum password length, the minimum password length must be set to this value or larger.

We see that the minimum password length in the Benchmark is 12. We can change this to another value, such as 8 characters. If we change the minimum password length check, the change will be saved in the Tailoring File – the Benchmark is not modified.

After selecting the SCAP Rules you wish to evaluate and modifying values as needed you run the scan by clicking on the SCAN button. The SCAP Scan is run and results displayed in the SCAP Workbench. You can also see the full SCAP report by clicking on the Show Report button, or save the full report by clicking Save Results.


February 14, 2015

Adding an LDAP backed domain to a Packstack install

I’ve been meaning to put all the steps together to do this for a while:

Got an IPA server running on Centos7
Got a Packstack all in one install on Centos 7. I registered this host as a FreeIPA client, though that is not strictly required.

Converted the Horizon install to be domain aware by editing /etc/openstack-dashboard/local_settings

OPENSTACK_API_VERSIONS = {
    "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_DEFAULT_DOMAIN = 'Default'

And restarting HTTPD.

sudo yum install python-openstackclient

The keystonerc_admin file is set for V2.0 of the identity API. To make it work with the v3 api, cp keystonerc_admin keystonerc_admin_v3 and edit:

export OS_AUTH_URL=http://10.10.10.25:5000/v3/
export OS_IDENTITY_API_VERSION=3
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default

And confirm:

$ openstack domain list
+----------------------------------+------------+---------+----------------------------------------------------------------------+
| ID                               | Name       | Enabled | Description                                                          |
+----------------------------------+------------+---------+----------------------------------------------------------------------+
| default                          | Default    | True    | Owns users and tenants (i.e. projects) available on Identity API v2. |
+----------------------------------+------------+---------+----------------------------------------------------------------------+

Add a domain for the LDAP backed domain like this:

$ openstack domain create YOUNGLOGIC
+---------+----------------------------------+
| Field   | Value                            |
+---------+----------------------------------+
| enabled | True                             |
| id      | a9569e236912496f9f001e73fc978baa |
| name    | YOUNGLOGIC                       |
+---------+----------------------------------+

To Enable domain specific backends, edit /etc/keystone.conf like this:

[identity]
domain_specific_drivers_enabled=true
domain_config_dir=/etc/keystone/domains

Right now this domain is backed by SQL for Both Identity and Assignments. To convert it to LDAP for Identity, create a file in /etc/keystone/domains. This directory and file needs to be owned by the Keystone user:

Here is my LDAP specific file /etc/keystone/domains/keystone.YOUNGLOGIC.conf

# The domain-specific configuration file for the YOUNGLOGIC domain

[ldap]
url=ldap://ipa.younglogic.net
user_tree_dn=cn=users,cn=accounts,dc=younglogic,dc=net
user_id_attribute=uid
user_name_attribute=uid
group_tree_dn=cn=groups,cn=accounts,dc=younglogic,dc=net


[identity]
driver = keystone.identity.backends.ldap.Identity

Restart Keystone. Juno RDO has Keystone running in Eventlet still, so use:

sudo systemctl restart openstack-keystone.service

Now, grant the admin user an admin role on the new domain:

openstack role add --domain YOUNGLOGIC --user  admin admin

Like most things, I did my initial test using curl:

{
    "auth": {
        "identity": {
            "methods": [
                "password"
            ],
            "password": {
                "user": {
                    "domain": {
                        "name": "YOUNGLOGIC"
                    },
                    "name": "edmund",
                    "password": "nottellingyou"
                }
            }
        }
    }
}
curl -si -d @token-request-edmund.json -H "Content-type: application/json" $OS_AUTH_URL/auth/tokens

That requests an unscoped token. It has the side effect of populating the id_mappings for the user and group ids from the user that connects.

You can then assign roles to the user like this:

openstack role add --project demo --user  9417d7b6e7d53d719106b192251e7b9560577b9c1709463a19feffdd30ea7513 _member_

I have to admit I cheated: I looked at the database:

$   echo "select * from id_mapping;"  |  mysql keystone  --user keystone_admin --password=stillnottelling 
public_id	domain_id	local_id	entity_type
7c3448d7dc5f51861444a7f974bc63c77a2685a057d8545341a1fbcafd754b96	a9569e236912496f9f001e73fc978baa	ayoung	user
9417d7b6e7d53d719106b192251e7b9560577b9c1709463a19feffdd30ea7513	a9569e236912496f9f001e73fc978baa	edmund	user
862caa65329a761556ded5e6317f3f0cbfcab839f01340b334bdd2be4e54f1c4	a9569e236912496f9f001e73fc978baa	ipausers	group
b14ecd0d21ffb1485261ffce9c95b2cf8fec3d8194bfcda4257bb0ac74f80b0e	a9569e236912496f9f001e73fc978baa	peter	user
4e2abec872c7559279612abd2834ba1a3919aad9c01035cb948a91241a831830	a9569e236912496f9f001e73fc978baa	wheel	group

But with that knowledge I can do:

openstack role add --project demo --group  862caa65329a761556ded5e6317f3f0cbfcab839f01340b334bdd2be4e54f1c4 _member_

And now every user in the ipausers group gets put in the demo project. This works upon first login to Horizon.

UPDATE: See this post for a necessary configuration change.</a.>

February 12, 2015

Debugging OpenStack with rpdb

OpenStack has many different code bases.  Figuring out how to run in a debugger can be maddening, especially if you are trying to deal with Eventlet and threading issues.  Adding HTTPD into the mix, as we did for Keystone, makes it even trickier.  Here’s how I’ve been handling things using the remote pythong debugger (rpdb).

rpdb is a remote python debugging tool.  To use it, you edit the python source and add the line

import  rpdb;  rpdb.set_trace()

and execute your program.  The first time that code gets hit, the program will stop on that line, and then open up a socket. You can connect to the socket with:

telnet localhost 4444

You can replace localhost with a ip address or the hostname.

In order to use this from within the unit tests run from tox on keystone client, for example,  you first need to get rpdb into the venv

. .tox/py27/bin/activate
pip install rpdb
deactivate

Note that if you put the rpdb line into code that is executed on multiple tests, the second and subsequent times tests hit that line of code, the rpdb code will report an error binding to the socket that the address is already in use.

I use emacs, and to run the code such that It matches up with the source in my local git repository, I use:

Meta-X pdb

and then I run pdb like this:

telnet localhost 4444

and gud is happy to treat it like a local pdb session.

February 04, 2015

Life-cycle of a Security Vulnerability

Security vulnerabilities, like most things, go through a life cycle from discovery to installation of a fix on an affected system. Red Hat devotes many hours a day to combing through code, researching vulnerabilities, working with the community, and testing fixes–often before customers even know a problem exists.

Discovery

When a vulnerability is discovered, Red Hat engineers go to work verifying the vulnerability and rating it to determine it’s overall impact to a system. This is a most important step as mis-identifying the risk could lead to a partial fix and leave systems vulnerable to a variation of the original problem. This also allows prioritization of fixes so that those issues with the greatest risk to customers are handled first and issues of low or minimal risk are not passed on to customers who also need to invest time in validating new packages for their environment.

Research

Many times a vulnerability is discovered outside of Red Hat’s domain. This means that the vulnerability must be researched and reproduced in-house to fully understand the risk involved. Sometimes reproducing a vulnerability leads to discovering other vulnerabilities which need fixes or re-engineering.

Notification

When a vulnerability has been discovered, Red Hat works with upstream developers to develop and ship a patch that fixes the problem. A CVE assignment will be made that records the vulnerability and links the problem with the fix among all applicable implementations. Sometimes the vulnerability is embedded in other software and that host software would acquire the CVE. This CVE is also used by other vendors or projects that ship the same package with the same code—CVEs assigned to software Red Hat ships are not necessarily Red Hat specific.

Patch development

One of the most difficult parts of the process is the development of the fix. This fix must remedy the vulnerability completely while not introducing any other problems along the way. Red Hat reviews all patches to verify it fixes the underlying vulnerability while also checking for regressions. Sometimes Red Hat must come up with our own patches to fix a vulnerability. When this happens, we fix not only our shipped software, but also provide this fix back upstream for possible inclusion into the master software repository. In other cases, the upstream patch is not applicable because the version of the software we ship is older, and in these cases Red Hat has to backport the patch to the version we do ship. This allows us to minimize any changes exclusively to those required to fix the flaw without introducing and possible regressions or API/ABI changes that could have an impact on our customers.

Quality assurance

As important as patch development, Red Hat’s QE teams validate the vulnerability fix and also check for regressions. This step can take a significant amount of time and effort depending on the package, but any potential delays introduced due to the quality assurance effort is worth it as it significantly reduces any possible risk that the security fix may be incomplete or introduces other regressions or incompatibilities. Red Hat takes the delivery of security fixes seriously and we want to ensure that we get it right the first time as the overhead of re-delivering a fix, not to mention the additional effort by customers to re-validate a secondary fix, can be costly.

Documentation

To make understanding flaws easier, Red Hat spends time to document what the flaw is and what it can do. This documentation is used to describe flaws in the errata that is released and in our public CVE pages. Having descriptions of issues that are easier to understand than developer comments in patches is important to customers who want to know what the flaw is and what it can do. This allows customers to properly assess the impact of the issue to their own environment. A single flaw may have much different exposure and impact to different customers and different environments, and properly-described issues allow customers to make appropriate decisions on how, when, and if the fix will be deployed in their own environment.

Patch shipment

Once a fix has made it through the engineering and verification processes, it is time to send it to the customers. At the same time the fixes are made available in the repositories, a Red Hat Security Advisory (RHSA) is published and customers are notified using the rhsa-announce list. The RHSA will provide information on the vulnerability and will point to errata that more thoroughly explain the fix.

Customers will begin to see updates available on their system almost immediately.

Follow-on support

Sometimes questions arise when security vulnerabilities are made public. Red Hat customers have access to our technical support team that help support all Red Hat products. Not only can they answer questions, but they can also help customers apply fixes.

Conclusion

Handling security issues properly is a complex process that involves a number of people and steps. Ensuring these steps are dealt with correctly and all issues are properly prioritized is one of the things Red Hat provides with each subscription. The level of expertise required to properly handle security issues can be quite high. Red Hat has a team of talented individuals who worry about these things so you don’t have to.

January 29, 2015

The Travelling Saxophone

The Saxophone is a harsh mistress. She demands attention every day. A musician friend once quoted to me: “Skip a day and you know. Skip two days and your friends know. Skip three days and everyone knows.” That quote keeps me practising nightly.

Playing Sax by the Seine

By the Seine: Photo by Jamie Lennox

My work on OpenStack has me travelling a bit more than I have had to for other software projects. While companies have been willing to send me to conferences in the past, only OpenStack has had me travelling four times a year: two for the Summit and two for mid-cycle meetups of the Keystone team. Keeping on a practice schedule while travelling is tough, sometimes impossible. But the nature of the places where I am visiting makes me want to bring along my horn and play there.

The Kilo OpenStack summit was in Paris in November. The thought of playing in Paris took residence in my imagination and wouldn’t leave. I brought the horn along, but had trouble finding a place and a time to play. The third night, I decided that I would skip the scheduled fun and go play in the middle of the Arc de Triomphe, a couple blocks away from my hotel. There is a walkway under the traffic circle with stairs that lead up to the plaza. However, a couple of police stationed at the foot of the stairs made me wonder if playing there would be an issue, and I continued on. As I approached the far end of the walkway, I heard an accordion.

The accordion player spoke no English. I spoke less French. However, his manner indicated he was overjoyed to let me play along with him.

I shut my case to indicate that tips would still be going in his box. I was certainly not playing for the money.

He struck up a tune, and I followed long, improvising. It worked. He next said the single word “Tango” and I kicked started one off with a growl. Another tune, and then he suggested “La Vie en Rouge” and I shrugged. He seemed astounded that I didn’t really know the tune. This would be the equivalent of being in New Orleans and not knowing “When the Saint’s Go Marching In.” I faked it, but I think his enthusiasm waned, and I packed up afterwards and headed back to the hotel.

I got one other chance to play on that trip. Saturday, prior to heading to the airport, Jamie Lennox and I toured a portion of the city, near the Eiffel tower. Again, I wasn’t playing for the money, and I didn’t want to gather crowds. So we headed down to the banks of the Seine and I played near a bridge, enjoying the acoustics of the stone.

The Keystone midcycle happened in January, and I brought my Sax again. This time, I played each night, usually in the courtyard of the hotel or down along the Riverwalk. The Keystone gang joined me one night, after dinner, and it was gratifying to play for people I knew. On the walk back to the Hotel, Dolph and Dave Stanek (maybe others) were overly interested in their cell phones. It turned out they were setting up ww.opensax.com.

Playing by the Riverwalk

Playing by the Riverwalk: Photo by Dolph Matthews

January 28, 2015

Security improvements in Red Hat Enterprise Linux 7

Each new release of Red Hat® Enterprise Linux® is not only built on top of the previous version, but a large number of its components incorporate development from the Fedora distribution. For Red Hat Enterprise Linux 7, most components are aligned with Fedora 19, and with select components coming from Fedora 20. This means that users benefit from new development in Fedora, such as firewalld which is described below. While preparing the next release of Red Hat Enterprise Linux, we review components for their readiness for an enterprise-class distribution. We also make sure that we address known vulnerabilities before the initial release. And we review new components to check that they meet our standards regarding security and general suitability for enterprise use.

One of the first things that happens is a review of the material going into a new version of Red Hat Enterprise Linux. Each release includes new packages that Red Hat has never shipped before and anything that has never been shipped in a Red Hat product receives a security review. We look for various problems – from security bugs in the actual software to packaging issues. It’s even possible that some packages won’t make the cut if they prove to have issues that cannot be resolved in a manner we decide is acceptable. It’s also possible that a package was once included as a dependency or feature that is no longer planned for the release. Rather than leave those in the release, we do our best to remove the unneeded packages as they could result in security problems later down the road.

Previously fixed security issues are also reviewed to ensure nothing has been missed since the last version. While uncommon, it is possible that a security fix didn’t make it upstream, or was somehow dropped from a package at some point during the move between major releases. We spend time reviewing these to ensure nothing important was missed that could create problems later.

Red Hat Product Security also adds several new security features in order to better protect the system.

Before its 2011 revision, the C++ language definition was ambiguous as to what should happen if an integer overflow occurs during the size computation of an array allocation. The C++ compiler in Red Hat Enterprise Linux 7 will perform a size check (and throw std::bad_alloc on failure) if the size (in bytes) of the allocated array exceeds the width of a register, even in C++98 mode. This change affects the code generated by the compiler–it is not a library-level correction. Consequently, we compiled all of Red Hat Enterprise Linux 7 with a compiler version that performs this additional check.

When we compiled Red Hat Enterprise Linux 7, we also tuned the compiler to add “stack protector” instrumentation to additional functions. The GCC compiler in Red Hat Enterprise Linux 6 used heuristics to determine whether a function warrants “stack protector” instrumentation. In contrast, the compiler in Red Hat Enterprise Linux 7 uses precise rules that add the instrumentation to only those functions that need it. This allowed us to instrument additional functions with minimal performance impact, extending this probabilistic defense against stack-based buffer overflows to an even larger part of the code base.

Red Hat Enterprise Linux 7 also includes firewalld. firewalld allows for centralized firewall management using high-level concepts, such as zones. It also extends spoofing protection based on reverse path filters to IPv6, where previous Red Hat Enterprise Linux versions only applied anti-spoofing filter rules to IPv4 network traffic.

Every version of Red Hat Enterprise Linux is the result of countless hours of work from many individuals. Above we highlighted a few of the efforts that the Red Hat Product Security team assisted with in the release of Red Hat Enterprise Linux 7. We also worked with a number of other individuals to see these changes become reality. Our job doesn’t stop there, though. Once Red Hat Enterprise Linux 7 was released, we immediately began tracking new security issues and deciding how to fix them. We’ll further explain that process in an upcoming blog post about fixing security issues in Red Hat Enterprise Linux 7.

January 20, 2015

Running SCAP Scans

OpenSCAP can be run from the command line, but there are easier ways to do it.

OpenSCAP support has been integrated into Red Hat Satellite and into the Spacewalk open source management platform.

Red Hat Satellite has the ability to push SCAP content to managed systems and to run the SCAP audit scans. Red Hat Satellite has the ability to schedule SCAP audit scans and to retrieve the reports and access them through the Red Hat Satellite Audit tab.

If you are going to be using SCAP in production, especially on large numbers of systems, you should really be using a management framework like Red Hat Satellite or Spacewalk.

For development, testing, tuning SCAP benchmarks, and small scale use, the SCAP Workbench is a friendly and flexible tool. We will cover this in more detail in the next post.


January 12, 2015

Security Tests – SCAP Content

While the SCAP technologies are interesting, they have limited value without security content – the actual set of security tests run by SCAP. Fortunately there is a good set of content available that can be used as a starting point.

The US Government has released a set of SCAP content that covers the baseline security required – the United States Government Configuration Baseline (USGCB), which contains the security configuration baselines for a variety of computer products which are widely deployed across federal agencies. USGCB content covers Internet Explorer, Windows, and Red Hat Enterprise Linux Desktop 5.

Also from the US Government is the Department of Defense STIG or Security Technical Implementation Guides. A specific example of this would be downloadable SCAP Content for RHEL 6, the Red Hat 6 STIG Benchmark, Version 1, Release 4.

A number of vendors include SCAP content in their products. This is often a sample or an example – it is enough to get you started, but does not provide a comprehensive security scan.

While the available SCAP content is a good start, most organizations will have additional needs. This can be addressed in two ways: by tailoring existing SCAP content and by writing new SCAP content.

Tailoring SCAP content involves choosing which SCAP rules will be evaluated and changing parameters.

An example of changing parameters is minimum password length. The default value might be 12 characters. You can change this in a tailoring file, perhaps to 8 characters or to 16 characters for a highly secure environment.

A common way to use SCAP is to have a large SCAP benchmark (content) which is used on all systems, and to select which rules will be used for each scan. This can be changed for each system and each run. You do this by providing the SCAP benchmark, an SCAP Tailoring file, and running the SCAP scanner.

Writing new SCAP content can be a daunting task. SCAP is a rich enterprise framework – in other words, it is complex and convoluted… If you are going to be writing SCAP content (and you really should), I suggest starting with Security Automation Essentials, getting very familiar with the various websites we’ve mentioned, studying the existing SCAP content, and being prepared for a significant learning curve.


Update on Red Hat Enterprise Linux 6 and FIPS 140 validations

Red Hat achieved its latest successful FIPS 140 validation back in April 2013. Since then, a lot has happened. There have been well publicized attacks on cryptographic protocols, weaknesses in implementations, and changing government requirements. With all of these issues in play, we want to explain what we are doing about it.

One of the big changes was that we enabled support of Elliptic Curve Cryptography (ECC) and Elliptic Curve Diffie Hellman (ECDH) in Red Hat Enterprise Linux to meet the National Institute of Standards and Technology’s (NIST’s) “Suite B” requirements taking effect this year. Because we added new ciphers, we knew we needed to re-certify. Re-certification brings many advantages to our government customers, who not only benefit from the re-certification, but they also maintain coverage from our last FIPS 140 validation effort. One advantage of re-certification is that we have picked up fixes for BEAST, Lucky 13, Heartbleed, Poodle, and some lesser known vulnerabilities around certificate validation. It should be noted that these attacks are against higher level protocols that are not part of any crypto primitives covered by a FIPS validation. But, knowing the fixes are in the packages under evaluation should give customers additional peace of mind.

The Red Hat Enterprise Linux 6 re-certification is now under way. It includes reworked packages to meet all the updated requirements that NIST has put forth taking effect Jan. 1, 2014, such as a new Deterministic Random Bit Generator (DRGB) as specified in SP 800-90A (PDF); an updated RSA key generation technique as specified in FIPS 186-4 (PDF); and updated key sizes and algorithms as specified in SP 800-131A (PDF).

Progress on the certification is moving along – we’ve completed review and preliminary testing and are now applying for Cryptographic Algorithm Validation System (CAVS) certificates. After that, we’ll submit validation paperwork to NIST. All modules being re-certified are currently listed on NIST’s Modules in Process page, except Volume Encryption (dm-crypt). Its re-certification is taking a different route because the change is so minor thus not needing CAVS testing. We are expecting the certifications to be completed early this year.

January 06, 2015

What’s worse?
<noscript>Take Our Poll</noscript><script type="text/javascript"> (function(d,c,j){if(!d.getElementById(j)){var pd=d.createElement(c),s;pd.id=j;pd.src='https://s1.wp.com/wp-content/mu-plugins/shortcodes/js/polldaddy-shortcode.js';s=d.getElementsByTagName(c)[0];s.parentNode.insertBefore(pd,s);} else if(typeof jQuery !=='undefined')jQuery(d.body).trigger('pd-script-load');}(document,'script','pd-polldaddy-loader')); </script>
Securing Secure Shell

I was passed an interesting article, this morning, regarding hardening secure shell (SSH) against poor crypto that can be a victim of cracking by the NSA and other entities.  The article is well written and discusses why the changes are necessary in light of recent Snowden file releases.


January 05, 2015

SCAP Component Technologies

We’re going to dig into SCAP in a fair amount of detail. So, let’s start by covering the various technologies that make up SCAP:

  • XCCDF – the Extensible Configuration Checklist Description Format. An XML based language for creating machine parsable security checklists.
  • OVAL – the Open Vulnerability and Assessment Language. Standardizes how to assess and report on the machine state of computer systems.
  • OCIL – the Open Checklist Interactive Language. Ask users questions. For example, “do you know who to report security breaches to?” and allowing the user to respond with yes or no – or perhaps the name and contact information of where to report security breaches.
  • CCE – Common Configuration and Enumeration. Uniquely identify configuration characteristics. For example, how do you identify minimum password length across Windows, Unix, Linux and Mac?
  • CPE – Common Platform Enumeration. A structured naming scheme for IT systems, software and packaging.
  • CVE – Common Vulnerability Enumeration. A standard way to uniquely identify computer vulnerabilities, for example HeartBleed – CVE-2014-0160.
  • CEE – Common Event Expression. A common way to record events – i.e. a standard logging format.
  • CRE – Common Remediation Enumeration. Describes how to remediate or mitigate security vulnerabilities.
  • CVSS – Common Vulnerability Scoring System. A consistent methodology for measuring and quantifying the impact and risk of vulnerabilities identified through CVE.

Some of these are widely used. For example, the CVE Database maintained by Mitre is the common resource used for sharing information on security vulnerabilities. It has been used by security professionals around the world for over 15 years.

Others are new, such as the use of XCCDF and OVAL to create standardized security content that can be shared across organizations and industries and be used by automated scanners.


December 18, 2014

Before you initiate a “docker pull”

In addition to the general challenges that are inherent to isolating containers, Docker brings with it an entirely new attack surface in the form of its automated fetching and installation mechanism, “docker pull”. It may be counter-intuitive, but “docker pull” both fetches and unpacks a container image in one step. There is no verification step and, surprisingly, malformed packages can compromise a system even if the container itself is never run. Many of the CVE’s issues against Docker have been related to packaging that can lead to install-time compromise and/or issues with the Docker registry.

One, now resolved, way such malicious issues could compromise a system was by a simple path traversal during the unpack step. By simply using a tarball’s capacity to unpack to paths such as “../../../” malicious images were able to override any part of a host file system they desired.

Thus, one of the most important ways you can protect yourself when using Docker images is to make sure you only use content from a source you trust and to separate the download and unpack/install steps. The easiest way to do this is simply to not use “docker pull” command. Instead, download your Docker images over a secure channel from a trusted source and then use the “docker load” command. Most image providers also serve images directly over a secure, or at least verifiable, connection. For example, Red Hat provides a SSL-accessible “Container Images”.  Fedora also provides Docker images with each release as well.

While Fedora does not provide SSL with all mirrors, it does provide a signed checksum of the Docker image that can be used to verify it before you use “docker load”.

Since “docker pull” automatically unpacks images and this unpacking process itself is often compromised, it is possible that typos can lead to system compromises (e.g. a malicious “rel” image downloaded and unpacked when you intended “rhel”). This typo problem can also occur in Dockerfiles. One way to protect yourself is to prevent accidental access to index.docker.io at the firewall-level or by adding the following /etc/hosts entry:

127.0.0.1 index.docker.io

This will cause such mistakes to timeout instead of potentially downloading unwanted images. You can still use “docker pull” for private repositories by explicitly providing the registry:

docker pull registry.somewhere.com/image

And you can use a similar syntax in Dockerfiles:

from registry.somewhere.com/image

Providing a wider ecosystem of trusted images is exactly why Red Hat began its certification program for container applications. Docker is an amazing technology, but it is neither a security nor interoperability panacea. Images still need to come from sources that certify their security, level-of-support, and compatibility.

December 17, 2014

Container Security: Isolation Heaven or Dependency Hell

Docker is the public face of Linux containers and two of Linux’s unsung heroes: control groups (cgroups) and namespaces. Like virtualization, containers are appealing because they help solve two of the oldest problems to plague developers: “dependency hell” and “environmental hell.”

Closely related, dependency and environmental hell can best be thought of as the chief cause of “works for me” situations. Dependency hell simply describes the complexity inherent in modern application’s tangled graph of external libraries and programs they need to function. Environmental hell is the name for the operating system portion of that same problem (i.e. what wrinkles, in particular which bash implementation,on which that quick script you wrote unknowingly relies).

Namespaces provide the solution in much the same way as virtual memory simplified writing code on a multi-tenant machine: by providing the illusion that an application suite has the computer all to itself. In other words,”via isolation”. When a process or process group is isolated via these new namespace features, we say they are “contained.” In this way, virtualization and containers are conceptually related, but containers isolate in a completely different way and conflating the two is just the first of a series of misconceptions that must be cleared up in order to understand how to use containers as securely as possible. Virtualization involves fully isolating programs to the point that one can use Linux, for example, while another uses BSD. Containers are not so isolated. Here are a few of the ways that “containers do not contain:”

  1. Containers all share the same kernel. If a contained application is hijacked with a privilege escalation vulnerability, all running containers *and* the host are compromised. Similarly, it isn’t possible for two containers to use different versions of the same kernel module.
  2. Several resources are *not* namespaced. Examples include normal ulimit systems still being needed to control resources such as filehandlers. The kernel keyring is another example of a resource that is not namespaced. Many beginning users of containers find it counter-intuitive that socket handlers can be exhausted or that kerberos credentials are shared between containers when they believe they have exclusive system access. A badly behaving process in one container could use up all the filehandles on a system and starve the other containers. Diagnosing the shared resource usage is not feasible from within
  3. By default, containers inherit many system-level kernel capabilities. While Docker has many useful options for restricting kernel capabilities, you need a deeper understanding of an application’s needs to run it inside containers than you would if running it in a VM. The containers and the application within them will be dependent on the capabilities of the kernel on which they reside.
  4. Containers are not “write once, run anywhere”. Since they use the host kernel, applications must be compatible with said kernel. Just because many applications don’t depend on particular kernel features doesn’t mean that no applications do.

For these and other reasons, Docker images should be designed and used with consideration for the host system on which they are running. By only consuming images from trusted sources, you reduce the risk of deploying containerized applications that exhaust system resources or otherwise create a denial of service attack on shared resources. Docker images should be considered as powerful as RPMs and should only be installed from sources you trust. You wouldn’t expect your system to remain secured if you were to randomly install untrusted RPMs nor should you if you “docker pull” random Docker images.

In the future we will discuss the topic of untrusted images.

December 12, 2014

How to really screw up TLS

I’ve noticed a few of my favorite websites failing with some odd error from Firefox.

Firefox's Unable to connect securely error messageThe Firefox error message is a bit misleading.  It actually has nothing to do with the website supporting SSL 3.0 but the advanced info is spot on.  The error “ssl_error_no_cypher_overlap” means that the client didn’t offer any ciphers that the server also supports.  Generally when I see this I assume that the server has been setup poorly and only supports unsafe ciphers.  In this case the website only supports the RC4 cipher.  I wondered why I was starting to see a reversal of removing RC4 from so many websites recently (especially since RC4 is very weak and is on the way out).  Apparently these websites all use the F5 load balancer that had a bad implementation of the TLS 1.0 standard causing a POODLE-like vulnerability.

Stepping back for a moment, back in October the POODLE vulnerability hit the streets and a mass exodus from SSL 3.0 happened around the world.  I was happy to see so many people running away from the broken cryptographic protocol and very happy to see the big push to implementing the latest version of TLS, TLS 1.2.  So with SSL 3.0 out of the way and the POODLE vulnerability being squelched why are we seeing problems in TLS 1.0 now?

Well, simply put, F5 load balancers don’t implement TLS 1.0 correctly.  The problem with SSL 3.0 is that the padding format isn’t checked.  Apparently in the F5 devices it’s still a problem in TLS 1.0.  And while the company did offer up patches to fix the issue, some really bad advice has been circulating the Internetz telling people to only support RC4, again.  Sigh.

When RC4 finally dies a fiery death I’ll likely throw a party.  I’m sure I won’t be the only one…


December 11, 2014

Rolekit (or “How I learned to stop thinking in terms of packages”)

What’s the problem?

Let’s start with a simplification and discuss the lifecycle of software at a high-level:

  1. Research and Development – In this phase, the software is designed, coded and (hopefully) tested.
  2. Packaging – Here, we take the compiled, tested bits of the software and bundle it up into some sort of package that can be used to deliver it to a user.
  3. Deployment – An end-user takes the package and does something interesting with it (for the purists out there, I’m lumping the test, staging and production environments into the “deployment” category).

Despite the brevity of the list above, there are a lot of moving parts here. I’m going to use the Fedora process to illustrate how this all works in a pre-rolekit world and then talk a little bit about the limitations, some of the alternatives and finally how rolekit addresses the issue. First, though, I’ll answer the question I posited in the header: “What’s the problem?”

The problem to be solved is how to get useful software up and running in an end-user’s environment with the least amount of difficulty for the user. The first and most important rule in software is this: software is a means to an end, not an end unto itself. People install a piece of software in order to achieve a goal. This goal could be something relatively simple, such as “I want to listen to this MP3 I bought” or as complex as “I run the IT department for a multinational manufacturing company and I want to keep track of all my products, the rate of their sales and margins as well as what my competitors are doing”. The job of software is to enable the user to get to that desired state. To that end, I would argue this: it is far more important to help the user get started than it is to offer them every possible feature.

Some of you may interject: “But if you don’t have the feature they need, won’t they go to someone who does?”. Sure, sometimes that will happen. But you will probably discover that people will make a different tradeoff than you might think: “I can get 90% of what I need and get it set up in a few weeks” is a far more compelling statement to make to a financial decision-maker than “This product provides everything we need, but I’ll need two more full-time people to get it running next year”.

What are we doing today?

Open source development is fairly unique compared to traditional software development. One of its major advantages for development can also become its biggest challenge to deployment. Because of the breadth of open source projects out there, there is almost always someone who has done at least a piece of what you want to do already. These other projects, such as coding libraries, web application frameworks, video game engines, etc. all provide the building blocks to start your work. The great thing here is that you can pick up the pieces that you need from somewhere else and then focus your attention only on the parts that make your project unique or exciting.

However, the challenge starts happening when you get to the packaging phase. Now that you have something you want to share with the world, you need to package it in a manner that allows them to use it. There are generally two schools of thought on how to do this, each with their own strengths and weaknesses.

  1. Grab the source code (or pre-built binaries) for everything that you depend on for your project to work and package them all together in a single deliverable.
  2. Package all of your dependencies separately in their own deliverables

I’m not going to go into the details of why, but the Fedora Project has policies that require the second option. (If you’re interested in the reasoning, I strongly recommend reading the Fedora Packaging Guidelines page on the subject). Fedora then provides a dependency-resolution mechanism that simplifies this case by ensuring that when you attempt to retrieve the package you want, it also automatically installs all of the packages that it depends on (and so on, recursively until they are all satisfied).

How do we deploy it now?

There are two schools of thought on this subject, which I will refer to as the “Fedora Approach” and the “Debian Approach”, since those two Linux distributions best represent them. (Note: my understanding of the Debian Approach is second-hand, so if I get any of the subtleties incorrect, please feel free to leave a comment and I’ll correct it).

The Debian Approach

In Debian and its derivatives (such as Ubuntu, Mint, etc.), when the package resolution is completed and the packages are downloaded, the user is required to indicate at that time their explicit decision on how the package must behave. Through a system called “debconf”, package installation is directly tied to deployment; the package installation cannot conclude without it being explicitly configured at that time. If the installation is non-interactive (such as if the installation was initiated by another service, rather than the user), the configuration must either be specified by an “answer file” (a configuration file passed to debconf stating the answers in advance) or else the package must provide a sensible set of defaults to automatically deploy it.

 The Fedora Approach

In Fedora and its derivatives (such as Red Hat Enterprise Linux, CentOS, Scientific Linux, etc.), when the package resolution is completed and the packages are downloaded, that’s it. In the vast majority of cases, the software is now on the system, but it is not configured to do anything at all. (There are a few specific exceptions which have been granted by the Fedora Engineering Steering Committee for things like the firewall). On these systems, nothing will happen until the user takes an explicit action to configure and start the services.

“That sounds like the Debian Approach is better!” you may say. However, there are concerns to be had here. For one, the above explanation I made about dependency-resolution comes into play; you as a user may not be fully aware of what packages are going to be pulled in by your dependencies (even accidentally). Furthermore, just because you installed a web-server package, it doesn’t mean that you necessarily want it running immediately. So, Fedora forces you to make these decisions explicitly, rather than implicitly. So when you’re ready, you configure the software and then start it up.

Where does this fall down?

The real problem is that the concept of “packages” derives very much from the engineering side of things. A package is a logical bundling of software for the developers. Not all problems can be solved with a single package, though. For example, the FreeIPA identity-management solution requires many top-level packages including an LDAP directory server, a certificate authority server, a DNS server and others. In this, the concept of a “package” gets more than a little fuzzy. In this particular case (as has been common historically), the solution was “Let’s make another package that glues them together!”. So the FreeIPA package just adds those other packages to its dependency chain.

But just adding more packages doesn’t necessarily solve the end-user concern: How do I easily deploy this?

Enter rolekit

Rolekit was designed to be specifically for handling the deployment situation and shield end-users from the concept of project-level packages. Instead, complete solutions will be “packaged” as Server Roles. Users will come to rolekit and declare a machine to be e.g. a Domain Controller, providing the minimum information necessary to set it up (today, that’s just an admin password in the Domain Controller example). Rolekit will handle all of the other necessary work under the hood, which involves downloading the appropriate packages, installing them on the system, setting up the configuration, starting the appropriate services and carefully opening up the firewall to allow access to it.

There are a lot of moving parts involved in deploying a role, but the user doesn’t really need to know what they are. If they can be shielded from much of the noise and churn inherent in package installation, configuration, service management and firewall settings, then they get back much of their time for solving the problems unique to their environments.

Fedora and Server Roles

As of Fedora 21, we have implemented the first release of the rolekit framework as well as a single representative Role: the Domain Controller. For Fedora 22, we’re working with the Cockpit project to produce a simple and powerful graphical interface to deploy the Domain Controller Role as well as building a new Database Server Role. As the project progresses, we very much hope that others will come forward to help us build more solutions. A few that I’d love to see (but don’t have time to start on yet):

  • A fileserver role that manages Samba and NFS file-shares (maybe [s]ftp as well).
  • A mail and/or groupware server role built atop something like Kolab
  • A backup server

Welcome to the post-package world, my friends!


December 10, 2014

Analysis of the CVE-2013-6435 Flaw in RPM

The RPM Package Manager (RPM) is a powerful command-line driven package management system capable of installing, uninstalling, verifying, querying, and updating software packages. RPM was originally written in 1997 by Erik Troan and Marc Ewing. Since then RPM has been successfully used in all versions of Red Hat Linux and currently in Red Hat Enterprise Linux.

RPM offers considerable advantages over traditional open-source software install methodology of building from source via tar balls, especially when it comes to software distribution and management. This has led to other Linux distributions to accept RPM as either the default package management system or offer it as an alternative to the ones which are default in those distributions.

Like any big, widely used software, over time several features are added to it and also several security flaws are found. On several occasions Red Hat has found and fixed security issues with RPM.

Florian Weimer of Red Hat Product Security discovered an interesting flaw in RPM, which was assigned CVE-2013-6435. Firstly, let’s take a brief look at the structure of an RPM file. It consists of two main parts: the RPM header and the payload. The payload is a compressed CPIO archive of binary files that are installed by the RPM utility. The RPM header, among other things, contains a cryptographic checksum of all the installed files in the CPIO archive. The header also contains a provision for a cryptographic signature. The signature works by performing a mathematical function on the header and archive section of the file. The mathematical function can be an encryption process, such as PGP (Pretty Good Privacy), or a message digest in the MD5 format.

If the RPM is signed, one can use the corresponding public key to verify the integrity and even the authenticity of the package. However, RPM only checked the header and not the payload during the installation.

When an RPM is installed, it writes the contents of the package to its target directory and then verifies its checksum against the value in the header. If the checksum does not match, that means something is wrong with the package (possibly someone has tampered with it) and the file is removed. At this point RPM refuses to install that particular package.

Though this may seem like the correct way to handle things, it has a bad consequence. Let’s assume RPM installs a file in the /etc/cron.d directory and then verifies its checksum. This offers a small race-window, in which crond can run before the checksum is found to be incorrect and the file is removed. There are several ways to prolong this window as well. So in the end we achieve arbitrary code execution as root, even though the system administrator assumes that the RPM package was never installed.

The approach Red Hat used to solve the problem is:

  • Require the size in the header to match with the size of the file in the payload. This prevents anyone from tampering with the payload, because the header is cryptographically verified. (This fix is already present in the upstream version of RPM)
  • Set restrictive permissions while a file is being unpacked from an RPM package. This will only allow root to access those file. Also, several programs, including cron, perform a check for permission sanity before running those files.

Another approach to mitigate this issue is the use of the O_TMPFILE flag. Linux kernel 3.11 and above introduced this flag, which can be passed to open(2), to simplify the creation of secure temporary files. Files opened with the O_TMPFILE flag are created, but they are not visible in the file system. As soon as they are closed, they are deleted. There are two uses for these files: race-free temporary files and creation of initially unreachable files. These unreachable files can be written to or changed same as regular files. RPM could use this approach to create a temporary, unreachable file, run a checksum on it, and either delete it or atomically link it to set the file up, without being vulnerable to the attack described above. However, as mentioned above, this feature is only available in Linux kernel 3.11 and above, was added to glibc 2.19, and is slowly making its way into GNU/Linux distributions.

The risk mentioned above is greatly reduced if the following precautions are followed:

  • Always check signatures of RPM packages before installing them. Red Hat RPMs are signed with cryptographic keys provided at https://access.redhat.com/security/team/key. When installing RPMs from Red Hat or Fedora repositories, Yum will automatically validate RPM packages via the respective public keys, unless explicitly told not to (via the “nogpgcheck” option and configuration directive).
  • Package downloads via Red Hat software repositories are protected via TLS/SSL so it is extremely difficult to tamper with them in transit. Fedora uses a whole-file hash chain rooted in a hash downloaded over TLS/SSL from a Fedora-run central server.

The above issue (CVE-2013-6435) has been fixed along with another issue (CVE-2014-8118), which is a potentially exploitable crash in the CPIO parser.

Red Hat customers should update to the latest versions of RPM via the following security advisories:
https://rhn.redhat.com/errata/RHSA-2014-1974.html
https://rhn.redhat.com/errata/RHSA-2014-1975.html
https://rhn.redhat.com/errata/RHSA-2014-1976.html

December 03, 2014

Disabling SSLv3 on the client and server

Recently, some Internet search engines announced that they would prefer websites secured with encryption over those that were not.  Of course there are other reasons why securing your website with encryption is beneficial.  Protecting authentication credentials, mitigating the use of cookies as a means of tracking and allowing access, providing privacy of your users, and authenticating your own server thus protecting the information you are trying to convey to your users.  And while setting up and using encryption on a webserver can be trivial, doing it properly might take a few additional minutes.

Red Hat strives to ship sane defaults that allow both security and availability.  Depending on your clients a more stringent or lax configuration may be desirable.  Red Hat Support provides both written documentation as well as a friendly person that can help make sense of it all.  Inevitably, it is the responsibility of the system owner to secure the systems they host.

Good cryptographic protocols

Protocols are the basis for all cryptography and provide the instructions for implementing ciphers and using certificates.  In the asymmetric, or public key, encryption world the protocols are all based off of the Secure Sockets Layer, or SSL, protocol.  SSL has come along way since its initial release in 1995.  Development has moved relatively quickly and the latest version, Transport Layer Security version 1.2 (TLS 1.2), is now the standard that all new software should be supporting.

Unfortunately some of the software found on the Internet still supports or even requires older versions of the SSL protocol.  These older protocols are showing their age and are starting to fail.  The most recent example is the POODLE vulnerability which showed how weak SSL 3.0 really is.

In response to the weakened protocol Red Hat has provided advice to disable SSL 3.0 from its products, and help its customers implement the best available cryptography.  This is seen in products from Apache httpd to Mozilla Firefox.  Because SSL 3.0 is quickly approaching its twentieth birthday it’s probably best to move on to newer and better options.

Of course the protocol can’t fix everything if you’re using bad ciphers.

Good cryptographic ciphers

Cryptographic ciphers are just as important to protect your information.  Weak ciphers, like RC4, are still used on the Internet today even though better and more efficient ciphers are available.  Unfortunately the recommendations change frequently.  What was suggested just a few months ago may no longer be good choices today.  As more work goes into researching the available ciphers weaknesses are discovered.

Fortunately there are resources available to help you stay up to date.  Mozilla provides recommended cipher choices that are updated regularly.  Broken down into three categories, system owners can determine which configuration best meets their needs.

Of course the cipher can’t fix everything if your certificate are not secure.

Certificates

Certificates are what authenticate your server to your users.  If an attacker can spoof your certificate they can intercept all traffic going between your server and users.  It’s important to protect your keys and certificates once they have been generated.  Using a hardware security module (HSM) to store your certificates is a great idea.  Using a reputable certificate authority is equally important.

Clients

Most clients that support SSL/TLS encryption automatically try to negotiate the latest version.  We found with the POODLE attack that http clients, such as Firefox, could be downgraded to a weak protocol like SSL 3.0.  Because of this many server owners went ahead and disabled SSL 3.0 to prevent the downgrade attack from affecting their users.  Mozilla has, with their latest version of Firefox, disabled SSL 3.0 by default (although it can be re-enabled for legacy support).  Now users are protected even though server owners may be lax in their security (although they are still at the mercy of the server’s cipher and protocol choices).

Much of the work has already been done behind the scenes and in the development of the software that is used to serve up websites as well as consume the data that comes from these servers.  The final step is for system owners to implement the technology that is available.  While a healthy understanding of cryptography and public key infrastructure is good, it is not necessary to properly implement good cryptographic solutions.  What is important is protecting your data and that of your users.  Trust is built during every interaction and your website it usually a large part of that interaction.

December 02, 2014

Security Audit Automation Made Easy with SCAP

Security automation can be defined as the use of standardized specifications and protocols to perform specific common security functions.

Which leads us to SCAP – the Security Content Automation Protocol, an industry and government initiative to automate security audits and compliance.

The basic concept of SCAP is that security guides should be executable content, not paper documents. You should be able to define your security requirements (or security content) in a form that can be run on a computer with no human intervention, and which produces an audit report that can be understood by both computers and people. You should be able to run these reports – effectively, to perform a complete security audit on a system – as frequently as you want.

Further, these security guides should be dynamic, extensible, customizable, and actionable.

  • Dynamic – as new security threats are discovered, the threat and how to respond to the threat should be added to the security guide.
  • Extensible – you should be able to get security content from multiple sources, as well as create your own specialized security content.
  • Customizable – you should be able to choose which security rules apply to which systems. For example, a web server in a DMZ, a database server and a development system will all have different security requirements.
  • Actionable – the security guide should not only identify security issues, it should also give you assistance in resolving these security issues. Specifically, it should help you understand what the issue is, what the risk is, and what the exposure is, as well as what steps can be taken to resolve or mitigate it.

And, of course, consistent. You may recall the discussion of password rules from a few posts back. You need to apply the same security rules across Windows, Solaris, AIX, HP-UX, Linux, Mac, and all other computers you have.

For people who want to jump ahead, good resources for SCAP include:


MySQL On Fedora 20 Setup

I’ve set up MySQL enough times figuring things out from docs that I decided I need to take notes.

This is a destructive re-install. Don’t do this if you value your data. In fact, just don’t do this.

Cleanup after old installs.

sudo systemctl stop mysqld.service
sudo yum erase mysql mariadb-libs
#remove files that have the vestiges of old installs
sudo rm -rf  /var/lib/mysql/
sudo rm -rf /etc/mysql/conf.d/
sudo rm -rf /etc/my.cnf.d/
sudo rm -rf /etc/my.cnf
#Install
sudo yum install mysql-server
# run the server
sudo systemctl start mysqld.service
#create a db
sudo mysqladmin create keystone

Connect to the database as root to do the basics. Yes, this could be scripted from the command line:

To Create a user for ayoung

$ sudo mysql keystone
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 5.5.39-MariaDB-wsrep MariaDB Server, wsrep_25.10.r4014

Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [keystone]> CREATE USER 'ayoung'@'localhost' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [keystone]> GRANT ALL PRIVILEGES ON * . * TO 'ayoung'@'localhost';
Query OK, 0 rows affected (0.00 sec)

MariaDB [keystone]> 

Log in as ayoung

$ mysql keystone --password
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 14
Server version: 5.5.39-MariaDB-wsrep MariaDB Server, wsrep_25.10.r4014

Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [keystone]> 

November 24, 2014

Security Checklists and the US National Checklist Program

If you are going to perform a security audit you need a checklist.

Let’s spend a minute on this. If you want a predictable outcome, you need a standard process – a standard set of steps to go through to reach that outcome. Basic stuff. But here is the tricky part: people are bad about remembering things and doing things the same way every time. If the results are important, you need a checklist.

Rather than spending a lot of time here, I’m going to hand out a reading assignment: The Checklist Manifesto: How to Get Things Right by Atul Gawande. This is one of the books I strongly recommend everyone should read. Go ahead, I’ll wait until you come back.

OK, welcome back.

Let’s take a look at applying checklists to security. The first suggestion I will make is don’t write checklists from scratch. Find one that is close to what you need and modify it. It takes several iterations and considerable experience to develop a solid process that works – the more you can build on other peoples experience, the less work you have to do. And the better your chances of getting it right!

A good resource for checklists on computer security is the US National Checklist Program. This is a repository of publicly available security checklists to provide detailed guidance on setting the security configuration of operating systems and applications.

Let’s start out with a written checklist – how about the HPLaserJet 4345 MFP Security Checklist. This is a 49 page document detailing how to secure a printer. Yes, a printer. Modern printers are actually servers with a print engine hanging off the side. They can be a major security risk. They have an internal disk drive that stores the documents being printed. Did you securely remove classified documents from the last printer you got rid of?

The document covers threat models, network security, printer settings, and ramifications of the various settings. It includes many screenshots of how to use the Web-based management interface to access and change the many settings.

The good news is that this security guide exists. The bad news is that it is a time consuming manual process to apply it. Speaking of which – who configured your printer when it was installed six years ago? Did they do it right? What has happened in the intervening time? Did someone disable security on the printer so that they could get their job done?

It looks like it is time to print out the security guide and start pointing your browser at all the printers in your organization!

There has got to be a better way to do this. And no, ignoring security until you show up on the front page of the newspaper or in front of a congressional committee isn’t a better way!…


November 19, 2014

Availability of OpenLMI in Various Linux Distributions

A quick update on the availability of OpenLMI:

I have tested Fedora, RHEL, CentOS, and OEL servers using the LMI CLI running on a Fedora system – the cross platform access works.

Fedora

Fedora is the primary development platform for OpenLMI. OpenLMI support has been included in Fedora starting with Fedora 18. We strongly recommend using Fedora 20 or the upcoming Fedora 21 release when using Fedora with OpenLMI, as these include the latest versions of OpenLMI. Fedora includes all OpenLMI capabilities: the CIMOM, all Providers, the client tools and all client scripts.

Red Hat Enterprise Linux

RHEL 7 includes the OpenLMI CIMOM and Providers. RHEL 7 includes the client side infrastructure (LMIShell and the LMI CLI). Many of the client scripts are available through the EPEL repository.

CentOS

CentOS 7 includes the OpenLMI CIMOM and Providers. CentOS 7 includes the client side infrastructure (LMIShell and the LMI CLI). Many of the client scripts are available through the EPEL repository.

Oracle Enterprise Linux

OEL 7 includes the OpenLMI CIMOM and Providers.

SuSE

SLES 12 includes a subset of the OpenLMI Providers. SuSE uses the sfcb CIMOM instead of the OpenPegasus CIMOM used by default in the other distributions (both sfcb and OpenPegasus ship it all of these Linux distributions).

SLES 12 includes the following OpenLMI Providers:

  • openlmi-fan
  • openlmi-hardware
  • openlmi-journald
  • openlmi-logicalfile
  • openlmi-pcp
  • openlmi-powermanagement
  • openlmi-python-base
  • openlmi-python-providers
  • openlmi-realmd
  • openlmi-service
  • openlmi-software

The SLES 12 documentation notes that “Only reading of management information is supported for the ‘openlmi’ providers.”.

SLES 12 does not include the OpenLMI storage or network Providers; thus, you can not use OpenLMI to query or configure storage or networks on a SLES 12 system.

Debian

OpenLMI support is not currently available in Debian.

Ubuntu

OpenLMI support is not currently available in Ubuntu.


November 18, 2014

LISA’14 – Are We Making Linux Too Easy?

LISA’14, the Large Installation System Administration conference, was held in Seattle last week. I had the opportunity to give a talk on Server Management – if you are interested, the slides are available here.

One of the questions caught me completely off guard: “Aren’t you afraid that you are making system management too simple and that people won’t learn how to really manage Linux? They will just learn a few simple commands and not go any further. Today, they have to learn how Linux works and how to solve problems. OpenLMI will leave them unprepared.”

Wow… Where to start?

Thinking about this further, it could happen. In fact it will happen! Many people are looking for the quickest fix to a problem – a common way of working is to Google what you need, find something that looks like it should work, try a quick cut and paste, and move on.

OpenLMI is designed to support this. The LMI CLI is task oriented, simple, and easy to use. All you really need to use the LMI CLI is “LMI help”. The LMIShell scripts are designed to do useful work, to be easy to read, and to be modified for specific tasks.

If someone is simply looking for a way to perform a specific task, use it, and move on the the next problem, OpenLMI is a good way to go. You can use OpenLMI at a shallow level, even use it to avoid having to learn how Linux really works.

On the other hand, OpenLMI can also be used to ease into a deep knowledge of Linux: Start with the LMI CLI and use it to perform tasks. Move into LMIShell and start using and developing scripts. From there it is straightforward to develop custom automation tools. You have several ways to dive deeper into Linux administration, perhaps even developing custom OpenLMI Providers.

I would suggest that OpenLMI makes Linux more approachable. Some people will only use OpenLMI, and will never go deeper – if they can do what they need to do, this seems like a reasonable approach. Some people will use OpenLMI as a tool and and entry point to mastering Linux administration; this is great.

I don’t believe everyone needs to master Linux to use it. Consider the car analogy: All some people want to do is drive a car – automatic transmissions are perfect for them. Some people want to be able to do light repairs such as oil changes. Some want to rebuild engines and repair major subsystems of the car. And some people want to design the eight speed computer controlled automatic transmissions that are part of the integrated drive train of modern cars!

What do you think? Do we face a real risk of making Linux “too easy”, or should we try to make Linux more approachable?


Automation – a Security Imperative

So far we have established:

  • Security Guides are a good idea and exist in almost all organizations.
  • Security audits are good and widely used.
  • Security guides are often poorly written, subject to interpretation, and difficult to apply.
  • Security audits are expensive and not performed as often as they should be.

Hmmm…. Well, computers are good at following rules and measuring things. And if security guide rules are precise enough to be implemented and measured, they are very close to what you need to create a computer program.

The obvious next step is to create computer programs to implement security rules and perform computer audits!

In fact, this is what has been done for years. Numerous programs have been written for security, many security capabilities are built into operating systems, and scripts to configure systems are widely used.

However: security at the enterprise level is a big, complex undertaking.

You need a large investment in tracking threats as they emerge. It would be terribly convenient if there were a standard way to talk about threats – for example, the first 6 people who identify a new computer virus are going to call it different things, unless something is done to create a standard definition.

The vast majority of computer security issues are quickly fixed after they are identified. Decades of experience show that most computer intrusions can be prevented by applying existing patches. The question is what patches need to be applied to each specific system? This is a more complex question than it appears to be – few organizations automatically apply all patches to all systems. Instead, they test patches and carefully apply specific patches to specific systems.

The challenge is knowing which patches have been applied, which patches are available, and which patches are needed for each system. What is the risk addressed by each patch, what is the impact, and how relevant is the exposure?

Creating a useful set of security rules is a huge undertaking. If each organization is 90% common with other organizations and 10% unique, it is incredibly wasteful for each organization to build 100% of the security rules themselves.

And enterprise systems are complex. You need a workflow and extensible frameworks to be able to effectively secure, manage and monitor them.

All of these things call out for an industry wide initiative to build a standard foundation for automating security.


November 17, 2014

Dynamic Policy in Keystone

Ever get that feeling that an epiphany is right around the corner? I spent a good portion of the OpenStack summit with that feeling. I knew that it would not be earth shattering, or lead me to want to rewrite Keystone, but rather a clarification of how a bunch of things should fall together. The “click” happened on the second to last day, and it can be summarized in a few key points.

When discussing the OAUTH1.0 extension to Keystone, several people commented on how it was similar to trusts, and that we should have a unified mechanism between them for delegation. During a discussion with David Chadwick, he mentioned that the role assignments themselves were a form of delegation, and lamented that we were losing the chain of delagtion by how we delegate roles. So the first point was this:

Keystone should have a single, unified mechanism for delegation.

One key feature that feeds into that is the ability to break a big role into a small one. I had posted a spec for hierarchical roles prior to the summit, but wasn’t clear for how to implement it; I could see how it coule be implemented on the token side, but all people I talked to insisted it made more sense on the enforcement side. That is the second big point.

Role inheritance should be expanded by policy enforcement.

Policy is almost all static. Each OpenStack project had it’s own policy file in its own it repo. Extending it to cover is user requests for things like project specific policy or more granular roles has not been possible.

UPDATE: I’ve been asked to make clearer what problems this addresses.

  1. Determine what roles a user can assign to another user
  2. Allow a user to determine what roles they need to perform some action
  3. Allow some user interface to determine what a user is capable of doing based on their roles
  4. Establish an iterative process solve the long-standing bug that a user with admin on any scope has admin on all scoped.
  5. Allow a user to delegate a subset of their capabilites to a remote service.

What we have now is a simple set of specs that build on each other that will, in the end, provide a much more powerful, flexible, and consistant delegation mechanism for Keystone. Here are the General steps:

  1. Graduate oslo policy to a library
  2. Add to the policy library the essential code to enforce policy based on a keystone token.  I’ve looked at both the Keystone and Nova pieces that do this, and they are similar enough that we should not have too much problem making this happen.
  3. Add in the ability to fetch the policy.json file from Keystone.
  4. Add a rule to the Keystone policy API to return the default policy file if no policy file is specified for an endpoint.
  5. Merge the current default policy files from all of the projects into a single policy file, with namespaces that keep the rules from conflicting across services.  Reduce the duplication of rules like “admin_or_owner”  so that we have a consistent catalog of capabilities across OpenStack.  Make this merged file the default that is served out of Keystone when an endpoint asks for a policy file and Keystone does not have an endpoint specific file to give it.
  6. Make a database schema to hold the rules from the policy file.  Use this to generate the policy files served by Keystone.  There should be no functional difference between the global file and the one produced in the above merge.
  7. Use the hierarchical role definitions to generate the rules for the file above.  For example, rules that essentially say “grant access to a user with any role on this project”  will now say  “grant access to any user with the member role, or with any role that inherits the member role.  The member role will be the lowest form of access.  Admin will inherit member, as will all other defined roles.
  8. Break member up into smaller roles.  For example,  we could distinguish between actions that can only read state from those that can change it:  “Observer”  and “Editor”  Member would inherit editor, and editor would inherit observer.
  9. Change the rules for specific API policy enforcement points to know about the new roles.  For example, the API to create a new image in glance might now require the editor role instead of the member role.  But, since member inherits editor, all current users will be able to perform the same set of operations.
  10. Change the role assignment mechanism so that a user can only assign a role that they themselves have on the designated scope.  In order to assign Member, the user must have the member role, or a role that inherits Member,such as admin.  Role assignment, trusts, oauth, and any other mechanism out there will follow this limitation.  We will have to perform additional limitations, such as determining what happens to a delegated role when the person that does the delegation has that role removed;  perhaps one will need a specific role in order to perform “sticky” role assignments that last past your employment, or perhaps we will allow a user to pass some/all their delegations on to another user.

 

This is still in the planning stage.  One nice thing about a plan like this is that each stage shows value on its own, so that if we only get as far as, say stage 3, we still have a better system than we do today.  Many of the details are still hiding in the weeds, and will require more design.  But I think the above approach makes sense, and will make Keystone do what a lot of people need it to do.

Minimal Token Size

OpenStack Keystone tokens can become too big to fit in the headers between mod_wsgi and the WSGI applications. Compression mitigates the problem somewhat, but if token sizes continue to grow, eventually they outpace the benefits of compression. How can we keep them to a minimal size?

There are two variables to the size of the tokens: the packaging, and the data inside. The packaging for a PKIZ token has a lower bound based on the the signing algorithm. An empty CMS document of compressed data is going to be no less than 650 bytes. An unscoped token with proper compression comes in at 930 bytes. This are for headers, but it means that we have to keep additional data inside the token body as small as possible.

Encoding

Lets shift gears back to the encoding. A recent proposal suggested using symmetric encryption instead of asymmetric. The idea is that a subset of data would be encrypted by Keystone, and the data would have to be sent back to Keystone to validate. What would this save us?

Lets assume for a moment that we don’t want to pay any of the overhead of the CMS message format. Instead, keystone will encrypt just the JSON and base64 the data. How much does that save us? Depends on the encryption algorithm. An empty token will be tiny: 33 bytes when encrypted like this:

openssl bf -salt -a -in cms/empty.json -out cms/empty.bf

Which, according to the openssl man page, is blowfish encrypted and base64 encoded. What about a non-trivial token? Turns out, our unscoped token is quite a bit bigger: 780 bytes for the comparable call:

openssl bf -d -k key.data -in cms/auth_token_unscoped.json -out cms/auth_token_unscoped.bf

Compared with the PKIZ format at 929 bytes, the benefit does not seem all that great.

What about for a scoped token with role data embedded in it, but no service catalog? It turns out the compression actually makes the PKIZ format more effecient: PKIZ is 917 bytes versus 1008 for the bf.

Content

What data is in the token?

Identification. This is what you would see in an unsigned token: user id and name, domain id and possibly name.

Scope: domain and project info Roles: specific to the scope. service catalog. The sets of services and endpoints that implement those services.

It is the service catalog that is so problematic. While we have stated that you can make tokens without a service catalog, doing so is rally not going to allow the endpoints to make any sort of decisions about where to get resources.

There is a lot of redundant data in the catalog. We’ve discussed doing ID only service catalogs. That implies that each endpoint is expandable on the endpoint size: the endpoints need to be able to fetch the service catalog and then look up the endpoints by ID.

But let us think in terms of scale. If there is a service catalog with, say, 512 endpoints, we are still going to be sending tokens that are 512 * length(endpoint_id)

Can we do better? According to Jon Bently in Programming Pearls, yes we can. We can use a bitmap. No, not the image format. Here a bitmap is an array of bits, each of which, when set, indicates inclusion of the member in the set.

We need two things. One, a cached version of the service catalog on the endpoints. But now we need to put a slightly stricter constraint on it: the token must match up exactly to a version of the service catalog, and the service catalog must contain that version number. I’d take the git approach, do a sha256 hash of the service catalog document, and include that version in the token.

Second, we need to enforce ordering on the service catalog. Each endpoint must be in a repeatable location in the list. I need to be able to refer to the endpoints, not by ID, but by sequence number.

Now, what the token would contain? Two things:

The hash of the service catalog. A bitmap of the included services.

Here’s a minimal service catalog

Index | Service name | endpoint ID
 0 | Nova | N1
 1 | Glance | G1
 2 | Neutron | T1
 3 | Cinder | C1

A service catlog that had all of the endpoints would be (b for binary) b1111 or, in Hex, 0xF

A service catalog with only Nova would be b0001 or 0x1.

Just cinder would be b1000 or 0x8

A service catalog with 512 endpoints would be 512 bits in length. That would be 64 characters long, the length of a string comparable to a sha256. A comparable list of uuids would take 16384 characters, not including the JSON overhead of commas and quotes.

I’ve done a couple tests with token data in both the minimized and the endpoing_id only formats. With 30 endpoint ids, the compressed token size is 1969 bytes. Adding one ID to that increases the size to 1989. The minimized format is 1117 when built with the following data:

"minimizedServiceCatalog": { 
    "catalog_sha256": "7c7b67a0b88c271384c94ed7d93423b79584da24a712c2ece0f57c9dd2060924",
    "entrymap": "Ox2a9d590bdb724e6d888db96f846c9fd8" },

The ID only one would scale up at rougly 20 bytes per entry point, the minimized one would stay fairly fixed in length.

Are there other options? If a token without a catalog assumed that all endpoints were valid, and auth_token middleware set the environment for the request appropriately, then there is no reason to even send a catalog on over.

Project filtering of endpoints could allow for definitions of the service catalog that is a subset of the overall catalog. These subordinate service catalogs could have their own ids, and be sent over in the token. This would minimize the size of data in the token at the expense of the server; a huge number of projects, each with their own service catalog would lead to a large synchronization effort between the endpoints and the keystone server.

If a token is only allowed to work with a limited subset of the endpoints assigned to the project, then maintaining strictly small service catalogs in their current format would be acceptable. However, this would require a significant number of changes on how users and service request tokens from Keystone.

November 12, 2014

Enterprise Linux 6.5 to 6.6 risk report

Red Hat Enterprise Linux 6.6 was released the 14th of October, 2014, eleven months since the release of 6.5 in November 2013. So lets use this opportunity to take a quick look back over the vulnerabilities and security updates made in that time, specifically for Red Hat Enterprise Linux 6 Server.

Red Hat Enterprise Linux 6 is in its fourth year since release, and will receive security updates until November 30th 2020.

Errata count

The chart below illustrates the total number of security updates issued for Red Hat Enterprise Linux 6 Server if you had installed 6.5, up to and including the 6.6 release, broken down by severity. It’s split into two columns, one for the packages you’d get if you did a default install, and the other if you installed every single package.

During installation there actually isn’t an option to install every package, you’d have to manually select them all, and it’s not a likely scenario. For a given installation, the number of package updates and vulnerabilities that affected you will depend on exactly what you selected during installation and which packages you have subsequently installed or removed.

Security errata 6.5 to 6.6 Red Hat Enterprise Linux 6 ServerFor a default install, from release of 6.5 up to and including 6.6, we shipped 47 advisories to address 219 vulnerabilities. 2 advisories were rated critical, 25 were important, and the remaining 20 were moderate and low.

Or, for all packages, from release of 6.5 up to and including 6.6, we shipped 116 advisories to address 399 vulnerabilities. 13 advisories were rated critical, 53 were important, and the remaining 50 were moderate and low.

You can cut down the number of security issues you need to deal with by carefully choosing the right Red Hat Enterprise Linux variant and package set when deploying a new system, and ensuring you install the latest available Update release.

 

Critical vulnerabilities

Vulnerabilities rated critical severity are the ones that can pose the most risk to an organisation. By definition, a critical vulnerability is one that could be exploited remotely and automatically by a worm. However we also stretch that definition to include those flaws that affect web browsers or plug-ins where a user only needs to visit a malicious (or compromised) website in order to be exploited. Most of the critical vulnerabilities we fix fall into that latter category.

The 13 critical advisories addressed 42 critical vulnerabilities across six different projects:

  • An update to php RHSA-2013:1813 (December 2013).  A memory corruption flaw was found in the way the openssl_x509_parse() function of the PHP openssl extension parsed X.509 certificates. A remote attacker could use this flaw to provide a malicious self-signed certificate or a certificate signed by a trusted authority to a PHP application using the aforementioned function, causing the application to crash or, possibly, allow the attacker to execute arbitrary code with the privileges of the
    user running the PHP interpreter.
  • An update to JavaOpenJDK
    • RHSA-2014:0026 (January 2014).  Multiple improper permission check issues were discovered in the Serviceability, Security, CORBA, JAAS, JAXP, and Networking components in OpenJDK. An untrusted Java application or applet could use these flaws to bypass certain Java sandbox restrictions.
    • RHSA-2014:0406 (April 2014).  An input validation flaw was discovered in the medialib library in the 2D component. A specially crafted image could trigger Java Virtual Machine memory corruption when processed. A remote attacker, or an untrusted Java application or applet, could possibly use this flaw to execute arbitrary code with the privileges of the user running the Java Virtual Machine.
    • RHSA-2014:0889 (July 2014).  It was discovered that the Hotspot component in OpenJDK did not properly verify bytecode from the class files. An untrusted Java application or applet could possibly use these flaws to bypass Java sandbox restrictions.
  • An update to ruby RHSA-2013:1764 (November 2014).  A buffer overflow flaw was found in the way Ruby parsed floating point numbers from their text representation. If an application using Ruby accepted untrusted input strings and converted them to floating point numbers, an attacker able to provide such input could cause the application to crash or, possibly, execute arbitrary code with the privileges of the
    application.
  • An update to nss and nspr RHSA-2014:0917 (July 2014).  A race condition was found in the way NSS verified certain certificates.  A remote attacker could use this flaw to crash an application using NSS or, possibly, execute arbitrary code with the privileges of the user running that application.
  • An update to bash (Shellshock) RHSA-2014:1293 (September 2014).  A flaw was found in the way Bash evaluated certain specially crafted environment variables. An attacker could use this flaw to override or bypass environment restrictions to execute shell commands. Certain services and applications allow remote unauthenticated attackers to provide environment variables, allowing them to exploit this issue.
  • An update to Firefox:
    • RHSA-2013:1812 (December 2013).   Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to terminate unexpectedly or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:0132 (February 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:0310 (March 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:0448 (April 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:0741 (June 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:0919 (July 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:1144 (September 2014). Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
    • RHSA-2014:1635 (October 2014).  Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox.
      A flaw was found in the Alarm API, which allows applications to schedule
      actions to be run in the future. A malicious web application could use this
      flaw to bypass cross-origin restrictions.

97% of updates to correct 42 critical vulnerabilities were available via Red Hat Network either the same day or the next calendar day after the issues were public.

Previous update releases

We generally measure risk in terms of the number of vulnerabilities, but the actual effort in maintaining a Red Hat Enterprise Linux system is more related to the number of advisories we released: a single Firefox advisory may fix ten different issues of critical severity, but takes far less total effort to manage than ten separate advisories each fixing one critical PHP vulnerability.

To compare these statistics with previous update releases we need to take into account that the time between each update release is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:

Security Errata per month Red Hat Enterprise Linux 6 Server Default InstallThis data is interesting to get a feel for the risk of running Enterprise Linux 6 Server, but isn’t really useful for comparisons with other major versions, distributions, or operating systems — for example, a default install of Red Hat Enterprise Linux 6 Server does not include Firefox, but Red Hat Enterprise Linux 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

See also: 6.5, 6.4, 6.3, 6.2, and 6.1 risk reports.

November 10, 2014

System Audits – There Has to be a Better Way!

We’re now at the point where we can discuss a system audit. We have defined what an audit is, what security requirements are, and what a security guide is.

At the most basic level, a system audit involves examining a system to verify that it conforms to specifications. This includes operational specifications for the role the system is performing, verifying the integrity and configuration of the system, and compliance against the company security guide.

In many cases system audits are manual processes. A team of people, either internally or from an external company hired to do the audit, go though a written set of checklists and manually verify system settings and configuration.

These audits are time consuming, tedious, and expensive. They are also error prone…

As a result, companies may only audit a system every six months, once a year, or even every two years.

There has to be a better way!