Fedora

A discussion last week made me think of the following:
Ansible as a mechanism to provide network/infrastructure-wide cron.

A couple of systems that do major administrative tasks could have a infra-cron file like:

01 04 * * * root run_system_wide_task
0 01 * * Sun root trigger_client_backups

Now, I’m sure lots of you are saying ‘yes, that’s cron, you don’t need another one’ but with ansible you could have an orchestrated cron. A cron that properly says ‘wait for the previous task to finish before you launch this other one’ or a cron that is able to better contingency handling if some of your systems are offline or disconnected.

I don’t have any code for this but I wanted to toss it out as a potentially odd idea that maybe someone would love.

Got a ridiculous process **cough**Jenkins**Cough** that you have to wait to create a dir before doing things?

This might help you as godawful ugly as it is.

– name: wait for a dir to exist – this is just ugly
shell: while `true`; do [ -d /var/lib/jenkins/plugins/openid/WEB-INF/lib/ ] && break; sleep 5; done
async: 1800
poll: 20

Hey folks,
Working on something for Spot I revived some code I had written a
few years ago and then discovered that other people had made much more
robust leveled topological sorts than I had written

Anyway – if you grab the files from:

http://skvidal.fedorapeople.org/misc/buildorder/

And run:

python buildorder.py /path/to/*.src.rpm

it will look up the interdependencies of the src.rpm to figure out a
build order. It outputs a bunch of different things:
1. a flat build order
2. a build order broken out by groups – you can build all the pkgs in
any group in parallel provided that all the pkgs in the previous group
have finished building.
3. outputs lists of direct loops between srpms.
4. probably will output A LOT of noise and garbage from the rpm
specfile parsing from the rpm.spec() module

But it might be worth a look at and, ideally, patches to make it a bit
more robust.

If you have a set of pkgs which you need to build but you can’t figure
out the buildorder this might help you out.

I’d love to know how often it is right or ‘right enough’.

Known Issues:
1. some spec files make the rpm.spec() parsing break in interesting
ways – sometimes tracing back
2. if a pkg is not dependent on any other pkg and nothing else depends
on it – they get lumped in the last grouping. Not really an issue -
just something someone noticed and was surprised.
3. It will handle file-buildreqs not at all, it will handle virtual
provide buildreqs, not at all, if your buildreqs are REALLY picky about
requiring <= Version – it will ignore all of that.
4. I fully expect that 2 or more level circular build deps (foo req bar
req baz req quux) will not be detected but will make the topological
sort function die). If so…. tough… go fix your packaging.

Anyway – give it a run and see if it helps you solve a problem.

If it does let me know about it. Some of us are curious if this could
fit well in mockchain or wrapped around/in mockchain.

We needed more space for cinder and had no nice way to expand it on our existing cinder server so after banging my head a bit I got assistance from Giulio Fidente who was able to show me a working config that let me figure out what I was missing. Below I document it so others might be able to find it, too.

NOTE: this works under folsom on rhel 6.4. I cannot vouch for anything else -but Giulio had it running on grizzly I think so…

Usage:

You have an existing cinder server setup and running – which includes
a volume server, an api service and a scheduler service. You need to
add more space and you have a system where that can run.

Here’s all you need to do:

1. install openstack-cinder on the server you want to be a new volume server

2. make sure your new system can access the mysql server on your primary
controller system

3. make sure tgtd knows to import the files /etc/cinder/volumes

add
include /etc/cinder/volumes/*
to:
/etc/tgt/targets.conf

4. make sure your other computer nodes can access the iscsi-target port
iscsi-target 3260/tcp on the system you want to add as an cinder-volume server

5. setup your /etc/cinder/cinder.conf
example:

[DEFAULT]
sql_connection = mysql://cinder_user:cinder_pass@mysqlhost/cinder
api_paste_config=/etc/cinder/api-paste.ini
auth_strategy = keystone
rootwrap_config = /etc/cinder/rootwrap.conf
rpc_backend = cinder.openstack.common.rpc.impl_qpid
qpid_hostname = qpid_hostname_ip_here
volume_group = cinder-volumes
iscsi_helper = tgtadm
iscsi_ip_address = my_volume_ip
logdir = /var/log/cinder
state_path = /var/lib/cinder
lock_path = /var/lib/cinder/tmp
volumes_dir = /etc/cinder/volumes

6. start tgtd and openstack-cinder-volume

service tgtd start
service openstack-cinder-volume start

7. check out /var/log/cinder/volume.log

8. Verifying it worked:
on your cloud controller run:
cinder-manage host list
you should see all of your volume servers there.

9. creating a volume. – just make a volume as usual – the scheduler
should default to the volume server with the most space available

10. on your new cinder-volume server run lvs to look for the new volume.

Things I learned today:

1. the predictable network device naming stuff in systemd is kinda arbitrary when it comes to cloud imgs that may run on a variety of virt systems – so to turn it off just add this to your %post in your kickstart:

# disable systemd ‘predictable’ device names for networks w/a hammer

ln -s /dev/null /etc/udev/rules.d/80-net-name-slot.rules

cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << EOF
DEVICE="eth0"
BOOTPROTO="dhcp"
ONBOOT="yes"
TYPE="Ethernet"
EOF

That last bit is just to make a generic ifcfg-eth0 so ifup eth0 works normally.

2. the hostonly initramfs that dracut makes now plays up when you are moving an image around. make sure you add

dracut-nohostonly

to %packages to get it to behave as you’d expect

3. if you don’t have a lot of memory then you may not  want tmpfs for /tmp  - to turn that off just do:

systemctl mask tmp.mount

in %post and it will be as you’d expect.

4. syslinux-extlinux is WAY nicer and simpler to use than grub2

Thanks to Mattdm for making the syslinux-extlinux option for anaconda happen.

A while back I wrote this for func – and I found I needed it ported to ansible.

I enhanced it to make it take more than just 2 systems. It can now compare any number of systems to the base system

http://fedorapeople.org/cgit/skvidal/public_git/scripts.git/tree/ansible/ans_rpm_compare.py

Takes a first argument of your ‘baseline’ host that’s the host all the other hosts package sets will be compared to.

It grabs the list of rpms installed on each system (just using rpm -qa, I’m lazy, or I could have used the yum list=installed option)

It transforms that output into a set – then does a difference on them each way.

Output looks like this

$ ans_rpm_compare.py app01.phx2.fedoraproject.org app02.phx2.fedoraproject.org
Packages on app01.phx2.fedoraproject.org not on app02.phx2.fedoraproject.org
words-3.0-17.el6.noarch
fedmsg-relay-0.6.8-1.el6.noarch
gdb-7.2-60.el6.i686

Packages on app02.phx2.fedoraproject.org not on app01.phx2.fedoraproject.org

Trivial but should be straightforward to follow how it works in the code.

No idea where else to put it so it goes into my scripts git repo.

If you see this traceback in your /var/log/mailman/error file

I’m trying to produce a simple list of instances on the fedora

openstack instance. I want to produce a list every 10m or so and diff

it against the last copy of that list and output the changes.

Here’s what I came up with:
http://fedorapeople.org/cgit/skvidal/public_git/scripts.git/tree/openstack/gather-diff-instances.py

it is based originally on nova-manage. It runs as root on the head system in our cloud and just dumps out json, then diffs the json.

Everything works but I’m trying to figure out if this is the ‘right’
way of going about this.
I thought about doing it via nova instead of using the nova-manage
direct-to-db api but I had 2 problems:

1. I would need to save the plaintext admin pw somewhere on disk to
poll for that info

2. or get a token which I would have to renew every 24 hours

We’re using the above the script as a simple cron job that lets us know
what things are changing in our cloud (who is bringing up new
instances, how many, what ips they are attaching to them, etc)

Additionally, is there a way in the db api to easily query the tenant and user info from keystone? I’d like to expand out the user uuid into username/project name.

gitproxy:
http://skvidal.fedorapeople.org/misc/gitproxy

Dealing with a potential problem we were trying to figure out a way to proxy/redirect git:// calls from one server to another one. This is a fairly ridiculous script I hacked up in the wee-small hours of thursday morning after talking to+Sitaram Chamarty on #git  for a while.

I fully expect this won’t work well under load but it does seem to function in my small tests here.

Ever need to diff 2 ini files but their sections and options aren’t in the right order?

Well, I do. I googled but I couldn’t find anything trivially available that did this.

I swear I wrote this once before but I couldn’t find it when I looked through my dirs of misc scripts so:

http://fedorapeople.org/cgit/skvidal/public_git/scripts.git/tree/inidiff

hope it is useful to someone.

A number of people have been surprised by this feature, even though it is documented, so I thought I’d mention it.

Ansible can run actions async. This means it connects to the client system, starts the process and disconnects.

In general you would want all your plays to be synchronous (do thing X, wait for it to be done/watch it, do thing Y).

However, there are times when what you want to do will take a VERY long time or could kill your ssh connection off.

An example is a yum update:

tasks:

- name: yum update

action: command yum -y update

That can take a long time, depending on what’s going on. You want to monitor what it does, but you don’t want a timeout or a reset ssh session/network to kill off that process.

So what do you do? You make it async:

tasks:

- name: yum update

action: command yum -y update

async: 7200

poll: 15

That means – run yum -y update – wait for up to 7200s and poll every 15s to check on the status of the action.

Here’s where we’re using it in fedora:

http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/playbooks/package-update.yml#n11

However, this means if your ssh or network were to die – the yum update process would still run to completion.

But if your connection does die and you cannot check on the status of the job what do you do?

Well -you can connect to any system as the user who was running the job and look in ~/.ansible_async

there will be a file in there for each job that was being run. It may just be a place holder and empty (if the job is still running) or it made be filled with the results if the job is finished.

Pretty handy for a variety of tasks.

So – the trick with resizing a qcow based instances is this:

1. either you have to resize the partitions in the initramfs of the instance (which is not yet available as something we can easily do, but we’re working on it

2. you have to resize the partition on the live instance and then reboot the instance.

Since 1 is going to take more time/testing – I went ahead to make 2 as painless as possible.

using the cloud-utils and ansible I came up with this:

http://skvidal.fedorapeople.org/misc/openstack-qcow-disk-resize.yml

Put in the hosts you want to run it against. It installs cloud-utils, resizes the partitions using growpart, reboots, waits for the instance to come back alive, then does the fs resizing.

I timed it – it took a total of 1m2s and that includes installing cloud-utils, waiting at minimum 10s for the instance to reboot and then resizing the  actual fs.

The example I gave checks the values from growpart – so it  won’t run more than once (and it won’t run if you cannot resize). So you can run this play over and over and not reboot it all the time. I’m thinking I’ll probably include this as a tasklist for a quick instance provisioning playbook.

dealing with image creation in all clouds is completely full of suck. I’d more or less come to terms on it with euca but now I’m trying to do the same thing with openstack and encountering some super-duper happy fun times.

I have a rhel6 img which works and boots – it is qcow2 based so it handles kernel updates, etc properly, yay. However it handles resizing  the / filesystem exactly not at all.

If I make the rhel6 img an ami and upload kernel/ramdisk – it’ll resize (well dump it out) into a large filesystem – no problem – but it handles kernel updates not at all.

I would like to have both, I think I deserve to have both, i’ll be damned if any of my testing comes up with both. I’ve done a fair amount of googling and LOTS of things I’ve found say that the qcow2 or raw img should just work in either openstack essex or folsom but I am not having that experience.

Anyone have a suggestion?

Is thinking that because something is working well that it will continue to do so.

Silly me.

Work continues on coprs but today someone asked me if I knew of a simpler buildsystem that would let them spin up a builder, stuff a bunch of packages at it, and return a pile of results.

I said well… ansible and mockremote could do it.

I started thinking about HOW that would work and realized I have all the pieces – but not a lot of motivation to put it together at the moment.

So I thought I’d tell folks how I would do it and maybe someone will:

1. write a playbook to spin up an instance in one of the public clouds using ansible and it’s ec2 module:

http://skvidal.fedorapeople.org/misc/ansible-euca-transient.yml

2. then provision the system with mock, a mockbuilder user and mockchain.

3. tell the user the ip the instance has

4. user submits jobs to it with:

mockremote.py -r chroot -b ip –destdir=/somewhere/ -c –recurse pkg1 pkg2 pkg3

5. user comes back whenever it feels like it

6. user runs a terminate playbook like:

http://infrastructure.fedoraproject.org/infra/ansible/files/copr/provision/terminatepb.yml

which shuts down the instance they had spun up.

I can see wrapping that whole thing in a single script or just letting the user do it a bit at a time. Either way – trivial to setup your own personal, temporary and pristine builder.

I’m still working on recaps from fudcon but I wanted to mention this to anyone who asked:

If you asked for access to coprs and you haven’t received a reply from me that is b/c I forgot that you asked. Please drop me an email and I can add you.

Thanks

We’ve gone through about 45 individual builds of a goodly number of pkgs thanks to the help of the testing volunteers. We’ve found a bunch of bugs and fixed many of them.

Here’s what’s changed:

1. we’re working on cli for copr – more info in this thread: https://lists.fedorahosted.org/pipermail/copr-devel/2013-January/000216.html

2. we’ve got some mockups from the UI team which should help mold where things can go

3. better logs and results for multiple builds

4. I took a first cut of an outline for setting up copr: http://git.fedorahosted.org/cgit/copr.git/tree/copr-setup.txt

Hi everyone, it is time for my annual plea to everyone going to fudcon to wash their hands many many times, try not to cough or sneeze on anyone and to do their best to not touch me at all.

It is winter and we’re in the middle of a massive flu and norovirus outbreak here in the US.

http://www.nytimes.com/2013/01/10/health/flu-widespread-leading-a-range-of-winters-ills.html?hp&_r=0

If at all possible I would like to come back from fudcon without bringing any sort of pestilence with me.

This is just a reminder that if you see me incased in a plastic ball don’t be alarmed just smile and wave.

My call for testers was successful. We have 21 people now who have access to coprs and a number of folks have started testing builds out. We’ve found some bugs and some feature enhancements needed. Also discussion has started on the copr devel mailing list and we’ve gotten some patches contributed to help us make a cli come to pass.

If anyone else wants to take a look at the code and help out, you can see it all here in copr cgit.

thanks to all the folks who are helping test and develop the copr code.

Right now, we have coprs working. It’s not fancy or beautiful and I am POSITIVE many bugs are lurking there (and some aren’t lurking at all just sitting there). So, we need people who will seriously undertake testing. I don’t want people who want to build things and don’t care about reporting issues. I also don’t want people who are going to be filled with destructive criticism.

If you want to test it out and cite issues please email me: skvidal at fedoraproject.org

you must already have an active fas account and some srpms you want to build.

I only need maybe 10 or 20 people at the moment. Not looking to overdo it

Thanks

Working with Pingou on this I merged the playbooks into a single playbook. We were setting up jenkins and wanted a way of putting all the systems together quickly and easily. All it does is spin up a master instance, provision it, spin up 2 workers (one in el6 the other f17) provision them and end:

It could easily be done in ec2, euca, openstack or pretty much any system. I know there are more ways to skin this cat but I thought this was simple enough to follow along and others might find it interesting.

This is the playbook:

http://infrastructure.fedoraproject.org/infra/ansible/playbooks/groups/jenkins-cloud.yml

all of our ansible playbooks and inventory files are available here:

http://infrastructure.fedoraproject.org/infra/ansible/

Bohuslav and I have been diligently working on the copr buildsystem code in the last month or so. We originally targetted the middle of november but that wasn’t to be. However, right now we have working rough-around-the-edges code. It has some completed successful builds so we know that part works.

A little background:

copr is intended to provide a safe place for packages to be built that are not packages that are not or cannot be built in fedora. Think of it as a place where you can scratch and chain build whatever you want (up to the obvious limits of legality . It takes your packages, builds them against themselves and puts the repo up available for download.

Why not just do this in koji?

1. koji is just for fedora builds
2. koji does not allow external, arbitrary repositories for build deps
3. koji’s builders are not destroyed and cleaned after each build
4. koji just isn’t designed to do what copr does and grafting it on top of koji is a dangerous proposition to the safety and security of fedora builds.

How does copr do things?

You submit your build request (name of the copr, chroots to build against, repos to use to build and the list of src.rpms to build) into the frontend. The backend takes this request, creates a builder in our private cloud, builds the pkgs, pulls back the results onto a webserver, provides you with a link to the results and destroys the builder.

What’s the status right now?

The backend code works, but there are a lot of rough edges to sand down. The frontend code works, but there is a bunch of ui work needed there (right now Ryan Lerch is creating some mockups for how things should be). The code is available from the link at the top. Right now everything is in two branches a front end branch (bkabrda-workspace) and a backend branch (skvidal-backend). We’ll be merging them into master in short order.

We’ll be sending out a call for folks to help test in the not-so-distant future. The whole goal is to provide a place for people to build packages they would like to have available but maybe cannot have in fedora for one reason or another.

Examples:

• alternative configuration/build options for an existing package
• not very easy to package in fedora (bundling, etc) but perfectly good software otherwise
• test builds
• your idea here
• builds based on other repos that are not included in fedora

Things which cannot be in a copr:

• software which are not legal to include in fedora

The fedora buildsys mailing list (buildsys@lists.fedoraproject.org) is where we will discuss things  going forward but I just wanted to let people know what we’ve been working on.

This week we held a Fedora Activity Day in Raleigh at RH HQ to get two-factor auth setup for Fedora Infrastructure. We had a pretty good plan and we knew the pieces we needed to put together – it was just a matter of doing it. We got our goals accomplished but that’s not what I wanted to talk about.

We ended up with 9 people there. I was originally a bit concerned that was too many people, that we would end talking more than accomplishing things and that would suck, especially if we had failed to get things implemented. I was keeping track of what everyone was doing, how they were helping. What I noticed was that everyone contributed in some way. There wasn’t anyone on the sidelines. At one point we had 2 people working on the package reviews of the pkgs we needed to get into fedora (totpcgi and pam_url). We had a person writing the cgi to let us use both yubikeys and google auth, a person working on the provisioning interface to get people setup using google auth, a person working on the puppet config, a person setting up the certs/pki needed to let pam_url connect securely. We had a person setting up cloud instances for us to use to test/blow things up and we had a couple of people writing/rewriting their yubikeys and auth secrets in order to test and retest and reretest.

The FAD just removed all friction (as someone else put it to me yesterday). It meant that instead of waiting a few days or more to solve the problems we only waited 20minutes. Like often said about mediation – good facilities are sometimes all that is required to get things done.

It was great having everyone there and able to work it was great being able to ONLY focus on this one thing. I think we will have this again in the future to help accomplish tasks which are just too involved to bite off a little at a time or will take years to get done at that rate.

Great presentation slidedeck:

https://speakerdeck.com/jpmens/ansible-an-introduction

introducing ansible.

As I find the need I write functionality I need into the existing euca2ools using their lovely cli python api.

I hate trying to remember an instance id. I know the ip of the host or I know the dns name of the hose. I don’t need to go find the instance id to know which one I want to kill.

But euca-terminate-instances is silly and won’t let me pass in an ip or a hostname. Nor will it let me specify globs

So I wrote this

http://fedorapeople.org/cgit/skvidal/public_git/scripts.git/tree/euca/my-terminate-instances.py

It takes public or private ips, public or private dns names (the ones euca or ec2 has) or instance ids.

It also lets you pass file-globs to them. So you can do things like:

my-terminate-instances i-\*

and kill everything you’re running. Isn’t that fun!

enjoy

A few days ago I posted about using ansible to provision persistent ip instances in a public or private cloud.

Today I finished up the next piece I needed to make this work for copr builders w/transient ips/instances.

I needed a way to create a new instance, provision it as if it was one of the normal inventory of our systems, and return back the ip of that was given to it via eucalyptus (or ec2) automatically. And when I was done with it – I didn’t want any record of it preserved in our inventory system. I wanted it gone.

The problem was ansible wants to act only on the hosts it knows about in its inventory. This is quite understandable. But since I’m not specifying an ip to this host and I don’t know it in advance I wanted a way, in a single playbook to do this.

So I wrote add_host an action_plugin for ansible. These let you interact with the internal state of the inventory/playbook/etc while executing a playbook.

All it does is to add the host to the in-memory inventory the ansible playbook object has. And it adds that host to a group you specify. This is so in the second play in the playbook you can say ‘oh operate on hosts in this special group name’ and that will be the only host in that group.

I’ve sent in a pull request for it but it’s not been accepted quite yet. However, if you want to try it out you can just toss it into your  action_plugins dir in ansible and call it.

Here’s an example playbook. It’s very similar to the one for creating a persistent instance. In the fedora public ansible repo we are, in fact, importing the same set of tasks from another file to set them up the same.

It just means if anyone wants an instance in our private cloud running f17 or el6 it is incredibly trivial to make one available for you.

You’ve just spun up a new instance and you need to give additional people access to the system as root. You have a common IDMS that houses ssh pub keys for your users. You want to trivially specify a list of those users and have their keys show up in root’s .ssh/authorized_keys file.

Here’s what you do:

 

- name: add root keys for other allowed users
action: authorized_key user=root key=’$PIPE(/path/to/script/for/keys ${root_auth_users})’
only_if: is_set(‘${root_auth_users}’}

In our infrastructure FAS houses all the pubkeys. So Toshio wrote this script: http://infrastructure.fedoraproject.org/infra/ansible/scripts/auth-keys-from-fas

So if you define a hostvar in your ansible inventory for this host – then the above will automatically populate your root authorized_keys with the right pub keys.

Kinda awesome, I think.

Since we’ve been working more and more with our private cloud setups on eucalyptus and openstack I’ve been trying to come up with a semi-sane way of having persistent hosts in the cloudlets.

So that if we shut things down or an instance dies – it can be brought back up automatically without someone having to explicitly say ‘go start this up’. I’ve also been working on setting up ansible such that we can run it via cron on our central command host to maintain our systems.

So, of course, I’ve combined those two things into one task

We want certain cloud instances to:

1. always exist
2. have the same ip each time they come up.

These are for things like the build master for COPRs/PPAs or for the jenkins-master server or for any number of random services where they need a persistent ip throughout recreations. For the ips we’re using euca-associate-address after allocating a number to our account.

So I started on a mechanism to create those instances and give me results I could use in ansible for other actions. I came up with ec2 which lets you do that from a local_action call in ansible.

Once you have that module in your ansible library path you just need to:

• add the host to your ansible inventory file
• take  this playbook example
• modify it for your host
• run it like: ansible-playbook thatplaybook.yml

you’ll need euca2ools installed along with the deps for ansible.

Here’s what the playbook does:

1. it looks to see if the host is running (by a simple nc connect to port 22)
2. if it is not running then it will run ec2 with your args
3. then associate the ip
4. wait for the host to get the ip and become available
5. then it goes on to the next play which is normal ansible provisioning

which is perfect, for the next time you run – steps 2-4 won’t happen – it just goes directly to provisioning. Since ansible playbooks are idempotent you can run them as many times as you want w/o maligning your server.

Now the example playbook is all merged together – but if you use host_vars in your ansible inventory and you setup the check/create tasks as an includable task file – then the playbook for creation and provisioning of any new cloud instance you want becomes VERY short.

More to come as I finish it.

I’ve been working with ansible a lot recently. Fedora has a new public repo:
http://infrastructure.fedoraproject.org/infra/ansible/
and
http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/

That’s just the beginnings of our repo for managing systems. There’s a number of details to work out, though and I have to merge over all the non-private things from our builder repo. I’ve started that process it’s just a matter of making reasonable decisions on where the tasks should live.

Recently I’ve been looking at:
https://github.com/ansible/ansible-provisioning

and figuring out how a euca2ools-based local_action module would work. I can spin up new instances, if they don’t already exist, and then provision them then I can create an entire network from more or less nothing.

Also working out how best to use euca-allocate-address and and associate-address in our infrastructure to provide some persistent and consistent servers/apps in the cloud. I think I have that bit hashed out but we’re discussing some tracking for cloud instances so we can sensibly shut them down when we need to w/o making everyone angry

I’m positive I can do all these things with ansible and euca/openstack. The question now is how long it will take to make them all happen. Anyone interested in helping out, come by #fedora-admin and talk to me.