Kalle Happonen

Ceph - a long debug

Hello again! Yet a longer time between posts, but at least my colleagues on https://pouta.blog.csc.fi are holding the fort!

Generic disclaimer: Ceph Hammer/Jewel, CentOS7

Most of the work was done by our Ceph team, yours truly was partially involved.


We've been using Ceph for a long time, and . . .

Read More

July 05, 2018

Resize problems

It's a while since the last blog. For some reason the technial blog posts stopped at the same time I took the product owner hat. Interpret that as you may.

Normal disclaimer: CentOS 7, RDO on OpenStack Newton

I can't resize!

Resizing and migrating VMs are some of the standard tools in the OpenStack arsenal . . . .

Read More

February 26, 2018

Rally and Tempest

Or: your cloud probably has problems

When you operate a complex system, like OpenStack, one of the most important things is to know the state of the system. What works, what's broken and what's broken for a known reason. Without this information it's really hard to make changes to the system.

I think pretty much all OpenStack installations have their own way . . .

Read More

July 20, 2017

Liberty to Newton

Sorry Mitaka

Here's my traditional OpenStack upgrade blog post. I usually write these after all of our major OpenStack upgrades.

Our stack: CentOS 7, RDO, Puppet + Ansible, Linuxbridges + VLAN, Ceph

We're a dinosaur who still has monolithic API nodes (Most services running on a pair of VMs), which is relevant for this procedure.

. . .

Read More

May 31, 2017

Packets - Part 2

Some time back, I wrote a post about hunting down dropped packets on our network nodes. Here's a small follow up with some performance hints.

More problems

My happy glow after fixing the previous issue didn't last for long. Our network nodes had hit the next problem.

Our network nodes are virtualized for . . .

Read More

May 16, 2017

Logical Volume Management

Or: Paying the Price for Technical Debt

We have run OpenStack for a while. I think we went into production around Grizzly. Back then we were new to running production OpenStack systems (who wasn't?). This means we made some decisions back then, which we are still paying for.

Credit where credit is due: Almost all the real work described in this post was done by my . . .

Read More

March 16, 2017

Packets, Drop by Drop

I originally started writing this blog to document all the tricky bugs I encountered, and google couldn't answer.

My, did I ever find one. And it wasn't one of the issues that makes you go beat yourself for not figuring it out sooner.

Network, gross work

We had a simple change in our cloud infrastructure. We . . .

Read More

January 30, 2017


This update link alerts you to new Silvrback admin blog posts. A green bubble beside the link indicates a new post. Click the link to the admin blog and the bubble disappears.

Got It!