Dienstag, 6. Dezember 2011

by-uuid, where are you?

0 Kommentare
Are you missing entries in /dev/disk/by-uuid ?

 Run "uuidgen" this will spawn a uuid like:
2edddab5-d555-414f-9db6-4c80c11897f5

After that, run tune2fs to assign the generated uuid to the filesystem on the device:
elbereth:~ # tune2fs /dev/yourdevice -U
2edddab5-d555-414f-9db6-4c80c11897f5

You're ready to go, don't forget to retrigger you udev rules with

udevadm trigger
Enjoy your new persistent name !



Dienstag, 4. Oktober 2011

Openais - zombies and the revenge of the undead

0 Kommentare
Are you trying to start your cluster and all you get out of it is a set of duplicate and defunct processes, similar to the following?

4129 ?        Ssl    0:01 /usr/sbin/corosync
4134 ?        S      0:00  \_ /usr/lib64/heartbeat/stonithd
4135 ?        S      0:00  \_ /usr/lib64/heartbeat/cib
4136 ?        Z      0:00  \_ [lrmd] <defunct>
4137 ?        S      0:00  \_ /usr/lib64/heartbeat/attrd
4138 ?        Z      0:00  \_ [pengine] <defunct>
4139 ?        Z      0:00  \_ [crmd] <defunct>
4141 ?        S      0:00  \_ /usr/lib64/heartbeat/stonithd
4142 ?        S      0:00  \_ /usr/lib64/heartbeat/cib
4143 ?        S      0:00  \_ /usr/lib64/heartbeat/lrmd
4144 ?        S      0:00  \_ /usr/lib64/heartbeat/attrd
4145 ?        S      0:00  \_ /usr/lib64/heartbeat/pengine
4652 ?        S      0:00  \_ /usr/lib64/heartbeat/crmd


Check /etc/corosync/service.d/
If in there you see a pcmk file *and* Pacemaker is specified as service in your /etc/corosync/corosync.conf  you have your answer.
Just leave your .config file as is and get rid of the pcmk file instead.
Reboot your node, and you' re ready to go.

Autoyast - yast2-samba-client: bits and pieces

0 Kommentare
Your autoinstallation fails to join your AD domain unless you run a net join as post installation script?
You can specify  the join credentials within the samba-client section of your profile like this  ( this goes after samba's </global> )
<join>         
     <password>whatever</password>
     <user>Administrator</user> 
</join>  

Also, are you setting your <realm> and the result in your smb.conf is an empty "realm=" entry?
Try setting it twice, in both <samba-client> and <samba-server> sections, that should do the trick.

Freitag, 22. Juli 2011

Missing the obvious, udp fragments and a short moment of frustration.

0 Kommentare
Do you know *that* final level boss, the one that seems impossible to beat even after hours and hours of attempts and uncountable retries? Yes, the very same that after a night spent lost in your own frustration and a day that kept you busy with some other business falls - piece of cake - at the first attempt?

Sometimes can be hard to keep track of the big picture when you are busy looking for details. All the small bits are in the end saturating your field of vision leaving you cluttered and even more confused than before.

This is exactly what I've been experiencing in the last 2 days, missing the obvious that was already in front of my eyes. This post is not meant as a technical explanation - the "solution" is way to obvious - but as a personal reminder, against the often unproductive habit to get lost in details.

Let's say that someone, for whatever reason, wants to DROP all udp fragments, and this using iptables.

No problem. Easy to write a rule for that, as long as you remember one detail: fragments are reassembled before entering the INPUT chain, and therefore would be best to do it in PREROUTING.
The rule by itself doesn't leave much space for mistakes:

iptables -t raw -A PREROUTING -p udp -j DROP

would do, and just to make sure, let's add some logging to make sure that all works as it should:

iptables -t raw -I PREROUTING -f -p udp -j LOG --log-prefix match_fragment_prerouting

A quick setup to test the whole, using elbereth as sender and mjolnir as receiver:

mjolnir:/# netcat -vv -l -u -p xxxx -s xxx.xxx.xxx.xxx.

elbereth:/# hping3 xxx.xxx.xxx.xxx. -V -2 -f -x --file /tmp/lotsofdata -d 2000 -p xxxx

And guess what? All my 2000 bytes are happily displayed and no packet matched my rule.

I will spare you all the inconclusive tests I did, and all the traces I've taken, while too blind to see what the problem really was, but please believe me, it was a lot. From doubts about the correct positioning of the rule, to datacenter's pixies and poltergeists.

This morning, I decided to reset the test environment completely and take two new traces, as clean as possible, and start again with it.
And there I finally saw captain obvious appearing from far away:

On the trace taken on mjolnir's side, I could see all the udp fragments *and * the reassembled packet.

And the captain gets closer.

A quick addition to my iptables' logging, to add an extra entry that would log not only the fragments, but any udp packet.

Run the test again, and my fear was confirmed, the rules are working, but no fragment ever reaches iptables, only reassembled packets.
Only reassembled packets I said?

And here captain obvious appeared next to me, enjoying the scene of me slapping my own head before, after and while typing

lsmod|grep conntrack

A pat on my shoulder, and he was away again, looking for someone else that missed the obvious, at least for a while.

Freitag, 24. September 2010

Growing a md device

0 Kommentare
I am not quite sure why so often people are  stumbling on this, but at this point I will invest a couple of line on the topic:

I am trying to grow my md, I already did this, and it used to work. This time I get the following error:

 Elbereth:~ # mdadm /dev/md2 --grow --size=max
 mdadm: Cannot set device size for /dev/md2: Device or resource busy

Short answer: You forgot your bitmap. Or at least there is a good chance you did.

Longer one: Currently, you cannot grow a md device that holds an internal bitmap. This is not too tragic, you can still remove the bitmap, grow your device and put the bitmap back, as long as you are not going to have a major failure during the process, you'll be just fine.
What if you happen to have the up mentioned failure? Well, do not focus on the bitmap too much, with a bad failure during a grow operation - bitmap or not - you are going to have quite a bit of trouble anyway.

Practically speaking:

mdadm --grow --bitmap=none /dev/mdX

to remove it and then grow you array as usual.
Once done, you put it back with a:

mdadm --grow --bitmap=internal /dev/mdX

Have fun!

Montag, 13. September 2010

Bits and pieces of a Monday

0 Kommentare


I should definitely start playing with SuseStudio a bit more.
Surprisingly enough, browsing trough the appliances, there is nothing already done for a quick setup of a dummy Openais node. Indeed, in this case a simple build is not enough, but I guess this gives me something to play with this week.
Well see if I manage to banish procrastination for long enough and actually get this done.

A couple of more or less proper topics deserving an entry from themselves are in the pipeline, but due the time and being Monday, for today I'll keep it short and confused.
So, here you go with random bits and pieces of what I've been asked. Probably nothing worthy to be noted, but you never know..

Given a NFS mount on my [opensuse|SLES|SLED] provided by my $filer (you haven't heard me saying Netapp here... ) due to a firewall dropping UDP packets, I see failures all over the place.
Captain obvious suggests to move to TCP, but unfortunately, using "mount -o tcp" does not really help, because mountd is still going to use UDP. 

As if it wouldn't be enough, using RH all my traffic is going as expected thorough TCP. What's wrong?
Quick answer

The  tcp  option  is an alternative to specifying proto=tcp.  What you are missing is mountproto=tcp
                   
Long answer
man nfs
/Using the mountproto mount option

On my Openais cluster I want to run several Dom-u resources but for whatever reason I don't want  automatic memory allocation.
I disabled it using the cluster GUI, but I still see changes in the amount of memory given to my Dom-u. What's wrong with me and with them?

With you, not sure, but with your cluster and Domu there is nothing wrong.
With the GUI, there was (this is currently fixed upstream, by the time you'll read this the answer will be make sure you are up to date!).
The problem was just a non valid value provided by the GUI, that would have let you choose between "True" and "False" while the resource agent was expecting a "0" in order to deactivate the feature.

I guess it is enough ranting for now, time to go back to my evening powered by Milky Oolong and Robochicken on the background.

qrcode

Mittwoch, 8. September 2010

Ode to autoreadonly

0 Kommentare
Have you ever come across unused md devices seen as autoreadonly?
 
This particular status is given to md devices lacking IO activity (as in, they never had any since they array was assembled).
If you are wondering why an md device should be started if no IO is taking place on it you are probably right - with one exception - it is legitimate to have swap on a md device.

This doesn't really cause any problem by itself, as soon as IO will start the device will automatically awake from this state, but there is something you should take care of.
Let's assume you are doing an autoinstallation, using (surprise surprise) Autoyast.
Without specifying a filesystem for those newly created mds, AY will do exactly what
it should - setting them up without further action.
Those are going to start in the (in)famous autoreadonly status, with a particularity - sync pending.
This is correct, the sides of the mirror never synced, and are currently readonly.
Practically it also means that you don't really have a working mirror.

If you do not manually take care of it, issuing for example a mdadm --readwrite in order to trigger a sync, you'll have quite some pain in case anything will happen to your storage.

Anything else? Well, do not forget that if by any chance your menu.lst contains a "resume=" option pointing to a md device, you'll get an autoreadonly status for free.
Either go for noresume, feel free to use a fake device to resume from or, if you have a real device that is not md, that one. If you change this early enough

AY related hint, try with a chrooted config:type="boolean">true</chrooted> in your chroot script

you can easily forget about all this rant, and live happily ever after.

Unrelated, but yummy QR of the day:
qrcode