Hozer Central

Lately I've been using lists of dictionaries a lot to store data to be processed sequentially. More often than not, the data gets loaded into the list in an ordered state, but I needed a way to force it to be ordered by a key of the dictionary. Luckily the sort() list function takes as an argument a function to do the comparison for me. Since the comparison is just a simple is this key less than another key I made my first venture into lambda functions:

l = [ {"name": "1234", "attribute1": "this is an attribute", "attribute2": "this is another"},
        {"name": "4325", "attribute1": "this is an attribute", "attribute2": "this is another"},
        {"name": "3145", "attribute1": "this is an attribute", "attribute2": "this is another"} ]

l.sort( lambda x, y: cmp( x["name"], y["name"] ) )

Ok, so i'll stop lying. The list of dictionaries is a list of Networker NSR Client entries. Each Networker entries is a dictionary relating to the attribute: value; of a Networker client. I need to sort this list so when we have multiple clients for the same server, they appear together when I process them. Using a lambda function as the function to pass to sort() accomplishes this since the lambda function is only calling the cmp() function on the 2 name attributes of 2 clients.

Quick, easy and reliable. Happy Hacking!

Recently on the Networker Mailing List, a windows admin was venturing into the promised land of cli access to networker and wanted some points in the right direction. Since I typed it up, I figure I'll post it here as well so I can find it next time someone asks.

In nsradmin you set a query and then act on it. A query is any set of key and value pairs (attributes). To set the query you use the `.` or `print` command. The benefit of `print` being that it sets the current query and then prints out the attributes returned by that query. Examples are probably easier to understand than theory.

(As I don't have a networker installation to access at the moment all the example output is from memory, the exact output of the various commands may vary)

1. To print a client's config:

   nsradmin> print type: nsr client; name: client.example.org
   

   This sets the query to any resources that match the named attributes. Here we are using the `type` attribute to restrict to `nsr client` and the `name` attribute to restrict the query to only those matching `client.example.org`. Since we used the `print` command, the results of the query are immediately displayed. The exact same output can be achieved by using:

   nsradmin> . type: nsr client; name: client.example.org
   Current query set.
   nsradmin> print
   

2. To update a client's config:

   nsradmin> . type: nsr client; name: client.example.org
   Current query set.
   nsradmin> update aliases: "www.example.org"
   

   First the query is set with the `.` command, and then the resources modified are updated. IMPORTANT: if you set a query that returns more than one resource you will update all those resources! Be careful with your queries and be as restrictive as possible. Doing something like:

   nsradmin> . type: nsr client; schedule: default
   Current query set.
   nsradmin> update aliases: "bad.example.org"
   

   Will set the alias for every client with the `default` schedule. Many a first time nsradmin adventurer have made mistakes like this.

   You could just as well use `print` instead of `.`, and I in fact recommend it as a safeguard to accidentally having to stay up for days fixing client configs. ;)

3. Deleting a group:

   nsradmin> delete type: nsr group; name: MyGroup
   <output of query>
   Delete? yes
   

   You could also split this out using the `.` command to set the query and then the `delete` command to act upon the query.

If you can, I would recommend setting up a linux box (or installing cygwin) and installing the Networker man pages. The syntax of nsradmin is very well documented in the nsr_resource man page.

Setting up multipathd on SLES9 is a hozer.

Material List

• NetApp FAS3020 (2 in cluster)
• QLA2422 HBA (2 per server)
• Switching Fabric of some sort (2)
• Target to bang head against (1)

Software List

• SLES9 SP3
• multipath-tools-0.4.5-020 (from novell's update site stock will work too)
• NetApp Linux Host Utils 3.0

multipath.conf

Take a look at the examples in /usr/share/doc/packages/multipath-tools. Although multipath will work without a configuration file, you will be limited to accessing the luns by their WWID. My expert recommendation is that at least the multipaths, devices, and devnode_blacklist be included.

devnode_blacklist

This section lists devices to be excluded from multipath (with regex matching). If the server boots from local SCSI disks, including those disks would be wise. Something along the lines of:

devnode_blacklist {
  devnode "^sda$"
  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
  devnode "^hd[a-z]"
  devnode "^cciss!c[0-9]d[0-9]*"
}

NOTE: It is important that any devnode starting with sd will need to be terminated with a $. LUNS can (and probaby will) be named sda[a-z][a-z].

multipaths

This is were you map your LUNS to a friendly name that you can work with, such as:

multipaths {
  multipath {
    wwid 360a9800043336a414c3a3954725a7869
    alias  my-lun0
  }
  multipath {
    wwid 360a9800043336a414c4a395871437a71
    alias  my-lun1
  }
}

Whoa, your saying to yourself, how do you figure out your WWID? If you run multipath -d -v2 -l you will get something like:

360a9800043336a414c3a3954725a7869
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:1:1  sdas 66:192 [active][ready]
 \_ 1:0:1:1  sdq  65:0   [active][ready]
\_ round-robin 0 [enabled]
 \_ 2:0:0:1  sdal 66:80  [active][ready]
 \_ 1:0:0:1  sdj  8:144  [active][ready]

360a9800043336a414c4a395871437a71
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [active]
 \_ 2:0:1:0  sdar 66:176 [active][ready]
 \_ 1:0:1:0  sdp  8:240  [active][ready]
\_ round-robin 0 [enabled]
 \_ 2:0:0:0  sdad 65:208 [active][ready]
 \_ 1:0:0:0  sdb  8:16   [active][ready]

The import part here is to look at the common LUN ID each path has. For the LUN with WWID 360a9800043336a414c4a395871437a71 you can see that it is LUN ID 0 in the group since all SCSI paths end in 0.

devices

With the devices section you define devices and options for them. Since we have a clustered NetApp, this is the section that will tell multipath which path to prefer since you can access a LUN via either filer head end (although they will bitch about it every now and again if you have autosupport turned on and are accessing a LUN via the "wrong" head end). Your section should look something like:

devices {
  device {
    vendor  "NETAPP"
    product  "LUN"
    path_grouping_policy  group_by_prio
    getuid_callout  "/sbin/scsi_id -g -u -s /block/%n"
    prio_callout  "/opt/netapp/santools/mpath_prio_ontap /dev/%n"
    features  "1 queue_if_no_path"
    path_checker  readsector0
    failback  immediate
  }  
}

Most importantly are the path_grouping_policy and prio_callout. By setting path_grouping_policy to group_by_prio paths to the LUN will be grouped via their priority as determined by which head end you are accessing the LUN through. The prio_callout entry tells multipath how to ask netapp what priority each path has.

Starting Multipathd

You can now start everything up with:

# /etc/init.d/boot.multipath start
# /etc/init.d/multipathd start

And add them to start on boot:

# insserv boot.multipath multipathd

The multipath devices can now be used in /dev/disk/by-name. To check everything out you can run multipath -d -v2 -ll and see the priority and grouping of the paths:

my-lun0 (360a9800043336a414c3a3954725a7869)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=8][active]
 \_ 2:0:1:1  sdas 66:192 [active][ready]
 \_ 1:0:1:1  sdq  65:0   [active][ready]
\_ round-robin 0 [prio=2][enabled]
 \_ 2:0:0:1  sdal 66:80  [active][ready]
 \_ 1:0:0:1  sdj  8:144  [active][ready]

my-lun1 (360a9800043336a414c4a395871437a71)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=8][active]
 \_ 2:0:1:0  sdar 66:176 [active][ready]
 \_ 1:0:1:0  sdp  8:240  [active][ready]
\_ round-robin 0 [prio=2][enabled]
 \_ 2:0:0:0  sdad 65:208 [active][ready]
 \_ 1:0:0:0  sdb  8:16   [active][ready]

You can see that now you not only get the WWID but now also the alias.

Rebooting

You may find that after rebooting your devices do not appear in /dev/disk/by-name. Run! The sky is falling! The streets will flow with the blood of the non-believers! Just kidding. What is happening is /etc/init.d/boot.multipath is being run too early. I'm still looking for the "correct" solution, but I have found that by editing /etc/init.d/multipathd to call it prior to the start of multipathd works. Here is my diff to the file:

--- multipathd.old      2007-01-17 21:35:41.091274231 -0600
+++ multipathd  2007-01-17 21:35:05.591133859 -0600
@@ -55,6 +55,7 @@
 case "$1" in
     start)
        echo -n "Starting multipathd"
+       /etc/init.d/boot.multipath start
 
        modprobe dm-multipath

This is just awsomely funny.

Have you ever accidentally deleted all of /? Of course you haven't you read the man page and know what the -f flag does. You don't just blindly include it whenever you need to delete something.

I ran across this post on the Ubuntu Forums and my heart just sank. No longer are the days when Unix/Linux users actually understand what they are doing. Nowadays users tend to just blinding paste anything they find from Google into a terminal.

This Ubuntu user had an excellent idea. Let us write a patch to rm to prevent it from deleting all of / without asking when you pass it the -rf flags. This was one of those Lewis Black moments for me. I had to reply.

But I can't just reply like all the other ramrods suggesting to alias `rm -i` to `rm` in bash. I actually wrote the patch.

Ladies and Gentlemen, I present to you the --idiot (-I) flag to rm:

diff -Naur coreutils-5.97.orig/src/remove.h coreutils-5.97/src/remove.h
--- coreutils-5.97.orig/src/remove.h    2005-05-14 02:58:37.000000000 -0500
+++ coreutils-5.97/src/remove.h 2007-01-13 00:15:13.268298115 -0600
@@ -33,6 +33,9 @@
   /* If true, recursively remove directories.  */
   bool recursive;
 
+  /* If true, the user is an idiot. */
+  bool idiot;
+
   /* Pointer to the device and inode numbers of `/', when --recursive.
      Otherwise NULL.  */
   struct dev_ino *root_dev_ino;
diff -Naur coreutils-5.97.orig/src/rm.c coreutils-5.97/src/rm.c
--- coreutils-5.97.orig/src/rm.c        2005-08-29 16:13:32.000000000 -0500
+++ coreutils-5.97/src/rm.c     2007-01-13 00:17:38.396568504 -0600
@@ -80,6 +80,7 @@
   {"directory", no_argument, NULL, 'd'},
   {"force", no_argument, NULL, 'f'},
   {"interactive", no_argument, NULL, 'i'},
+  {"idiot", no_argument, NULL, 'I'},
 
   {"no-preserve-root", no_argument, NULL, NO_PRESERVE_ROOT},
   {"preserve-root", no_argument, NULL, PRESERVE_ROOT},
@@ -142,6 +143,7 @@
                            supports `unlink' for nonempty directories)\n\
   -f, --force           ignore nonexistent files, never prompt\n\
   -i, --interactive     prompt before any removal\n\
+  -I, --idiot           let the ramrod who can't read the man page use -rf\n\ 
 "), stdout);
       fputs (_("\
       --no-preserve-root do not treat `/' specially (the default)\n\
@@ -183,6 +185,7 @@
   x->ignore_missing_files = false;
   x->interactive = false;
   x->recursive = false;
+  x->idiot = true;
   x->root_dev_ino = NULL;
   x->stdin_tty = isatty (STDIN_FILENO);
   x->verbose = false;
@@ -209,7 +212,7 @@
 
   rm_option_init (&x);
 
-  while ((c = getopt_long (argc, argv, "dfirvR", long_opts, NULL)) != -1)
+  while ((c = getopt_long (argc, argv, "dfiIrvR", long_opts, NULL)) != -1)
     {
       switch (c)
        {
@@ -227,6 +230,10 @@
          x.ignore_missing_files = false;
          break;
 
+    case 'I':
+      x.idiot = false;
+      break;
+
        case 'r':
        case 'R':
          x.recursive = true;
@@ -275,6 +282,11 @@
        error (EXIT_FAILURE, errno, _("failed to get attributes of %s"),
               quote ("/"));
     }
+  
+  if (x.recursive & x.ignore_missing_files & !x.interactive & x.idiot)
+  {
+      error (EXIT_FAILURE, 0, "You are an idiot. Rerun with the --idiot flag.");
+  }
 
   {
     size_t n_files = argc - optind;

I've also uploaded the the file here.