Wednesday, June 16, 2021

Sharing SSH session variables across multiple sessions

0 comments
In Unix-like systems, we're used to using ssh-agent to keep track of our private keys making it easier to log into remote systems. When using tmux or screen, it can be difficult to do this using just one agent unless you store certain values for later retrieval. Here is a script that makes that much easier / possible.

In order to use this script, place this file into your $HOME/bin directory and set the permissions of that file to 0755 if you want to share, or 0700 if you don't.

#!/bin/sh
#
# Save SSH environment variables for later
# retrieval to keep from having to start
# multiple ssh-agent executables.
#
# Use the following alias to automatically
# reconnect to the "old" session after using
# grabssh. Note that it assumes that your
# private keys all end in '.pem'.
#
# alias fixssh='source $HOME/bin/fixssh_helper 2>/dev/null ; temp="`ssh-add -l >/dev/null 2>/dev/null`" ; if [ $? -ne 0 ] ; then eval "`ssh-agent -s` ; $HOME/bin/grabssh ; ssh-add $HOME/.ssh/*.pem" ; else echo "Reconnected to ssh-agent" ; ssh-add -l ; fi'
#
SSHVARS="SSH_AUTH_SOCK SSH_AGENT_PID DISPLAY"

for x in ${SSHVARS} ; do
    (eval echo $x=\$$x) | sed  's/=/="/
                                s/$/"/
                                s/^/export /'
done 1>$HOME/bin/fixssh_helper

chmod 600 $HOME/bin/fixssh_helper

echo "Saved SSH auth information for later retrieval"

Make sure that $HOME/bin is in your PATH environment variable so no matter where you are, it'll find this script.

The script itself gives us a suggestion to use an alias in our shell's rc file. In my case, that's .zshrc, but you may be using .bashrc, .kshrc, or some other default file in your home directory. Your mileage may vary but this has been thoroughly tested with zsh and bash.

Example usage (running for the first time):

$ fixssh
command not found: fixssh

This is a good thing. We're not going to mess with any system commands named fixssh. :-)

$ alias fixssh='source $HOME/bin/fixssh_helper 2>/dev/null ; temp="`ssh-add -l >/dev/null 2>/dev/null`" ; if [ $? -ne 0 ] ; then eval "`ssh-agent -s` ; $HOME/bin/grabssh ; ssh-add $HOME/.ssh/*.pem" ; else echo "Reconnected to ssh-agent" ; ssh-add -l ; fi'

This installs the script we were asked to run (above) for our current shell.

If you don't have any SSH keys, there are lots of articles out there on how to generate SSH keys. I won't duplicate their efforts here.

Now, let's make sure we have a .pem file for it to use. I'll assume that you usually have id_rsa as your primary SSH private key. If there are others, you'll want to follow this same process with each private key file. There is no need to do this with public keys (.pub files).

$ mv $HOME/.ssh/id_rsa $HOME/.ssh/id_rsa.pem

Now we have an SSH key we can use with this system.

$ fixssh
Agent pid 32977
Saved SSH auth information for later retrieval
Identity added: *****.pem (*****.pem)

What happens if you close your ssh session? You can use fixssh again to reconnect to your ssh-agent. What if you open another session in parallel? As above, use fixssh to reconnect to your existing ssh-agent.

This is a great tool for folks that need to log into a jumpbox without having to set up their ssh-agent each time.

Note: I found the grabssh and fixssh methods on the web *many* years ago and while I have long since forgotten where that came from, my goal is not to plagiarize that method. This method of using fixssh has evolved greatly from the original. Hats off to the original poster.

Tuesday, August 2, 2016

Efficient MySQL Date Verification in Javascript?

0 comments
I'm not the best person I know at determining what is efficient in JavaScript (ECMAScript) though I would like to think that this could help someone.

/**
 * Make sure that the passed value is valid for the proposed condition. If
 * isRequired is true, dateString must not be blank or null as well as being
 * a valid date string. If isRequired is false, dateString may be blank or null,
 * but when it's not, it must be a valid date string. A valid date string looks
 * like YYYY-MM-DD
 *
 * @param dateString {String}
 * @param isRequired {Boolean}
 * @returns {Boolean}
 */
function isDateValid( dateString, isRequired ) {
    var regex = /^\d\d\d\d-\d\d-\d\d$/ ;
    var retVal = true ;

    if ( ! isRequired ) {
        if ( ( null == dateString ) || ( '' == dateString ) ) {
            return true ;
        }
    }
    else {
        retVal = ( ( null !== dateString ) && ( '' !== dateString ) ) ;
    }
    retVal = ( retVal && ( null !== dateString.match( regex ) ) ) ;
    if ( retVal ) {
        var daysInMonths = [ 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 ] ;
        var yr = parseInt( dateString.substring( 0, 4 ) ) ;
        var mo = parseInt( dateString.substring( 5, 7 ) ) ;
        var da = parseInt( dateString.substring( 8, 10 ) ) ;
        if ( ( yr % 4 ) && ( ( yr % 400 ) || ! ( yr % 100 ) ) ) {
                daysInMonths[ 1 ]++ ; // Leap day!
        }
        if  ( ( yr < 2000 ) || ( yr > 2038 )
           || ( mo < 1 ) || ( mo > 12 )
           || ( da < 1 ) || ( da > daysInMonths[ mo ] )
            ) {
         retVal = false ;
        }
    }
    return ( retVal ) ;
}
If you know of a more efficient way to handle a MySQL (YYYY-DD-MM) date validation, please reply to this post. :-)

Thursday, March 26, 2015

Estimating MySQL rollback time in InnoDB

0 comments
Estimating (computing) rollback time in MySQL can be a bit of a pain (art), but it can be done. MySQL provides data about rollbacks in InnoDB through SHOW ENGINE INNODB STATUS output. Here's a sample:

---TRANSACTION 2920ACF08, ACTIVE 12568 sec rollback
ROLLING BACK 2426027 lock struct(s), heap size 216037816, 52624206 row lock(s), undo log entries 3226386
MySQL thread id 5669944, OS thread handle 0x2b126bd21940, query id 2028903424 10.2.3.4 user41
# Query_time: 8736.352709  Lock_time: 0.000151 Rows_sent: 0  Rows_examined: 52624206
SET timestamp=1427378149; 
(query being rolled back)


For estimating rollback time, the important bits here are:
ROLLING BACK 2426027 lock struct(s), heap size 216037816, 52624206 row lock(s), undo log entries 3226386
The "undo log entries" value is the number of undo logs remaining to be rolled back. This value decreases over time. To get to the time remaining, we need at least two samples of this ROLLING BACK line along with timestamps of when those were taken. Put those together and you'll get the rollback rate. Here's rollback rate per second:
rbr = (staring log entries - ending log entries) / (end time in seconds - start time in seconds)
My experience has been that as undo log entries approaches zero, the rollback rate tends to increase. Having said that, if you have your $HOME/.my.cnf set up with your credentials in it, you can use something like this from a Unix (/bin/sh) shell prompt to see a likely up-to-date prediction of when rollback will complete:

$ rbr=161 # Rollback rate per second 
$ while true ; do clear ; x=“`mysql -h HOSTNAME -e 'show engine innodb status \G' | grep ROLLING\ BACK`" ; echo "$x" ; echo -n "Minutes to go: " ; expr `echo "$x" | cut -d, -f4- | cut -d' ' -f5-` / $rbr / 60 ; sleep 5 ; done

It's technically possible to dynamically adjust the value of $rbr, but I've found that it tends to lead to more frustration.

Saturday, March 2, 2013

GTD Done Wrong?

0 comments
So I just saw this in the GTD blog and wondered if I agree. I'm pretty sure I don't agree.

Priorities and Goals... The GTD Achilles Heel? David Allen Company Forums by commmmodo on 3/2/2013 1:24 PM
After using GTD since 2007, I have found priorities and goals to be t?he system's achilles heal.
I think there's a fair chance I'm doing something wrong, so I want to give the community a chance to correct me and defend GTD.
Time in life is short and finite. I have reached a conclusion that if you want to achieve big career and life goals, you have to cut out all of the unnecessary projects/tasks and focus exclusively on the absolutely best project that will advance you to that goal. There are lots of tasks and projects we could do, but 80% of our energy should be put into the 20% most important projects.
So I ran a little GTD experiment recently. At my weekly review, I started setting top priority projects for a 3-10 day span and timeboxing it. The idea is to find the project that is holding me back from the next level of success in life, and get it completed in a set number of days. So for example: until March 10th I am working on our fundraising documents for people to invest in our company, and after March 10th it's being marked DONE.
During my experiment, I replied to as few emails that don't deal with this project as possible, put off meetings on other projects, and anything that isn't directly achieving the goal I set. I went in my office and closed the door, metaphorically and literally. Because, really, I can do all of the medium-priority tasks I want... and they're not bad things to be working on... but if I really want to advance my career and my company to the next level as quickly as possible, this top-priority project is all I should be focusing on. It's a harsh reality. I guess an analogy would be, as Warren Buffet says, "Putting all of your eggs in 1 basket and watching it carefully." Instead of watering a thousand roses with my finite water bucket of time, I am watering 1 flower with a lot of water until it's bloomed big and strong.
I was a little upset at how well this experiment went, since I have trusted David Allen and GTD to tell me the best thing to do for 6+ years. The results? I got what would have taken 20 days done in about 4. I achieved my goal, and it moved the company and my life forward in a really big way.
GTD's answer to this, as I understand it, is pretty simple: set 50,000ft, 30,000ft, and 20,000ft altitudes (areas of responsibility and major goals) and review them at your weekly review. Then, as you go through your day, pick out next actions based on context, time, energy, and priority.
The problem with this GTD goal and priority system is: you're never picking out 1 30,000ft goal that should be done next, and systematizing it into your daily routine. There's context lists and project lists... but there's no "Do This Project and Nothing Else if You Want to Advance your Life And Career" list. There's no part of GTD that focuses you on that next most important goal. Instead, you're assessing goals and priorities every 5 minutes, and that creates a mental fatigue of sorts. That 3-10 day goal is never written down, making it easy to lose sight of what you really should be doing, even though you may identify this important project during those precious moments of weekly review zen.
Out of practicality, I've started doing a new activity during my weekly review: "What is the next most important project to complete that will advance my life and career more than anything else?" I write it down, open up Omnifocus, and hide all other projects except that one.
Therefore, I've started to see GTD as a sort of hamster on a wheel, a way to spend time on a lot of stuff that doesn't matter and avoid the harsh reality that I should be focused on the one project that actually matters, and saying "f*** everything else."
My question is: why aren't priorities and goals a part of GTD? Is GTD just that? Getting THINGS done. Don't we really want GTMITD? Getting THE MOST IMPORTANT THINGS done? Okay, okay, the acronym isn't as sexy. But life is short, time is finite, and priorities (as defined by your larger goals) need to be systematized. I need something where I can go on autopilot during the work day. That's the whole point of mind like water, is I don't need to be thinking about my task system all day long. I need a better answer than, "Set up your 20,000ft review, and then reanalyze your priorities every time you complete a task." It's not working for me.
I hope this explains the problem clearly. It's a complex situation, therefore I may not have explained everything you need to know to render a reply. Please feel free to ask followup questions and I'll respond to them promptly. Thank you.

Does anyone watching this blog have any comments on why this GTD practitioner should feel he wasn't using GTD while working on the fundraising documents given the information provided?

I don't consider myself a GTD expert, but I do think "commmmodo" was actually using GTD properly during the "experiment" because he/she elected to prioritize the fundraising document above nearly everything else for a limited time. Maybe I'm missing something.

Thoughts anyone?

Tuesday, November 27, 2012

NoSQL vs. SomeSQL

0 comments
Linux Journal had a fantastic article (SQL vs. NoSQL) some time back. While I know this is a bit of hopping on the bandwagon, I like the point this video is trying to make: http://www.xtranormal.com/watch/6995033/mongo-db-is-web-scale. Caution: The language used in this "video" may not be appropriate for some viewers.

There are lots of folks out there that like to tout numbers on performance and how sometimes performance is really fast under "ideal" conditions, but as both point out, the trick is to know how to balance performance/scalability, reliability, and availability. /dev/null is extremely scalable and available but it's completely unreliable. In MySQL - the Blackhole storage engine has a lot of the same performance metrics but used properly, can be a great way to "pass through" data in a replication ring.

Sunday, February 12, 2012

A basic shared-nothing data sharding system

0 comments
There's a lot of buzz about sharding data. Today, I'll provide a very brief overview of how sharding helps systems I manage run more efficiently and how we're addressing keeping individual shards balanced.

The goal of sharding data in our environment involves: 1) make the structure of the data consistent across all the shards, 2) dividing data up so it can be found easily, 3) automatically and continuously re-balance the shards, and 4) allow for changes in scale (like adding a new shard or different shard sizes).

Item 1 is a snap - all we do there is to deploy the same data structures in each of the shards with all the supporting data required to answer questions related to a user. Some of this data is user-specific, some is globally replicated. In any case, this goal makes it easy to use one set of code to access data in any of the shards without having to cross to another shard or database to get the answer for a question. This reduces workload in the application and on other database servers.

Item 2 is done by hashing our key data. Let's say that we have a set of widgets that users are concerned with. Some users have a few widgets, some have a lot, but each user is very different from another. Widgets are pretty common and well defined. Each user has a user ID and any question we ask the system always involves a specific user ID. So - our key data we hash against in this case would be the user ID. Data about the widgets is replicated to all the shards, but data about each user is only kept on the shard where that user's data lives.

Item 3 is handled by a separate process that utilizes the same API the application uses. Balancing the data between shards is simple - the balancer asks the API if there are any users that need to move. If yes, the balancer lets the API know to lock that user temporarily, moves the data, then unlocks the users for use on the new shard. What this means for applications is each time a location is returned for a specific user, that location is only guaranteed for a given window of time (30 seconds for example). So - when the balancer tells the API it's moving a user's records, any requests for that user's records are held up until the user's data is moved. The API is smart enough to only let the balancer move data that has not been accessed recently. This doesn't prevent all lock collisions, but it handles most of them.

Item 4 is handled through the configuration of the API. Because we use an API to tell the application where the data is for a given user, we've abstracted away where data actually lives. This makes it easy to add and remove servers from the sharding pool. We've extended this to include allowing a shard to be marked as in a draining state. When a shard is draining, the API will ask the balancer to move rows from the draining shard and redistribute that information onto other members of the sharding pool. This makes it possible to take a shard out of rotation for routine maintenance without the loss of data.

Notice that I didn't mention any specific software here. I didn't tell you what language the application is written in, what language the API is written in, or what the actual data store was. The technique of sharding data is pretty simple and can be done with nearly any persistence layer using any programming language.

The beauty of this system is that once the API is written, the balancer can be a complete "black box" to the application. This type of system could be implemented with a data store when just starting out and be expanded to multiple stores as the need expands. Also - sharding key needs to change, again, the application doesn't need to change - just the API and the balancer.

One other big benefit to sharding data like this - it's often a lot cheaper to buy several smaller systems than to buy and maintain one very large system. If one of the systems in the sharding pool goes off-line, the worst possible exposure in a shared-nothing sharding system is the data stored on the member that went down. In a monolithic system, you stand to lose a lot more.

While I wouldn't suggest trying to do this type of work on top of every data set out there, I do see that there is a lot of benefit when the types of questions being asked of a data set can be divided up easily while still making it relatively easy to answer the "question at hand" from a single source. The secret in the sauce is making sure that any common data is shared among all the systems in the pool.

Sunday, January 15, 2012

Managing incoming emails

0 comments
Reading emails all day long tends to be very counter-productive for me. I usually end up responding faster than anyone else which generally gets me a lot more work than I need. At the same time, I have a responsibility during my times as primary and secondary on-call to respond within our service level agreement. So - how do I find balance? My team and I use mailing lists to help us manage those truly urgent issues versus those issues that can be handled as time allows. We have three lists:

group_primary@foo.com
group_secondary@foo.com
group_admin@foo.com

We've published these three lists to our operations center. Everyone else just gets the admin list. We don't tell others about the primary and secondary lists because anything we'd get on primary or secondary would need to come via the operations center anyway. We also don't want our over 600 co-workers (not on our team and not in the NOC) to email us willy-nilly using our on-call emails.

Next, on each of our team's smart phones, we've set them up to recognize emails going specifically to the primary and secondary emails so our phones will either go off like a pager or (in my case) read the sender and destination email (think "Inbound Primary email from the NOC"). That prevents me from having to look at my phone every time a new message comes in but lets me know when there's something that requires my attention.

The other thing we do is to make it easy to change the destination for the primary address easily so that only primary gets notified. Secondary is notified in the same way but on my two-man team, there are only two of us so secondary always goes to the whole team (for now).

Finally, to help us have reasonable sanity, I do what I can to only check the "other" emails twice a day.

The  net result of this process is I am able to focus on getting project work done between routine email readings and it lets others figure things out for themselves or wait a bit for an answer. If it was truly urgent, the sender could simply ask the NOC to reach out to the on-call person to get a faster response.

How do you deal with your on-call processes and email?

Monday, November 28, 2011

Watch this video for instructions on how to use indexes better

0 comments
This is the first video I've ever seen that visually represents how indexing works. I've seen good stuff before but this ... wow. Yes - it has some stuff about Tokutek in it, but that's not why I suggest watching it - it's because it makes you re-think how to define good indexes.


http://www.youtube.com/watch?v=AVNjqgf7zNw

Sunday, November 27, 2011

Flashback to summer riding

0 comments
With all the recent crummy weather, I thought I'd resurrect a summer fun photo (from around July 4th, 2011) :-) If you like wind farms and/or motorcycling, you'll like this pic of my 2003 Honda Silverwing:


For those that are wondering, it gets about 52 MPG normally during the summer and around 45 MPG in the winter and will do over 100MPH.

Thursday, October 20, 2011

On-Call Silliness

0 comments
Are you on-call? Do you get called up for stuff that just makes no sense for you to attend to? Check out this video...

THE FROBNICATOR WON'T FROBNICATE!
by: kbcmdba



The trick to handling on-call is to make sure that you're getting called when your expertise is really needed. If you're getting called for things you can automate, automate them. If you're getting called for stuff that doesn't apply to you, show them this video. Just because the frobnicator won't frobnicate for someone else, doesn't mean it's your responsibility to fix it or even tell the right person about it.

Monday, June 20, 2011

mysqldumpslow and Rows Examined

0 comments
I have found that watching the ratio of rows sent to rows examined helps me get a much better idea of how hard the server has to work to get a result set. Unfortunately, mysqldumpslow in MySQL 5.1 doesn't provide that capability. Rather than keeping it to myself, I'm sharing my fix with the world in hopes that it gets included in all the current and future versions of MySQL. :-)

This patch is officially entered into the public domain.

--- mysqldumpslow 2010-05-07 10:17:19.000000000 -0500
+++ mysqldumpslow 2011-06-20 11:46:44.000000000 -0500
@@ -8,8 +8,8 @@
 use strict;
 use Getopt::Long;

-# t=time, l=lock time, r=rows
-# at, al, and ar are the corresponding averages
+# t=time, l=lock time, r=rows sent, e=rows examined
+# at, al, ar, and ae are the corresponding averages

 my %opt = (
     s => 'at',
@@ -83,8 +83,8 @@
     s/^#? Time: \d{6}\s+\d+:\d+:\d+.*\n//;
     my ($user,$host) = s/^#? User\@Host:\s+(\S+)\s+\@\s+(\S+).*\n// ? ($1,$2) : ('','');

-    s/^# Query_time: ([0-9.]+)\s+Lock_time: ([0-9.]+)\s+Rows_sent: ([0-9.]+).*\n//;
-    my ($t, $l, $r) = ($1, $2, $3);
+    s/^# Query_time: ([0-9.]+)\s+Lock_time: ([0-9.]+)\s+Rows_sent: ([0-9.]+)\s+Rows_examined: ([0-9.]+).*\n//;
+    my ($t, $l, $r, $e) = ($1, $2, $3, $4);
     $t -= $l unless $opt{l};

     # remove fluff that mysqld writes to log when it (re)starts:
@@ -121,6 +121,7 @@
     $s->{t} += $t;
     $s->{l} += $l;
     $s->{r} += $r;
+    $s->{e} += $e;
     $s->{users}->{$user}++ if $user;
     $s->{hosts}->{$host}++ if $host;

@@ -129,10 +130,11 @@

 foreach (keys %stmt) {
     my $v = $stmt{$_} || die;
-    my ($c, $t, $l, $r) = @{ $v }{qw(c t l r)};
+    my ($c, $t, $l, $r, $e) = @{ $v }{qw(c t l r e)};
     $v->{at} = $t / $c;
     $v->{al} = $l / $c;
     $v->{ar} = $r / $c;
+    $v->{ae} = $e / $c;
 }

 my @sorted = sort { $stmt{$b}->{$opt{s}} <=> $stmt{$a}->{$opt{s}} } keys %stmt;
@@ -141,13 +143,13 @@

 foreach (@sorted) {
     my $v = $stmt{$_} || die;
-    my ($c, $t,$at, $l,$al, $r,$ar) = @{ $v }{qw(c t at l al r ar)};
+    my ($c, $t,$at, $l,$al, $r,$ar, $e,$ae) = @{ $v }{qw(c t at l al r ar e ae)};
     my @users = keys %{$v->{users}};
     my $user  = (@users==1) ? $users[0] : sprintf "%dusers",scalar @users;
     my @hosts = keys %{$v->{hosts}};
     my $host  = (@hosts==1) ? $hosts[0] : sprintf "%dhosts",scalar @hosts;
-    printf "Count: %d  Time=%.2fs (%ds)  Lock=%.2fs (%ds)  Rows=%.1f (%d), $user\@$host\n%s\n\n",
-           $c, $at,$t, $al,$l, $ar,$r, $_;
+    printf "Count: %d  Time=%.2fs (%ds)  Lock=%.2fs (%ds)  Rows Sent=%.1f (%d) Rows Examined=%.1f (%d), $user\@$host\n%s\n\
n",
+           $c, $at,$t, $al,$l, $ar,$r, $ae,$e, $_;
 }

 sub usage {
@@ -163,11 +165,13 @@

   -v           verbose
   -d           debug
-  -s ORDER     what to sort by (al, at, ar, c, l, r, t), 'at' is default
+  -s ORDER     what to sort by (ae, al, at, ar, c, e, l, r, t), 'at' is default
+                ae: average rows examined
                 al: average lock time
                 ar: average rows sent
                 at: average query time
                  c: count
+                 e: rows examined
                  l: lock time
                  r: rows sent
                  t: query time

Saturday, May 28, 2011

Heavy improper connection termination creates disk full risk

2 comments
Logging warnings, is generally a good thing for a DBA. Without them, we may not be able to understand when there are problems that are otherwise hard to find. One of the warnings mysqld logs looks a bit like this:

110528 11:35:44 [Warning] Aborted connection 4185247 to db: 'db_name' user 'user_name' host: 'a.b.c.com' (Got an error reading communication packets)
There are two primary candidates for this type of warning: a) the client application, or b) the network.

On the client side, this is often caused by programs that terminate the connection without calling mysql_close after completing use of the connection. This problem is so common, in fact, that it is documented in http://dev.mysql.com/doc/refman/5.5/en/communication-errors.html
I can't count how many times I've had to speak with developers about making sure they close connections properly in order to prevent this kind of warning from occurring.

At the root of this issue, mysqld expects to receive some sort of information back from each client every so many (interactive_timeout or wait_timeout) seconds. If that time elapses without receiving anything, mysqld assumes that the connection has died. Rather than letting the valuable information go unnoticed, it logs that as a warning. mysql_close() tells mysqld that the connection is closing normally and the server should do its normal cleanup (no need to log normal closures).

If your code or the library it utilizes on doesn't call mysql_close when your program exits, mysqld will eventually log an error like the one above. If you utilize persistent connections, you will need to make sure that your connection talks to mysqld on a regular basis to keep that connection alive. If your library handles that for you, great. Otherwise, you'll need to "ping" mysqld at least once every interactive_timeout-1 or wait_timeout-1 seconds depending on your connection type. Most people should use wait_timeout for programs and interactive_timeout for mysql command-line client connections.

In rare occasions, some developers may also be concerned with net_read_timout - the amount of time mysqld will wait for data being transmitted to it in order to complete an operation. If you're sending data to mysqld and the sending process gets interrupted for a long time, this may also trigger the error listed above.

On the network side, I've seen this type of error is when the network is having a problem between the application layer and the database. How do we know which is which? We do all we can to make sure that the application layer isn't causing these issues so that when this error pops up, it's far less likely to be a preventable software coding issue.

Saturday, May 7, 2011

Just for fun...

0 comments
There are moments in every life when we ask ourselves - why do I do this?

One of the reasons I enjoy doing what I do is to get away at times and go ride... In my case, a Honda Silver Wing (FSC600). I enjoy the "SWing" so much that I've made it a point to take it with me when I can. :-)

Here are some of the places I've ridden so far... (updated 30May11)


Tuesday, September 14, 2010

Dude's Law: Meaningful Thinking

0 comments
I had never really thought much about how coaches get to be really good at what they do. I (like many, I assume) think that being a really good player makes someone a good coach of players. Then I thought about the adage - "Practice doesn't make perfect, practice makes permanent. Right practice makes perfect." The problem is - without the right feedback, there is little chance of knowing that practice is right.

The article I'm pointing to here inspired me to think more about "why" I do things. Why do I go to work each day besides earning a paycheck? Why do I blog? Why do I play disc golf? Why did I get married? Why do I stay married? Why did I make so many of the decisions I've made in my life?

The more I ask why, the more I have to answer "so that." By being conscious of the "so that's," I think living life can become much easier.

Friday, May 14, 2010

Timestamps in tables please

0 comments
As MySQL DBA's, it helps us to understand when rows were added in a table and when those rows were last updated. As a result, we're requesting that all tables have the following two columns in them to help us track those changes:

, created TIMESTAMP NOT NULL DEFAULT '0000-00-00 00:00:00'
, updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP

This should help when it comes time to maintain rows in all your tables because the database knows how to handle these two columns by itself given the right scenario:

mysql> CREATE TABLE foo (
->     foo_id SMALLINT NOT NULL AUTO_INCREMENT PRIMARY KEY
->   , bar varchar(24) NOT NULL DEFAULT ''
->   , created TIMESTAMP NOT NULL DEFAULT '0000-00-00 00:00:00'
->   , updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
-> ) ;
Query OK, 0 rows affected (0.04 sec)

mysql> INSERT foo ( bar, created, updated )
-> VALUES ( 'hello', NULL, NULL )
->      , ( 'world', NULL, NULL )
->      ;
Query OK, 2 rows affected (0.00 sec)
Records: 2  Duplicates: 0  Warnings: 0

Notice here that we intentionally insert NULL values into the created and updated columns. These will be rows 1 and 2 below.

mysql> INSERT foo ( bar ) VALUES ( 'hello again' ) ;
Query OK, 1 row affected (0.00 sec)

This time, I'm not setting the created field at all so you can see what happens when we skip it. I'm not crazy about this inconsistency, but it's what we have to work with for now.

mysql> UPDATE foo SET bar = 'from here' WHERE foo_id = 2 ;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> SELECT * FROM foo ;
+--------+-------------+---------------------+---------------------+
| foo_id | bar         | created             | updated             |
+--------+-------------+---------------------+---------------------+
|      1 | hello       | 2010-05-13 15:23:01 | 2010-05-13 15:23:01 |
|      2 | from here   | 2010-05-13 15:23:01 | 2010-05-13 15:23:19 |
|      3 | hello again | 0000-00-00 00:00:00 | 2010-05-13 15:23:03 |
+--------+-------------+---------------------+---------------------+
3 rows in set (0.00 sec)

Notice that row 3's created column has the 0000-... default value in it. Rows 1 and 2 don't because we actually inserted the NULL value into the NOT NULL column. That demonstrates that inserting NULL into a TIMESTAMP column created with NOT NULL will cause the column to use the current time when an action is performed (INSERT/UPDATE/REPLACE). This does not work with DATE, TIME, and DATETIME fields.

Want to learn more? Check out this link.

Sunday, March 28, 2010

Two of my favorite "toys"

0 comments


Doing work all day long can get rather boring.  So, here, I am showing off one of my hobbies.  The magazine is there for perspective in size.  The small helicopter is a Blade CP+ and the larger is a Bergen Gasser though not fully assembled.  As you can see, the Gasser is about 2.5 times the size of the Blade even with the training gear on.


Monday, February 8, 2010

Automatically Generate Configuration Files (Part 1)

0 comments

This article begins a series on using m4 to generate configurations for MySQL but applies to other configurations as well. Why would I want to generate configurations versus just managing each configuration separately? How often do we feel we need to make a change that applies to several instances all at once? Wouldn't it be nice if there was an easy way to change something in one place so that wold be reflected in all the configuration files under our care? What about when we want to see the same change in all the files except one or two? What about when we want to see that change happen in only one or two files?

There are a plethora of templating, macro, and text processing systems available that can help administrators deal with this type of challenge. I am using m4 for my own needs because it is very simple to understand and maintain while retaining the flexibility I need to go far beyond standard search and replace. I use m4 to help me generate configurations for mysql, httpd, bind(DNS), and other software. Generating configurations makes it much easier for me to maintain, test, and version my changes across several systems.

Enough of the lead-in, let's get started using m4 in a meaningful way.

m4 is available on most Unix-based systems automatically. If it's not available on yours, check your repositories. On Windows, m4 is available in Cygwin. If you don't already have Cygwin, it is a way to get many of the tools that run in Unix-based environments without having to run Unix free of charge.

m4 is a macro processor that has its roots in the C language processors. It takes a stream of text, looks for certain keywords and uses those as macros that are expanded based on the needs presented to it. m4 can handle loops, conditionals, and other common programming constructs. m4 is a tool that helps users by keeping repetition down to a minimum.

m4 uses processing instructions to determine when it needs to do something. I've told it only to read processing instructions that start with m4_ because I don't want it confusing my instructions with real information that needs to be left alone. It's just a precaution, but a good one and I encourage you to use it as well. To get m4 to require m4_, simply add a -P to your m4 command (as shown below).

The first instruction I'll show you is m4_define - the root of all that's m4. What good would a macro processor be if we couldn't define macros to be expanded? Isn't that like having a sail boat without a sail? Anyway, I digress. The syntax for m4_define looks like this: m4_define(`defined_label', `macro expanded') with the defined_label being the text that m4 will substitue and macro expanded being what it will replace defined_label with. Notice that I've wrapped these items with a back-tick and a regular tick. This is important and I'll explain more about that in upcoming articles. For now, let's get with the program! ;-)

$ cat my.cnf.m4
m4_define(`DEF_SERVER_ID', 1)

[mysqld]
server-id = DEF_SERVER_ID
$ 

If we run this file through m4, we will get the following:

$ m4 -P < my.cnf.m4


[mysqld]
server-id = 1
$ 
Notice that the definition of the DEF_SERVER_ID variable seems to have disappeared. That's true - m4 "ate" the definition macro. The new line is still there but that's all that's left of the first line of the file. What's cool, however, is we now have one place where the server ID was specified and another where it was used. No big deal there - what value does that give me, you ask? None at all unless you really like repeating yourself a lot. Of course there is more. Let's make this a little more meaningful.
$ cat my.cnf.m4
m4_define(`DEF_MYSQLDIR', `/var/lib/mysql')
m4_define(`DEF_SERVER_ID', 1)

[mysqld]
server-id = DEF_SERVER_ID
log = DEF_MYSQLDIR/logs/mysql.log
socket = DEF_MYSQLDIR/mysql.sock
pid-file = DEF_MYSQLDIR/run/mysql.pid
datadir = DEF_MYSQLDIR/data
$ 
Did you think through what m4 will do for us? Think about your answer before you read further.
$ m4 -P < my.cnf.m4



[mysqld]
server-id = 1
log = /var/lib/mysql/logs/mysql.log
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/run/mysql.pid
datadir = /var/lib/mysql/data
$ 
Pretty cool, eh? Well - maybe not yet. Did you get it right? What if I decided I needed to move my MySQL instance to another directory like /opt/mysqlInstance1 ? Rather than having to search for and replace all the text of "/var/lib/mysql" changing it to "/opt/mysqlInstance1", all I need to do is change the definition of DEF_MYSQLDIR at the top of the m4 template, then re-run m4 on my template and I get the new ouptut file. That's what I call slick. :-)
$ cat my.cnf.m4
m4_define(`DEF_MYSQLDIR', `/opt/mysqlInstance1')
m4_define(`DEF_SERVER_ID', 1)

[mysqld]
server-id = DEF_SERVER_ID
log = DEF_MYSQLDIR/logs/mysql.log
socket = DEF_MYSQLDIR/mysql.sock
pid-file = DEF_MYSQLDIR/run/mysql.pid
datadir = DEF_MYSQLDIR/data
$ m4 -P < my.cnf.m4



[mysqld]
server-id = 1
log = /opt/mysqlInstance1/mysql.log
socket = /opt/mysqlInstance1/mysql.sockm4 
pid-file = /opt/mysqlInstance1/run/mysql.pid
datadir = /opt/mysqlInstance1/data
$ 
So - that's nice, but there's a lot more to m4 than just being able to define macros. One of the most powerful tools that m4 offers is the ability to pull in other files. I'll explain by example:
$ cat my.cnf.m4
[mysqld]
server-id = DEF_SERVER_ID
log = DEF_MYSQLDIR/logs/mysql.log
socket = DEF_MYSQLDIR/mysql.sock
pid-file = DEF_MYSQLDIR/run/mysql.pid
datadir = DEF_MYSQLDIR/data

$ cat instance1.cnf.m4
m4_define(`DEF_MYSQLDIR', `/opt/mysqlInstance1')m4_dnl
m4_define(`DEF_SERVER_ID', 1)m4_dnl
m4_include(`my.cnf.m4')
$ cat instance2.cnf.m4
m4_define(`DEF_MYSQLDIR', `/opt/mysqlInstance2')m4_dnl
m4_define(`DEF_SERVER_ID', 2)m4_dnl
m4_include(`my.cnf.m4')

$ m4 -P < instance1.cnf.m4
[mysqld]
server-id = 1
log = /opt/mysqlInstance1/mysql.log
socket = /opt/mysqlInstance1/mysql.sock
pid-file = /opt/mysqlInstance1/run/mysql.pid
datadir = /opt/mysqlInstance1/data

$ m4 -P < instance2.cnf.m4
[mysqld]
server-id = 2
log = /opt/mysqlInstance2/mysql.log
socket = /opt/mysqlInstance2/mysql.sock
pid-file = /opt/mysqlInstance2/run/mysql.pid
datadir = /opt/mysqlInstance2/data

$ 

So - I added two things: m4_include and m4_dnl. m4_include says basically go read this file and continue processing as if the text were part of this file. m4_dnl says discard to next line. It's a form of comment but I often use it to mean "don't new line."

You notice in the example above, I created two very small files with the m4_define statements I needed (defining my macros) and then included the main template where those definitions were expanded as explained in the smaller files. So - rather than creating one my.cnf file with the definitions for one instance then copying and hand-modifying that second file for the second instance, I am able to write one template and a couple of small definition files to get the same result. If I need ot make a change to both files at the same time, I can simply change my my.cnf.m4 then re-generate my my.cnf files for each instance.

In today's tutorial, you saw how to create and get m4 to expand basic macros, how to include other files into an m4 macro file, and how to create comments. In our next episode, I'll explain how to make m4 smarter by showing you how to make it ask questions and do things differently depending on the answer. I'll also explain how to get the make command to automatically handle creating all your configuration files for you as well as getting the files into source control.

Do you like this series? Please post a comment! :-)

How to COUNT in SQL

0 comments
Often, I see users attempting to count rows of a table with COUNT(*) as a part of their query. Regardless of what database you're using, this probably not an optimal way to count.

COUNT(*) is likely to cause the system to do a table scan to find out exactly how many rows are available. Table scans are a very bad thing because it causes the system to go to disk to read the table. COUNT(columnName) on a column that is indexed will allow the database to use the index to compute the count (meaning it reads less data and may even be able to do it entirely from RAM if the index is already loaded there). The difference between COUNT(*) and COUNT(columnName) is a simple one - COUNT(*) counts every row where COUNT(columnName) counts the rows where columnName has a non-NULL value.

My favorite way to do counts (when I don't already have a summary table telling me how many rows are in the result set I'm looking for) is to count the non-contextual primary key in the table since I make it a habit of putting a non-contextual primary key in every table I can. This takes advantage of every possible optimization I can and keeps the amount of data the system must read down to a minimum.

This is a great practice to be in because it helps not only in MySQL, but in other databases as well.

Thursday, January 7, 2010

MySQL Styles

0 comments
There are nearly as many ways of writing SQL code as there are people writing it. Each person has his or her own style. While that may make it easy at one time or another for the writer, I suggest that using a well-recognized style makes it easier to read and maintain code (SQL or otherwise). Good style also makes it easier to spot problems with code. Whether you like the style I prefer or not, simply thinking about why others use one coding style over another can help us all communicate through our code better.

Before I launch into the methods I'm using, I suggest you check out http://sqlinform.com/ (I'm not associated with the site, however, I use the free on-line SQL formatting tool there rather frequently). When I'm trying to decipher someone else's code formatted in some other way (often just mashed together on a single or a few lines), this tool gives me a quick/easy way to re-format my code in a way that helps me read through the code. It clearly delineates the pieces of the code structure and helps guide my eye through it. Given the proper settings, I am able to read code so much easier that I am encouraging all the DBD's I work with to utilize this tool whether or not they're writing SQL for MySQL.

Imagine you were handed a file that looks like this:


USE DBFOO; START TRANSACTION;
DELIMITER ;;
// SP_GET_FOO_DATA CREATED BY SOME BEGINNER TO MAKE IT EASIER TO MAINTAIN HOW FOO DATA IS RETRIEVED
CREATE PROCEDURE SP_GET_FOO_DATA(V_FOO VARCHAR(2048)) SQL SECURITY INVOKER BEGIN SELECT SPLATTER.FOO, BLIP.BAR, BLOP.BAZ FROM BLIP LEFT JOIN SPLAT SPLATTER USING(SPLAT_ID), BLOP WHERE FOO = V_FOO AND BLIP.BLIP_ID = BLOP.BLIP_ID; END;; COMMIT;;
DELIMITER ;


Don't run for the hills... That's some funky code but let's see what we can do to clean it up:

1) Using START TRANSACTION and COMMIT are both unnecessary here because DDL statements (like CREATE PROCEDURE, ALTER TABLE, CREATE TABLE, etc) implicitly call COMMIT for you whether or not you're in an active transaction. MySQL silently ignores COMMITs when dealing with non-transactional storage engines but the point is, adding a COMMIT here doesn't add anything, so let's get rid of the excess transactional code.

2) Adding a USE statement may or may not be appropriate. At a minimum it makes deploying the code to a different database that much more difficult because the person applying it will need to change the USE statement accordingly.

3) Am I the only one who finds it hard to read text in all caps all the time? Okay - don't answer that. Obviously, I'm not the only one who would rather see text mostly lower case and all caps only when it's time to draw the reader's attention to something special.

4) The comment that is placed right before the CREATE PROCEDURE statement makes it hard to identify where the comment begins and ends versus where the code begins and ends. These lines need to be a) split apart at a minimum. It makes sense to me that the code would look and read better if the comment became a part of the DDL so it will show up in the SHOW PROCEDURE STATUS statement when called.

5) Long lines are no fun to read when the line is longer than your screen width setting. To get the whole picture, one must either decrease the font size to a sometimes unreadable value, or manually edit the file to break things up. When I do reviews, I want to spend time reading code, not reformatting it.

6) As I mentioned above, all caps makes it harder to follow text. It's a very common standard used by SQL writers that SQL clauses (like SELECT, WHERE, AND, VARCHAR, etc) are typically expressed in capital letters while database, table, column and variable names are expressed in lower case. This makes it much easier to find the clause or parameters to a clause much easier when utilized in conjunction with the other methods below.

7) Breaking up lines into logical groups helps the reader get the big picture fast. When I am writing SQL code, I like to see my column listings in a group, each JOIN in a line group, the WHERE clause in a line group, etc.

8) Combining implicit and explicit joins in MySQL 5 with a sql_mode of STRICT will cause MySQL to complain loudly. Even if we hadn't used the explicit join in the middle, adding the implicit join after the explicit join throws many for a loop. By writing explicit joins, we lead the reader through the code.

9) When aliasing columns or tables, use the word AS even though it's not required. This helps the reader quickly see that the table or column is being aliased. Without it, the alias isn't as easy to find.

10) Combining JOIN USING with implicit joins is typically a problem for MySQL and will likely cause a syntax error. Consider being more explicit when using the USING clause.

11) Positional sensitivity creates fragile code. It's better to use order and group by with the names of the columns than their positional counterparts because the positional order can change without negatively impacting the functionality.

I have some other rules I use for my own code that make it easier for me to read SQL. Can you find them below?


USE dbFoo ;
DELIMITER ;;
CREATE
PROCEDURE sp_get_foo_data(
v_foo VARCHAR(2048)
)
SQL SECURITY INVOKER
COMMENT 'Make it easier to maintain how foo data is retrieved'
BEGIN
SELECT s.foo
, b1.bar
, b2.baz
FROM blip
AS b1

LEFT
JOIN splat
AS s
ON s.SPLAT_ID = b1.SPLAT_ID

INNER
JOIN blop
AS b2
ON b2.blip_id = b1.blip_id

WHERE s.foo = v_foo
;
END ;;
DELIMITER ;


You can see I like to keep operators and SQL clauses to the left of a visual vertical column (right justified) and indent according to the level of code (new blocks like the begin/end block are indented). I also prefer to be explicit with table aliases for each table in-use. I go through this extra step when tables are being joined even if those names don't conflict. This makes it easier to quickly indicate which table the column comes from. When I have long lists of comma-separated lists of items (columns in this case), I like to treat the comma as an operator and start a new line right before the comma then indent according to the list. I put the comma first because it makes adding and removing lines from the list much easier that way. When it makes sense to do so, I also like to vertically align arithmetic, comparison, and assignment operators.

I could go on, I think you might be able to see some of the why behind my own formatting preferences. What preferences do you have and why?

Wednesday, December 2, 2009

Motorola Droid versus Blackberry 8830 World Edition

0 comments
As a DBA, I often need to be available to perform work on databases I am responsible for when I am away from the office. I have two phones - a corporate Blackberry 8830 World Edition and my personal Motorola Droid. Both phones are fully featured and support a large number of applications. Both phones are impressive in their own right on capabilities, though the thing I feel I miss most on the Blackberry is the turn-by-turn verbally announced directions (versus VZ Navigator on my old Glyde). The ssh client in the Droid (ConnectBot) seems far more capable than the ssh client I use on my Blackberry. I had difficulty learning the Blackberry development environment and while I didn't try terribly hard, I was not able to find a simulator to help me test my applications before deploying them to a Blackberry. I did not have that problem at all with my Droid. The Droid simulator was very easy to find and works wonderfully for me. The integration into Eclipse also makes it very easy to utilize.

With what I know about the Droid versus the Blackberry after a few months of use, for my needs, the Droid beats my 8830 hands down.