So long, and thanks for all the fish

In May of 1998 I was laid-off.  It’s the one and only time in my life that has happened, but it did, and I expected it.  I was working in the semiconductor industry and lay-offs happened on an 18 month cycle.  I was fortunate at the time to have very few bills, outrageously low rent and was able to survive quite nicely on my unemployment benefits and my severance.  By the end of July I was bored – bored, bored, bored; and my unemployment was close to running out.  It was time to find the next job, so I started the job hunt in earnest instead of just applying for the required 3 jobs a week and spending the remainder of my time reading in coffee shops, playing around on my favorite BBS, and drinking beer with friends over a game of billiards.  At the time I had a friend that had been working for this jewelry company in town – they were a mail order place, with a call center and warehouse and they were always hiring… He liked it, but was moving on to something else, and strongly suggested I apply.  Worst case they paid decently for the area and had amazing benefits, and the CEO wished every single employee a good morning, every single day.

Needless to say I applied for a call center position – I had done call center work before, and the pay was okay – not as much as I had been making in the fab, but certainly more than unemployment and at this point I needed to get back to work.

After the strangest pre-employement assessments and interview process I had ever been involved in I eventually got a call that they wanted me to start at the end of August.

On August 24, 1998 I started my next job in my short history of jobs as an adult.  I really thought it was just a job – something with decent hours, a set schedule, low stress – I mean, even the most upset customer had to admit that no one ever died from a jewelery emergency!  Looking back now, I couldn’t have been more wrong about the path I was taking that late summer day.

I did a very short tenure in the call center, and was pushed by my business coach to apply for an open position with the Information Systems Operations team.  I’ve written about that before, so won’t rehash it again here.  I will say that if it hadn’t been for that one single push, my life would be very different today, and I can’t thank my coach enough for it.

Today – after 14 years and a handful of months I am turning in my keys to that building, and I am pulling out of the parking lot for the last time, and I’m likely shedding a tear or twelve while I do it.

I’ve turned over the systems I’ve managed and supported for the last decade, I’ve dumped as much historical and technical knowledge as humanly possible to my teammates, and said my good byes.

While I was technically an adult when I started my tenure at Rio Grande, I can honestly say that I grew up inside those walls.  I learned that I can stand up for myself in the face of sexism, I learned that I can excel at any damn thing I put my mind to, and I learned that I am the only thing standing in the way of my happiness and success.  I have struggled to live a principled life – sometimes succeeding, and sometimes falling flat on my face.  I learned what it means to be really, truly, on the hook for keeping the system that runs the business up and running.  I learned how to give and receive feed back in a positive manner regardless of the content of what is said.  I learned how to  have fun while working harder than I’d worked before.  I learned how to trust that everyone had the same goal in their sights.  I learned how to admit that I don’t know an answer, but would be happy to find it.  I learned how to crimp CAT5 very late one night in Tuscon AZ because of Rio.  I learned that I can – if needed spend 5 straight days at work to fix a critical problem.  I learned that I enjoy making things work, and making them work better than they did before.   I learned that no matter what I know now, there is always something more to learn.  I found my community with #SQLFAMILY I learned who I was and who I don’t want to be inside those walls.

I made more friends than I can count while working at Rio – some of those friendships fleeting, and others are life long, and I can only hope that I will work with the same caliber of people again in my life.

So, why leave?  Because all good things must end and it’s time I took my career into my own hands.  ExactTarget offers me experiance that I can’t get anywhere else.  I will be joining Eddie Wuerch and the rest of his highly intelligent, super cool DB Performance team on Monday to learn the ropes of high volume, high demand performance tuning.  ET’s set of 8 principles mesh well with the principles I’ve worked under previously, and the work excites me in a way that I can only compare with the high I get when I get together with my #SQLFAMILY.  It’s time for a challenge, and ET offered me that & the icing on the cake is that I don’t have to relocate to get it.

Now, almost 15 years later, I find myself unemployed.  For the first time in more than a decade I’m not on the hook for a production system, and I’m only carrying a single cell phone.  At least this time it’s only 48 hours or so before the next chapter starts at ExactTarget.  I sure hope they are ready – ET has big shoes to fill after Rio!

Advertisement

Post SQL Install checklist

I’m finding myself in an uncomfortable (for me) situation these days.  I have SQL servers being built and configured on my network with out my input or assistance and there’s not much I can do about it – other than wait for the VAR to be finished and give me access to the boxes.

To prep for that day I’ve started documenting all those things that I generally do as part of my installation process, but will now have to do after the fact.  This list is by no means complete, nor in a particular order – and really a reminder for myself, but hopefully it will help some of you as well.

1.  Alerts for severity 16-25, errors 823, 824, 825

2.  Operators

3.  DB Mail

4.  Backups

5.  Index maintenance

6.  Check DB scheduled

7.  STATs maintenance

8.   Check permissions for ‘Perform vol Maintenance tasks’

9.  PageFile Size

10.  Check Memory configs

11. Default file paths for DB’s

12.  Check model db configs

13.  Create DBA database

14.  Install SP_whoisactive

15.  Install & run SP_Blitz

16.  Check TEMP db setup – how many files, autogrowth, etc

Edited 12/5/12

17.  Verify windows power management settings

18. Verify SQL Agent is mail enabled

 

I will continue to add to this list as I think of things.

XDBA 2012 follow up

I’ve just returned from Summit 2012 where I was officially recognized as the Exceptional DBA of 2012.  The love I felt from my peers and the community at large was almost overwhelming – in the very best way 🙂
Several times I had folks ask me what I did to win this award, so here are my answers to the XDBA 2012 interview questions for the world to see.

-Enjoy!

Hey look! that has my name on it!

 

Q1 of 5: Briefly describe your job history as a DBA.

I am a self trained, home grown DBA. I’ve been with the same company for the last 14 years, starting out on the sales floor and moved quickly into IT. Within months I moved from my position as the ‘help desk girl’ to administrating business critical systems. When our company migrated from our legacy VMS system to a SQL Server based ERP I stepped up to be a backup resource for our lonely DBA. When he left the company I took over as the solo DBA and I’ve happily been in that position for the last 6 years

 

Q2 of 5: What has been your biggest accomplishment so far in their DBA career?

I feel like I have several accomplishments in my DBA career, but the one I’m most proud of to date is that I’ve managed to maintain the availability and performance the company needs out of our Dynamics GP databases as we’ve grown as a business. We are one of the largest installs of GP in the US, have a highly customized database and have a high (for GP) daily transaction rate. Despite that I’ve kept it running well for the last 6 years.

I am also very proud of my community involvement. Over the last three years I have become quite involved in the PASS family as the virtual chapter leader of WIT, an officer on the board for my local PASS chapter, offering assistance on #SQLHELP and speaking at SQL Saturdays.

 

Q3 of 5: What is the biggest mistake you have made as a DBA, and how did you deal with it?

To date the biggest mistake I’ve made as a DBA was a total rookie mistake! I was in the midst of preparing a test server for use. In our shop that includes a full restore of production databases followed by cleaning up database users and a handful of other configurations that reside in the databases. This is a process I’ve done so many times I could almost do it in my sleep, have completely documented and scripted. It was a busy day in the office I share with the rest of the Operations team and I was distracted. I finished the restore and before I could start the script to drop users from the databases I was called to handle a quick issue on production.

I took care of the prod issue and then opened my cleanup script and hit F5. It took all of 30 seconds for my phone to start ringing – I had dropped all users from production at 2pm on a Tuesday!

Needless to say it didn’t take long for me to figure out what I had done, and I asked my teammates to handle the phone and questions while I fixed the issue. Fortunately I had just restored the previous nights backup so I had the users listed out on my test system and was able to move them over to prod using scripts I had saved from a previous migration. I think it took less than 10 minutes to get us back up and running.

Once I made sure the business was functional again I set out to modify my cleanup scripts to prevent this from happening again. I added a simple bit of T-SQL to the beginning of all of my scripts for setting up a test system that checks the instance name. If that instance name is our production system the script fails.

My last task for that day was to write up what had happened, how I modified my process to prevent it from happening, and what the business impact was and send that out to the leadership team at my company.

 

Q4 of 5: What SQL Server community activities have you participated in?

I am a sometimes blogger at meredithryan.worpress.com, have a few articles over at sswug.com, and participate on #SQLHELP as time permits. I also try to get to at least 4 SQL Saturdays in the southwest US a year either as a speaker or attendee. I have been attending the PASS Summit every year since 2003, and for the last three years have been the chapter leader of the WIT VC. For the last year I have been on the board of my local PASS chapter in Albuquerque, as the Communications Officer and Vice President and plan on continuing in that role for as long as they will have me.

 

Q5 of 5: Finally, why do you think you should win the Exceptional DBA Award?

I’ve spent the last 6 years trying very hard to balance the needs of my business with best practices for SQL Server and ensure the safety and reliability of our databases. This hasn’t always been easy, and has taken some creativity on my part and my coworkers parts, but I think I’ve done a top notch job of it. I don’t always get to follow best practices with my databases, and some of the processes and practices we have in place would make other DBA’s cringe, (some of them make me cringe too), but I know for a fact that every decision I’ve made has been to give the business what it needs to maintain growth, and in turn meet the needs of our customers.

I am by no means an expert in SQL Server, but I go out of my way to pick up the skills the business needs me to have, and to admit when I don’t know something. Once I identify that knowledge gap I either go learn what I need to learn or help find the appropriate resource to fill it.

On Being Exceptional

I was recently named as Redgate’s Exceptional DBA for 2012 and let me be the first to tell you that winning that award floored me.  Seriously, completely, absolutely floored me. 

Let me tell you a little secret here:  I don’t think I’m really all that exceptional. 

I think I do my best, and I try very, very hard to meet all of the needs put on my plate, and I go out of my way to try to anticipate needs, but I don’t think that makes me exceptional….

and that’s okay. 

Time for another secret:  Since being named as the 2012 Exceptional DBA I’ve received tons of congratulations, a few cards and even a money tree.  The attention has been a bit overwhelming at times, and every time I think it’s fading another round of emails and phone calls come in.  This is a strange place for me to be.  It really is.  Don’t get me wrong – I like to hear the occasional ‘Good job!’ as much as the next girl, but I’ve discovered that I have no desire to be a super star.  I’ve found myself averting my eyes, and mumbling a thank you when folks stop me in the hall at work to congratulate me, blushing when the award is mentioned in meetings, and down right uncomfortable when I am introduced as an award winning DBA.

My comfort zone is off in the wings, offering support to those I can, and solving problems in my own quiet way.

So, you might ask:  “Meredith, if you know you don’t like the full glare of the flood lights why did you enter the Exceptional DBA contest in the first place?”  – great question! 

I entered for one simple reason.  I’m not very good at singing my own praises.  I am one of those people that deflects praise quickly and deftly.  If I solve a problem I will quickly include everyone that helped in any way.  I try very hard to include my coworkers in any small victory.   I have also come to realize that not everyone operates in the same way, and if I am not ready to tell the world what I do, have done, know, or can find out no one else will – at least not all the time.  I entered the Exceptional DBA contest as a way to practice answering questions about me and my work in an honest and flattering way.  I had no expectation that I would be named a finalist, let alone win.

So, this is my long and rambling way of saying thank you to the community members that voted for me, and to the judges that felt I deserved to be elevated into the final five in the first place, and also a way to remind everyone else out there that just because you don’t feel like you are exceptional all the time, chances are good that you are exceptional to the people that need you to be.

 

 

 

 

Changing Gears again.. it’s the life of a DBA

When I last wrote my big project was to upgrade our primary business systems to SQL 2008 while we sourced a new application to replace our legacy one.  At the same time I’m a resource for the new application sourcing.

As I began planning upgrading my existing windows 2003/sql 2005 cluster I realized that the resources I needed for my upgrade were already hip deep into the sourcing project.  This kind of resources contention is never a good thing, especially because we are all still on the hook to support the environment and take care of the day to day business needs.

Being a cautious DBA I was planning a side by side upgrade instead of an in-place upgrade and this meant that all of our client workstations, reports, Access routines (gasp!  I know, we have them and rely on them), SSIS packages, etc would need to be pointed to the new cluster and tested before we could cut over.  It sounds like a short list and my first reaction was that it would be easy – we know where all the reports, access routines, etc live so no problem…

Once we dug in and detailed out all the different places that the instance name would have to be changed it became quite daunting very quickly.  Especially considering these systems, reports, packages, and Access routines would all be scrapped in just about 12 months with the new software we are sourcing.
I made a quick couple of lists to help me decide if we should press on – it’s what I do as a first step when making a decision like this.
Pro’s to upgrading to 2008

* we’re back on a platform that  is supported under mainstream support

* I get to use page/row compression

Con’s to upgrading to 2008

* risk that we will miss something in testing and cause an outage for the business

* over allocation of people resources

* we have a stable system now, and upgrades introduce instability

I know there are more benefits to upgrading from 2005 to 2008 than I’ve listed, but to be honest, those are the two that we would have taken advantage of immediately.  After thinking a bit and talking to my team we made the decision to scrap the 2008 upgrade for these business systems.

So, it’s time to switch gears.  I’m focusing on gearing up with SQL 2012, evaluating Always On for HA, planning a new architecture for the new business systems, and keeping my old 2005 cluster running for the next 12 months.

My plan right now, is to make notes of what I’m learning as I go along here on my blog for my own reference – hopefully it will help you as well.

It’s never easy is it?

We recently made a business decision that upgrading some of our business systems to current versions didn’t make much sense – they don’t really meet the needs of the business and the upgrade project would take almost as many resources as replacing them.  The replacement project has kicked off and it looks like we will have new systems in place within 18 months or so.

This is all well and good – migrations are always a more interesting project for me than upgrades, but it left me in a bit of a quandary about what to do with the SQL install that supports these systems.  I’m still running these databases on SQL 2005 (I know! I know!) and could really take advantage of page/row compression for some of them.  Of course, to do that I need to be running at least SQL 2008 so it is time to start planning and testing my upgrade paths.  After doing some research on what versions of SQL the applications supported with our current version of software I realized that as much as I wanted to I couldn’t upgrade to SQL 2008 R2, and instead was limited to SQL 2008.  It’s not great, but better than supporting  these systems on 2005 for another 18 months.

I have a big old honking test system that holds five named instances of SQL for test/dev/qa.  Because we were going to upgrade the software to current it was a mixed bag as far as my SQL versions.

My current reality looked like this:

Instance1 – 2005 sp2

Instance2 – 2005 sp2

Instance3 – 2005 sp2

Instance4 – 2008 R2

Instance5 – 2008 R2

 

I needed it to get to here:

 

Instance1 – 2008 sp3

Instance2 – 2008 sp3

Instance3 – 2008 R2

(Instances 4 and 5 could be removed – they were installed for the application upgrade, and not needed any more).

Easy-peasy right?  Not so much!

Initially I (naively) assumed that because I was dealing with named instances I could upgrade the 2005 instances to 2008 with ease even though I had a those two 2008R2 instances living out there.. I mean, I don’t care what version of SSMS is running, I just cared about the engine version for each instance.

That was my first poor assumption.  After attempting the upgrade and it failing for Instance2 I realized that I needed to remove the 2008 R2 bits from the server.

No problem I thought – I’ll remove 2008 R2 via the control panel since I don’t really need it currently and go on with my upgrades.  If only it were that easy!

I went round and round with this, and eventually wound having to use Aaron Bertrand’s guidance in this blog post to remove all of the 2008 bits completely.

Once that was complete I was able to perform in place upgrades to SQL 2008 SP3 for Instance2 and Instance3 and only once those were complete did I dare to attempt the upgrade on Instance1 to 2008 R2.

All in all I spent entirely too many hours on this, but it would have been even longer had Aaron not blogged about removing the eval bits of 2008R2 back in 2010.

 

For me it was just another reminder that you should ALWAYS test before attempting upgrades or changes in production.   Certainly not my favorite way to spend a Sunday afternoon, but it would have been much worse had it been in production and I had a steadily decreasing maintenance window.

T-SQL Tuesday #028 – Jack of All Trades, Master of None?

Argenis Fernandez is hosting this month’s T-SQL Tuesday and he asks us the following questions:

Are you specialized? On something? Or anything at all? Has that been a good or a bad thing? Why?

Are you the SQL guy at work? Or the one who does everything?

Do you code? And configure wireless routers at work also?

If you had to pick one thing to specialize on, what would it be?

I’ve mentioned at least once or twice that I fell into, not only DBA work, but IT work in general several years ago.  When I did get that help desk position back in 1999 event though my primary responsibility was to provide first level support, answer the help desk and escalate calls to other, more experienced technicians I found myself quickly picking up the skills needed to resolve an ever increasing number of trouble tickets on my own.  In addition, I found myself taking it upon myself to take on systems themselves.  I worked in a small IT Operations team – there were 6 of us in total and we managed everything from the desktops, network, phone system, server systems, application support, licensing, etc, etc…  I found myself picking up systems that no one else on the team wanted or had time to properly manage.  Things like our old VMS system, Anti-Virus system, and even hardware support.  At one point I could rebuild an Okidata printer in less than 30 minutes – just because I did it so often.

Because I tended to pick up the red headed step systems in our network I have gathered a general knowledge about many areas of IT, and I don’t consider myself an expert in any of them – and honestly I’m okay with that.  It kept me on my toes for the early years of my career.  I never knew if I would be working on a queue problem in VMS or a virus outbreak when I got to work.

Even now – with my official title of DBA I tend to wear a lot of different hats.  I manage licensing for the business, I manage most of our hardware purchases, I am a back up Exchange administrator, I still help out at the Help Desk when needed, and I’m a resource for the newer folks on our Operations Team.  There are times, when my databases are behaving themselves that I don’t even open Management Studio for days at a stretch.

It’s a mixed blessing in my opinion.  Taking this path into IT and being more a generalist than a specialist has given me the confidence that I can tackle any area of IT and become proficient, but the flip side of that coin is that I, at times don’t see a clear path to becoming a specialist, let alone an expert in any single area.

With all that being said, I suspect that I am the most specialized person in my team at this point, and I agree with what many others have said today.  Specialization is best when it follows a well rounded base knowledge.  You have an edge that others might not have with that base knowledge.  You can often see the bigger picture quicker, and you have alternate paths to explore when troubleshooting issues.

So, go be a Jack of all Trades for awhile – and when you’re done doing that think about the one area of IT that really caught your attention and focus on that for a while.  I would guess that it will serve you well just like it has me.

Slip Sliding Away

Time is creeping away from me these days.  I’ve had some major life changes in the last year and while I’m not going to air those here, I’ve at times felt it was all I could do to keep up with my day job, community responsibilities, and all those things that must be done to keep life rolling.  Needless to say, keeping up with this blog has been at the end of my long list of high priorities while I’ve attempted to find my new rhythm.  I think (hope) that I’m back to a place that will allow me to keep this semi-regularly updated from here on out.

Here goes nothing!

While I was away I managed to pack in Summit 2011, SQL Saturday 104, and SQL Saturday 109 as well as an ERP Vendor Shootout.

I’m not going to spend time trying to summarize my experiances at Summit and SQL Saturday 104 – It’s been too long, and I certainly wont’ do justice to either.  You should know that Summit 2011 was one of the best yet  – every year Summit gets better and better.  SQL Saturday 104 was great as well.  I delivered a Backup Basics session to a great group of folks, and the organizers did a fantastic job doubling the size of their event (or did they triple it?) from the first year.  Colorado Springs will be an event I don’t plan on missing in the future – and you should make the effort to join them next year.

I will tell you a little bit about SQL Saturday 109 though.  I have to hand it to Mark Ginnenbaugh, Ross Mistry and their team of volunteers.  They put on a great show, with an all star cast of speakers for their inaugural event.  I believe the attendee count was right at 450, which is a feat in itself for the first SQL Saturday in Silicon Valley.  The session selection was top notch – in fact I had tough choices to make for each session slot, and really wished that sessions had been recorded for playback after the event.  I think the top session for me was Grant Babb’s session on Pro-I : open source security monitoring on SQL2008 because it was completely new material for me.  Don’t get me wrong – security isn’t a new concept for me, but I’d not considered what it would take to create a proactive monitoring system end to end using SQL Server and the work he and his team did is fascinating to me.  In fact I plan on sharing it with my co-workers this week as something to consider in our shop.

Other sessions I attended included Angel Abundez’s session on Reporting Services for mobile platforms, Wendy Pastrick’s Replication session, Denise McInerney’s session on Begin.. Commit, and Allan Hirt’s session on High Availability in 2012.  The day was rounded out with the Women in Tech lunch panel.  I had the pleasure of sitting on the panel with Wendy Pastrick (@wendy_dance), Jen Stirrup (@jenstirrup), and Nikila Srinivasan (@nikisrinivasan), moderated by Denise McInerney (@denisemc06).  We had a great discussion about why we keep talking about Women in Tech.  Both Jen and Nikila brought diversity and insights to the panel that we’ve not always had at previous SQL Saturday WIT events.

I’m still gathering my thoughts on the ERP Shootout – while it wasn’t really a SQL centric event in any way, the format was interesting and I can see a place for that type of event in our community – imagine if we had an event to see all of the 3 party packages for say, backups following the same scripts, using the same databases, on the same hardware, back to back.  Would that be valuable to you?  I could see how it would be if your company was in the process of evaluating toolsets.   I have more to say on the event in general and the format as well.

Let’s see some sparks people!

First a disclaimer – this post has very little to do with SQL, or even IT; rather it is all about community and sparks.

I attended Ignite NM 11 last night, and was very disappointed in myself that I managed to miss the first 10 events.  That’s right.  I missed 10 other opportunities over the last 3 years to participate in my community, and watch other people talk about their passions.  For those of you that are unfamiliar with Ignite talks they are 5 minute presentations similar to the Lightning Talks at PASS Summit (speaking of, go submit now you have until August 17, 2011) with a few differences.  At an Ignite talk you have 5 minutes, 20 slides, 15 seconds a slide.  Speakers submit abstracts and the community votes on their favorites.  The result is truly an evening full of diverse presentations by folks with differing levels of public speaking skills.  Last night I was introduced to organizations ranging from The Metropolitan Homelessness Project to the Challenger Learning Centers   and in the same evening learned a little bit about alternate plant sources for the paper industry, what it is like to live in a dune shack for 3 weeks, and how to publish a ‘zine.   All of that, and I missed at least 3 Sparks while I took a phone call outside.  The event ran just over 2 hours, included ample networking time before hand, during an intermission, and after the last spark and was a great way to reconnect with old and dear friends as well as meet a few new people that I otherwise would never run into.

The evening started as a way for me to get out of the house, avoid yet another night of talking to the dog and cats about SQL server and things happening at work, and wound up being just like a first date.  I left feeling energized about the community I live in.  I left with sparks of my own floating in my head.  I left with a long list of things and groups to research, a desire to help out with Quelab, and a reminder that there are folks out there that are passionate about the things they know in my local community.

That reminder is important to me for a couple of reasons.  I find myself energized when I surround myself with passionate people.  I thrive in situations that allow me to tap into someone else’s enthusiasm to recharge my own batteries.  Generally I get my recharge by attending PASS Summit, SQL Saturdays and even ABQSQL meetings.  To find yet another venue that is close to home to charge myself up is a fantastic thing.  To find another venue that introduces me to even more places to find passionate people is better than fantastic… fantabulous maybe?

So, what are my next steps?  Fortunately I have SQLSaturday #95 booked in September, PASS Summit in October and SQLinTheCity in late October so I can get my DBA fix, and in the time between those events I plan on meeting up with the gang at Quelab to see what I might contribute to their efforts, and I might just develop a spark of my own for IgniteNM 12.

You should do the same…

Bad Blogger, no cookie

It dawned on me this morning that it’s been just over a month since my last post, and for that I apologize.

I’ve got a long list of reasons excuses I could share, but I think for now I will just leave you with a promise that I will be back in action next week with a long post on how to be an effective IT admin without encroaching on others.