On Being Awarded my Microsoft MVP on the Data Platform

The email came on Saturday, April 1st.  It was already a crazy day, with my daughter competing in regionals, several hours away that evening.  Things were a bit of a whirlwind that morning getting her to dress rehearsal in full hair & makeup, costume ready to go and packing bags for our over night stay.  I'm having a very late, very rushed lunch with my boyfriend as 3 MVP calendar invites show up in my inbox.  There's been some mistake, I think.  It is April Fool's Day after all. I wasn't awarded an MVP with this latest round.  Was I?  

Time to check the Social & Promotional folders on my Gmail. And there it is

Congratulations 2017 Microsoft MVP!

I'm actually at a loss for words for a couple of minutes.  I place my phone, face down on the table and quietly say "I've been awarded the MVP.  I can think of so many other people that are more deserving."  To be considered among such company is incredibly humbling.  I'm immediately aware of the responsibility now upon my shoulders.   

So what is an MVP & how did I become one?

According to Microsoft, this award is given to exceptional community leaders who share their remarkable passion, real-world knowledge and technical expertise with other through a demonstration of exemplary commitment.  I suppose I do that.  I'm very active with my local user group and I organize SQL Saturday Atlanta (register or submit NOW!).  I speak at SQL Saturdays all across the US every year.  I'm an Idera ACE for 2017 where they encourage me to blog, answer questions in their forums, host #SQLChat and present webinars for them.  I'm co-leader of the PASS Women in Technology Virtual Chapter and host 15+ webinars a year where women speak on technical topics and highlight women technical speakers weekly as they're speaking at different events, presenting webinars or Pre Cons.  I mentor other women to GET OUT THERE and speak.  

Know someone else that should be?

Do you know someone that does all of this and probably more?  If so, I encourage you to NOMINATE that person to be a Microsoft MVP as well.  It applies for women in business and the same holds true for being an MVP.  Once you make it through the door, it's your responsibility to drag someone along with you.  I've already nominated someone that I felt was truly deserving of the award.  

Creating a Disaster Recovery Plan

        So your boss asked for a copy of your DR plan.  Once you've wiped that deer-in-the-headlights look off your face, you realize "We've got database backup.", isn't exactly a plan.  You'll need to define what a disaster could be, document the business impact identify your limitations.  So where to you start? Well, that's the easy part. 

Define what's important... before a disaster.

    There are lots of questions to ask before you have a disaster. They're even more important before you build your disaster recovery plan.  It's essentially the WHO, WHAT & WHEN of your plan. How long can you be down & how much data you can lose. Define what data is important.  Who is responsible for declaring a disaster? Who is responsible for doing the actual recover work? Even knowing where you store your plan is something to think about now.

Who are my stakeholders?

    In a single statement, they're the people that can answer all of the questions we're asking here.  This starts with C-level execs, since they'll be the ones that have to answer to the board after a disaster.  They're also the ones that have to pay for it.  Next, identify the people that will be affected by any data loss or a system outage.  Who can't do their job? Go through each application and ask yourself "Who cares?" and then follow up with them.  Finally, talk to the people that will have to implement the DR plan: System admins, networking & security, Operations, Storage & Infrastructure, DBA team, etc.

RPO & RTO

These are terms you see thrown around in sessions, in meetings and on twitter.  The official definitions are little wordy & not as helpful as you might think.

    Your Recovery Time Objective is a way of defining how long can you be down.  Or put another way, how long until you have to come back up.  So find out how your business defines RTO and then build toward that.  Of course, this can vary based on the nature of your disaster.   Did you lose a database, an instance or did someone kick the storage out from under everything? Did you have to fail over to your DR site?  Were you the victim of a DDOS attack, either directly or indirectly?

    Your Recovery Point Objective is easily summed up by asking "How much data can you lose?"  Some systems, like payments and healthcare systems, that answer will be zero.  The approach and expense for those systems is greatly different than others.  The infrastructure, build and design will vary by a large degree when you're allowed to lose milliseconds of data versus up to 15 minutes. For slow changing systems, could you restore on Tuesday from a Sunday full backup and Monday's differential? In some instances, rebuilding is faster and easier than restoring, so be sure to explore that as an option.  

    While Execs can help you define RTO & RPO, they aren't the ones who have to make it happen.  If you can't meet their requirements, be honest. You also need to be aware of contractual obligations to external clients.  While these are out of your direct control, you should encourage sales & legal to work with you while you prep a contract.  If they're promising 5 9's in uptime but you can only deliver 3 9's, they need to know.  Use phrases like "claw back", "refund of fees" or "violation of terms".  That should get their attention. 

Building a Backup Strategy

    Start with the basics.  Now that you know how much data you can lose and how long you've got to get things back up and running, set about making that happen.  Start documenting your backup processes then put them into place across the enterprise. Make sure all of your servers are following the rules you've established.  If you can lose 15 minutes worth of data and take 2 hours to come back online, then set the schedule.  Make sure you know how to restore the tail end of a log. Here, Tim Radney [T|B] shows us how in an older blog post that's timeless.

    We're not just talking about database backups.  If you use it, you'll need it.  Defining items to backup other than databases means an end to end examination of our business.  Plan on having to recovery things like Active Directory, Application configs, development source code, external files, encryption keys and passwords.  Script out some things ahead of time like SQL Agent jobs, a create logins script, database restores, service accounts, etc.  

Building a Recovery Strategy

    Any DBA is only as good as their last restore. That means you should be doing that regularly.  You'll want to establish recovery baselines.  During a disaster, the longer something takes, the more likely you are to panic. Make sure you know ahead of time everything you'll need to do for a restore and how long you'll expect it to take.  Make sure you have copies of database backups locally and a copy at your DR site. You should know how long it's going to take to restore.  Of course, the biggest pay off to practicing restores is knowing that your backups work.  

Test your plan. 

  

    Some sage advice from Allan Hirt, "If you don't test your D/R plan, you don't have a plan.  You have a document." Document your DR plan.  Practice.  Automate. Adjust. Document. Practice again.  It's the only way to really be sure.  That being said, if your business can't support a full fail-over test, consider a tabletop test instead.  Once your plan is documented, have a meeting with all of the people responsible for recovery.  Go through each and every step of your DR plan.  Make sure every one agrees this should work.  Make sure every one understands their role.  Make sure they have all the pieces they need to put this plan into action.

    Don't just practice for the big disasters either.  Practice for those smaller disasters & disruptions.  Things like DDOS, ISP going down, a data center power outage, natural and unnatural disasters.  While these aren't directly a DBA problem, if they cause you to fail to your secondary data center, it just became your problem.  Then there are the smaller but potentially devastating events: forgetting a where clause in production, drive failure, storage corruption or a malicious insider.  A pissed off employee can cause real issues deleting or modifying data that is vital to your company.  Be prepared to do an object level restore or a side-by-side restore to recover data that may have been compromised. 

Building it out. 

    Let's be honest.  Most companies can't afford to build & maintain a hot standby environment equal to your current production.  If yours can, good for you.  Feel free to skip this section.  But if they're like many companies, DR is currently sharing real estate with staging, QA or UAT.  It's not a hot site or it barely has the processing power you'd need to run your business.  

    Identify your wants versus your needs.  Lay out what you WANT your DR site to look like and how you need for it to function. Identify how you're going to keep it up to date.  Then lay out what you NEED to have for your disaster recovery site.   I suspect the final product will fall somewhere in the middle of your wants and needs.  

    Don't forget hardware, licensing and maintenance.  Plan for enough storage space for the live databases and backups. You'll need enough web servers to run your applications.  Don't forget to factor in enough time each & the peop0le required to build this out and maintain it on a monthly basis.  You'll have to patch all those servers & keep versions aligned. 

How much will it cost you to build?  Probably less than it will cost in lost revenue, client trust and public relations.

Let Her Finish: Supporting Women's Voices from Staff Meetings to the Board Room

It happens all the time.  Maybe you don’t notice because it’s so frequent, like a fish not knowing it’s in water.  It's just the way things are, right? Let's lay out the problem, then we'll talk about how to be part of the solution.  Women are interrupted in meetings at 3 times the rate that men are. It doesn't have to be that way.  

•       Studies from 1975 showed men were responsible for 47 out of 48 conversation interruptions.

But it’s gotten better, right?

•       Study in 2015 found that men not only interrupted twice as often as women, they were 3 times as likely to interrupt a woman.

•       Australian study findings include:

1.    women don’t speak as much without interruption as men.

2.    of 311 interruptions that questioned a speaker's authority and credibility, 213 were directed towards women.

3.    female witnesses were called emotional, unreasonable or words to similar effect 163 times, and 120 of these comments were made by men.  

4.    women were more likely to be punished for their interruptions, than their male peers, by the chair during public hearings.

Even on Diversity Panels

SXSW Gender & Diversity Panel – Google Exec Chair Eric Schmidt repeatedly interrupted U.S. Chief Technology Officer Megan Smith, the sole female panelist on stage with him during the talk. Finally, he was reprimanded by an audience member, Judith Williams, who also happened to be Google's global diversity manager.

Especially if it makes them uncomfortable

Most recently, Senator Elizabeth Warren (D-Massachusetts) was officially silenced while reading a letter on the senate floor.  The letter, authored by Coretta Scott King, was read into the Congressional record in 1986 by Ted Kennedy (D-MA).  The 30 year old letter directly addressed the topic at hand, the ability of Jeff Sessions (R-AL) to support the civil rights of all Americans. 

     ‘She was warned. She was given an explanation. 

      Nevertheless, she persisted’

        -Senator Mitch McConnell

Think it’s just an example of politics and not sexism?  Four other senators were allowed to read the letter all or in part just hours after Sen. McConnell silenced Sen. Warren.  I've listed them below.  Just shout it right now if you notice anything different about them.  Come on, I know you know the answer.

Tom Udall (D-New Mexico)
Jeff Merkley (D-Oregon)
Sherrod Brown (D-Ohio)
Bernie Sanders (D-Vermont)

Yup, you got it.  They’re men.  Do you think that’s a coincidence? I’m willing to bet that most working women know it’s business as usual. 

So how have other people solved these problems?

A No Interruptions Policy

Years ago, while producing the hit TV series “The Shield,” Glen Mazzara noticed that two young female writers were quiet during story meetings. He pulled them aside and encouraged them to speak up more.

Watch what happens when we do, they replied.

Almost every time they started to speak, they were interrupted or shot down before finishing their pitch. When one had a good idea, a male writer would jump in and run with it before she could complete her thought.

He found a clever way to change the dynamics that were holding those two female employees back. He announced to the writers that he was instituting a no-interruption rule while anyone — male or female — was pitching. It worked, and he later observed that it made the entire team more effective.

A Seat at the Table

Women working for the Obama White House complained they weren’t being let into important meetings.  When they were, they were ignored.

Female staffers adopted a meeting strategy they called “amplification”: When a woman made a key point, other women would repeat it, giving credit to its author. This forced the men in the room to recognize the contribution — and denied them the chance to claim the idea as their own.

How can you help solve the problem? 

To start, women need to learn a few key phrases. 

"I'm not finished." 

"Stop interrupting me."

"I just said that."

But we can't do it alone, gentlemen.  Pay attention the next time you're in a meeting.  When you hear this happen to your co-worker, be her ally.  Try this:

"Let her finish."
"Don't interrupt."
"She just said that."

So what are your ideas about solving this problem?  I'd love to hear them.  Either comment below or tweet me @IrishSQL

http://www.nytimes.com/2015/01/11/opinion/sunday/speaking-while-female.html?_r=0

https://www.youtube.com/watch?v=9soYj3O4Ud8

http://nytlive.nytimes.com/womenintheworld/2015/03/19/google-chief-blasted-for-repeatedly-interrupting-female-government-official/

http://www.womensagenda.com.au/talking-about/editors-agenda/item/7525-men-interrupting-women

https://www.facebook.com/senatorelizabethwarren/videos/724337794395383/

http://www.theglobeandmail.com/889d1711-bd46-429a-b11b-3686468601ad

SQL Server Migrations Lessons Learned

With the recent announcements surrounding all the wonderful things available in SQL Server 2016 SP1, I couldn't help but think about migrating my servers NOW.  I must be crazy, right?  I managed a migration from SQL Server 2005 to SQL Server 2014 in March of this year.  It's only been 8 months.  Don't I ever learn?  Actually it turns out, I do.  I learned a lot.  I was asked to share that experience with Idera via their April #SQLChat on Twitter.  

What is #SQLChat? I'm glad you asked.  If you already know, skip ahead to Question 1.  Still here? Good.  #SQLChat is an Idera Software sponsored SQL Community conversation that takes place entirely on Twitter.  A series of questions are asked & answered on a predecided topic.  There's a host (in today's example, I played that role) that came up with the questions & solicits answers. There's the @Idera_Software twitter account that poses the questions.  There's a dozen or more SQL professionals that help answer these questions,  Finally, there are other SQLFamily twitter users that ask follow up questions.  It's less Q&A and more conversation. 

Here's what I did. What worked for you? 

On that note, I need to know a way to solve X. 

Here's a solution that might work. 

If you use TweetDeck or similar Twitter Interface Apps, trying following the #SQLChat hashtag.  If you can participate this month, great!  If not, read over the questions & answers at your leisure.  I promise, you'll learn something every time.

Q1: What are your deciding factors for migrating/upgrading #SQLServer and to what version? #SQLChat

My thoughts:  Would upgrade benefit the business? Are new features worth the time/effort/money? While some companies wait as late as possible, right up until End of Life for a product, others chose to be proactive. How does this benefit us? Is the payoff worth all the time, effort, planning, money, etc? This answer can vary by company and project.

I really loved the answers my SQL colleagues came up with.  They asked some great follow up questions & had great advice.  Answers varied on when they’d make the decision, what the driving factors would be, how they’d handle it, etc.  I’ve compiled some of those here for you.

Am I currently on a supported version? If not, time to upgrade.  Rob Volk @sql_r

"Do you wait for engine to seize before changing oil & filter in car?" ;) #sqlchat Rob Volk @sql_r

Business decision + vendor app support. Go to latest/greatest app will run on, preferably 2012 or 2014. #SQLChat -- Paul Timmerman @mnDBA

If I need\want to use newer features… If I have available licenses (we dont have an EA) #sqlchat – Monica Rathbun @sqlespresso

@Idera_Software How high of a SQL version will the related software support?  --Dave Mason @BeginTry

@Idera_Software Waiting til the last minute can lead to panic, mistakes AND loss of compliance certs #sqlchat  -- Rie Irish @IrishSQL

New features/functionality benefits, whether the system is covered under SA, and highest ver w/ app support. – Jamie Wich @Jamie_Wick

Q2: How do you decide your timeline from initial planning to push to production? #SQLChat

My thoughts:  We actually started with the date we wanted to Go Live & then built our calendar backwards from there.  We had to leave in a little time to spin tires & get stuck in the mud, but that Go Live date was the goal line.  It wasn’t moving.  To accomplish this, we had to do a few things.  We had to get buy in from sales & executive teams.  I’ve found if those guys, the ones with their eye on money want something, it usually happens.  Try to have someone on the development champion the change from inside. This will help drive them to reach their dates.  Get the migration put on their project roadmap early.  This will help guide them when they’re choosing which project to work on.  The one that’s already on the roadmap, of course!  Some people felt they didn’t have a voice in when the change happened but they seemed to keep their sense of humor about it.

Upper Management dictates most of the time .. and I just have to hit the mark #sqlchat – Monica Rathbun @sqlespresso

What date did the PM pull out of thin air and tell everyone else before consulting DBAs?  -- Bob Pusateri @SQLBob

I've never been a decision maker for upgrade timeline. I'm lucky to find out before it happens. – Rob Volk @sql_r

In my pharma jobs, I got RDBMS classified as infra, which gave us more flexibility for upgrading – Joey D’Antoni @jdanton

Regardless, the timeline always ends up shorter than you'd like #SQLChat James Medlin @jmedlinz

Follow up question, do you allow development/QA time in your initial deadline? How much time is allotted? John Morehouse @SQLrus

Q3: Who do you get involved in the project? What teams have a say? Who leads? #SQLChat

My thoughts:  We started with management.  Not just executives, but all levels of management. We met with Director of Development to discuss what we wanted to do.  He was immediately on board and so was the people on his team doing the work.  We're a Software as a Service shop, so we aren't a vendor shipping out software.  These changes were going to be in-house. 

We identify 3 of our clients each release for a 3 month BETA test. Onsite engagements as well at that time.  --Jim Donahoe @SQLFlipFlopsDBA

Again, I'm not usually a decision maker. I'll add DEV, operations, support if they're not included by mgmt. – Rob Volk @sql_r

Collaborative effort. As a DB architect, we usually lead the charge but definitely get other teams involved. – John Morehouse @SQLRUs

A3: DBA Team is brought into project as a resource. We drive discussions around version of SQL Server. #sqlchat  --Paul Timmerman @mnDBA

A3 It Depends! Totally on the scope of the project, who gets most benefit, etc. #sqlchat -- Mike Essen @AtlSQL

Q4: What's the one thing you'd change or do differently with your next migration? #SQLChat

My thoughts:  So this is the easiest one for me to answer.  Frankly, this blog post may not have enough space to hold what I’d change or do differently.  First, have a project manager.  A good PM will think of all those details your techie brain won’t.  She’ll make sure people are staying on task & help identify what isn’t getting done.  Second, create a flight plan for the actual migration.  I learned this lesson many years ago and it’s stuck with me.  We create an excel spreadsheet with a row for every “to do” item, from building servers to restoring DBs for mirroring.  We provide an estimated start & end time, the person responsible, how we test it, etc. It might seem too detailed but trust me, this attention to detail pays off.  And this should be a living document.  Don’t be afraid to rewrite steps, times, etc. as you move through Staging, UAT and into Production.  Practice.  Repeatedly.  Until you’re sick of it.  Make sure you all know the steps to the dance.  A migration should be like a well-choreographed Broadway show.

 

Once again, my SQL colleagues didn’t let me down.  They came up with some great advice based on their experience.  Again, they maintained their sense of humor.

A4: Practice. Automate. Practice again. Automate more. Refine until I don't need to be there. #SQLChat –Rob Volk @sql_r

A4: Does "go back in time 5 years and do something differently in the first place" count as an answer? #sqlchat  -- Vicky Harp @VickyHarp

A4:To let somebody else manage it! HA Just to practice and practice until it goes smoothly with little interaction  -- Jim Donahoe @SQLFlipFlopsDBA

This! Having template processes helps a ton. Saves us a bunch of time. – Paul Timmerman @mnDBA

Not having to deal w/ multiple domains would be nice. :)  -- Dave Mason @BeginTry

Q5: Did you do a rolling migration or did you plan downtime? #SQLChat

My thoughts:  We chose to do a rolling migration on the databases but turned off client access to the applications.  Argenis Fernandez (T|B) gives a session on Rolling Upgrades that really cemented this as a choice for us.  The downside in using this method to upgrade versions is rollback.  Once you bring up the new version (with mirroring); you can’t fail back.  If you’ve turned off apps like we did, you wouldn’t lose transactions by bringing up the old DB though. We spent the week, days & hours leading up to the migration restoring backups.  First, restoring the full backup from the prior weekend, next the differential from the night before and finally a day’s worth of t-log backups.

Based on feedback on this question, most people agreed with this method.

A5: I've done both rolling upg. and planned downtime. Downtime is always an option, whether you plan for it or not #SQLChat – Rob Volk @sql_r

A5: I also always try to do rolling upgrade to new hardware/environment, easier and faster to roll back if needed. #SQLChat – Rob Volk @sql_r

I prefer a rolling migration with as little downtime as possible. Time = money usually. #sqlchat – John Morehouse @SQLRUs

A5: AGs are rolling migration. Stand-alone have planned downtime. Unsupported dbs are migrated off at upgrade. #sqlchat – Jamie Wick @JamieWick

A5: Rolling migration with downtime only for last log application. Standard process. #SQLChat – Paul Timmerman @mnDBA

Q6: What role did High Availability play in your decision to migrate? #SQLChat

My thoughts:  Most people agreed that moving to High Availability played a large role in their decision to migrate.  I work for a company that has high transactions with 24/7 uptime.  We needed to modernize the infrastructure and increase availability.  Moving to Availability Groups provided a sense of security & stability that execs required.  Just 2 weeks after migration from 2005 to 2014 (virtualized), we experienced corruption in our main production database.  These things never happen at a reasonable hour, so in the wee hours on a Sunday morning, I was able to use the readable secondary on the AG to repair the damaged data.  It took me a couple of hours, running select statements that attempted to read from the damaged pages, but with each error SQL Server replaced the damaged/missing data with corrected data.  What a payoff!  To be able to go to the executive team at work and say “We had an issue over the weekend that could have resulted in lost client data, but because of the investment YOU made in our infrastructure, we didn’t.”

A6: Huge! Recently were moving from SQL Server 2005. More confidence in newer renditions of HA features. #SQLChat – Paul Timmerman @mnDBA

A6: The new HA features of SQL definitely will play a role. #sqlchat  -- John Morehouse @SQLRUs

A6: HA is major factor for most prod dbs. #sqlchat – Jamie Wick @JamieWick

A6: None, Mirroring & Log Shipping have been around forever. Haven't ventured into AG yet. #sqlchat – Mike Essen @AtlSQL

A6 my biggest problem with AG is WFC requirement. Multiple servers, physical clusters, diverse geography. #SQLChat – Scott @ppcx