Data Culture: Roles and Responsibilities

In last week’s post and video we looked at how business intelligence tools have evolved, with each evolution solving one set of problems and introducing another set. The video described how self-service BI tools have enabled business users to work with data in ways that used to require specialized technical skills, freeing up IT to focus on other tasks – but at the same time introducing challenges around oversight, consistency, and control.

And that brings us to today’s video.

The video includes a few different graphics, but for this post I want to highlight this one, which I’ve taken from the Power BI Adoption Framework[1].

2020-07-31-20-44-46-355--POWERPNT

Successful cultures balance freedom and restriction in ways that benefit the culture as a whole. It’s a compromise – no one gets everything they want, but everyone gets the things they need the most.

For a data culture, this typically involves letting IT do the things they do best, and letting business get down to business.

When an organization adopts a managed self-service model for business intelligence[2], the central BI team in IT[3] does the heavy lifting. This means they prepare, cleanse, warehouse, and deliver the biggest and most important[4] data. They deliver the data that would be hard for a business user to create using a self-service tool, and which will be used by a large audience. They do the things that have broad reach, strategic impact, and strategic importance. They create things that need to be correct, consistent, and supported.

And business users do the rest. This is a broad bucket, but it’s a reasonable generalization as well. Business users create their own reports using the IT-managed data sources and data models. And they prepare and use their own data where there is no IT-managed data for a given purpose.

Over time a given application or a given data source may become more or less managed, as the culture adopts, adapts, and improves.

Managed self-service BI isn’t the only way to be successful, but in my experience working with many large Power BI customers it’s the best way and most predictable way. By having clearly defined roles and responsibilities – who does what – moving from either extreme can overcome the problems that that extreme creates, without going too far in the other direction.

Does your organization take this approach?

If it doesn’t today, what will it take to make this change a reality?


[1] Which I have shamelessly stolen from the Power BI Adoption Framework, because it is an awesome graphic and because I love to stand on the shoulders of giants.

[2] Which is the approach most likely to drive short-and long term efficiencies and successes.

[3] Please understand that this is a gross simplification. This “central BI team in IT” may be a single central team for the whole company, it may be a single central team for a specific business unit, or it may be one of dozens of BI teams that are designed to support a distributed global enterprise. This is less about the technology and the culture than it is about organizational structure, so I don’t expect to ever try to tackle this diversity of approaches in this series.

[4] Next week’s video will talk a little more about what data is likely to qualify for this distinction. And since the video is done already, I’m pretty confident to say “next week” this week.

Data Culture: A brief history of business intelligence

Why is a data culture important?

Why does building a data culture in the 2020s involve the challenges and goals that it does?

Why am I publishing in July a video recorded in January?

battleship-389274_1280
BI Application Internals, Circa 1986

The first two questions are probably the easier ones to answer, so I’ll save the third one for last.

For the first question, I’ll refer you to the first post and video in this series, which introduces the key concepts for a data culture.

For the second question, you need to understand how we got to where we are today – how the data and BI technologies we use today evolved over the past decades. And that’s where today’s video[1] comes in. Check out a brief history of business intelligence for context on the problems that a successful data culture needs to solve, and what your starting point might be.

I won’t go into depth here (I spent six months on that video, so darned right I’m not going to spoil it!) but there are basically four phases in this brief history:

  1. The dark ages, also known as “back when I was your age” when there was no BI, and decisions were made based on human knowledge, instinct, and experience
  2. The good old days, when BI solutions were developed by highly skilled BI professionals using complex and difficult tools, and when ETL consultants could charge exorbitant rates[2] without clients blinking an eye
  3. The SSBI era where every business user got up to their elbows in data, and IT wasn’t slowing anyone down any more
  4. Nirvana[3], where IT and business work together as partners, each contributing based on their skills and priorities to common shared goals

And each phase brought (or brings) with it its own challenges:

  1. Dependence on specific individuals with specialized knowledge, slow decisions, and probably giant hair you spent hours sculpting with Aqua Net
  2. Expensive development efforts, long turnaround times for insights, IT becomes a bottleneck
  3. Inconsistent results, untrusted (and untrustworthy) insights, producing misinformed business decisions
  4. Difficulty in aligning business and IT and making the necessary cultural changes… also flannel?

And for that third question, please let me just say that 2020 is kicking my butt in ways I never expected. Between work and family and the global pandemic that’s ravaging the globe, I’ve needed to harshly triage all non-essential work. Back in January I was sure I would publish this video a week late, but still in January. Back on February 24th I made this optimistic comment:

In case you’ve been wondering why my blog and YouTube output has dried up this month, it’s because real life has been kicking my ass. I think I can finally see the light at the end of the tunnel, so hopefully we’ll be back with regular content before too long. Hopefully.

That was about a week before the coronavirus outbreak at the nursing home down the road from my house hit the news, and it’s all been downhill from there.

I’m tempted to say I again see the light at the end of the tunnel, but I’m not going to do that. The last time the light turned out to be a train. I will instead say that I’ll do my best to get more blog and video content created and published as consistently as possible and that I appreciate your patience and your support[4].


[1] Today’s video was recorded on January 19th as “take two” that delayed its publishing. Basically I was filming the video at the end of the day when I had planned on publishing it, so I figured it would end up being a week late. Yeah.

[2] This may not be how everyone remembers those times, but this is my blog and my story and I’m sticking with it.

[3] Or Pearl Jam. Maybe Soundgarden. I never really did keep up with grunge, so I’m honestly not sure.

[4] I don’t want to get anyone’s hopes up, but between Friday evening when I wrote this post and finished editing the video, I have recorded the next two videos in the data culture series, and edited the first of them. Only time will tell if August is kinder to me than recent months have been, but barring unforeseen circumstances you’ll see the next two videos on the next two Monday.

5’ve got all the swords

Power BI turns five years old this week. On July 24, 2015 Microsoft announced the general availability of Power BI, a new cloud service for self-service business intelligence.

In the five years since then Power BI has evolved to be much more than a self-service tool. Power BI today includes capabilities for self-service and enterprise BI, adding data preparation, information management, machine learning and more, while still including powerful tools for data visualization and reporting.

Today[1] the Power BI team at Microsoft is commemorating this milestone by saying thank you.

PowerBI_5YearAnni_1584x396-04-1024x256

Even though I’m on the Power BI team at Microsoft, this is my personal blog, and I wanted to say thank you as well. So I put together a short video that includes five of my favorite things:

  1. Power BI
  2. Swords
  3. Bad jokes
  4. My awesome new “5’ve got the Power BI” t-shirt
  5. Worse jokes[2]

The one thing I couldn’t include is all of you.

Thanks for the comments and feedback on the blog and the videos, and thanks even more for helping make Power BI what it is today.

 


[1] Yes, I know today is July 23rd here in North America, but by commemorating this milestone today we can be more inclusive for folks in parts of the world where it’s already the weekend.

[2] There is one joke in the video that I suspect I am the only person who will understand enough to laugh at it. If you get it please tell me so I don’t feel so alone.

Foster Kittens and Managed Self-Service BI

My family is a foster family for cats and kittens from the cat shelter where we’ve adopted some of our own cats. Usually a litter will stay with us for a month or two, but it depends on the kittens themselves, and on external factors.

While the kittens are with us, it’s our responsibility to help them grow, both physically and socially. The experts at the shelter are always available if we need help, but for the most part we have the knowledge and tools we need to be successful. In many cases we’re their first real exposure to humans, and we can prepare them to be loving and playful members of a family. Just not our family.

Once the kittens are ready to be adopted, we take them to the shelter, where they will be carefully matched with their forever family. This last part is important – it’s hard enough to let go, and knowing that each of them will find a good home is what makes it possible.

It’s really the best of both worlds – kind of like managed self-service BI with Power BI.[1]

 

Not unlike fostering kittens, managed self-service BI can be the best of both worlds. As an analyst working in Power BI, you can often pick up projects when the scope is still small and manageable, and when you can have fun playing around with the data and seeing what it’s likely to become.

I’m emphasizing the “managed” in managed self-service BI, because it’s best to not be completely on your own. Having someone backing you up, someone with the expertise and resources to get you through challenging spots with a helping hand, is just as important with BI as it is with kittens. An author on his own may make avoidable mistakes with long-term consequences, but a center of excellence or community of practice can provide training up front, and assistance along the way so the finished self-service solution is ready to grow up – and growing up is an important goal.

Just as my family includes our adult cats, that analyst working in Power BI has a day job. If we kept each litter of kittens we foster, things would soon become messy and unmanageable. If an analyst retained ownership of every Power BI solution he developed, he would struggle to stay on top of his core priorities.

Being able to hand off a self-service solution to a central BI team is what gives this story a happy ending. The BI team can give the analyst’s work the long-term home it deserves, and the analyst can get back to his job… while also keeping an eye open for the next self-service BI challenge to come along and steal his heart.

Of all the kittens I have loved, I miss Tiny the most.
Your head. I will bite it now.[2]
If you’re interested in learning more about the shelter where we volunteer, please visit the Meow Cat Rescue web site. Please also consider donating while you’re there – the global pandemic is making it harder for their awesome staff and volunteers to do what they do, and kitten season is upon us. If you appreciate the BI Polar blog and its companion YouTube channel, there’s no better way to say thank you than to donate to Meow. Even a small donation will help.


[1] I hope you saw that one coming

[2] This footage of Tiny attacking my head didn’t fit into the Power Kittens video, but I shared it on Twitter because it was just too cute to not share.

Being a Program Manager at Microsoft

My awesome friend Christopher also works at Microsoft, but his career and mine have taken different paths since we joined. Here we are together back in 2008, before either one of us worked at Microsoft. He’s the one not wearing a Manowar t-shirt.

gateslunch1
No, no, the other one.

Our backgrounds are remarkably similar. We each spent years before joining Microsoft as Microsoft Certified Trainers, with an emphasis on software development rather than systems administration. We each became involved in the MCT community, and in the broader technical community, and to one extent or another this helped us find our first positions at Microsoft.

These days Christopher is working with student developers – I’ve seen him tweeting a lot recently about Django on Twitch, which probably means that he’s still teaching people about software development… or may be watching Quentin Tarantino movies while drinking too much coffee. Honestly, either one is possible.

Anyway… Christopher reached out to me last month and asked if I’d be interested in talking to college students about to graduate about what it’s like being a program manager at Microsoft. I said yes[1], then I said this:

I’ll let the video speak for itself. I tell my story for the first 15 minutes or so, and around the 14:50 mark I talk with Will Thompson and Tessa Hurr from the Power BI team and ask them to share their experiences as well. But there are a few things that I want to add that didn’t really fit well into the video.

First of all, every team has a need for many types of program managers. I’ve blogged about diversity enough that you probably know how I feel about the value of diverse teams, and this applies to PM teams at Microsoft as well. Since this video is targeted at college students and recent college graduates, I’d like to focus briefly on the career diversity dimension.

What does “career diversity dimension” mean? Looking at most PM teams I see program managers falling into three broad groups:

  1. New to career – program managers who are starting their careers after college, or after switching from a non-IT discipline.[2]
  2. Industry hires – program managers who are new to Microsoft, but who have an established career in a related field.[3] This is often someone who has been a consultant, developer or administrator who works hands-on with Microsoft or competitive tools and technologies.
  3. Veterans – program managers who have been at Microsoft long enough to succeed a few times, fail a few times, and understand what PM success can look like on multiple teams.[4]

Reading this list it may be easy to think that there is a progression of value implied, but this is not the case. A successful team will find ways to get from each group the things that only they can contribute. A PM in one group will be able to see things and do things that a PM in other groups will not, and an experienced team leader will be able to direct each PM to the problems that they are best suited to solve, and can add the most value.

The second thing I wanted to add to the video is that each team is different. I said this in the video, but I want to elaborate here. Each team will have its own culture, and some teams will be a better fit than others for a given PM. I’ve worked on multiple teams where I didn’t think I was contributing effectively, or where I felt that my contributions weren’t valued[5]. At one point this culture mismatch almost led me to leave Microsoft, but with the support of my manager at the time I instead found another team in another org where I could thrive.

And this leads me to the the final point I wanted to add: Microsoft is huge. I’ve been a PM at Microsoft since October 2008, but I’ve had 4 or 5 major career changes since then, with very different responsibilities after each change, requiring very different contributions from me.

When I was interviewing for my first position in 2008, the hiring manager asked me why I wanted to work for Microsoft. I already had a successful career as a data warehousing and ETL consultant, and becoming a Microsoft employee would include a reduction in income, at least in the short term. Why give up what I’d built?

I hadn’t expected this question, and my answer was authentic and unscripted, and I’ve thought a lot about it over the past 11+ years. I told him that I wanted to join Microsoft because Microsoft had bigger and more challenging problems to solve than I would ever see as a consultant, and would never run out of new problems for me to help solve. If I joined Microsoft, I would never be bored.

I wanted to join Microsoft because Microsoft had bigger and more challenging problems to solve than I would ever see as a consultant, and would never run out of new problems for me to help solve. If I joined Microsoft, I would never be bored.

And I was right.

If you’re a PM at Microsoft, please share your thoughts and experiences.

If you’re thinking about becoming a PM at Microsoft, please share your questions.

If you’ve joined Microsoft as a PM after watching this video and reading this blog, please send me an email, because I would love to say hi.


[1] It’s a good thing he asked when he did. I haven’t had a haircut since late February, and I don’t know when I’ll let anyone point a camera at my head again…

[2] This was kind of me when I first applied for a position at Microsoft in the 90s, although I already had a few years’ experience. I was not hired.

[3] This was me in 2008 when I was hired.

[4] This is me in 2020. I tend to talk about the successes more, but it’s the failures I think about the most, and where I learned the most along the way. Success is awesome, but it’s a lousy teacher.

[5] Yes, this sucked as much as you could imagine.

Data culture: Executive sponsorship

Continuing on our series on data culture, we’re examining the importance of having an executive sponsor. This is one of the least exciting success factors for implementing Power BI and getting more insights from more data to deliver more value to the business… but it’s also one of the most important factors.

Let’s check it out:

Ok, what did we just watch?

This video (and the series it’s part of) includes patterns for success I’ve observed as part of my role on the Power BI CAT team[1]. and will complement the guidance being shared in the Power BI Adoption Framework.

The presence of an executive sponsor is one of the most significant factors for a successful data culture. An executive sponsor is:

  • Someone in a position of authority who shares the goals having important business decisions driven by accurate and timely data
  • A leader who can help remove barriers and make connections necessary to build enterprise data solutions
  • A source of budgetary[2] and organizational support for data initiatives
skyscraper-3184798_1920
Because executives fly on planes, right?

Without an executive sponsor, the organizational scope of the data culture is often limited by the visibility that departmental BI successes can achieve. The data culture will grow gradually and may eventually attract executive attention… or may not.

Without an executive sponsor, the lifetime of a data culture is often limited by the individuals involved. When key users move to new roles or take on new challenges and priorities, the solutions they’ve developed can struggle to find new owners.

Without an executive sponsor, all of the efforts you take to build and sustain a data culture in your organization will be harder, and will be more likely to fail.

Who is your executive sponsor?

Update: A Twitter conversation about this video sparked a follow-up post. You can check it out here: Tough Love in the data culture.


[1] This is your periodic reminder that this is my personal blog, and all posts and opinions are mine and mine alone, and do not reflect the opinions of my employer or my teenage children.

[2] This aspect of sponsorship is a bigger deal than we’re going to cover in this post and video – organizations fund what’s important to them, and they don’t fund what’s not.

Building a data culture

tl;dr – to kick off 2020 we’re starting a new BI Polar video series focusing on building a data culture, and the first video introduces the series. You should watch it and share it.

Succeeding with a tool like Power BI is easy – self-service BI tools let more users do more things with data more easily, and can help reduce the reporting burden on IT teams.

Succeeding at scale with a tool like Power BI is not easy. It’s very difficult, not because of the technology, but because of the context in which the technology is used. Organizations adopt self-service BI tools because their existing approaches to working with data are no longer successful – and because the cost and pain[1] of change has become outweighed by the cost and pain of maintaining course.

Tool adoption may be top-down, encouraged or mandated by senior management as a broad organization-wide effort. Adoption may be bottom-up, growing organically and virally in the teams and departments least well served by the existing tools and processes in place.

Both of these approaches[2] can be successful, and both of these approaches can fail. The most important success factor is a data culture in which the proper use of self-service BI tools can deliver the greatest value for the organization.

The most important success factor is a data culture

books-1655783_640
There must be a data culture on the other side of this door.

Without an organizational culture that values, encourages, recognizes, and rewards users and teams for their use of data, no tool and no amount of effort and skill is enough to achieve the full potential of the tools – or of the data.

In this new video series we’ll be covering practices that will help build a data culture. More specifically, we’ll introduce common practices that are exhibited by large organizations that have mature and successful data cultures. Each culture is unique, but there are enough commonalities to identify patterns and anti-patterns.

The content in this series will be informed by my work with enterprise Power BI customers as part of my day job[3], and will complement nicely[4] the content and guidance in the Power BI Adoption Framework.

Back in November when the 100th BI Polar blog post was published, I asked what everyone wanted to read about in the next 100 posts. There were lots of different ideas and suggestions, but the most common theme was around guidance like this. Hopefully you’ll enjoy the result – and hopefully you’ll let me know either way.


[1] I strongly believe that pain is a fundamental precursor to significant change. If there is no pain, there is no motivation to change. Only when the pain of not changing exceeds the perceived pain of going through the change will most people and organizations consider giving up the status quo. There are occasional exceptions, but in my experience these are very rare.

[2] Including any number of variations – these approaches are common points on a wide spectrum, but should not be interpreted as the only ways to adopt Power BI or other self-service BI tools.

[3] By day I’m a masked crime-fighter. Or a member of the Power BI customer advisory team. Or both. It varies from day to day.

[4] Hopefully this will be true. I’m at least as interested in seeing where this ends up as you are.

Power BIte: Power Platform dataflows

INTERIOR: pan over cubicles of happy, productive office workers

CLOSE-UP: office worker at desk

NARRATOR: Is that Susie I see, giving Power Platform dataflows a try?

SUSIE: That’s right! With dataflows I can have all of the data I need, right where I need it!!

NARRATOR: Dataflows. They’re not just for Power BI anymore.

OK, you may not remember that orange juice ad campaign from the late 1970s and early 80s[1], but I’ve had it stuck in my head since I started working on this post and video. I couldn’t figure out how to work it into the video itself, so here it is in written form.

Anyway, with that awkward moment behind us, you probably want to watch the video. Here is it:

As the video discusses, Power Apps now have a dataflows capability that is a natural complement to Power BI dataflows. Power Platform dataflows have been generally available since November 2019, and have been in preview since summer.

Power Platform dataflows use Power Query Online – and the same set of connectors, gateways, and transformation capabilities as Power BI dataflows. But there are a few key differences that are worth emphasizing.

Power Platform dataflows can load data into the Common Data Service, either into the standard Common Data Model entities used by Dynamics 365 apps, or into custom entities used by custom Power Apps. This is important – this makes dataflows more like a traditional ETL tool like SSIS data flows in that at the end of the dataflow creation process you can map the columns in your queries to the columns in these existing tables[2].

Power Platform dataflows can load data into ADLSg2 for analytical scenarios, but Power Apps doesn’t have the same concept of “built-in storage” that Power BI does. That means if you want to use Power Platform dataflows to create CDM folders, you must configure your Power Apps environment to use an ADLSg2 resource in your Azure subscription.

The “link to data lake” feature in Power Apps feels to me like a better integration experience than what’s currently available in Power BI. In Power Apps you define the link at the environment level, not the tenant level – this provides more flexibility, and enables non-tenant admins[3] to configure and use the data lake integration.

2019-12-23-11-07-30-619--ApplicationFrameHost

The first time you create a Power Platform dataflow and select the “analytical entities” option, you’ll be prompted – and required – to link the Power Apps environment to an Azure Data Lake Storage resource. You’ll need to have an Azure subscription to use, but the process itself if pretty straightforward.

2019-12-23-11-19-10-160--ApplicationFrameHost.png

I can’t wait to hear what you think of this new capability. Please let me know in the comments or via Twitter.

See you in the new year!


[1] I just realized that this was 40 years ago. Were you even born yet?

[2] CDS entities aren’t tables by the strictest definition, but it’s close enough for our purposes today.

[3] I honestly don’t know enough about Power Apps security to go into too much depth on this point, but I am not a Power Apps admin and I was able to create a trial environment and link it to my own ADLSg2 resource in my own Azure subscription without affecting other users.

Video: A most delicious analogy

Every time I cook or bake something, I think about how the tasks and patterns present in making food have strong and significant parallels with building BI[1] solutions. At some point in the future I’m likely to write a “data mis en place” blog post, but for today I decided to take a more visual approach, starting with one of my favorite holiday recipes[2].

Check it out:

(Please forgive my clickbaitey title and thumbnail image. I was struggling to think of a meaningful title and image, and decided to have a little fun with this one.)

I won’t repeat all of the information from the video here, but I will share a view of what’s involved in making this self-service BI treat.

2019-12-17-12-52-36-894--VISIO

When visualized like this, the parallels between data development and reuse are probably a bit more obvious. Please take a look at the video, and see what others jump out at you.

And please let me know what you think. Seriously.


[1] And other types of software, but mainly BI these days.

[2] I published this recipe almost exactly a year ago. The timing isn’t intentional, but it’s interesting to me to see this pattern emerging as well…

Power BIte: Dataflows enhanced compute engine

The enhanced compute engine in Power BI dataflows has been in preview since June. It’s not really new, and I’ve posted about it before. But I still keep hearing questions about it, so I thought it might make sense to record a video[1].

This video.


I won’t go into too much more depth here – just watch the video, and if you want more details check out one of these existing posts:

Now to get back on schedule with that next video…


[1] Also, I’m behind on my video schedule – this was a motivating factor as well. November was an unexpectedly busy month[2], and between work, life, and not really having the video editing skills I need to keep to a schedule… Yeah.

[2] And I expected it to be very, very busy.