Tough love in the data culture

When I shared on Twitter the most recent installment in the BI Polar “data culture” series, BI developer Chuy Varela replied to agree with some key points from the video on executive sponsorship… and to share some of his frustrations as well[1].

2020-01-13-08-46-03-439--msedge

Chuy’s comments made me think of advice that I received a few years back from my manager at the time. She told me:

Letting others fail is a Principal-level behavior.

Before I tie this tough love into the context of a data culture and executive sponsorship, I’d like to share the context in which the advice was given.

At the time I had just finished a few 80-hour work weeks, including working 12+ hour days through a long holiday weekend. Much of that time was spent performing tasks that weren’t my responsibility – another member of the extended team was dropping the ball, and with a big customer-facing milestone coming up I wanted to make everything as close to perfect as possible.

Once the milestone had been met, my manager pulled me aside and let me know how unhappy she was that I had been making myself ill to deliver on someone else’s commitments[4]. She went on to elaborate that because of my well-intentioned extra effort, there were two negative consequences:

  1. The extended team member would not learn from the experience, so future teams were more likely to have the same challenges.
  2. The executive sponsors for the current effort would believe that the current level of funding and support was enough to produce the high quality and timely deliverables that actually required extra and unsustainable work.

She was right, of course, I’ve added two phrases to my professional vocabulary because of her advice.

The first phrase[5] gets used when someone is asking me to do work:

That sounds very important, and I hope you find the right folks to get it done. It sounds too important for me to take on, because I don’t have the bandwidth to do it right, and it needs to be done right.

The second phrase gets used when I need to stop working on something:

After this transition I will no longer be performing this work. If the work is still important to you, we’ll need to identify and train someone to replace me. If the work is no longer important, no action is required and the work will no longer be done.

Now let’s think about data culture and executive sponsorship in a bottom-up organization. How does a sponsor – or potential sponsor – react when a BI developer or analyst is delivering amazing insights when it’s not really their job?

If the sponsor is bought in to the value of a data culture, their reaction is like to include making it their job, and ensuring that the analyst gets the support and resources to make this happen. This can take many different forms[6], but should always include an explicit recognition of the work and the value it delivers.

And what if the analyst who has enabled more data-driven decisions is ready to move on to new challenges? What happens to the solutions they’ve built and the processes that rely on their work? What then?

Again, if the sponsor is committed to a data culture, then their reaction will involve making sure that it becomes someone’s responsibility to move the analyst’s work forward. They will assign the necessary resources – data resources, human resources, financial resources – to ensure that the data-driven insights continue.

If you’re working on helping to build a data culture from the bottom up, there are a few opportunities to apply this advice to that end. Please understand that there is risk involved in some of these approaches, so be certain to take the full context into consideration before taking action.

  1. Work with your immediate manager(s) to amplify awareness of your efforts. If your managers believe in the work that you’re doing, they should be willing to help increase visibility.
  2. Ensure that everyone understands that your work needs support to be sustained. You’ve done something because it was the right thing to do, but for you to keep doing it, but you can’t keep doing it with all of your other responsibilities.
  3. When it comes time to change roles, ask who will be taking over the solutions you’ve built in your current role, and how you can transition responsibility to them.

Each of these potential actions is the first step down a more complicated path, and the organization’s response[7] will tell you a lot about the current state of the data culture.

For the first potential course of action, managerial support in communicating the value and impact of current efforts can demonstrate the art of the possible[8] to a potential executive sponsor who may not be fully engaged. If you can do this for your current team or department, imagine what a few more empowered folks could do for the business unit! You see where I’m going…

If this first course of action is the carrot, the other two might be the stick. Not everyone will respond the way we might like to our efforts. If you’re building these solutions today without any additional funding or support, why would I give you more resources? Right? In these cases, the withdrawal or removal of existing capabilities may be what’s necessary to communicate the value of work that’s already been completed.

Wow, this post ended up being much longer than planned. I’ll stop now. Please let me know what you think!


[1] I don’t actually believe in New Years resolutions, but I am now retroactively adding “start a blog post with a sentence that contains six[2] or more hyperlinks to my goals for 2020[3].

[2] Yes, I had to go back to count them.

[3] Nested footnotes achievement unlocked!

[4] If your reaction at this point is “what sort of manager would yell at you for doing extra work?” the answer is “an awesome manager.” This may have been the best advice I ever received.

[5] I’ve probably never said either phrase exactly like this, but the sentiment is there.

[6] Including promotion, transfer to a new role, and/or adding capacity to the analyst’s team so the analyst can focus more on the new BI work.

[7] Remember that a culture is what people do, not what people say – have I mentioned that before?

[8] The art of the possible is likely to warrant its own post and video before too long, so I won’t go into too much depth here.

Data culture: Executive sponsorship

Continuing on our series on data culture, we’re examining the importance of having an executive sponsor. This is one of the least exciting success factors for implementing Power BI and getting more insights from more data to deliver more value to the business… but it’s also one of the most important factors.

Let’s check it out:

Ok, what did we just watch?

This video (and the series it’s part of) includes patterns for success I’ve observed as part of my role on the Power BI CAT team[1]. and will complement the guidance being shared in the Power BI Adoption Framework.

The presence of an executive sponsor is one of the most significant factors for a successful data culture. An executive sponsor is:

  • Someone in a position of authority who shares the goals having important business decisions driven by accurate and timely data
  • A leader who can help remove barriers and make connections necessary to build enterprise data solutions
  • A source of budgetary[2] and organizational support for data initiatives
skyscraper-3184798_1920
Because executives fly on planes, right?

Without an executive sponsor, the organizational scope of the data culture is often limited by the visibility that departmental BI successes can achieve. The data culture will grow gradually and may eventually attract executive attention… or may not.

Without an executive sponsor, the lifetime of a data culture is often limited by the individuals involved. When key users move to new roles or take on new challenges and priorities, the solutions they’ve developed can struggle to find new owners.

Without an executive sponsor, all of the efforts you take to build and sustain a data culture in your organization will be harder, and will be more likely to fail.

Who is your executive sponsor?

Update: A Twitter conversation about this video sparked a follow-up post. You can check it out here: Tough Love in the data culture.


[1] This is your periodic reminder that this is my personal blog, and all posts and opinions are mine and mine alone, and do not reflect the opinions of my employer or my teenage children.

[2] This aspect of sponsorship is a bigger deal than we’re going to cover in this post and video – organizations fund what’s important to them, and they don’t fund what’s not.

Building a data culture

tl;dr – to kick off 2020 we’re starting a new BI Polar video series focusing on building a data culture, and the first video introduces the series. You should watch it and share it.

Succeeding with a tool like Power BI is easy – self-service BI tools let more users do more things with data more easily, and can help reduce the reporting burden on IT teams.

Succeeding at scale with a tool like Power BI is not easy. It’s very difficult, not because of the technology, but because of the context in which the technology is used. Organizations adopt self-service BI tools because their existing approaches to working with data are no longer successful – and because the cost and pain[1] of change has become outweighed by the cost and pain of maintaining course.

Tool adoption may be top-down, encouraged or mandated by senior management as a broad organization-wide effort. Adoption may be bottom-up, growing organically and virally in the teams and departments least well served by the existing tools and processes in place.

Both of these approaches[2] can be successful, and both of these approaches can fail. The most important success factor is a data culture in which the proper use of self-service BI tools can deliver the greatest value for the organization.

The most important success factor is a data culture

books-1655783_640
There must be a data culture on the other side of this door.

Without an organizational culture that values, encourages, recognizes, and rewards users and teams for their use of data, no tool and no amount of effort and skill is enough to achieve the full potential of the tools – or of the data.

In this new video series we’ll be covering practices that will help build a data culture. More specifically, we’ll introduce common practices that are exhibited by large organizations that have mature and successful data cultures. Each culture is unique, but there are enough commonalities to identify patterns and anti-patterns.

The content in this series will be informed by my work with enterprise Power BI customers as part of my day job[3], and will complement nicely[4] the content and guidance in the Power BI Adoption Framework.

Back in November when the 100th BI Polar blog post was published, I asked what everyone wanted to read about in the next 100 posts. There were lots of different ideas and suggestions, but the most common theme was around guidance like this. Hopefully you’ll enjoy the result – and hopefully you’ll let me know either way.


[1] I strongly believe that pain is a fundamental precursor to significant change. If there is no pain, there is no motivation to change. Only when the pain of not changing exceeds the perceived pain of going through the change will most people and organizations consider giving up the status quo. There are occasional exceptions, but in my experience these are very rare.

[2] Including any number of variations – these approaches are common points on a wide spectrum, but should not be interpreted as the only ways to adopt Power BI or other self-service BI tools.

[3] By day I’m a masked crime-fighter. Or a member of the Power BI customer advisory team. Or both. It varies from day to day.

[4] Hopefully this will be true. I’m at least as interested in seeing where this ends up as you are.

Power BIte: Power Platform dataflows

INTERIOR: pan over cubicles of happy, productive office workers

CLOSE-UP: office worker at desk

NARRATOR: Is that Susie I see, giving Power Platform dataflows a try?

SUSIE: That’s right! With dataflows I can have all of the data I need, right where I need it!!

NARRATOR: Dataflows. They’re not just for Power BI anymore.

OK, you may not remember that orange juice ad campaign from the late 1970s and early 80s[1], but I’ve had it stuck in my head since I started working on this post and video. I couldn’t figure out how to work it into the video itself, so here it is in written form.

Anyway, with that awkward moment behind us, you probably want to watch the video. Here is it:

As the video discusses, Power Apps now have a dataflows capability that is a natural complement to Power BI dataflows. Power Platform dataflows have been generally available since November 2019, and have been in preview since summer.

Power Platform dataflows use Power Query Online – and the same set of connectors, gateways, and transformation capabilities as Power BI dataflows. But there are a few key differences that are worth emphasizing.

Power Platform dataflows can load data into the Common Data Service, either into the standard Common Data Model entities used by Dynamics 365 apps, or into custom entities used by custom Power Apps. This is important – this makes dataflows more like a traditional ETL tool like SSIS data flows in that at the end of the dataflow creation process you can map the columns in your queries to the columns in these existing tables[2].

Power Platform dataflows can load data into ADLSg2 for analytical scenarios, but Power Apps doesn’t have the same concept of “built-in storage” that Power BI does. That means if you want to use Power Platform dataflows to create CDM folders, you must configure your Power Apps environment to use an ADLSg2 resource in your Azure subscription.

The “link to data lake” feature in Power Apps feels to me like a better integration experience than what’s currently available in Power BI. In Power Apps you define the link at the environment level, not the tenant level – this provides more flexibility, and enables non-tenant admins[3] to configure and use the data lake integration.

2019-12-23-11-07-30-619--ApplicationFrameHost

The first time you create a Power Platform dataflow and select the “analytical entities” option, you’ll be prompted – and required – to link the Power Apps environment to an Azure Data Lake Storage resource. You’ll need to have an Azure subscription to use, but the process itself if pretty straightforward.

2019-12-23-11-19-10-160--ApplicationFrameHost.png

I can’t wait to hear what you think of this new capability. Please let me know in the comments or via Twitter.

See you in the new year!


[1] I just realized that this was 40 years ago. Were you even born yet?

[2] CDS entities aren’t tables by the strictest definition, but it’s close enough for our purposes today.

[3] I honestly don’t know enough about Power Apps security to go into too much depth on this point, but I am not a Power Apps admin and I was able to create a trial environment and link it to my own ADLSg2 resource in my own Azure subscription without affecting other users.

New resource: Generating CDM folders from Azure Databricks

Most of my blog posts that discuss the integration of Azure data services and Power BI dataflows via Common Data Model folders[1][2][3] include links to a tutorial and sample originally published in late 2018 by the Azure team. This has long been the best resource to explain in depth how CDM folders fit in with the bigger picture of Azure data.

Now there’s something better.

Microsoft Solutions Architect Ted Malone has used the Azure sample as a starting point for a GitHub project of his own, and has extended this sample project to start making it suitable for more scenarios.

2019-12-20-15-39-41-744--msedge

The thing that has me the most excited (beyond having Ted contributing to a GitHub repo, and having code that works with large datasets) is the plan to integrate with Apache Atlas for lineage and metadata. That’s the good stuff right there.

If you’re following my blog for more than just Power BI and recipes, this is a resources you need in your toolkit. Check it out, and be sure to let Ted know if it solves your problems.


[1] Power BIte: Creating dataflows by attaching external CDM folders

[2] Quick Tip: Working with dataflow-created CDM folders in ADLSg2

[3] Dataflows, CDM folders and the Common Data Model

Video: A most delicious analogy

Every time I cook or bake something, I think about how the tasks and patterns present in making food have strong and significant parallels with building BI[1] solutions. At some point in the future I’m likely to write a “data mis en place” blog post, but for today I decided to take a more visual approach, starting with one of my favorite holiday recipes[2].

Check it out:

(Please forgive my clickbaitey title and thumbnail image. I was struggling to think of a meaningful title and image, and decided to have a little fun with this one.)

I won’t repeat all of the information from the video here, but I will share a view of what’s involved in making this self-service BI treat.

2019-12-17-12-52-36-894--VISIO

When visualized like this, the parallels between data development and reuse are probably a bit more obvious. Please take a look at the video, and see what others jump out at you.

And please let me know what you think. Seriously.


[1] And other types of software, but mainly BI these days.

[2] I published this recipe almost exactly a year ago. The timing isn’t intentional, but it’s interesting to me to see this pattern emerging as well…

The Power BI Adoption Framework – it’s Power BI AF

You may have seen things that make you say “that’s Power BI AF” but none of them have come close to this. It’s literally the Power BI AF[1].

That’s right – this week Microsoft published the Power BI Adoption Framework on GitHub and YouTube. If you’re impatient, here’s the first video – you can jump right in. It serves as an introduction to the framework, its content, and its goals.

Without attempting to summarize the entire framework, this content provides a set of guidance, practices, and resources to help organizations build a data culture, establish a Power BI center of excellence, and manage Power BI at any scale.

Even though I blog a lot about Power BI dataflows, most of my job involves working with enterprise Power BI customers – global organizations with thousands of users across the business who are building, deploying, and consuming BI solutions built using Power BI.

Each of these large customers takes their own approach to adopting Power BI, at least when it comes to the details. But with very few exceptions[2], each successful customer will align with the patterns and practices presented in the Power BI Adoption Framework – and when I work with a customer that is struggling with their global Power BI rollout, their challenges are often rooted in a failure to adopt these practices.

There’s no single “right way” to be successful with Power BI, so don’t expect a silver bullet. Instead, the Power BI Adoption Framework presents a set of roles, responsibilities, and behaviors that have been developed after working with customers in real-world Power BI deployments.

If you look on GitHub today, you’ll find a set of PowerPoint decks broken down into five topics, plus a few templates.

2019-12-12-12-09-19-166--msedge

These slide decks are still a little rough. They were originally built for use by partners who could customize and deliver them as training content for their customers[3], rather than for direct use by the general public, and as of today they’re still a work in progress. But if you can get past the rough edges, there’s definitely gold to be found. This is the same content I used when I put together my “Is self-service business intelligence a two-edged sword?” presentation earlier this year, and for the most part I just tweaked the slide template and added a bunch of sword pictures.

And if the slides aren’t quite ready for you today, you can head over to the official Power BI YouTube channel where this growing playlist contains bite-size training content to supplement the slides. As of today there are two videos published – expect much more to come in the days and weeks ahead.

The real heroes of this story[4] are Manu Kanwarpal and Paul Henwood.  They’re both cloud solution architects working for Microsoft in the UK. They’ve put the Power BI AF together, delivered its content to partners around the world, and are now working to make it available to everyone.

What do you think?

To me, this is one of the biggest announcements of the year, but I really want to hear from you after you’ve checked out the Power BI AF. What questions are still unanswered? What does the AF not do today that you want or need it to do tomorrow?

Please let me know in the comments below – this is just a starting point, and there’s a lot that we can do with it from here…


[1] If you had any idea how long I’ve been waiting to make this joke…

[2] I can’t think of a single exception at the moment, but I’m sure there must be one or two. Maybe.

[3] Partners can still do this, of course.

[4] Other than you, of course. You’re always a hero too – never stop doing what you do.

Power BIte: Dataflows enhanced compute engine

The enhanced compute engine in Power BI dataflows has been in preview since June. It’s not really new, and I’ve posted about it before. But I still keep hearing questions about it, so I thought it might make sense to record a video[1].

This video.


I won’t go into too much more depth here – just watch the video, and if you want more details check out one of these existing posts:

Now to get back on schedule with that next video…


[1] Also, I’m behind on my video schedule – this was a motivating factor as well. November was an unexpectedly busy month[2], and between work, life, and not really having the video editing skills I need to keep to a schedule… Yeah.

[2] And I expected it to be very, very busy.

Power BIte: Turning datasets into dataflows

At this point I’ve said “Power BI dataflows enable reuse” enough times that I feel like a broken record[1]. What does this phrase actually mean, and how can you take advantage of dataflows to enable reuse in your Power BI applications?

This Power BIte video is a bit longer than its predecessors, and part of this is because it covers both the problem and the solution.

The problem is that self-service BI applications often start out as one-off efforts, but don’t stay that way. At least in theory, if the problem solved by the application was widespread and well understood, there would be an existing solution already developed and maintained by IT, and business users wouldn’t need to develop their own solutions.

Successful applications have a tendency to grow. For self-service BI, this could mean that more and more functionality gets added to the application, or it could mean that someone copies the relevant portions of the application and uses them as the starting point for a new, different-but-related, application.

Once this happens, there is a natural and gradual process of drift[2] that occurs, as each branch of the application tree grows in its own direction. A predictable consequence of this drift in Power BI applications is that query definitions that start off as common will gradually become out of sync, meaning that “the same data” in two datasets will actually contain different values.

Moving queries that need to be shared across multiple applications from multiple datasets into a single dataflow is a simple and effective solution to this problem. There’s no dedicated tooling for this solution in Power BI today, but the steps are still simple and straightforward.

P.S. This is the first Power BIte video recorded in my home office. After struggling unsuccessfully to get decent audio quality in my office at work, I’m trying out a new environment and some new tools. I know there’s still work to be done, but hopefully this is a step in the right direction. As always, I’d love to know what you think…


 

[1] For my younger readers, this phrase is a reference to when Spotify used to be called “records” and the most common service outage symptom was a repeat of the audio buffer until the user performed a hard reset of the client application. True story.

[2] Is there a better term for this? I feel like there should be an existing body of knowledge that I could reference, but my searching did not yield any promising results. The fact that “Logical Drift” is the name of a band probably isn’t helping.

Power BI Premium Dedicated Capacity Load Assessment Tool on GitHub

Last month[1] at the Microsoft Business Applications Summit (MBAS), Power BI program managers David Magar and Swati Gupta showed off a new load assessment tool for Power BI Premium capacity workloads.

premium tool

This new tool was included as part of the BRK2046  session on Power BI Premium at MBAS. The whole session is valuable, but the tool itself comes in around the the 32 minute mark. There’s a demo at the 37 minute mark. The tool is available today on github.

This tool will help Power BI Premium customers better plan for how their specific workloads (reports, dashboards, datasets, dataflows, and patterns of access) will perform on a given Premium capacity.

The tool is built on top of a PBIE load generation tool my teammate Sergei Gundorov has built to help ISVs better handle load on their PBIE capacities. The tool grabs a user’s token and uses it to render reports again and again, cycling through preset filter values and incrementing a “render counter”. The tool stops rendering when the authentication token expires, so the result is an empirical benchmark: “report X can run against capacity Y, Z times in 1 hour”.

The tool that’s available publicly used Sergei’s work as the starting point and uses PowerShell to turn it into a simple “menu-based” UX that anybody can run. The tool enables users to:

  • choose multiple reports to run at once
  • choose the credentials used for each report
  • define filter values to cycle through between renders for each report
  • define how many users (browser windows) should request the report at once

Once all definitions are set the tool launches multiple browser windows, each targeting different reports and the users can see the load test happening on screen.

The tool was an effective way for David and Swati to generate “interesting” load scenarios for their MBAS workshop. They used it demonstrate how phenomena related to overloaded capacities (such as query wait time build up and frequent evictions) are visible using the Power BI Premium metrics app. If you haven’t already watched the session recording, be sure to check it out.

The dedicated capacity load assessment tool is published on GitHub for anyone to use. There’s a post introducing it on the Power BI blog.

The folks at Artis Consulting have already taken the tool that Sergei developed and which was shown at MBAS and have released a “Realistic” load test tool, also on GitHub. This tool build on the original one, and makes it easier to simulate users interacting with reports in a more realistic manner, such as selecting filters and bookmarks.

If you’re interested in understanding how your Power BI application will scale under load on your dedicated capacity[2], check out these tools and consider how to incorporate them into your development and deployment processes.


[1] Yes, MBAS took place in June, and this is getting posted in October. I originally wrote this post in early July, and I put it on hold to wait for the official blog post to be done so I could include a link. It’s been languishing in my drafts ever since. Life comes at you fast…

[2] It’s worth emphasizing that this tool and this post apply only to dedicated Power BI capacity. If you are using shared capacity, you should not use this tool.