One diagram to rule them all

A few weeks back MVP Paul Turley blogged on Power Query performance and diagnostics. It was a good, useful post, but I wasn’t really the target audience and I probably would have forgotten about it if it weren’t for one thing.

This diagram.

pbi
It really says it all, doesn’t it?

Look at it.

Look at it again, and pause to thoughtfully consider its elegance and beauty.

In the time since Paul shared this post, I’ve been involved in any number of conversations[1] where customer stakeholders had questions about Power BI application performance. This type of conversation isn’t particularly new, but now I’ve started using this diagram[2] as a point of reference.

The results have been very positive. Although nothing in the diagram is new or particularly interesting on its own, having this simple visual reference for the components that make up the canonical end-to-end flow in a Power BI application have made my conversations more useful and productive. Less time is required to get all stakeholders to a point of shared understanding – more time can be devoted to identifying and solving the problem.

I don’t know if Paul truly appreciates the beauty of what he’s created. But I do. And you should too.


[1] In case you’ve been wondering why my blog and YouTube output has dried up this month, it’s because real life has been kicking my ass. I think I can finally see the light at the end of the tunnel, so hopefully we’ll be back with regular content before too long. Hopefully.

[2] This beautiful, simple, elegant diagram.

Common sense solutions to simple problems

Have you ever looked at a problem and immediately known how to solve it?

Have you ever seen that there’s already a group of people who are responsible for solving this problem this problem, and although they’re working on a fix they obviously have not seen the common sense solution that is so apparent to you?

ford-498244_640
I knew we could park here! Problem solved!

Yeah[1].

In situations like these, there are three[2] potential reasons for what you’re seeing:

  1. The other people are just too stupid to to see the simple and obvious solution that’s been sitting right under their noses all this time.
  2. The other people, due to the time that they have spent closely examining the problem and potential solutions, have realized that the simple and obvious solution doesn’t actually address the problem – even though common sense suggests that it should – or that the solution has unintended side-effects that outweigh its benefits.
  3. The other people are responsible for additional problems and solutions across a broader or more strategic scope, and although the simple solution may make sense in isolation, implementing it has been deferred to enable work on other, higher priority problems.

(At this point I feel compelled to point out that this is something I’ve thought about for decades, and I’ve had a draft blog post on this subject for almost a year. I’m not trying to subtweet[3] anyone. I’ve had several people recently reach out to me to make sure that they’re not the reason I tweet periodic reminders that lazy communication is theft, so a disclaimer here feels appropriate.)

In my experience, many people seem to automatically gravitate to the first choice, and that bias seems exacerbated by differences in perspective. If the observer is in a business group and the solution team is in IT – or vice versa[4] – it’s easy for them to miss the complexity and nuance in something that appears simple and straightforward from the outside.

Part of this challenge is inherent in this type of relationship. It’s natural and appropriate to hide internal complexity from external stakeholders. I’m pretty sure I even learned about this in college… not that I remember college[5].

But part of it is the choice we make when we engage in the cross-team relationship. Every time we see a “simple problem” with an “obvious” or “easy” or “common sense” solution we get to decide how to react. We can assume that we know everything important and that the “other guy” is an idiot… or we can consider the option that there might be something we don’t already know.

The choice is ours.


[1] This is less of an actual footnote than a tool to enforce a dramatic pause as you read. Assuming you read footnotes as you go through the post, rather than reading them all at the end. I may need more telemetry to better understand use behavior. Why aren’t any simple problems actually as simple as they first seem?

[2] If you can think of other reasons that aren’t just variations on these three, I would love to hear about them in the comments. My list used to only include the first two reasons, but over the last few years I’ve added the third.

[3] Sub-blog? is that a thing?

[4] There are scores of other variations on this theme – I picked this one because it’s likely to be familiar to the people I imagine read my blog.

[5] Maybe this?

Data culture and the centerline

I’m running behind on my own YouTube publishing duties[1], but that doesn’t keep me from watching[2] the occasional data culture YouTube video produced by others.

Like this one:

Ok… you may be confused. You may believe this video is not actually about data culture. This is an easy mistake to make, and you can be forgiven for making it, but the content of the video make its true subject very clear:

A new technology is introduced that changes the way people work and live. This new technology replaces existing and established technologies; it lets people do what they used to do in a new way – easier, faster, and further. It also lets people do things they couldn’t do before, and opens up new horizons of possibility.

The technology also brings risk and challenge. Some of this is because of the new capabilities, and some is because of the collision[3] between the new way and the old way of doing things. The old way and the new way aren’t completely compatible, but they use shared resources and sometimes things go wrong.

At the root of these challenges is users moving faster than any relevant authorities. Increasing numbers of people are seeing the value of the new technology, assuming the inherent risk[4], and embracing its capabilities while hoping for the best.

Different groups see the rising costs and devise solutions for these challenges. Some solutions are tactical, some are strategic. And eventually some champions emerge to push for the creation of standard solutions. Or standards plural, because there always seems to be more than one of those darned things.

Not everyone buys into the standards at first, but over time the standards are refined and… actually standardized.

This process doesn’t slow down the technology adoption. The process and the standards instead provide the necessary shape and structure for adoption to take place as safely as possible.

With the passage of time, users take for granted the safety standards as much as they take for granted the capabilities of the technology… and can’t imagine using one without the other.

For the life of me I can’t imagine why they kept doubling down on the “lane markings” analogy, but I’m actually happy they did. This approach may get more people paying attention – I can’t find any other data culture videos on YouTube with 488K views…

road-220058_640


[1] Part of this is because my wife has been out of town, and my increased parental responsibilities have reduced the free time I would normally spend filming and editing… but it’s mainly because I’m finding that talking coherently about data culture is harder for me than writing about data culture. I’ll get better, I assume. I hope.

[2] In this case, I watched while I was folding laundry. As one does.

[3] Yes, pun intended. No, I’m not sorry.

[4] Either through knowledge or through ignorance.

Have you looked at the Power BI roadmap lately?

In case you missed it, Microsoft has published the “2020 release wave 1” release plan for the Power Platform, including Power BI.

You can find the goodness here: Power Platform: 2020 release wave 1 plan.

globe-trotter-1828079_640
I have the map, and the road… where are dataflows on this thing?

Even though you won’t see the term “roadmap” anywhere in the release plan[1] docs, this is how I think of them – because they’re the best, most current, and most complete public view of what Microsoft is planning for Power BI and the rest of the Power Platform.

Check it out today, and also check back in as the release plan is updated periodically[2] as the teams have more clarity and detail to share.


[1] Yes, these were called “release notes” not too long ago. No, I don’t know why picking a name and sticking with it is so hard. Yes, I will do my best to call these “roadmap” even though this isn’t their official name. Hashtag power rebel.

[2] I think the docs team publishes updates every week, but not every article gets modified in each update. I’m also not 100% sure about the weekly publishing schedule, which is why I buried this in a footnote that no one will actually read.

Power BI and ACLs in ADLSg2

In addition to using Azure Data Lake Storage Gen2 as the location for Power BI dataflows data, Power BI can also use ADLSg2 as a data source. As organizations choose ADLSg2 as the storage location for more and more data, this capability is key to enabling analysts and self-service BI users to get value from the data in the lake.

boje-2914324_640
Oh buoy, that is one big data lake!

But how do you do this in as secure a manner as possible, so that the right users have the minimum necessary permissions on the right data?

The short answer is that you let the data source handle secure access to the data it manages. ADLSg2 has a robust security model, which supports both Azure role-based access control (RBAC) and POSIX-like access control lists (ACLs)[1].

The longer answer is that this robust security model may make it more difficult to know how to set up permissions in the data lake to meet your analytics and security requirements.

Earlier this week I received a question from a customer on how to get Power BI to work with data in ADLSg2 that is  secured using ACLs. I didn’t know the answer, but I knew who would know, and I looped in Ben Sack from the dataflows team. Ben answered the customer’s questions and unblocked their efforts, and he said that I could turn them into a blog post. Thank you, Ben![2]

Here’s what you should know:

1 – If you’re using ACLs, you must at least specify a filesystem name in the URL to load in the connector (or if you access ADLS Gen2 via API or any other client).

i.e. Path in Power BI Connector must at least be: https://storageaccountname.dfs.core.windows.net/FileSystemName/

2 – For every file you want to read its contents, all parent folders and filesystem must have the “x” ACL. And the file must have a “r” ACL.

i.e. if you want to access the file: https://StorageAccountName.dfs.core.windows.net/FileSystemName/SubFolder1/File1.csv

3 – For files you want to list, all parent folders and filesystem must have the “x” ACL. The immediate parent folder must also have a “r” ACL.

i.e. if you want to view and access the files in this subfolder: https://StorageAccountName.dfs.core.windows.net/FileSystemName/SubFolder1/

4 – Default ACLs are great way to have ACLs propagate to child items. But they have to be set before creating subfolders and files, otherwise you need to explicitly set ACLs on each item.[3]

5 – If permission management is going to be dynamic, use groups as much as possible rather than assigning permissions to individual users[4]. First, ACL the groups to folders/files and then manage access via membership in the group.

6 – If you have an error accessing a path that is deep in the filesystem, work your way from the filesystem level downwards, fixing ACL settings in each step.

i.e. if you are having trouble accessing https:/StorageAccountName.dfs.core.windows.net/FileSystemName/SubFolder1/SubFolder2/File(s)

First try: https://StorageAccountName.dfs.core.windows.net/FileSystemName

Then: https://StorageAccountName.dfs.core.windows.net/FileSystemName/SubFolder1

And so on.

Update: James Baker, a Program Manager on the Azure Storage team has published on GitHub a PowerShell script to recursively set ACLs. Thanks to Simon for commenting on this post to make me aware of it, Josh from the Azure support team for pointing me to the GitHub repo, and of course to James for writing the actual script!


[1] This description is copied directly from the ADLSg2 documentation, which you should also read before acting on the information in this post.

[2] Disclaimer: This post is basically me using my blog as a way to get Ben’s assistance online so more people can get insights from it. If the information is helpful, all credit goes to Ben. If anything doesn’t work, it’s my fault. Ok, it may also be your fault, but it’s probably mine.

[3] This one is very important to know before you begin, even though it may be #3 on the list.

[4] This is a best practice pretty much everywhere, not just here.

Tough love in the data culture

When I shared on Twitter the most recent installment in the BI Polar “data culture” series, BI developer Chuy Varela replied to agree with some key points from the video on executive sponsorship… and to share some of his frustrations as well[1].

2020-01-13-08-46-03-439--msedge

Chuy’s comments made me think of advice that I received a few years back from my manager at the time. She told me:

Letting others fail is a Principal-level behavior.

Before I tie this tough love into the context of a data culture and executive sponsorship, I’d like to share the context in which the advice was given.

At the time I had just finished a few 80-hour work weeks, including working 12+ hour days through a long holiday weekend. Much of that time was spent performing tasks that weren’t my responsibility – another member of the extended team was dropping the ball, and with a big customer-facing milestone coming up I wanted to make everything as close to perfect as possible.

Once the milestone had been met, my manager pulled me aside and let me know how unhappy she was that I had been making myself ill to deliver on someone else’s commitments[4]. She went on to elaborate that because of my well-intentioned extra effort, there were two negative consequences:

  1. The extended team member would not learn from the experience, so future teams were more likely to have the same challenges.
  2. The executive sponsors for the current effort would believe that the current level of funding and support was enough to produce the high quality and timely deliverables that actually required extra and unsustainable work.

She was right, of course, I’ve added two phrases to my professional vocabulary because of her advice.

The first phrase[5] gets used when someone is asking me to do work:

That sounds very important, and I hope you find the right folks to get it done. It sounds too important for me to take on, because I don’t have the bandwidth to do it right, and it needs to be done right.

The second phrase gets used when I need to stop working on something:

After this transition I will no longer be performing this work. If the work is still important to you, we’ll need to identify and train someone to replace me. If the work is no longer important, no action is required and the work will no longer be done.

Now let’s think about data culture and executive sponsorship in a bottom-up organization. How does a sponsor – or potential sponsor – react when a BI developer or analyst is delivering amazing insights when it’s not really their job?

If the sponsor is bought in to the value of a data culture, their reaction is like to include making it their job, and ensuring that the analyst gets the support and resources to make this happen. This can take many different forms[6], but should always include an explicit recognition of the work and the value it delivers.

And what if the analyst who has enabled more data-driven decisions is ready to move on to new challenges? What happens to the solutions they’ve built and the processes that rely on their work? What then?

Again, if the sponsor is committed to a data culture, then their reaction will involve making sure that it becomes someone’s responsibility to move the analyst’s work forward. They will assign the necessary resources – data resources, human resources, financial resources – to ensure that the data-driven insights continue.

If you’re working on helping to build a data culture from the bottom up, there are a few opportunities to apply this advice to that end. Please understand that there is risk involved in some of these approaches, so be certain to take the full context into consideration before taking action.

  1. Work with your immediate manager(s) to amplify awareness of your efforts. If your managers believe in the work that you’re doing, they should be willing to help increase visibility.
  2. Ensure that everyone understands that your work needs support to be sustained. You’ve done something because it was the right thing to do, but for you to keep doing it, but you can’t keep doing it with all of your other responsibilities.
  3. When it comes time to change roles, ask who will be taking over the solutions you’ve built in your current role, and how you can transition responsibility to them.

Each of these potential actions is the first step down a more complicated path, and the organization’s response[7] will tell you a lot about the current state of the data culture.

For the first potential course of action, managerial support in communicating the value and impact of current efforts can demonstrate the art of the possible[8] to a potential executive sponsor who may not be fully engaged. If you can do this for your current team or department, imagine what a few more empowered folks could do for the business unit! You see where I’m going…

If this first course of action is the carrot, the other two might be the stick. Not everyone will respond the way we might like to our efforts. If you’re building these solutions today without any additional funding or support, why would I give you more resources? Right? In these cases, the withdrawal or removal of existing capabilities may be what’s necessary to communicate the value of work that’s already been completed.

Wow, this post ended up being much longer than planned. I’ll stop now. Please let me know what you think!


[1] I don’t actually believe in New Years resolutions, but I am now retroactively adding “start a blog post with a sentence that contains six[2] or more hyperlinks to my goals for 2020[3].

[2] Yes, I had to go back to count them.

[3] Nested footnotes achievement unlocked!

[4] If your reaction at this point is “what sort of manager would yell at you for doing extra work?” the answer is “an awesome manager.” This may have been the best advice I ever received.

[5] I’ve probably never said either phrase exactly like this, but the sentiment is there.

[6] Including promotion, transfer to a new role, and/or adding capacity to the analyst’s team so the analyst can focus more on the new BI work.

[7] Remember that a culture is what people do, not what people say – have I mentioned that before?

[8] The art of the possible is likely to warrant its own post and video before too long, so I won’t go into too much depth here.

Data culture: Executive sponsorship

Continuing on our series on data culture, we’re examining the importance of having an executive sponsor. This is one of the least exciting success factors for implementing Power BI and getting more insights from more data to deliver more value to the business… but it’s also one of the most important factors.

Let’s check it out:

Ok, what did we just watch?

This video (and the series it’s part of) includes patterns for success I’ve observed as part of my role on the Power BI CAT team[1]. and will complement the guidance being shared in the Power BI Adoption Framework.

The presence of an executive sponsor is one of the most significant factors for a successful data culture. An executive sponsor is:

  • Someone in a position of authority who shares the goals having important business decisions driven by accurate and timely data
  • A leader who can help remove barriers and make connections necessary to build enterprise data solutions
  • A source of budgetary[2] and organizational support for data initiatives
skyscraper-3184798_1920
Because executives fly on planes, right?

Without an executive sponsor, the organizational scope of the data culture is often limited by the visibility that departmental BI successes can achieve. The data culture will grow gradually and may eventually attract executive attention… or may not.

Without an executive sponsor, the lifetime of a data culture is often limited by the individuals involved. When key users move to new roles or take on new challenges and priorities, the solutions they’ve developed can struggle to find new owners.

Without an executive sponsor, all of the efforts you take to build and sustain a data culture in your organization will be harder, and will be more likely to fail.

Who is your executive sponsor?

Update: A Twitter conversation about this video sparked a follow-up post. You can check it out here: Tough Love in the data culture.


[1] This is your periodic reminder that this is my personal blog, and all posts and opinions are mine and mine alone, and do not reflect the opinions of my employer or my teenage children.

[2] This aspect of sponsorship is a bigger deal than we’re going to cover in this post and video – organizations fund what’s important to them, and they don’t fund what’s not.