Migrating to Power BI

One aspect of building a data culture is selecting the right tools for the job. If you want more people working with more data, giving the tools they need to do that work is an obvious[1] requirement. But how many tools do you need, and which tools are the right tools?

Migrating to the cloud

It should be equally obvious that the answer is “it depends.” This is the answer to practically every interesting question. The right tools for an organization depend on the data sources it uses, the people who work with that data, the history that has gotten the organization to the current decision point, and the goals the organization needs to achieve or enable with the tools it selects.

With that said, it’s increasingly common[2] to see large organizations actively working to reduce the number of BI tools they support[3]. The reasons for this move to standardization are often the same:

  • Reduce licensing costs
  • Reduce support costs
  • Reduce training costs
  • Reduce friction involved in driving the behaviors needed to build and grow a data culture

Other than reducing the licensing costs[4], most of these motivations revolve around simplification. Having fewer tools means learning and using fewer tools. It means everyone learning and using fewer tools, which often results in less time and money spent to get more value from the use of those tools.

One of the challenges in eliminating a BI tool is ensuring that the purpose that tool fulfilled is now effectively fulfilled by the tool that replaces it. This is where migration comes in.

The Power BI team at Microsoft has published a focused set of guidance articles focused specifically on migrating from other BI tools to Power BI.

This documentation was written by the inestimable Melissa Coates of Coates Data Strategies, with input and technical review by the Power BI customer advisory team. If you’re preparing to retire another BI tool and move its workload to Power BI – or if you’re wondering where to start – I can’t recommend it highly enough.


[1] If this isn’t obvious to a given organization or individual, I’m reasonably confident that they’re not actively trying to build a data culture, and not reading this blog.

[2] I’m not a market analyst but I do get to talk to BI, data, and analytics leaders at large companies around the world, and I suspect that my sample size is large and diverse enough to be meaningful.

[3] I’m using the word “support” here – and not “use” – deliberately. It’s also quite common to see companies remove internal IT support from deprecated BI tools, but also let individual business units continue to use them – but also to pay for the tools and support out of their own budgets. This is typically a way to allow reluctant “laggard” internal customer groups to align with the strategic direction, but to do it on their own schedules.

[4] I’m pretty consistent in saying I don’t know anything about licensing, but even I understand that paying for two things costs more than paying for one of those things.

Data Culture: Training for the Community of Practice

The last few posts and videos in this series have introduced the importance of a community where your data culture can grow, and ways to help motivate members of the community, so your data culture can thrive.

But what about training? How do we give people the skills, knowledge, and guidance that they need before they are able do work with data and participate in the data culture you need them to help build?

Training is a key aspect of any successful data culture, but it isn’t always recognized as a priority. In fact the opposite is often true.

I’ve worked in tech long enough, and have spent enough of that time close to training to know that training budgets are often among the first things cut during an economic downturn. These short-term savings often produce long-term costs that could be avoided, and more mature organizations are beginning to realize this.

In my conversations with enterprise Power BI customers this year, I’ve noticed a trend emerging. When I ask how the COVID-19 pandemic is affecting how they work with data, I hear “we’re accelerating our efforts around self-service BI and building a data culture because we know this is now more important than ever” a lot more than I hear “we’re cutting back on training to save money.” There’s also a clear correlation between the maturity of the organizations I’m talking with and the response I get. Successful data cultures understand the value of training.

I’ll let the video speak for itself, but I do want to call out a few key points:

  1. Training on tools is necessary, but it isn’t enough. Your users need to know how to use Power BI[1], but they also need to know how to follow organizational processes and work with organizational data sources.
  2. Training material should be located as close as possible to where learners are already working – the people who need it the most will not go out of their way to look for it or to change their daily habits.
  3. There is a wealth of free Power BI training available from Microsoft (link | link | link) as well as a broad ecosystem of free and paid training from partners.

The most successful customers I work with use all of the resources that are available. Typically they will develop internal online training courses that include links to Microsoft-developed training material, Microsoft product documentation, and community-developed content, in a format and structure[2] that they develop and maintain themselves, based on their understanding of the specific needs of their data culture.

Start as small as necessary, listen and grow, and iterate as necessary. There’s no time like the present.


[1] Or whatever your self-service BI tool of choice may be – if you’re reading this blog, odds are it’s Power BI.

[2] I’m tempted to use the term “curriculum” here, but this carries extra baggage that I don’t want to include. Your training solution can be simple or complex and still be successful – a lot of this will depend on your company culture, and the needs of the learners you’re targeting.

Webcast: Patterns for adopting dataflows in Power BI

I haven’t been posting a lot about dataflows in recent months, but that doesn’t mean I love them any less. On Wednesday September 23rd, I’ll be sharing some of that love via a free webcast hosted by the Istanbul Power BI user group[1]. You can sign up here.

In this webcast I’ll be presenting practices for successfully incorporating dataflows into Power BI applications, based on my experience working with enterprise Power BI customers. If you’re interested in common patterns for success, common challenges to avoid, and answers to the most frequently asked dataflows questions, please sign up today.

This webcast won’t cover dataflows basics, so if you’re new to dataflows in Power BI or just need a refresher, please watch this tutorial before joining!


[1] In case it needs to be said, yes, the session will be delivered in English.

Data Culture: Motivation and Encouragement

The last post in our ongoing series on building a data culture focused on the importance of community, and on ways organizations can create and promote successful communities around data. But while a community is where the data culture can grow, how can you motivate people to participate, to contribute, and to be part of that growth?

Business intelligence is more about business than it is about technology, and business is really about people. Despite this, many BI professionals focus their learning largely on the technology – not the people.

Do you remember the first time you were involved in a performance tuning and optimization effort? The learning process involved looking at the different parts of the tech stack, and in understanding what each part does, how it does it, and how it relates to all of the other parts. Only when you understood these “internals” could you start looking at applying your knowledge to optimizing a specific query or workload.

You need to know how a system works before you can make it work for you. This is true of human systems too.

This video[1] looks at motivation in the workplace, and how you can motivate the citizen analysts in your data culture to help it – and them – grow and thrive. If you think about these techniques as “performance tuning and optimization” for the human components in a data culture, you’re on the right track.

This image makes a lot more sense after you’ve watched the video – I promise

People are motivated by extrinsic motivators (doing something to get rewards) and intrinsic motivators (doing something because doing it makes them happy)[2], and while it’s important to understand both types of motivators, it’s the intrinsic motivators that are more likely to be interesting – and that’s where we spend the most time in the video.

When you’re done with the video, you probably want to take a moment to read this Psychology Today article, and maybe not stop there. People are complicated, and if you’re working to build a data culture, you need to understand how you can make people more likely to want to participate. Even with an engaged executive sponsor, it can be difficult to drive personal change.

In my personal experience, task identity and task significance are the biggest success factors when motivating people to contribute in a data culture. If someone knows that their work is a vital part of an important strategic effort, and if they know that their work makes other people’s lives better, they’re more likely to go the extra mile, and to willingly change their daily habits. That’s a big deal.


[1] If you’re not old enough to recognize the opening line in the video, please take a moment to appreciate how awesome commercials were in the 1980s.

[2] Yes, I’m oversimplifying.

Data Culture: The Importance of Community

The last two videos  in our series on building a data culture covered different aspects of  how business and IT stakeholders can partner and collaborate to achieve the goals of the data culture. One video focused on the roles and responsibilities of each group, and one focused on the fact that you can’t treat all data as equal. Each of these videos builds on the series introduction, where we presented core concepts about cultures in general, and data culture in particular.

Today’s video takes a closer look at where much of that business/IT collaboration takes place – in a community.

Having a common community space – virtual, physical, or both – where your data culture can thrive is an important factor in determining success. In my work with global enterprise Power BI customers, when I hear about increasing usage and business value, I invariably hear about a vibrant, active community. When I hear about a central BI team or a business group that is struggling, and I ask about a community, I usually hear that this is something they want to do, but never seem to get around to prioritizing.

Community is important.[1]

woman-1594711

A successful data culture lets IT do what IT does well, and enables business to focus on solving their problems themselves… but sometimes folks on both sides of this partnership need help. Where do they find it, and who provides that help?

This is where the community comes in. A successful community brings together people with questions and people with the answer to these questions. A successful community recognizes and motivates people who share their knowledge, and encourages people to increase their own knowledge and to share it as well.

Unfortunately, many organizations overlook this vital aspect of the data culture. It’s not really something IT traditionally owns, and it’s not really something business can run on their own, and sometimes it falls through the cracks[2] because it’s not part of how organizations think about solving problems.

If you’re part of your organization’s journey to build and grow a data culture and you’re not making the progress you want, look more closely at how you’re running your community. If you look online you’ll find lots of resources that can give you inspiration and ideas, anything from community-building ideas for educators[3] to tips for creating a corporate community of practice.


[1] Really important. Really really.

[2] This is a pattern you will likely notice in other complex problem spaces as well: the most interesting challenges come not within a problem domain, but at the overlap or intersection of related problem domains. If you haven’t noticed it already, I suspect you’ll start to notice it now. That’s the value (or curse) of reading the footnotes.

[3] You may be surprised at how many of these tips are applicable to the workplace as well. Or you may not be surprised, since some workplaces feel a lot like middle school sometimes…

Power BI dataflows PowerShell scripts on GitHub

Last week I shared a post highlighting a common pattern for making API data available through dataflows in Power BI, and included a few diagrams to show how a customer was implementing this pattern.

In the post I mentioned that I was simplifying things a bunch to only focus on the core pattern. One of the things I didn’t mention is that the diagrams I shared were just one piece of the puzzle. Another part was the need to define dataflows in one workspace, and then use those as a template for creating other dataflows in bulk.

This is simple enough to do via the Power BI portal for an individual dataflow, but if you need to do it for every dataflow in a workspace, you might need a little more power – PowerShell, to be specific.

The rest of this post is not from me – it’s from the dataflows engineering team. it describes a set of PowerShell scripts they’ve published on GitHub, and which address this specific problem. The rest is pretty self-explanatory, so I’ll just mention that these are unsupported scripts presented as-is, and I’ll let the rest speak for itself.

nautilus-1029360_1920

Microsoft Power BI dataflows samples

The document below describes the various PowerShell scripts available for Power BI dataflows. These rely on the Power BI public REST APIs and the Power BI PowerShell modules.

Power BI Dataflow PowerShell scripts

Below is a table of the various Power BI PowerShell modules found in this repository.

Description Module Name Download
Export all dataflows from a workspace ExportWorkspace.ps1 GitHub Location
Imports all dataflows into a workspace ImportWorkspace.ps1 GitHub Location
Imports a single dataflow ImportModel.ps1 GitHub Location

For more information on Powershell support for Power BI, please visit powerbi-powershell on GitHub

Supported environments and PowerShell versions

  • Windows PowerShell v3.0 and up with .NET 4.7.1 or above.
  • PowerShell Core (v6) and up on any OS platform supported by PowerShell Core.

Installation

  1. The scripts depend on the MicrosoftPowerBIMgmt module which can be installed as follows:
Install-Module -Name MicrosoftPowerBIMgmt

If you have an earlier version, you can update to the latest version by running:

Update-Module -Name MicrosoftPowerBIMgmt
  1. Download all the scripts from the GitHub Location into a local folder.
  2. Unblock the script by right click on the files and select “Unblock” after you download. Otherwise you might get a warning when you run the script.

Uninstall

If you want to uninstall all the Power BI PowerShell cmdlets, run the following in an elevated PowerShell session:

Get-Module MicrosoftPowerBIMgmt* -ListAvailable | Uninstall-Module -Force

Usage

The APIs below supports two optional parameters:

  • -Environment: A flag to indicate specific Power BI environments to log in to (Public, Germany, USGov, China, USGovHigh, USGovMil). Default is Public
  • -V: A flag to indicate whether to produce verbose output. Default is false.

Export workspace

Exports all the dataflow model.json from a Power BI workspace into a folder:

.\ExportWorkspace.ps1 -Workspace "Workspace1" -Location C:\dataflows

Import workspace

Imports all the dataflow model.json from a folder into a Power BI workspace. This script also fixes the reference models to point to the right dataflow in the current workspace:

.\ImportWorkspace.ps1 -Workspace "Workspace1" -Location C:\dataflows -Overwrite

Import dataflow

Imports a dataflow model.json into a Power BI workspace:

.\ImportModel.ps1 -Workspace "Workspace1" -File C:\MyModel.json -Overwrite

Data Culture: Picking Your Battles

Not all data is created equal.

One size does not fit all.

In addition to collaboration and partnership between business and IT, successful data cultures have something  else in common: they recognize the need for both discipline and flexibility, and have clear, consistent criteria and responsibilities that let all stakeholders know what controls apply to what data and applications.

2020-08-01-19-55-59-794--POWERPNT

Today’s video looks at this key fact, and emphasizes this important point: you need to pick your battles[1].

If you try to lock everything down and manage all data and applications rigorously, business users who need more agility will not be able to do their jobs – or more likely they will simply work around your controls. This approach puts you back into the bad old days before there were robust and flexible self-service BI tools – you don’t want this.

If you try to let every user do whatever they want with any data, you’ll quickly find yourself in the “wild west” days – you don’t want that either.

Instead, work with your executive sponsor and key stakeholders from business and IT to understand what requires discipline and control, and what supports flexibility and agility.

One approach will never work for all data – don’t try to make it fit.


[1] The original title of this post and video was “discipline and flexibility” but when the phrase “pick your battles” came out unscripted[2] as I was recording the video, I realized that no other title would be so on-brand for me. And here we are.

[2] In case you were wondering, it’s all unscripted. Every time I edit and watch a recording, I’m surprised. True story.

Dataflows and API data sources

These days more and more data lives behind an API, rather than in a more traditional data source like databases or files. These APIs are often designed and optimized for small, chatty[1] interactions that don’t lend themselves well to use as a source for business intelligence.

beehive-337695_1280
I’d explain the choice of image for APIs, but it doesn’t take a genus to figure it out

These APIs are often slower[3] than a database, which can increase load/refresh times. Sometimes the load time is so great that a refresh may not fit within the minimum window based on an application’s functional requirements.

These APIs may also be throttled. SaaS application vendors often have a billing model that doesn’t directly support frequent bulk operations, so to avoid customer behaviors that affect their COGS and their bottom line, their APIs may be limited to a certain number of calls[4] for a given period.

The bottom line is that when you’re using APIs as a data source in Power BI, you need to take the APIs’ limitations into consideration, and often dataflows can help deliver a solution that accommodates those limitations while delivering the functionality your application needs.

I’ve covered some of the fundamental concepts of this approach in past blog posts , specifically this one from 2018 on Power BI Dataflows and Slow Data Sources, and this one from 2019 on Creating “data workspaces” for dataflows and shared datasets.  I hadn’t planned on having a dedicated new blog post, but after having this pattern come up in 4 or 5 different contexts in the past week or so, I thought maybe a post was warranted.

In late July I met with a customer to discuss their Power BI dataflows architecture. They showed me a “before” picture that looked something like this:

2020-08-01-13-40-14-887--POWERPNT

One of their core data sources was the API for a commercial software application. Almost every Power BI application used some data from this API, because the application supports almost every part of their business. This introduced a bunch of familiar challenges:

  • Training requirements and a steeper-then-necessary learning curve for SSBI authors due to the complex and unintuitive API design
  • Long refresh times for large data extracts
  • Complex, redundant queries in many applications
  • Technical debt and maintenance due to the duplicate logic

Then they showed me an “after” picture that looked something like this:

2020-08-01-13-40-32-810--POWERPNT

They had implemented a set of dataflows in a dedicated workspace. These dataflows have entities that pull data from the source APIs, and make them available for consumption by IT and business Power BI authors. All data transformation logic is implemented exactly once, and each author can easily connect to trusted tabular data without needing to worry about technical details like connection strings, API parameters, authentication, or paging. The dataflows are organized by the functional areas represented in the data, mapping roughly to the APIs in the source system as viewed through the lens of their audience.

The diagram above simplifies things a bit. For their actual implementation they used linked and computed entities to stage, prepare, and present the dataflows, but for the general pattern the diagram above is probably the right level of abstraction. Each IT-developed or business-developed application uses the subset of the dataflows that it needs, and the owners of the dataflows keep them up to date and current.

Life is good[5].


[1]There’s a lot of information available online covering chatty and chunky patterns in API design, so if you’re not sure what I’m talking about here, you might want to poke around and take a peek[2] at what’s out there.

[2] Please let me know if you found the joke in footnote[1].

[3] Possibly by multiple orders of magnitude.

[4] Or a given data volume, etc. There are probably as many variations in licensing as there are SaaS vendors with APIs.

[5] If you’re wondering about the beehive image and the obtuse joke it represents, the genus for honey bees is “apis.” I get all of my stock photography from the wonderful web site Pixabay. I will typically search on a random term related to my blog post to find a thumbnail image, and when the context is completely different like it was today I will pick one that I like. For this post I searched on “API” and got a page full of bees, which sounds like something Bob Ross would do…

Data Culture: Roles and Responsibilities

In last week’s post and video we looked at how business intelligence tools have evolved, with each evolution solving one set of problems and introducing another set. The video described how self-service BI tools have enabled business users to work with data in ways that used to require specialized technical skills, freeing up IT to focus on other tasks – but at the same time introducing challenges around oversight, consistency, and control.

And that brings us to today’s video.

The video includes a few different graphics, but for this post I want to highlight this one, which I’ve taken from the Power BI Adoption Framework[1].

2020-07-31-20-44-46-355--POWERPNT

Successful cultures balance freedom and restriction in ways that benefit the culture as a whole. It’s a compromise – no one gets everything they want, but everyone gets the things they need the most.

For a data culture, this typically involves letting IT do the things they do best, and letting business get down to business.

When an organization adopts a managed self-service model for business intelligence[2], the central BI team in IT[3] does the heavy lifting. This means they prepare, cleanse, warehouse, and deliver the biggest and most important[4] data. They deliver the data that would be hard for a business user to create using a self-service tool, and which will be used by a large audience. They do the things that have broad reach, strategic impact, and strategic importance. They create things that need to be correct, consistent, and supported.

And business users do the rest. This is a broad bucket, but it’s a reasonable generalization as well. Business users create their own reports using the IT-managed data sources and data models. And they prepare and use their own data where there is no IT-managed data for a given purpose.

Over time a given application or a given data source may become more or less managed, as the culture adopts, adapts, and improves.

Managed self-service BI isn’t the only way to be successful, but in my experience working with many large Power BI customers it’s the best way and most predictable way. By having clearly defined roles and responsibilities – who does what – moving from either extreme can overcome the problems that that extreme creates, without going too far in the other direction.

Does your organization take this approach?

If it doesn’t today, what will it take to make this change a reality?


[1] Which I have shamelessly stolen from the Power BI Adoption Framework, because it is an awesome graphic and because I love to stand on the shoulders of giants.

[2] Which is the approach most likely to drive short-and long term efficiencies and successes.

[3] Please understand that this is a gross simplification. This “central BI team in IT” may be a single central team for the whole company, it may be a single central team for a specific business unit, or it may be one of dozens of BI teams that are designed to support a distributed global enterprise. This is less about the technology and the culture than it is about organizational structure, so I don’t expect to ever try to tackle this diversity of approaches in this series.

[4] Next week’s video will talk a little more about what data is likely to qualify for this distinction. And since the video is done already, I’m pretty confident to say “next week” this week.

Weekend reading: New dataflows docs

In case you didn’t have any plans for the weekend, the Power Query team has released a treasure trove of new documentation for dataflows in Power BI and Power Apps.

You can find the new docs here: https://docs.microsoft.com/en-us/power-query/dataflows/overview-dataflows-across-power-platform-dynamics-365

There are over 20 articles in this new resource, and the content they contain answers many of the questions I hear most frequently. For some reason WordPress isn’t letting me upload the screen shot I took of the content outline, so instead here is a picture of me[1] when I first learned about this new content:

woman-591576_1280

Please check out this new information, and please submit feedback to the documentation team using the feedback form on each page. When you let the docs team know what you like, what you don’t like, or what is missing, they read every comment!


[1] Me in my avatar as a photogenic young woman.