Category: Azure

Power BI dataflows, Premium, and ADLSg2

On By Matthew RocheIn Azure, Dataflows, Power BI1 Comment

I received a question today via Twitter, and although I know the information needed to answer it is available online, I don’t believe there’s a single concise answer anywhere[1]. This is the question, along with a brief elaboration following my initial response:

Billing Twitter

Here’s the short answer: When you use an organizational ADLSg2 account to store dataflow data, your Azure subscription will be billed for any storage and egress based on however Azure billing works[2].

Here’s the longer answer:

  • Power BI dataflows data counts against the same limits as Power BI datasets. Each Pro license grants 10 GB of storage,  and a Premium capacity node includes 100 TB of storage.
  • Integrating Power BI dataflows with ADLSg2 is not limited to Power BI Premium.
  • When you’re using Power BI dataflows in their default configuration, dataflow data is stored to this Power BI storage, and counts against the appropriate quota.
  • When dataflow data is saved to Power BI storage, it can only be accessed by Power BI – no other services or applications can read the data.
  • When you configure your dataflows to use an organizational ADLSg2 account, the dataflow data is saved to the Azure resource you specify, and not to the Power BI storage, so it doesn’t count against the Pro or Premium storage quota. This is particularly significant when you’re not using Power BI Premium, as ADLSg2 storage will scale to support any scenario, and not be limited by the 10 GB Pro storage limit.
  • When dataflow data is saved to ADLSg2, the CDM folders can be accessed by any authorized client via Azure APIs, and by Power BI as dataflow entities. This is particularly valuable for enabling collaboration between analysts and other Power BI users, and data scientists and other data professionals using Azure tools.

Hopefully this will help clear things up. If you have any questions, please let me know!

[1] Please note that I didn’t actually go looking to make sure, because I was feeling lazy and needed an excuse to blog about something vaguely technical.

[2] I add that final qualifier because I am not an authority on Azure or Power BI billing, or on licensing of any sort. For any specific information on licensing or billing, please look elsewhere for expert advice, because you won’t find it here.

 

Status Check: Power BI dataflows and ADLSg2

On By Matthew RocheIn Azure, Dataflows, Power BI5 Comments

In the last few weeks I’ve seen a spike in questions related to the integration of Power BI dataflows and Azure Data Lake Storage Gen2. Here’s a quick “status check” on the current state of this feature to get the answers out for as many people as possible.

  • Power BI dataflows are generally available (GA) for capabilities that use the built-in Power BI-managed storage.
  • Power BI dataflows integration with Azure is currently in preview – this includes the “Bring your own storage account” capabilities where you can configure Power BI to use your ADLSg2 storage account for dataflows storage, instead of using the built-in Power BI-managed storage.
  • During preview, there are several known limitations:
    • Only a single ADLSg2 storage account can be configured for a Power BI tenant.
    • The storage account, once configured, cannot be changed.
    • The setup process to connect Power BI with ADLSg2 is somewhat lengthy and step-intensive.
    • To grant users other than the owner of the dataflow access to the dataflow in Power BI, you must grant them permissions to access the workspace in Power BI and grant them access to the CDM folder in ADLSg2.

These limitations will be addressed when this capability hits GA, but you should definitely be aware of them in the meantime. (You may also want to take a look at this MBAS session for an overview of the roadmap for the rest of this calendar year.)

I’ve seen customers take different approaches:

  1. Some customers delay their integration of Power BI and ADLSg2, and are waiting for these limitations to be removed before they move forward.
  2. Some customers adopt within the constraints of the preview, and choose a workload or set of workloads where the current limitations are acceptable.
  3. Some customers set up a demo tenant of Power BI and use it to test and validate the ADLSg2 integration while deciding on option 1 or option 2.

I hope this helps. If you or your customers have any questions on this topic that aren’t answered here, please let me know!

[1] And they’re all documented. Nothing in this blog post is new, but hopefully it will help to have this summary online and sharable.

Power BI dataflows and CDM Sessions from MBAS 2019

On By Matthew RocheIn Azure, Dataflows, Power BI, Power Query, Presentations2 Comments

Last week Microsoft held its annual Microsoft Business Applications Summit (MBAS) event in Atlanta. This two-day technical conference covers the whole Business Applications platform – including Dynamics, PowerApps, and Flow – and not just Power BI, but there was a ton of great Power BI content to be had. Now that the event is over, the session recordings and resources are available to everyone.

MBAS 2019 Banner

There’s a dedicated page on the Power BI community site with all of the sessions, but I wanted to call out a few sessions on dataflows and the Common Data Model that readers of this blog should probably watch[1].

Power BI dataflows sessions

Microsoft Power BI: Democratizing self-service data prep with dataflows

This session is something of a “deep technical introduction” to dataflows in Power BI. If you’re already familiar with dataflows a lot of this will be a repeat, but there are some gems as well.

Microsoft Power BI: Enterprise-grade BI with Power BI dataflows

This session is probably my favorite dataflows session from any conference. This is a deep dive into the dataflows architecture, including the brand-new-in-preview compute engine for performance and scale.

Common Data Model sessions

As you know, Power BI dataflows build on CDM and CDM folders. As you probably know, CDM isn’t just about Power BI – it’s a major area of investment across Azure data services as well. The session lineup at MBAS reflected this importance with three dedicated CDM sessions.

Common Data Model: All you need to know

This ironically-named session[2] provides a comprehensive overview of CDM. It’s not really everything you need, but it’s the right place to begin if you’re new to CDM and want to the big-picture view.

Microsoft Power BI: Common Data Model and Azure Data Services

This session covers how CDM and CDM folders are used in Power BI and Azure data services. If you’ve been following dataflows and CDM closely over the past six months much of this session might be review, but it’s an excellent “deep overview” nonetheless.

Microsoft Power BI: Advanced concepts in the Common Data Model

This session is probably the single best resource on CDM available today. The presenters are the key technical team behind CDM, and goes into details and concepts that aren’t available in any other presentation I’ve found. I’ve been following CDM pretty closely for the past year or more, and I learned a lot from this session. You probably will too.

Once you’re done watching these sessions, remember that there’s a huge library of technical sessions you can watch on-demand. Also some less-technical sessions.

[1] I have a list of a dozen or more sessions that I want to watch, and only a few of them are dataflows-centric. If you look through the catalog you’ll likely find some unexpected gems.

[2] If this is all you need to know, why do we have these other two sessions?

[3] Including Jeff Bernhardt, the architect behind CDM. Jeff doesn’t have the rock star reputation he deserves, but he’s been instrumental in the design and implementation of many of the products and services on which I’ve built my career. Any time Jeff is talking, I make a point to listen closely.

Session resources: Power BI dataflows and Azure Data Lake integration

On By Matthew RocheIn Azure, Dataflows, Power BI, Presentations10 Comments

Last week I delivered two online sessions on the topic of integrating Power BI dataflows with an organizational Azure Data Lake Storage Gen2 storage account. I’ve blogged on this topic before (link | link | link | link) but sometimes a presentation and demo is worth a thousand words.

On April 30th I delivered a “Power BI dataflows 201 – beyond the basics” session for the PASS Business Intelligence Virtual Chapter. The session recording is online here, and you can download the slides here.

On May 4th I delivered a “Integrating Power BI and Azure Data Lake with dataflows and CDM Folders” session for the SQL Saturday community event in Stockholm, Sweden. I was originally planning to deliver the Stockholm session in person, but due to circumstances beyond my control[1] I ended up presenting remotely, which meant that I could more easily record the session. The session recording is online here, and you can download the slides here.

Each of these sessions covers much of the same material. The Stockholm presentation got off to a bit rocky start[2] but it gets smoother after the first few minutes.

Please feel free to use these slides for your own presentations if they’re useful. And please let me know if you have any questions!

[1] I forgot to book flights. Seriously, I thought I had booked flights in February when I committed to speaking, and by the time I realized that I had not booked them, they were way out of my budget. This was not my finest hour.

[2] The presentation was scheduled to start at 6:00 AM, so I got up at 4:00 and came into the office to review and prepare. Instead I spent the 90 minutes before the session start time fighting with PC issues and got everything working less than a minute before 6:00. I can’t remember ever coming in quite this hot…

Upcoming Dataflows Presentations

On By Matthew RocheIn Azure, Dataflows, Power BI1 Comment

I’m not dead!

After having a prolific first few months with this blog, the year-end holidays disrupted my routine, and I’ve been struggling to get everything balanced again. 2019 has been great so far, but its also been stupid crazy busy, and while blogging has been on my backlog, it just hasn’t made the bar for implementation.

Until now, I guess…

Last week I was in Darmstadt, Germany for the SQL Server Konferenz 2019 event, where I presented on Power BI dataflows. My session was well-attended and well-received[1] but I realized that I’d never actually told anyone about it. Now it’s time to correct this oversight for some upcoming events. These are the public events where I’ll be speaking over the next few months:

Event: SQL Saturday #826

Location: Victoria, BC, Canada

Date: March 16, 2019

Session: Introduction to Power BI dataflows

Event: Intelligent Cloud Conference

Location: Copenhagen, Denmark

Date: April 9, 2019

Session: Integrating Power BI and Azure Data Lake with dataflows and CDM Folders

Pro tip: If you’re attending Intelligent Cloud, be sure to attend Wolfgang Strasser‘s “Let your data flow” session earlier in the day. This session will provide a great introduction to Power BI dataflows and will provide the prerequisite knowledge necessary to get the most out of my session.

Event: SQL Saturday #851

Location: Stockholm, Sweden

Date: May 4, 2019

Session: Hopefully two dataflows sessions[2], details still being ironed out.

 

[1] Except for that one guy who rated the session a 2. I’d love to know what I could have done to improve the presentation and demos

[2] Also, swords.

Dataflows, or Data Flows?

On By Matthew RocheIn Azure, Azure Data Factory, Dataflows3 Comments

I don’t hear this question as often as I used to[1], but I still hear it: What’s the difference between dataflows in Power BI and Data Flows in Azure Data Factory?

I’ve already written extensively on the Power BI side of things, and now the awesome Merrill Aldrich from BlueGranite has published an excellent overview of the ADF feature on the BlueGranite blog. You should check it out here: https://www.blue-granite.com/blog/ssis-inspired-visual-data-transformation-comes-to-azure-data-factory

I’m not going to summarize the similarities and differences in this post, but after you’ve read Merrill’s article, I’d love to hear your specific questions.

[1] This may or may not be because I’ve been on vacation for the past three weeks.

Choosing Between Power BI Premium and Azure Analysis Services

On By Matthew RocheIn Azure, Power BI7 Comments

Yesterday I posted an article comparing Power BI dataflows, Power BI datasets, and Azure Analysis Services. Although I’d like to believe that the article was useful, I used the disclaimer “I’m not an expert” in multiple places where I was talking about the differences between models in Power BI and AAS. I may not be an expert, but I do know quite a few people who are.

Specifically I know Gabi Münster and Oliver Engels from oh22data AG in Germany, and Paul Turley from Intelligent Business LLC in the United States.

Gabi and Oliver presented last month at the PASS Summit conference in Seattle on this very topic. Their session “Azure Analysis Services or Power BI – Which service fits you best?” looked at the history of the two services, their current capabilities and strengths, and future capabilities that Microsoft has announced in its business intelligence release notes. They even included a decision flowchart!

If you weren’t able to attend their session in November, I have good news and I have bad news and I have more good news.

The good news is that this session is included in the PASS Summit session recordings, which you can purchase and download today.

The bad news is that the session recordings cost $699, which may be difficult to justify if this is the only session you’re interested in[1].

The good news is that Oliver and Gabi were kind enough to share the slide deck and let me share it with you. You can download it here: AAS or PBI – Which service fits – from PASS Summit 2018.

And I’m very happy to see that their conclusions line up pretty well with my previous post.

PBI AAS

Paul has also presented a conference session[2] related to this topic, and has also recently blogged with an excellent feature comparison table between the different options in SQL Server Analysis Services, Azure Analysis Services, and Power BI.

If you’re in a position where you need to select a BI platform, I highly recommend checking out these resources, as they includes both valuable information, and a framework for using that information in different common scenarios.

Update: Check out this new post from James Fancke at selfservedbi.com: Why Azure Analysis Services is a great value proposition. This article provides a great counterpoint to my post, and drill-down specifically into Azure Analysis Services, and is well worth a read.

Update: Also check out this new post from Dave Ruijter at the moderndata.ai: Guide To Selecting Between Azure Analysis Services And Power BI Premium.

And if after reading these posts and this slide deck you still have unanswered questions, please seek professional help. Specifically, please find a Microsoft Partner who specializes in business intelligence, or a similar expert consultant who can help evaluate your specific needs and how the different available technical options can be applied to address them.

[1] The conference had many awesome sessions, so this should not be the only one you’re interested in.

[2] A delightfully themed conference session, at that.