Automatically refresh dataset when dataflow refresh completes

Back in August I highlighted the new dataflows PowerShell script repo on GitHub. These scripts provide an accelerated starting point to working with the dataflows REST APIs and to automate common dataflows tasks.

This week the dataflows team has released two new REST APIs for managing dataflows transactions (think “refresh”) and a new parameterized PowerShell script to make using the APIs easier.

You can find the APIs documented here and the PowerShell script here.

The most exciting thing about these APIs and scripts[1] is that they enable one of the most frequently requested capabilities: you can now trigger a dataset refresh (or the refresh of a dataflow in another workspace) when a dataflow refresh completes.

Boom.

If you’re reading this and are thinking “that’s all well and good, but I don’t want to use an API or PowerShell[2]” please don’t fret. The Power BI team is working on non-API-based experiences to make the end-to-end refresh experience simpler and more complete. I can’t share dates or details here before they’re publicly announced, but I did want to share proactively before anyone asked about a code-free UX for refresh orchestration.


[1] And the best thing to happen in 2020 so far, if James Ward is to be believed.

[2] Or even “this should be ‘clicky-clicky-draggy-droppy’ and not ‘typey-typey-scripty-codey’.”

11 thoughts on “Automatically refresh dataset when dataflow refresh completes

  1. Hey buddy! Imagine someone is allergic to PowerShell. Is it possible for them to leverage Power Automate for this? And be able to do this before the Power BI team figures out their non-API experience? Asking for a friend. šŸ™‚

    Keep on being awesome!

    Like

    1. Short answer: I don’t know.

      I have only the most trivial Power Automate knowledge (I did a tutorial once back before it got its fancy new name) so I’m not qualified to say yes or no. I think/assume that you can use Power Automate to simplify calling the REST API (something like this https://docs.microsoft.com/en-us/power-automate/get-started-flow-dev) but I haven’t used it myself and honestly don’t know if I’m giving you a solution by sharing this link, or just sending you down a dead end path…

      Like

    2. Ian Felton

      @Micah
      Yes you can. You can use power automate to create a custom connector, which can use the API definition file (containing all Power BI APIs), and then call any API from that one connector. This is really useful for orchestration.

      Like

  2. Pingback: Automatically refresh dataset when dataflow refresh completes ā€” BI Polar | ERP and BI

  3. Pingback: Refresh a Dataset when Dataflow Refresh Completes – Curated SQL

  4. What about using a service principal to perform this but also to perform the import of the JSON file as a dataflow in a workspace (https://ssbipolar.com/2019/10/03/importing-model-json-to-create-a-new-dataflow/). Apparently that throws an error: “Import Failed with code [DataflowUnauthorizedError] and details []”.

    I believe the Post Import in Group is not an admin API to have this restriction around Service Principles. Long term, in production in order to fully automate things, it does not make any sense to leverage an actual account compared to a service principle that has the key secret stored in AKV. Also, cannot find any documentation around limitations of using Service Principals with the Post Import in group Rest API as to only statement around required scope is: Dataset.ReadWrite.All

    Like

    1. Hi Catalin – if you’re looking for assistance, your best bet is to post this on community.powerbi.com. I do not believe that this API supports service principals today, but I’m also not an expert in this area, so my belief is not guaranteed to be correct…

      Like

  5. Pingback: Automatically refresh dataset when dataflow refresh completes – ScienceGeek

  6. derek

    the best practise is to have extraction dataflow and transformation dataflow after it, with linked entities.
    the refresh command given on the extraction dataflow.

    however, these powershell scripts dont recognize the dataset (that gets data from the transformation dataflow) as dependent on the extraction dataflow. Still some work to be done.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s