Month: January 2021

Will it fold? Look in Power Query Online!

On By Matthew RocheIn Dataflows, Power BI, Power Query3 Comments

Query folding is a key capability in Power Query. In short, query folding is the process by which the Power Query mashup engine takes the steps defined in the Power Query editor and translates them into processing to be performed by the data source from which the query is extracting data.

The use of query folding can produce significant performance improvements. Consider a query that includes a grouping step. Without query folding, all of the detailed data required must be extracted from the data source, and the aggregation performed by the mashup engine. With query folding, the mashup engine can add a GROUP BY clause[1] to the data source query it generates, and lets the data source perform the costly operation – only the aggregated data leaves the data source.

Because of the performance benefit that query folding provides, experienced query authors are typically very careful to ensure that their queries take advantage of the capabilities of their data sources, and that they fold as many operations as possible. But for less experienced query authors, telling what steps will fold and which will not has not always been simple…

Until now.

This week the Power Query team announced the introduction of query folding indicators in Power Query Online. If you’re authoring a dataflow in Power BI or Power Apps, you will now see visual indicators to let you know which steps will fold, and which will not.

Each step will include an icon[2] that shows how that step will be handled. The announcement blog post goes into more details (because connectivity and data access are complicated topics, and because of the huge diversity in the data sources to which Power Query can connect) but the short version of the story is that life just got simpler for people who build dataflows.

After the last 12 months anything that makes my life simpler is a good thing.

[1] Or whatever the syntax is for the data source being used.

[2] Or is it a logo?

Old videos, timeless advice

On By Matthew RocheIn Career, Communication, Generic Blather, PatternsLeave a comment

It’s 2021, which means my tech career turns 25[1] this year.

It me

Back in 1995 an awesome book was published: Dynamics of Software Development by Jim McCarthy. Jim was a director on the Visual C++ team at Microsoft back when that was a big deal[2]. The book is one of the first and best books I read as a manager of software development teams, and it has stood the test of time – I still have it on the bookshelf in my office[3].

Of course, I wasn’t managing a software development team in 1995. I discovered Jim’s wisdom in 1996 or 1997 when I was preparing to teach courses on the Microsoft Solutions Framework[4]. When I opened the box that the trainer kit came in, there was a CD-ROM with videos[5] of Jim McCarthy presenting to an internal Microsoft audience on his “rules of thumb” for reliably shipping great software. When I first watched them, it was eye-opening… almost life-changing. Although I did have a mentor at the time, I did not have a mentor like Jim.

The second time I watched the videos, it was with my whole team. Jim’s rules from these videos and his book became part of the team culture[6], and I’ve referred back to them many times over the decades.

Now you can too – the videos are available on YouTube:

Some of the rules may feel a bit dated if you’re currently shipping commercial software or cloud services[7] but even the dated ones are based on fundamental and timeless truths of humans and teams of humans.

I’m going to take some time this week to re-watch these videos, and to start off the new year with this voice from years gone by. If you build software and work with humans, maybe you should too.

[1] Damn, that is weird to type, since I’m sure I can’t even be 25 years old yet.

[2] It may also be a big deal today, but to me at least it doesn’t feel quite as much.

[3] Not that I’ve seen my office or my bookshelf in forever, so this assertion should be taken with a grain of 2020.

[4] Fun fact: if you remember MSF, you’re probably old too.

[5] Because in those days “video” and “internet” weren’t things that went together. This may or may not have been because the bits had to walk uphill both ways in the snow.

[6] To the extent the team could be said to have a shared culture. We were all very young, and very inexperienced, and were figuring things out as we went along.

[7] Back in 1995 CI/CD wasn’t part of the industry vocabulary, and I can count on zero hands the clients I worked with before joining Microsoft who had anything resembling a daily build.