Problems, not solutions

Imagine walking into a restaurant.

No, not that one. Imagine walking a nicer restaurant than the one you thought of at first. A lot nicer.

restaurant-2697945_640
Even nicer than this.

Imagine walking into a 3-star Michelin-rated best-in-the-world restaurant, the kind of place where you plan international travel around reservations, the kind of place where the chef’s name is whispered in a kind of hushed awe by other chefs around the world.

Now imagine being seated and then insisting that the chef cook a specific dish in a specific way, because that’s what you’re used to eating, because you know what you like and what you want.

knife-1088529_640
I’ll just leave this here for no particular reason.

In this situation, one of three things is likely to happen:

  1. The chef will give you what you ask for, and your dining experience will be diminished because your request was granted.
  2. The chef will ask you to leave.
  3. The chef will instruct someone else to ask you to leave.[1]

Let’s step back from the culinary context of this imaginary scenario, and put it into the context of software development and BI.

Imagine a user emailing a developer or software team[2] and insisting that they need a feature developed that works in some specific way. “Just make it do this!” or maybe “It should be exactly like <legacy software feature> but <implemented in new software>!!”

I can’t really imagine the restaurant scene playing out – who would spend all that money on a meal just to get what they could get anywhere? But I don’t need to imagine the software scene playing out, because I’ve seen it day after day, month after month for decades, despite the fact that even trivial software customization can be more expensive than a world-class meal. I’ve also been on both sides of the conversation – and I probably will be again.

When you have a problem, you are the expert on the problem. You know it inside and out, because it’s your problem. You’ve probably tried to solve it – maybe you’ve tried multiple solutions before you asked for help. And while you were trying those ineffective solution approaches, you probably thought of what a “great” solution might look like.

So when you ask for help, you ask for the solution you thought of.

This is bad. Really bad.

“Give me this solution” or “give me this feature” is the worst thing to ask for. Because while you may be the expert on your problem, you’re not an expert on the solution. If you were, you wouldn’t be asking for help in the first place.

And to make matters worse, most of the people on the receiving end aren’t the IT equivalents of 3-star Michelin-rated chefs. They’re line cooks, and they give you what you asked for because they don’t know any better. And because the customer is always right, right?

Yeah, nah.

As a software professional, it’s your job to solve your customers’ problems, and to do so within constraints your customers probably know nothing about, and within an often-complex context your customers do not understand[3]. If you simply deliver what the customer asks for, you’ve missed the point, and missed an opportunity to truly solve the fundamental problem that needs to be solved.

If you’re a BI professional, every project and every feature request brings with it an opportunity. It’s the opportunity to ask questions.

Why do you need this?

When will you use it?

What are you doing today without the thing you’re asking for?

When will this be useful?

Who else will use it?[4]

As a software or BI professional, you’re the expert on the solution, just as your customer is the expert on the problem. You know where logic can be implemented, and the pros and cons of each option. You know where the right data will come from, and how it will need to be transformed. You know what’s a quick fix and what will require a lot of work – and might introduce undesirable side-effects or regressions in other parts of the solution.

With this expertise, you’re in the perfect position to ask the right questions to help you understand the problem that needs to be solved. You’re in the perfect position to take the answers to your questions and to turn them into what your customer really needs… which is often very different from what they’re asking for.

You don’t need to ask these questions every time. You may not even need to ask questions of your customers most of the time[5]. But if you’re asking these questions of yourself each time you’re beginning new work – and asking questions of your customers as necessary – the solutions you deliver will be better for it.

And when you find yourself on the requesting side (for example, when you find yourself typing into ideas.powerbi.com) you’re in the perfect position to provide information about the problem you need solved – not just the solution you think you need. Why not give it a try?

This is a complex topic. I started writing this post almost 100 years ago, way back in February 2020[6]. I have a lot more that I want to say, but instead of waiting another hundred years I’ll wrap up now and save more thoughts for another post or two.

If you’ve made it this far and you’re interested in more actual best practices, please read Lean Customer Development by Cindy Alvarez. This book is very accessible, and although it is targeted more at startups and commercial software teams it contains guidance and practices that can be invaluable for anyone who needs to deliver solutions to someone else’s problems.


 

[1] This seems like the most likely outcome to me.

[2] This could be a commercial software team or “the report guy” in your IT department. Imagine what works for you.

[3] If you’re interested in a fun and accessible look at how the Power BI team decides what features to build, check out this 2019 presentation from Power BI PM Will Thompson. It’s only indirectly related to this post, but it’s a candid look at some of the “often-complex context” in which Power BI is developed.

[4] Please don’t focus too much on these specific questions. They might be a good starting point, but they’re just what leaped to mind as I was typing, not a well-researched list of best practice questions or anything of the sort.

[5] If you’re a BI developer maintaining a Power BI application for your organization, you may have already realized that asking a ton of questions all the time may not be appreciated by the people paying your salary, so please use your own best judgment here.

[6] This probably explains why I so casually mentioned the idea of walking into a restaurant. I literally can’t remember the last time I was in a restaurant. Do restaurants actually exist? Did they ever?

Upcoming webcast: Unleashing your personal superpower

This damned pandemic[1] has been getting the best of me, but Talking About Mental Health is Important, and it’s more important now than ever. So when I learned that the Data Platform Women in Tech user group was hosting a free day-long online event focused on mental health and wellness, I knew I wanted to participate.

Today I am excited to announce that I will be presenting on “Unleashing your personal superpower” on Friday May 7th:

Building a successful career in tech is hard. Every day is a battle, and sometimes the barriers placed in your way seem insurmountable. Wouldn’t it all be easier if you had a superpower?

Maybe you do.

Greta Thunberg famously described her Asperger’s syndrome diagnosis as being a superpower: “I’m sometimes a bit different from the norm. And – given the right circumstances- being different is a superpower.” We can all learn something from Greta.

In this informal session, Microsoft program manager Matthew Roche will share his personal story – including the hard, painful parts he doesn’t usually talk about. Matthew will share his struggles with mental health, how he found his own superpower, how he tries to use it to make the world a better place… and how you might be able to do the same.

Please join the session, and join the conversation, because talking about mental health is important, and because the first step to finding your superpower is knowing where to look.

You can learn more about the event here, and you can sign up here.

I hope you’ll join me!


[1] Of course it’s more than just the damned pandemic, but when I first typed “this year” I realized this year has been going on for decades now, and I figured I would just roll with it because finding the perfect phrase wasn’t really important to the webcast announcement and anyway I expect things will probably suck indefinitely and isn’t this what run-on sentences in footnotes are for, anyway?

Deep thoughts on dataflows

As you may have noticed, life is complicated and keeps getting in the way of my plans to be a more active blogger and YouTuber[1]. I haven’t released a new dataflows video of my own in far too long[2], but my teammate Kasper is helping out by doing the hard work for me:

Last week I had an awesome conversation on “everything dataflows” with Kasper and the video is now available on his excellent Kasper On BI YouTube channel. In this hour-long video we talk about a lot of the “big picture” topics related to dataflows, including how dataflows fit into a larger data estate – and when you should use them, or avoid using them.

Please check it out, and let me know what you think!


[1] If this isn’t the name of a social media platform for potatoes, it should be.

[2] To add insult to the injury that is life in a global pandemic, my video editing PC died a few weeks ago. I now have it back up and running, but I lost all of my project templates and works in progress, which is likely to introduce more delays. FFS.

Being a PM at Microsoft – Part 2

Last year I published a blog post and video on Being a Program Manager at Microsoft, in which I shared some of my personal experience and advice based on my decade plus at Microsoft. I’ve received an incredible amount of feedback[1] on that 2020 post…

…and now it has an unexpected follow-up:

My teammate Kasper de Jonge has started a YouTube channel, and in addition to some great technical deep dive content, he has a series of interviews with members of the Power BI product team and the broader data platform community.

This week he’s published an interview with Power BI Group Program Manager Kim Manis, where they spend 45 minutes talking about being a PM in the Power BI team… and it’s fascinating.

In the interview Kim provides an insider’s perspective on topics ranging from how we decide what features to build, how we build and ship them, to what are you looking for in a PM candidate when you’re hiring.

I’ve been a PM at Microsoft for well over a decade, and have worked on Power BI for almost three years now… and I learned a lot from their conversation. If you’re reading this post I am very confident you’ll learn something too, and will enjoy the whole delightful conversation.


[1] I’ve come to believe that this video may have been the best thing I did in 2020. I’ve had dozens of conversations with people around the world who want to be program managers, which feels like a chance to make a difference in the world when the world is doing its best to make me feel powerless.

Will it fold? Look in Power Query Online!

Query folding is a key capability in Power Query. In short, query folding is the process by which the Power Query mashup engine takes the steps defined in the Power Query editor and translates them into processing to be performed by the data source from which the query is extracting data.

The use of query folding can produce significant performance improvements. Consider a query that includes a grouping step. Without query folding, all of the detailed data required must be extracted from the data source, and the aggregation performed by the mashup engine. With query folding, the mashup engine can add a GROUP BY clause[1] to the data source query it generates, and lets the data source perform the costly operation – only the aggregated data leaves the data source.

Because of the performance benefit that query folding provides, experienced query authors are typically very careful to ensure that their queries take advantage of the capabilities of their data sources, and that they fold as many operations as possible. But for less experienced query authors, telling what steps will fold and which will not has not always been simple…

Until now.

This week the Power Query team announced the introduction of query folding indicators in Power Query Online. If you’re authoring a dataflow in Power BI or Power Apps, you will now see visual indicators to let you know which steps will fold, and which will not.

Each step will include an icon[2] that shows how that step will be handled. The announcement blog post goes into more details (because connectivity and data access are complicated topics, and because of the huge diversity in the data sources to which Power Query can connect) but the short version of the story is that life just got simpler for people who build dataflows.

After the last 12 months anything that makes my life simpler is a good thing.


[1] Or whatever the syntax is for the data source being used.

[2] Or is it a logo?

Old videos, timeless advice

It’s 2021, which means my tech career turns 25[1] this year.

It me

Back in 1995 an awesome book was published: Dynamics of Software Development by Jim McCarthy. Jim was a director on the Visual C++ team at Microsoft back when that was a big deal[2]. The book is one of the first and best books I read as a manager of software development teams, and it has stood the test of time – I still have it on the bookshelf in my office[3].

Of course, I wasn’t managing a software development team in 1995. I discovered Jim’s wisdom in 1996 or 1997 when I was preparing to teach courses on the Microsoft Solutions Framework[4]. When I opened the box that the trainer kit came in, there was a CD-ROM with videos[5] of Jim McCarthy presenting to an internal Microsoft audience on his “rules of thumb” for reliably shipping great software. When I first watched them, it was eye-opening… almost life-changing. Although I did have a mentor at the time, I did not have a mentor like Jim.

The second time I watched the videos, it was with my whole team. Jim’s rules from these videos and his book became part of the team culture[6], and I’ve referred back to them many times over the decades.

Now you can too – the videos are available on YouTube:

Some of the rules may feel a bit dated if you’re currently shipping commercial software or cloud services[7] but even the dated ones are based on fundamental and timeless truths of humans and teams of humans.

I’m going to take some time this week to re-watch these videos, and to start off the new year with this voice from years gone by. If you build software and work with humans, maybe you should too.


[1] Damn, that is weird to type, since I’m sure I can’t even be 25 years old yet.

[2] It may also be a big deal today, but to me at least it doesn’t feel quite as much.

[3] Not that I’ve seen my office or my bookshelf in forever, so this assertion should be taken with a grain of 2020.

[4] Fun fact: if you remember MSF, you’re probably old too.

[5] Because in those days “video” and “internet” weren’t things that went together. This may or may not have been because the bits had to walk uphill both ways in the snow.

[6] To the extent the team could be said to have a shared culture. We were all very young, and very inexperienced, and were figuring things out as we went along.

[7] Back in 1995 CI/CD wasn’t part of the industry vocabulary, and I can count on zero hands the clients I worked with before joining Microsoft who had anything resembling a daily build.

Automatically refresh dataset when dataflow refresh completes – now with 100% less code!

Back in October I blogged about using PowerShell and/or the Power BI dataflows REST APIs to trigger a dataset refresh on dataflow refresh completion. This was a new capability and was well received… but lots of people wanted to do it without needing to write any code.

For those people, Christmas came early.

This week the Data Integration team at Microsoft announced the public preview of the dataflows connector for Power Automate.

This new Power Automate connector[1] will enable any authorized user to take action based on dataflow refresh completion, including:

  • Sending a notification – so you can keep additional users and groups informed
  • Trigger a dataflow or dataset refresh – so you don’t need to maintain separate refresh schedules or live with unnecessary processing delays
  • Trigger a Power Automate flow – so you can do pretty much whatever you want, because Power Automate has a ton of capabilities this blog post won’t even attempt to mention

You can also use the connector to refresh dataflows and monitor dataflow refresh status.

To make it even easier, the preview includes samples for these scenarios and more. You can read all about it and get started today in this post on the official Power BI Blog. What are you waiting for?


[1] Which uses the APIs mentioned in the earlier blog post.

Data Culture Presentation Resources

On Thursday, December 10th I joined the Glasgow Data User Group for their December festivities. Please don’t tell anyone[1] but I’ll be bringing the gift of data culture!

The session recording is now available on YouTube:

If you’re interested, you can also download my session slides here: Glasgow Data UG – Building a Data Culture with Power BI.


[1] I want it to be a surprise! Also this footnote makes even less sense now that the session is in the past…

A hat full of of dataflows knowledge

Life and work have been getting the best of me this month, and I haven’t found the time[1] to keep up on blogging and video now that my series on building a data culture has wrapped up. I’ve been watching the dataflows and Power Query teams releasing all sorts of exciting new capabilities, and realizing that I’m not going to be writing about them in a timely manner.

Thankfully Laura Graham-Brown is picking up the slack – and then some.

Laura is a Microsoft MVP whose “Hat Full of Data” blog has become one of my favorite morning reads, and whose YouTube channel seems to include all of the videos I’ve been thinking about making, but not actually finding the time to make them.

Like this one on the new Power Query Online diagram view that is now available in public preview for dataflows in Power BI:

If you’ve been waiting for new dataflows content, you should definitely head over to Laura’s blog today to check out the awesome work she’s been doing.

I hope to be writing more regularly in December after my work-related “crunch mode” has passed, but if 2020 has taught me anything[2] it’s that I have no idea what’s waiting around the corner. In the meantime, you should follow Laura, because she’s doing awesome work.


[1] Or the spare creative mental energies, which seem to be in sparser supply than spare minutes and hours.

[2] If 2020 has taught me anything, it’s that 2020 has taught me nothing.

“Why dataflows?” webcast recording now online

A lot of the questions I get about dataflows in Power BI boil down to the simplest[1] question: “Why dataflows?”

On Saturday November 7 I joined MVP Reid Havens for a YouTube live stream where we looked at this question and a bunch of other awesome dataflow questions from the 60+ folks who joined us.

The stream recording is now available for on-demand viewing. You should check it out!


[1] And therefore most difficult to answer concisely. That’s just how it goes.