Data Culture: Motivation and Encouragement

The last post in our ongoing series on building a data culture focused on the importance of community, and on ways organizations can create and promote successful communities around data. But while a community is where the data culture can grow, how can you motivate people to participate, to contribute, and to be part of that growth?

Business intelligence is more about business than it is about technology, and business is really about people. Despite this, many BI professionals focus their learning largely on the technology – not the people.

Do you remember the first time you were involved in a performance tuning and optimization effort? The learning process involved looking at the different parts of the tech stack, and in understanding what each part does, how it does it, and how it relates to all of the other parts. Only when you understood these “internals” could you start looking at applying your knowledge to optimizing a specific query or workload.

You need to know how a system works before you can make it work for you. This is true of human systems too.

This video[1] looks at motivation in the workplace, and how you can motivate the citizen analysts in your data culture to help it – and them – grow and thrive. If you think about these techniques as “performance tuning and optimization” for the human components in a data culture, you’re on the right track.

This image makes a lot more sense after you’ve watched the video – I promise

People are motivated by extrinsic motivators (doing something to get rewards) and intrinsic motivators (doing something because doing it makes them happy)[2], and while it’s important to understand both types of motivators, it’s the intrinsic motivators that are more likely to be interesting – and that’s where we spend the most time in the video.

When you’re done with the video, you probably want to take a moment to read this Psychology Today article, and maybe not stop there. People are complicated, and if you’re working to build a data culture, you need to understand how you can make people more likely to want to participate. Even with an engaged executive sponsor, it can be difficult to drive personal change.

In my personal experience, task identity and task significance are the biggest success factors when motivating people to contribute in a data culture. If someone knows that their work is a vital part of an important strategic effort, and if they know that their work makes other people’s lives better, they’re more likely to go the extra mile, and to willingly change their daily habits. That’s a big deal.


[1] If you’re not old enough to recognize the opening line in the video, please take a moment to appreciate how awesome commercials were in the 1980s.

[2] Yes, I’m oversimplifying.

Representation and visibility

tl;dr: We need more representation in tech, in part because career opportunities come from kicking ass where people can see you, and that comes from knowing that you belong, and that knowing comes from seeing people like you already belonging.

Image by Gerd Altmann from Pixabay

Representation is a complex topic. Being a straight, white, cisgender, American man, I don’t have a lot of personal experiences with the lack of representation. But I do have one, and it involves swords.

Back in 2004 a friend of mine sent me a copy of the first edition of The Swordsman’s Companion by Guy Windsor. At this  point I had no idea that people were recreating medieval martial arts and fighting with steel weapons[1], so I read through the book and eventually sold it at a yard sale. I was interesting, and I loved the idea of being able to do what the people in the book were doing, but deep inside I knew that people like me didn’t actually do things like that. I was wrong, of course, but at the time the totality of my life experience told me I was right, and I simply didn’t question this knowledge.[2] Ten years passed before this opportunity presented itself again – ten years in which I could have been studying, practicing, competing, improving.

Now take this almost-trivial example and apply it to your career. What if you had never seen someone like yourself in a job role? How would you know that this was something that you could do, that this was a path that was open for you to travel?

When I look back on my career in IT, I can see tipping points – places where everything changed, and my life was forever improved. All but one of them (we’ll come back to this one) happened because someone saw me kicking ass and said “I want you to come kick ass with me.”

  • In 1996 when Dave saw me excelling as an applications trainer and offered me the opportunity to become a technical trainer and MCT.
  • In 1997 when Jeff saw me teaching Windows NT networking and offered me a job with a much higher salary and responsibilities that included consulting and application development.
  • In 2003 when Carolyn offered me a full-time contract as I was leaving my then-collapsing employer and starting my own consultancy, and when I simply didn’t know how I would pay my mortgage in six months, or pay the hospital bills for my impending second child.
  • In 2005 when Corey saw me teaching a beta SQL Server 2005 course, and said “you need to come work at TechEd this year.”
  • In 2007 when I quoted Ted almost double my then-standard daily rate to get him to quit nagging me about working for him, and he instantly agreed and asked how soon I could start.
  • In 2008 when Ken and Dave told Shakil that I was the only person who could do the job he needed done, and he ended up offering me my first job at Microsoft.
  • In 2011 when Matt learned I was looking for a new role and said “we have an open PM position on the SSIS team – let me introduce you to the hiring manager so you can see if you want to apply.”
  • In 2017 when Kasper kept subtly mentioning how much he loved the team he was on, until I finally figured out he wanted me to join it.

This is my story, and I’m including names both to say “thank you” and to make it real – these are the people who enabled me to be who I am today, where I am today, doing what I love so much today. I’ve heard variations on this story from many of my colleagues and peers, but this one is mine.

Please don’t get me wrong – I don’t believe anyone was giving me any handouts. At every step along the way I was doing the work – I was working hard, pushing, trying, struggling to kick as much ass as I possibly could. I was really good, because I had studied and practiced and put in the long hours to learn and improve – but without these people seeing and recognizing my hard work, the opportunities simply would not have existed for me. At any of these tipping points, if I hadn’t been where these people could see me, the moment would have passed, and the opportunity would have been lost. Each opportunity was dependent on the opportunities that came before it – each one depended on my being where I was, being visible and being seen.

This brings me back to that very first tipping point. In 1996 I saw a classified ad[3] for a job teaching people how to use Microsoft Windows and Office, and I applied for it.

And this brings us back to representation.

I applied for that first job because I could see myself in the role – I knew that I could do it. I had no difficulty picturing my 20-something Christian white male self in that role, and neither did the folks doing the hiring.[4]

But what if I was female, or Black, or transgender? What if I looked at an industry that was overwhelmingly male and overwhelmingly white and overwhelmingly cis, and knew that I didn’t belong, because there was no one like me doing things like that? Would I have opened that very first door, on which every later door depended?

I can’t answer that, but looking at the still-white and still-male faces around me every day in tech, I have every reason to believe that many candidates would see this lack of representation and infer from it a lack of opportunity – and never take that first step. And for those who do take the first steps I only need to look around to see the barriers that the tech industry put in place for women and minorities, letting them know every day that they’re not welcome.

Back in July I shared a post that referenced an amazing webcast by UX Designer Jasmine Orange on “Designing for the Ten Percent.” In that post the link was buried in a footnote, so I want to call it out here explicitly – if you made it this far into my post, you really want to watch this webcast. Jasmine’s premise[5] is that by designing for underrepresented users/communities/audiences/people you end up making better products for everyone.

I believe the same is true of tech in general – by building organizations, teams, and cultures that are welcoming to people who have been traditionally excluded, we build organizations, teams, and cultures that are better and more welcoming for people who have never been excluded. Diverse teams are stronger, more resilient, more agile, and more productive. Diverse cultures thrive and grow where monocultures collapse and die.

Why do I care? Does any of this even affect me directly?

Yes, it does.

On one hand, being part of a diverse team means that I will be more likely to be part of a successful team, with the financial rewards that come with success. On the other hand, I feel more welcome and more at home as part of a diverse team.

Still, it’s not about me, is it?

If you’re considering a role in tech and you’re not sure if you should apply for a job, do it. Even if you don’t feel like you’re a perfect fit, or don’t believe you meet every single requirement in the job posting. The white guys are going to apply because they know they below. You belong too, even if you don’t know it yet.

If you’re coming into tech from an underrepresented minority, you not only belong, you’re vitally needed. You bring something that no one else can bring – your background, your experiences, your perspectives – all the lessons you learned the hard way that an all-male, all-white team might pay an expensive consultant to tell them about after they’ve failed enough times.

Not every employer will recognize this value, but that doesn’t mean it’s not there, or that it’s not real. Some employers will judge every candidate against the “tech bro ideal” and will try to make every hire fit into this mold.

If you’re an employer, don’t be this guy. If you’re an employer, recognize the strategic value of having a diverse team. And recognize that this diverse team won’t happen on its own. Support organizations like Black Girls Code, because this is where your hiring pipeline will come from. Value the diverse perspectives that are already represented on your team, and promote them. Give authority and power to people who will hire diverse candidates. Give authority and power to people who look like the people you’re trying to recruit.

And if you’re working in tech today, please be be the person who notices. When you see someone kicking ass – doing an amazing job and demonstrating talent and potential to do more – be the person who says something. Even if you can’t see yourself in that person. Even if they don’t look like you and the rest of your team.

Representation is important, because it helps more people know that they can open that first door. Visibility is important, because it provides the opportunity for change and growth – visibility is what opens the second door, and the third, and….

Ok…

I started off writing a completely different post, but this is the one that wanted to come out today. At some point in the next few weeks[6] I’ll follow up with a post on community and how community is a key technique for increasing your visibility, but this post has gotten long enough, and then some.

Thanks for sticking with me – I’d love to hear what you think.


[1] I learned in 2014 when I first saw this video and fell in love.

[2] Underlying this false knowledge was likely the experience of growing up rural and poor, and not having a lot of opportunities as a child, or many experiences as an adult. It wasn’t until 2005 when I first travelled to Europe for a Manowar concert that my mindset started to change, but that’s probably not relevant to this post.

[3] Classified ads are what we called Craigslist before it was LinkedIn. Something like that.

[4] At this point I feel compelled to mention my friend Kathy, who actually got the job I applied for. I didn’t get the job, so I ended up taking a job as a bank teller. When the training center had another opening a few weeks later they called me back to ask if I was still available, and I said yes. True story. Hi Kathy!

[5] More accurately, this is my interpretation of Jasmine’s premise.

[6] LOL @ me, and LOL @ you if you haven’t learned by now not to trust any predictions of future output on my part. We should both know better by now.

Thoughts on effective communication

Early in my career I worked with an incredible network engineer. This was back when networking was a bigger part of my job, and I learned a lot from him[1]. But he was not an effective communicator. Any time we were in a meeting and he was presenting a topic I cared about, I needed to stand up, get closer to the front of the room, and focus exclusively on what he was saying, forcing myself to pay attention.

Because he just shared facts.

Page after page, slide after slide, minute after agonizing hour of facts.

arrows-2980845_640

This was before I knew about the value of asking questions, so I took the burden on myself, because I cared. Most people who could have gotten value from his expertise simply tuned out, or fell asleep[2].

Although this colleague of days gone by will always stand out as the exemplar of ineffective communication, I still see people every day who communicate by sharing facts without context, and without taking their audience into consideration.

Here are a bunch of things that I know and now you know them too!

Yeah, nah. When communication is simply a dump of facts, an opportunity is lost.

If you find yourself falling into this pattern of communication, please pause and ask yourself these questions:

  1. Who is my audience?
  2. What is my goal for communicating with this audience?
  3. What does my audience care about in general – what motivates them?
  4. Why will my audience care about what I am telling them?
  5. How will my communication motivate my audience to take action to meet my goal?

Or, as a colleague summed it up nicely in a chat:

“Target audience and value proposition. FFS.”[3]

You may be looking at these questions and thinking “that sounds like a lot of work.” I’d like to respond to that excellent-but-false thought in two ways.

First, asking yourself these questions is simply a habit to develop. It will take a little time at the beginning, but over time it becomes second nature.[4]

Second, choosing to not ask yourself these questions is like choosing the low bidder when picking a home improvement contractor. You save a little up-front, but you end up needing to re-do the work more quickly, and no one is ever happy with the results.

These questions apply to any form of communication – email, presentations and conference talks, personal  conversations, you name it.

For lower-risk, lower-value communication like a chat with a coworker you might ask yourself these questions during a conversation, as a safety check to help you keep on track. For a high-risk, high-value communication like a business review or product pitch or large presentation, you might ask yourself these questions many times over weeks or months as you define and refine the narrative you want to drive.

For the past year or so, my great-grandboss was an amazing leader named Lorraine. One of the things I loved about working with Lorraine was how she structured large-audience emails. Any time she sent out a newsletter or other message that would reach hundreds of people, she consistently included these sections at the top of the mail:

2020-08-05-10-57-47-549--OUTLOOK

  1. Big bold header so you know instantly what you’re looking at
  2. tl;dr section so you can read in 2 or 3 sentences the most important things in the mail
  3. Read this if section so you know if you’re the target audience for the mail

Every time I see one of these emails I am impressed. Lorraine is a communication wizard, guru, ninja, and I am in awe of her effortless-seeming skills.[5]

But… these effortless results probably come from years of mindful practice. As with any skill, you only get the results if you put in the effort. Why not start today?

P.S. While this post was written and waiting to be published, someone mentioned to me the Minto Pyramid Principle. I’d heard of it before, but I never actually knew what it was called, and now that I do know I’ve done some additional reading on it. The Minto Pyramid Principle is a communication tool that take a similar approach to the “know your audience” approach I’ve presented above. Unlike my random blog post, the Minto Pyramid Principle has stood the test of time – it’s been around since the 70s, and there are lots of different resources including books and articles out there to support it with training and information. You should check it out too.


[1] 20+ years later I am still reminded regularly that 90% of communication failures are caused by problems in the physical layer. Please do not throw sausage pizza away.

[2] Yes, literally fell asleep in work meetings. It was a running joke.

[3] Full disclosure: I added the FFS, not him. But it fits.

[4] Think back to when you were first learning the Farfalla di Ferro, and how you needed to think about every cut, every step, and every transition – and how today you could do it with your eyes closed.

[5] Lorraine is no longer in my reporting chain after a recent re-org, and you can be certain that I sent her a “thank you” message during the transition period. Some things are too awesome to not acknowledge explicitly.

Data Culture: The Importance of Community

The last two videos  in our series on building a data culture covered different aspects of  how business and IT stakeholders can partner and collaborate to achieve the goals of the data culture. One video focused on the roles and responsibilities of each group, and one focused on the fact that you can’t treat all data as equal. Each of these videos builds on the series introduction, where we presented core concepts about cultures in general, and data culture in particular.

Today’s video takes a closer look at where much of that business/IT collaboration takes place – in a community.

Having a common community space – virtual, physical, or both – where your data culture can thrive is an important factor in determining success. In my work with global enterprise Power BI customers, when I hear about increasing usage and business value, I invariably hear about a vibrant, active community. When I hear about a central BI team or a business group that is struggling, and I ask about a community, I usually hear that this is something they want to do, but never seem to get around to prioritizing.

Community is important.[1]

woman-1594711

A successful data culture lets IT do what IT does well, and enables business to focus on solving their problems themselves… but sometimes folks on both sides of this partnership need help. Where do they find it, and who provides that help?

This is where the community comes in. A successful community brings together people with questions and people with the answer to these questions. A successful community recognizes and motivates people who share their knowledge, and encourages people to increase their own knowledge and to share it as well.

Unfortunately, many organizations overlook this vital aspect of the data culture. It’s not really something IT traditionally owns, and it’s not really something business can run on their own, and sometimes it falls through the cracks[2] because it’s not part of how organizations think about solving problems.

If you’re part of your organization’s journey to build and grow a data culture and you’re not making the progress you want, look more closely at how you’re running your community. If you look online you’ll find lots of resources that can give you inspiration and ideas, anything from community-building ideas for educators[3] to tips for creating a corporate community of practice.


[1] Really important. Really really.

[2] This is a pattern you will likely notice in other complex problem spaces as well: the most interesting challenges come not within a problem domain, but at the overlap or intersection of related problem domains. If you haven’t noticed it already, I suspect you’ll start to notice it now. That’s the value (or curse) of reading the footnotes.

[3] You may be surprised at how many of these tips are applicable to the workplace as well. Or you may not be surprised, since some workplaces feel a lot like middle school sometimes…

Manowar – Munich, 2007

tl;dr: this post has nothing to do with data or business intelligence, and there’s no reason to read it unless you want to learn how 2007-era Past Matthew stumbled his way through Bavaria. But “heavy metal” is listed in the banner of the blog, so I’d say this fits. You’ve been warned.

Over the past 30 years I’ve seen the heavy metal band Manowar almost 75 times, across the US and Europe, in Japan, in Israel, and above the arctic circle in Svalbard. Manowar are not the most prolific musicians, but their music is inspiring and motivating, and they put on a live concert like no one else.

My first international show was in 2005 – that was also the first time I’d ever visited a non-English-speaking country, and only the second time I’d ever left the US. That 2005 festival was a life-changing experience – it opened my eyes to experiences and possibilities that I had never previously imagined, and in 2007 I was ready to do it again.

When Manowar announced their 2007 spring tour, I bought tickets and made plans to travel with my wife and two friends to see the show in Munich. It was going to be our first vacation after we had kids, and it would be the first time seeing Manowar in concert for any of them. Family health problems caused those plans to fall apart, but at the last minute – less than a week before the show – I realized that I had enough frequent flyer miles to make the trip for free. I also realized that if I flew in the day of the show and flew back the next morning I would only need to miss two days of work. (This sounded like a good idea at the time.)

The rest, as they say, is history. In any event, the rest of the post was written in March 2007, so… ancient history.

IMG_4308
Before the concert

I can’t believe I’ve made it this far. As I type I’m sitting in a Boeing 767 on the tarmac in Atlanta, waiting for takeoff for the transatlantic flight to Munich, Germany. The show is tomorrow; I’ll be on the ground in Germany for just over 24 hours before flying back home.

This was supposed to be a different trip. I was supposed to have a nice, relaxing week with my wife and two friends. Fly into London, spend a few days there. Train to Dover. Ferry to Calais. Rent a car in Calais, and take a few days to drive across France and Germany, arriving in Munich on the day of the show. Sounds like a good idea, no? Of course, it was not meant to be. The best laid plans of mice and Manowar fans, I suppose. Plans change, and here I am, flying alone into the face of a whirlwind one-day invasion of Germany, hoping I’ll be able to sleep on the flight.

The day itself started off well. I was chatting with the woman at the airline counter (they’re always extra nice to us frequent flyers) and she asked me what I was doing in Munich on such a short trip. I told her I was going to see Manowar in concert, “They’re a heavy metal band,” I said. She broke into a huge smile – apparently Joey is her buddy. He’s always very nice when he flies (Manowar and I happen to share the same airline and airport, although I’ve never run into them on the road) and she helped the band when they were flying out to start the tour early last week. Sadly I could not turn my Manowar fandom into a free first class upgrade – when you’re flying for free in the first place, they tend to frown on free upgrades – but that was OK. I was going to see Manowar – what more could I need?

Things went downhill from there… My flight from New York was delayed half an hour. I had a 90-minute connection in Atlanta, so it would be tight, but I should make it. The half hour delay turned into 45 minutes, then an hour. No!!! Well, obviously I made it, or I would not be bothering to write up a tour report, right? We made up some time in the air, and although my flight to Munich was boarding by the time I made it to the gate, I did make the flight. The plane is pretty full, and no one is sitting next to me yet – do I have a chance at enough room to almost be comfortable?

Yes!! We’re in the air, flying north along the eastern seaboard, and I have room to spare. It’s not first class – as the flatulent young German woman sitting in front of me keeps reminding me as she tries to make re seat recline even further – but it will do. There are worse ways to cross the Atlantic, I’m sure.

Well, that wasn’t too bad. Waking up over The Netherlands, flying into morning, around an hour out of Munich has a certain appeal. The clouds are laid out below the plane like a giant layer of cotton, and the sun is rising with exaggerated speed ahead of us. And they’re bringing food and coffee. Well, something that they call coffee. It’s not great, but it will have to do until we get on the ground. I hope there’s decent coffee in the Munich airport…

…but if there is, I can’t locate it.

And pissing me off even more is my mobile phone. It’s a quad-band phone that should work everywhere in the world, and I went out of my way to make sure that it would work in Germany. Weeks before the trip I talked to my service provider: “Oh, yes sir, you won’t have any problems. You’ll have great service in Germany.” Right. Replace the word “problems” with “coverage” and the word “service” with “headaches” and you’ll be much closer to the truth. I’d made plans with people from six different countries to stay in touch via text messages while I was on the ground. Now it looks like I’ll just have to wing it, in a city I’ve never visited, in a country where I don’t speak the language, trying to meet with people who can’t get in touch with me, and most of whom I’ve met only once before, at the Earthshaker fest in 2005. Oh boy!

At the airport train station I met up with Alex, a German Manowar fan I met back in 2005. He was travelling to Munich for the show as well, with two other German friends. Alex had also caught the show in Berlin the night before, and was planning on catching two more shows before the German tour was over. I’m a little bit jealous, but I doubt I’ll be moving to Germany any time soon, so I should not complain. The four of us rode the train together from the airport to the central Munich train station, and had breakfast and coffee together in downtown Munich. We told stories about past Manowar shows and the trials and travails involved in getting to them sometimes, and paid way too much for coffee.

After breakfast, Alex and his friends went on to the show venue, and I went out into Munich to kill a few hours. My Dutch friend Joyce was arriving on a 2:00 train, and she and I were planning to get our tickets and head over to the hall together. Munich was beautiful, but the main square was pretty much overrun with tourists, which was a bit annoying. Still, I didn’t have any trouble keeping busy until 2:00. I found still water (hurrah!!) and some decent food, and watched people.

One thing that really surprised me was how many people were smoking, both indoors and out. I’d forgotten that Germany was so permissive about indoor smoking – it’s been so many years since smoking indoors has been legal in the States that I just never think about it. But here, in the airports, train stations, stores and restaurants, everyone seems to be puffing on a cigarette. The upside to the wait, however, is that I found still water. I picked up two 1.5 liter bottles at a grocery store near the train station and have them ready for the show. A small victory, but a victory nonetheless.

But where the fuck is Joyce? 2:00 has come and gone. I’ve been waiting in the train station since 11:00 or so. For the last hour I’ve been standing where all of the escalators come up from the lower levels where the trains arrive. And there is no sign of her. Since she is going to be picking up the tickets at the box office, she and I need to get together at some point or this whole trip has been for naught. And where the fuck is David, for that matter? He was supposed to arrive on a 3:00 train, and that time has also passed. Of course, I’ve never met David in person, so it may well be that we passed without even knowing. I should just send them text messages to find out where they are, right? Right.

So by a little after 3:00 it’s become apparent that there’s nothing to gain by waiting in the train station, other than sore feet. I take another train to the Olympiapark. The show is going to take place is at the Olympiahalle, a big concert and sporting venue in the Olympiapark, and my hotel is nearby. Although I won’t be sleeping there, it seemed like money well spent to have a secure place to leave my things and maybe take a shower after the show. I would feel very bad for anyone having to sit down for ten hours next to someone who had spent four hours or more in the crowd at a Manowar show. So I stop off at the hotel, check in, drop off my bag, pack a small sack with my water and some snacks, and take a taxi back to the Olympiahalle.

When I arrive at the hall, it’s still over two hours before the doors are scheduled to open, and there are only a few dozen people there. The first group I meet includes my friend Alex. He and the friends with whom I had coffee this morning were standing at the head of a line with a bunch of their other German Manowar friends. It’s amazing how many faces were familiar from Earthshaker, and how many people remembered me as well. Everyone seemed amazed that I would travel from New York to see Manowar play in Germany, and not once, but twice. I’ll bet that there are people in Germany who paid more (in time, money and pain – however you want to look at it) to be there than I did, but I guess the miles you travel adds to momentary celebrity. In any event, it’s always nice to have people be glad to see you.

On the other side of the hall, waiting in another line (yes, to hear the battle cry) was a group of Italian Manowar friends, including my friend Nicola. This guy is amazing. He takes tons of photographs under the name “Thundersteel” and has a great attitude about everything. To give you an idea of how great this guy is, when Gods of War was released, he bought me a copy of the special edition CD and mailed it to me – from Italy. All this because he knew that it would not be available for sale in the US until early April. What an amazing guy. He introduced me to a bunch of his friends as well – again, many of them were familiar from Earthshaker, but there were a few “Manowar virgins” as well, although I’ve met a few of them online over the years. Everyone was excited about the impending show.

In addition to being excited, I was also getting a bit worried. I had friends to meet, tickets to pick up and no success to date in doing either. But there was one place remaining where I had not yet looked: The Manowar show truck. It was parked on the far side of the hall, so I made my way over there. There were a bunch of fans hanging out near it, but basically it was just a big tractor trailer truck painted with the King of Kings artwork. Very cool, but also very closed. There were some familiar people unloading boxes of t-shirts and other merchandise from the truck, but it wasn’t being used for anything other than transportation and storage that I could tell. And of course, there was no Joyce and no David. And the clock was ticking…

Having seen all that there was to see at the Manowar truck, I walked back around to the front of the Olympiahalle to hang out with Alex and his group of friends. But no sooner had I arrived than whom did I see walking toward the hall? Joyce, her friend Hans and our Norwegian friend Erman who was such a great guy at Earthshaker. He wasn’t wearing his signature wool, leather and metal outfit, but he was still ready to kick ass. After we said hi and exchanged stories (Joyce and Hans had apparently also waited in the train station until 3:00, looking for me – what was up with that?) we all headed over to the Manowar truck together. By the time we were done looking at the truck once more, the beer garden and food stand on that side of the hall had opened, and everyone (well, everyone but me – I hate beer even when I’m in Bavaria) had to settle down for a beer or three. And who did we find waiting near the beer? Our English friend David, who came up and greeted us. Finally! Everyone who needed to be together WAS together and everyone had his own ticket. What a relief. Although I did not have any beer, I did have a giant pretzel – thanks, Hans! – and couldn’t finish the whole thing. It was bigger than my head.

After shooting the shit for a while – talking about Manowar and concerts and kids and more – we all headed together back to the front of the hall. We got in line behind Alex and his group, and none to soon. Shortly after we got situated, large groups of fans began arriving. At this point it was around an hour before the doors were scheduled to open, and people were arriving in serious numbers. And among those numbers was my Austrian friend Mark. Mark and I had talked earlier about the possibility of his making it to the show in Munich, but I’d thought that he was not going to be able to make it. I was delighted to see him – and rightfully so, as later events were to show.

As we waited, the sun was setting. And with the setting of the sun, the temperature started to drop. It was snowing when I landed in Munich in the early morning, and it has been chilly all day. As the sun went down, it was getting damned cold. I’d left my jacket and over-shirt at the hotel, knowing how hot it was going to be once the show started. At least the cold weather would keep my hard-earned water nice and cool. I had only drunk a little from one of the bottles. I knew how much I would need water once I got down into the crowd – dehydration was what undid me at Earthshaker, and I was determined not to make the same mistakes again. I had water, I had snacks, I was all set.

Or so I’d thought. When the doors opened (nearly half an hour late) the first thing the security people did was confiscate my water. Bastards! I suppose it makes sense, but if you’re not going to sell it in the hall (and no, of course there was no still water for sale anywhere in the hall) then why keep people from bringing in their own? God damn it.

And this disappointment was quickly followed by another. The security goons were not letting anyone into any areas other than the ones printed on their tickets. And although I had a good seat (and for the price I paid for the damned ticket, it had better be good) it was still only that: a seat. I had no desire to sit down and watch a Manowar concert from the comfort of a chair. But what to do?

I decided to go spend some money. I went up to the top floor of the Olympiahalle and found the merchandise stand. I’m not normally one to buy crap at concerts – especially Manowar concerts, where the t-shirts tend to have naked women and guitar-wielding demons on them – but this time I made an exception. I got a few “Sign of the Hammer” shirts, some patches, some Manowar ear plugs (how could I resist?) and two toy Manowar show trucks for the kids. What fun!

I also remembered that Manowar were giving away “International Road Warrior” t-shirts to fans who came to a show from a different country. These were being distributed v e r y s l o w l y at another table separate from the main merchandise stand. I went over there and resigned myself to waiting in line for a while. (Not that I had to worry about anyone stealing my assigned seat, right?) The system that the one person at the desk was using to distribute these free shirts went a little bit like this.

  • Equipment: One black sharpie marker, one ball point pen, one stack of white paper.
  • Organization: One piece of paper per country. The name of the country is written at the top of the paper, in bold black sharpie. The names of fans from said country are written below, in pen.
  • Process: When a fan arrives and shows his passport, the t-shirt guy locates the appropriate page, or allocates a new one if no fan from that country has yet gotten a shirt. If there are existing fans from that country listed, he cross-references the current fan’s ID to ensure that he has not already gotten a shirt. If the fan is not a return customer, then the t-shirt guy writes down his name and ID number, asks what size t-shirt is desired, and hands over the shirt. Next customer, please!

As you can probably imagine, this was taking FOREVER!! I was fortunate in that I arrived relatively early, but it still took me 10 or 15 minutes in line. By the time I had my shirt (which is wonderful, by the way!) the line was five times as large as when I’d arrived. The database and software architectural consultant in me was screaming “there’s a better way to do this” and thinking of simple applications that would dramatically speed things up at future shows. I’m a geek, I know…

But waiting in line was interesting too. There were lots of people from Austria and Italy, but also people from Slovenia, Great Britain and right in front of me was a guy from Australia! People thought I’d come a long way to see Manowar play, but this Aussie had come at least twice as far as I. And as I was walking away with my t-shirt, I saw my Austrian friend Mark. If there was a Nobel prize for Metal, this guy would win it. Why, you ask? Because he said to me “Wait here – I think I have a way to get you down to the show floor.” Even now, 4,300 miles and 20 hours later, I get chills as I type this. Although I’m sure that I would have enjoyed the show from a distance, I was bitterly disappointed that I could not get up front for the show. And here was my redemption, my salvation.

And five minutes later, there I was, four or five rows back from the front, watching HolyHell start the show. I was the happiest man in Germany that night.

HolyHell put on a great show. They played everything from their new single, plus a song called “Eclipse” and a cover of Yngwie Malmsteen’s “Rising Force.” It was the same basic set they’d played back in 2005 – with the exception of one or two new original songs and the omission of a Rhapsody cover – but they were in much better form. Maria sounded (and looked) great, Rhino pounded his drums, Tom and Joe shredded the hell out of their guitars, Jay mastered his bass and Francisco was a wizard with his keyboards. My one complaint about HolyHell’s act back in 2005 was that they didn’t seem to be a single unit yet. Everyone did their own thing well, but they didn’t have the chemistry that you want to see on stage. They got better each time I saw them in 2005, but even at Earthshaker they weren’t quite there yet. But tonight they were on. Not only did each individual musician kick ass, the band as a whole really seemed to be a single cohesive unit, and that made a big difference. I sang along to every song, and at the end of the night, back at my hotel, I found that it was HolyHell going through my head, not Manowar. Interesting…

And of course, Eric’s duet with Maria on “Phantom of the Opera” was the highlight of the show. I don’t think anyone was surprised like they were back in 2005, but it worked wonderfully nonetheless. Their voices were so clear and so strong and it was so obvious that they were having a great time on stage together. If you’ve ever seen Manowar live you know the look that Eric gets on his face when he’s having a lot of fun, and it was there during this entire song. I swear, Eric should record a side project of Andrew Lloyd Weber cover songs. Weber wrote so many great musicals, and many of his songs are written to really push the singer’s capabilities, to light up only when being sung by the best. And who could be better than Eric Adams? Following Eric’s duet with Sarah Brightman (Andrew Lloyd Weber’s ex-wife, and the female singer for whom “Phantom of the Opera” was originally written) would it really be that big of a stretch? I’d buy it!

Sadly, the HolyHell set was quite short. I realize that they’re the first act of three, and don’t even have a full album out yet, but I’d much rather watch (and listen to) them than Rhapsody. Of Fire. Oh boy.

I suppose that Rhapsody put on a good show, but I just could not get into it. I own most of their albums, and really like a few of their songs. I even think that they’re much better live than in the studio. But I just could not get into their show in Munich. It was pretty much the same show they’d done in 2005, and to me all of the songs sounded basically the same to me. I mainly used their set as an opportunity to rest for Manowar, conserve my strength, and gradually move closer and closer to the stage. By the end of Rhapsody’s act there were only two people between me and the security barrier.

And my Italian brother Nicola was beside me! He and a few friends had gotten down on the floor through some subterfuge (probably similar to my own) and were ready to experience Manowar up close and personal.

The intermission between Rhapsody and Manowar was incredibly short, by Manowar standards anyway. They unveiled more amps, Scott’s Drums of Doom, a giant video screen behind the stage and a giant Viking longboat below the video screen. The longboat is obviously a set piece and not a real boat, but it was pretty well done regardless. The press of the crowd became intense as everyone tried to get closer to the stage.

And then the lights went down. And the music swelled. And the announcement came, as Orson Welles’s voice filled the hall. “Ladies and gentlemen, from the United states of America – All hail Manowar!”

And all hell broke loose.

Which, of course, was very much to be expected.

The setlist for the show was pretty much the same as the Pardubice show earlier in the tour. I’m really grateful to Manowar for posting the a photo of the setlist on their blog, because I would never have remembered what songs were played or in what order. Instead of listing each song, I’ll instead focus on my impressions from the pit.

Overall: The show was amazing. Although there were a few places in a few songs where Joey’s bass seemed distorted or obscured, it only happened two or three times, and never lasted for more than a second or two. The sound was just great. The band seemed motivated, inspired, energized and simply delighted to be on stage in Munich. In short, they were simply ON for the show.

The Setlist: It was the best of times; It was the worst of times. Or something like that. Everything on the setlist was great. There were some songs that I really would have liked to hear (“Hail and Kill” and “Gates of Valhalla” leap to mind) but by the end of the show I was practically begging for it to end. I don’t know if I could have taken another song or two without dying or getting carried away.

Volume: For the “main” set, the volume was perfect. It was loud enough to feel every pound of the drums both through the air and through the floor, but not loud enough to hurt. When the crowd was really into a song you could hear them clearly around you, but you could always hear Eric singing as well. It was a great balance and whoever was responsible for it (Jeff Hair?) should be commended. When the “Gods of War” setlist began, however, it seemed like the volume jumped a good 5 decibels. It went from being just right to my being “too old,” because for me it was just too loud. I had earplugs in my pocket – I didn’t need them for the main set – but there was no way for me to get my hands into the pocket to get them out, so I just had to suffer. My ears stopped ringing by the time I got to the airport, so it wasn’t as bad as I’d feared, but I don’t think I’ll be cranking up any music for weeks to come.

The Crowd: This was the first time I attended a regular concert in Germany, so I don’t know if the crowd was typical or not, but the people around me up front were totally and overwhelmingly into the show. They sang along to the standard songs that Manowar plays on every tour. They sang along to the classic songs that Manowar has not played live in many years. They sang along to the brand new songs that have never been played live before this tour. Fists truly filled the air. It was hot, sweaty, smelly, smoky and packed as tight as I’ve seen any pit since Manowar played B.B. King’s in NYC in 2005. There was a lot of attrition – I saw at least 10 people get helped/pulled out of the crowd by security, one of whom was carried off in a stretcher. There were just so many people packed into too little a space, and people were getting crushed. There was also fierce competition for space up front. I reached the security barricade during “Each Dawn I Die” and managed to stay there for the rest of the show, but it was a constant battle to stay there.

Highlights: Seeing “Holy War” performed live was worth the trip to Germany in and of itself. I’ve never seen it live before, although (as I’ve mentioned in several forums online over the years) I spent the first half of my first ever Manowar show (at the start of the Triumph of Steel tour) screaming “Holy War!” at the top of my lungs until Eric actually paused and told me they weren’t going to play it that night. And the most amazing thing was that when Joey started playing it, I didn’t  know what song it was. The bass-heavy opening sounded so different from the album version – lacking as it was the segue from “Drums of Doom” – and I turned to Nicola beside me to ask what it was before it struck me. Even though I knew they were going to play it, it still took me completely by surprise. And it was so far beyond my hopes and expectations it’s hard for me to believe how great it was. The song performed live had such a different sound and feel to it from the studio version, but it was still completely true to the heart of the original. It just had more power, more strength, and in essence more holy war.

The other older songs were great too. I’ve never been much of a fan of “Gloves of Metal,” but the crowd loved it, and it was hard to not enjoy the journey back in time. The other older songs, especially “Mountains,” “The Oath” and “Secrets of Steel” were much closer to my heart. I’d never dreamed of hearing “Mountains” live, but it had all of the majesty, clarity, serenity (if I can use this adjective to describe a Manowar song) and power of the original, and then some.

The Vikings: With the thunderous conclusion of “Black Wind, Fire and Steel” the lights came back up, and the stage crew got to work getting ready for the second set, uncovering pyrotechnics and flame pots on the stage. They also hung shields on the side of the longboat, and the shields were decorated with the four band members’ symbols. I thought that was a nice touch. After just a few minutes (the merest blink of the eye if you’re accustomed to waiting for hours for Manowar stages to be set up) the lights went down again and the tape began to roll.

That’s right – playback. The second set was the final “real” tracks from the “Gods of War” CD – everything from “The Blood of Odin” through “Hymn of the Immortal Warriors.” For “The Blood of Odin” they played back the studio narration, instead of having Joey recite it live on stage. They didn’t “pull a Rhapsody” however – instead of simply rolling the tape while lights flash and smoke machines pump out smoke, Manowar brought out the Jomsvikings warriors both in video and in person. They had filmed short video segments to go along with “The Blood of Odin” and “Glory, Majesty, Unity” and more, acting out Joey’s interpretations of the songs. The films were very well done, so I suspect Neil Johnson was involved in their production. For some parts of some songs as well, the Vikings were also live on stage – either up above near the longboat or down below fighting next to the band members – with sword and shield in hand.

Now let me add a disclaimer: I fenced competitively when I was in college, and fought some of the best fencers in the world. (Yes, they kicked my ass, but that’s neither here nor there.) I’ve also done enough other forms of sword fighting, both for theater and for competition, that I have a very critical eye when I see things like this performance. I didn’t really care for their stage show during the Manowar Fanconvention in Geiselwind 2005, but they did a decent job on stage in Munich. It looked like a lot of work and planning went into their performance, and unlike in Geiselwind the “fighting” was only a small portion of what was going on. So the Vikings, to me, were good but not great, although they were probably the least of the available evils.

Why do I say this? First and foremost, I was delighted by the inclusion of a significant “Gods of War” set at the concert. I love the album, and the parts of it that they played are some of the best parts. And since it is a narrative slice of a concept album, it really needs to be played end to end. (Can you imagine Queensryche not playing “Suite Sister Mary” during Operation: Livecrime because it was too long?) And that means including the narrative interludes. And I can’t think of any way to perform songs like “The Blood of Odin” and “Glory, Majesty, Unity” live that would be better than what they did. The video told/enhanced the story, the Vikings made it real, and the band played along whenever it made sense.

To be completely honest, if I were given the choice between a “classic” Manowar live set and a “Gods of War” theatrical set, I’d choose the former. But Manowar didn’t make me choose – they gave us the best of both worlds, and it totally worked.

The show ended with an amazing rendition of “Hymn of the Immortal Warriors” that was a perfect ending to the night. I can’t describe the emotions I felt as the song (and the show) ended. Not only was it a perfectly delivered version of one of my favorite new songs, but it also marked the end of the battle I’d been fighting for the last three hours or more – the battle to reach the front row and to stay there. It seemed to me that Eric met my eye time and time again during this song, as I screamed along with him to its very end. But, of course, I was awfully punchy by that point.

Then the lights came up, “The Crown and the Ring” started playing, and the show was nothing more than a sweet memory. As I stood there, completely drained and exhausted from my ordeal, I noticed one of the cameramen (the small guy with the shaved head – I always call him “Handycam” but I really should learn his real name) gesturing wildly to one of the photographers to move out of his way – apparently Nicola and I made a good shot, because he kept the camera on us for the rest of the song. If they end up making a DVD of this show, I should be all over it. I saw the cameras on me time and again. I guess that’s just one more example of why being in the front row is so great: to the victor go the spoils.

MunichItaly_640
After the concert

And then it was time for me to go. Although there was some water shared with the crowd during the show (a drop here and there, not nearly enough) I was seriously dehydrated and needed fluids immediately. I said goodbye to my Italian friends, stumbled up the stairs and staggered to the nearest refreshment stand. “Do you have any still water?” “Uh… No.” I desperately look around for something neither alcoholic nor carbonated. “I’d like an orange juice please.” The guy pours me a pint glass of orange juice. By the time he’s turned to the cash register, I’ve drained the glass. “Could you make that two, please?” He pours me another. It, too, disappears. Sadly, he only had enough juice on hand for three glasses. But it was enough to keep me from dying, and that is probably what matters.

At this point I had no idea what time it was. My only timekeeping devices were my phone and my iPod, and both were back at my hotel. And that is exactly where I needed to go. There was a change of clean clothes waiting for me there, and even more importantly, there was a giant bathtub just waiting for me to fill it with scalding water and soak my bruised, battered and sore legs before heading back to the airport. My hotel was right around the corner from the Olympiahalle, right next to the train station, so how hard could it be to find it, right?

Right?

I was about to find out. After the sweltering heat of the crowd, the outdoor air was bitterly cold. It was probably around 35 degrees Fahrenheit, give or take. It wasn’t snowing yet, but it wasn’t too far from it. I left the Olympiahalle through a main entrance (which I did not realize at the time was not the same main entrance as the one I used to enter) and noticed a few things in rapid succession. First, there was a large crowd of people all walking off in the same direction. Second, there was a big sign pointing in the direction they were going, with both “Taxi” and “U” (as in “U Bahn” or “subway”) on it. Finally, I realized I had no idea where I was or where I was going, so stunned and dazed was I following the journey to Valhalla I’d just endured. So I figured that the wisest course would be to follow the crowd and the signs. In the best case I‘d end up at the train station and I could walk to my hotel from there in a minute or two. In the worst case I’d hail a taxi and pay a few Euro for the trip to the hotel. It sounds good, right? Well, it did at the time…

Around five minutes later, the crowd had almost completely thinned out. There had been no signs after that first one, and I was on a long path winding off into the distance. I was covered with frozen sweat; the cool still felt good, but I had a feeling that it wasn’t going to feel good for too much longer. Then, up ahead of me, I heard two people talking in English, and they were both wearing “International Road Warrior” t-shirts. And they mentioned something about the U Bahn. I took this as a good sign. “Hey, are you guys going to the train station too?” “Yeah.” “Do you have any idea where it is?” “Um…” “Mind if I follow you, and we can be lost together?” “Sure.” The two guys were in town for the show. They’d flown down from Dublin and were trying to catch a train back to central Munich to the hostel where they were staying for the show. They were both 21 years old, it was the first Manowar show either one had attended. They both seemed amazed that I’d flown to Munich just for the day, just to see Manowar play, and that it wasn’t the first time. But now that they’d seen the spectacle of might that is Manowar on stage, I think they understood.

But that didn’t get us any closer to the train station or to my hotel. We found more “U” signs, but they had no relationship to the actual location of any U Bahn station in this reality. We followed them for what seemed like a long time, to no avail. We tried navigating by the giant tower (nicknamed “Joey’s Dick” by my Dutch friend Joyce earlier in the day) near the Olympiahalle, as one of the Irish boys had marked it as a landmark on their walk to the show earlier in the day. All of this was to no avail – we just seemed to be getting more and more lost. Finally I flagged a taxi, and we rode together back to the train station, and I rode back to the hotel from there. I learned later on that we’d walked for a little over an hour, and had gone over two miles from the hall – but in the wrong direction! Still, it all ended out well, and it was a great experience.

One high point of The Lost Walk was realizing how Manowar is a force that crosses cultural boundaries. I’d met and bonded with people from over 10 countries over the course of one day, and these two young men were another example of the type of person that follows Manowar: strong, self-confident, friendly, and empowered with a great sense of humor. The other high point was that they both thought I was in my 20s. When I told them I was pushing 40 (we got on the subject when I was telling about my visit to Dublin with my wife back in 2000 before we had kids) they wouldn’t believe it. I guess Metal not only makes you strong, it also keeps you young.

So around 1:30 I finally made it back to my hotel. To coin a phrase from “Blow Your Speakers” I only had one thing on my mind: taking a hot bath. My legs were KILLING ME after a day spent standing up waiting on hard tile floors in the train station and then fighting for position in the crowd during the show. And the two-mile hike after the show hadn’t helped either. When I’d checked in earlier in the day I’d noted how the huge size of the tub. It was long enough for me to stretch out and deep enough for me to completely submerge in – I’m not a small guy, so that’s a rare thing in a hotel bathroom. So I take a shower, get clean and start to get warm, and then plug the drain and start filling up the tub. I sit down and wait for the scalding water to rise and cover my aching legs. And I wait, and wait, and wait… And after a little investigation learn that the drain to the tub doesn’t actually seal. I’d been looking forward to this bath for hours and hours. I felt deeply and profoundly betrayed…

Twenty minutes later I was clothed, packed, checked out of the hotel and waiting for my taxi. I took the taxi to the central Munich train station, from there to take the S8 train back to the airport. The ride through Munich in the early morning hours was beautiful; snow was falling and everything was quiet and peaceful and slightly alien to my American eyes. We drove past a handful of giant billboards advertising the Manowar show; I felt filled with deep sense of accomplishment at the day’s events. Despite all the world had thrown in my way – both today and over the past few months – I had persevered.

When the taxi driver dropped me off at the train station, I went inside and had a series of revelations. Of course, I was very tired at this point, having been awake for the last 24 hours, so this series of revelations came very slowly. More like rumbles of thunder than flashes of lightning. It went a little like this:

Revelation 1: This looks like a completely different building than the train station where I arrived. Am I at the wrong place? No, the sign says “Hauptbahnhof” so that’s right…

Revelation 2: This place is huge! Those trains sitting there (it was around 3:00 AM at this point, and no trains were running) are much bigger than the trains I’ve been riding.

Revelation 3: Oh, THAT’S why I never saw Joyce and Hans yesterday!

That’s right. I’d come into Munich on a local train from the airport, and was waiting around in that part of the train station, underground. Joyce and Hans were coming in on a regional train from The Netherlands, and were coming in to a completely different part of the station that I didn’t even realize existed. Sad, but true.

So 20 minutes later and I’m riding the train out to the airport, around a 45 minute ride. There were quite a few Manowar fans back at the train station, but none that I saw going back to the airport to catch a flight home. Perhaps this meant that none were flying, but perhaps it meant that the rest of the long-distance warriors were a lot smarter than I, and decided to sleep before heading home. I was planning to sleep on the plane back to America, of course. By the time that I was in the air I would have been awake for 28 hour straight, with lots of walking and a Manowar concert thrown in for good measure. What could keep me awake?

Coach. Coach class could keep anyone awake, at least when combined with my neighbor. I was sitting next to a middle-aged German woman wearing too much perfume, who kept getting up to go talk to her husband who was sitting 10 rows behind us. Why she couldn’t bother the woman sitting on the other side of her – the one who WAS NOT SLEEPING, I might add – is a mystery to me. And just to make matters worse, this woman smelled like she had eaten a bushel of rotten fruit, and it wasn’t agreeing with her digestion. She had the most horrendously pungent gas that I have ever smelled; I never thought that a human being could make such a smell. So I did my best to smile and ignore her for the 10 hour flight from Munich to Atlanta. I caught a few hours sleep in bits and pieces on the way, but no real rest to speak of.

And now, after an interminable layover in Atlanta, I’m on the two-hour flight back home. The kids will be asleep by the time I arrive, but I suppose that’s for the best – this way I can give them their presents and tell them about the trip and the concert when they’re awake in the new day, instead of when they’re tired and whiny at bedtime. There’s always a silver lining, and as we all know, for every day that stings, two better days it brings.

So if you’ve made it all the way to the bottom of this novella (I just looked at the page count and even surprised myself) you’re probably wondering what just happened. So please let me summarize. There are two things that I’m taking away from this wonderful ordeal. First and foremost, if you are a Manowar fan you should do anything possible to see Manowar play in Germany on this tour. They are in the finest form possible, and the show they’re putting on is second to none. Second, and perhaps even more important, is that Manowar fans are the best people in the world. I met friends – people I hardly know, but whom I would be proud to call Brother and Sister – from eight countries, and each one did everything in their power to make my trip the best it could be. Mark, Joyce, Nicola, Alex, Erman, David – if you are ever in my part of New York, you’d better look me up.

Power BI dataflows PowerShell scripts on GitHub

Last week I shared a post highlighting a common pattern for making API data available through dataflows in Power BI, and included a few diagrams to show how a customer was implementing this pattern.

In the post I mentioned that I was simplifying things a bunch to only focus on the core pattern. One of the things I didn’t mention is that the diagrams I shared were just one piece of the puzzle. Another part was the need to define dataflows in one workspace, and then use those as a template for creating other dataflows in bulk.

This is simple enough to do via the Power BI portal for an individual dataflow, but if you need to do it for every dataflow in a workspace, you might need a little more power – PowerShell, to be specific.

The rest of this post is not from me – it’s from the dataflows engineering team. it describes a set of PowerShell scripts they’ve published on GitHub, and which address this specific problem. The rest is pretty self-explanatory, so I’ll just mention that these are unsupported scripts presented as-is, and I’ll let the rest speak for itself.

nautilus-1029360_1920

Microsoft Power BI dataflows samples

The document below describes the various PowerShell scripts available for Power BI dataflows. These rely on the Power BI public REST APIs and the Power BI PowerShell modules.

Power BI Dataflow PowerShell scripts

Below is a table of the various Power BI PowerShell modules found in this repository.

Description Module Name Download
Export all dataflows from a workspace ExportWorkspace.ps1 GitHub Location
Imports all dataflows into a workspace ImportWorkspace.ps1 GitHub Location
Imports a single dataflow ImportModel.ps1 GitHub Location

For more information on Powershell support for Power BI, please visit powerbi-powershell on GitHub

Supported environments and PowerShell versions

  • Windows PowerShell v3.0 and up with .NET 4.7.1 or above.
  • PowerShell Core (v6) and up on any OS platform supported by PowerShell Core.

Installation

  1. The scripts depend on the MicrosoftPowerBIMgmt module which can be installed as follows:
Install-Module -Name MicrosoftPowerBIMgmt

If you have an earlier version, you can update to the latest version by running:

Update-Module -Name MicrosoftPowerBIMgmt
  1. Download all the scripts from the GitHub Location into a local folder.
  2. Unblock the script by right click on the files and select “Unblock” after you download. Otherwise you might get a warning when you run the script.

Uninstall

If you want to uninstall all the Power BI PowerShell cmdlets, run the following in an elevated PowerShell session:

Get-Module MicrosoftPowerBIMgmt* -ListAvailable | Uninstall-Module -Force

Usage

The APIs below supports two optional parameters:

  • -Environment: A flag to indicate specific Power BI environments to log in to (Public, Germany, USGov, China, USGovHigh, USGovMil). Default is Public
  • -V: A flag to indicate whether to produce verbose output. Default is false.

Export workspace

Exports all the dataflow model.json from a Power BI workspace into a folder:

.\ExportWorkspace.ps1 -Workspace "Workspace1" -Location C:\dataflows

Import workspace

Imports all the dataflow model.json from a folder into a Power BI workspace. This script also fixes the reference models to point to the right dataflow in the current workspace:

.\ImportWorkspace.ps1 -Workspace "Workspace1" -Location C:\dataflows -Overwrite

Import dataflow

Imports a dataflow model.json into a Power BI workspace:

.\ImportModel.ps1 -Workspace "Workspace1" -File C:\MyModel.json -Overwrite

Data Culture: Picking Your Battles

Not all data is created equal.

One size does not fit all.

In addition to collaboration and partnership between business and IT, successful data cultures have something  else in common: they recognize the need for both discipline and flexibility, and have clear, consistent criteria and responsibilities that let all stakeholders know what controls apply to what data and applications.

2020-08-01-19-55-59-794--POWERPNT

Today’s video looks at this key fact, and emphasizes this important point: you need to pick your battles[1].

If you try to lock everything down and manage all data and applications rigorously, business users who need more agility will not be able to do their jobs – or more likely they will simply work around your controls. This approach puts you back into the bad old days before there were robust and flexible self-service BI tools – you don’t want this.

If you try to let every user do whatever they want with any data, you’ll quickly find yourself in the “wild west” days – you don’t want that either.

Instead, work with your executive sponsor and key stakeholders from business and IT to understand what requires discipline and control, and what supports flexibility and agility.

One approach will never work for all data – don’t try to make it fit.


[1] The original title of this post and video was “discipline and flexibility” but when the phrase “pick your battles” came out unscripted[2] as I was recording the video, I realized that no other title would be so on-brand for me. And here we are.

[2] In case you were wondering, it’s all unscripted. Every time I edit and watch a recording, I’m surprised. True story.

Dataflows and API data sources

These days more and more data lives behind an API, rather than in a more traditional data source like databases or files. These APIs are often designed and optimized for small, chatty[1] interactions that don’t lend themselves well to use as a source for business intelligence.

beehive-337695_1280
I’d explain the choice of image for APIs, but it doesn’t take a genus to figure it out

These APIs are often slower[3] than a database, which can increase load/refresh times. Sometimes the load time is so great that a refresh may not fit within the minimum window based on an application’s functional requirements.

These APIs may also be throttled. SaaS application vendors often have a billing model that doesn’t directly support frequent bulk operations, so to avoid customer behaviors that affect their COGS and their bottom line, their APIs may be limited to a certain number of calls[4] for a given period.

The bottom line is that when you’re using APIs as a data source in Power BI, you need to take the APIs’ limitations into consideration, and often dataflows can help deliver a solution that accommodates those limitations while delivering the functionality your application needs.

I’ve covered some of the fundamental concepts of this approach in past blog posts , specifically this one from 2018 on Power BI Dataflows and Slow Data Sources, and this one from 2019 on Creating “data workspaces” for dataflows and shared datasets.  I hadn’t planned on having a dedicated new blog post, but after having this pattern come up in 4 or 5 different contexts in the past week or so, I thought maybe a post was warranted.

In late July I met with a customer to discuss their Power BI dataflows architecture. They showed me a “before” picture that looked something like this:

2020-08-01-13-40-14-887--POWERPNT

One of their core data sources was the API for a commercial software application. Almost every Power BI application used some data from this API, because the application supports almost every part of their business. This introduced a bunch of familiar challenges:

  • Training requirements and a steeper-then-necessary learning curve for SSBI authors due to the complex and unintuitive API design
  • Long refresh times for large data extracts
  • Complex, redundant queries in many applications
  • Technical debt and maintenance due to the duplicate logic

Then they showed me an “after” picture that looked something like this:

2020-08-01-13-40-32-810--POWERPNT

They had implemented a set of dataflows in a dedicated workspace. These dataflows have entities that pull data from the source APIs, and make them available for consumption by IT and business Power BI authors. All data transformation logic is implemented exactly once, and each author can easily connect to trusted tabular data without needing to worry about technical details like connection strings, API parameters, authentication, or paging. The dataflows are organized by the functional areas represented in the data, mapping roughly to the APIs in the source system as viewed through the lens of their audience.

The diagram above simplifies things a bit. For their actual implementation they used linked and computed entities to stage, prepare, and present the dataflows, but for the general pattern the diagram above is probably the right level of abstraction. Each IT-developed or business-developed application uses the subset of the dataflows that it needs, and the owners of the dataflows keep them up to date and current.

Life is good[5].


[1]There’s a lot of information available online covering chatty and chunky patterns in API design, so if you’re not sure what I’m talking about here, you might want to poke around and take a peek[2] at what’s out there.

[2] Please let me know if you found the joke in footnote[1].

[3] Possibly by multiple orders of magnitude.

[4] Or a given data volume, etc. There are probably as many variations in licensing as there are SaaS vendors with APIs.

[5] If you’re wondering about the beehive image and the obtuse joke it represents, the genus for honey bees is “apis.” I get all of my stock photography from the wonderful web site Pixabay. I will typically search on a random term related to my blog post to find a thumbnail image, and when the context is completely different like it was today I will pick one that I like. For this post I searched on “API” and got a page full of bees, which sounds like something Bob Ross would do…

Data Culture: Roles and Responsibilities

In last week’s post and video we looked at how business intelligence tools have evolved, with each evolution solving one set of problems and introducing another set. The video described how self-service BI tools have enabled business users to work with data in ways that used to require specialized technical skills, freeing up IT to focus on other tasks – but at the same time introducing challenges around oversight, consistency, and control.

And that brings us to today’s video.

The video includes a few different graphics, but for this post I want to highlight this one, which I’ve taken from the Power BI Adoption Framework[1].

2020-07-31-20-44-46-355--POWERPNT

Successful cultures balance freedom and restriction in ways that benefit the culture as a whole. It’s a compromise – no one gets everything they want, but everyone gets the things they need the most.

For a data culture, this typically involves letting IT do the things they do best, and letting business get down to business.

When an organization adopts a managed self-service model for business intelligence[2], the central BI team in IT[3] does the heavy lifting. This means they prepare, cleanse, warehouse, and deliver the biggest and most important[4] data. They deliver the data that would be hard for a business user to create using a self-service tool, and which will be used by a large audience. They do the things that have broad reach, strategic impact, and strategic importance. They create things that need to be correct, consistent, and supported.

And business users do the rest. This is a broad bucket, but it’s a reasonable generalization as well. Business users create their own reports using the IT-managed data sources and data models. And they prepare and use their own data where there is no IT-managed data for a given purpose.

Over time a given application or a given data source may become more or less managed, as the culture adopts, adapts, and improves.

Managed self-service BI isn’t the only way to be successful, but in my experience working with many large Power BI customers it’s the best way and most predictable way. By having clearly defined roles and responsibilities – who does what – moving from either extreme can overcome the problems that that extreme creates, without going too far in the other direction.

Does your organization take this approach?

If it doesn’t today, what will it take to make this change a reality?


[1] Which I have shamelessly stolen from the Power BI Adoption Framework, because it is an awesome graphic and because I love to stand on the shoulders of giants.

[2] Which is the approach most likely to drive short-and long term efficiencies and successes.

[3] Please understand that this is a gross simplification. This “central BI team in IT” may be a single central team for the whole company, it may be a single central team for a specific business unit, or it may be one of dozens of BI teams that are designed to support a distributed global enterprise. This is less about the technology and the culture than it is about organizational structure, so I don’t expect to ever try to tackle this diversity of approaches in this series.

[4] Next week’s video will talk a little more about what data is likely to qualify for this distinction. And since the video is done already, I’m pretty confident to say “next week” this week.

Weekend reading: New dataflows docs

In case you didn’t have any plans for the weekend, the Power Query team has released a treasure trove of new documentation for dataflows in Power BI and Power Apps.

You can find the new docs here: https://docs.microsoft.com/en-us/power-query/dataflows/overview-dataflows-across-power-platform-dynamics-365

There are over 20 articles in this new resource, and the content they contain answers many of the questions I hear most frequently. For some reason WordPress isn’t letting me upload the screen shot I took of the content outline, so instead here is a picture of me[1] when I first learned about this new content:

woman-591576_1280

Please check out this new information, and please submit feedback to the documentation team using the feedback form on each page. When you let the docs team know what you like, what you don’t like, or what is missing, they read every comment!


[1] Me in my avatar as a photogenic young woman.