Successfully measuring / measuring success

In 2008 I was hired to solve a problem.

At this point almost 12 years later, the problem itself is no longer relevant[1]. While digging around on an unrelated task today I found this chart, which is. You should look at it now.

2020-03-11-14-44-54-465--POWERPNT
Before Power BI we had PowerPoint, but data was still Power, even back then

The scope of the problem is measured by the blue series on this chart. You should look at it again. Just look at it!

Both the blue series and the yellow series are net satisfaction (NSAT) scores. There’s a lot of context behind the numbers[2], but for the purposes of this post let’s say that on this scale anything over 150 is “time for a team party and a big round of bonuses” and anything under 100 is “you probably won’t include this job on your resume, and you’re thinking about this a lot because you’ve been sending your resume out a lot this week.”

There are two stories that leap out from this chart.

The first story is pretty obvious: something changed in FY06. That change had a dramatically negative impact on the blue series, and a small (and probably acceptable) negative impact on the yellow series.

The second story may not be as obvious, but it’s vitally important: the yellow series was being used to track the impact of the change. Something changed in FY06, and the people that made the change were measuring its impact.

They were tracking the wrong thing.

Until I joined the team, no one had a chart like this. It wasn’t that the blue series wasn’t being tracked – it was. It just wasn’t recognized as the true success metric until things were well into resume-polishing territory.[3]

meme

The lesson here isn’t that someone made a bad decision and didn’t realize it. The lesson is that sometimes the metric you’re tracking doesn’t mean what you think it means.

As is the case in my personal story, the problem is usually quite obvious in retrospect, but it’s also usually quite opaque in the moment. Although most large companies have a culture of measurement, it’s more rare to see a culture that consistently questions those measurements. Although this approach may not work for everyone, I recommend using this three-year-old approach to defining your most important metrics.

I don’t mean that the approach is three years old. I mean that you should approach the problem like a three-year-old would: by repeatedly asking “why?”

When someone[4] suggests measuring using a given metric, ask why. “Why do you think this is the right way to measure this thing?” When you get an answer, ask why again. “Why do you believe that?” Keep asking why – the more important the metric, the more times you should ask why and expect to get a well-considered answer[5]. And if the answers aren’t forthcoming or aren’t credible… that is an important point to recognize before you’ve invested too much in a project or solution, isn’t it?


[1] Which is why I’m not going to talk about the problem or the solution here, except in the most general, hand-wavey terms.

[2] You can read this article if you’re curious.

[3] I should also point out that I wasn’t the person who figured out that we’d been measuring the wrong thing. The person who hired me had figured it out, which was why I was hired. Credit where credit is due.

[4] This someone may or may not be you. But definitely question yourself in the same way, because it’s always hardest to see your own biases.

[5] The person who introduced me to this idea called it “five whys” but I wouldn’t read too much into that specific number. He also never explained what he meant by this, and for months I thought he was referring to some five word phrase where each word started with the letter Y. True story.

Viral adoption: Self-service BI and COVID-19

I live 2.6 miles (4.2 km) from the epicenter of the coronavirus outbreak in Washington state. You know, the nursing home that’s been in the news, where over 10 people have died, and dozens more are infected.[1]

As you can imagine, this has started me thinking about self-service BI.

2020-03-10-17-44-57-439--msedge
Where can I find information I can trust?[2]
When the news started to come out covering the US outbreak, there was something I immediately noticed: authoritative information was very difficult to find. Here’s a quote from that last link.

This escalation “raises our level of concern about the immediate threat of COVID-19 for certain communities,” Dr. Nancy Messonnier, director of the CDC’s National Center for Immunization and Respiratory Diseases, said in the briefing. Still, the risk to the general public not in these areas is considered to be low, she said.

That’s great, but what about the general public in these areas?

What about me and my family?

When most of what I saw on Twitter was people making jokes about Jira tickets[3], I was trying to figure out what was going on, and what I needed to do. What actions should I take to stay safe? What actions were unnecessary or unhelpful?

Before I could answer these questions, I needed to find sources of information. This was surprisingly difficult.

Specifically, I needed to find sources of information that I could trust. There was already a surge in misinformation, some of it presumably well-intentioned, and some from deliberately malicious actors. I needed to explore, validate, confirm, cross-check, act, and repeat. And I was doing this while everyone around me seemed to be treating the emerging pandemic as a joke or a curiosity.

I did this work and made my decisions because I was a highly-motivated stakeholder, while others in otherwise similar positions were farther away from the problem, and were naturally less motivated at the time.[4]

And this is what got me thinking about self-service BI.

In many organizations, self-service BI tools like Power BI will spread virally. A highly-motivated business user will find a tool, find some data, explore, iterate, refine, and repeat. They will work with untrusted – and sometimes untrustworthy – data sources to find the information they need to use, and to make the decisions they need to make. And they do it before people in similar positions are motivated enough to act.

But before long, scraping together whatever data is available isn’t enough anymore. As the number of users relying on the insights being produced increases – even if the insights are being produced by a self-service BI solution – the need for trusted data increases as well.

Where an individual might successfully use disparate unmanaged sources successfully, a population needs a trusted source of truth.

At some point a central authority needs to step up, to make available the data that can serve as that single source of truth. This is easier said than done[5], but it must be done. And this isn’t even the hard part.

The hard part is getting everyone to stop using the unofficial and untrusted sources that they’ve been using to make decisions, and to use the trusted source instead. This is difficult because these users are invested in their current sources, and believe that they are good enough. They may not be ideal, but they work, right? They got me this far, so why should I have to stop using them just because someone says so?

This brings me back to those malicious actors mentioned earlier. Why would someone deliberately share false information about public health issues when lies could potentially cost people their lives? They would do it when the lies would help forward an agenda they value more than they value other people’s lives.

In most business situations, lives aren’t at stake, but people still have their own agendas. I’ve often seen situations where the lack of a single source of truth allows stakeholders to present their own numbers, skewed to make their efforts look more successful than they actually are. Some people don’t want to have to rebuild their reports – but some people want to use falsified numbers so they can get a promotion, or a bonus, or a raise.

Regardless of the reason for using untrusted sources, their use is damaging and should be reduced and eliminated. This is true of business data and analytics, and it is true of the current global health crisis. In both arenas, let’s all be part of the solution, not part of the problem.

Let us be a part of the cure, never part of the plague – we’ll only be remembered for what we create.[6]


[1] Before you ask, yes, my family and I are healthy and well. I’ve been working from home for over a week now, which is a nice silver lining; I have a small but comfortable home office, and can avoid the obnoxious Seattle-area commute.

[2] This article is the best single source I know of. It’s not authoritative source for the subject, but it is aggregating and citing authoritative sources and presenting their information in a form closer to the solution domain than to the problem domain.

[3] This is why I’ve been practicing social media distancing.

[4] This is the where the “personal pandemic parable” part of the blog post ends. From here on it’s all about SSBI. If you’re actually curious, I erred on the side of caution and started working from home and avoiding crowds before it was recommended or mandated. I still don’t know if all of the actions I’ve taken were necessary, but I’m glad I took them and I hope you all stay safe as well.

[5] As anyone who has ever implemented a single source of truth for any non-trivial data domain can attest.

[6] You can enjoy the lyrics even if Kreator’s awesome music isn’t to your taste.

Lazy communication is theft

If you follow me on Twitter[1], you have likely seen me post something like this:

You’ve probably seen it more than once. But you’ve only seen it an order of magnitude less than I’ve thought it, because if I posted it multiple times each day I would be part of the problem. Typically when I tweet a variation on this theme, it’s because someone has been lazy, and has stolen my time, and the time of others.

Consider these scenarios.

Have you ever forwarded a lengthy email thread to a group, with “FYI” or “this is interesting” as your only addition, without adding a summary of the thread? If you have, then each person who receives your mail needs to read through the thread to understand what is important for them.

Have you ever sent an email with a meaningless and non-descriptive subject line that’s unrelated to the message content? If you have, then each person who receives your mail needs to read through the message to understand your intent and to prioritize any follow-up actions.

Have you ever sent an email that includes a document or link to a valuable resource, but you don’t include any relevant search terms in the subject or body? If you have, then when your recipients need to find and use that link or document they will not be able to easily search to locate it. You’ve forced each recipient to implement their own discovery process.

Have you ever sent an email that references a shared resource like a web site or an Excel workbook on a SharePoint site, and didn’t include a link to that resource? If you have, then each recipient has needed to manually locate the shared resource – you have wasted the time of every person who received the mail. And to make matters worse, your laziness has introduced ambiguity, and increased the likelihood that people will end up using the wrong resources.

Have you ever sent an email that includes a general description of a specific problem for which you are requesting assistance? If you have, then you are offloading the responsibility for identifying the problem cause to the recipients – and this often means that multiple people are duplicating the effort that you should have put in proactively.

Have you ever sent an email that includes an acronym that you have not explicitly defined? If you have, then you’re again forcing the recipients to do the heavy lifting to figure out what you mean, when you could have saved them this effort by putting in a little effort on your own…

Have you ever sent an email related to an event – a technical conference “call for content” announcement, for example – and you haven’t bothered to include the event dates in the mail? If you have, then you have forced every recipient to look up this information before they can act on your mail.

Have you ever asked someone for help solving a technical problem or error, but you haven’t clearly articulated the scope of the problem? Maybe you couldn’t even be bothered to include key details like error messages? If this is the case, you’ve very clearly told the people who could be helping you that you do not value their time, and that you are choosing to make your problem their problem.

Of course, the impact of this laziness isn’t limited to email – email just happens to be where I personally experience it the most. My most recent[2] periodic reminder came when someone on Twitter asked for help, and included an undefined acronym. By the time I noticed the conversation, three or four members of the Power BI team had replied, either asking for clarification or proposing possible answers if the acronym meant what they thought it meant. (I did not join that conversation.)

The common theme of these scenarios – and many more like them – is that a small effort to be mindful in your communication can help reduce the cost on the people with whom you are communicating. If you choose not to put in that effort, your lazy communication is stealing time and productivity from your teammates, peers, and colleagues.

Is that what you want?[3]

Each of these bad habits is easily and simply corrected. In most situations it only takes a moment to clarify the meaning and context of your message, to add a subject, or summary, or link. A moment of your time can save many minutes wasted by every person who receives your communication.

Will you choose to spend that time, and to respect the time of others?

Or will you steal their time?


[1] I wouldn’t recommend it.

[2] Most recent when I started writing this post, anyway. That was back in early November. It’s taken me so long to finish and publish this post because people keep stealing my time.

[3] If it is, please don’t tell me.

Common sense solutions to simple problems

Have you ever looked at a problem and immediately known how to solve it?

Have you ever seen that there’s already a group of people who are responsible for solving this problem this problem, and although they’re working on a fix they obviously have not seen the common sense solution that is so apparent to you?

ford-498244_640
I knew we could park here! Problem solved!

Yeah[1].

In situations like these, there are three[2] potential reasons for what you’re seeing:

  1. The other people are just too stupid to to see the simple and obvious solution that’s been sitting right under their noses all this time.
  2. The other people, due to the time that they have spent closely examining the problem and potential solutions, have realized that the simple and obvious solution doesn’t actually address the problem – even though common sense suggests that it should – or that the solution has unintended side-effects that outweigh its benefits.
  3. The other people are responsible for additional problems and solutions across a broader or more strategic scope, and although the simple solution may make sense in isolation, implementing it has been deferred to enable work on other, higher priority problems.

(At this point I feel compelled to point out that this is something I’ve thought about for decades, and I’ve had a draft blog post on this subject for almost a year. I’m not trying to subtweet[3] anyone. I’ve had several people recently reach out to me to make sure that they’re not the reason I tweet periodic reminders that lazy communication is theft, so a disclaimer here feels appropriate.)

In my experience, many people seem to automatically gravitate to the first choice, and that bias seems exacerbated by differences in perspective. If the observer is in a business group and the solution team is in IT – or vice versa[4] – it’s easy for them to miss the complexity and nuance in something that appears simple and straightforward from the outside.

Part of this challenge is inherent in this type of relationship. It’s natural and appropriate to hide internal complexity from external stakeholders. I’m pretty sure I even learned about this in college… not that I remember college[5].

But part of it is the choice we make when we engage in the cross-team relationship. Every time we see a “simple problem” with an “obvious” or “easy” or “common sense” solution we get to decide how to react. We can assume that we know everything important and that the “other guy” is an idiot… or we can consider the option that there might be something we don’t already know.

The choice is ours.


[1] This is less of an actual footnote than a tool to enforce a dramatic pause as you read. Assuming you read footnotes as you go through the post, rather than reading them all at the end. I may need more telemetry to better understand use behavior. Why aren’t any simple problems actually as simple as they first seem?

[2] If you can think of other reasons that aren’t just variations on these three, I would love to hear about them in the comments. My list used to only include the first two reasons, but over the last few years I’ve added the third.

[3] Sub-blog? is that a thing?

[4] There are scores of other variations on this theme – I picked this one because it’s likely to be familiar to the people I imagine read my blog.

[5] Maybe this?

Where has the time gone?

My last post was apparently my 100th post since BI Polar kicked off last October.

2019-11-13-19-30-18-503--msedge

That’s an average of right around two posts per week, although my actual writing output has been much less even and predictable than this number might suggest.

2019-11-13-19-33-01-422--msedge

After 100 posts and a little over one year, where should BI Polar go?

What are the topics you would like to see emphasized in the next year and the next hundred posts?

For the past month or so I’ve been sticking to a three-posts-per-week schedule, but I don’t know how sustainable that is. I’m thinking about switching to a Monday-Wednesday schedule, with one post each Monday to accompany the week’s new YouTube video, plus one additional post each week. This feels like a much more reasonable long-term plan.

So… what topics or themes are you interested in? What would you like to see more of, based on what you’ve seen over the last year? I can’t promise I’ll do what you want, but I can promise that I’ll read every comment, and I expect I’ll be inspired by whatever ideas you have…

It all comes down to culture

I talk about data culture a lot, and in my presentations I often emphasize how the most important success factor when adopting a tool like Power BI[1] is the culture of the organization, not the tool itself.

I talk about this a lot, but I think Caitie McCaffrey may have just had the final word.[2]

2019-10-24-12-47-22-722--msedge

I don’t think that Caitie was talking about the enterprise adoption of self-service business intelligence, but she could have been.

In my day job I get to talk to leaders from large companies around the world, and to see how they’re adopting and using Power BI, Azure. Before today I didn’t think of Moby Dick – I thought of Leo Tolstoy’s classic Anna Karenina, which starts with this classic line:

All happy families are alike; each unhappy family is unhappy in its own way.

Although the details vary, large companies that have successfully adopted managed self-service BI at scale have cultures with important aspects in common:

  • Leaders empower business users to work with data
  • Leaders trust business users to use data to make better decisions
  • IT supports business users with platforms and tools and with curated data sources
  • Business users work with the tools from IT and the guidance from leaders, and work within the guardrails and guidelines given to them for this use
  • Business and IT collaborate to deliver responsive solutions and mature/stable solutions, with clearly defined responsibilities between them

Companies that are successful with managed self-service BI do these things. Companies that are not successful do not. The details vary, but the pattern holds up again and again.

How do these roles and responsibilities relate to culture?

In many ways a culture is defined by the behaviors it rewards, the behaviors it allows, and the behaviors it punishes. A culture isn’t what you say – it’s what you do.

In the context of BI, having a culture with shared goals that enable business and IT to work together with the support from the company leaders is the key. If you have this culture, you can be successful with any tool. Some tools may be more helpful than others, and the culture will enable the selection of better tools over time, but the tool is not the most important factor. The culture – not the tool – inevitably determines success.

This is not to say that BI tools should not improve to be a bigger part of the solution. But to paraphrase Caitie… maybe you should let that white whale swim past.

 


[1] But definitely not only Power BI.

[2] He says unironically, before writing many more words.

BI is dead. Long live BI!

As I was riding the bus home from jury duty the other day[1] I saw this tweet come in from Eric Vogelpohl.

 

There’s a lot to unpack here. and I don’t expect to do it all justice in this post, but Eric’s thought-provoking tweet made me want to reply, and I knew it wouldn’t fit into 280 characters… but I can tackle some of the more important and interesting elements.

First and foremost, Eric tags me before he tags Marco, Chris, or Curbal. I am officially number one, and I will never let Marco or Chris forget it[2].

With that massive ego boost out of the way, let’s get to the BI, which is definitely dead. And also definitely not dead.

Eric’s post starts off with a bold and simple assertion: If you have the reactive/historical insights you need today, you have enough business intelligence and should focus on other things instead. I’m paraphrasing, but I believe this effectively captures the essence of his claim. Let me pick apart some of the assumptions I believe underlie this assertion.

First, this claim seems to assume that all organizations are “good w/ BI.” Although this may be true of an increasing number of mature companies, in my experience it is definitely not something that can be taken for granted. The alignment of business and technology, and the cultural changes required to initiate and maintain this alignment, are not yet ubiquitous.

Should they be? Should we be able to take for granted that in 2019 companies have all the BI they need? [3]

The second major assumption behind Eric’s first point seems to be that “good w/ BI” today translates to “good w/ BI” tomorrow… as if BI capabilities are a blanket solution rather than something scoped and constrained to a specific set of business and data domains. In reality[4], BI capabilities are developed and deployed incrementally based on priorities and constraints, and are then maintained and extended as the priorities and constraints evolve over time.

My job gives me the opportunity to work with large enterprise companies to help them succeed in their efforts related to data, business intelligence, and analytics. Many of these companies have built successful BI architectures and are reaping the benefits of their work. These companies may well be characterized as being “good w/ BI” but none of them are resting on their laurels – they are instead looking for ways to extend the scope of their BI investments, and to optimize what they have.

I don’t believe BI is going anywhere in the near future. Not only are most companies not “good w/ BI” today, the concept of being “good w/ BI” simply doesn’t make sense in the context in which BI exists. So long as business requirements and environments change over time, and so long as businesses need to understand and react, there will be a continuing need for BI. Being “good w/ BI” isn’t a meaningful concept beyond a specific point in time… and time never slows down.

If your refrigerator is stocked with what your family likes to eat, are you “good w/ food”? This may be the case today, but what about when your children become teenagers and eat more? What about when someone in the family develops food allergies? What about when one of your children goes vegan? What about when the kids go off to college? Although this analogy won’t hold up to close inspection[5] it hopefully shows how difficult it is to be “good” over the long term, even for a well-understood problem domain, when faced with easily foreseeable changes over time.

Does any of this mean that BI represents the full set of capabilities that successful organizations need? Definitely not. More and more, BI is becoming “table stakes” for businesses. Without BI it’s becoming more difficult for companies to simply survive, and BI is no longer a true differentiator that assures a competitive advantage. For that advantage, companies need to look at other ways to get value from their data, including predictive and prescriptive analytics, and the development of a data culture that empowers and encourages more people to do more things with more data in the execution of their duties.

And of course, this may well have been Eric’s point from the beginning…

 


[1] I’ve been serving on the jury for a moderately complex civil trial for most of August, and because the trial is in downtown Seattle during business hours I have been working early mornings and evenings in the office, and taking the bus to the courthouse to avoid the traffic and parking woes that plague Seattle. I am very, very tired.

[2] Please remind me to add “thought leader” to my LinkedIn profile. Also maybe something about blockchain.

[3] I’ll leave this as an exercise for the reader.

[4] At least in my reality. Your mileage may vary.

[5] Did this analogy hold up to even distant observation?