Month: March 2020

Metadata is not a “nice to have”

On By Matthew RocheIn Cooking and Baking, Data Culture, Data Governance, Metadata, Metaphors, Power BI15 Comments

I love to cook, and over the past few days I’ve made a few of my favorite Indian recipes that I haven’t made in a while. So, of course, this has me thinking about metadata.

IMG_20200319_193323
Homemade naan and chicken tikka masala

Going from Indian cooking to metadata isn’t as big a leap as you might think. The bridge is one of my favorite cookbooks: Julie Sahni’s Classic Indian Vegetarian and Grain Cooking.

If you’re just here for the food, you should immediately make this spectacularly delicious Bengal red lentil recipe taken from this book, because it is absolutely phenomenal. If you’re here for the metadata, remember the link but don’t click on it yet.

IMG_20200321_080342

Every recipe I’ve made from this cookbook has produced fantastic results. It’s one of those go-to cookbooks where I know that anything I try will be good. And yet, I almost never seek it out when I want to cook, except for the recipes I already know. The reason is metadata.

It doesn’t matter how good your data is – without effective and available metadata, your investment in quality data will be undermined.

Let’s look at the recipe for saag paneer. Say those words out loud (“saag paneer”) and images of that rich, vibrant green sauce will start running through your mind.

IMG_20200321_080455

I found this recipe easily because I have a bookmark. But let’s say I didn’t – it should still be easy to find, because cookbooks have indexes, and indexes are the perfect tool for finding recipes. Let’s find the recipe for saag paneer.

IMG_20200321_080549
Oh, there’s no entry for saag paneer, or for saag?
IMG_20200321_080628
There’s no entry for paneer? There are so many paneer recipes in this book!
IMG_20200321_080656
Ok, we’re making progress… I think.
IMG_20200321_080727
There it is. Maybe? It still doesn’t say saag paneer anywhere.

Literally the only place the phrase “saag paneer” exists in this book is below the recipe header. This means that the only way to find the saag paneer recipe is to flip through the book page by page, or to know the specific and arbitrary phrase the author uses to describe the recipe for Western readers. This is why my copy of the book looks like this[1]:

IMG_20200321_083450

This systemic problem is exacerbated by the book’s complete lack of photos; there’s also no way to skim through the book and quickly visually identify recipes of relevant interest. The reader is forced to carefully evaluate each recipe in turn, looking at ingredients and processes to decide if the recipe is worth making.

At this point you may be asking what this has to do with metadata[2] or you may see the connection already.

The reason I immediately thought of metadata may be related to a BI effort I’m working on. Without going into too much detail, I have built a small Power BI app that presents information from a program I run and makes that information available to other members of my extended team.

I’m currently at the point where my app needs to include data from other sources in order to increase its value. Fortunately, that data already exists, and to make it even easier to work with, it is available as a set of Power BI dataflows. I was able to email the owner to get access[3] and to learn which dataflows to look in, and I was off. But not for very far, or for very long.

Very quickly I was back where this post started: I was faced with the high-quality data I needed, and I lacked the metadata to efficiently use it. I needed to manually evaluate each dataflow and each entity to understand its contents and context and to decide if it was right for me. I made some early progress, but because of the lack of metadata the effort will likely take days not hours, and this means it probably won’t get done this month or next.

Let that sink in: because of a lack of effective metadata, quality curated data is going unused, and business insights are being delayed by weeks or months[4].

Just like these fantastic recipes sitting on my shelf, largely unused and unmade because a fantastic cookbook lacks a usable index, these fantastic dataflows are going largely unused, at least by me. All because metadata was treated as a “nice to have” rather than as a fundamental high-priority requirement.

Does your data have the metadata it needs, in a format and location that serves the needs of your users? How do you know? Remember that last picture of all the bookmarks[5]?

These bookmarks are a symptom of the underlying metadata problem. Bookmarks aren’t a problem themselves, but if you’re paying attention you can see that they’ve been implemented as a workaround to a problem that might not otherwise be apparent. If you’re familiar with the concept of “code smells”, you probably see where I’m going.

When your data lacks useful metadata to enable its effective use, people will start to take actions because of this lack. Things like emailing you to ask questions. Things like building their own ad hoc data dictionaries. Things like using alternate or derivative sources instead of using your authoritative data source – like the recipe link I shared above.

The more of these actions you identify, the more urgency you should feel about closing the metadata gap. Not every data source is a werewolf, but every data source requires metadata to be effectively and efficiently used.

Requires.

[1] Remember this picture. There will be a quiz later.

[2] You may also be asking if there’s anything in life that doesn’t make me think about metadata. This is a fair question.

[3] I knew the owner’s email because I had bookmarked it earlier.

[4] To be fair, my full schedule is also contributing to this delay – I’m not trying to say that the lack of metadata is independently costing months. But it is a key factor: my schedule could accommodate two or three hours for this work, but it doesn’t have room for two or three days until the end of April.

[5] I told you there would be a quiz.

Think before you GIF

On By Matthew RocheIn Diversity, Mental Health2 Comments

Animated GIF images are an inescapable part of our online experiences, and more and more tools make it easier and easier to include them in our written communication. Sometimes this can be a good thing.

Sometimes. Not all times.

Before you include a GIF image – especially one that flashes or blinks or strobes – in your next chat message, please pause to consider the impact that this may have on the recipients.

burnout-3721062_640

Before you include a GIF, ask yourself:

  • How many people will see this – is this a 1:1 chat, or is it a large group?
  • Does the GIF include flashing, strobing, blinking, fast-moving images?
  • Do any of the people who will see the GIF have photosensitive conditions like seizures or migraines?
  • Are you sure?
  • Does the software tool you’re using allow users to disable GIF autoplay?

I include this last point because Microsoft Teams does not provide any option to disable GIF autoplay. Seriously – even tough the UserVoice forum for Teams says that this was done in 2017, the Teams UX today does not provide any option for users to prevent GIFs from playing for them.

So if you are on a Teams meeting with 100 people and you post a GIF, everyone sees it. And odds are, that GIF you posted will mean that someone on the call will need to leave the call, or close the chat, or maybe end up in a dark room in pain for the rest of the day.

So please think before you GIF.

You can’t avoid problems you can’t see

On By Matthew RocheIn Data Culture, Data Governance, Power BI4 Comments

The last post was about the dangers inherent in measuring the wrong thing – choosing a metric that doesn’t truly represent the business outcome[1] you think it does. This post is about different problems – the problems that come up when you don’t truly know the ins and outs of the the data itself… but you think you do.

This is another “inspired by Twitter” post – it is specifically inspired by this tweet (and corresponding blog post) from Caitlin Hudon[2]. It’s worth reading her blog post before continuing with this one – you go do that now, and I’ll wait.

Caitlin’s ghost story reminded me of a scary story of my own, back from the days before I specialized in data and BI. Back in the days when I was a werewolf hunter. True story.

Around 15 years ago I was a consultant, working on a project with a company that made point-of-sale hardware and software for the food service industry. I was helping them build a hosted solution for above-store reporting, so customers who had 20 Burger Hut or 100 McTaco restaurants[3] could get insights and analytics from all of them, all in one place. This sounds pretty simple in 2020, but in 2005 it was an exciting first-to-market offering, and a lot of the underlying platform technologies that we can take for granted today simply didn’t exist. In the end, we built a data movement service that took files produced by the in-store back-of-house system and uploaded them over a shared dial-up connection[4] from each restaurant to the data center where they could get processed and warehoused.

The analytics system supported a range of different POS systems, each of which produced files in different formats. This was a fun technical challenge for the team, but it was a challenge we expected. What we didn’t expect was the undocumented failure behavior of one of these systems. Without going into too much detail, this POS system would occasionally produce output files that were incomplete, but which did not indicate failure or violate any documented success criteria.

To make a long story short[5], because we learned about the complexities of this system very late in the game, we had some very unhappy customers and some very long nights. During a retrospective we engaged with of the project sponsors for the analytics solution because he had – years earlier – worked with the development group that built this POS system. (For the purposes of this story I will call the system “Steve” because I need a proper noun for his quote.)

The project sponsor reviewed all we’d done from a reliability perspective – all the validation, all the error handling, all the logging. He looked at this, then he looked at the project team and he said:

You guys planned for wolves. ‘Steve’ is werewolves.

Even after all these years, I still remember the deadpan delivery for this line. And it was so true.

We’d gone in thinking we were prepared for all of the usual problems – and we were. But we weren’t prepared for the horrifying reality of the data problems that were lying in wait. We weren’t prepared for werewolves.

Digging through my email from those days, I found a document I’d sent to this project sponsor, planning for some follow-up efforts, and was reminded that for the rest of the projects I did for this client, “werewolves” became part of the team vocabulary.

2019-10-30-12-37-25-597--WINWORD

What’s the moral of this story? Back in 2008 I thought the moral was to test early and often. Although this is still true, I now believe that what Past Matthew really needed was a data catalog or data dictionary with information that clearly said DANGER: WEREWOLVES in big red letters.

This line from Caitlin’s blog post could not be more wise, or more true:

The best defense I’ve found against relying on an oral history is creating a written one.

The thing that ended up saving us back in 2005 was knowing someone who knew something – we happened to have a project stakeholder who had insider knowledge about a key data source and its undocumented behavior. What could have better? Some actual <<expletive>> documentation.

Even in 2020, and even in mature enterprise organizations, having a reliable data catalog or data dictionary that is available to the people who could get value from it is still the exception, not the rule. Business-critical data sources and processes rely on tribal knowledge, time after time and team after team.

I won’t try to supplement or repeat the best practices in Caitlin’s post – they’re all important and they’re all good and I could not agree more with her guidance. (If you skipped reading her post earlier, this is the perfect time for you to go read it.) I will, however, supplement her wisdom with one of my favorite posts from the Informatica blog, from back in 2017.

I’m sharing this second link because some people will read Caitlin’s story and dismiss it because she talks about using Google Sheets. Some people will say “that’s not an enterprise data catalog.” Don’t be those people.

Regardless of the tools you’re using, and regardless of the scope of the data you’re documenting, some things remain universally true:

  • Tribal knowledge can’t be relied upon at any meaningful scale or across any meaningful timeline
  • Not all data is created equal – catalog and document the important things first, and don’t try to boil the ocean
  • The catalog needs to be known by and accessible to the people who need to use the data it described
  • Someone needs to own the catalog and keep it current – if its content is outdated or inaccurate, people won’t trust it, and if they don’t trust it they won’t use it
  • Sooner or later you’ll run into werewolves of your own, and unless you’re prepared in advance the werewolves will eat you

When I started to share this story I figured I would find a place to fit in a “unless you’re careful, your data will turn into a house when the moon is full” joke without forcing it too much, but sadly this was not the case. Still – who doesn’t love a good data werehouse joke?[6]

Maybe next time…

[1] Or whatever it is you’re tracking. You do you.

[2] Apparently I started this post last Halloween. Have I mentioned that the past months have been busy?

[3] Or Pizza Bell… you get the idea.

[4] Each restaurant typically had a single “data” phone line that used the same modem for processing credit card transactions. I swear I’m not making this up.

[5] Or at least short-ish. Brevity is not my forte.

[6] Or this data werehouse joke, for that matter?

Successfully measuring / measuring success

On By Matthew RocheIn Data Culture, Generic Blather2 Comments

In 2008 I was hired to solve a problem.

At this point almost 12 years later, the problem itself is no longer relevant[1]. While digging around on an unrelated task today I found this chart, which is. You should look at it now.

2020-03-11-14-44-54-465--POWERPNT
Before Power BI we had PowerPoint, but data was still Power, even back then

The scope of the problem is measured by the blue series on this chart. You should look at it again. Just look at it!

Both the blue series and the yellow series are net satisfaction (NSAT) scores. There’s a lot of context behind the numbers[2], but for the purposes of this post let’s say that on this scale anything over 150 is “time for a team party and a big round of bonuses” and anything under 100 is “you probably won’t include this job on your resume, and you’re thinking about this a lot because you’ve been sending your resume out a lot this week.”

There are two stories that leap out from this chart.

The first story is pretty obvious: something changed in FY06. That change had a dramatically negative impact on the blue series, and a small (and probably acceptable) negative impact on the yellow series.

The second story may not be as obvious, but it’s vitally important: the yellow series was being used to track the impact of the change. Something changed in FY06, and the people that made the change were measuring its impact.

They were tracking the wrong thing.

Until I joined the team, no one had a chart like this. It wasn’t that the blue series wasn’t being tracked – it was. It just wasn’t recognized as the true success metric until things were well into resume-polishing territory.[3]

meme

The lesson here isn’t that someone made a bad decision and didn’t realize it. The lesson is that sometimes the metric you’re tracking doesn’t mean what you think it means.

As is the case in my personal story, the problem is usually quite obvious in retrospect, but it’s also usually quite opaque in the moment. Although most large companies have a culture of measurement, it’s more rare to see a culture that consistently questions those measurements. Although this approach may not work for everyone, I recommend using this three-year-old approach to defining your most important metrics.

I don’t mean that the approach is three years old. I mean that you should approach the problem like a three-year-old would: by repeatedly asking “why?”

When someone[4] suggests measuring using a given metric, ask why. “Why do you think this is the right way to measure this thing?” When you get an answer, ask why again. “Why do you believe that?” Keep asking why – the more important the metric, the more times you should ask why and expect to get a well-considered answer[5]. And if the answers aren’t forthcoming or aren’t credible… that is an important point to recognize before you’ve invested too much in a project or solution, isn’t it?

[1] Which is why I’m not going to talk about the problem or the solution here, except in the most general, hand-wavey terms.

[2] You can read this article if you’re curious.

[3] I should also point out that I wasn’t the person who figured out that we’d been measuring the wrong thing. The person who hired me had figured it out, which was why I was hired. Credit where credit is due.

[4] This someone may or may not be you. But definitely question yourself in the same way, because it’s always hardest to see your own biases.

[5] The person who introduced me to this idea called it “five whys” but I wouldn’t read too much into that specific number. He also never explained what he meant by this, and for months I thought he was referring to some five word phrase where each word started with the letter Y. True story.

Viral adoption: Self-service BI and COVID-19

On By Matthew RocheIn Data Culture, Data Governance, Generic Blather, Heavy Metal, Metaphors, Patterns, Power BI1 Comment

I live 2.6 miles (4.2 km) from the epicenter of the coronavirus outbreak in Washington state. You know, the nursing home that’s been in the news, where over 10 people have died, and dozens more are infected.[1]

As you can imagine, this has started me thinking about self-service BI.

2020-03-10-17-44-57-439--msedge
Where can I find information I can trust?[2]
When the news started to come out covering the US outbreak, there was something I immediately noticed: authoritative information was very difficult to find. Here’s a quote from that last link.

This escalation “raises our level of concern about the immediate threat of COVID-19 for certain communities,” Dr. Nancy Messonnier, director of the CDC’s National Center for Immunization and Respiratory Diseases, said in the briefing. Still, the risk to the general public not in these areas is considered to be low, she said.

That’s great, but what about the general public in these areas?

What about me and my family?

When most of what I saw on Twitter was people making jokes about Jira tickets[3], I was trying to figure out what was going on, and what I needed to do. What actions should I take to stay safe? What actions were unnecessary or unhelpful?

Before I could answer these questions, I needed to find sources of information. This was surprisingly difficult.

Specifically, I needed to find sources of information that I could trust. There was already a surge in misinformation, some of it presumably well-intentioned, and some from deliberately malicious actors. I needed to explore, validate, confirm, cross-check, act, and repeat. And I was doing this while everyone around me seemed to be treating the emerging pandemic as a joke or a curiosity.

I did this work and made my decisions because I was a highly-motivated stakeholder, while others in otherwise similar positions were farther away from the problem, and were naturally less motivated at the time.[4]

And this is what got me thinking about self-service BI.

In many organizations, self-service BI tools like Power BI will spread virally. A highly-motivated business user will find a tool, find some data, explore, iterate, refine, and repeat. They will work with untrusted – and sometimes untrustworthy – data sources to find the information they need to use, and to make the decisions they need to make. And they do it before people in similar positions are motivated enough to act.

But before long, scraping together whatever data is available isn’t enough anymore. As the number of users relying on the insights being produced increases – even if the insights are being produced by a self-service BI solution – the need for trusted data increases as well.

Where an individual might successfully use disparate unmanaged sources successfully, a population needs a trusted source of truth.

At some point a central authority needs to step up, to make available the data that can serve as that single source of truth. This is easier said than done[5], but it must be done. And this isn’t even the hard part.

The hard part is getting everyone to stop using the unofficial and untrusted sources that they’ve been using to make decisions, and to use the trusted source instead. This is difficult because these users are invested in their current sources, and believe that they are good enough. They may not be ideal, but they work, right? They got me this far, so why should I have to stop using them just because someone says so?

This brings me back to those malicious actors mentioned earlier. Why would someone deliberately share false information about public health issues when lies could potentially cost people their lives? They would do it when the lies would help forward an agenda they value more than they value other people’s lives.

In most business situations, lives aren’t at stake, but people still have their own agendas. I’ve often seen situations where the lack of a single source of truth allows stakeholders to present their own numbers, skewed to make their efforts look more successful than they actually are. Some people don’t want to have to rebuild their reports – but some people want to use falsified numbers so they can get a promotion, or a bonus, or a raise.

Regardless of the reason for using untrusted sources, their use is damaging and should be reduced and eliminated. This is true of business data and analytics, and it is true of the current global health crisis. In both arenas, let’s all be part of the solution, not part of the problem.

Let us be a part of the cure, never part of the plague – we’ll only be remembered for what we create.[6]

[1] Before you ask, yes, my family and I are healthy and well. I’ve been working from home for over a week now, which is a nice silver lining; I have a small but comfortable home office, and can avoid the obnoxious Seattle-area commute.

[2] This article is the best single source I know of. It’s not authoritative source for the subject, but it is aggregating and citing authoritative sources and presenting their information in a form closer to the solution domain than to the problem domain.

[3] This is why I’ve been practicing social media distancing.

[4] This is the where the “personal pandemic parable” part of the blog post ends. From here on it’s all about SSBI. If you’re actually curious, I erred on the side of caution and started working from home and avoiding crowds before it was recommended or mandated. I still don’t know if all of the actions I’ve taken were necessary, but I’m glad I took them and I hope you all stay safe as well.

[5] As anyone who has ever implemented a single source of truth for any non-trivial data domain can attest.

[6] You can enjoy the lyrics even if Kreator’s awesome music isn’t to your taste.