Lately it’s copilot before and after in all Microsoft communications and announcements. Also, during the Fabric Community Conference in Las Vegas where I was last week, it was hard to keep count of the number of times copilot was mentioned. Next to Microsoft communications, everyday conversations with customers and colleagues include copilots everywhere too. Which is fine, as long as we manage to keep things in perspective.

In this blog I will further elaborate on my experiences with copilot, where I experienced it as a useful addition to my day-to-day job and also where we should be careful. And no, this is not a rant on copilot, but just sharing my vision and everyday experience when talking about copilot to my customers.

Which copilot?

Let’s start with defining which copilot we’re talking about here. Cause copilot is everywhere! Github copilot, M365 copilot, Power BI copilot, Fabric copilot, Bing Chat Enterprise, oh wait… that’s copilot too now. That’s already where a lot of confusion starts. In all fairness, it is easy that everything has the same name on one end, but on the other hand it also leads to a lot of confusion.

Again, copilot is everywhere, and it will for sure change the way how we do our work. Even in Microsoft Teams it is tightly integrated as part of the M365 copilot. And I tend to love it and hate it at the same time. It will help us all to make things easier. Let’s say summarizing a meeting, getting those meeting notes out there. All tasks that I personally dislike and love copilot taking that duty for me.

Does it do what I expect it to do?

Right now, I think copilot is a bit hyped. You see it everywhere in slick demos. However, when you try it in practice, it is not always such a smooth ride as the demo showed. There are many factors influencing the outcome of copilot, at least in my experience. Again, let’s take the example of writing meeting notes. The quality of the output it is really dependent on the meeting, context and above all language! Most of my meetings are in English and then all is fine. However, when having meetings in Dutch, it doesn’t really provide me the output I expect it to.

Let’s take the example of having copilot in Power BI. It’s pretty awesome in helping users understand the context of their semantic model. Which columns it includes, grasping an overall idea what could be in the semantic model. However, at the same time language is key. Also, regularly I come across semantic models that don’t have pretty names like “Client Name” or “Sales Amount” but more things like “ClientName” or “_CalcSalesAmt”. Having proper naming is a good idea anyway for understandability of your semantic model. I tried this feature on both AdventureWorks demo data, as well as my drone fligh analysis, which is not the typical semantic model you will come across. The results where fine, but again both models have decent naming of tables and columns.

Also, having copilot to suggest a first report page for you looks awesome. But having tried that on multiple different semantic models with different contexts, it’s no longer fancy what it comes up with. Every report will roughly look the same and return the same.

Also, in above experience I was happily surprised with the understanding of my drone data. However, when I asked copilot to generate a report for it by just using the prompt: “build me a first report” it came up with something totally pointless… Counting LensDetails? Average relative altitude by LensDetails? Sum of an absolute altitude? In all fairness, the only relevant metric it came up with, is the average of relative altitude. That is actually something useful to have a look at.

The average conversation on copilot with customers

The average conversation with customers nowadays leads pretty quickly to “copilot will fix that for me, right?” Well, I would not be too sure about that. Building a decent data model is still crucial to get relevant output. Also, as we have seen in above example, understanding the context of the semantic model is fine, but building the report is still something you have to do yourself for now. Also, swapping fields used on a visual is not available yet.

Not saying everyone does, but there seems to be a tendency that copilot is expected like an autopilot for many. Whatever copilot comes up with, as a user you have to sense the output. In that light, it is not different than responses that ChatGPT or any other LLM model provides you. As user, you are responsible for sensing the output of the model and deciding what to do with it. At least in my experience, it is not yet where most people expect it to be (at the time of writing this blog – April 2024).

Putting it in perspective

Let’s face it, would you happily board a plane that is fully controlled by copilot? No real pilots on board anymore… I would not! Also, on planes the actual pilots and operators are still in control to monitor systems, validate what the aircraft is doing and carrying the end responsibility.

So, why would you rely completely on copilot to build your critical business report, semantic modeling, DAX expressions to include business logic and so on? You should still be the operator to sense, validate and control the output whatever copilot is providing you. This applies to each and every copilot there is nowadays, at least in my perspective.

Am I telling you here to not use it? Definelty not! Go for it, experience it and get the best out of it! I do the same with Power BI, Fabric, M365, Bing and wherever I can use copilot or other Large Language Model implementations like ChatGPT. But just be careful with the output it provides, sense and validate before your take action.

It’s a copilot, not an autopilot!