Back in 2019, Ton Swart and I wrote a blog post on Versioning and CI/CD for Power BI with Azure DevOps. Back then, there were not many options available for versioning and professionalizing your ALM story for Power BI. Till today, this blog post is till my most read blog on Data-Marc.com which makes us proud.
In May 2020, Microsoft announced Power BI deployment pipelines, a native ALM feature as part of the Power BI Service. As Microsoft stated: Deployment pipelines help enterprise BI teams build an efficient and reusable process by maintaining development, test, and production environments. BI creators can incrementally transition new or updated content between environments, reconfiguring them with the appropriate data connections and permissions.
As I wrote in July 2020 in my blog about why you should care about Power BI deployment pipelines, there were some functionalities lacking in Power BI deployment pipelines, which are easy to build and configure yourself in Azure DevOps as you have full flexibility to customize operations to match your organizational processes. In this blog I highlighted some strengths of Power BI Deployment pipelines, like incremental and metadata deployment. Though, versioning was a missing functionality at the time.
As you might have seen, during the recent MS Build conference, Microsoft announced API support for Power BI Deployment Pipelines. My colleague Ton and I had the privilege to get our hands on this in an early stage, which gave us the opportunity to build an Azure DevOps extension (read along!) based on these all-new APIs. In this blog we will run you through the configuration of the DevOps extension, where you can download it and get you up to speed!
Get you up to speed with the ALM topic
In case you are not familiar yet with the topics described in above introduction, I encourage you to first look at below resources that will help you to get up to speed.
- Read more about the general setup of DTAP for Power BI in the blog about Multi-tier architecture and continuous delivery with Power BI
- The first version of Azure DevOps integration to setup versioning and CI/CD for Power BI in the blog Versioning and CI/CD for Power BI with Azure DevOps
- Introduction to Power BI Deployment Pipelines in the Microsoft documentation
- Find our more details about the comparison between Azure DevOps and native Power BI deployment pipelines this blog about Why you should care about Power BI Deployment Pipelines
Above topics will help you to better understand the end-to-end story of Deployment Pipelines. The current gaps we identify in the native Power BI Deployment Pipelines are the lack of dataflow support, versioning capabilities and DevOps integration.
Looking at the Power BI release plan, dataflow support for Deployment Pipelines is coming up shortly! Currently it is scheduled for June 2021 to reach the public preview state. Versioning and DevOps integration go hand-in-hand to our opinion. With Azure DevOps Git integration, we can overcome the versioning challenge while integrating with Azure DevOps at the same time, as described in the previous blog in 2019. Today, we release a new version of the DevOps implementation which uses native Power BI functionality. Stay tuned!
As we really like the metadata deployment and the ease of setup a pipeline in the Power BI Service, Ton and I decided to setup an Azure DevOps extension based on the recently released Power BI REST APIs for Deployment Pipelines. Although Microsoft promised to come-up with a native DevOps extension over time, we decided to go for it. Time to bridge the gap!
Setup your ALM story with Deployment Pipelines and Azure DevOps
In this section, we will walk you through all the steps you need to take to integrate Azure DevOps and Power BI Deployment pipelines. Step by step, we will describe each step you have to take.
Setup Deployment Pipelines in the Power BI Service
To not go into much detail, it all starts with setting up your Power BI Deployment Pipeline in the Power BI Service. Without Deployment Pipeline, we will not be able to configure the DevOps extension. To configure the pipeline, we need to have a workspace that we assign to the pipeline. Today, it is mandatory to first create a pipeline manually. As announced in the release blog, the API will support pipeline creation as well in the future.
It all starts with having a Power BI workspace. For this example, we have setup a workspace “Deployment Sample [DEV]” as this will be our development workspace. In this workspace we have a dataset, report and dashboard like shown below.
Notice that the workspace is part of Premium (Premium per User in this example). This is required to use Power BI Deployment Pipelines.
In the top ribbon of the workspace, you will see the button “Create a pipeline”. By clicking this button, you will be able to setup a new Power BI deployment pipeline that includes this workspace. Secondly, you need to assign the workspace to one of the stages in the pipeline. In this example, we assigned the workspace to the Development stage.
The Deployment Pipeline in the Power BI Service will automatically be created. During the configuration, the pipeline will contain only one workspace that is assigned to the development stage. For now, this is sufficient as the other workspaces will be generated during deployment and run of the pipeline.
Currently, you are limited to assign only one workspace to a Deployment Pipeline. In case you have your DTAP configuration already set, you must choose which workspace you assign to the pipeline. This will change in the future, when you will be able to assign multiple workspaces to a pipeline. You can find this future plan on the public roadmap.
Setup Azure DevOps
In Azure DevOps we need to import the DevOps extension first. You can find the extension via this link. If you are new to DevOps extensions, this documentation will guide you through installing extensions step by step.
The DevOps task, can be configured in multiple ways. In our example we use Release Pipelines in Azure DevOps. In the pipeline we use the imported extension for the Test and Production stage to trigger the Power BI Deployment Pipeline to run, like shown in the above example. Depending on your configuration, you can configure pre-deployment approvals or other settings if you like.
If we go into edit mode of the pipeline, we can configure different settings for each pipeline stage. The DevOps extension includes multiple tasks that can be set. Below is the setup as we have invented it:
- Development: Import Task, to upload PBIX files from the GIT Repository to the Development Workspace.
- Test: Pipeline task, to move content from Development to the Test Workspace.
- Production: Pipeline task, to move content from Test to the Production Workspace.
Below we describe in more detail what all different settings mean and how you can configure them in specific for the Deployment Pipeline Task. Underneath you will find a screenshot where all these settings can be found.
- Displayname: this is the name of the task executed on pipeline stage execution. If you want to combine different tasks to run on the Agent Job, you can give each task an explanatory name for your own reference. The name is not used elsewhere.
- Power BI service connection: this is the connection that is configured as Service Connection in Azure DevOps and used for authentication to the Power BI Service and Power BI Deployment Pipeline. The service connection can be a Username/Password combination as well as a Service Principal. Please know that Username/Password combination can only be used without MFA configured.
Power BI Details
- Pipeline name or id: this represents the name or the GUID from the Power BI Deployment pipeline as configured in the earlier section of this blog. Following the example, the name should be ”Demo pipeline for blogpost”.
- Target stage name: this dropdown represents the name you want to deploy the content to. If you want to deploy content from Development to Test, you have select “Test”, if you want to deploy from Test to Production, you have to select “Production” in the dropdown menu.
- Deploy Action: this defines what type of deployment you want to execute. In the dropdown menu you can select “Deploy all content from a stage” or “Deploy specific content from a stage”.
- All content: If you select all content, there are no further fields you must fill in the configuration.
- Specific content: If you want to deploy that one report or dashboard only, you can choose to deploy specific content. Additional variables will appear on the screen to fill. In these fields you can specify the Dashboards, Reports and/or Datasets you want to deploy. You can specify the GUIDs from these objects or the names, both works. If you want to specify multiple items, you can specify them in order, semicolon ( ; ) separated.
- Purge data from the target artifact: This checkbox is created in the deployment with which you can allow the deployment to delete the current data from the target artifact in case of a schema mismatch, such as a change in datatype that does not match the current dataset. For more information, you can check the documentation about Power BI Deployment pipelines troubleshooting.
In the previous section, we explained that the “Deploy specific content from a stage” allows you to deploy selected items from a given stage in a Power BI Deployment pipeline. Included at this configuration, we also have the checkbox to include related artifacts. Enabling this checkbox will investigate artifact dependencies to find all related items and include them in the deployment.
Dependency investigation only works upstream at this moment. Which means as much as, the dependency investigation only flows from Dashboard > Report > Dataset, but not the other way around. Please take this into consideration when you specify the names and/or IDs of the artifacts you want to deploy.
In order to easily investigate the artifacts that will be included with the Include related artifacts setting enabled, I encourage you to use the lineage view in the Power BI Service.
Taking the above example of the lineage view, the following scenarios could be defined, all with the setting for related artifacts enabled.
- Specify Dashboard name: in the scenario where the dashboard name “Demo Dashboard” is specified in the DevOps task variables, this will also deploy both downstream reports, “DIAD Final Report” and “Demo Report”, as well as the dataset “DIAD Final Report”.
- Specify Report name: when the report name “Demo Report” is specified in the deployment, the downstream dataset “DIAD Final Report” will also be deployed. But other report “DIAD Final Report” as well as the upstream dashboard “Demo Dashboard” will not be deployed.
- Specify Dataset name: the last scenario where the dataset name “DIAD Final Report” is specified in the dataset variable, only the dataset will be deployed. As you might notice, the setting to deploy related artifacts is already included here. This will be useful in the future when dataflows are supported to investigate downstream dataflows that also must be deployed. The dependency investigation already supports this scenario today, but deployment pipelines cannot deploy dataflows yet.
Wrap up and remarks
We believe the integration between Azure DevOps and native Power BI Deployment pipelines eases the configuration and setup of DT(A)P story for Power BI and at the same time professionalizes the end-to-end ALM story. With this integration it will be easier and more visual for end-users to configure their pipeline in the Power BI service while still being able to trigger pipeline runs in Azure DevOps when Power BI artifact development is part of a larger data platform deployment.
Besides being able to trigger the Power BI Deployment Pipeline, versioning capabilities can be professionalized in this way. Combining the previous version of DevOps deployments as described in an earlier blog, combined with this new extension, allows you to version your Power BI files through DevOps GIT integration and deploy from stage to stage using native deployment pipelines. You can configure the previous extension for your development stage in DevOps Release Pipeline to upload new artifacts from your repository to the Power BI Service and use the newly added Deployment Pipelines tasks to deploy to Test and Production workspaces.
As API support for Power BI Deployment Pipelines is still in a public preview state, please know that things might change over time. Ton and I will do our best to keep the DevOps Extension updated according to the most recent changes of these APIs. If there is anything you bump into, feel free to leave a comment underneath or in the DevOps Marketplace.
Finally, while dataflow deployment is still a missing piece (early June 2021) in the native Power BI Deployment Pipelines, you might have spotted dataflow support as preview in the earlier screenshots. With permission of the responsible Program Manager in the Microsoft Power BI team, we were allowed to share these screenshots, which also means that it will not be long before dataflows will be supported!
With that, I also want to do a shout out to the team we worked with at Microsoft and thank them for their close collaboration, feedback and allow us to build this extension in such an early stage. Those involved know that it concerns them. Thanks a lot!
- Azure DevOps Extension: Macaw Power BI Extension
- Blog: Versioning and CI/CD for Power BI with Azure DevOps
- Blog: Why you should care about Power BI Deployment Pipelines
- Blog: Multi-tier architecture and continuous delivery with Power BI
- Microsoft blog: announced Power BI deployment pipelines
- Microsoft blog: Microsoft announced API support for Power BI Deployment Pipelines
- Microsoft documentation: Power BI release plan – dataflow support
- Microsoft documentation: Power BI release plan – assign multiple workspaces
- Microsoft documentation: Power BI REST APIs for Deployment Pipelines
- Microsoft documentation: Power BI Deployment pipelines troubleshooting
- Microsoft documentation: Power BI lineage view in the Power BI Service
- Microsoft documentation: Installing Azure DevOps Extensions
- Microsoft documentation: Release Pipelines in Azure DevOps
- Microsoft documentation: Service Connection in Azure DevOps
This blog and solution are co-authored by my colleague, Ton Swart. Ton is a Data & Analytics DevOps Engineer at Macaw and has over many years of experience in IT. This includes roles as BI Consultant, Data warehouse specialist and IT consultant. His focus is on Business Intelligence. Ton his expertise is CI/CD with Azure DevOps, PowerShell as automating Power BI deployments.