How to test new Azure DevOps extension version before publishing it for everyone

When testing a new version of your extension, you need to use either use a different ExtensionID or a different PublisherID. And the test extension must be marked public: false.

There are multiple ways to make this process easy. I personally use the Azure DevOps Extension Tasks in different ways.

For my own private extension I have a build definition that builds either the public version or the private version. In the past I used to have 2 separate build definitions, but with YAML available I’ve started to consolidate this into a single definition. extensionTag is appended to the existing extensionId.

steps:
- task: ms-devlabs.vsts-developer-tools-build-tasks.package-extension-build-task.PackageVSTSExtension@1
  displayName: 'Package Extension: $(Build.SourcesDirectory)'
  inputs:
    rootFolder: '$(Build.SourcesDirectory)'
    outputPath: '$(Build.BinariesDirectory)\vsix\jessehouwing.azure-pipelines-snyk-task.vsix'
    outputVariable: CreateExtension.OutputPath
    publisherId: jessehouwing
    extensionId: 'vsts-snyk'
    extensionVersion: '$(Build.BuildNumber)'
    updateTasksVersion: true
    updateTasksVersionType: patch
    extensionVisibility: public

- task: ms-devlabs.vsts-developer-tools-build-tasks.publish-extension-build-task.PublishExtension@1
  displayName: 'Publish Extension Private'
  inputs:
    connectedServiceName: 'Jesse Houwing'
    fileType: vsix
    vsixFile: '$(CreateExtension.OutputPath)'
    extensionTag: '-develop'
    extensionVisibility: private
  condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/master'))

- task: ms-devlabs.vsts-developer-tools-build-tasks.publish-extension-build-task.PublishExtension@1
  displayName: 'Publish Extension Public'
  inputs:
    connectedServiceName: 'Jesse Houwing'
    fileType: vsix
    vsixFile: '$(CreateExtension.OutputPath)'
    extensionVisibility: public
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))

I use the conditions to trigger either the public or the private publish function.

The end result looks like this in my publisher:

On the ALM Rangers account we use a build definition that builds a single vsix during build and then we use binary promotion to publish the vsix with different overrides. You don’t need to use a different publisher in this case, but the Rangers do. The reason for this is that the Rangers used to publish to the MsDevDiv publisher and Microsoft didn’t want to give every contributor access to that publisher and all of its extensions. So the separate publisher is used more to separate the concerns of developing the extension from providing support, answering questions and responding to reviews.

In order to test I publish the extension to a different Azure DevOps organisation. This is because you can’t have two extensions installed to the same Azure DevOps organisation if both extensions contain the same Build Task Ids. In my case I use dev.azure.com/jessehouwing to build my extensions and use dev.azure.com/jessehouwing-dev to test the changes before making them publicly available.

For some extensions I publish a second private extension for early adopters:

• Extension ID: jessehouwing.snyk-develop privately shared to jessehouwing-dev for testing.
• Extension ID: jessehouwing.snyk-canary privately shared to a couple of select users for early adopters.
• Extension ID: jessehouwing.snyk for public use.

A couple of my customers have a special case where they work on an extension pack with multiple developers at the same time. To not have to provide each developer with a separate Azure DevOps organisation and build agents, they publish the test and public extension to a single account. In this case:

• Extension ID: publisher.extension private for standard usage.
• Extension ID: publisher.extension-branch, private, preview for internal development and canary releases. There can be multiple of these active at the same time.

To allow this each build task must have a unique Task ID for the Build tasks in in the extension. The Azure DevOps Extension Tasks have a special feature to generate unique IDs based on the publisher, extension-id, taskname. This feature is detailed in these release notes.

I recently presented at the MVP summit on these usage patterns. The presentation is shared here.

If you want to roll your own scripts then you can follow the following pattern:

• vss-extension.json – store all common properties of the extension in this file. Don’t specify extensionid nor galleryflags nor public.
• vss-extension-test.json – store the values unique to the test version. These include: extensionid, galleryflags: preview, public: false.
• vss-extension-release.json – store values unique to release version. These include: extensionid, galleryflags: public, public: true.

Then invoke:

// deploy test
tfx extension publish --manifest-globs vss-extension.json vss-extension-test.json
// deploy release
tfx extension publish --manifest-globs vss-extension.json vss-extension-release.json

To publish the combined manifests.

Or use an override manifest:

• vss-extension.json – store all the details for the public extension
• vss-extension-override-test.json – store a json-patch file with the values you want to override. Again: extensionid, galleryflags: preview, public.

Then use

// deploy test
tfx extension publish --manifest-globs vss-extension.json --override-file vss-extension-override-test.json
// deploy release: 
tfx extension publish --manifest-globs vss-extension.json

If you’re rolling your own scripts, then you can use vsts-bump to auto-increase the version of your build tasks.