Category: azure-pipelines

Auto Added by WPeMatico

How to test a core pipeline repo against another, dependent repo with Azure DevOps PR build validation?

Context

In my Azure DevOps project, I have multiple project repos using a common Azure DevOps pipeline. I have accomplished this by defining the pipeline in a core pipeline repo and then referring to it from each project repo.

Repo: core-pipeline

File: pipeline.yml

jobs:
  - job: exampleJob
    displayName: Example job
    steps:
      - checkout: core_pipeline # Not magic; see azure-pipelines.yml below.
        path: $(PIPELINE_DIR_RELATIVE)
      - checkout: self
        path: $(PROJECT_DIR_RELATIVE)
      - task: ShellScript@2
        displayName: Show project repo info
        inputs:
          scriptPath: $(PIPELINE_DIR)/showProjectInfo.sh
          cwd: $(PROJECT_DIR)

variables:
  __SOURCES_DIR__: s
  PIPELINE_DIR_RELATIVE: $(__SOURCES_DIR__)/core-pipeline
  PIPELINE_DIR: $(Pipeline.Workspace)/$(PIPELINE_DIR_RELATIVE)
  PROJECT_DIR_RELATIVE: $(__SOURCES_DIR__)/$(Build.Repository.Name)
  PROJECT_DIR: $(Pipeline.Workspace)/$(PROJECT_DIR_RELATIVE)

File: showProjectInfo.sh

#!/bin/bash

pwd
ls -al

Repo: example-project

File: azure-pipelines.yml

resources:
  repositories:
    - repository: core_pipeline # Must match `checkout:` in core pipeline repo.
      type: git
      name: MyAzureDevOpsProject/core-pipeline
      ref: master

trigger:
  - master

extends:
  template: pipeline.yml@core_pipeline # Must match `repository:` above.

(This is the only piece of code duplicated across project repos.)

Problem

The setup described above works well. For example, I can have build validation for pull requests in example-project, and I can trigger pipeline runs manually from the Azure DevOps web GUI. In the latter case, I can optionally select another branch for the core_pipeline repository resource, which is useful for testing changes to the core-pipeline repo before merging them.

However, there is nothing preventing me from merging PRs in core-pipeline without having tested them whatsoever. I have to remember to manually test each project repo against the core-pipeline PR branch before merging the PR, which is somewhat tedious and, above all, very error-prone. There is effectively no protection against making arbitrarily bad changes to the core pipeline, breaking the workflow of all project repos depending on it.

I can add example-project‘s pipeline to the build validation policies for PRs in core-pipeline, but then core-pipeline‘s master branch is used for the validation build, which is useless; the point is to use the PR branch.

Goal

I would like to have a build validation policy for PRs in the core-pipeline repo such that example-project must pass a pipeline run using that PR branch of core-pipeline before the PR can me merged.

It is not necessary that all project repos using the core pipeline repo automatically be covered by said build validation; I’d be perfectly happy if I had to manually select (once) which project repos should be included.

Important note

The core pipeline consists of two equally important parts:

  • the YAML template – processed at template compile time, before the pipeline run is started
  • the Bash script – cloned at runtime in the checkout task

Checking out a specific ref using inline syntax, e.g.

- checkout: 'git://MyAzureDevOpsProject/core_pipeline@${{ parameters.ref }}'
  path: $(PIPELINE_DIR_RELATIVE)

is not a (complete) solution, because even if parameters.ref can be populated with the name of the PR branch, only the Bash script will be affected; the YAML template will still be from the master branch, not the PR branch.

Go to Source
Author: Simon Alling

Resource collision in Azure Devops Pipeline

The system I’m working with is using a private agent pool rather than hosted.

Roughly, my yaml defined procedure is the following and each step is dependent on the last:

  1. Build (to check for build failures)
  2. Test
  3. Deploy (I simply build and output the files into another predefined folder of servers on the local network)

The Issue in Detail

For different pipelines, the devops service is able to create different folders for each pipeline and I have little issues. However, regarding on the subject matter of different branches on the same pipeline; the issue I am running into is that if the CI/CD trigger gets scheduled too closely, the build and test directories for the pipeline will race condition causing at least one of the runs to fail. Furthermore, they can also interfere with each other’s “deploy” if the build tries to write to the same remote folder.

  • Is there a way to define a separate workspace for each branch so that building and testing happens in isolation from other runs?
  • Is there some sort of locking system I can implement so that only one
    job can execute my entire pipeline job atomically?

FYI

Scouring through the DevOps documentation, it appears you’re not supposed to be able to define the output folders or workspace folder through normal means.

I tried using the Exclusive Lock feature, but couldn’t find any documentation or examples on how to make this work with my pipeline (I instead, found an open issue on the Microsoft documentation GitHub saying the documentation was missing). It is my understanding this feature is extremely new.

I also noticed that the Hosted Agents use a separate vm container for each run that largely eliminate resource collision. Are we supposed to setup something similar on the private agent?

For reference, I am relatively new to Azure DevOps.

Go to Source
Author: Bennett Yeo

Azure Pipeline PowerShell@2 Task can’t run ‘invoke-build’ command as inline script (build PowerShell Module)

I’m trying to build a powershell module in an Azure pipeline (dev.azure.com).

I’ve setup an initial pipleline and then building it up incrementally in small steps, because inexplicable errors are encountered far too easily on Azure.

I’ve added a ‘Prepare’ stage:

name: $(Build.DefinitionName)_$(Date:yyyyMMdd))

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  major: 0
  minor: 0
  patch: $(Build.BuildID)
  buildVersion: $(major).$(minor).$(patch)

stages:
- stage: Prepare
  jobs:
    - job: Prepare
      steps:
      - powershell: .bootstrap.ps1
        displayName: 'Install pre-requisites'

This is based on the original yaml that you get by default when you start a new Pipeline, so the part invoking the bootstrap file is what I got generated for free (This part works ok).

The bootstrap uses a dependencies file into which you can declare your dependencies and these are subsequently imported. One of those dependencies is InvokeBuild which contains the Invoke-Build command. So I would expect to be able to use the Invoke-Build command in subsequent stages.

The next part of the yaml pipeline is as follows:

- stage: Build
  jobs:
    - job: Build
      steps:
        - task: PowerShell@2
          inputs:
            targetType: 'inline'
            script: 'invoke-build build'

The invoke-build build, in itself is ok valid and works on a local host.

The Build job fails with the following error:

/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/edb3758b-8e16-448d-bea2-9ba18acb9693.ps1'
Invoke-Build: /home/vsts/work/1/s/do-build.ps1:3
Line |
   3 |  Invoke-Build build
     |  ~~~~~~~~~~~~
     | The term 'Invoke-Build' is not recognized as the name of a
     | cmdlet, function, script file, or operable program. Check the
     | spelling of the name, or if a path was included, verify that
     | the path is correct and try again.

I discovered there is another way to invoke the build with the PowerShell@2 task using the filePath input:

- stage: Build
  jobs:
    - job: Build
      steps:
        - task: PowerShell@2
          inputs:
            filePath: '$(System.DefaultWorkingDirectory)/do-build.ps1'

the do-build.ps1 script is defined as:

Set-Location .Elizium.FakeBuddy
Invoke-Build build

(Elizium.FakeBuddy is the name of my PowerShell test module being built and that directory contains the InvokeBuild build script)

but this fails with the exact same error.

So why is the invoke-build command not available in the powershell session at the build stage, even though the Prepare stage has already imported the module.

The only thing I can think of is that different stages somehow don’t share the same powershell session, but I don’t know if this is correct, this seems unlikely, it doesn’t make sense that different stages dont share the same session.

EDIT: I tried an alternative version where the Build job is declared inside the same stage as Prepare, but this makes no difference, still a problem with Invoke-Build command.

Go to Source
Author: Plastikfan

How do i formulate file path for testsettings on the Runsettings file parameter

Whats the proper way to form relative file path/ or file path for the runsettings parameter

<SettingsFile>xxxxxAutomation.testsettings</SettingsFile>

In my local environment this works but when i push it to build release agents it fails since
its looking for the file in a different directory.

This is the directory its looking at E:Agent_workr10axxxxxxxxx.testsettings,

here is my current way i have it set

<SettingsFile>xxxxxAutomation.testsettings</SettingsFile>

here is the error message
Error: The test settings file E:Agent_workr10axxxxxxxxxx.testsettings, specified in the MSTestAdapter settings, is not available. Either access to the file is denied or the file does not exist. Ensure that the test settings file is available and try again.

instead of
E:Agent_workr10a_XXXXX CI BuilddropXXXXX_Automation_TestbinReleasexxxxxxxxx.testsettings

There is no documentation that states how to formulate the file path on msdn and didn’t see
anything online.

Go to Source
Author: skinnyWill