Appendix C. Continuous Integration Pipeline

Table of Contents

Example setups for Jenkins and GitLab CI pipelines originally based on Maven.gitlab-ci.yml.

C.1. Resources

C.2. GitLab CI Pipeline

Just add a YAML file gitlab-ci.yml to a project to tell the GitLab runner what to do. The latest version can be found at https://gitlab.com/freedumbytes/setup/blob/master/.gitlab-ci.yml.

Important

Some pipeline settings depend on setting from the Maven POM base/parent setup.

C.2.1. Build Stage (test-)compile

The first stage is created to compile the main en test sources and the resulting target and optional generated content is collected as an artifact for the next stage:

+variables:
+  MAVEN_OPTS: "-Djava.awt.headless=true"
+  MAVEN_CLI_OPTS: "--batch-mode --errors --fail-fast --show-version"
+
+
+
+
+
+.maven-compile: &maven-compile
+  stage: build
+  script:
+    - 'mvn $MAVEN_CLI_OPTS test-compile -PopenSource'
+  artifacts:
+    paths:
+      - "target"
+      - "*/target"
+      - "*/*/target"
+      - "src/main/generated"
+      - "*/src/main/generated"
+      - "*/*/src/main/generated"
+    expire_in: 1h
+  tags:
+    - docker
+
+
+
+
+
+maven-compile:jdk8:
+  <<: *maven-compile
+  image: maven:3.5.4-jdk-8

Note

Alas the artifacts cannot be collected recursively. Thus initial setup is for a maven project with 2 levels deep modules hierarchy.

Important

The original template example used --fail-at-end instead of --fail-fast, but this makes the build run longer in case of failure and causing additional logging.

C.2.2. Test Stage deploy

The second stage is created to test and deploy into Nexus the compiled/generated class dependencies from the build stage and saving the test results for the next stage:

+.maven-artifacts: &maven-artifacts
+  stage: test
+  script:
+#    - 'mvn $MAVEN_CLI_OPTS deploy -PopenSource'
+    - 'mvn $MAVEN_CLI_OPTS verify'
+  artifacts:
+    paths:
+      - "target/surefire-reports/*"
+      - "*/target/surefire-reports/*"
+      - "*/*/target/surefire-reports/*"
+      - "target/failsafe-reports/*"
+      - "*/target/failsafe-reports/*"
+      - "*/*/target/failsafe-reports/*"
+      - "target/coverage-reports/*"
+      - "*/target/coverage-reports/*"
+      - "*/*/target/coverage-reports/*"
+    expire_in: 1h
+  tags:
+    - docker


 maven-compile:jdk8:
   <<: *maven-compile
   image: maven:3.5.4-jdk-8
+
+
+
+maven-artifacts:jdk8:
+  <<: *maven-artifacts
+  image: maven:3.5.4-jdk-8
+  dependencies:
+    - maven-compile:jdk8

Note

Used verify instead of deploy since there isn't a secret variable for ossrh configured (yet). Also disabled openSource profile since there isn't a secret variable for gpg.keyname configured (yet).

Important

Instead of only saving surefire/failsafe TEST-*.xml and *.txt files, these paths will also fetch a copy of any [date]-jvmRun[N].dump, date]-jvmRun[N].dumpstream and [date].dumpstream files for artifacts download in case of an error.

C.2.3. Test Stage no recompile - Changes detected

Prevent unnecessary recompiling by maven-compiler-plugin when Changes detected - recompiling the module! due to the absolute .java file path in target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst:

 variables:
+  MAVEN_PREVENT_RECOMPILE: "-Dmaven.compiler.useIncrementalCompilation=false"
   MAVEN_OPTS: "-Djava.awt.headless=true"
-  MAVEN_CLI_OPTS: "--batch-mode --errors --fail-fast --show-version"
+  MAVEN_CLI_OPTS: "--batch-mode --errors --fail-fast --show-version $MAVEN_PREVENT_RECOMPILE"

Note

This happens for example when Jenkins CI runs parallel jobs in different workspace locations (see also MCOMPILER-209).

C.2.4. Test Stage no recompile - Compiling xx source files

Prevent unnecessary recompiling by maven-compiler-plugin when Compiling xx source files instead of Nothing to compile - all classes are up to date due to git clone action in the test job causing the .java file being more recent than the corresponding .class file:

 .maven-artifacts: &maven-artifacts
   stage: test
   script:
+    - 'find . -name "*.class" -exec touch {} \+'
 #    - 'mvn $MAVEN_CLI_OPTS deploy -PopenSource'
     - 'mvn $MAVEN_CLI_OPTS verify'
   artifacts:

Note

This happens for example after upgrading to GitLab CI 9.x.

C.2.5. Test Stage unit vs integration phase

Separate mvn unit and integration tests:

   stage: test
   script:
     - 'find . -name "*.class" -exec touch {} \+'
-#    - 'mvn $MAVEN_CLI_OPTS deploy -PopenSource'
-    - 'mvn $MAVEN_CLI_OPTS verify'
+    - 'mvn $MAVEN_CLI_OPTS test -PopenSource'
+#    - 'mvn $MAVEN_CLI_OPTS deploy -PopenSource,integrationTestsOnly'
+    - 'mvn $MAVEN_CLI_OPTS verify -PintegrationTestsOnly'
   artifacts:
     paths:
       - "target/surefire-reports/*"

Note

Usage of custom profile integrationTestsOnly.

C.2.6. Test Stage unit vs integration job

Separate jobs for unit and integration tests:

-.maven-artifacts: &maven-artifacts
+.maven-unit: &maven-unit
   stage: test
   script:
     - 'find . -name "*.class" -exec touch {} \+'
     - 'mvn $MAVEN_CLI_OPTS test -PopenSource'
-#    - 'mvn $MAVEN_CLI_OPTS deploy -PopenSource,integrationTestsOnly'
-    - 'mvn $MAVEN_CLI_OPTS verify -PintegrationTestsOnly'
   artifacts:
     paths:
       - "target/surefire-reports/*"
       - "*/target/surefire-reports/*"
       - "*/*/target/surefire-reports/*"
+      - "target/coverage-reports/*"
+      - "*/target/coverage-reports/*"
+      - "*/*/target/coverage-reports/*"
+    expire_in: 1h
+  tags:
+    - docker
+
+
+
+.maven-integration-artifacts: &maven-integration-artifacts
+  stage: test
+  script:
+    - 'find . -name "*.class" -exec touch {} \+'
+#    - 'mvn $MAVEN_CLI_OPTS deploy -PopenSource,documents,integrationTestsOnly'
+    - 'mvn $MAVEN_CLI_OPTS verify -Pdocuments,integrationTestsOnly'
+  artifacts:
+    paths:
       - "target/failsafe-reports/*"
       - "*/target/failsafe-reports/*"
       - "*/*/target/failsafe-reports/*"

Note

Also added custom profile documents to attach sources and javadoc artifacts when applicable.

-maven-artifacts:jdk8:
-  <<: *maven-artifacts
+maven-unit:jdk8:
+  <<: *maven-unit
+  image: maven:3.5.4-jdk-8
+  dependencies:
+    - maven-compile:jdk8
+
+
+
+maven-integration-artifacts:jdk8:
+  <<: *maven-integration-artifacts
   image: maven:3.5.4-jdk-8
   dependencies:
     - maven-compile:jdk8

Note

Usage of custom profile integrationTestsOnly.

Important

One could debate that unit and integration tests shouldn't run in the same stage. But the intended integration tests are those between classes which do not required a deployed system. How to run system integration tests will be added at a later date.

C.2.7. GitLab CI Cache

Cache downloaded dependencies and plugins between jobs:

 variables:
   MAVEN_PREVENT_RECOMPILE: "-Dmaven.compiler.useIncrementalCompilation=false"
-  MAVEN_OPTS: "-Djava.awt.headless=true"
+  MAVEN_ALTERNATIVE_REPO_LOCATION: ".m2/repository"
+  MAVEN_OPTS: "-Djava.awt.headless=true -Dmaven.repo.local=$MAVEN_ALTERNATIVE_REPO_LOCATION"
   MAVEN_CLI_OPTS: "--batch-mode --errors --fail-fast --show-version $MAVEN_PREVENT_RECOMPILE"



+cache:
+  paths:
+    - .m2/repository
+  key: "$CI_COMMIT_SHA"

In case of timeouts during caching on the GitLab CI open source server just disable it again:

-  MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository -Djava.awt.headless=true"
+#  MAVEN_ALTERNATIVE_REPO_LOCATION: ".m2/repository"
+#  MAVEN_OPTS: "-Djava.awt.headless=true -Dmaven.repo.local=$MAVEN_ALTERNATIVE_REPO_LOCATION"
+  MAVEN_OPTS: "-Djava.awt.headless=true"
   MAVEN_CLI_OPTS: "--batch-mode --errors --fail-fast --show-version $MAVEN_PREVENT_RECOMPILE"



-cache:
-  paths:
-    - .m2/repository
-  key: "$CI_COMMIT_SHA"
+#cache:
+#  paths:
+#    - "$MAVEN_ALTERNATIVE_REPO_LOCATION"
+#  key: "$CI_COMMIT_SHA"

C.2.8. Show Time

Display time in Maven logging:

 variables:
+  MAVEN_SHOW_TIME: "-Dorg.slf4j.simpleLogger.showDateTime=true"
+  MAVEN_FORMAT_TIME: "-Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS"
   MAVEN_PREVENT_RECOMPILE: "-Dmaven.compiler.useIncrementalCompilation=false"
   MAVEN_ALTERNATIVE_REPO_LOCATION: ".m2/repository"
-  MAVEN_OPTS: "-Djava.awt.headless=true -Dmaven.repo.local=$MAVEN_ALTERNATIVE_REPO_LOCATION"
+  MAVEN_OPTS: "-Djava.awt.headless=true $MAVEN_SHOW_TIME $MAVEN_FORMAT_TIME -Dmaven.repo.local=$MAVEN_ALTERNATIVE_REPO_LOCATION"
   MAVEN_CLI_OPTS: "--batch-mode --errors --fail-fast --show-version $MAVEN_PREVENT_RECOMPILE"

C.2.9. Show Date/Time

Display date and time in Maven logging:

 variables:
   MAVEN_SHOW_TIME: "-Dorg.slf4j.simpleLogger.showDateTime=true"
-  MAVEN_FORMAT_TIME: "-Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS"
+  MAVEN_FORMAT_TIME: "-Dorg.slf4j.simpleLogger.dateTimeFormat=EEE..yyyy-MM-dd...HH:mm:ss.SSS..z"
   MAVEN_SUPPRESS_DOWNLOAD_LOGS: "-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN"
   MAVEN_PREVENT_RECOMPILE: "-Dmaven.compiler.useIncrementalCompilation=false"
   MAVEN_ALTERNATIVE_REPO_LOCATION: ".m2/repository"
   MAVEN_OPTS: "-Djava.awt.headless=true $MAVEN_SHOW_TIME $MAVEN_FORMAT_TIME -Dmaven.repo.local=$MAVEN_ALTERNATIVE_REPO_LOCATION"

Important

The dots between for example date and time are necessary because spaces would cause the following error Error: Could not find or load main class … (see also MNG-4559).

C.2.10. Suppress download messages

Suppress download messages for dependencies and plugins:

 variables:
   MAVEN_SHOW_TIME: "-Dorg.slf4j.simpleLogger.showDateTime=true"
   MAVEN_FORMAT_TIME: "-Dorg.slf4j.simpleLogger.dateTimeFormat=EEE..yyyy-MM-dd...HH:mm:ss.SSS..z"
+  MAVEN_SUPPRESS_DOWNLOAD_LOGS: "-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN"
   MAVEN_PREVENT_RECOMPILE: "-Dmaven.compiler.useIncrementalCompilation=false"
   MAVEN_ALTERNATIVE_REPO_LOCATION: ".m2/repository"
-  MAVEN_OPTS: "-Djava.awt.headless=true $MAVEN_SHOW_TIME $MAVEN_FORMAT_TIME -Dmaven.repo.local=$MAVEN_ALTERNATIVE_REPO_LOCATION"
+  MAVEN_OPTS: "-Djava.awt.headless=true $MAVEN_SHOW_TIME $MAVEN_FORMAT_TIME $MAVEN_SUPPRESS_DOWNLOAD_LOGS -Dmaven.repo.local=$MAVEN_ALTERNATIVE_REPO_LOCATION"
   MAVEN_CLI_OPTS: "--batch-mode --errors --fail-fast --show-version $MAVEN_PREVENT_RECOMPILE"

C.2.11. Deploy Stage site:stage

Generate project site

+.maven-site: &maven-site
+  stage: deploy
+  script:
+    - 'mvn $MAVEN_CLI_OPTS site -PopenSource,enableUpdatesReports'
+    - 'mvn $MAVEN_CLI_OPTS site:stage -PopenSource,enableUpdatesReports'
+  artifacts:
+    paths:
+      - "target/staging"
+    expire_in: 1h
+  tags:
+    - docker
+
+
 maven-compile:jdk8:
   <<: *maven-compile
   image: maven:3.5.4-jdk-8
   image: maven:3.5.4-jdk-8
   dependencies:
     - maven-compile:jdk8
+
+
+
+maven-site:jdk8:
+  <<: *maven-site
+  image: maven:3.5.4-jdk-8
+  dependencies:
+    - maven-compile:jdk8
+    - maven-unit:jdk8
+    - maven-integration-artifacts:jdk8

C.2.12. GitLab CI Deploy Pages Stage

With GitLab Pages you can create static websites for your GitLab projects, groups, or user accounts:

+stages:
+  - build
+  - test
+  - deploy
+  - deploy-pages
+
+
+
 variables:
   MAVEN_SHOW_TIME: "-Dorg.slf4j.simpleLogger.showDateTime=true"
   MAVEN_FORMAT_TIME: "-Dorg.slf4j.simpleLogger.dateTimeFormat=EEE..yyyy-MM-dd...HH:mm:ss.SSS..z"
 maven-site:jdk8:
   <<: *maven-site
   image: maven:3.5.4-jdk-8
   dependencies:
     - maven-compile:jdk8
     - maven-unit:jdk8
     - maven-integration-artifacts:jdk8
+
+
+
+pages:
+  stage: deploy-pages
+  script:
+  - 'mkdir .public'
+  - 'cp -r target/staging/* .public'
+  - 'mv .public public'
+  dependencies:
+    - maven-site:jdk8
+  artifacts:
+    paths:
+    - public
+    expire_in: 1h

C.2.13. Maven Site Staging issue workaround

Sometimes the staged site isn't created in target/staging but next to it in target/<artifactId>:

   script:
     - 'mvn $MAVEN_CLI_OPTS site -PopenSource,enableUpdatesReports'
     - 'mvn $MAVEN_CLI_OPTS site:stage -PopenSource,enableUpdatesReports'
+    - 'rm -r target/site'
+    - 'mv `find ./target -type f -name "team.html" | sed -r "s|/[^/]+$||" | sort -u | head -1` target/pages'
   artifacts:
     paths:
-      - "target/staging"
+      - "target/pages"
     expire_in: 1h
   tags:
     - docker

Note

As a workaround just move the first folder in target containing a team.html report to tarrget/pages.

 pages:
   stage: deploy-pages
   script:
   - 'mkdir .public'
-  - 'cp -r target/staging/* .public'
+  - 'cp -r target/pages/* .public'
   - 'mv .public public'
   dependencies:
     - maven-site:jdk8

C.2.14. GitLab CI Deploy Pages Stage take 2

Alas the pages job should better be in the deploy stage because the extra pages:deploy job entry is placed in that stage anyway:

 stages:
   - build
   - test
+  - docs
   - deploy
-  - deploy-pages
 .maven-site: &maven-site
-  stage: deploy
+  stage: docs
   script:
     - 'mvn $MAVEN_CLI_OPTS site -PopenSource,enableUpdatesReports'
     - 'mvn $MAVEN_CLI_OPTS site:stage -PopenSource,enableUpdatesReports'
 pages:
-  stage: deploy-pages
+  stage: deploy
   script:
   - 'mkdir .public'
   - 'cp -r target/pages/* .public'

C.2.15. Site Generation revisited

Only update the site when running on the master. Also generate the site in case of a failure (requires test/verify artifacts set to when: always):

       - "*/target/coverage-reports/*"
       - "*/*/target/coverage-reports/*"
     expire_in: 1h
+    when:
+      always
   tags:
     - docker
 maven-site:jdk8:
   <<: *maven-site
   image: maven:3.5.4-jdk-8
   dependencies:
     - maven-compile:jdk8
     - maven-unit:jdk8
     - maven-integration-artifacts:jdk8
+  only:
+    - master
+
+
+
+maven-site-failure:jdk8:
+  <<: *maven-site
+  image: maven:3.5.4-jdk-8
+  dependencies:
+    - maven-compile:jdk8
+    - maven-unit:jdk8
+    - maven-integration-artifacts:jdk8
+  artifacts:
+    paths:
+      - "target/pages"
+    expire_in: 1 day
+  when:
+    on_failure
 pages:
     paths:
     - public
     expire_in: 1h
+  only:
+    - master

Note

Alas due to the nature of the pages job it is not possible to deploy the site on failure also (for now just download the generated site artifacts, which will expire in 1 day, to browse the test reports).

C.2.16. Effective Pom

Convenience job to dump the effective-pom in case of failure:

+.maven-effective-pom: &maven-effective-pom
+  stage: docs
+  script:
+    - 'mvn $MAVEN_CLI_OPTS help:effective-pom -PopenSource'
+  tags:
+    - docker
+
+
+
+maven-effective-pom:jdk8:
+  <<: *maven-effective-pom
+  image: maven:3.5.4-jdk-8
+  when:
+    on_failure

C.2.17. Optional JDK Check

Add animal-sniffer and enforcer in case the maven.compiler.target is not the same as the jdk used by the docker image:

     - 'find . -name "*.class" -exec touch {} \+'
 #    - 'mvn $MAVEN_CLI_OPTS deploy -PopenSource,documents,integrationTestsOnly'
     - 'mvn $MAVEN_CLI_OPTS verify -Pdocuments,integrationTestsOnly'
+    - 'mvn $MAVEN_CLI_OPTS animal-sniffer:check -PopenSource'
+    - 'mvn $MAVEN_CLI_OPTS enforcer:enforce -PopenSource'
   artifacts:
     paths:
       - "target/failsafe-reports/*"

Note

TODO see JModalWindow project.

C.2.18. Optional OWASP Test Stage

Add dependency-check to master branch only:

+.maven-owasp: &maven-owasp
+  stage: test
+  script:
+    - 'mvn $MAVEN_CLI_OPTS org.owasp:dependency-check-maven:aggregate -PopenSource'
+  artifacts:
+    paths:
+      - "target/owasp-reports/dependency-check-report.xml"
+    expire_in: 1h
+  tags:
+    - docker
+  only:
+    - master
+
+
+
+maven-owasp:jdk8:
+  <<: *maven-owasp
+  image: maven:3.5.4-jdk-8
+  dependencies:
+    - maven-compile:jdk8

C.2.19. Optional SonarQube Docs stage

Add sonar to master branch only:

+.maven-sonar: &maven-sonar
+  stage: docs
+  script:
+    - 'mvn $MAVEN_CLI_OPTS sonar:sonar -PopenSource,sonarCloud'
+  tags:
+    - docker
+  only:
+    - master
+
+
+
+#maven-sonar:jdk8:
+#  <<: *maven-sonar
+#  image: maven:3.5.4-jdk-8
+#  dependencies:
+#    - maven-unit:jdk8
+#    - maven-integration-artifacts:jdk8
+#    - maven-owasp:jdk8

Note

Currently disabled since there isn't a secret variable for sonar.login configured (yet).

C.2.20. SonarQube Dashboard issue workaround

Maven report link to SonarQube is no longer working (see also issue 6 and Section H.3.4, “Open Source Patches”):

   stage: docs
   script:
     - 'mvn $MAVEN_CLI_OPTS site -PopenSource,enableUpdatesReports'
+    - 'find . -type f -name "sonar.html" | xargs --no-run-if-empty sed -i "s/\/project\/index\//\/dashboard\/index\//g"'
     - 'mvn $MAVEN_CLI_OPTS site:stage -PopenSource,enableUpdatesReports'
     - 'rm -r target/site'
     - 'mv `find ./target -type f -name "team.html" | sed -r "s|/[^/]+$||" | sort -u | head -1` target/pages'

C.2.21. Optional OWASP Docs Stage

When running the Dependency Check Aggregate on a completely new version of a multi-module project, it fails (TODO reword) when a referenced submodule isn't available in the local or remote repository already with the following error [ERROR] Could not find artifact …. So move it from test to docs stage while possibly reducing any build time bonus:

-.maven-owasp: &maven-owasp
-  stage: test
-  script:
-    - 'mvn $MAVEN_CLI_OPTS org.owasp:dependency-check-maven:aggregate -PopenSource'
-  artifacts:
-    paths:
-      - "target/owasp-reports/dependency-check-report.xml"
-    expire_in: 1h
-  tags:
-    - docker
-  only:
-    - master
-
-
-
 .maven-sonar: &maven-sonar
   stage: docs
   script:
+    - 'mvn $MAVEN_CLI_OPTS org.owasp:dependency-check-maven:aggregate -PopenSource'
     - 'mvn $MAVEN_CLI_OPTS sonar:sonar -PopenSource,sonarCloud'
   tags:
     - docker
-maven-owasp:jdk8:
-  <<: *maven-owasp
-  image: maven:3.5.4-jdk-8
-  dependencies:
-    - maven-compile:jdk8
-
-
-
 #maven-sonar:jdk8:
 #  <<: *maven-sonar
 #  image: maven:3.5.4-jdk-8
 #  dependencies:
 #    - maven-unit:jdk8
 #    - maven-integration-artifacts-deploy:jdk8
-#    - maven-owasp:jdk8

C.2.22. Artifacts verify vs deploy

On branches run verify only instead of deploy because it is otherwise no longer clear which SNAPSHOT artifacts will be downloaded from Nexus (the latest master or branch build):

 .maven-integration-artifacts: &maven-integration-artifacts
   stage: test
   script:
     - 'find . -name "*.class" -exec touch {} \+'
-#    - 'mvn $MAVEN_CLI_OPTS deploy -PopenSource,documents,integrationTestsOnly'
-    - 'mvn $MAVEN_CLI_OPTS verify -Pdocuments,integrationTestsOnly'
+#    - 'mvn $MAVEN_CLI_OPTS $MVN_PHASE -PopenSource,documents,integrationTestsOnly'
+    - 'mvn $MAVEN_CLI_OPTS $MVN_PHASE -Pdocuments,integrationTestsOnly'
     - 'mvn $MAVEN_CLI_OPTS animal-sniffer:check -PopenSource'
     - 'mvn $MAVEN_CLI_OPTS enforcer:enforce -PopenSource'
   artifacts:
-maven-integration-artifacts:jdk8:
+maven-integration-artifacts-verify:jdk8:
   <<: *maven-integration-artifacts
   image: maven:3.5.4-jdk-8
+  variables:
+    MVN_PHASE: "verify"
   dependencies:
     - maven-compile:jdk8
+  except:
+    - master
+    - tags
+
+
+
+maven-integration-artifacts-deploy:jdk8:
+  <<: *maven-integration-artifacts
+  image: maven:3.5.4-jdk-8
+  variables:
+#    MVN_PHASE: "deploy"
+    MVN_PHASE: "verify"
+  dependencies:
+    - maven-compile:jdk8
+  only:
+    - master
+    - tags
 #maven-sonar:jdk8:
 #  image: maven:3.5.4-jdk-8
 #  dependencies:
 #    - maven-unit:jdk8
-#    - maven-integration-artifacts:jdk8
+#    - maven-integration-artifacts-deploy:jdk8
 #    - maven-owasp:jdk8
 maven-site:jdk8:
   <<: *maven-site
   image: maven:3.5.4-jdk-8
   dependencies:
     - maven-compile:jdk8
     - maven-unit:jdk8
-    - maven-integration-artifacts:jdk8
+    - maven-integration-artifacts-deploy:jdk8
   only:
     - master
 maven-site-failure:jdk8:
   <<: *maven-site
   image: maven:3.5.4-jdk-8
   dependencies:
     - maven-compile:jdk8
     - maven-unit:jdk8
-    - maven-integration-artifacts:jdk8
+    - maven-integration-artifacts-verify:jdk8
+    - maven-integration-artifacts-deploy:jdk8
   artifacts:
     paths:
       - "target/pages"

After this change the Animal Sniffer Check fails for multi-module projects, when a referenced submodule isn't available in the local or remote repository already with the following error [ERROR] Failed to execute goal on project sample: Could not resolve dependencies for project … which is Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact … or [ERROR] … Undefined reference: …. So switch from verify to install:

   variables:
-    MVN_PHASE: "verify"
+    MVN_PHASE: "install"

C.2.23. Artifacts verify on fork

For the same reason as branches in the prior paragraph run verify only for master and tags on forks by adding the group and the project of for example https://gitlab.com/freedumbytes/setup to all only and except options:

-    - master
-    - tags
+    - master@freedumbytes/setup
+    - tags@freedumbytes/setup

C.2.24. Javadoc Reports

Enable the Javadoc Reports for the generated project site:

 .maven-site: &maven-site
   stage: docs
   script:
-    - 'mvn $MAVEN_CLI_OPTS site -PopenSource,enableUpdatesReports'
+    - 'mvn $MAVEN_CLI_OPTS site -PopenSource,enableUpdatesReports,enableJavadocReports'
     - 'find . -type f -name "sonar.html" | xargs --no-run-if-empty sed -i "s/\/project\/index\//\/dashboard\/index\//g"'
-    - 'mvn $MAVEN_CLI_OPTS site:stage -PopenSource,enableUpdatesReports'
+    - 'mvn $MAVEN_CLI_OPTS site:stage -PopenSource,enableUpdatesReports,enableJavadocReports'
     - 'rm -r target/site'
     - 'mv `find ./target -type f -name "team.html" | sed -r "s|/[^/]+$||" | sort -u | head -1` target/pages'
   artifacts:

C.2.25. Dependency Check Report

Enable the Dependency Check Reports for the generated project site, because SonarCloud isn't using that information yet:

 .maven-site: &maven-site
   stage: docs
   script:
-    - 'mvn $MAVEN_CLI_OPTS site -PopenSource,enableUpdatesReports,enableJavadocReports'
+    - 'mvn $MAVEN_CLI_OPTS site -PopenSource,enableUpdatesReports,enableJavadocReports,enableDependencyCheckReport'
     - 'find . -type f -name "sonar.html" | xargs --no-run-if-empty sed -i "s/\/project\/index\//\/dashboard\/index\//g"'
-    - 'mvn $MAVEN_CLI_OPTS site:stage -PopenSource,enableUpdatesReports,enableJavadocReports'
+    - 'mvn $MAVEN_CLI_OPTS site:stage -PopenSource,enableUpdatesReports,enableJavadocReports,enableDependencyCheckReport'
     - 'rm -r target/site'
     - 'mv `find ./target -type f -name "team.html" | sed -r "s|/[^/]+$||" | sort -u | head -1` target/pages'
   artifacts:

C.2.26. Site Generation on release only

Optionally switch to updating the generated project site on release only:

 maven-site:job:
   <<: *maven-site
   image: $DOCKER_IMAGE
   dependencies:
     - maven-compile:job
     - maven-unit:job
     - maven-integration-artifacts-deploy:job
   only:
-     - master@freedumbytes/setup
+#     - master@freedumbytes/setup
+     - tags@freedumbytes/setup
 pages:
   stage: deploy
   script:
   - 'mkdir .public'
   - 'cp -r target/pages/* .public'
   - 'mv .public public'
   dependencies:
     - maven-site:job
   artifacts:
     paths:
     - public
     expire_in: 1h
   only:
-     - master@freedumbytes/setup
+#     - master@freedumbytes/setup
+     - tags@freedumbytes/setup

C.2.27. Docker Image

When only building for a certain jdk version anyway, just create a docker image environment variable and thus requiring only one change to upgrade the script:

stages:


 variables:
+  DOCKER_IMAGE: "maven:3.5.4-jdk-8"
-maven-integration-artifacts-deploy:jdk8:
+maven-integration-artifacts-deploy:job:
   <<: *maven-integration-artifacts
-  image: maven:3.5.4-jdk-8
+  image: $DOCKER_IMAGE
   variables:
 #    MVN_PHASE: "deploy"
     MVN_PHASE: "install"
   dependencies:
-    - maven-compile:jdk8
+    - maven-compile:job
   only:
     - master@freedumbytes/setup
     - tags@freedumbytes/setup

Should the project be configured to create jdk7 code for example with default POM properties overrides for compiler, animal-sniffer, enforcer and javadoc plugins:

  <properties>
    <maven.compiler.compilerVersion>1.8</maven.compiler.compilerVersion>
    <maven.compiler.source>${maven.compiler.target}</maven.compiler.source>
    <maven.compiler.target>1.7</maven.compiler.target>
    <maven.compiler.testSource>${maven.compiler.target}</maven.compiler.testSource>
    <maven.compiler.testTarget>${maven.compiler.target}</maven.compiler.testTarget>
    <!--
      Requires Java9+, otherwise: 'mvn compile' results in "javac: invalid flag: --release".
      <maven.compiler.release>7</maven.compiler.release>
    -->

    <mojoSignatureArtifactId>java17</mojoSignatureArtifactId>
    <ignoreDependencies>true</ignoreDependencies>

    <extraEnforcerRulesMaxJdkVersion>${maven.compiler.target}</extraEnforcerRulesMaxJdkVersion>
    <extraEnforcerRulesFail>true</extraEnforcerRulesFail>

    <javadocVersion>1.7.0</javadocVersion>
  </properties>

It is now possible to switch to a lower version of the jdk in the above configured DOCKER_IMAGE:

stages:


 variables:
-  DOCKER_IMAGE: "maven:3.5.4-jdk-8"
+  DOCKER_IMAGE: "maven:3.5.4-jdk-7"
+  DOCKER_IMAGE_SONAR: "maven:3.5.4-jdk-8"

This way we don't run the risk of the animal-sniffer and enforcer missing the issue of TODO.

C.3. Jenkins CI Pipeline

Just add a file Jenkinsfile to a project to tell the Jenkins pipeline what to do. The latest version can be found at https://gitlab.com/freedumbytes/setup/blob/master/Jenkinsfile.

Important

Some pipeline settings depend on setting from the Maven POM base/parent setup.

C.3.1. Build Stage (test-)compile

The first stage is created to compile the main en test sources and the resulting target and optional generated content is collected as an artifact for the next stage:

+pipeline
+{
+  agent
+  {
+    label "windows"
+  }
+
+
+
+  environment
+  {
+    MAVEN_OPTS = "-Djava.awt.headless=true"
+    MAVEN_CLI_OPTS = "--batch-mode --errors --fail-fast --show-version"
+  }
+
+
+
+  stages
+  {
+    stage("build")
+    {
+      steps
+      {
+        mvnCompileNode nodeName: "windows", profile: "", branchNames: []
+      }
+    }
+  }
+}
+
+
+
+
+
+def mvnCompileNode(Map args)
+{
+  node("${args.nodeName}")
+  {
+    stage("compile")
+    {
+      echo "Clean Up..."
+      deleteDir()
+
+      echo "Check Out..."
+      checkout scm
+      stash name: "sourcecode", includes: "**", excludes: "**/target/**"
+
+      echo "Compile Source..."
+      sh "mvn $MAVEN_CLI_OPTS test-compile ${args.profile}"
+      stash name: "bytecode", includes: "**/target/**,**/generated/**", allowEmpty: true
+    }
+  }
+}

Note

Alas the artifacts cannot be collected recursively. Thus initial setup is for a maven project with 2 levels deep modules hierarchy.

Important

The original template example used --fail-at-end instead of --fail-fast, but this makes the build run longer in case of failure and causing additional logging.

C.3.2. Test Stage deploy

The second stage is created to test and deploy into Nexus the compiled/generated class dependencies from the build stage and saving the test results for the next stage:

         mvnCompileNode nodeName: "windows", profile: "", branchNames: []
       }
     }
+
+    stage("test")
+    {
+      steps
+      {
+        mvnArtifactsNode nodeName: "windows", profile: "", branchNames: []
+      }
+    }
   }
 }
+def mvnArtifactsNode(Map args)
+{
+  node("${args.nodeName}")
+  {
+    stage("artifacts")
+    {
+      echo "Clean Up..."
+      deleteDir()
+
+      echo "Source Code..."
+      unstash "sourcecode"
+
+      echo "Bytecode..."
+      unstash "bytecode"
+
+      try
+      {
+        echo "Artifacts..."
+        sh "mvn $MAVEN_CLI_OPTS deploy ${args.profile}"
+      }
+      finally
+      {
+        stash name: "testresults", includes: "**/target/surefire-reports/*,**/target/failsafe-reports/*,**/target/coverage-reports/*", allowEmpty: true
+      }
+    }
+  }
+}

Note

Used verify instead of deploy since there isn't a secret variable for ossrh configured (yet). Also disabled openSource profile since there isn't a secret variable for gpg.keyname configured (yet).

C.3.3. Test Stage unit vs integration phase

Separate mvn unit and integration tests:

       try
       {
-        echo "Artifacts..."
-        sh "mvn $MAVEN_CLI_OPTS deploy ${args.profile}"
+        echo "Unit Test..."
+        sh "mvn $MAVEN_CLI_OPTS test ${args.profile}"
+
+        echo "Integration Test..."
+        sh "mvn $MAVEN_CLI_OPTS deploy -PintegrationTestsOnly"
       }
       finally
       {
-        stash name: "testresults", includes: "**/target/surefire-reports/*,**/target/failsafe-reports/*,**/target/coverage-reports/*", allowEmpty: true
+        stash name: "unittests", includes: "**/target/surefire-reports/*,**/target/coverage-reports/*", allowEmpty: true
+        stash name: "integrationtests", includes: "**/target/failsafe-reports/*,**/target/coverage-reports/*", allowEmpty: true
       }
     }
   }

Note

Usage of custom profile integrationTestsOnly.

C.3.4. Test Stage unit vs integration job

Separate jobs for unit and integration tests:

     {
       steps
       {
-        mvnArtifactsNode nodeName: "windows", profile: "", branchNames: []
+        parallel (
+          "unit":
+          {
+            mvnUnitTestNode nodeName: "windows", profile: "", branchNames: []
+          },
+
+          "integration-artifacts":
+          {
+            mvnIntegrationTestNode nodeName: "windows", profile: "-Pdocuments,integrationTestsOnly", branchNames: ['master']
+          }
+        )
       }
     }
   }

Note

Also added custom profile documents to attach sources and javadoc artifacts when applicable.

-def mvnArtifactsNode(Map args)
+def mvnUnitTestNode(Map args)
 {
   node("${args.nodeName}")
   {
-    stage("artifacts")
+    stage("unit")
     {
       echo "Clean Up..."
       deleteDir()
       {
         echo "Unit Test..."
         sh "mvn $MAVEN_CLI_OPTS test ${args.profile}"
+      }
+      finally
+      {
+        stash name: "unittests", includes: "**/target/surefire-reports/*,**/target/coverage-reports/*", allowEmpty: true
+      }
+    }
+  }
+}
+
+
+
+def mvnIntegrationTestNode(Map args)
+{
+  node("${args.nodeName}")
+  {
+    stage("integration-artifacts")
+    {
+      echo "Clean Up..."
+      deleteDir()
+
+      echo "Source Code..."
+      unstash "sourcecode"

+      echo "Bytecode..."
+      unstash "bytecode"
+
+      try
+      {
         echo "Integration Test..."
-        sh "mvn $MAVEN_CLI_OPTS deploy -PintegrationTestsOnly"
+        sh "mvn $MAVEN_CLI_OPTS deploy ${args.profile}"
       }
       finally
       {
-        stash name: "unittests", includes: "**/target/surefire-reports/*,**/target/coverage-reports/*", allowEmpty: true
         stash name: "integrationtests", includes: "**/target/failsafe-reports/*,**/target/coverage-reports/*", allowEmpty: true
       }
     }

Note

Usage of custom profile integrationTestsOnly.

Important

One could debate that unit and integration tests shouldn't run in the same stage. But the intended integration tests are those between classes which do not required a deployed system. How to run system integration tests will be added at a later date.

C.3.5. Test Stage no recompile - Changes detected

Prevent unnecessary recompiling by maven-compiler-plugin when Changes detected - recompiling the module! due to the absolute .java file path in target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst:

   environment
   {
+    MAVEN_PREVENT_RECOMPILE = "-Dmaven.compiler.useIncrementalCompilation=false"
     MAVEN_OPTS = "-Djava.awt.headless=true"
-    MAVEN_CLI_OPTS = "--batch-mode --errors --fail-fast --show-version"
+    MAVEN_CLI_OPTS = "--batch-mode --errors --fail-fast --show-version ${MAVEN_PREVENT_RECOMPILE}"
   }

Note

This happens for example when Jenkins CI runs parallel jobs in different workspace locations (see also MCOMPILER-209).

C.3.6. Show Time

Display time in Maven logging:

   environment
   {
+    MAVEN_SHOW_TIME = "-Dorg.slf4j.simpleLogger.showDateTime=true"
+    MAVEN_FORMAT_TIME = "-Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS"
     MAVEN_PREVENT_RECOMPILE = "-Dmaven.compiler.useIncrementalCompilation=false"
-    MAVEN_OPTS = "-Djava.awt.headless=true"
+    MAVEN_OPTS = "-Djava.awt.headless=true ${MAVEN_SHOW_TIME} ${MAVEN_FORMAT_TIME}"
     MAVEN_CLI_OPTS = "--batch-mode --errors --fail-fast --show-version ${MAVEN_PREVENT_RECOMPILE}"
   }

C.3.7. Show Date/Time

Display date and time in Maven logging:

   environment
   {
     MAVEN_SHOW_TIME = "-Dorg.slf4j.simpleLogger.showDateTime=true"
-    MAVEN_FORMAT_TIME = "-Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss.SSS"
+    MAVEN_FORMAT_TIME = "-Dorg.slf4j.simpleLogger.dateTimeFormat=EEE..yyyy-MM-dd...HH:mm:ss.SSS..z"
     MAVEN_SUPPRESS_DOWNLOAD_LOGS = "-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN"
     MAVEN_PREVENT_RECOMPILE = "-Dmaven.compiler.useIncrementalCompilation=false"
     MAVEN_OPTS = "-Djava.awt.headless=true ${MAVEN_SHOW_TIME} ${MAVEN_FORMAT_TIME} ${MAVEN_SUPPRESS_DOWNLOAD_LOGS}"

Important

The dots between for example date and time are necessary because spaces would cause the following error Error: Could not find or load main class … (see also MNG-4559).

C.3.8. Suppress download messages

Suppress download messages for dependencies and plugins:

   environment
   {
     MAVEN_SHOW_TIME = "-Dorg.slf4j.simpleLogger.showDateTime=true"
     MAVEN_FORMAT_TIME = "-Dorg.slf4j.simpleLogger.dateTimeFormat=EEE..yyyy-MM-dd...HH:mm:ss.SSS..z"
+    MAVEN_SUPPRESS_DOWNLOAD_LOGS = "-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN"
     MAVEN_PREVENT_RECOMPILE = "-Dmaven.compiler.useIncrementalCompilation=false"
-    MAVEN_OPTS = "-Djava.awt.headless=true ${MAVEN_SHOW_TIME} ${MAVEN_FORMAT_TIME}"
+    MAVEN_OPTS = "-Djava.awt.headless=true ${MAVEN_SHOW_TIME} ${MAVEN_FORMAT_TIME} ${MAVEN_SUPPRESS_DOWNLOAD_LOGS}"
     MAVEN_CLI_OPTS = "--batch-mode --errors --fail-fast --show-version ${MAVEN_PREVENT_RECOMPILE}"
   }

C.3.9. Deploy Stage site-deploy

Generate project site

+    stage("deploy")
+    {
+      steps
+      {
+        mvnSiteNode nodeName: "windows", profile: "-PenableUpdatesReports", branchNames: ['master']
+      }
+    }
+def mvnSiteNode(Map args)
+{
+  node("${args.nodeName}")
+  {
+    stage("site")
+    {
+      if (args.branchNames.contains(env.BRANCH_NAME))
+      {
+        echo "Clean Up..."
+        deleteDir()
+
+        echo "Source Code..."
+        unstash "sourcecode"
+
+        echo "Bytecode..."
+        unstash "bytecode"
+
+        echo "Testresults..."
+        unstash "unittests"
+        unstash "integrationtests"
+
+        echo "Site Deployment..."
+        sh "mvn $MAVEN_CLI_OPTS site-deploy ${args.profile}"
+      }
+      else
+      {
+        echo "Skip Site Deployment for branch '${env.BRANCH_NAME}'."
+      }
+    }
+  }
+}

C.3.10. Jenkins CI Test Report

Reduce the need to look through the console logging in case of test failures:

+      post
+      {
+        always
+        {
+          jenkinsTestReportNode nodeName: "windows", profile: "", branchNames: []
+        }
+      }
+def jenkinsTestReportNode(Map args)
+{
+  node("${args.nodeName}")
+  {
+    stage("test-report")
+    {
+      echo "Clean Up..."
+      deleteDir()
+
+      echo "Source Code..."
+      unstash "sourcecode"
+
+      echo "Bytecode..."
+      unstash "bytecode"
+
+      echo "Testresults..."
+      unstash "unittests"
+      unstash "integrationtests"
+
+      junit testResults: "**/target/*-reports/*.xml", allowEmptyResults: true
+    }
+  }
+}

Note

This is different from the GitLab CI solution where the Maven project site is generated for user download to browse the test reports.

C.3.11. Effective Pom

Convenience job to dump the effective-pom in case of failure:

+  post
+  {
+    failure
+    {
+      mvnEffectivePom nodeName: "windows", profile: "", branchNames: []
+    }
+  }
+def mvnEffectivePom(Map args)
+{
+  node("${args.nodeName}")
+  {
+    stage("pom")
+    {
+      echo "Clean Up..."
+      deleteDir()
+
+      echo "Source Code..."
+      unstash "sourcecode"
+
+      echo "Effective POM..."
+      sh "mvn $MAVEN_CLI_OPTS help:effective-pom ${args.profile}"
+    }
+  }
+}

C.3.12. Optional JDK Check

Add animal-sniffer and enforcer in case the maven.compiler.target is not the same as the jdk used by the docker image:

       {
         echo "Integration Test..."
         sh "mvn $MAVEN_CLI_OPTS deploy ${args.profile}"
+        sh "mvn $MAVEN_CLI_OPTS animal-sniffer:check ${args.profile}"
+        sh "mvn $MAVEN_CLI_OPTS enforcer:enforce ${args.profile}"
       }
       finally
       {

Note

TODO see JModalWindow project.

C.3.13. Optional OWASP Test Stage

Add dependency-check to master branch only:

           "integration-artifacts":
           {
             mvnIntegrationTestNode nodeName: "windows", profile: "-Pdocuments,integrationTestsOnly", branchNames: ['master']
+          },
+
+          "dependency-check":
+          {
+            mvnOWASPNode nodeName: "windows", profile: "", branchNames: ['master']
           }
         )
       }
+def mvnOWASPNode(Map args)
+{
+  node("${args.nodeName}")
+  {
+    stage("owasp")
+    {
+      if (args.branchNames.contains(env.BRANCH_NAME))
+      {
+        echo "Clean Up..."
+        deleteDir()
+
+        echo "Source Code..."
+        unstash "sourcecode"
+
+        echo "Dependency Check..."
+        sh "mvn $MAVEN_CLI_OPTS org.owasp:dependency-check-maven:aggregate ${args.profile}"
+        stash name: "owaspreport", includes: "**/target/owasp-reports/dependency-check-report.xml", allowEmpty: true
+      }
+      else
+      {
+        echo "Skip Dependency Check for branch '${env.BRANCH_NAME}'."
+      }
+    }
+  }
+}

C.3.14. Optional SonarQube Deploy stage

Add sonar to master branch only:

     {
       steps
       {
-        mvnSiteNode nodeName: "windows", profile: "-PenableUpdatesReports", branchNames: ['master']
+        parallel (
+          "quality-assurance":
+          {
+            mvnSonarQubeNode nodeName: "windows", profile: "", branchNames: ['master']
+          },
+
+          "documentation":
+          {
+            mvnSiteNode nodeName: "windows", profile: "-PenableUpdatesReports", branchNames: ['master']
+          }
+        )
       }
     }
   }
+def mvnSonarQubeNode(Map args)
+{
+  node("${args.nodeName}")
+  {
+    stage("sonarqube")
+    {
+      if (args.branchNames.contains(env.BRANCH_NAME))
+      {
+        echo "Clean Up..."
+        deleteDir()
+
+        echo "Source Code..."
+        unstash "sourcecode"
+
+        echo "Bytecode..."
+        unstash "bytecode"
+
+        echo "Testresults..."
+        unstash "unittests"
+        unstash "integrationtests"
+
+        echo "Dependency Check..."
+        unstash "owaspreport"
+
+        echo "Quality Assurance..."
+        sh "mvn $MAVEN_CLI_OPTS sonar:sonar ${args.profile}"
+      }
+      else
+      {
+        echo "Skip SonarQube for branch '${env.BRANCH_NAME}'."
+      }
+    }
+  }
+}

C.3.15. Optional OWASP Docs Stage

When running the Dependency Check Aggregate on a completely new version of a multi-module project, it fails when a referenced submodule isn't available in the local or remote repository already with the following error [ERROR] Could not find artifact …. So move it from test to docs stage while possibly reducing any build time bonus:

           {
             mvnIntegrationTestNode nodeName: "windows", profile: "-Pdocuments,integrationTestsOnly", branchNames: ['master']
           },
-
-          "dependency-check":
-          {
-            mvnOWASPNode nodeName: "windows", profile: "", branchNames: ['master']
-          }
         )
       }
-def mvnOWASPNode(Map args)
-{
-  node("${args.nodeName}")
-  {
-    stage("owasp")
-    {
-      if (args.branchNames.contains(env.BRANCH_NAME))
-      {
-        echo "Clean Up..."
-        deleteDir()
-
-        echo "Check Out..."
-        checkout scm
-
-        echo "Dependency Check..."
-        sh "mvn $MAVEN_CLI_OPTS org.owasp:dependency-check-maven:aggregate ${args.profile}"
-        stash name: "owaspreport", includes: "**/target/owasp-reports/dependency-check-report.xml", allowEmpty: true
-      }
-      else
-      {
-        echo "Skip Dependency Check for branch '${env.BRANCH_NAME}'."
-      }
-    }
-  }
-}
-
-
-
 def mvnSonarQubeNode(Map args)
 {
         …
         unstash "integrationtests"

         echo "Dependency Check..."
-        unstash "owaspreport"
+        sh "mvn $MAVEN_CLI_OPTS org.owasp:dependency-check-maven:aggregate ${args.profile}"

         echo "Quality Assurance..."
         sh "mvn $MAVEN_CLI_OPTS sonar:sonar ${args.profile}"

C.3.16. Jenkins CI (un)stash vs checkout scm

Use checkout scm instead of (un)stash "sourcecode" to reduce storage usage:

       echo "Check Out..."
       checkout scm
-      stash name: "sourcecode", includes: "**", excludes: "**/target/**"

       echo "Compile Source..."
       sh "mvn $MAVEN_CLI_OPTS test-compile ${args.profile}"
       echo "Clean Up..."
       deleteDir()

-      echo "Source Code..."
-      unstash "sourcecode"
+      echo "Check Out..."
+      checkout scm

       echo "Bytecode..."
       unstash "bytecode"
       echo "Clean Up..."
       deleteDir()

-      echo "Source Code..."
-      unstash "sourcecode"
+      echo "Check Out..."
+      checkout scm

       echo "Bytecode..."
       unstash "bytecode"
       echo "Clean Up..."
       deleteDir()

-      echo "Source Code..."
-      unstash "sourcecode"
-
-      echo "Bytecode..."
-      unstash "bytecode"
-
       echo "Testresults..."
       unstash "unittests"
       unstash "integrationtests"
         echo "Clean Up..."
         deleteDir()

-        echo "Source Code..."
-        unstash "sourcecode"
+        echo "Check Out..."
+        checkout scm

         echo "Bytecode..."
         unstash "bytecode"
         echo "Clean Up..."
         deleteDir()

-        echo "Source Code..."
-        unstash "sourcecode"
+        echo "Check Out..."
+        checkout scm

         echo "Bytecode..."
         unstash "bytecode"
       echo "Clean Up..."
       deleteDir()

-      echo "Source Code..."
-      unstash "sourcecode"
+      echo "Check Out..."
+      checkout scm

       echo "Effective POM..."
       sh "mvn $MAVEN_CLI_OPTS help:effective-pom ${args.profile}"

C.3.17. Test Stage no recompile - Compiling xx source files

Prevent unnecessary recompiling by maven-compiler-plugin when Compiling xx source files instead of Nothing to compile - all classes are up to date due to git clone action in the test job causing the .java file being more recent than the corresponding .class file:

       echo "Check Out..."
       checkout scm

       echo "Bytecode..."
       unstash "bytecode"
+      sh "find . -name \"*.class\" -exec touch {} \\+"

Note

This happens after dropping unstash "sourcecode" in favor of checkout scm.

C.3.18. Extra Open Source Stage

Because GitLab CI isn't configured yet to deploy the artifacts to Sonatype Open Source Software Repository Hosting and to SonarSource SonarCloud Continuous Code Quality let the local hosted Jenkins CI do this also with an extra stage:

+    stage("open-source")
+    {
+      steps
+      {
+        parallel (
+          "quality-assurance":
+          {
+            mvnSonarQubeNode nodeName: "windows", profile: "-PopenSource,sonarCloud", branchNames: ['master']
+          },
+
+          "artifacts":
+          {
+            mvnIntegrationTestNode nodeName: "windows", profile: "-DskipTests -PopenSource,documents", branchNames: ['master']
+          }
+        )
+      }
+    }

Note

The sonarCloud profile is configured in settings.xml with the sonar.login token for https://sonarcloud.io/.

C.3.19. Artifacts verify vs deploy

On branches run verify only instead of deploy because it is otherwise no longer clear which SNAPSHOT artifacts will be downloaded from Nexus (the latest master or branch build):

       try
       {
         echo "Integration Test..."
-        sh "mvn $MAVEN_CLI_OPTS deploy ${args.profile}"
+
+        if (args.branchNames.contains(env.BRANCH_NAME))
+        {
+          sh "mvn $MAVEN_CLI_OPTS deploy ${args.profile}"
+        }
+        else
+        {
+          sh "mvn $MAVEN_CLI_OPTS verify ${args.profile}"
+        }
+
         sh "mvn $MAVEN_CLI_OPTS animal-sniffer:check ${args.profile}"
         sh "mvn $MAVEN_CLI_OPTS enforcer:enforce ${args.profile}"
       }

After this change the Animal Sniffer Check fails for multi-module projects, when a referenced submodule isn't available in the local or remote repository already with the following error [ERROR] Failed to execute goal on project sample: Could not resolve dependencies for project … which is Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact …. So switch from verify to install:

         else
         {
-          sh "mvn $MAVEN_CLI_OPTS verify ${args.profile}"
+          sh "mvn $MAVEN_CLI_OPTS install ${args.profile}"
         }

         sh "mvn $MAVEN_CLI_OPTS animal-sniffer:check ${args.profile}"

C.3.20. Javadoc Reports

Enable the Javadoc Reports for the generated project site:

           "documentation":
           {
-            mvnSiteNode nodeName: "windows", profile: "-PenableUpdatesReports", branchNames: ['master']
+            mvnSiteNode nodeName: "windows", profile: "-PenableUpdatesReports,enableJavadocReports", branchNames: ['master']
           }

C.3.21. Release Tags

Just as for the 'master' branch also deploy the artifacts to Nexus for release tags, because Jenkins CI doesn't have a tags notion as GitLab CI:

   environment
   {
     …
     MAVEN_CLI_OPTS = "--batch-mode --errors --fail-fast --show-version ${MAVEN_PREVENT_RECOMPILE}"
+
+    COMMIT_HASH = sh(returnStdout: true, script: "git log -1 --pretty=format:'%H'").trim()
+    COMMIT_HASH_SHORT = sh(returnStdout: true, script: "git log -1 --pretty=format:'%h'").trim()
+    COMMIT_MESSAGE = sh(returnStdout: true, script: "git log -1 --pretty=format:'%B'").trim()
+    COMMIT_COMMITER = sh(returnStdout: true, script: "git log -1 --pretty=format:'%an'").trim()
   }
 def mvnCompileNode(Map args)
 {
       …

       echo "Check Out..."
       checkout scm

+      echo "Commit Hash: ${env.COMMIT_HASH}"
+      echo "Commit Hash Short: ${env.COMMIT_HASH_SHORT}"
+      echo "Commit Message: '${env.COMMIT_MESSAGE}'"
+      echo "Committer: ${env.COMMIT_COMMITER}"
+
+      if (hasReleaseTag())
+      {
+        echo "Commit Tags: '${env.COMMIT_TAGS}'"
+      }
+
       echo "Compile Source..."
 def mvnIntegrationTestNode(Map args)
 {
       …

       {
         echo "Integration Test..."

-        if (args.branchNames.contains(env.BRANCH_NAME))
+        if (args.branchNames.contains(env.BRANCH_NAME) || hasReleaseTag())
         {
           sh "mvn $MAVEN_CLI_OPTS deploy ${args.profile}"
         }
 def mvnSonarQubeNode(Map args)
 {
   node("${args.nodeName}")
   {
     stage("sonarqube")
     {
-      if (args.branchNames.contains(env.BRANCH_NAME))
+      if (args.branchNames.contains(env.BRANCH_NAME) || hasReleaseTag())
       {
         …

         echo "Quality Assurance..."
         sh "mvn $MAVEN_CLI_OPTS sonar:sonar ${args.profile}"
+def boolean hasReleaseTag() {
+  if (!env.COMMIT_TAGS)
+  {
+    env.COMMIT_TAGS = sh(returnStdout: true, script: "git show-ref --tags -d | grep '^${COMMIT_HASH_SHORT}' | sed -e 's,.* refs/tags/,,' -e 's/\\^{}//'").trim()
+  }
+
+  return (env.COMMIT_TAGS != "" && env.COMMIT_MESSAGE.contains("[maven-release-plugin] prepare release"))
+}

Note

A tag is marked as release tag when the commit message also contains [maven-release-plugin] prepare release, which is placed by the Maven Release Plugin.

Important

When pushChanges is turned off for the Maven SCM Plugin make sure to not push the HEAD which contains [maven-release-plugin] prepare for next development iteration but the commit before it with for example:

git push origin HEAD~:<release-branch-name>

In case Jenkins CI failed or missed building the tag just use the alternative release sh script mentioned below.

Since GitLab CI doesn't use SNAPSHOT artifacts from Sonatype Open Source Software Repository Hosting (OSSRH), because that repository isn't defined in the used Maven Docker images, we might as well don't upload them anymore:

     stage("open-source")
     {
       steps
       {
         parallel (
           "quality-assurance":
           {
             mvnSonarQubeNode nodeName: "windows", profile: "-PopenSource,sonarCloud", branchNames: ['master']
           },

           "artifacts":
           {
-            mvnIntegrationTestNode nodeName: "windows", profile: "-DskipTests -PopenSource,documents", branchNames: ['master']
+            mvnOpenSourceReleaseNode nodeName: "windows", profile: "-DskipTests -PopenSource,documents", branchNames: ['master']
           }
+def mvnOpenSourceReleaseNode(Map args)
+{
+  node("${args.nodeName}")
+  {
+    stage("release-artifacts")
+    {
+      if (hasReleaseTag())
+      {
+        echo "Clean Up..."
+        deleteDir()
+
+        echo "Check Out..."
+        checkout scm
+
+        echo "Bytecode..."
+        unstash "bytecode"
+        sh "find . -name \"*.class\" -exec touch {} \\+"
+
+        echo "Open Source Release for '${env.COMMIT_TAGS}'"
+
+        sh "mvn $MAVEN_CLI_OPTS deploy ${args.profile}"
+      }
+      else
+      {
+        echo "Skip Open Source SNAPSHOT for '${env.COMMIT_MESSAGE}'."
+      }
+    }
+  }
+}

C.3.21.1. Alternative release sh script

Alternatively release after git tagging with the following script /p/dev/apps/windows/batch/git-mvn-deploy.sh:

#!/bin/bash

if [ $# -lt 1  ]
then
  echo "Usage: git-mvn-deploy <git-tag> -Pdocuments[,openSource,sonarCloud]"
  exit 9
else
  git_tag="$1"
  git_profile="$2"

  git checkout $git_tag

  if [ "$?" -ne 0 ]; then
    echo "Git checkout $git_tag failed!"
    exit 1
  fi

  mvn clean

  if [ "$?" -ne 0 ]; then
    echo "Maven clean unsuccessful!"
    exit 2
  fi

  mvn deploy $git_profile

  if [ "$?" -ne 0 ]; then
    echo "Maven deploy $git_tag unsuccessful!"
    exit 3
  fi

  mvn org.owasp:dependency-check-maven:aggregate $git_profile

  if [ "$?" -ne 0 ]; then
    echo "Maven OWASP dependency check unsuccessful!"
    exit 4
  fi

  mvn sonar:sonar $git_profile

  if [ "$?" -ne 0 ]; then
    echo "Maven SonarQube unsuccessful!"
    exit 5
  fi

  git checkout master

  if [ "$?" -ne 0 ]; then
    echo "Git checkout master failed!"
    exit 6
  fi

  mvn clean

  if [ "$?" -ne 0 ]; then
    echo "Maven clean unsuccessful!"
    exit 7
  fi

  mvn test-compile

  if [ "$?" -ne 0 ]; then
    echo "Maven compile unsuccessful!"
    exit 8
  fi
fi

To release to the self-hosted Nexus use for example:

/p/dev/apps/windows/batch/git-mvn-deploy.sh setup-4.3.4 -Pdocuments

To release to Sonatype Open Source Software Repository Hosting (OSSRH) use for example:

/p/dev/apps/windows/batch/git-mvn-deploy.sh setup-4.3.4 -Pdocuments,openSource,sonarCloud