## Publishing with Gradle to Sonatype and Maven Central on Travis

I spent many hours yesterday trying to automate the release of the Java client library for NATs. The goal was to upload a release to Sonatype's open source Nexus server whenever we tag a release in GitHub. Sonatype will then upload it to Maven Central. This involves uploading to Sonatype from Travis with my Gradle script, which turned out to be a pretty big deal. While I found a lot of help on the internet, no one had a single full deploy so I thought I might provide one for those that find it useful. All of the files/code involved are available in the client library repo linked above.

There is a lot of groovy for the gradle setup and yaml for the travis, but I hope that seeing the full setup will save someone some time. This "tutorial" assumes that you are deploying to the open source Sonatype Nexus from Travis. I suspect you can do a bit of work to tweak it for other situations. But I will assume that you have access to Travis and that you have a Sonatype username and password.

## The Signing Key

You will need a signing key so that you can sign your POM.xml and Jars. If you already have a PGP key, you can skip some of these steps, but it may be worth reading through in case you haven't registered your public key in a public place.

To create a signing key, install gpg from here. Choose the identity you want to associate with the key, which should match your organization or personal identity in some way. We are going to register this key in a public place so you want people to know it is you.

To create the actual key pair run:

$gpg --gen-key this will create a key. You can get the id using: $ gpg --list-keys --keyid-format short

The private key is stored locally and you don't want to check it into GitHub or anything like that. We will go through the process of hiding it in a moment. But first, we need to upload the public key to some registries that Sonatype will look for it on.

Export the public key with some armor that relates to your organization or identity, I used "nats.io" for the client library:

$gpg --export -a "nats.io" > public.key Now we want to upload it to a few well known repositories, starting with http://keys.gnupg.net:11371/, http://keyserver.ubuntu.com/ and http://pool.sks-keyservers.net/pks/add. You can do this manually, or on the command line: $ gpg --keyserver keyserver.ubuntu.com --send-keys <keyid>
$gpg --keyserver pool.sks-keyservers.net --send-keys <keyid>$ gpg --keyserver keys.gnupg.net --send-keys <keyid>
$gpg --keyserver pgp.mit.edu --send-keys <keyid> The entire key process is rather painful, and seems to take time to propagate. But once it is done Sonatype will be able to identify you as you. More information is available on the Sonatype documentation site. We will come back to the key again, when we set up the travis.yml file. ## Gradle Our gradle file will assume that five values are available: • Your sonatype user name • Your sonatype password • Your signing key id (you got this from the list-keys command above) • The password you used to sign the key • The location of the signing key (this will be .travis/nats.travis.gpg in the example below) You can set these values in a ~/.gradle/gradle.properties file for local work: signing.keyId=<key id> signing.password=<your pw> signing.secretKeyRingFile=<where you put the gpg keyring> ossrhUsername=<your sonatype user name> ossrhPassword=<your sonatype password> or you can use environment variables, which is what we will do for travis using secrets discussed below. The remaining discussion breaks down the build.gradle file and discusses each section. You can see the full file at github. ### Imports The Nats Java client has a number of imports for the gradle file. plugins { id 'java-library' id 'java' id 'jacoco' id 'maven' id 'maven-publish' id 'signing' id 'com.github.kt3k.coveralls' version '2.8.4' id 'osgi' id 'io.codearte.nexus-staging' version '0.21.0' id "de.marcphilipp.nexus-publish" version "0.3.0" } The most important ones for this tutorial are maven-publish which is the newer alternative to uploadArchives, signing to sign the pom and jars, io.codearte.nexus-staging and de.marcphilipp.nexus-publish to manage the sonatype interactions. There was a major travis-to-sonatype issue related to ip address that these plugins solve, they also allow us to close and release a repository. ### Variables, Values and Versions The java library stores the version in 2 modes, one as pieces and one as a full string. At some point this should come from the tag but I haven't gotten that done yet. We also use the travis branch and tag to make some choices. def versionMajor = 2 def versionMinor = 6 def versionPatch = 4 def versionModifier = "" def jarVersion = "2.6.4" def branch = System.getenv("TRAVIS_BRANCH"); def tag = System.getenv("TRAVIS_TAG"); archivesBaseName = 'jnats' group = 'io.nats' The version used or files is calculated based on the version components and an optional -SNAPSHOT string that is there whenever the tag is missing. This snapshot flag will play a role in deployment as well. Any version with a snapshot will go to the snapshot repo on Sonatype. Non-snapshots, or releases, will go to the staging repository, before being closed and released to the releases repository. def getVersionName = { -> if ("".equals(tag)) { versionModifier = "-SNAPSHOT" } if (versionModifier != null && versionModifier.length() > 0) { return "" + versionMajor + "." + versionMinor + "." + versionPatch + versionModifier } else { return "" + versionMajor + "." + versionMinor + "." + versionPatch } } version = getVersionName() Now we get to the properties mentioned earlier. If the SONATYPE_USERNAME environment variable exists, we assume that all the values are in the environment. Otherwise they need to be in the gradle.properties file. if (System.getenv('SONATYPE_USERNAME') != null) { project.ext['ossrhUsername'] = System.getenv('SONATYPE_USERNAME') project.ext['ossrhPassword'] = System.getenv('SONATYPE_PASSWORD') project.ext['signing.secretKeyRingFile'] = System.getenv('GPG_KEYRING_FILE') project.ext['signing.keyId'] = System.getenv('GPG_KEY_ID') project.ext['signing.password'] = System.getenv('GPG_KEY_PASSPHRASE') } ### Task management We will enable and disable some tasks. Signing is on only if this is a release build. Close and release are on only if we are going to be doing a staging release and we can get the id for the staging repository. closeRepository and releaseRepository come from the plugins we added above. nexusPublishing also refers to those plugins. I wasn't able to automate the close/release without those. tasks { signing { onlyIf {!version.endsWith("SNAPSHOT")} } closeRepository { onlyIf { nexusPublishing.useStaging.get() } } releaseRepository{ onlyIf { nexusPublishing.useStaging.get() } } } ### Dependencies and Source Steps The java build only depends on a couple modules. Junit for tests and the eddsa module for an ed25519 implementation. Your project will differ here of course. repositories { jcenter() } dependencies { compile 'net.i2p.crypto:eddsa:0.3.0' testImplementation 'junit:junit:4.12' } The source sets are broken into the main library, examples and tests. sourceSets { main { java { srcDirs = ['src/main/java','src/examples/java'] } } test { java { srcDirs = ['src/test/java'] } } } osgiClasses { exclude("io/nats/examples/**") } Everything but the examples are part of the OSGI registration. ### Basic Task Setup First we set up the Jar with the appropriate attributes, and without the example files. jar { manifest { attributes('Implementation-Title': 'Java Nats', 'Implementation-Version': jarVersion, 'Implementation-Vendor': 'nats.io') instruction "Import-Package", "!net.i2p.crypto.eddsa.math" instruction "Import-Package", "net.i2p*" instruction "Import-Package", "io.nats*" } exclude("io/nats/examples/**") } Next we configure the tests. The java client uses random numbers which we have found can be an issue on linux if you don't set up the urandom source. test { maxHeapSize = "2g" if (org.gradle.internal.os.OperatingSystem.current().isLinux()) { jvmArgs '-Djava.security.egd=file:/dev/./urandom' } testLogging { exceptionFormat = 'full' events "started", "passed", "skipped", "failed" } } Next we configure the java doc. You may find the doLast step interesting. We set the favicon.ico for all the files as well as copy images into the java doc folder. The implementation and examples are excluded from the official Java doc. javadoc { options.overview = 'src/main/javadoc/overview.html' // relative to source root source = sourceSets.main.allJava title = "NATS.IO Java API" excludes = ['io/nats/client/impl', 'io/nats/examples'] classpath = sourceSets.main.runtimeClasspath doLast { exec { println "Updating favicon on all html files" workingDir 'build/docs/javadoc' // Only on linux, mac at this point commandLine 'find', '.', '-name', '*.html', '-exec', 'sed', '-i', '-e', 's#<head>#<head><link rel="icon" type="image/ico" href="favicon.ico">#', '{}', ';' } copy { println "Copying images to javadoc folder" from 'src/main/javadoc/images' into 'build/docs/javadoc' } } } The examples go in a jar of their own. task examplesJar(type: Jar) { classifier = 'examples' manifest { attributes('Implementation-Title': 'Java Nats Examples', 'Implementation-Version': jarVersion, 'Implementation-Vendor': 'nats.io') } from(sourceSets.main.output) { include "io/nats/examples/**" } } The Java doc and source each go in a jar: task javadocJar(type: Jar) { classifier = 'javadoc' from javadoc } task sourcesJar(type: Jar) { classifier = 'sources' from sourceSets.main.allSource } We build a "fat jar" that has all the dependencies to make life easy during development. This is not uploaded to sonatype/nexus. task fatJar(type: Jar) { classifier = 'fat' manifest { attributes('Implementation-Title': 'Java Nats With Dependencies', 'Implementation-Version': jarVersion, 'Implementation-Vendor': 'nats.io') } from { configurations.compileClasspath.collect { it.isDirectory() ? it : zipTree(it) } } with jar } Finally, we use jacocoTestReport to build a coverage report. jacocoTestReport { reports { xml.enabled = true // coveralls plugin depends on xml format report html.enabled = true } afterEvaluate { // only report on main library not examples classDirectories = files(classDirectories.files.collect { fileTree(dir: it, exclude: ['**/examples**']) }) } } ### The Artifacts The sonatype deployment will release four artifacts, the main jar plus the javadoc, sources and examples. artifacts { archives javadocJar, sourcesJar, examplesJar } We will sign all of these artifacts when signing is enabled. Note, we will do something special for the pom.xml below. signing { sign configurations.archives } ### The Nexus/Sonatype Magic Now we get to the magic. First, we have to configure the nexusStaging plugin with the package group and our authentication information. nexusStaging { packageGroup = group username = project.getProperty('ossrhUsername') password = project.getProperty('ossrhPassword') } You can also provide a staging repo id, but we will rely on the one that Sonatype provides for our group. Finally we want to configure the publishing. There is a lot here, and I want to call out a few very important sections without chunking the file any more than I already have. First, you have to list the artifacts you want to publish. This includes the components.java plus the jars we created. Second, you set the pom here, that is no big surprise. But you have to have the pom.withXML section to sign your pom. Otherwise it will not be signed and if it is not signed the sonatype close operation will fail. Third, all of the signatures have to be classified in the project.tasks.signArchives.signatureFiles.each section. This code is based on the nexus plugins readme files but I had to add the examples jar to the regular expression match. If you have other artifacts you will need to add them here. Fourth, you have to give the nexusPublishing plugin the authentication information. This is not shared with the staging plugin above. At least it wasn't for me, and the error messages were not that helpful ;-) Fifth, and finally, you have to have the model section to require signing before publishing, especially the signing of the pom. Note the naming of the final section publishMavenJavaPublicationToSonatypeRepository is based on the nexusPublishing repositories sonatype hierarchy before it. publishing { publications { mavenJava(MavenPublication) { from components.java artifact sourcesJar artifact examplesJar artifact javadocJar pom { name = 'jnats' packaging = 'jar' groupId = group artifactId = archivesBaseName description = 'Client library for working with the NATS messaging system.' url = 'https://github.com/nats-io/nats.java' licenses { license { name = 'The Apache License, Version 2.0' url = 'http://www.apache.org/licenses/LICENSE-2.0.txt' } } developers { developer { id = "synadia" name = "Synadia" email = "info@synadia.com" url = "https://nats.io" } } scm { url = 'https://github.com/nats-io/nats.java' } } pom.withXml { def pomFile = file("${project.buildDir}/generated-pom.xml")
writeTo(pomFile)
def pomAscFile = signing.sign(pomFile).signatureFiles[0]
artifact(pomAscFile) {
classifier = null
extension = 'pom.asc'
}
}

// create the signed artifacts
artifact(it) {
def matcher = it.file =~ /-(sources|javadoc|examples)\.jar\.asc$/ if (matcher.find()) { classifier = matcher.group(1) } else { classifier = null } extension = 'jar.asc' } } } } nexusPublishing { repositories { sonatype { username = project.getProperty('ossrhUsername') password = project.getProperty('ossrhPassword') } } } model { tasks.generatePomFileForMavenJavaPublication { destination = file("$buildDir/generated-pom.xml")
}
}
}
}
}

With this gradle file, we are ready to move on to Travis.

## Travis

Most of your travis configuration is specific your project, but there are two pieces that you will need to set up to work with gradle. The first is getting your key ready. The second, is setting up the secrets. I discuss those two things first, but really you need a travis.yml file to perform them.

### Storing the Key For Travis

In order for Travis to run the Gradle signing plugin you need to add your key to the repository.

The first step is to install the travis CLI. I use the gem discussed here, which also has information about encryption keys.

Of course you don't want to do this in plain text, so instead you can export the secret key. I put it in the .travis folder.

$gpg --export-secret-key <TRAVIS_KEY_ID> > nats.travis.gpg$ travis encrypt-file .travis/nats.travis.gpg

This will create a .enc version of the file. Delete the original, and do not check it in.

Travis will print out a line of code for you to put in your build script:

- openssl aes-256-cbc -K $<some key> -iv$<some iv> -in <in file path> -out <out file path> -d

Now you have a pgp key that you can safely check in, but that you can also use in Travis builds.

Next you want to update your travis.yml with four secrets:

• The sonatype user name and password
• The GPG key id
• The GPG Key passphrase

We will add these as secrets to the travis.yml which will result in them being environment variables when we run the script.

You can use the --add command below.

$travis encrypt SONATYPE_USERNAME="<YOUR_JIRA_USER_NAME>" --add$ travis encrypt SONATYPE_PASSWORD="<YOUR_JIRA_PASSWORD>" --add
$travis encrypt GPG_KEY_ID="<TRAVIS_GPG_KEY_ID>" --add$ travis encrypt GPG_KEY_PASSPHRASE="<TRAVIS_KEY_PASSPHRASE>" --add

or use

$travis encrypt -i to avoid passwords in shell history and add them to the global section manually. Travis will want to know the repository this is for, if it can't figure it out use -r <repo owner>/<repo> for example -r nats-io/nats.java. NOTE - If your password has special characters to bash, which may be required by Sonatype, you need to escape them before you encrypt them. If you do not then bash will mess them up when Travis tries to set them. ### The Travis YAML File For the NATS Java client, the final travis file looks like this (if you unchunk it): First some standard settings for the image and JDK versions. dist: trusty language: java sudo: required jdk: - openjdk8 - openjdk9 - openjdk10 - openjdk11 - oraclejdk8 - oraclejdk9 For the tests we grab the NATS server binary. before_script: - wget "https://github.com/nats-io/nats-server/releases/download/$nats_server_version/nats-server-$nats_server_version-linux-amd64.zip" -O tmp.zip - unzip tmp.zip - mv nats-server-$nats_server_version-linux-amd64 nats-server

Then we decrypt the signing key (see above):

before_install:
- openssl aes-256-cbc -K $encrypted_f07928735f08_key -iv$encrypted_f07928735f08_iv
-in .travis/nats.travis.gpg.enc -out .travis/nats.travis.gpg -d

We assemble and run the tests, as well as manage the cache:

install:
script:
#for testing - "./gradlew build -x test"
before_cache:
- rm -f  $HOME/.gradle/caches/modules-2/modules-2.lock - rm -fr$HOME/.gradle/caches/*/plugin-resolution/
cache:
directories:
- "$HOME/.gradle/caches/" - "$HOME/.gradle/wrapper/"

#### The Magic

After the build succeeds we use three conditionals. I found these conditionals to be a bit of a pain at first, not the - | syntax to allow the shell script. Also i had to use -z for the empty check on the tag and not == "".

We deploy the oraclejdk8 build for compatibility. We publish on a test branch travis_deploy and on master if the tag is not set. Travis will set the branch value to the tag value if one is provided. In that case we do a publish and a close/release.

publishToSonatype comes from the nexus plugins and is related to that nexusPublishing repositories sonatype hierarchy in the gradle publishing configuration. For non-release versions, ie SNAPSHOTS, this step will overwrite the last snapshot. Dependents can use:

https://oss.sonatype.org/content/repositories/snapshots/

to access the snapshot artifacts.

The closeAndReleaseRepostiory also comes from the plugins and will try to close the staging repository and release it. If this succeeds the artifacts will eventually be uploaded to maven central, which takes a couple hours. They will be available on sonatype in a few minutes. Dependents can use:

https://oss.sonatype.org/content/repositories/releases/

to access the release artifacts, or get them from Maven central when they get there.

after_success:
- |
if [ "${TRAVIS_BRANCH}" == "travis_deploy" ] && [ -z "${TRAVIS_TAG}" ] && [ "${TRAVIS_JDK_VERSION}" == "oraclejdk8" ]; then ./gradlew -i sign publishToSonatype fi - | if [ "${TRAVIS_BRANCH}" == "master" ] && [ -z "${TRAVIS_TAG}" ] && [ "${TRAVIS_JDK_VERSION}" == "oraclejdk8" ]; then
fi
- |
if [ "${TRAVIS_BRANCH}" == "${TRAVIS_TAG}" ] && [ ! -z "${TRAVIS_TAG}" ] && [ "${TRAVIS_JDK_VERSION}" == "oraclejdk8" ]; then
fi

#### The Environment Setup

Remember those secrets, those go in the env/global section. The NATS Java client also uses a variable for the nats-server version and the path we install it at. The final environment variable points to the signing key file. This environment variable, along with the secret ones are used to cross the boundary to the gradle file.

env:
global:
- nats_server_version=v2.0.4
- nats_server_path=\$TRAVIS_BUILD_DIR/nats-server/nats-server
- GPG_KEYRING_FILE=.travis/nats.travis.gpg
- secure: isW18c01AJEDAPUUl6rKcewHxOqItTW0TiiEIrWQqQP/C3O06WgAbiFYVFPJ9zCi6me0Wj3YMmEoxiYBhFdgH/O5xoQnnU7xIfD9hcmByglsoyGsK/Wz0wcERoVf9bfbVQkj9q/Mg7kaUZCMWqcFR3CqHEGu8UH5x7ecDW5FXfAQDjN5czT1j1VAwhHZCfIktJuy/GzoFGgRJpvnFPSlHmi0I8fApoX43tmOCkTVHnaXt9CDL3A5EIKtok5dwu0FF5d9hQFncJB8gqGxd+r8a3W3+0Gfgdou3x+AlGTf3R62LgB03GY0MFrMVfanWJE1ORdV0o9hC3AiwOsKBTungZ0arQeXtDXHSeMY52O6u7C8MCwQgbTmzO2YsmMwwTL98PPQxEJ6c8r7WBAfxzxxRTJ/QjPqQdyWV9dFWOnsmEhBLM2Wi858dJlw5fDEoHgy8EUZTQcquUWqEzTJca1VdrLza/PlND8dqfAjxqINtpsXu88JsLUu5VjFiLwln5NpdNKfcY4oaPiLLYdrSgdxBfHCCISP+r8iqgKLDguFwza3xcPSFwqtEq8aYmy0fjgd0c9hlz6oe0NvLc4kPJf4q9NDjffUXBciiv8VXdL3YyRG67h9AF+ndbM8NHsup5FfmALfq2bGIpe4USIqoOAZFUSa35hPDW87C7Z4vvPvb9I=
- secure: n1r8PLtKgQbW0v0C+MzKRg+EcSXsBBnBe1u8/OyPdC84X6Zwe7JRv/C55Zx71I3yzLhlY6zjBtsNBTO1hgViDPlnDGFl3BL2+fQEi6sofqThgaTSB4UIqTVY8jDYvXp7fcLqN4jM83ElTlfWRSlQHyQ5FKLdla81EcGOB+cL/BObE+Adf9CGGcf7oUd+2MVU+mrEZZTRvaUk32eXzYqEHfib1EHOvyf4ACXbGHe7bIaO6dxAhVNlTjbDdyEdSDFTrtNCFA+t0780IcPhf7uaOy/JBFZ2uL33IqUGih44/GsHj2zJjJmHEL2NtcBhLeuAbRFT6VuE+13DtaB7eNwuXCD4d553vFWK0jT3eKmVgePswHmB1Vs8wsGa92LQLP8vHDCqeMBYzCxWK3hlJj+bzok9AXdyn0IECMTMXc9KLNDIfHrvvmu1DfpuKklmTETII4zSkcPkMAO+VU+kxJ+krf7CYb519KDGpF97bLBWnBboBj3WOx70ux1Fm9ah28YEMPuGHp3Ft6o7ozi2eP8a4YxXo8wQUxe62UIx0e4hDv43aa2Qzvnu9ldEL21WduzlBGjjPTD2GxtkIaB8H1nmH9sgShGVfNwcba3LlO/xU5nUTbrQVcP6WEvlvkdfQmbX4AGh1jCWLWVvxjyPxfZrlARE1JRV5x0oOeH79KA9a18=

## Conclusions

That is a lot of configuration. The key places I found friction:

• The two nexus plugins are key. In particular they use the session repo id which is required for travis deployments to avoid the ip shuffle that happens.
• You have to configure the authentication for both nexus plugins.
• You must manually set up the pom.xml signature, without that you can't close a staging repository.
• Travis sets the branch to the tag if their is one, the branch won't be master in that case.
• Use - | to set up bash conditionals for travis
• You have to set up the model dependencies to make sure signing happens for the publish step

For an open source project there is also the issue that when someone wants to build locally they may not have the gradle.properties file. Turning off the sign step in most cases is an important piece of making the project friendly to developers.

I hope that helps someone.