rdf4j logo Eclipse rdf4j: documentation

Home

Developer Workflow and Project Management

In this document the Eclipse RDF4J project workflow and developer best practices are explained. It contains information on how to create branches, tag releases, manage pull requests, create and schedule issues, and so on. Some of this information is targeted specifically at the project lead(s), other information is relevant to every committer.

1. Semantic Versioning

RDF4J strives to apply Semantic Versioning principles to its development:

  1. We use a MAJOR.MINOR.PATCH versioning template.

  2. A PATCH release (2.2.1, 2.2.2, etc.) is a release that contains only bug fixes that are backwards compatible.

  3. A MINOR release (2.0, 2.1, 2.2, etc.) is a release that can contain improvements and new features but makes no backward-incompatible changes to existing functionality.

  4. A MAJOR release (1.0, 2.0, 3.0, etc) is a release that can contain changes to the public API that are not backward compatible.

It is currently not fully specified what the boundaries of the RDF4J public API are. Until this is resolved (see issue #619), we allow changes to public or protected methods/classes/interfaces in minor releases under the following conditions:

  1. any renamed interface is declared an extension of the old interface. The old interface is marked deprecated with Javadoc containing a reference to the new name;

  2. any renamed class is declared a superclass of the old class. The old class is marked deprecated with Javadoc containing a reference to the new name;

  3. any renamed member is added next to the old member name. The old member is declared deprecated with Javadoc containing a reference to the new name.

These conditions are to ensure that existing user code will continue to work when upgrading to the new release. If there is any doubt about a change being backwards-compatible, it can not be made part of a minor release.

For patch releases we never allow changes in the public API, unless the change is specifically to fix a bug that aligns the actual behavior of the code with the publicly documented behavior.

Only release branches (more on those later) include the full major.minor.patch version numbers included with releases. The major and minor version of the master and develop branches do not include a patch number and always have a SNAPSHOT tag. The master and develop versions always matches the pattern MAJOR.MINOR-SNAPSHOT.

The master version always has the same major and minor number as the latest release while the develop version uses the next expected major/minor release number, such as 2.1-SNAPSHOT and 2.2-SNAPSHOT respectively (after a 2.1.x release, but before any 2.2.x releases are made public).

2. Workflow

Every issue, no matter how small, gets its own branch while under development. The milestone label of the issue is set to the planned release version for the issue, but that could change by the time a PR is merged. Issue branch names are always prefixed with issues/, followed by the issue number in the GitHub issue tracker, followed by one or two dash-separated keywords for the issue.

For example: issues/#276-rdfxml-sax is the branch for a fix for GitHub issue #276, which has to do with RDF/XML and the SAX parser.

RDF4J uses a git branching model where collaborative feature development takes place on branches from the develop branch. This is where all development for the next (minor or major) release happens. These branches should have the prefix features/ to indicate they are either not complete or have not been scheduled for the next minor release.

Once a issue is complete and tested, a Pull Request should be created for peer review. If the issue is part of a larger feature, the PR should be branched from (and target) the corresponding features/ branch. If the issue is a new feature in and of itself, the PR should be branched from (and target) the develop branch. If the issue is a bug fix, the PR should be branched from (and target) the master branch. The milestone label of the PR (and issue) are updated as part of the review process and later merged into the target branch.

2.1. Patch Requests

If the change is a bug fix, contains no new features, and does not change any public or protected APIs then

  1. Create an issue branch by forking the master branch.

  2. Make the necessary changes and verify the codebase.

  3. Create a Pull Request that targets the master branch.

  4. Peers and project committers now have a chance to review the PR and make suggestions.

  5. Any modifications can be made to the issue branch as recommended.

  6. Once any necessary changes have been made, project committers can mark the PR as approved.

  7. Project committers should then determine what patch release this fix will be included in by updating the milestone label of both the PR and the issue.

  8. Only when a Pull Request is approved and scheduled for the next patch release it should be merged into the master branch then

  9. the issue branch can be delete (if desired).

  10. After a PR has been merged into the master branch, the master branch should then be merged into the develop branch by the project committer that merged the PR, any conflicts (such as due to new features) should be resolved. This can be done from the command line from a clean checkout as follows:

    git checkout develop
    git pull origin master
    git push origin develop

2.2. Feature Requests

Pull Requests that add a self contained new features to the public API follow the same steps as a Patch Request but should fork and target the develop branch. However, Pull Requests that make changes to a feature that has not been scheduled for the next minor release, should fork and target the corresponding features/ branch. Only PRs that have been scheduled for the next minor release should be merged into the develop branch.

Project committers that are contributing to a features/ branch should periodically pull changes from the develop branch to minimize conflicts later on. Once a features is complete another PR should be created using the features/ branch and target the develop branch. Then follow similar steps to a patch request to schedule and merge into develop.

Minor and major releases require a formal release review, and because this is the case, they need to be planned well in advance, and the project lead needs to manage what can go into each release, and prepare necessary documentation (both technical and legal) for review. For this reason approved Pull Requests may stay open (not scheduled or merged) for some time until a release plan that incorporates these changes and any required documentation is in place. The comment section in the PR can be used to keep everyone informed of the progress.

3. Patch releases

Patch releases are created by branching the master branch into a release branch, and when complete, tagging this release branch with the version number before release deployment. Once release deployment is complete, the release branch is deleted.

It is important that the master branch is always in a release ready state (build passes, no new features, docs are upto date), so a patch release from the master branch can be done in an ad-hoc fashion, without the need for a formal review.

Plans to do a patch release are announced by the project lead on the rdf4j-dev@eclipse.org mailinglist, usually about a week in advance, with an open invitation for contributors to propose additional fixes to include, which are done as Pull Requests to the master branch.

3.1. Creating a patch release branch

Any fixes to be included in the a patch release must be merged into the master branch first. A patch release branch should differ from the master branch, at the time of release, only by the version number - a patch release branch has a patch number version, while the master branch has a SNAPSHOT version. To create a patch release branch, follow these steps:

  1. Check out the master branch.

    E.g. if we’re preparing a release 2.2.1, the master branch will have the version 2.2-SNAPSHOT:

    git checkout master
  2. Create a new release branch, named releases/<version>:

    git checkout -b releases/2.2.1
  3. Fix maven version numbers in the release branch.

    We need to set the project version to 2.2.1 by using:

    mvn versions:set

    This will ask for the new version number. Enter 2.2.1 to indicate that this is the release branch for 2.2.1 release.

    After this is done, execute

    mvn versions:commit

    which will remove backup files.

    Finally, commit the version number changes:

    git commit -s -a -m "release branch for 2.2.1"
  4. Tag the version and push the tag upstream:

    git tag 2.2.1
    git push origin 2.2.1

4. Hotfix releases

Hotfix release are patch releases that target an prior minor version (not the latest stable release). These are needed when a critical bug was found in a production deployment using an earlier version.

A Hotfix release use a preceding release as its basis. This means we need to create a release branch not by simply branching from the current master branch, but by branching from a specific release tag. To create a patch release branch, follow these steps:

  1. Check out the tag of the previous release.

    E.g. if we’re preparing a release 2.1.6, the preceding release is 2.1.5, so we do:

    git checkout 2.1.5
  2. Create a new release branch, named releases/<version>:

    git branch releases/2.1.6
  3. Fix maven version numbers in the release branch.

    We need to set the project version to 2.1.6-SNAPSHOT by using:

    mvn versions:set

    This will ask for the new version number. Enter 2.1.6-SNAPSHOT to indicate that this is the development branch for the upcoming 2.1.6 release.

    After this is done, execute

    mvn versions:commit

    which will remove backup files.

    Finally, commit the version number changes:

    git commit -s -a -m "release branch for 2.1.6"
  4. Push the newly created branch, as follows:

    git push origin releases/2.1.6

Bug fixes are typically added to a hotfix release branch by cherry-picking the relevant commits from the master branch.

This works as follows:

  1. Check out the patch release branch.

  2. In the git commit history, identify the commit for the fix you wish to add to the release. You can usually easily find this by looking at the original Pull Request for the (normally the PR can be found by looking through the issue coments on GitHub). You’re looking for a message in the PR about the merge, usually at the end, that looks like this:

    jeenbroekstra merged commit 5d13554 into eclipse:master

    The commit number (5d13554) is what you’re after.

  3. Add this fix to the release branch by executing the following command:

    git cherry-pick -m 1 5d13554

    The -m 1 flag is necessary because this is a merge-commit, which has two parents: we need inform git which parent commit it needs to use as the base (we select 1, which is the master branch, to ensure that only changes introduced by this fix are included).

Once all fixes are applied to the release branch, and the build is stable (NB verify by executing mvn clean verify), we can tag and finalize the release:

  1. Set the maven pom version numbers.

    We need to set the project version to from 2.1.6-SNAPSHOT to 2.1.6 by using:

    mvn versions:set

    This will ask for the new version number. Enter 2.1.6 to indicate that this is the actual code for 2.1.6 release.

    After this is done, execute

    mvn versions:commit

    which will remove backup files.

    Finally, commit the changes and push:

    git commit -s -a -m "patch release 2.1.6"
    git push
  2. Tag the version and push the tag upstream:

    git tag 2.1.6
    git push origin 2.1.6

Once the release is complete, the hotfix branch needs to be deleted. Although this can of course be done from the command line, it is cumbersome, and we recommend using a decent Git client (like SourceTree) that can do this for you.

Note that, although the branch is deleted, the release tag is still in place, for future use of further hotfix releases.

5. Release distribution deployment

RDF4J has two separate distributions:

  1. the SDK and onejar archives, downloadable via http://www.rdf4j.org/download .

  2. the Maven artifacts, available via The Central Repository.

RDF4j has three separate announcements:

5.1. Building and uploading Maven artifacts

We use the Eclipse RDF4J Hudson CI instance to build and deploy new releases to the Central Repository. To do this, simply log in to Hudson, and start the job named rdf4j-deploy-release-ossrh-central. The job will ask for the release tag as an input parameter, e.g. '2.2.1'.

This Hudson job will automatically check out the release tag, build the project, and upload all artifacts to OSS Sonatype. After successful upload, it will also automatically invoke synchronization with the Central Repository. Note that after successful completion, the artifacts may not be available on the Central Repository for several hours.

5.2. Building and uploading SDK and onejar

The SDK and onejar archives are hosted on https://www.eclipse.org/downloads/ . The archives need to be built locally, and uploaded manually, via secure FTP.

5.3. Building the SDK and onejar

  1. Check out the release tag

  2. Enter the core directory:

    cd core
  3. From the core directory, execute the following:

    mvn -Passembly clean package

Once this completes, the SDK and onejar can be found in core/assembly/target. Verify that the SDK is complete by inspecting its contents (in particular, check that javadoc is included).

5.4. Uploading the SDK and onejar

  1. SFTP to build.eclipse.org. You will need to provide your eclipse username and password.

  2. Go to remote directory /home/data/httpd/download.eclipse.org/rdf4j.

  3. Upload the SDK and onejar archives to this directory (NB we currently only distribute the SDK zip file, not the tar.gz file)

6. Minor and Major releases

Minor and major releases require a formal release review, and because this is the case, they need to be planned well in advance, and the project lead needs to manage what can go into each release, and prepare necessary documentation (both technical and legal) for review.

We plan each release about 8 weeks in advance. At this stage, the final feature set is not etched in stone but a number of priority features/improvements is identified (via discussion on the mailinglist and/or via issue tracker comments and PRs) and scheduled. A first draft of a release plan is created by the project lead on the Eclipse RDF4J project site, and the necessary milestones are created in the RDF4J issue tracker. In addition, the RDF4J Pull Requests milestones are updated as the code is ready for inclusion.

6.1. Review planning and application

A release can only be done once its review is successfully concluded. Eclipse release review are announced in regular cycles, and always complete on the first or third Wednesday of each month. For this reason, we schedule our releases to happen on a first or third Thursday.

A release review runs for a week. Although mostly a formality, it does need some careful preparation and planning. It needs to be formally applied for, and this application in turn requires that several pieces of documentation are in order:

  1. The project’s IP log needs to be filed and approved by the Eclipse legal team;

    The IP log can be automatically generated and submitted to the legal team. Obtaining approval may require several days, so it’s good practice to submit this at least two weeks before the planned release date.

  2. The project’s review documentation, part of the application, needs to be in order.

Typical review documentation can be a simple reiteration of the most important new features, a link to the issue tracker/release notes and documentation, and a remark about compatibility (if applicable). Once the review documentation is up, a mail needs to be sent to technology-pmc@eclipse.org to ask for approval. Here’s an example of such a message, which was to get approval for the RDF4J 2.2 release:

Dear PMC members,

Can I get your approval for RDF4J release 2.2, scheduled for February 2.

Release review info: https://projects.eclipse.org/projects/technology.rdf4j/reviews/2.2-release-review

Issue tracking the release: https://bugs.eclipse.org/bugs/show_bug.cgi?id=510577

Kind regards,

Jeen Broekstra

When IP log approval and review approval have been given, the review can be scheduled. To do this, emo@eclipse.org needs to be notified. This can happen through the eclipse project governance page (accessible through the project site), which will show a link at the top of the page for the planned release.

For more detailed information about the release review process, see the Eclipse Project Handbook.

6.2. Branching minor releases

Minor releases are created by branching the develop branch into a release branch, and when complete, tagging this release branch with the version number before release deployment. Once release deployment is complete, the release branch is deleted.

It is important that only features and fixes that have already been scheduled for release (via PR milestone labels) be merged into the develop branch, so that there is no confusion as to what will be included in the next minor release.

The steps to create a minor release are very similar to the patch release steps, but are based on the develop branch (not the master branch).

Once a minor release is published the develop branch should be merged into the master branch (along with the develop branch’s version). This will increment the master version to the latest major/minor SNAPSHOT version. Then the develop minor version should be incremented to the next SNAPSHOT version and any approved features that are scheduled for this next minor version should be merged into develop branch.

7. Further reading

Some generic sources of information about projects hosted by Eclipse: