Development policies

Development Policies

Writing Code

License

Project code is licensed under the Apache License, version 2.0. All modules, as well as other code-like artifacts, need to contain a license notice at the top in appropriate comment format. For Java/Groovy modules, the notice shall read as follows:

/*
 * Copyright (c) <current-year> by the original author or authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

This works as long as the actual author(s) are identified further down in the code commentary; otherwise it's fine to replace "original author" with actual names.

NOTE: This may not be correct for work done under grant sponsorship in the U.S. because of University copyright policies. Please check to make sure you can claim copyright personally; otherwise you will need to claim copyright for your institution instead of yourself.

In IntelliJ Idea, go to Settings > Copyright > Copyright Profiles and create a new profile with the copyright text that is not wrapped into a Java comment. In Settings > Copyright add a new entry with the scope All and save. This will automatically add the copyright information to files you create or are working on.

In Eclipse, simply load powerac-server/powertac-style.xml.

Version Control

After cloning the git repository, it's a good idea to start a new local branch before you start working. When you're done with your work you may pull the latest changes from the server into your master branch, then merge your working branch with the master branch and finally commit and push the master branch.

Coding Guidelines

• Please follow our Coding Guidelines.
• Consistency and readability are at least as important as performance, except in very unusual circumstances. If you find yourself in one of those circumstances, please include commentary that explains why you felt a need to diverge from project standards.
• For small standalone tools (shell scripts, Python programs), include a header that explains how to use your code.
• For all other code, include a header comment that explains what the code is, what role it plays, and how to use it.

Unit Testing

For server and broker modules, we use JUnit 4.x as a unit-testing framework. For modules using the Maven directory layout, each class under src/test/java will be assumed to be a unit test if it has at least one method annotated with @Test. Maven will run them in nor particular order given the test goal (as in mvn clean test).

Unit tests are normally packaged in the same packages as their targets, to allow access to package-visible resources in units under test. Thus it often makes sense to provide test-accessible attributes methods in your production code under package visibility. Most modern IDEs will generate test stubs for you, given a class with a set of methods. However, good unit testing is generally focused on features rather than just on method coverage, so a better approach is to write test methods for each significant feature, and test methods for simple variable setters and getters are mostly a waste of time.

Package (namespace) usage

The top-level package name for all Power TAC code will be org.powertac. Because of the large number of institutions involved in development, it makes no sense to use the names of individual institutions to name packages, and the project owns the powertac.org url.

Dependency management

We are using Apache Maven for dependency management and build automation. One place to start learning it is Maven by Example. Specific tips and procedures for its use are at Using Maven.

Third-Party packages

Third-party components may be used after careful testing and after consideration of architectural impact, as long as they are automatically resolvable with the maven dependency resolution process. If a component is going to be used by multiple Power TAC components, be sure you test with all of them, and that dependency versions are kept in sync.

Issue Tracking

We use GitHub's issue tracking system.

Raising an issue

If you see something that does not work, something that needs to be done, or something that seems incorrect or confusing, and there is not already an issue that describes the problem, please create a new issue. Describe the problem as clearly as you can, and include relevant error messages and details on how you can re-create the problem. If you suspect the problem is due to your own configuration or lack of preparation (it's a complex system, so that applies to all of us on occasion), then you may of course ask for clarification on the developers Nabble list first.

When you raise an issue, do your best to classify it by attaching the appropriate green and red labels. If you think you need a label that's not there, then you may add a label, but it's also important to keep the number of labels under control.

Clarifying and refining an issue

If you see an issue for which you have ideas or information to contribute, please feel free to add comments to the issue. Just make sure that you poke the "comment" button and not the "comment and close" button when you are finished. Note that you can link wiki pages from issue comments as ../wiki/PageName.

Assigning issues

If you see an issue that you would like to work on, feel free to assign it to yourself, by attaching the label with your name. If there is no label with your name, then create one if you have the necessary authority, or ask to have it added.

Important issues that are not claimed by project members will need to be assigned by project leadership.

Working an issue

Generally, this involves a series of steps, many of which should result in comments added to the issue so we can see your status.

• Understand and confirm the issue. For software defects and enhancement requests, the best way to do this is to write a test case that demonstrates the issue, under the assumption that software defects are indications of missing tests. There can be several outcomes to this process:
  • It's not really an issue, but a misunderstanding or miscommunication. In this case, it may make sense to treat it as a documentation issue, or simply to close it after clearing up the miscommunication and adding an appropriate comment to the issue.
  • It's a duplicate of some other issue. Add a comment with the other issue number, and close it.
  • There is some other issue that must be understood and resolved before progress can be made. If this issue is already posted, then add a comment about the dependency, otherwise raise the new issue and note the dependency in the original issue. It may make sense at this point to remove the assignment.
  • It's one that you understand and can start work on. If appropriate, add a comment that describes your plan for resolving the issue, and gives an estimate of when you expect to complete the work.
• Develop the fix. This may involve software and unit-test changes as well as documentation changes. In almost all cases, your work should be done on your own fork, which you will need to update before starting work. The only exception is when the fix is very simple, you are confident that it can be fixed in a single short step, you test carefully before committing, and you have commit access to the associated repository.
• Once your fix is complete, add a comment to the issue describing what you did and the outcome of your testing. Also make sure any relevant documentation has been updated to reflect the changes you have made.
• Issue a pull request to get your changes incorporated into the trunk.
• The module gatekeeper will inspect your work and run tests, before merging your work into the trunk, and will then close the issue.

Dealing with unexpected problems

If you have made a completion estimate and are unable to meet it, please add a comment that either gives a new completion estimate, or explains why you are unable to make progress. Please do this before the expected completion date if possible. If necessary, we can defer (lower the priority) or reassign the issue in these cases.

Resolving an issue

For all but the most trivial issues, the fix should be confirmed by a second person before the issue is closed. If the fix is the subject of a pull request, then the second person is the one responding to the pull request. In any case, the issue should be closed with a comment describing how the fix was confirmed.

Testing

Thoughtful design and careful implementation is the first line of defense against defects. Next is attention to compiler errors and warnings. Careful unit and integration testing is also a strong defense against nasty surprises at the system level.

Committing Code

• Get familiar with Git and GitHub. Note that the server setup uses submodules; if you want to know how submodules work, see (Submodules on Pro Git, Git Book or some useful blog posts like http://longair.net/blog/2010/06/02/git-submodules-explained/ or http://pacific.mpi-cbg.de/wiki/index.php/Git_submodule_tutorial)
• All substantial work, especially work that involves multiple modules, should be done on a branch or fork (forks are generally easier), and preferably merged by someone else, for two reasons: (1) it allows you to commit partially-completed work and share development with a colleague before the work is complete, and (2) it forces someone else to look at your code and run your tests before it becomes generally available.
• When new work or bugfixes are ready for general consumption, issue a pull request on the fork, and one of the "owner group" will do the inspection, merge, and testing, and commit your changes to the master branch (the development trunk).
• In order to commit code, you should be a collaborator on the repository you are working on. You can always commit code to your own fork. In order to become a collaborator on one of the shared archives, ask for collaboration rights.
• If you intend to push changes to a shared master branch rather than your own fork, be sure you know what you are doing and completely understand the consequences.

Local installation and SNAPSHOT deployment

After updating server code or other shared artifacts, such as the broker core, it is important to run full-system tests in a local development environment. As long as code depends on the current SNAPSHOT version, the Maven install goal will install artifacts in your local Maven repository ~/.m2/repository where they will override downloads from the Sonatype SNAPSHOT repository. If the common module has been changed, then in general it is necessary to do this in three steps: powertac-core, powertac-server, broker-core. Installation is done with maven:

  mvn clean install

SNAPSHOT deployments are done by substituting deploy for install in these commands. Note that whenever any of the modules in powertac-core is re-installed or re-deployed, you must also install or deploy both powertac-server and broker-core, because otherwise the updates to common or other powertac-core modules will not show up in the running code. For the same reason, if you update a server module and want to re-install it without doing a full re-install of powertac-server, you must also re-install all modules down the dependency tree. Fortunately the dependency tree is shallow. It is a very bad idea to do partial deployments because of possible negative impacts on colleagues.

Releasing code and other artifacts

A release is a stable foundation on which our stakeholders can base their work. A particular release may contain all or many of these elements:

• A release identifier, typically of the form a.b.c, where a is incremented for major releases, b for minor releases, and c for bugfix releases. The first release of the Java/Spring version is 0.1.0. The version for the 2012 competition should be 1.0.x.
• A compatible set of versions of a number of code modules. At least the top-level modules must be labeled with the release identifier.
• A written description of the release, (here's an example) with a list of features and known issues. The issues should all be in the github issue tracker, and the release notes should link to the individual issues. The feature list in most cases should concentrate on differences from the previous release.
• A tag, with the name of the release, in the git repos of all modules included in the release.
• For all major and minor releases, a new branch, with the name of the release in the git repos of all included modules. This is the branch on which maintenance of the release will take place, from where bugfix releases will originate.
• A set of jars, including binary, source, and javadoc, as required for promotion to Maven Central.
• A distribution package, generated from the server-distribution module. This package must be named for the release, and packaged into a compressed tar and uploaded as a "download package" on the powertac-server download page.
• An update of the game specification that matches the code.

Creating a release package

A release package contains packaged versions of the server with its components, the broker framework, and the common package, along with descriptive documentation in the form of release notes that describe the capabilities of the release and list known outstanding issues.

We use the Sonatype maven repo for snapshot artifacts, and Maven Central for release artifacts. The process of posting snapshots and releases is described in detail on the Sonatype site. If you need access to post snapshots and release packages, please contact John. For the code, the maven-release-plugin takes care of generating many of the artifacts required for a release. It does not generate the distribution package for the server or the source package for the sample broker.

Released artifacts cannot depend on SNAPSHOT artifacts. Major and minor releases are always created from the master branch, and bugfix snapshots and releases are always created from a release branch. In order to satisfy dependencies at each stage of the release process, the server must be released in three stages:

1. powertac-core comes first, unless a server release is being made without a corresponding release of one or more of the powertac-core modules. The standard maven release:prepare - release:perform process works well.

1. Next we need to prepare powertac-server for release. All modules must be updated to use the release version of powertac-core modules. Once the powertac-core modules are synced on maven central, all the dependencies in the individual modules should be satisfied as long as they depend on the release version of powertac-core modules, and as long as they use the powertac.version property correctly.
2. The powertac-server super-module should be released in one step using release:prepare and release:perform.

1. The server-distribution module is not released using maven; rather, it's a download package. So all that's necessary it to edit the dependencies by hand, tag, commit, and push.
2. Whenever a powertac-core module is updated and released, generally broker-core must also be re-released. The process is the same as for the server modules.
3. Close and release all the modules using the Sonatype Nexus server.
4. Finally, update the version numbers in server-master/server-distribution/pom.xml.
5. Push all changes up to github, including the tags (use git push --tag).
6. If broker-core was released, then sample-broker must be updated to use it, committed, tagged, and pushed.
7. For major and minor releases only (not bugfix releases), you need to create the maintenance branches for each released component. You do this as follows: check out the tag that was created by maven during the release:prepare step, then create the branch locally as git checkout -b release-a.b tagname, then git push origin release-a.b.

Once all the components are available on Maven Central, you can create and post the actual release package. You do that by tagging the server-distribution release version, then create the release in github.

Post-release development

After a release, there will always be a main development branch for work on the next release, and a release branch for maintenance of the release. Depending on the project state, after a release x.y.z, the pom versions on the main development branch (the master branch) will be updated to either x.y+1.0-SNAPSHOT or x.y.z+1-SNAPSHOT, depending on project plans. The pom versions on the maintenance release branch will start at x.y.1-SNAPSHOT, and only the bugfix id will be incremented as new versions of individual modules are released. The release branch will be named release-x.y.

Maintaining backward compatibility

A core idea of the competitive-simulation approach to research is that broker implementations can be shared among the community. In many cases, these shared brokers will be binary packages rather than sources, either because research groups don't want to give away their code, or perhaps because they don't want to have to document it for others. As a result, it is very important to maintain backward compatibility between the broker and server, such that older broker implementations can work with newer versions of the server. There are several basic practices that can maintain backward compatibility, although in general it's very hard to test for without giving server developers in-depth knowledge of individual broker designs and their data and sequence dependencies. At the very least, server developers should observe at least the following principles when making changes or adding features to the server:

1. Existing message types must not be removed.
2. Data fields in messages should not be removed, nor should their meanings be changed. It should be OK to add fields; XStream should ignore extra fields when deserializing messages. If in doubt, test.
3. Remember that not all the messages sent to brokers are in the common package. Almost all of them are either in the common or common.msg packages, but server configuration is transmitted to brokers in the form of a Properties instance.
4. Server configuration data is generated for a module just in case (1) there are fields or setter/getter methods decorated with @ConfigurableValue annotations having publish = true; and (2) the module calls serverProps.publishConfiguration(this). So if it is necessary to move, rename, or change the meaning of a published bit of configuration, the original name and semantics must be retained, keeping in mind that the property name is generated from the package and classname of the module that originally published the item.