Creating (KNIME) update sites with Buckminster

If you are a KNIME user you are probably familiar with the mechanism that lets you install additional extensions and update an existing installation with later versions. If you are a KNIME developer you have probably wondered what kind of magic is involved to make this possible: getting from hundreds of line of Java code to an online update that allows users to install and update extensions. In this week's blog post we will reveal some (if not all) of this magic.

The wizard we are using in KNIME is called Buckminster (not be confused with Richard Buckminster Fuller). Buckminster's product sheet describes it as "a set of frameworks and tools for automating build, assemble & deploy (BA&D) development processes in complex or distributed component-based development". The nice thing about Buckminster is that you can use it in your SDK but there is also a headless application which is suitable for fully automating the process.

Although Buckminster is quite well documented (the BuckyBook), initially it can be overwhelming. Therefore we would like to take a closer look at the essentials that are required to build an update site from scratch.

In the following we assume that you have an Eclipse SDK with the following extensions installed:

• Subclipse or Subversive for connecting to Subversion
• Buckminster including
  • Buckminster Core
  • Buckminster PDE Support
  • Buckminster SVN Support (either Subclipse or Subversive, depending on what you are using)

All files and projects discussed in this article are available in the KNIME Community SVN (in fact they are automatically checked out in the process below).

Buckminster starts from scratch, literally. You have a completely empty workspace and the first thing you need to do is get your source projects and their dependencies. This is done using a so-called CQUERY (component query) which tells Buckminster what component (plug-in, feature, ...) to fetch. To do this, the query needs to know where to get stuff, therefore you also need to provide an RMAP (resource map). Let's have a look at a CQUERY:

This query will fetch the component my.example.update (which is a feature project, as we will see in a moment) using the resource map knime.rmap. The advisor node instructs Buckminster to get all components from my.example in source form and not as binaries. The two properties are used later on for connecting to the SVN repository.

The corresponding resource map looks as follows:

This file is a bit larger and can grow even larger over time. It basically says that all components starting with my.example (the pattern is a regular expression therefore the need to escape dots) can be found via the search path my-stuff and everything else via the search path knime-stuff. Both search paths are defined after that. my-stuff looks into an SVN repository under the specified path (the placeholders are replaced by the nested properties) whereas knime-stuff uses the public KNIME update site to get all required dependencies (if they are not already part of the target platform).

If you have both files in a project in your workspace you can simply double-click the CQUERY file and then click "Resolve and materialize". If you are using the headless application, simply run

. Buckminster will now fetch the requested root component and all dependencies to your workspace. Either as source projects or as part of the target platform, depending on availability and the constraints defined in the query.

Let's quickly return to the root component requested by our query. I already said that my.example.update is a feature project. In Eclipse, features are usually small projects that define a unit, which you can install into Eclipse (or KNIME) via an update site. Buckminster re-uses this project type for defining the update site itself (which is different from the standard Eclipse way as it uses a special Update Site Project). The setup is as simple as it can be: every feature and plug-in listed in either the "Included Features" or the "Included Plug-ins" tabs will end up on the update site. Which means you simply list all features that the user should be able to install - and that's it (usually you don't need to add individual plug-ins, but there may be cases when it's useful). Continuing with our example the update site feature may look like this:

In the build.properties file in the update site's feature you can define categories for the feature(s) on the update site by adding the following lines:

Now we are almost ready to build the update site. Before this, we need to tell Buckminster where to put the created files and how the magic ".qualifier" in version numbers should be replaced (among other possible options). This is done via a standard properties file (e.g. bm.properties):

In the example we use the last modification date of the project in SVN as the qualifier. The "largest" qualifier of all plug-ins included in a feature will be propagated into the qualifier of this feature. So you don't need to touch the feature in order have a recent qualifier (although in most cases you should increase the other version number digits anyway).

Almost there now. In the SDK you can now right-click the update site feature and select Buckminster ⇒ Invoke Action. In the list select site.p2 and as properties file select the file from above and click OK. Now Buckminster will compile all projects (if not already done by Eclipse itself) and put the results in the specified output folder. In our example the update site files will be placed into /tmp/KNIME-Build/my.example.update_0.0.1-eclipse.feature/site.p2. Done.

If you are using the headless application you need to issue

and

. You can also put the three headless commands (import, build, perform) into a script file and call

.

If you have used the KNIME Help you may have noticed that all node descriptions are also available there and you can search in the full description. Generating these files for the Eclipse help system can also be automized with Buckminster. For this to work, you need to install the KNIME Build System extension from http://update.knime.org/build/3.1 (in your SDK and/or in the headless Buckminster application). Additionally you have to tell Buckminster that there is an additional build step to be performed on every plug-in. This is done by modifying the metadata of all plug-in projects using a so-called CSPEX (CSpec extension) file. It has to be put into to the root of the plug-in and named buckminster.cspex. The contents should look as follows:

I will spare you the details what this exactly does. If you are desperate you can read through the BuckyBook (or offer me a non-alcoholic drink).

After you created the magic file(s), when you do Buckminster ⇒ Invoke Action and select site.p2 you will notice that the process takes considerably longer. This is because for every plug-in a KNIME instance will be launched in the background that creates the node documentation from the XML files.

Using the headless application you can now write a few shell scripts that do the job. Or do it properly once and for all using my favourite CI system Jenkins (which is one of the best inventions since sliced bread in my opinion). There is a nice Jenkins plug-in that adds a Buckminster build step in free-style jobs. The plug-in basically takes care of installing a headless Buckminster and running the headless application as part of the job.

There is one important change you need to do to resource map file that is used by the CQUERY: usually you let Buckminster check out the sources into the job's workspace. This means they are already available locally and there is no need for Buckminster to check them out again. Instead it can use the existing folders in the jobs's workspace. Therefore our resource map from above for Jenkins will look as follows:

Instead of using an svn reader we use a local reader that simply imports projects from the local file system into the Eclipse workspace (the workspace.root property will be set by the Buckminster plug-in in Jenkins). The knime-stuff provider and the rest of the configuration stay unchanged.

Finally you add an "Run Buckminster" build step to your job and configure it as shown in the screen shot:

The Buckminster Installation option may change depending on how you set up the Buckminster plug-in the the system configuration (the my.example.update project contains a buckminster.json file that can be used by the Jenkins plug-in to install Buckminster with all required extensions).

If you didn't get everything right upon your first attempt, don't give up. Here at KNIME it took me about a month (seriously) to get the system running smoothly with the additional functionality and a few more tweaks I haven't described here. But I have assisted a number of customers of ours with setting up their internal system and they had it running in less than a day. Which did cost them a few non-alcoholic drinks (*hint*) but I guess they will agree it was worth it!