The Smart Data Blog

Resolving OSGi "Uses Constraint Violations" Using Anzo

Posted by Aron Lurie on Aug 10, 2015 7:45:00 PM

Summary

In large Java projects that use OSGi, there can arise a well-defined problem known as a "uses constraint violation". This problem arises when the classes available to a certain bundle - its class space - contains two versions of the same package. An OSGi system's bundles, packages, and dependencies can be modeled as a graph inside of Anzo Smart Data Platform (Anzo SDP), thus providing a framework for visualization and complex analysis. This article provides multiple methods for using this framework to determine the class space of a bundle. It also provides a SPARQL query and technique for determining the causes and possible solutions of a uses constraint violation, and discusses a concrete example.

Background

OSGi is a specification for building modular or component-based systems in Java. Its basic component unit is a bundle. Each bundle defines a set of Java packages that it exports, and a set that it imports. This is how, for example, a class inside of one bundle can import a class in a different bundle. When importing, a bundle specifies which version of the package it requires, and when exporting, it specifies which version it is supplying.

Neil Bartlett does an excellent job of explaining uses constraints[1], from which I will copy an extract here:

Starting with the basics, in OSGi we have dependencies based on Java packages. Some bundle exports a package, possibly with a version number, and another bundle imports it, possibly specifying an acceptable range of versions:

Osgi simple import.png

This notation is borrowed from the OSGi specification: a black rectangle is an export and a white rectangle is an import. The surrounding yellow blobs are bundles. The line between the export and the import means that OSGi has chosen that specific export as a match for the import — they have been “wired” together. So B imports package foo (version 1.0) from A.

Let’s build this up. Suppose bundle B, in addition to importing package foo 1.0, also exports package bar (the version of bar is unimportant). The C bundle imports package bar and gets wired to B:

Osgi three bundles.png

So far so good. Now let’s complicate things a bit more: let’s say that bundle C, in addition to importing package bar, also imports package foo… BUT it imports version 2.0. Wait, we don’t have foo version 2.0! Never fear, there’s another bundle that does export foo 2.0. We’ll call him D:

Osgi four bundles.png

Are you surprised that this works? Perhaps not, you may have heard that OSGi supports multiple versions of the same library at the same time, and here it is in action. However there are limits: while we can have multiple versions of the foo package, we must still be able to construct a consistent “class space” for each bundle that has exactly one version of every class. The “class space” for bundle C is shown by the shaded blue area:

Osgi class space.png

Notice how the shaded area avoids the import of foo inside bundle B. This is only possible if the package bar exported by B has no internal dependency on foo does not “expose” foo via its signature (thanks to BJ for this correction).

Exposing foo means that a type in foo is visible through the signature of a type in bar, for example:

package bar;

import foo.Foo;

public class Bar extends Foo {   // exposure via subclassing
    public Foo getFoo();         // exposure via method return type
    public void setFoo(Foo foo); // exposure via method parameter
}

If package bar does expose package foo, as in this example, then we have a “uses constraint”. We illustrate this with a little rubber band, like so:

Osgi uses.png

Now bundle C cannot be resolved. The rubber band means we cannot exclude the import of foo 1.0 from the blue shaded area, i.e. C’s class space must contain foo 1.0. But it is not allowed to contain both foo 1.0 and foo 2.0, so C’s second import cannot be satisfied.

The Dependency Graph

A natural representation for the bundles and packages and their relationships is as a graph. Each element (bundle, import package, and export package) is a node, and the relationships between them (imports, exports, source, uses) are edges. Let me skip ahead for a moment to show how we would visualize a concrete example of the first image from Neil's article as a graph:

Simple import anzo.png

The color scheme is preserved in this diagram, while the edges are labeled with the relationships between them.

This image was generated using the Anzo Network Visualizer. In order to do that, we had to load the dependency graph into Anzo. Since Anzo itself runs on OSGi, we did this by adding a bundle to the system that starts last and introspects all resolved bundles, writing that information to Anzo as a graph. See Appendix A for a code snippet of introspecting the resolved bundles.

Calculating Class Spaces

Once we have the dependency graph in Anzo, we can use a couple different tools to do analysis of the graph. One is the Anzo Network Visualizer, which produced the above screenshot. Another is SPARQL, which is a powerful, standardized query language that allows us to quickly answer questions about the graph.

What is a question we might want to answer? Well, for a given bundle, we might like to know its class space. This can be done in the Network Visualizer by expanding the relevant edges. For example, this is the class space for com.novell.ldap as it is resolved in Anzo:

Novell ldap class space.png

I chose this bundle because its diagram is simple; there are no uses constraints in this bundle's class space, and it fits neatly on a page. However, some class spaces include hundreds of bundles with intricate use constraint interconnections. Those can be analyzed using SPARQL. See Appendix B for a substantially more complex class space diagram.

Let's say we wanted to write a SPARQL query that would give us a list of the Exported Packages (the black nodes) and their versions in the class space of a given bundle. We would want it to start with the bundle node, and then find all nodes that can be reached by following any combination of edges expressing an imports, source, or uses relationship. In SPARQL parlance, it would be called an arbitrary length path query. Since it would also give us Import Packages (white nodes), we also want to express in our query a filter to limit it to Export Packages. Here is what that would look like, for the same bundle com.novell.ldap:

PREFIX wiring: <http://openanzo.org/ontologies/2015/06/OSGIWiring#>
SELECT DISTINCT ?pname ?pversion
WHERE {
    { <osgi:bundle/com.novell.ldap_4.6.0> (wiring:imports | wiring:source | wiring:uses)+ ?package .
      ?package a ?ptype .
      ?package wiring:packageVersion ?pversion .
      ?package wiring:packageName ?pname . }
    FILTER (?ptype = wiring:ExportPackage)
}

Running the above query returns the following:

?pname                                   ?pversion
---------------------------------------  ---------
"javax.net"                              "0.0.0"
"javax.security.auth.callback"           "0.0.0"
"org.apache.commons.httpclient"          "3.1.0"
"javax.xml.parsers"                      "1.3.1"
"org.apache.commons.httpclient.methods"  "3.1.0"
"org.w3c.dom"                            "3.0.0"
"org.xml.sax"                            "2.0.2"
"org.xml.sax.helpers"                    "2.0.2"
"javax.net.ssl"                          "0.0.0"

In Practice

Note that this tool only generates graph data for resolved bundles. Yet, it is meant to be used when there is a uses constraint violation, and thus bundles cannot be resolved. Thus the usage pattern looks something like this:

  1. Add a dependency and realize that it has caused a uses constraint violation
  2. Back off the change so that bundles resolve normally
  3. Use graph analytics to assess the cause of the violation
  4. Develop solution or action plan
  5. Implement solution and bundles resolve successfully

Let's go through each of those steps in a concrete example.

  1. Add a dependency and realize that it has caused a uses constraint violation

    When I began, the bundle com.owlike.genson in our system imported only javax.xml.datatype, javax.xml.bind.annotation, and javax.xml.bind.annotation.adapters. However, that is not indicative of all the bundles that are referenced in its code. For example, com.owlike.genson.ext.jaxrs.GensonJsonConverter implements javax.ws.rs.ext.MessageBodyReader, but javax.ws.rs.ext was not declared as an import. This was fine because we had never had a code path which tried to load com.owlike.genson.ext.jaxrs.GensonJsonConverter, but once I added such a code path, I ran into a ClassNotFoundException at runtime.

    To resolve the ClassNotFoundException, I added javax.ws.rs.ext as an import-package in the manifest of com.owlike.genson. This produced a uses constraint violation, and Anzo would no longer load.

  2. Back off the change so that bundles resolve normally

    I undid my change to the manifest of com.owlike.genson, and Anzo loaded once more.

  3. Use graph analytics to assess the cause of the violation

    For this task, I want to express in the graph the relationship between com.owlike.genson and javax.ws.rs.ext: the fact that although I backed off my change and the one does not import the other, that my goal is to have the one import the other. That way, I can use SPARQL to compute the theoretical class space, and see which packages are present more than once. I express this relationship using an edge which I add manually labelled "desiredImports". It looks like this:
    Genson desired import.png
    Next, I calculate my SPARQL theoretical class space by altering my prior SPARQL query so that it also uses the desiredImports relationship in addition to the ones it already uses. The query looks like this:
    PREFIX wiring: <http://openanzo.org/ontologies/2015/06/OSGIWiring#>
    SELECT DISTINCT ?pname ?pversion
    WHERE {
        { <osgi:bundle/com.owlike.genson_1.0.0> (wiring:imports | wiring:source | wiring:uses | wiring:desiredImports)+ ?package .
          ?package a ?ptype .
          ?package wiring:packageVersion ?pversion .
          ?package wiring:packageName ?pname . }
        FILTER (?ptype = wiring:ExportPackage)
    }
    This returns:
    ?pname                                ?pversion
    ------------------------------------  -----------
    "javax.ws.rs.ext"                     "2.0.0.m10"
    "javax.ws.rs"                         "2.0.0.m10"
    "javax.xml.bind.annotation"           "2.1.7"
    "javax.ws.rs.core"                    "2.0.0.m10"
    "javax.xml.bind.annotation.adapters"  "2.1.7"
    "javax.xml.datatype"                  "1.3.1"
    "javax.xml.bind"                      "2.1.7"
    "javax.xml.parsers"                   "1.3.1"
    "javax.xml.transform.dom"             "1.3.1"
    "javax.xml.transform"                 "1.3.1"
    "org.w3c.dom"                         "3.0.0"
    "javax.xml.bind.annotation"           "0.0.0"
    "javax.xml.bind.annotation.adapters"  "0.0.0"
    "javax.xml.namespace"                 "1.3.1"
    "javax.xml.bind.attachment"           "2.1.7"
    "javax.xml.validation"                "1.3.1"
    "org.xml.sax"                         "2.0.2"
    "javax.xml.stream"                    "1.0.1"
    "javax.activation"                    "1.1.1"
    A cursory glance tells us that we have two packages appearing twice in the list: javax.xml.bind.annotation and javax.xml.bind.annotation.adapters. We can tell this quickly from the list because it is small, or because we can copy it into a text file and use some clever bash scripting. However, another way we can find items which appear in the list twice is to use the SPARQL GROUP BY construct. Instead of showing the package names and versions, we can show package names and how often they appear in the list. Here, I will implement this using a subquery. Also, I will filter out packages that only appear once, with a HAVING clause. In total it looks like this:
    PREFIX wiring: <http://openanzo.org/ontologies/2015/06/OSGIWiring#>
    SELECT ?pname (COUNT(?pname) as ?pcount)
    WHERE
    {
      {
        SELECT DISTINCT ?pname ?pversion
        WHERE {
            { <osgi:bundle/com.owlike.genson_1.0.0> (wiring:imports | wiring:source | wiring:uses | wiring:desiredImports)+ ?package .
              ?package a ?ptype .
              ?package wiring:packageVersion ?pversion .
              ?package wiring:packageName ?pname . }
            FILTER (?ptype = wiring:ExportPackage)
        }
      }
    }
    GROUP BY ?pname
    HAVING(?pcount > 1)
    Running this, the output is:
    ?pname                                ?pcount (xsd:integer)
    ------------------------------------  ---------------------
    "javax.xml.bind.annotation.adapters"  2
    "javax.xml.bind.annotation"           2
  4. Develop solution or action plan

    Knowing that the javax.xml.bind.annotation.adapters and javax.xml.bind.annotation packages were the source of my problem, I started to dig into what it really meant to have both version 0.0.0 and version 2.1.7 in my workspace. It turns out that version 0.0.0 is the version that OSGi assigns to packages for which an export version is not specified. The export version is not specified in this case because it is a package provided by the Java Runtime. Since I am running Java 7, this table tells me that the true version of those packages is 2.2.4-1 or higher, which is actually higher than the other version in my workspace. So, I decided to lower require-version of those packages in the com.owlike.genson manifest. This way the packages with version 0.0.0 would be used across the class space.

  5. Implement solution and bundles resolve successfully

    Once I changed the com.owlike.genson manifest, lowering the require-version of javax.xml.bind.annotation.adapters and javax.xml.bind.annotation, and adding the imports javax.ws.rs.ext, javax.ws.rs, javax.ws.rs.core, and javax.xml.namespace (the latter three required as dependencies of javax.ws.rs.ext), I was able to load Anzo without any bundle resolve errors or ClassNotFoundError's.

Appendices

Appendix A

import org.osgi.framework.Bundle;
import org.osgi.framework.BundleContext;

import org.osgi.framework.wiring.BundleCapability;
import org.osgi.framework.wiring.BundleRevision;
import org.osgi.framework.wiring.BundleWire;
import org.osgi.framework.wiring.BundleWiring;

public class BundleWiringIntrospection {
    public BundleWiringIntrospection(BundleContext bundleContext) {
        Bundle[] existingBundles = context.getBundles();
        for (Bundle b : existingBundles) {
            processBundle(b);
        }
    }

    private void processBundle(Bundle b) {
        BundleWiring bundleWiring = b.adapt(BundleWiring.class);

        if (bundleWiring == null) {
            //Not resolved
            return;
        }

        BundleRevision revision = bundleWiring.getRevision();
        List<BundleCapability> capabilities = bundleWiring.getCapabilities(null);
        List<BundleWire> requiredWires = bundleWiring.getRequiredWires(null);
        ...
    }
}

Appendix B

A complex class space diagram (based on the jcalais bundle):

Complex class space.png

 

Aron can be reached at [email protected]

Cambridge Semantics is hiring engineers! Send a note to [email protected]

Topics: Anzo, OSGi