State(Art)

Kivakit Manifesto

Tue, 09 Aug 2022 00:00:00 +0000

The KivaKit Manifesto — A New Vision for Java

The KivaKit project is more than a modular application framework for Java.

It’s a new way to think about coding in Java.

The Mission

The mission of KivaKit is to provide a new vision for the development of Java software by providing a design system that takes code reuse to a new level.

The primary goal of the KivaKit coding system is to deeply unify Java coding so that units of reuse (classes and packages) can be discovered and reused blindly, without the need to understand the most common API inconsistencies.

Reuse

Code reuse has improved through various means since the early days of Java. The JDK has improved, providing better APIs as well as modular Java. Third party frameworks like Spring, OSGi, JSF and Apache Wicket, have provided ways to build and reuse code.

But the JDK does not have interfaces consistent enough to discover and use blindly. For example, the java.net package and the java.io package are both concerned with accessing resources, and they have common integration points (streams), they do not have the same look and feel to the developer. Network resources and files don’t have the same semantics. The contents of a package are not accessed in the same way as a folder. Similar criticisms can be made of aspects of many third-party frameworks.

What we mean by discoverability and blind access here is that many, if not most, aspects of reusable units should not have a learning curve. It should be possible for a developer to discover a new class or package and know how it is structured and how it should behave without learning too much.

Reusable units should have a common structure and consistent behavior that can be relied on.

This sounds like a lofty goal, but how can it work in practice?

KivaKit API Design

KivaKit’s system for designing APIs provides discoverability, common structure, and consistent behavior through:

1. Interface Composition

KivaKit enables the consistent composition of components and packages. Since objects have the same structure they are easier to recognize. They are also more discoverable because they have the same recognizable features. KivaKit makes it easy to put objects together in complex ways without abandoning the familiarity of the Java extends and implements keywords. For example, BaseComponent and ComponentMixin are defined in terms of many small pieces, each with only a few methods at most:

Fine-grained interface composition permits more reuse because code accepting interfaces is applicable to more objects.

Fine-grained interface composition is less fragile because interfaces are smaller and simpler, producing stable contracts that are easy to implement.

2. Mixins

The common structure of reusable units in KivaKit also provides recognizable behavior in the form of inherited and composited features. In KivaKit, objects often provide behavior in two forms. Behavior can be inherited by extending a base class or it can he inherited by implementing an interface:

class Alien extends BaseComponent     // Extend abstract class or
class Alien implements ComponentMixin // implement mixin

Here, both forms of inheritance provide Alien with the Component API (see the diagram above), freeing the developer to build and compose objects without regard to the limitations of Java’s inheritance model.

3. Error Handling

Through messaging behavior (see the diagram above), KivaKit provides consistent and flexible error handling. Errors are handled in the same way, in every component, and by every piece of code. This means we don’t need to make a choice when writing an API. The choice is already made:

All components in KivaKit obey these error reporting rules:

Any exceptions caught are converted to broadcast messages

Any exceptions thrown must be for flow control of fluent method chains

All code handles errors by listening to messages

This error handling system is flexible enough that we can choose how we want to handle errors for any component at the point of use. Since all KivaKit components report and consume errors in the same way, this relieves an important barrier to both design and consumption of components. We no longer need to know anything about error handling to build or use a component.

4. Application Design

Using consistent error handling and composition, KivaKit provides commonly needed pieces that enable quick and easy application and component design. These components cover common needs, including project structure, applications, components, settings, object discovery, paths, resources, package access, serialization, progress reporting, conversion, validation and so on. KivaKit also unifies key JDK functionality with the KivaKit coding model. This allows most Java code to look and feel the same.

Conclusion

The KivaKit project provides a new way to create objects with common structure and behavior that are easy to discover and use.

Status Messaging

Mon, 07 Feb 2022 00:00:00 +0000

2022.02.07

Why Messaging is a Better Way to Report Status

In a typical Java program, the status of a method call is typically reported in one of three ways:

The method returns a value containing status information
The method throws an exception containing status information
The method logs or otherwise stores status information

One result of this is that every API has different error reporting semantics. We have all seen code similar to these examples:

Status Reporting via Return Values

var sender = new EmailSender();

var result = sender.send(email);
if (result != null)
{
    logger.error(result);
}

if (!sender.send(email))
{
    logger.error("Could not send");
}

var error = sender.send(email);
if (error < 0)
{
    logger.error("Couldn't send email: Error #" + error);
}

Status Reporting via Exceptions

var sender = new EmailSender();

try
{
    sender.send(email);
}
catch (SendException e)
{
    logger.error("Unable to send", e);
}

try
{
    sender.send(email);
}
catch (SendException e)
{
    throw new EmailException("Couldn't send email", e);
}

An alternative to these common idioms is to report status “out-of-band” by broadcasting messages to interested listeners. This has a few immediate advantages:

It standardizes error reporting, allowing API users to treat all components in the same way, reducing the learning curve for new APIs
It allows return values and exceptions to be used only for control flow
It simplifies and clarifies the responsibilities of components by decoupling status reporting from status handling
It allows multiple listeners and chains of listeners to handle the same status information in different ways

This status reporting model is implemented by the Java Open Source microservices framework, KivaKit. In KivaKit, our EmailSender class would implement Repeater (which is a kind of Broadcaster) by extending BaseRepeater or (more likely) BaseComponent.

class EmailSender extends BaseRepeater
{
    Connector connector = listenTo(new Connector());
    
    boolean send(Email email)
    {
        if (!connector.connect())
        {
            problem("Could not send");
            return false;
        }

            [...]
            
        return true;
    }
}

class Connector extends BaseRepeater
{
    boolean connect()
    {
        if ( [...] )
        {
            problem("Cannot connect to server");
        }
    }
}

class Client extends BaseRepeater
{
    EmailSender sender = listenTo(new EmailSender());

    void sendReport()
    {
        sender.send(composeReport());
    }
}

Here, we can see that the Connector object created during construction is passed to listenTo(). The listenTo() method, inherited from BaseRepeater, causes EmailSender to listen for any messages broadcast by Connector. Since EmailSender is a Repeater, it will repeat any messages it receives from Connector to its own listeners in Client, forming a listener chain:

Connector ==> EmailSender ==> Client ==> [...]

Note: In KivaKit, broadcasting messages does not involve message queueing or concurrency. Messages are delivered to listeners synchronously via the method Listener.onMessage().

What’s important here is that Connector and EmailSender only report problems regarding their own status. If Connector is unable to connect(), it simply calls problem() explaining what it couldn’t do. The problem() method then transmits a Problem message to EmailSender, and EmailSender in turn transmits it to Client. In this way, EmailSender indirectly provides “out-of-band” information about Connector’s status to Client. Of course, EmailSender also broadcasts its own Problem message if the Connector.connect() method fails, without needing to worry about reporting why the connection failed.

The flow of control for KivaKit messaging is shown in this UML sequence diagram:

KivaKit's "Out-of-Band" Messaging

Client calls EmailSender.send(), which calls Connector.connect(). During the execution of each of these methods, status messages may be transmitted down the listener chain (as shown by the orange lines) when a method like problem() is called.

The rules of listener chains are:

Components are Repeaters which listenTo() any components they create to form a listener chain
Each component in the chain minds its own business, broadcasting any relevant status messages during method calls
Listeners can be added at any point along the chain to capture and process status information

Use of EmailSender in Client now looks like this:

var sender = listenTo(new EmailSender());
if (!sender.send(email))
{
    problem("Unable to send report: $", email);
    return false;
}
return true;

Because the semantics of KivaKit status reporting are so regular, it’s possible to condense this idiom even further:

return isTrueOr(sender.send(), "Unable to send report $", email);

The isTrueOr() method here is inherited from the LanguageTrait interface via the Component interface. If the boolean argument to isTrueOr() is not true, it broadcasts the message specified by the remaining arguments as a Problem. It then returns the boolean value. This eliminates a lot of boilerplate if/else nesting when checking multiple conditions. Code like this (23 lines):

if (isTimeToSend())
{
    if (email != null)
    {
        if (sender.send())
        {
            information("Email sent.");
            return true;
        }
        else
        {
            problem("Unable to send report");
        }
    }
    else
    {
        problem("No email to send");
    }
}
else
{
    problem("Not time to send yet");
}

can be reduced to just this (8 lines):

if (isTrueOr(isTimeToSend(), "Not time to send yet") &&
    isNonNullOr(email, "No email to send") &&
    isTrueOr(sender.send(email), "Unable to send report"))
{
    information("Email sent.");
    return true;
}
return false;

A Few KivaKit Listeners and Repeaters

KivaKit has hundreds of Listeners and Repeaters. This includes all KivaKit Components. To give an idea of the range of Listener implementations:

Listener	Purpose
Logger	Log status messages
MutableCount	Count status messages
MessageAlarm	Trigger an alarm if there are too many problems
StatusPanel	Show status messages in a Swing panel
MessageList	Capture messages in a list
ValidationIssues	Capture problems during validation
Converter	Broadcast conversion errors
ThrowingListener	Throws an exception when a problem is received
ConsoleWriter	Writes status messages directly to the console
MicroservletRequest	Captures microservlet request handling errors
Application	Terminal listener that logs and counts messages

Conclusion

In this short article we took a look at the advantages of broadcaster/listener messaging in status reporting over more typical idioms. We then took a look at how KivaKit implements this system. More information on KivaKit messaging is available on my blog as well as on GitHub.

Questions? Comments? Tweet yours to @OpenKivaKit or post here:

Docker Build

Fri, 28 Jan 2022 00:00:00 +0000

2022.01.28

KivaKit - Docker Build Environment

KivaKit 1.2.3 provides a Docker build environment that makes it easy to build KivaKit without installing software or configuring the build environment.

Launching the Build Environment

Once Docker has been installed, the KivaKit Docker build environment can be launched with a few simple commands:

export KIVAKIT_WORKSPACE=~/workspaces/kivakit

mkdir -p $KIVAKIT_WORKSPACE 
cd $KIVAKIT_WORKSPACE
git clone --branch develop https://github.com/Telenav/kivakit.git
bash $KIVAKIT_WORKSPACE/kivakit/setup/setup-repositories.sh

export KIVAKIT_BUILD_IMAGE=1.2.3-snapshot

docker run \
    --volume "$KIVAKIT_WORKSPACE:/host/workspace" \
    --volume "$HOME/.m2:/host/.m2" \
    --volume "$HOME/.kivakit:/host/.kivakit" \
    --interactive --tty "jonathanlocke/kivakit:$KIVAKIT_BUILD_IMAGE" \
    /bin/bash

The build environment can build source code in a workspace on the host as well as inside Docker, so the first line above specifies the host workspace. The next four lines clone all the required repositories into that workspace. The second export line then specifies the KivaKit Docker tag to launch. Finally, the docker run command launches the build environment.

Build Scripts

Several build scripts are available in the Docker build environment, and they are all made available on our shell’s PATH. To build KivaKit, we would use kivakit-build.sh. Since this script is on our PATH, we can build KivaKit without changing our current working directory with cd:

kivakit-build.sh

We can see all available KivaKit scripts by typing kivakit- followed by TAB. Here are some of the more important scripts:

KivaKit Script	Purpose
kivakit-version.sh	Show KivaKit version
kivakit-build.sh	Build KivaKit
kivakit-build-documentation.sh	Build KivaKit documentation
kivakit-git-pull.sh	Pull changes from GitHub **
kivakit-git-checkout.sh [branch]	Check out the given branch **
kivakit-docker-build-workspace.sh	Switch between host and Docker workspaces
kivakit-feature-start.sh [branch]	Start a git flow feature branch **
kivakit-feature-finish.sh [branch]	Finish a git flow feature branch **

** executes the command in each kivakit repository

Build Workspaces

When the Docker build environment starts up, it is set up to build a workspace provided inside the container. This baseline build should work nearly all of the time, because the build environment is standardized, and the build should have worked at the time the Docker image was created. Having a common, standardized build environment provides us with an easy way to determine if there’s a problem in the source code versus a build environment problem. If the Docker workspace builds successfully, we can be fairly sure that there is nothing wrong with code, even if the host workspace build fails.

The kivakit-docker-build-workspace.sh script allows us to switch between the Docker and host workspaces. Initially, the workspace inside the Docker container is set up to build. If we want to switch to the host workspace, we can issue this command (use TAB to complete the long script name):

kivakit-docker-build-workspace.sh host

Switching to the host workspace will allow us to do an initial build of the code we cloned into that workspace above. Once the host workspace has been built, KivaKit projects can be imported into an IDE like IntelliJ.

But how does workspace switching work?

Notice that the docker run command has three Docker –volume mounts. The –volume switch makes a host folder visible inside a Docker container:

--volume [host-folder]:[docker-folder]

So, the docker run command we issued above makes these three host folders available under the /host folder inside Docker:

Docker Mount Folder	Host Folder
/host/workspace	$KIVAKIT_WORKSPACE
/host/.m2	$HOME/.m2
/host/.kivakit	$HOME/.kivakit

When we start up the Docker build environment, the KIVAKIT_WORKSPACE environment variable points to a workspace in the /root folder inside our container, and the symbolic links /root/.m2 and /root/.kivakit point to folders under /root/developer, also inside our container:

Docker Reference	Target Folder
KIVAKIT_WORKSPACE	/root/workspace
/root/.m2	/root/developer/.m2
/root/.kivakit	/root/developer/.kivakit

When we switch to the host workspace, the KIVAKIT_WORKSPACE environment variable and the symbolic links are switched to point to folders that have been mapped into the Docker container from the host (using the –volume mount switches, as described above). Now our environment looks like this:

Docker Reference	Target Folder
KIVAKIT_WORKSPACE	/host/workspace
/root/.m2	/host/.m2
/root/.kivakit	/host/.kivakit

and, when we execute kivakit-build.sh in Docker, the host’s workspace will be built, using our standardized Docker build environment.

Code

The scripts for the KivaKit Docker build environment are here:

Questions? Comments? Tweet yours to @OpenKivaKit or post here:

Formatting

Wed, 29 Dec 2021 00:00:00 +0000

2021.12.25

Kernel - Message Formatting and Template Expansions

The module kivakit-kernel supports a simple variable substitution syntax. This syntax can be used when formatting messages, or when substituting variables into templates.

Formatting a Message

Basic message formatting is achieved with the Message.format() method:

var formatted = Message.format("Hello my name is $", name);

The symbol $ is an expansion marker, and the corresponding argument is substituted into the formatted string at the marker’s location:

var formatted = format("argument1 = $, argument2 = $", argumentOne, argumentTwo);

Here, argumentOne’s string value will be substituted for the first $ and argumentTwo’s string value will be substituted for the second $. Both arguments will be converted to string values using Strings.toString(Object).

In addition to this basic subsitution syntax, arguments can be formatted in a variety of ways using the syntax ${format}, where format is one of the following:

Format	Description
$$	evaluates to a literal ‘$’
${class}	converts a Class argument to a simple (non-qualified) class name
${hex}	converts a long argument to a hexadecimal value
${binary}	converts a long argument to a binary string
${integer}	converts an integer argument to a non-comma-separated numeric string
${long}	converts a long argument to a non-comma-separated numeric string
${float}	converts a float argument to a string with one digit after the decimal
${double}	converts a double argument to a string with one digit after the decimal
${debug}	converts an argument to a string using DebugString.toDebugString() if that interface is supported.
${object}	uses an ObjectFormatter to format the argument by reflecting on it
${flag}	converts boolean arguments to ‘enabled’ or ‘disabled’
${name}	converts the argument to the name returned by Named.name() if the argument is Named
${nowrap}	signals that the message should not be wrapped

For example:

format("The file named '${name}' was read in $", file, start.elapsedSince());

The reason for the ${long} and ${integer} formats in the table above is that int, long and Count objects are formatted by default with comma separators. This code:

Count lines;

    [...]

format("Processed $ lines.", lines);

will produce a string like:

Processed 1,457,764 lines.

Using ${integer} like this:

Count lines;

    [...]

format("Processed ${integer} lines.", lines);

produces string this instead:

Processed 1457764 lines.

Broadcasting a Message

When broadcasting a message from a component, each message broadcasting method (information(), warning(), problem(), etc.) accepts the same parameters as Message.format() and they are handled in the same way:

information("Processed $ lines.", lines);

Templates

The VariableMap and PropertyMap classes allow for easy template substitutions using a similar syntax:

var properties = PropertyMap.load(this, file);
properties.put("home", "/users/shibo");
properties.put("job", "/var/jobs/job1");
var expanded = properties.expand("Home = ${home}, Job = ${job}");

Here, the marker ${home} is replaced with the value returned by properties.get(“home”), and ${job} is replaced by properties.get(“job”).

Putting it all together, we can load a .properties file and a template, and expand the template like this:

Resource template;
Resource properties;

    [...]

var expanded = PropertyMap.load(this, properties).expand(template.string());

The properties file resource properties is read with load(), broadcasting any warning or problem messages to this object. The property map is then expanded into the template read from the resource template with Resource.string(). Yes, reading and expanding a template in KivaKit is a one-liner.

Code

The code discussed above is available on GitHub:

kivakit-kernel

The KivaKit kernel, is available on Maven Central at these coordinates:

<dependency>
    <groupId>com.telenav.kivakit</groupId>
    <artifactId>kivakit-kernel</artifactId>
    <version>1.2.0</version>
</dependency>

Questions? Comments? Tweet yours to @OpenKivaKit or post here:

Aws Lambda

Sat, 25 Dec 2021 00:00:00 +0000

2021.12.25

KivaKit and AWS Lambda

KivaKit 1.2 adds seamless support for AWS Lambda. Lambdas for REST and GRPC can be added to a KivaKit Microservice without alteration (which will make this a short article).

Creating a Lambda

We have already seen a KivaKit request handler for REST in the Microservices article. We will simply reuse this code as our Lambda request handler. As a reminder the code from that article looks like this:

@OpenApiIncludeType(description = "Request for divisive action")
public class DivisionRequest extends BaseMicroservletRequest
{
    @OpenApiIncludeType(description = "Response to a divide request")
    public class DivisionResponse extends BaseMicroservletResponse
    {
        @Tag(1)
        @Expose
        @OpenApiIncludeMember(description = "The result of dividing",
                              example = "42")
        int quotient;

        public DivisionResponse()
        {
            this.quotient = dividend / divisor;
        }

        public String toString()
        {
            return Integer.toString(quotient);
        }
    }

    @Tag(1)
    @Expose
    @OpenApiIncludeMember(description = "The number to be divided",
                          example = "84")
    private int dividend;

    @Tag(2)
    @Expose
    @OpenApiIncludeMember(description = "The number to divide the dividend by",
                          example = "2")
    private int divisor;

    public DivisionRequest(int dividend, int divisor)
    {
        this.dividend = dividend;
        this.divisor = divisor;
    }

    public DivisionRequest()
    {
    }

    @Override
    @OpenApiRequestHandler(summary = "Divides two numbers")
    public DivisionResponse onRequest()
    {
        return listenTo(new DivisionResponse());
    }

    @Override
    public Class<DivisionResponse> responseType()
    {
        return DivisionResponse.class;
    }

    @Override
    public Validator validator(ValidationType type)
    {
        return new BaseValidator()
        {
            @Override
            protected void onValidate()
            {
                problemIf(divisor == 0, "Cannot divide by zero");
            }
        };
    }
}

Adding a Lambda Service

In a similar fashion to adding a REST service, a Lambda service is added like this:

public class DivisionMicroservice extends Microservice
{
    [...]

    @Override
    public MicroserviceLambdaService onNewLambdaService()
    {
        return new DivisionLambdaService(this);
    }
}

The onNewLambdaService() method returns an instance of DivisionLambdaService, which extends MicroserviceLambdaService:

public class DivisionLambdaService extends MicroserviceLambdaService
{
    [...]

    @Override
    public void onInitialize()
    {
        mount("division", "1.0", DivisionRequest.class);
    }
}

When the service is initialized, a call to the mount() method in onInitialize() is used to associate the name of our lambda and its version with the handler DivisionRequest. Nothing more is required.

Code

The code discussed above is available on GitHub:

The KivaKit Microservice API, including support for AWS Lambda, is available on Maven Central at these coordinates:

<dependency>
    <groupId>com.telenav.kivakit</groupId>
    <artifactId>kivakit-microservice</artifactId>
    <version>1.2.0</version>
</dependency>

Questions? Comments? Tweet yours to @OpenKivaKit or post here:

Clustering

Wed, 22 Dec 2021 00:00:00 +0000

2021.12.22

KivaKit Clustering

KivaKit provides built-in support for clustering of microservices using Apache Zookeeper. It supplies a cluster model that is updated as members join and leave the cluster, and an implementation of the SettingsStore interface that stores settings in Zookeeper.

Joining and Leaving a KivaKit Microservice Cluster

To use KivaKit in a cluster, Apache Zookeeper must be running according to the instructions. The default port for Zookeeper is 2181.

The source code for a clustered microservice should be organized like this:

├── deployments
│   └── mycluster
│       ├── ZookeeperConnection.properties
│       └── MyMicroserviceSettings.properties
└── MyMicroservice

The ZookeeperConnection.properties file here configures ZookeeperConnection as specified by kivakit-configuration:

class       = com.telenav.kivakit.settings.stores.zookeeper.ZookeeperConnection$Settings
ports       = 127.0.0.1:2181
timeout     = 5m
create-mode = PERSISTENT

The Microservice subclass, MyMicroservice, is then parameterized on a class that holds information about cluster members. Each cluster member will have its own instance of this object, describing that particular member. An instance of this object is created during initialization by the onNewMember() method:

public class MyMicroservice extends Microservice<MyMicroserviceSettings>
{
    [...]
    

    protected MicroserviceClusterMember<MyMicroserviceSettings> onNewMember()
    {
        return new MicroserviceClusterMember<>(require(MyMicroserviceSettings.class));
    }
}

Here, the MicroserviceClusterMember model returned by this method references an instance of the MyMicroserviceSettings, which is created and registered during initialization, typically by a settings file in the deployment like MyMicroserviceSettings:

class    = myapp.MyMicroserviceSettings
port     = 8081
grpcPort = 8082
server   = true

Once the MicroserviceClusterMember model object has been created, it is stored in Zookeeper. Other microservice instances in the cluster then receive a notification that a new member has joined through this Microservice method:

protected void onJoin(MicroserviceClusterMember<MyMicroserviceSettings> member)
{
    announce("Joined cluster: $", member.identifier());
}

When a member leaves the cluster, its model object will disappear from Zookeeper, and the remaining cluster members will be notified with a call to:

protected void onLeave(MicroserviceClusterMember<MyMicroserviceSettings> member)
{
    announce("Left cluster: $", member.identifier());
}

Cluster Elections

A cluster has an elected leader at any given time. Each time a member joins or leaves the cluster, an election is held to determine which member should lead the cluster. The election takes place automatically and the elected member will have the first MicroserviceClusterMember.identifier() value alphabetically. This identifier is currently the DNS name of the host and the process number, but is only guaranteed to be unique and is subject to change in the future.

To determine if a cluster member is the elected leader:

if (member.isLeader())
{
    [...]
}

Zookeeper Settings Stores

Although some applications may require only the settings object provided by MicroserviceClusterMember, others may need to store settings for other components in Zookeeper. This can be accomplished by registering a ZookeeperSettingsStore instance in Microservice.onInitialize():

var store = listenTo(register(new ZookeeperSettingsStore(PERSISTENT)));

Settings can be loaded from this store with

registerSettingsIn(store);

and settings can be saved to this store with:

saveSettingsTo(store, settings);

Because settings are retrieved in KivaKit using a pull model, any changes to settings objects will be automatically available when the desired object is next retrieved with require(Class).

A typical idiom is to read any existing settings from the store, and if the desired settings object is not present, save a default value. Other members will then read that value using the same logic:

registerSettingsIn(store);
if (!hasSettings(MyMicroserviceSettings.class))
{
   store.save(new MyMicroserviceSettings());
   registerSettingsIn(store);
}

Manually modifying the GSON value in Zookeeper will have the identical effect to saving a value with saveSettingsTo().

Code

The code discussed above is available on GitHub:

The KivaKit Microservice API is available on Maven Central at these coordinates:

<dependency>
    <groupId>com.telenav.kivakit</groupId>
    <artifactId>kivakit-microservice</artifactId>
    <version>1.2.0</version>
</dependency>

The KivaKit Zookeeper settings store API is at these coordinates:

<dependency>
    <groupId>com.telenav.kivakit</groupId>
    <artifactId>kivakit-settings-stores-zookeeper</artifactId>
    <version>1.2.0</version>
</dependency>

Questions? Comments? Tweet yours to @OpenKivaKit or post here:

Xml Streaming

Fri, 12 Nov 2021 00:00:00 +0000

2021.11.12

KivaKit XML Streaming

Since Java 1.6 in 2006, Java has had a built-in XML streaming API in the package javax.xml.stream. This API is known as StAX (Streaming API for XML), and it is a very efficient “pull parser”, allowing clients to iterate through the sequence of elements in an XML document. Other approaches to working with XML are event-handling “push parsers”, and full-blown, in-memory DOMs (Document Object Models). Although StAX is convenient and very fast, it can be significantly harder to work with than a DOM, because the hierarchy of the document that’s being streamed is lost. Our code sees just one element at a time.

KivaKit’s New XML Mini-framework

KivaKit 1.1 quietly added a small, but useful mini-framework to the kivakit-extensions repository called kivakit-data-formats-xml. The project contains just two simple classes: StaxReader and StaxPath. The StaxReader class adds a layer of convenience to the Java StAX API by making it easy to:

Open and close XML streams
Get information about the reader’s stream position
Advance through XML elements
Determine the reader’s hierarchical positioning in the stream

Moving Through an XML Stream

The static StaxReader.open(Resource) method is used to start reading an XML stream. The method either returns a valid StaxReader that’s ready to go, or it throws an exception. Since StaxReader implements Closeable, it can be used within a try-with-resources statement:

try (var reader = StaxReader.read(file))
{
    [...]
}

Within our try-with-resources block, we can advance through the stream with these methods:

hasNext()
next()
at()
nextAttribute()
nextCharacters()
nextOpenTag()
nextCloseTag()
nextMatching(Matcher)

When we reach the end of the stream, hasNext() will return false. So, processing an XML file looks like this:

try (var reader = StaxReader.read(file))
{
    for (; reader.hasNext(); reader.next())
    {
        var element = reader.at();
        
        [...]
        
    }        
}

As we stream our way through an XML document, a few simple methods can help us to identify what kind of tag the reader is currently at:

isAtEnd()
isAtCharacters()
isAtOpenTag()
isAtCloseTag()
isAtOpenCloseTag()

Streaming Through an XML Hierarchy

Although the underlying StAX API can only move through a document in sequential order, StaxReader adds functionality that allows us to determine where we are in the document hierarchy as we move along. Using the hierarchical path to our current position the stream, we can search for specific elements in the nested document structure, and we can process data when we reach those elements.

Okay. Let’s make this concrete. Here is a simple document:

<a>   <---- The path here is a
    <b>   <---- The path here is a/b
        <c>   <---- The path here is a/b/c
        </c>
    </b>
</a>

The StaxPath class represents hierarchical paths in our XML document. As seen above, the path at <a> is a. The path at <b> in our document is a/b and the path at <c> is a/b/c.

As we read elements from the XML stream, StaxReader tracks the current path using a stack of elements. When the reader encounters an open tag, it pushes the name of the tag onto the end of the current path. When it encounters a close tag, it pops the last element off the end of the path. So, the sequence of steps as StaxReader streams through our document is:

Step  Element     Action    StaxPath
   <a>         push a    a
     <b>       push b    a/b
       <c>     push c    a/b/c
       </c>    pop       a/b
     </b>      pop       a
   </a>        pop       

The current StaxPath for a StaxReader can be obtained by calling StaxReader.path().

Finding Elements in the Document Hierarchy

The following methods test the current path of our StaxReader reader against a given path. The reader is considered at a given path if the the reader’s path is equal to the given path. For example, if the reader is at the path a/b/c and the given path is a/b/c, the reader is at the given path. The reader is inside the given path if the given path is a prefix of the reader’s current path. For example, if the reader is at a/b/c/d and the path is a/b/c, then the reader is inside the given path. Finally, the reader is outside the given path in the reverse situation, where the reader is at a/b and the path is a/b/c.

isAt(StaxPath) - Returns true if the reader is at the given path
findNext(StaxPath) - Advances until the given path or the end of the document is reached
isInside(StaxPath) - Returns true if the reader is inside the given path
isAtOrInside(StaxPath) - Returns true if the reader is at or inside the given path
isOutside(StaxPath) - Returns true if the reader is outside the given path.

Putting it All Together: Fiasco

And now, the reason for doing all of this. I am working on a build tool for Java projects called Fiasco (named for Stanislaw Lem’s 1986 science fiction novel, Fiasko).

Fiasco is in the design stages still. When it is completed, it will be a pure-Java build tool with these design goals, and non-goals:

Goals

Intuitive API
Quick learning curve
Easy to understand and debug builds
Modular design
Easy to create new build tools
Reasonably fast

Non-Goals

Incremental compilation
Build security

How Fiasco Reads POM Files with StaxReader

To build a Java project, Fiasco needs to access artifacts in Maven repositories, like Maven Central. To be able to do this, it is necessary to parse Maven pom.xml files, and since Fiasco will be parsing a lot of these files, it is desirable to do it fairly efficiently. The PomReader class demonstrates how StaxReader can be used to parse a complex, hierarchical XML file like a Maven POM file. The relevant details are found in the method read(MavenRepository, Resource), which opens a pom.xml resource, parses it and returns a Pom model:

public class PomReader extends BaseComponent
{
    StaxPath PROPERTIES_PATH = StaxPath.parseXmlPath("project/properties");
    StaxPath DEPENDENCY_PATH = StaxPath.parseXmlPath("project/dependencies/dependency");

    [...]
    
    Pom read(MavenRepository repository, Resource resource)
    {
        [...]
        
        try (var reader = StaxReader.open(resource))
        {
            var pom = new Pom(resource);

            for (reader.next(); reader.hasNext(); reader.next())
            {
                [...]
                
                if (reader.isAt(PROPERTIES_PATH))
                {
                    pom.properties = readProperties(reader);
                }

                if (reader.isAt(DEPENDENCY_PATH))
                {
                    pom.dependencies.add(readDependency(reader));
                }
                
                [...]
             }
         }
         
     [...]        

 }

The code here that opens our pom.xml resource and moves through the elements of the XML document is essentially the same as we saw earlier. Within the for loop, is where we deal with the POM hierarchy. The StaxReader.isAt(StaxPath) method is used to determine when the reader lands on the open tag for the given path. When PROPERTIES_PATH (project/properties) is reached, we call a method that reads the properties nested within the <properties> open tag (see readProperties(StaxReader) for the gory details). For each DEPENDENCY_PATH (project/dependencies/dependency) we reach (there may be many of them), we read the <dependency> with logic similar to this:

Dependency readDependency(StaxReader reader)
{
    MavenArtifactGroup artifactGroup = null;
    String artifactIdentifier = null;
    String version = null;
    var scope = DEPENDENCY_PATH;

    // Skip past the <dependency> open tag we landed on,
    reader.next();

    // and while we're not outside the <dependency> tag scope,
    for (; !reader.isOutside(scope); reader.next())
    {
        // populate any group id,
        if (reader.isAt(scope.withChild("groupId")))
        {
            artifactGroup = MavenArtifactGroup.parse(this, reader.enclosedText());
        }

        // any artifact id,
        if (reader.isAt(scope.withChild("artifactId")))
        {
            artifactIdentifier = reader.enclosedText();
        }

        // and any version.
        if (reader.isAt(scope.withChild("version")))
        {
            version = reader.enclosedText();
        }
    }

So, there we have it. It is true that our code here is not as succinct as it would be with a DOM model. However, we can parse POM files well enough with StaxReader, and we save the time and memory required by a full, in-memory DOM model.

Code

The code discussed above is available on GitHub:

kivakit-data-formats-xml
Fiasco (GitHub)
PomReader.java

The KivaKit XML API is available on Maven Central at these coordinates:

<dependency>
    <groupId>com.telenav.kivakit</groupId>
    <artifactId>kivakit-data-formats-xml</artifactId>
    <version>${kivakit.version}</version>
</dependency>

Questions? Comments? Tweet yours to @OpenKivaKit or post here:

Microservices

Fri, 01 Oct 2021 00:00:00 +0000

2021.10.14

KivaKit Microservices

KivaKit is designed to make coding microservices faster and easier. In this blog post, we will examine the kivakit-microservice module. As of this date, this module is only available for early access via SNAPSHOT builds and by building KivaKit. The final release of KivaKit 1.1 will include this module and should happen by the end of October, 2021.

What does it do?

The kivakit-microservice mini-framework makes it easy to implement REST-ful GET, POST and DELETE request handlers, and to mount those handlers on specific paths. It also provides transparent support for GRPC using the same request handlers. Most of the usual plumbing for a REST microservice is taken care of, including:

Configuration and startup of Jetty web server
Handling GET, POST and DELETE requests
Serialization of JSON objects with Json
Error handling with KivaKit messaging
Generating an OpenAPI specification
Viewing the OpenAPI specification with Swagger
Starting an Apache Wicket web application

Microservices

The DivisionMicroservice class below is a Microservice that performs arithmetic division (in the slowest and most expensive way imaginable). The Microservice superclass provides automatic configuration and startup of Jetty server:

public class DivisionMicroservice extends Microservice
{
    public static void main(final String[] arguments)
    {
        new DivisionMicroservice().run(arguments);
    }

    @Override
    public MicroserviceMetadata metadata()
    {
        return new MicroserviceMetadata()
                .withName("division-microservice")
                .withDescription("Example microservice for division")
                .withVersion(Version.parse("1.0"));
    }

    @Override
    public void onInitialize()
    {
        // Register components here 
    } 
        
    @Override
    public DivisionRestService onNewRestService()
    {
        return new DivisionRestService(this);
    }
}

Here, the main(String[] arguments) method creates an instance of DivisionMicroservice and starts it running with a call to run(String[]) (the same as with any KivaKit application). The metadata() method returns information about the service that is included in the REST OpenAPI specification (mounted on /open-api/swagger.json). The onNewRestService() factory method creates a REST service for the microservice, and the webApplication() factory method optionally creates an Apache Wicket web application for configuring the service and viewing its status. Any initialization of the microservice must take place in the onInitialize() method. This is the best place to register components used throughout the application.

When the run(String[] arguments) method is called, Jetty web server is started on the port specified by the MicroserviceSettings object loaded by the -deployment switch. The -port command line switch can be used to override this value.

When the microservice starts, the following resources are available:

Resource Path	Description
/	Apache Wicket web application
/	KivaKit microservlet REST service
/assets	Static resources
/docs	Swagger OpenAPI documentation
/open-api/assets	OpenAPI resources (.yaml files)
/open-api/swagger.json	OpenAPI specification
/swagger/webapp	Swagger web application
/swagger/webjar	Swagger design resources

REST Services

A REST service is created by extending the MicroserviceRestService class:

public class DivisionRestService extends MicroserviceRestService
{
    public DivisionRestService(Microservice microservice)
    {
        super(microservice);
    }
    
    @Override
    public void onInitialize()
    {
        mount("divide", POST, DivisionRequest.class);
    }
}

Request handlers must be mounted on specific paths inside the onInitialize() method (or an error is reported). If the mount path (in this case “divide”) doesn’t begin with a slash (“/”), the path “/api/[major-version].[minor-version]/” is prepended automatically. So, “divide” becomes “/api/1.0/divide” in the code above, where the version 1.0 comes from the metadata returned by DivisionMicroservice. The same path can be used to mount a single request handler for each HTTP method (GET, POST, DELETE). However, trying to mount two handlers for the same HTTP method on the same path will result in an error.

The gsonFactory() factory method (not shown above) can optionally provide a factory that creates configured Gson objects. The Gson factory should extend the class MicroserviceGsonFactory. KivaKit will use this factory when serializing and deserializing JSON objects.

For anyone interested in the gory details, the exact flow of control that occurs when a request is made to a KivaKit microservice, is detailed in the Javadoc for MicroserviceRestService.

Microservlets

Microservlets handle MicroservletRequests by producing MicroservletResponses. They are mounted on paths in the same way that request handlers are mounted. Although Microservlet is a public API, so far the only use case is in dispatching requests to request handlers. The MicroservletRestService class uses an anonymous Microservlet class to forward requests to MicroservletRequestHandlers. You can see this in MicroserviceRestService in the method mount(String path, HttpMethod method, Class<MicroservletRequest> requestType).

Microservlet Request Handlers

Request handlers are mounted on a MicroserviceRestService with calls to mount(String path, HttpMethod method, Class<MicroserviceRequest> requestType).

Note that the BaseMicroservletRequest class implements both MicroservletRequest and MicroservletRequestHandler:

public abstract class BaseMicroservletRequest extends BaseComponent implements
        MicroservletRequest,
        MicroservletRequestHandler

This is crucial in terms of object-oriented design and encapsulation: the request object itself has the data and therefore knows how to produce a response using that data.

KEY DESIGN PRINCIPLE

Don’t ask for the information you need to do the work; ask the object that has the information to do the work for you.

By employing this principle we avoid getters and setters, and our abstraction does not “leak”.

Below we see a request handler, DivisionRequest, that divides two numbers. The request contains a dividend and a divisor. Its onRequest() method produces the response, which is implemented by the nested class DivisionResponse. The response has access to the dividend and the divisor in the outer request class. So, it can create the response quotient in its constructor:

public DivisionResponse()
{
    this.quotient = dividend / divisor;
}

The request and response can also perform self-validation, again following the key design principle above, by overriding the Validatable.validator() method, as seen below.

Note that an OpenAPI specification is generated using information from @OpenApi annotations

@OpenApiIncludeType(description = "Request for divisive action")
public class DivisionRequest extends BaseMicroservletRequest
{
    @OpenApiIncludeType(description = "Response to a divide request")
    public class DivisionResponse extends BaseMicroservletResponse
    {
        @Tag(1)
        @Expose
        @OpenApiIncludeMember(description = "The result of dividing",
                              example = "42")
        int quotient;

        public DivisionResponse()
        {
            this.quotient = dividend / divisor;
        }

        public String toString()
        {
            return Integer.toString(quotient);
        }
    }

    @Tag(1)
    @Expose
    @OpenApiIncludeMember(description = "The number to be divided",
                          example = "84")
    private int dividend;

    @Tag(2)
    @Expose
    @OpenApiIncludeMember(description = "The number to divide the dividend by",
                          example = "2")
    private int divisor;

    public DivisionRequest(int dividend, int divisor)
    {
        this.dividend = dividend;
        this.divisor = divisor;
    }

    public DivisionRequest()
    {
    }

    @Override
    @OpenApiRequestHandler(summary = "Divides two numbers")
    public DivisionResponse onRequest()
    {
        return listenTo(new DivisionResponse());
    }

    @Override
    public Class<DivisionResponse> responseType()
    {
        return DivisionResponse.class;
    }

    @Override
    public Validator validator(ValidationType type)
    {
        return new BaseValidator()
        {
            @Override
            protected void onValidate()
            {
                problemIf(divisor == 0, "Cannot divide by zero");
            }
        };
    }
}

We can appreciate now how time-tested OO design principles have improved encapsulation, eliminated boilerplate and increased code readability.

Accessing KivaKit Microservices in Java

The kivakit-microservice module includes MicroserviceClient, which provides easy access to KivaKit microservices in Java. The client can be used like this:

public class DivisionClient extends Application
{
    public static void main(String[] arguments)
    {
        new DivisionClient().run(arguments);
    }

    @Override
    protected void onRun()
    {
        var client = listenTo(new MicroservletClient(
            new MicroserviceGsonFactory(), 
            Host.local().https(8086), 
            Version.parse("1.0"));

        var response = client.post("divide", 
            DivisionRequest.DivisionResponse.class, 
            new DivisionRequest(9, 3));

        Message.println(AsciiArt.box("response => $", response));
    }
}

Here, we create a MicroservletClient to access the microservice we built above. We tell it to use the service on port 8086 of the local host. Then we POST a DivisionRequest to divide 9 by 3 using the client, and we read the response. The response shows that the quotient is 3:

-------------------
|  response => 3  |
-------------------

Path and Query Parameters

A request handler does not access path and query parameters directly. Instead they are automatically turned into JSON objects. For example, a POST to this URL:

http://localhost:8086/api/1.0/divide/dividend/9/divisor/3

does the exact same thing as the POST request in the DivisionClient code above. The dividend/9/divisor/3 part of the path is turned into a JSON object like this:

{
    "dividend": 9,
    "divisor": 3
}    

The microservlet processes this JSON just as if it had been posted. This feature can come in handy when POST-ing “flat” request objects (objects with no nesting). Note that when path variables or query parameters are provided, the body of the request is ignored.

OpenAPI

The “/docs” root path on the server provides a generated OpenAPI specification via Swagger:

The annotations available for OpenAPI are minimal, but effective for simple REST projects:

Annotation	Purpose
@OpenApiIncludeMember	Includes the annotated method or field in the specification
@OpenApiExcludeMember	Excludes the annotation method or field from the specification
@OpenApiIncludeMemberFromSuperType	Includes a member from the superclass or superinterface in the specification
@OpenApiIncludeType	Includes the annotated type in the specification schemas
@OpenApiRequestHandler	Provides information about a request handling method (onGet(), onPost() or onDelete())

GRPC

Google’s remote procedure call protocol (GRPC) is transparently supported by the kivakit-microservice mini-framework thanks to the Protostuff project. To enable the GRPC protocol (on a separate port specified by the -grpc-port switch), two changes are required. The first is to create a trivial subclass of MicroserviceGrpcService. The second is to instantiate this class in onNewGrpcService(). That’s it. Done.

public class DivisionGrpcService extends MicroserviceGrpcService
{
    public DivisionGrpcService(Microservice microservice)
    {
        super(microservice);
    }
}

public class DivisionMicroservice extends Microservice
{
    [...]
        
    @Override
    public DivisionGrpcService onNewGrpcService()
    {
        return new DivisionGrpcService(this);
    }
}

Note: The @Tag annotations in the DivisionRequest and DivisionResponse objects above tell Protostuff what order the fields should go into the protocol buffer. This is necessary to allow for schema evolution. See the Protostuff documentation for details.

Code

The code discussed above is a working example in the kivakit-examples repository. It can be instructive to trace through the code in a debugger.

The KivaKit Microservice API is available for early access in the develop branch of the kivakit-microservice module of the kivakit-extensions repository in KivaKit.

<dependency>
    <groupId>com.telenav.kivakit</groupId>
    <artifactId>kivakit-microservice</artifactId>
    <version>${kivakit.version}</version>
</dependency>

Questions? Comments? Tweet yours to @OpenKivaKit or post here:

State Watcher

Tue, 14 Sep 2021 00:00:00 +0000

2021.07.20

Signaling and waiting for concurrent state changes

Java’s concurrency library (java.util.concurrent) provides a mutual-exclusion (mutex) Lock called ReentrantLock. This lock maintains a queue of threads that are waiting to own the lock, allowing access to a protected resource. A thread can be added to the lock’s wait queue by calling lock(). When the lock() method returns, the thread will own the lock. Once the thread obtains the lock in this way, it can mutate any shared state protected by the lock, and then it can release its ownership by calling unlock(), allowing another thread to get its turn at owning the lock and accessing the shared state. Because the lock is reentrant, a thread can call lock() multiple times, and the lock will only be released to the next waiting thread when all nested calls to lock() have been undone with calls to unlock(). The flow of a reentrant thread using a lock looks like this:

lock() 
    lock() 
        lock() 
        unlock()
    unlock()
unlock()

KivaKit provides a simple extension of this functionality that reduces boilerplate calls to lock() and unlock(), and ensures that all lock calls are balanced by unlock calls:

public class Lock extends ReentrantLock
{
    /**
     * Runs the provided code while holding this lock.
     */
    public void whileLocked(Runnable code)
    {
        lock();
        try
        {
            code.run();
        }
        finally
        {
            unlock();
        }
    }
}

Use of this class looks like:

private Lock lock = new Lock();

[...]

lock.whileLocked(() -> mutateSharedState());

In addition to mutual exclusion, ReentrantLock (and in fact, all Java Lock implementations) provides an easy way for one thread to wait for a signal from another thread. This behavior makes ReentrantLock a condition lock, as declared in Java’s Lock interface:

public interface Lock
{
    void lock();
    void unlock();
    Condition newCondition();
}

The Condition implementation returned by newCondition has methods for threads that own the lock to signal or wait on the condition (similar to Java monitors). A simplification of the Condition interface looks like this:

public interface Condition
{
    void await() throws InterruptedException;
    void signal();
}

KivaKit uses condition locks to implement StateWatcher, which provides a way to signal and wait for a particular state.

For example:

enum State
{
    IDLE,     // Initial state where nothing is happening
    WAITING,  // Signal that the foreground thread is waiting
    RUNNING,  // Signal that the background thread is running
    DONE      // Signal that the background thread is done
}

private StateWatcher state = new StateWatcher(State.IDLE);

[...]

new Thread(() ->
{
    state.waitFor(WAITING); 
    state.signal(RUNNING);

    doThings();
    
    state.signal(DONE);
    
}).start();

state.signal(WAITING);
state.waitFor(DONE);

In this example, you might expect that this code has a race condition. It is okay if the thread starts up and reaches waitFor(WAITING) before the foreground thread reaches signal(WAITING). But what if the foreground thread signals that it’s WAITING and proceeds to wait for DONE before the background thread even starts? With Java monitors (or Conditions), the signal would be missed by the background thread. It would then hang forever waiting for a WAITING signal that will never come. The foreground thread would also hang waiting for a DONE signal that will never arrive. A classic deadlock scenario.

StateWatcher solves this issue by making signaling and waiting stateful operations. In our race condition case, the foreground thread calls signal(WAITING), as before. But the signal isn’t lost. Instead, StateWatcher records that it is in the WAITING state before proceeding to wait for DONE. If the background thread then finishes starting up and it calls waitFor(WAITING), the current state retained by StateWatcher will still be WAITING and the call will return immediately instead of waiting. Our deadlock is eliminated, and with a minimal amount of code. The state that StateWatcher keeps to allow this to happen is commonly known as a condition variable.

But how exactly does StateWatcher implement this magic?

StateWatcher has a State value that can be updated, and a (KivaKit) Lock that it uses to protect this state. It also maintains a list of Waiters, each of which has a Condition to wait on (created from the Lock) and a Predicate that it needs to be satisfied.

When the *waitFor(Predicate)* method is called (if the watcher isn't already in the desired *State*), a new *Waiter* object (see below) is created with the *Predicate* and a *Condition* created from the *Lock*. The *waitFor()* method then adds the *Waiter* to the wait list and *awaits()* future signaling of the condition.

When signal(State) is called, the current state is updated, and each waiter is processed. If a waiter’s predicate is satisfied by the new state, its condition object is signaled, causing the thread awaiting satisfaction of the predicate to be awakened.

Finally, waitFor(State) is simply implemented with a method reference to equals() as a predicate:

waitFor(desiredState::equals)

A simplified version of StateWatcher is shown below. The full StateWatcher class is available in kivakit-kernel in the KivaKit project.

public class StateWatcher<State>
{
    /**
     * A thread that is waiting for its predicate to be satisfied
     */
    private class Waiter
    {
        /** The predicate that must be satisfied */
        Predicate<State> predicate;

        /** The condition to signal and wait on */
        Condition condition;
    }

    /** The re-entrant (KivaKit) lock */
    private Lock lock = new Lock();

    /** The clients waiting for a predicate to be satisfied */
    private List<Waiter> waiters = new ArrayList<>();

    /** The most recently reported state */
    private State current;
    
    public StateWatcher(State current)
    {
        this.current = current;
    }

    /**
     * Signals any waiters if the state they are waiting for has arrived
     */
    public void signal(final State state)
    {
        lock.whileLocked(() ->
        {
            // Update the current state,
            current = state;

            // go through the waiters
            for (var watcher : waiters)
            {
                // and if the reported value satisfies the watcher's predicate,
                if (watcher.predicate.test(state))
                {
                    // signal it to wake up.
                    watcher.condition.signal();
                }
            }
        });
    }

    /**
     * Waits for the given boolean predicate to be satisfied based on changes * to the observed state value
     */
    public WakeState waitFor(Predicate<State> predicate)
    {
        return lock.whileLocked(() ->
        {
            // If the predicate is already satisfied,
            if (predicate.test(current))
            {
                // we're done.
                return COMPLETED;
            }

            // otherwise, add ourselves as a waiter,
            var waiter = new Waiter();
            waiter.predicate = predicate;
            waiter.condition = lock.newCondition();
            waiters.add(waiter);

            try
            {
                // and go to sleep until our condition is satisfied.
                if (waiter.condition.await())
                {
                    return TIMED_OUT;
                }
                else
                {
                    return COMPLETED;
                }
            }
            catch (InterruptedException e)
            {
                return INTERRUPTED;
            }
        });
    }

    /**
     * Wait forever for the desired state
     */
    public WakeState waitFor(State desired)
    {
        return waitFor(desired::equals);
    }
}

Code

The StateWatcher class is available in the kivakit-kernel module in KivaKit.

<dependency>
    <groupId>com.telenav.kivakit</groupId>
    <artifactId>kivakit-kernel</artifactId>
    <version>${kivakit.version}</version>
</dependency>

Questions? Comments? Tweet yours to @OpenKivaKit or post here:

Kivakit Build

Tue, 07 Sep 2021 00:00:00 +0000

2021.09.07

A poor man’s multiple-repository build system

Refactoring feature branches across multiple repositories

A common use case when working with multiple, dependent repositories is to use git flow to create multiple feature branches:

kivakit            [feature/simplify-log-api]
kivakit-extensions [feature/simplify-log-api]

If project(s) in kivakit-extensions here depend on projects in kivakit, refactoring code in kivakit can propagate code changes to kivakit-extensions. Then both feature branches are modified.

When we commit and push these feature branches (ideally all at the same time for convenience), it would be nice to know that our continuous integration (CI) build system will build them correctly. There are existing solutions to the problem of repository build order, including the KIE build chain tool. However, for KivaKit, we decided it was desirable to have a bit more flexibility than GitHub Actions provide with their .yaml configuration files, so we created a few simple scripts to manage our multi-repository builds. Remember Perl?

The KivaKit build system

In our poor man’s solution to this problem, each repository has its own .github/scripts/build.pl file that is invoked from a set of .yaml workflows. The relevant portion of a workflow .yaml file is very simple. It passes all the build responsibility off to the Perl script build.pl:

- name: Build
   run: |
     perl ./.github/scripts/build.pl package

The build.pl script for a given repository takes a build-type argument, which can be one of two values:

package - build the repository’s projects
publish - build the repository’s projects and publish them to OSSRH

The build script clones the repository Telenav/cactus-build into the GitHub Action workspace and includes Perl functions from a shared build-include.pl file in .github/scripts:

system("git clone --branch develop --quiet https://github.com/Telenav/cactus-build.git");

require "./cactus-build/.github/scripts/build-include.pl";

The script then uses the provided functions to clone and build any dependent repositories in the proper order, followed by the repository itself:

my ($build_type) = @ARGV;
my $github = "https://github.com/Telenav";

clone("$github/kivakit", "dependency");
clone("$github/kivakit-extensions", "build");

build_kivakit("package");
build_kivakit_extensions($build_type);

This script in kivakit-extensions clones kivakit and kivakit-extensions into the GitHub Actions workspace (the second parameter determines which branch gets checked out). It then builds the project kivakit before building the dependent project, kivakit-extensions.

Conclusion

This simple build system was created over a fun (and nostalgic!) weekend with Perl. It is true that this is not the most efficient way to build dependent repositories. And for private repositories, that inefficiency will make GitHub a little richer every day. However, this approach to clone and build all required projects, in order, on every repository build action, is conceptually simple, robust, flexible and easy to debug offline.

Caveat - Out of order pushes

It is useful to note here that if the kivakit-extensions build action clones kivakit before it has been pushed it will build against the wrong branch. Out-of-order pushes are a problem with any build method. Imagine that someone pushes a kivakit-extensions feature branch and then goes to lunch without pushing the corresponding kivakit feature branch. Failure is always possible with GitHub CI, because GitHub doesn’t know enough about repositories and branches, and how they relate to each other.

The KivaKit clone() function in build-include.pl makes this issue less likely (assuming you have a high bandwidth Internet connection) by simply delaying for 15 seconds before cloning dependencies. It is no substitute for a proper locking mechanism (which could ensure that cross-repository CI builds would never fail), but in practice it works nearly all the time when all interdependent feature branches are pushed at the same time.

Code

The KivaKit build system is somewhat specific to KivaKit in a few places, but it could easily be modified to work in other situations. Feel free to download it and tailor it to your needs. The Perl code for the kivakit project’s build is here:

The build and workflow files for other KivaKit projects are available in the same location in those projects.

Lexakai also uses the KivaKit build system.

Questions? Comments? Tweet yours to @OpenKivaKit or post here: