Drone.io, First Thoughts

I’ve been a happy user of Travis-ci for cloud CI/CD needs for a long time.  It’s straightforward to use, reliable, and free for open source projects. That said, it’s always worth looking at alternatives right?  Recently Drone.io started offering a free cloud based service so I gave it a spin.

The basics are very similar to Travis-ci. You add a YAML file to your project that describes you CI/CD pipeline. When you push your code, it signals the site to read your YAML file, fire up a clean environment, and perform your requests.  At first glance, save the syntax details of the commands, you’d be hard pressed differentiate the two.  But Drone.io requires a bit more nuts and bolts work right away.

  • Drone.io’s cloud is brand spanking new, and the site had a bit of a craigslist UX feel to it still. Travis-ci has been at it a while and the site is more powerful and elegant.
  • Drone.io, even in simple cases, demands you supply the docker image to create your environments with. Even to build a pure Java project I had to go over to docker hub and lookup a OpenJdk docker image to use. With Travis-ci I’ve always been able to use their basic built in images.
  • The Drone.io shell commands where pretty finicky about pipelines and file redirects. Some of my CI/CD basic steps took some finesse to achieve.
  • Drone.io’s git clone stripped off my my git tags(?!).  I use git tags to differentiate snapshots from releases, performing some tasks, like deployments, only for release tags.  I’m sure I could have figured out a solution, but honestly by then I’d lost interest. 

Drone.io was consistently just a bit more effort on everything, it felt a bit faster, and was perhaps more versatile, but I just didn’t feel it was worth it.

Advertisements

Gradle bootRun Task Might Deceive You

At work we had a Spring Boot RESTful microservice that lost some data in its responses only when deployed. The endpoint did what was expected in unit testing supposidly but when deployed some of the values in the JSON responses where empty (?).  Immediately suspicious of the quality of the unit tests I fired up the service using ./gradlew bootRun and hit the end point.  The data I was looking for was there. Head scratching ensured. Being thorough, and systematic, next I tried ./gradlew bootRepackage and ran the resultant jar file and tried the endpoint again. Bingo, the data disappeared.  

To be fair, I’ve left out a key fact. The data I was looking for was included in property file resources that were included in another dependency.  The RESTful endpoint was returning a catalog of information that pulled certain longer, internationalized, text descriptions from a dependant jar.  Those property file resources are ultimately loaded with ClassLoader.getResourceAsStream(). Now this worked in the dependency’s unit test. It worked in the service’s unit tests. It worked with ./gradlew bootRun but when you did ./gradlew bootRepackage and created a fat jar it failed. Why?

The issue was the fat jar. Fat jars are a little bit of a hack. Java likes to think of jars as a bundle of classes and resources. Java wants you to use jars in an additive manner, meaning if you have classes in multiple jars, you use the classpath to include the multiple jars.  It’s not a recursive model, Java doesn’t want a jar full of jars. But if you want to distribute an application, creating a single package is, for simplicity, where you want to end up.  So there’s a mismatch between the goal of a single package, and Java’s desire to treat an program as a collection of many jars.  To address this, folks started tampering with jar files to allow programs to be distributed as a single jar, but somehow include all the code come from a collection of jars. 

Initial attempts at this were somewhat mechanical, they’d unpack all the stuff from all the jars and put it all together in one jar.  That had issues because individual jar’s didn’t consider this, so they didn’t worry about uniquely naming resources, and when you dumped all the contents of all the jars into one jar resources started to collide and get overwritten. 

When the “pour it all out into one jar” approach proved awkward, the next approach was to mess with the ClassLoader to allow for recursion. Java supports custom class loaders so this seemed to be the right way to go. Make a class loader that allowed a class to be found in a jar in another jar. This meant you didn’t tamper with the jars you included, those jars all were just stuffed unaltered into another jar. 

And that is why bootRun isn’t the same as running the jar resulting from bootRepackage.  BootRun is really just the old school class path of all the dependencies included. BootRepackage creates a fat jar, and provides a class loader that can handle the jars in jars. 

… and that was our problem. Spring’s class loader behaved just a bit differently than the normal class loader when responding to getResourceAsStream to support the jars in jars.

Once we pinpointed the problem, we tweaked our code a bit and, without problem found idiomatic Java that worked properly in either scenario.  It was the getting there that was tricky. 

Gradle Kotlin DSL: First Impressions

As an learning exercise, now that Gradle has a Kotlin DSL, I decided to migrate a Gradle build from the standard Groovy DSL over to the Kotlin one. My Kotlin SVG DSL project was the perfect candidate, being a pure Kotlin project already.   This article won’t attempt to cover all the details, but I’ll discuss the basics and my impressions.

The Migration Path

The basics of moving the Gradle build from Groovy to Kotlin are simple. Gradle itself now supports either, so none of the tools themselves change. What changes is your build.gradle script.  If Gradle sees build.gradle is expects Groovy, if it sees build.gradle.kts it expects Kotlin.  So the obvious change is to simply rename your build script and deal with the resultant issues. That said, there are script elements common to both, so before I took the leap and renamed the file, I worked in the Groovy version to insure that the common bits like the plugin management were up to current standards.

So my steps were:

  • Update Gradle to the latest and greatest (4.10.2 at that point).
  • Migrate the Groovy plugin management to the newest method
  • Update all the plugins
  • Review all the strings, Groovy allows several flavors of strings including single quoted, Kotlin mostly wants double quoted strings. The double quoted work in either, so I moved all the single quoted over to double quoted before the migration.
  • Rename build.gradle to build.gradle.kts

And KABOOM!

The Kaboom

Yes there was an earth shattering kaboom.  The renamed file, even with the common bit prepared was not a compliant Kotlin script.  The Kotlin DSL is familiar in spirit, but the code must be kotlin and so there are many syntactic and a few semantic differences around configuring and managing tasks.

All of my plugins still worked, but configuring them, and any ad hoc code I’d used to enhance them had to be reworked to varying degrees.

What I Observed

  • Many changes we sane and mechanical once you knew what to do.
  • Finding out what to do was the challenge:
    • The Kotlin DSL isn’t yet covered in a lot of documentation. Much of the documentation out there pre-dates the Kotlin DSL.
    • The Kotlin DSL is young and about 30% of the documentation out there is based on prior working/transitional versions and not right.
    • Some tried and true Groovy tricks for things like collecting dependency list etc. were hard to reimagine in Kotlin.
    • Kotlin’s type awareness meant that things you’d done in Groovy had just worked without knowing the types, so no one ever bothered documenting them. I ended up digging through the source code to some plugins to find type information needed by Kotlin.

Impressions

I mostly like the resultant Kotlin version better and have switch the master branch over to it.  But I’m not going to switch over all my projects, at least not yet.  In particular, plugins that employed too much magic, like Spring Boot with it’s automagic versioning resolution, or code generation tools, I don’t think the reward is worth the work to figure out the migration.

Kotlin’s ‘use’ A Mixed Bag

I’ve been working with Kotlin as much as possible of late. I really like the language, and it challenges my thinking about how to code solutions.  Many times the challenges revolve around using some Kotlin feature to better implement something. Usually if I employ the feature well I’m really happy with the result.

Employing ‘use’

I’ve done a lot of work on systems that depend heavily resources that require management. Any time you involve networking or persistence you’re going to have objects that, if not managed, will leak resources: sockets, database connections, file descriptors, etc.  If you don’t free them up properly you’ll run out, and properly can get a little involved when you start figuring in all the possible exception conditions.

Java Historically

Prior to Java 7 managing a Closable resource involved a fair bit of boiler plate code to ensure things were done properly.

That had a lot of extraneous code, and you could go wrong in several different ways.

But then try-with-resources  was added and things got much simpler.

Kotlin Introduces ‘use’

Kotlin introduced use which works as follows.

Looks good… but then I added the exception management…

Okay it lost a little luster there.  And then if you have to manage a couple of resources it really didn’t look like a win anymore.  If you compare a Java example to an idiomatic Kotlin one, well, I’d pick the Java.

Java:

Kotlin:

That’s a first for me, something I don’t like better in Kotlin.

My First Open Source Kotlin: KSvg

With Java, I’ve regularly open sourced my personal work.  I’ve a goto tool chain to make open sourcing projects straight forward.  Recently I’ve been doing more and more Kotlin coding and wrote a Kotlin DSL for generating SVG output that I thought I’d share.  This post will discuss both the project, and the tool chain I used to open source it.

KSvg, a Kotlin DSL For Creating SVG Images

Kotlin has great support for creating domain specific languages that integrate seamlessly into further Kotlin code.  My requirement was to be able to efficiently generate simple charts from a Kotlin microservice.  The chart could rendered with about a half page of inline SVG and a trivial Kotlin DSL for generating SVG output let me generate that SVG efficiently from the Kotlin server.

The Result: KSVG

Ksvg allow you to write code generating SVG output with a simple DSL.  An example would look like:

The Kotlin behind the DSL is only about 350 lines and supports various SVG elements and their attributes.  You can see the project here.

Open Source Tool Chain For Kotlin

One of the pleasures of Kotlin is that it’s tool chain is rich and can draw heavily from the Java ecosystem. I ended up using almost all the same tools I’m familiar with, with little or no issue.  To go from source to publically available repository I employed:

  • IntelliJ to code
  • JUnit for testing
  • Gradle to build
  • GitHub for source code control
  • Travis IO for CI/CD
  • Codecov for coverage reports
  • Bintray to populate JCenter
  • GitHub & GitHub Pages for the public site

It was all straight forward and you can see how it all technically came together here.

Where Kotlin is Still Weak

I’ve noted before, Kotlin’s static code analysis is still not really up to grade. The tools I employed where:

  • IntelliJ’s analysis features
  • ktlint
  • detekt
  • JaCoCo

These were all straight forward and worked as advertised but not as complete as the Java ecosystem offers. I’m not confident that I could adequately manage code quality for a team of developers, something you need to do, particularly with various skill level folks, learning a new technology, in a distributed team.

Revenge of Mojave

I posted previously  that I had to buy a new drive to get OS X Mojave to install.  Well I’m a glutton for punishment, or just stupid, because when the first Mojave Beta patch came down I applied it…. and the new drive gave an error 22 and the welcome/setup hung AGAINAt least with this drive, I could reformat and reinstall – that failed on the prior drive.  But the ~4 hours of install and Migration Assistant is getting REALLY old fast.

OS X Mojave Upgrade Hang solved!

Mojave Hangs

I’ve a MacBookPro9,2 at home. It’s older but I like it. Has sufficient horsepower and the memory, disk, battery are all still maintainable. I attempted the OS X Mojave upgrade and it would complete but hang at the gray Apple logo on the final reboot before kicking off the setup/welcome app.   I tried all the reported fixes without success.

The Fix

Abandoning the reported solutions I dug deeper, and did:

  1. Reboot in recovery mode (Command-R power)
  2. Open the Terminal
  3. Look in the target volume’s /var/log

I found there an install.log and read that.  Near the end there was an attempt to remount the drive with a specific hardware option and it was returning an error 22. This mount option must be new to Mojave since prior upgrades have worked.

So, I bought a new SSD, installed on that, and Viola! Apparently the 750GB HGST drive I had didn’t support the newly incorporated feature.

Would have been nice if the install detected that before it blew hours of transferring data etc.