Java Application Reloader

I’ve some micro services that I’ve been moving off commercial PaaS platforms to a Raspberry Pi on my home network. It’s been fun and worked out well, but I did lose one thing in the move, support for continuous delivery.  Previously Travis-CI was able to deploy the services to the PaaS, but with the new setup I had to figure out a new solution.

Looking at the service, and seeing as it’s a single jar file, I felt I ought to be able to come out with a simple solution.  I knew I could get the jar files onto the Pi, all I needed was some why to cleanly reload the service from the new jar.

A bit of searching turned up this older article on how to Programmatically Restart a Java Application.  It was close to what I wanted, if I could just enhance it to behave like the update feature you often see in apps where it doesn’t just restart, but also updates the version.

Reloader is Born

Starting from the article mentioned, I developed reloader.  Reloader will restart a java application, but additionally it can:

  • Act as a signal handler, so that you can kick it off by sending a signal to the application, optionally creating a pid file.
  • Find the newest version of the jar containing the application, and switch to that, allowing for pseudo in place upgrades.

In addition to adding features I brushed six years of dust off the code,  and deployed the package to jcenter for easier public use.

Using Reloader

Using reloader couldn’t be much simpler. It does pretty much everything on its own.  All you need to do is call one method explicitly:

    Reloader.restartApplication();

Or set it up as a signal handler:

    Reloader.onSignal("USR2");

That’s all it takes.

My Use Case

To achieve continuous delivery from Travis-CI  I added two very restricted and hardened breaches in my Raspberry Pi’s security:

  1. A jar can be pushed to the server remotely
  2. The service can receive a USR2 signal remotely

I then set up Travis-CI to perform those two actions after any successful build. With reloader added to the service, those two actions were all that was needed.

In Conclusion

By its nature reloader is a bit rough around the edges and while it has some test coverage I’ve really only burned it in for my use case. That said if you want to give it a try, or just look at its bits and pieces head over to github and go wild.

From OpenShift to a Raspberry Pi

For some time I’ve parked a personal snippets service I wrote on OpenShift’s free tier. It seemed like the free ride wasn’t going to last so it was time to move on.  I use my snippets on a regular basis, but I didn’t really want to pay for a 24×7 internet presence… and I had a Raspberry Pi laying about, so I thought, why the heck not?

Could it Work?

My first step was to see if this was even doable…

  • Could a Pi host my service?  Java 8, enough memory, enough storage, enough CPU. Pass √
  • Was my home network up to it?  I’ve a decent connection, and I verified my ISP does allow inbound connections. My router can port map. I don’t have a registered domain name or a fixed IP but I knew how to deal with that.  Pass √
  • Was it a sane choice? The answer to that has never stopped me before so … Pass √

Bringing the Pi Out of Retirement

I had a Pi that I hadn’t used in a year or so just gathering dust.  All I had was the Pi and it’s power supply though.  No SD card. No compatible keyboard.  A quick trip a chain drug store up the road got me an 16GB SD card for 7$ on sale.  Google yielded a slew of tutorials when asked “set up Raspberry Pi without keyboard”.   So…

  1. Format the SD
  2. Load it with Raspbian
  3. Pop it in the Pi
  4. Cable Pi to network
  5. Power Pi up
  6. SSH in…. denied ?!?  FAIL X

The tutorials were mostly out of date with regards to a security update in Raspbian. It no longer has the ssh daemon running by default. The solution is to put the SD into a computer and create an empty file named ssh on the FAT32 /boot partition. Return the SD card to the Pi, reboot, and I could ssh in. Pass √

Before I went further, I loaded my service onto the Pi ran it, and sure enough it ran like a charm.  Pass √

Opening a Port and Dynamic DNS

Now to make the Pi accessible to the outside world.  I set up my router to port map the selected port and targeted my networks external IP address with the port and it went right through to the Pi and the server worked! Pass √

Since the external IP address can change, and using raw IP addresses is clunky I set about setting up a Dynamic DNS service. I chose a free one: ChangeIP.com. These services require some client software running on you Pi to keep your possibly changing IP address in sync with the name you chose.  So back on the Pi I set up ddclient which worked fine with very little effort. Did an nslookup and got the right IP address.  Pass √

A note here, on a partial fail, apparently my jobs http proxy rejects dynamic DNS names out of simplistic paranoia so I have to use the IP address, which ironically I can find with and nslookup which the network does allow (?!).

The Final Test

Okay, using my phone, turned its wifi off, and targeted the appropriate URL.  I used my phone without wifi to insure I was really coming in from outside world – no cheating.   And… denied ?!?  FAIL X

Okay, so I said my ISP allowed inbound connections right?  Well, a bit of further investigation taught me that they allowed them, but not by default, again a security enhancement.  Turns out you have to go to their website and turn them on… and the feature was broken.  I got on chat with a support person, and after, following their required script where I checked for them again all the things I already had done, they made the change for me. And…  Pass !!!

I could get in from my phone and it all worked wonderfully.  Easy peasy :-)

Is it Free When it Costs so Much Time…

Clearly not.  So why do I bother with the free/cheap stuff? Ok, I’ll admit that, even though I know it’s not free if it costs my time, I can’t help liking cheap/free.  In particular I know the free tiers of most services rarely stay around, at least not as free. But, the moving from one to the next forces one to learn new technologies, and teaches you how to write good loosely coupled software. Also, the quality and longevity of a free service can be a good indicator of the general value of a product. If it’s unstable and evaporates in a few months, well it’s not something I’m going to suggest at work or to others.

The Rime of the Ancient Mariner in Greece

I had a somewhat eclectic education, and as a grade school child I was tasked with memorizing The Rime of the Ancient Mariner.  So you too are not forced to wrap your head around this epic poem, and for the sake of my tale, I’ll summarize. The poem tells of an old sailor that’s impelled to continuously accost passers by and tell them his tale of how his misdeeds doomed a ship and crew to destruction, and how from that he learned to act with compassion. Why do I mention the poem?  Because I’ve just lived it, or sort of. Your call.

I’m in the process of ending a six year relationship with someone. I’m not seeking judgement here, I accept my share of failures in the relationship, perhaps even the larger share, but for the sake of this tale it’s worth noting that they had chosen to end things and left.  So when they ended up sick in a foreign land, Greece, and turned to me, some very sane people asked me, well what’s that to you?  The poem didn’t come immediately to mind, but when I decided to help the person get back from Greece, I did find myself compelled over and over to explain the situation, and the image of that ancient mariner telling his crazy tale to one person after the next seemed on point.

This wasn’t going so be a simple case of sending a bit of cash you see. They’d ended up twice in the hospital, were diagnosed, treated and recuperating, but were simply in no shape to travel on their own.  Any option I could find to get others to help, US Embassy, insurance, private medical evacuation, all seemed to end in being stuck alone in Greece for an indeterminate length of time, and at huge cost… so realistically someone just had to go there and bring them home.  And that someone really was, for a number of reasons, me. So I did.  I flew to Greece, organized the return trip, and flew them back.  A three day world tour, NYC, Amsterdam, Athens, Rome, NYC.  And it was touch and go at moments. Getting to the airport, through transfers, customs, and back home, with someone so exhausted that five minutes on their feet left them dizzy and shaking like a leaf, and with a stomach so weak that any peculiar smell had them gagging and even vomiting, was tough. But we got back home.

So, as I said, at first I recalled the poem because working through the logistics of missed work, travel plans, finances, etc., I was accosting one person after the the next, having to explain why I was doing this seemingly crazy thing.  But now, with it done, I think maybe too, the poem came to mind because of its conclusion. I’m not sure I even consciously understood it as a child, but the message, to live compassionately, is pretty clear:

He prayeth well, who loveth well
Both man and bird and beast.

He prayeth best, who loveth best
All things both great and small;
For the dear God who loveth us,
He made and loveth all.

While I didn’t magically repair the relationship, or bring about any great change to my life, the fact that I acted compassionately and resolved at least this one crises in a good way… well that part feels deeply right.

The Lie That Snowballed

I won’t speak for everyone, but I’d guess that most people have, at some point, been warned not to tell even small lies because they just take on a life or their own, and you’ll end up owning them till the messy end.   I know I heard that repeatedly as a child, and I certainly saw the truth of it, but I can’t say I deep down believed it.  I hate confrontations, and certainly as a child, I know I’d blurt out what I considered a harmless or trivial lie, to avoid things getting too emotionally intense.  The following is what drove the lesson home for me, made be a true believer.

I was asked by work to travel to Mexico City to help close a deal for my employer.  From the start the trip had a bad wind about it.  I was convinced I wasn’t the right person to send, as I thought it was a bad deal for my employer in the first place, and I was just a programmer without any marketing skills.  They gave me only a couple days warning, and I had to cancel a long awaited trip to the circus with my son. Regardless, I put on a happy face and tried to make the best of it. By the week’s end in Mexico City, I’d encountered one problem after another, nothing had gone to plan, from travel plans, to business negotiations, and to top things off the deal did not look like it would go through. My positive facade had worn very thin.

As I left for the airport, the hotel, trying their best to make amends for various issues during my stay, insisted on collecting my luggage in my room and providing a car to the airport.  Usually, I’m the sort that manages my own details, but I relented and let them take care of things.

Arriving at the airport with ample time, things were looking up.  I was heading home. I decided to pack my overcoat into my luggage for convenience, opened my bag and…. didn’t recognize the contents!  It was the same model bag, but at some point the hotel or car had switched mine with someone else’s.  Did I mention my Spanish was pretty weak Okay … no need to panic. This was before one could just look things up on their mobiles, but I like to think I’m a pretty resourceful person so I went and bought a local phone card valued at 10 US dollars, which was a goodly sum at that time,  and went to a pay phone with someone using it. When the caller hung up, in broken Spanish, I explained that I would give them the entire phone card if they made just one call for me.  They agreed happily and navigated the operator and directory assistance, and I was speaking to my hotel in less then a minute.  The hotel quickly admitted their error, and agreed to send a car back with the correct bag.  Done.

I awaited the bag switch and it seemed to go off without a hitch and I made it to my gate with almost an hour to spare. No worries…. or so I thought. Apparently Mexico City had been battling travelers that booked a number of outbound flights and then just showed up for whichever one they pleased. In response, a policy was instituted of giving away seats not confirmed in the airport one hour in advance.  Five minutes ago my seat had been given away and the flight was now fully booked. Right, I still wasn’t going to let this get to me. I went to the counter, explained what had happened with the bag, and asked about the next flight out.  I received a very disinterested, “None till tomorrow.”

That’s when I snapped internally. I’d reached my limit. I just could not stay even one more night there. So, thinking all I needed to do was get them actually interested in helping me, I told the lie. I said to the counter official, “Please, are there any other flights? I’ll take anything! I’ve just found out my pregnant wife is in labor and I must get home!”  No, my wife wasn’t in labor, nor even pregnant, I just thought if I could make a human connection the official might actually try and be helpful.

It worked. Their eyes lit up, their fingers flew on the console, and in a regretful voice they said, “Sir, I could route you through Mexico City to LA, LA to Seattle, Seattle to Minneapolis, Minneapolis to NYC. It would get you in only four hours before our first morning flight, and you’ll fly all night.”  Okay I thought, rather then spend another night there, “Thank you so much! I’ll take it.”

And then it began. The lie now had it’s own life and began to thrive.  In moments I was escorted across the airport on a cart with a flashing light. The driver clearing the crowds shouting, what with my broken Spanish, seemed to be “Man with Baby!”.

I arrived at the plane and they were holding it for me!  Everyone was seated with expectant looks as I took the walk of shame to my seat. But they weren’t upset. Some people even seemed to give me well wishes about the baby as I took my seat. I felt horrible.

The woman in the seat beside me immediately said, “You must be the man who’s wife is in labor. Congratulations! I’m the US Ambassador’s wife, I’ll see what I can do!”  Oh no…. well I spent the leg to LA talking with her, never owning my lie of course, but we mostly spoke about general American expat type topics so … it was all good right?

I think, in fact, that after she got off the plane, she made a call of some sort.  What I can say with certainty is that every one of my transfers I was ridden through the airports on a cart and the plane held if needed.

When I arrived in NYC I tried to slip off the plane like any other passenger, but I saw airline officials point me out to uniformed men, who immediately came to me and explained they were there to give me escort to the hospital.  At this point I simply did not have the courage to set things straight.  I thanked them profusely, and threw another lie on the fire “My cousin is waiting just outside, I’ve had an update and the baby hasn’t come yet, so let me just find him.”  Miraculously, and undeservedly, it worked. I managed to slink off to a taxi and head home without further event.

I’ll never forget that. How that moment of dishonesty blossomed into a mess of lies and repercussions.  I told my son that story later in life when I gave him the “Don’t tell a lie because it will snowball” talk, and I like to think hearing such an idiot’s tale from his father maybe gave the message some impact.  It certainly impacted me, and I will say publicly (the legal statute of limitation has expired right?), that if any of those kind people I deceived that night, particularly the Ambassador’s wife happen across this story, I am truly sorry for the trouble I caused …

Life Allegories…

If you’ve ever had to share knowledge with another, perhaps you’ll agree, that simply stating what you believe is the conclusion is rarely effective.  With science you often need to engage the listener in the steps to arrive at the conclusion, and with matters of life you try to engage the listener in the journey you took to find your conclusion. The proverb, “give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime” seems to vaguely apply.  Simply providing a conclusion might feed them in the moment, but engaging them in the path to the conclusion may actually provide them with something useful forever.

I’ve always believed the above, and since I was a child loved a well told story that’s a life allegory.  It’s a skill I’ve tried to develop for myself, the telling of a good story. I won’t say I’ve mastered it, but I will say that some of my personal experiences, as I relate them, listeners have seemed to enjoy.

So, if you come to my blog for technical info, don’t leave in disgust. I’ve not found a religion or some such, I’m just posting a few non technical pieces to cleanse my palate. You can easily ignore the pieces tagged “Non Tech”.  But if you wish, you can indulge me and read them and maybe enjoy the diversions as well.