Progress Report - Nixie MyRobotLab

Ahoy !

On Christmas we released MyRobotLab Manticore - and now we have been busy working on the very big changes for the next release "Nixie"

(for those of you with GitHub Accounts -> https://github.com/orgs/MyRobotLab/projects/2 )

Some of the biggest changes are how we build MyRobotLab.  This includes the very important managing dependencies.  Managing dependencies is how MyRobotLab gets all that great functionality from the huge resource of open source libraries around the world.  

Previously we managed our very small customized repo (called "repo") in github.  Now, we have taught MyRobotLab to use "real" Maven repos.  Maven repos house most of the dependencies on Earth, so our conversion potentially gives us easy access to a ginormous resource !  Yay !

There are 2 places where we now use the new "maven" dependency management.

  • The "Building" of MyRobotLab
  • The Running and Installing of Services in MyRobotLab

At this time we basically never have to worry about ever "checking" in jars or artifact into the github repo.  In fact this will probably get deleted eventually.

In the near future - the "only" place you will need to update is the Service's getMetaData function. 
For example :

meta.addDependency("org.bytedeco", "javacv-platform", "1.3.3");

That is the single line need to pull in JavaCV, JavaCPP and all of the correct native libraries for OpenCV.
Nice Right ?

Switching to a new version is as easy as just incrementing that number.

If you remember there are 2 places which reference this info, the build & and a myrobotlab "install". 
The build gets this information from the pom.xml. When you "install" a service in a running instance of myrobotlab, mrl gets the information from a serviceData.json file, which is packaged with myrobotlab.jar when it was built.  We don't want to have to update and maintain 2 different files to keep them in sync.  Instead we will make a function that updates one from the other.

Now that myrobotlab appears to be downloading, installing and unzipping correctly, I intend to make a function in the Repo class generate the pom.xml dependencies.  This will make it easier for us to keep getMetaData & pom.xml in sync.  Less Maintenance !  Yay !  Potentially, this function could be smart, and figure out a problem which we occasionally run into.  Which is when one service requires one version of some library, another service might depend on the same library but a different version.

This is a Transitive Depenendy Conflict Problem :(

If your lucky it means just using the "latest" version of the library in question.

I'll show you what I mean - because its happening now:

Above is the result of some new functionality of the Repo class.  It now has the ability to download dependencies from maven repos into a different directory for each Service.

We can easily see Deeplearning4j uses commons-codec-1.10.jar but AcapelaSpeech, Esp8226_01, GoPro, HttpClient, IndianTts, NaturalReaderSpeech, Polly Solr, Test, VoiceRss & WolframAlpha depend on commons-codec-1.9.jar

And GoogleCloud & WikiDataFetcher depend on commons-codec-1.6.jar

Vision depends on the oldest commons-codec-1.2.jar

If the service includes this dependency directly, then we would upgrade it and test.
 
If the service included this dependency, because it had a dependency which included another dependency, which at some point included commons-codec-1.6.jar - it might be desirable to exclude this dependency.
 
Its much less desirable to dump all of these different library versions into the same directory, because the results will be difficult to determine and probably "not good"
 
So back to Polishing !
 
Update 20170110

I've been playing with the MyRobotLab Repo class functions - and wanted to show how (at the moment) dependencies are being delivered.

The Repo class uses and embedded Apache Ivy in order to "resolve" & "retrieve" dependencies

  • resolve - is finding out you need a library, and finding out where it exists and downloading it to the cache 
  • retrieve - is copying those libraries from the cache to where you want them

 

When the retrieve process copies the libraries from the cache to the myrobotlab/libraries location it does so using the following "retrieve pattern"

Manticore

libraries/[type]/[artifact]-[revision].[ext]

Nixie

libraries/[type]/[originalname].[ext]

and below you can see what the results are

Manticore Install  Nixie Install

 

What this means is means is the public repos have a lot more [type]s .  This concerns me a little bit in that I see a "so" directory.  "so" is a Linux dll or shared object native code and its platform dependent.  In our own private repo we controlled what "native" code went down to a system based on configuration passed up by the running instance.  I think the "easiest" is to dump everything in the "jar" directory and not to use [type] variable in maven to sort the material. It seems like a messy solution, but its probably the best chance of worky I know of ...

 


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Ray.Edgley's picture

It was worth it

Now you have it worky, I bet your glad it done.

 

Good job well done