20180225 Update "Lots Learned"

I have learned quite a bit about Dependency Managers , and .
Three very big, complex, popular, and well designed tools used for building software.

Dependency Management - is a system to get the right libraries from the right places so everything is worky.  Sounds simple enough right ?  It isn't.  Dependency Management is used at two different times. It is used during the building of myrobotlab.jar (compile-time) and its also used when myrobotlab is running (runtime)  ....managing libraries and other files at these two times is complex, fragile, and has been an ongoing challenge since nearly the beginning of software.  And if you do not do it right, you're sure to burn in Dependency Hell   Therefore, its probably worth looking into what works best for MyRobotLab software.

	Pros	Cons
	Ant and Ivy work well together.  Ivy is only a dependeny manager, while Ant is the build framework.  Ivy is very well designed and highly modularized.  It can be used "with" other systems or by its self as a standalone.  Up to 2011 Gradle used Ivy as dependency manager too.	Ant is the "oldest" build system and is beginning to lose popularity in new projects.  There was a big bug in it regarding maven repos and artifacts with classifiers - but I think I fixed that.
	Nixie build uses maven currently.  Its very popular with many Java projects. Maven's mantra is "convention" over "configuration".  Which I heartily agree with, however, maven is not easy to extend. And it also can be difficult for those who do not know its conventions.	Maven does not "modularize" its dependency management. It also expects environment variables to be set and has some other requirements. These requirements make it challeging to bundle with MyRobotLab software. Maven has an embedded project, but it appears to be poorly documented and not working at all in the most recent releases
	Gradle is the new kid on the block.  The build files are much smaller than Maven, and it takes "some" of the paradigms used in Maven while at the same time can extend those conventions easily.   Gradle is the build system of choice for Android projects. It also prides itself on being "faster" than Maven.	Gradle as an application does not "modularize" its dependency management, so we can not utilize this in myrobotlab.jar easily.

I'm now in the process of making MyRobotLab be capable of generating its own build files for all 3 of these build systems. After evaluating the results we can go forward with the most solid foundation, which we all can build on !

Results of generated full build pulls of dependencies:

20180108 Initial Report

Ahoy !

On Christmas we released MyRobotLab Manticore - and now we have been busy working on the very big changes for the next release "Nixie"

(for those of you with GitHub Accounts -> https://github.com/orgs/MyRobotLab/projects/2 )

Some of the biggest changes are how we build MyRobotLab.  This includes the very important job of managing dependencies.  Managing dependencies is how MyRobotLab gets all that great functionality from the huge resource of open source libraries around the world.  

Previously we managed our very small customized repo (called "repo") in github.  Now, we have taught MyRobotLab to use "real" Maven repos.  Maven repos house most of the dependencies on Earth, so our conversion potentially gives us easy access to a ginormous resource !  Yay !

There are 2 places where we now use the new "maven" dependency management.

• The "Building" of MyRobotLab
• The Running and Installing of Services in MyRobotLab

At this time we basically never have to worry about ever "checking" in jars or artifact into the github repo.  In fact this will probably get deleted eventually.

In the near future - the "only" place you will need to update is the Service's getMetaData function. 
For example :

meta.addDependency("org.bytedeco", "javacv-platform", "1.3.3");

That is the single line need to pull in JavaCV, JavaCPP and all of the correct native libraries for OpenCV.
Nice Right ?

Switching to a new version is as easy as just incrementing that number.

If you remember there are 2 places which reference this info, the build & and a myrobotlab "install". 
The build gets this information from the pom.xml. When you "install" a service in a running instance of myrobotlab, mrl gets the information from a serviceData.json file, which is packaged with myrobotlab.jar when it was built.  We don't want to have to update and maintain 2 different files to keep them in sync.  Instead we will make a function that updates one from the other.

Now that myrobotlab appears to be downloading, installing and unzipping correctly, I intend to make a function in the Repo class generate the pom.xml dependencies.  This will make it easier for us to keep getMetaData & pom.xml in sync.  Less Maintenance !  Yay !  Potentially, this function could be smart, and figure out a problem which we occasionally run into.  Which is when one service requires one version of some library, another service might depend on the same library but a different version.

This is a Transitive Depenendy Conflict Problem :(

If your lucky it means just using the "latest" version of the library in question.

I'll show you what I mean - because its happening now:

Above is the result of some new functionality of the Repo class.  It now has the ability to download dependencies from maven repos into a different directory for each Service.

We can easily see Deeplearning4j uses commons-codec-1.10.jar but AcapelaSpeech, Esp8226_01, GoPro, HttpClient, IndianTts, NaturalReaderSpeech, Polly Solr, Test, VoiceRss & WolframAlpha depend on commons-codec-1.9.jar

And GoogleCloud & WikiDataFetcher depend on commons-codec-1.6.jar

Vision depends on the oldest commons-codec-1.2.jar

Update 20180110 Dynamic Dependency Retrieval

---

I've been playing with the MyRobotLab Repo class functions - and wanted to show how (at the moment) dependencies are being delivered.

The Repo class uses and embedded Apache Ivy in order to "resolve" & "retrieve" dependencies

• resolve - is finding out you need a library, and finding out where it exists and downloading it to the cache 
• retrieve - is copying those libraries from the cache to where you want them

 

When the retrieve process copies the libraries from the cache to the myrobotlab/libraries location it does so using the following "retrieve pattern"

Manticore

libraries/[type]/[artifact]-[revision].[ext]

Nixie

libraries/[type]/[originalname].[ext]

and below you can see what the results are

Manticore Install	Nixie Install

 

What this means is means is the public repos have a lot more [type]s .  This concerns me a little bit in that I see a "so" directory.  "so" is a Linux dll or shared object native code and its platform dependent.  In our own private repo we controlled what "native" code went down to a system based on configuration passed up by the running instance.  I think the "easiest" is to dump everything in the "jar" directory and not to use [type] variable in maven to sort the material. It seems like a messy solution, but its probably the best chance of worky I know of ...

Update 20180119 Dependency Management

Dependency Management is something that can make or break a project.  Its managing all those libraries and cool features which the Services depend on.  In the case of OpenCV or Deeplearning4j it can be a very large set of libraries from the internet.  Sometimes the libraries from 2 services will "overlap". For example
 

OpenCV version 1.3.3 depends on a library commons-codec-1.2.jar  however
Deeplearning4j version 0.9.1 depends on  commons-codec-1.10.jar

In the past we manually resolved conflicts like this. Manually is not the way !  This needs to be automated. 
We currently are using 2 programs to do dependency management.

Maven - which is a very popular, very capable "build" tool. One of its necessary sub-functions is dependency management. We use Maven to build & test myrobotlab.jar.
Ivy - ivy is explicitly only for dependency management. This application is used inside myrobotlab.jar when a Service is installed.

At the moment when Maven builds myrobotlab.jar it knows about all of the desired dependencies.  Ivy knows only a single dependency.  This must change so that Ivy can process conflicts exactly the same as Maven.

"Current" installation of a Service is done by invoking Ivy's retrieve function - 
Each dependency listed in Service.getMetaData is processed seperately.
A temporary Ivy file is created from Mrl's Repo class.
Below is an example of javacv-platform 1.3.3

It took some work to figure out how to interface with Ivy programmatically.  Ivy is a great application written very well, but it was no "made" for programmatical interfaces.  Primarily, its used as a standalone build tool.
Here is the code which mrl currently interfaces with Ivy. 
Its a bit challenging to understand and work with.

Therem is one very nice advantage of working with Ivy "in-process", and that is the ability to collect errors.

My goal is to simply generate the Ivy xml file without using Ivy components, then call Ivy's Main.main (standalone entrypoint) to process the file and resolve & retrieve all dependencies.  It's make developing and construction a dynmic Ivy file trivial, while still being able to process errors (in-process).  This would be a win-win.

Update 20180121 Dependency Management

318 files 238 Mb in the "jar" directory

Ivy has a definate "sub-set" of Maven's dependencies - many appear to be different platform related info.
Which makes me wonder if Maven during compile scope pulls "all" of this stuff into the classpath ?
If you look at ffmpeg-3.2.1-1.3-windows-x86_64.jar on the Ivy side, you can also see an "older" ffmpeg 2.8.1 on the Maven side ! .. FTW ?

There seem to be several cases of Ivy getting more recent releases of libraries, perhaps that's just because I did it a couple minutes later ?  Or does it have something to do with the details of the algorithm which handles transitive conflicts ?

Update 20180123 Classifiers and Ivy Challenges

I've learned a lot in the last week regarding ivy & maven.  I've leared that maven's copy dependency plugin copies "all scopes" unless configured to do otherwise.  

The issue I'm trying to resolve now is the one below.  On the right is Maven's copy "all scope" dependecies.  On the left is Ivy's "default" configuration.  Maven uses scopes & classifiers, Ivy uses configurations.  The "mapping" between the two is a little challenging to follow.  

I believed the desired state is to have Ivy download "roughly" the same thing as Maven.  We are about 80% there.  Most of the issues are around native jars in projects.  They are typically labeled with maven "classifiers" - and these "classifiers" can be free-form text which can really be any value (not much standardization between different operating system types)

The results are the maven side downloads all referenced jars - all ones which have classifiers too.
The Ivy appears to take the "first" one defined.  

I thought I came close by adding a "extra" definition to the ivy file - then adding its own form of classification.
It worked, in that I could explicitly add artifacts to download, however in the "retrieve" the "first" artifact referenced would be retrieved with an un-parsed retrieve value .. 
So when Ivy previously downloaded jinput-2.0.7-native-platform-osx.jar and leave linux & windows, after adding the classification entries .. the linux & windows jar would download, but the osx one would have filename [originalname].jar :P

Update 20180131 Change of Direction !

So the Ivy path was a lot more sticky and dark than I was expecting.  Ivy I still believe is a great tool and the coding design I know is really impressive. I think Ivy was designed exceptionally well to do an "extremely" difficult job of managing dependencies.

Unfortunately, the way it interfaces with maven central dependency classifiers, it will not guarantee the appropriate jars to come down. Regrettably, I am not able to change its behavior without hacking that design.  So, at this point I have abandoned Ivy as our dependency manager.  Goodbye Ivy, you have served us well.

Maven is what builds MRL, and maven manages the jars which are downloaded to do that.  Soooo, the easiest way to match that during runtime is to have the same tool which manages dependencies during "compile" time do it for "runtime".  Soo, this means packaging an embedded maven.  Maven is a bit more complicated than Ivy.  Ivy is a single jar, Maven contains a Maven Home environment variable, and many jars in a lib with a bootstrap and config :P.   To manage all this complexity I need a "Installer"

• http://maven.apache.org/ref/3.1.0/maven-embedder/cli.html
• https://dzone.com/articles/embedding-maven
• https://techblog.bozho.net/embedding-maven/
• https://stackoverflow.com/questions/8243930/how-can-i-embed-maven-2-int…

In order to bundle and use maven with mrl - it should require an "installer".  If we are going to that effort, we "should" have a simple executable for each operating system.

GAH !!! so much more stuff to learn now ...

• Good Overview - https://www.excelsior-usa.com/articles/java-to-exe.html#launch
• https://docs.oracle.com/javase/tutorial/deployment/selfContainedApps/in… (native now to jvm - is it any good in Java 8???)
  https://docs.oracle.com/javase/8/docs/technotes/guides/deploy/packager…
• Open Source Popular - https://sourceforge.net/projects/launch4j/ (but make note - it "failed" first time for me :P
• http://izpack.org/ another popular one

I need to test the best open source installers and see what best can handle all our requirements

• appropriate native executable on all systems max-osx, windows 32 & 64, linux 32 & 64
• capable of installing a jvm
• installs maven (multiple jars & specific directory path)
• can launch