20180225 Update "Lots Learned"


I have learned quite a bit about Dependency Managers , and .
Three very big, complex, popular, and well designed tools used for building software.

Dependency Management - is a system to get the right libraries from the right places so everything is worky.  Sounds simple enough right ?  It isn't.  Dependency Management is used at two different times. It is used during the building of myrobotlab.jar (compile-time) and its also used when myrobotlab is running (runtime)  ....managing libraries and other files at these two times is complex, fragile, and has been an ongoing challenge since nearly the beginning of software.  And if you do not do it right, you're sure to burn in Dependency Hell   Therefore, its probably worth looking into what works best for MyRobotLab software.

  Pros Cons
Ant and Ivy work well together.  Ivy is only a dependeny manager, while Ant is the build framework.  Ivy is very well designed and highly modularized.  It can be used "with" other systems or by its self as a standalone.  Up to 2011 Gradle used Ivy as dependency manager too. Ant is the "oldest" build system and is beginning to lose popularity in new projects.  There was a big bug in it regarding maven repos and artifacts with classifiers - but I think I fixed that.

Nixie build uses maven currently.  Its very popular with many Java projects.

Maven's mantra is "convention" over "configuration".  Which I heartily agree with, however, maven is not easy to extend. And it also can be difficult for those who do not know its conventions.

Maven does not "modularize" its dependency management. It also expects environment variables to be set and has some other requirements. These requirements make it challeging to bundle with MyRobotLab software.

Maven has an embedded project, but it appears to be poorly documented and not working at all in the most recent releases

Gradle is the new kid on the block.  The build files are much smaller than Maven, and it takes "some" of the paradigms used in Maven while at the same time can extend those conventions easily.  
Gradle is the build system of choice for Android projects.
It also prides itself on being "faster" than Maven.
Gradle as an application does not "modularize" its dependency management, so we can not utilize this in myrobotlab.jar easily.

I'm now in the process of making MyRobotLab be capable of generating its own build files for all 3 of these build systems. After evaluating the results we can go forward with the most solid foundation, which we all can build on !

Results of generated full build pulls of dependencies:

  files size  
ivy 2.4.0 un-patched 548 547 Mb similar to gradle missing all the platform native jars
ivy 2.4.0 patched 517 714 Mb very similar to gradle (not exact...)
ivy develop / unstable 415 425 Mb zoiks !
gradle 517 714 Mb  
maven 520 732 Mb ver close to gradle - however on occasion gets older versions

 

20180108 Initial Report


Ahoy !

On Christmas we released MyRobotLab Manticore - and now we have been busy working on the very big changes for the next release "Nixie"

(for those of you with GitHub Accounts -> https://github.com/orgs/MyRobotLab/projects/2 )

Some of the biggest changes are how we build MyRobotLab.  This includes the very important job of managing dependencies.  Managing dependencies is how MyRobotLab gets all that great functionality from the huge resource of open source libraries around the world.  

Previously we managed our very small customized repo (called "repo") in github.  Now, we have taught MyRobotLab to use "real" Maven repos.  Maven repos house most of the dependencies on Earth, so our conversion potentially gives us easy access to a ginormous resource !  Yay !

There are 2 places where we now use the new "maven" dependency management.

  • The "Building" of MyRobotLab
  • The Running and Installing of Services in MyRobotLab

At this time we basically never have to worry about ever "checking" in jars or artifact into the github repo.  In fact this will probably get deleted eventually.

In the near future - the "only" place you will need to update is the Service's getMetaData function. 
For example :

meta.addDependency("org.bytedeco", "javacv-platform", "1.3.3");

That is the single line need to pull in JavaCV, JavaCPP and all of the correct native libraries for OpenCV.
Nice Right ?

Switching to a new version is as easy as just incrementing that number.

If you remember there are 2 places which reference this info, the build & and a myrobotlab "install". 
The build gets this information from the pom.xml. When you "install" a service in a running instance of myrobotlab, mrl gets the information from a serviceData.json file, which is packaged with myrobotlab.jar when it was built.  We don't want to have to update and maintain 2 different files to keep them in sync.  Instead we will make a function that updates one from the other.

Now that myrobotlab appears to be downloading, installing and unzipping correctly, I intend to make a function in the Repo class generate the pom.xml dependencies.  This will make it easier for us to keep getMetaData & pom.xml in sync.  Less Maintenance !  Yay !  Potentially, this function could be smart, and figure out a problem which we occasionally run into.  Which is when one service requires one version of some library, another service might depend on the same library but a different version.

This is a Transitive Depenendy Conflict Problem :(

If your lucky it means just using the "latest" version of the library in question.

I'll show you what I mean - because its happening now:

Above is the result of some new functionality of the Repo class.  It now has the ability to download dependencies from maven repos into a different directory for each Service.

We can easily see Deeplearning4j uses commons-codec-1.10.jar but AcapelaSpeech, Esp8226_01, GoPro, HttpClient, IndianTts, NaturalReaderSpeech, Polly Solr, Test, VoiceRss & WolframAlpha depend on commons-codec-1.9.jar

And GoogleCloud & WikiDataFetcher depend on commons-codec-1.6.jar

Vision depends on the oldest commons-codec-1.2.jar

If the service includes this dependency directly, then we would upgrade it and test.
 
If the service included this dependency, because it had a dependency which included another dependency, which at some point included commons-codec-1.6.jar - it might be desirable to exclude this dependency.
 
Its much less desirable to dump all of these different library versions into the same directory, because the results will be difficult to determine and probably "not good"
 
So back to Polishing !
 


Update 20180110 Dynamic Dependency Retrieval


I've been playing with the MyRobotLab Repo class functions - and wanted to show how (at the moment) dependencies are being delivered.

The Repo class uses and embedded Apache Ivy in order to "resolve" & "retrieve" dependencies

  • resolve - is finding out you need a library, and finding out where it exists and downloading it to the cache 
  • retrieve - is copying those libraries from the cache to where you want them

 

When the retrieve process copies the libraries from the cache to the myrobotlab/libraries location it does so using the following "retrieve pattern"

Manticore

libraries/[type]/[artifact]-[revision].[ext]

Nixie

libraries/[type]/[originalname].[ext]

and below you can see what the results are

Manticore Install  Nixie Install

 

What this means is means is the public repos have a lot more [type]s .  This concerns me a little bit in that I see a "so" directory.  "so" is a Linux dll or shared object native code and its platform dependent.  In our own private repo we controlled what "native" code went down to a system based on configuration passed up by the running instance.  I think the "easiest" is to dump everything in the "jar" directory and not to use [type] variable in maven to sort the material. It seems like a messy solution, but its probably the best chance of worky I know of ...

 

Update 20180119 Dependency Management


Dependency Management is something that can make or break a project.  Its managing all those libraries and cool features which the Services depend on.  In the case of OpenCV or Deeplearning4j it can be a very large set of libraries from the internet.  Sometimes the libraries from 2 services will "overlap". For example
 

OpenCV version 1.3.3 depends on a library commons-codec-1.2.jar  however
Deeplearning4j version 0.9.1 depends on  commons-codec-1.10.jar

In the past we manually resolved conflicts like this. Manually is not the way !  This needs to be automated. 
We currently are using 2 programs to do dependency management.

Maven - which is a very popular, very capable "build" tool. One of its necessary sub-functions is dependency management. We use Maven to build & test myrobotlab.jar.
Ivy - ivy is explicitly only for dependency management. This application is used inside myrobotlab.jar when a Service is installed.

At the moment when Maven builds myrobotlab.jar it knows about all of the desired dependencies.  Ivy knows only a single dependency.  This must change so that Ivy can process conflicts exactly the same as Maven.

"Current" installation of a Service is done by invoking Ivy's retrieve function - 
Each dependency listed in Service.getMetaData is processed seperately.
A temporary Ivy file is created from Mrl's Repo class.
Below is an example of javacv-platform 1.3.3

OpenCV.getMetaData Temp Ivy file
meta.addDependency("org.bytedeco", "javacv-platform", "1.3.3");
meta.exclude("commons-codec", "commons-codec");
 
<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0">
<info organisation="org.bytedeco"
module="javacv-platform-caller"
revision="working"
status="release"
publication="20180120124820"
default="true"
/>
<configurations>
<conf name="default" visibility="public"/>
</configurations>
 
<publications>
 <artifact name="javacv-platform-caller" type="jar" ext="jar" conf="default"/>
</publications>
<dependencies>
  <dependency org="org.bytedeco" name="javacv-platform" rev="1.4" conf="">
  <exclude org="commons-codec" module="commons-codec" name="" type="jar" ext="jar" conf="" matcher="exactOrRegexp"/>
</dependency>
</dependencies>
</ivy-module>
 

It took some work to figure out how to interface with Ivy programmatically.  Ivy is a great application written very well, but it was no "made" for programmatical interfaces.  Primarily, its used as a standalone build tool.
Here is the code which mrl currently interfaces with Ivy. 
Its a bit challenging to understand and work with.

Therem is one very nice advantage of working with Ivy "in-process", and that is the ability to collect errors.

My goal is to simply generate the Ivy xml file without using Ivy components, then call Ivy's Main.main (standalone entrypoint) to process the file and resolve & retrieve all dependencies.  It's make developing and construction a dynmic Ivy file trivial, while still being able to process errors (in-process).  This would be a win-win.

 

Update 20180121 Dependency Management


I'm finally starting to understand Ivy's configurations ..

The best resource links are :

So my idea is to begin testing on ivy.xml & ivysettings.xml files and ivy command

java -jar ivy-2.4.0.jar -retrieve "libraries/[type]/[originalname].[ext]" -confs "runtime"

with the goal of matching the same output as the maven command

mvn -DoutputDirectory=libraries/jars dependency:copy-dependencies 

I found out you can convert a pom.xml to an ivy.xml.
Strangely its not a command line parameter, but it seems to require ant & a build.xml file ;P

I converted our current pom.xml to an ivy.xml 

 

Ivy reported the following when processing the converted pom.xml
java -jar ivy-2.4.0.jar -retrieve  "ivyToPom/[type]/[originalname].[ext]"

---------------------------------------------------------------------
|                  |            modules            ||   artifacts   |
|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
|      default     |   17  |   0   |   0   |   3   ||   14  |   0   |
|      master      |   0   |   0   |   0   |   0   ||   0   |   0   |
|      compile     |   17  |   0   |   0   |   3   ||   14  |   0   |
|     provided     |  398  |   7   |   2   |   63  ||  336  |   2   |
|      runtime     |   17  |   0   |   0   |   3   ||   14  |   0   |
|       test       |   17  |   0   |   0   |   3   ||   14  |   0   |
|      system      |   0   |   0   |   0   |   0   ||   0   |   0   |
|      sources     |   0   |   0   |   0   |   0   ||   0   |   0   |
|      javadoc     |   0   |   0   |   0   |   0   ||   0   |   0   |
|     optional     |   0   |   0   |   0   |   0   ||   0   |   0   |
---------------------------------------------------------------------

318 files 238 Mb in the "jar" directory

maven (without provided) ivy (convert pom.xml to ivy.xml with ivysettings.xml)
mvn dependency:copy-dependencies java -jar ivy-2.4.0.jar -retrieve  "ivyToPom/libraries/[originalname].[ext]"
"provides removed" - 448 files - 702,394,923 bytes
---------------------------------------------------------------------
|                  |            modules            ||   artifacts   |
|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
|      default     |  410  |   5   |   0   |   68  ||  343  |   0   |
|      master      |   0   |   0   |   0   |   0   ||   0   |   0   |
|      compile     |  390  |   5   |   0   |   66  ||  325  |   0   |
|     provided     |   0   |   0   |   0   |   0   ||   0   |   0   |
|      runtime     |  410  |   5   |   0   |   68  ||  343  |   0   |
|       test       |  410  |   5   |   0   |   68  ||  343  |   0   |
|      system      |   0   |   0   |   0   |   0   ||   0   |   0   |
|      sources     |   0   |   0   |   0   |   0   ||   0   |   0   |
|      javadoc     |   0   |   0   |   0   |   0   ||   0   |   0   |
|     optional     |   0   |   0   |   0   |   0   ||   0   |   0   |
---------------------------------------------------------------------
 
343 files on disk - 398,190,773 bytes

Ivy has a definate "sub-set" of Maven's dependencies - many appear to be different platform related info.
Which makes me wonder if Maven during compile scope pulls "all" of this stuff into the classpath ?
If you look at ffmpeg-3.2.1-1.3-windows-x86_64.jar on the Ivy side, you can also see an "older" ffmpeg 2.8.1 on the Maven side ! .. FTW ?

There seem to be several cases of Ivy getting more recent releases of libraries, perhaps that's just because I did it a couple minutes later ?  Or does it have something to do with the details of the algorithm which handles transitive conflicts ?

 

Update 20180123 Classifiers and Ivy Challenges


I've learned a lot in the last week regarding ivy & maven.  I've leared that maven's copy dependency plugin copies "all scopes" unless configured to do otherwise.  

The issue I'm trying to resolve now is the one below.  On the right is Maven's copy "all scope" dependecies.  On the left is Ivy's "default" configuration.  Maven uses scopes & classifiers, Ivy uses configurations.  The "mapping" between the two is a little challenging to follow.  

I believed the desired state is to have Ivy download "roughly" the same thing as Maven.  We are about 80% there.  Most of the issues are around native jars in projects.  They are typically labeled with maven "classifiers" - and these "classifiers" can be free-form text which can really be any value (not much standardization between different operating system types)

The results are the maven side downloads all referenced jars - all ones which have classifiers too.
The Ivy appears to take the "first" one defined.  

I thought I came close by adding a "extra" definition to the ivy file - then adding its own form of classification.
It worked, in that I could explicitly add artifacts to download, however in the "retrieve" the "first" artifact referenced would be retrieved with an un-parsed retrieve value .. 
So when Ivy previously downloaded jinput-2.0.7-native-platform-osx.jar and leave linux & windows, after adding the classification entries .. the linux & windows jar would download, but the osx one would have filename [originalname].jar :P

 

Update 20180131 Change of Direction !


So the Ivy path was a lot more sticky and dark than I was expecting.  Ivy I still believe is a great tool and the coding design I know is really impressive. I think Ivy was designed exceptionally well to do an "extremely" difficult job of managing dependencies.

Unfortunately, the way it interfaces with maven central dependency classifiers, it will not guarantee the appropriate jars to come down. Regrettably, I am not able to change its behavior without hacking that design.  So, at this point I have abandoned Ivy as our dependency manager.  Goodbye Ivy, you have served us well.

Maven is what builds MRL, and maven manages the jars which are downloaded to do that.  Soooo, the easiest way to match that during runtime is to have the same tool which manages dependencies during "compile" time do it for "runtime".  Soo, this means packaging an embedded maven.  Maven is a bit more complicated than Ivy.  Ivy is a single jar, Maven contains a Maven Home environment variable, and many jars in a lib with a bootstrap and config :P.   To manage all this complexity I need a "Installer"

In order to bundle and use maven with mrl - it should require an "installer".  If we are going to that effort, we "should" have a simple executable for each operating system.

GAH !!! so much more stuff to learn now ...

 

I need to test the best open source installers and see what best can handle all our requirements

  • appropriate native executable on all systems max-osx, windows 32 & 64, linux 32 & 64
  • capable of installing a jvm
  • installs maven (multiple jars & specific directory path)
  • can launch

 

 

 

Ray.Edgley

6 years 2 months ago

Now you have it worky, I bet your glad it done.

 

Good job well done

 

Hello Grog,

I like all your dated updates explaining the process you have been working through for almost 3 weeks now.

I am a little confused as to why the updates are 12 month old?

Its now 20180122....

Just an observation... ;-)

At lease you have been consistent with the date :-)