Ahoy !
This is the current dependency references in ProgramAB service
 
    meta.addDependency("program-ab", "program-ab-data", "1.1", "zip");
    meta.addDependency("program-ab", "program-ab-kw", "0.0.8.4");
 
@kwatters you said the initial service is supposed to support only english and Japanese
 
This is the file structure of "program-ab", "program-ab-data", "1.1", "zip"
Problems :
  • it unzips into the mrl directory and not under {mrl}/resource
  • if its suppose to only be "alice" - it looks like alot more stuff - so what should a ProgramAB service install ?
 
This is the file structure of InMoov2
Problems :
  • its under InMoov2 vs being under resource/ProgramAB (which will do a merge on install)
  • the bots are under a subdirectory chatbots, while if its merged with /resource/ProgramAB the bots should be under /resource/ProgramAB/bots
  • There is confusion between languages vs languages+locale ... these probably should have been just languages as the en-US part isn't distinguishable in the data ..  but if we are going to continue with {language}-{Locale} then {language} should be removed (I think these directories might just be created for saving config)
  • These InMoov bots collide with ProgramAB , yet there content is not the same :(  .. There is considerable differences between the aiml here which was forked from InMoov and the aiml in the ProgramAB directories ... if they are supposed to be different then their names should not be the same

My Plan :

Unless advised differently I'm going to:

  • make the InMoov2 package install/merge into ProgramAB all its bots under the /resource/ProgramAB/bots directory.  (removing the chatbot subdirectory)
  • find the offending code that writes config to the {lang} directory
  • delete ProgramAB bots except english if thats alice, then rename it to alice and the jp-JP one under advisement from kwatters (but this needs to be renamed too - no collisions with InMoov2 bots)
  • do a pr so ProgramAB installs into the correct /resource/ProgramAB directory
  • make a ProgramAB JenkinsFile pipeline job for ProgramAB and ProgramAB data - that will get dumped in the repo .. as this is another modular service.

After this is done, InMoov2 specific stuff will be in /resource/InMoov2, its ProgramAB, JMonkeyEngine, and WebGui assets will all be bundled and auto-magically merge on install (with no collisions)

kwatters

4 years 9 months ago

ok.. so..  you're missing a few dependencies.

    meta.addDependency("org.apache.lucene", "lucene-analyzers-common", "8.4.1");
    meta.addDependency("org.apache.lucene", "lucene-analyzers-kuromoji", "8.4.1");
 
Those are for japanese support.  (the japanese support that existed in ProgramAB before was horrible.. this is a decent shot at that.)
 
Next,  you're correct,  the dependencies for ProgramAB (the default chat bots ) should NOT have anything to do with inmoov.  I tried expliaining this in the past to folks, but it wasn't quite understood.  ProgramAB should ship only with a few default pure AIML chat bots.  (Alice2 ,  maybe some others?)
 
Bots have names.  A bot should have a concept of the Locale, however for some reason the InMoov bots are named after the locale.  I think this is odd.  
 
Next.  I like the idea of the resources/ProgramAB/bots folder to contain the bots  (I would even like to remove the "bots" part of the path, however, that will require a change to ProgramAB itself.  perhaps it's time for that?)
 
Also..  I don't think the InMoov2 resources should be added to any directory other than resources/InMoov2 ... As mentioned in the shoutbox before.   Containing resources for a service to that services resource directory will make things more modular and easier to upgrade.  If resources spread their files over multiple directories, it's not very clean.
 
Additionally, I think we need to consider the resources/Service directories as read only.  I think that ProgramAB should be updated to write out incremental AIML into the data/ProgramAB/BotName directory instead.. When loading the AIML, the aiml should first be loaded from the resources directory and then the incremental AIML should be loaded from the data directory.   On creating a new session, all the files under the bots/botname/config directory should also be under the data/resource/ProgramAB/botName directory instead...   The idea / hope is that you would never edit the default aiml, but rather only craft your own overrides in the data directory  (not the resource directory)
 
Lastly... don't break or change the unit test for ProgramAB.. it is very particular about what it tests.  When this new behavior is added , we should add tests to ProgramABTest to validate the desired behavior.
 
On.. more lastly.  I don't think we need to worry about a CI/CD build/deployment of the programab stuff..  It's largely static.. a one time re-packaging should be sufficient. 

ok.. so..  you're missing a few dependencies.

    meta.addDependency("org.apache.lucene", "lucene-analyzers-common", "8.4.1");
    meta.addDependency("org.apache.lucene", "lucene-analyzers-kuromoji", "8.4.1");
 
Those are for japanese support.  (the japanese support that existed in ProgramAB before was horrible.. this is a decent shot at that.)
 
ok, so they are for jar/binary support ... i'm primarily focused on the /resource/ProgramAB directory .. and my guess was these do not contribute to it
 
Next,  you're correct,  the dependencies for ProgramAB (the default chat bots ) should NOT have anything to do with inmoov.  I tried expliaining this in the past to folks, but it wasn't quite understood.  ProgramAB should ship only with a few default pure AIML chat bots.  (Alice2 ,  maybe some others?)
 
That's what I figured...  Can you package a "pure" Alice2 for ProgramAB and/or others ?
 
Bots have names.  A bot should have a concept of the Locale, however for some reason the InMoov bots are named after the locale.  I think this is odd.  
 
I had nothing to do with the naming, but I suspect this had to do with trying to make it clear for a noobie user where their language exists.  moz4r and others tried very hard to make things mult-lingual, which I can certainly appreciate.  Having a robot speak in a foreign tongue is not nearly as cool that speaks with you natively.  This could be enhanced to have bot names, and the whole language thing could be re-worked using Java locals ... but this would be a very large project I'm not ready to take on.
 
Next.  I like the idea of the resources/ProgramAB/bots folder to contain the bots  (I would even like to remove the "bots" part of the path, however, that will require a change to ProgramAB itself.  perhaps it's time for that?)
 
I'm keeping it bots at the moment.  I don't care, but yes it does need to be synchronized. 
 
Also..  I don't think the InMoov2 resources should be added to any directory other than resources/InMoov2 ... As mentioned in the shoutbox before.   Containing resources for a service to that services resource directory will make things more modular and easier to upgrade.  If resources spread their files over multiple directories, it's not very clean.
 
I mostly agree with that, except I'm not sure you understand the complexity of what it means.  
I absolutely agree that nothing should be "overwritten" - and that's why I started this post, because I saw that InMoov2 and ProgramAB would be in conflict (actually they would miss .. but that is due to more inconsistencies :P).  
 
I think "not writing to other directories" is definately preferred, but it means services will need to be more clever regarding their resources, or there needs to be a resource manager and not just a single path.  For example, the webgui has a single root and it has a shit-tonne of its own resources.  It's not very complient at setting 2+ roots.
 
ProgramAB has a single root - at the moment I don't think ProgramAB ships with anything that require its single root to not be at /resource/ProgramAB/bots.  ie InMoov2 can just programmatically switch ProgramAB's single root to brain.setPath({InMoov2's resource root}/bots}) ... this would mean the user doesn't get to play with Alice2 if they start InMoov2 ... not a big deal, but its an attempt to show you some of the detail.
 
I was thinking that this could be done in stages :
  • Step 1 : cleanup the conflict and chaos between ProgramAB & InMoov2.  Make sure no file overwrites happen and establish a solid directory struture for "merging" resources
    e.g.
    zip path inside ProgramAB would be
          /resource/ProgramAB/bots/Alice2
    zip path inside InMoov2 would be 
          /resource/ProgramAB/bots/en-US
          /resource/ProgramAB/bots/fr-FR
          /resource/InMoov2/gestures
          ...
          /resource/WebGui/app/views/InMoov2Gui.html
          /resource/JMonkeyEngine/assets
    this is how it works now (at least up to the point of ProgramAB and InMoov2 conflict mess)
    It may not be clear during runtime where all the resources came from, but at least its clear in the build and packaging where stuff is.
  • Step 2 : Make any peer resource capable of programmatically accepting more resources. I don't want a copy of all WebGui resources in InMoov2 ... 
  • Step 3 : have complete self-containment and use the new programmatic utilities to add resources dynamically
    once this was done, zip path inside InMoov2 would change to be
          /resource/InMoov2/ProgramAB/bots/en-US
          /resource/InMoov2/ProgramAB/bots/fr-FR
          /resource/InMoov2/gestures
          ...
          /resource/InMoov2/WebGui/app/views/InMoov2Gui.html
          /resource/InMoov2/JMonkeyEngine/assets
     

I thought about these details for a while, and I think its possible to go directly to step 3... but I have Servos to take care of now

Additionally, I think we need to consider the resources/Service directories as read only.  I think that ProgramAB should be updated to write out incremental AIML into the data/ProgramAB/BotName directory instead.. When loading the AIML, the aiml should first be loaded from the resources directory and then the incremental AIML should be loaded from the data directory.   On creating a new session, all the files under the bots/botname/config directory should also be under the data/resource/ProgramAB/botName directory instead...   The idea / hope is that you would never edit the default aiml, but rather only craft your own overrides in the data directory  (not the resource directory)
 
Agreed, who is going to do it and when?
 
Lastly... don't break or change the unit test for ProgramAB.. it is very particular about what it tests.  When this new behavior is added , we should add tests to ProgramABTest to validate the desired behavior.
 
Agreed, who is going to do it and when?
 
On.. more lastly.  I don't think we need to worry about a CI/CD build/deployment of the programab stuff..  It's largely static.. a one time re-packaging should be sufficient. 
 
I don't really care much, aside from I'm very exicited utilizing the new capability I've added to the CI/CD
  • The JenkinsFile currently in InMoov2 should be a copy/paste/search/replace
  • Creating a new one + the build the correctly deploys it and polls for changes should take literally a couple minutes
  • With the new "latest" dependency feature all building/deploying/releasing is zero maintenance after the setup