Services written in Native Python

Hello all!

Over the past week, I've been working on a little addition to MRL's service capabilities, I wanted to create a Google Assistant service, but the API is in Python (a Java imlementation exists, but it is hard to follow and has far fewer features compared with the Python example). So, to remedy these, I built a Native Python API, called mrlpy. This API currently supports registering as a service, process synchronization, a compatibility layer for scripts written for the Jython interpreter, and proxy service generation.

The API can be downloaded using Pip: 

$ sudo pip install mrlpy

The source code and API documentation can be found on my github page

mrlpy will be updated shortly to support full compatibility and full mesage passing. The companion service (PythonProxy) will be committed to the development branch soon (just a little refactoring left). The WebGui service will have to be updated so that PythonProxy can access the list of clients and determine which one is the native python service.

After finishing up these problems, I will start work on a Google Assistant service and possibly a ROS bridge.

P.S.

I noticed that recent committs show that user grperry checked in a blank Proxy class in the develop branch. Perhaps we could combine our efforts?


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
GroG's picture

Wow .. I read this and it was

Wow .. I read this and it was a really enjoyable surprise.  I'm very impressed AutonomicPerfectionist.  I began reading your post and thought, "cool, another person has worked on the MRL API and found some use out of it" ..  however, the more I read the more excited I got.

"The API currently supports registering as a service, process synchronization, .... "

You had me at registering as a service ;)

"mrlpy will be updated shortly to support full compatibility and full mesage passing"

Excellent !
Google Assistant ?  ROS bridge ?  Awesome.

The WebGui service will have to be updated so that PythonProxy can access the list of clients and determine which one is the native python service.

I'd like to know in what way .. I have a large commit coming which will "add to" the existing API perhaps what you need is in the future commit, depends...  I'd need more info, perhaps the Proxy service can suffice .. 

Ya, I'm grperry when I'm too lazy to change my .gitconfig global config :)

WebGui and the processing of the MRL API has been refactored substantially.  The API itself hasn't change "much".  It has a few changes which are important, but you might be able to disregard them.  I'd have to see how you start communication.

The purpose of the refactoring was to support something currently called WebGuiClient service.  This currently is a websocket client which can attach from one MRL instance to another instance running WebGui.  It would allow distributed messaging over multiple instances of MRL.  

WebGuiClient is a java implementation of a client which sounds exactly like what your mrlpy does.  I have been working on a java websocket client, but it was my full intention that the changes I have done were enough to fully support clients from any language ... e.g. JavaScript (which is our Angular display), Python, Go, etc ....

The "Proxy" service is what you probably already guessed - a "representitive of a Service in the registry which is "not" Java.   It is a placeholder for a remote service and a message & integration point for all the services currently running in Java MRL. 

I'll try to go over the code you have checked into github, and I am excited to see your project.

 
AutonomicPerfectionist's picture

Hey, thanks GroG! Yeah, the

Hey, thanks GroG!

Yeah, the WebGuiClient service is probably what I need. I've written a workaround that just uses webgui.broadcast() to forward messages, but that's inefficient and can also cause problems where there is more than one service with the same name. I've tried to check in my code, but I don't seem to have access. I've instead pushed it to a fork on my github page for the time being.

Thanks again!

GroG's picture

The WebGuiClient is what you

The WebGuiClient is what you want to do .. but in Python.
The WebGuiClient is a consumer of the WebGui's API - but over JSON serialization, which Python, Javascript, etc.. can use.  

You won't "need" the WebGuiClient - You'll probably want to follow what its doing - and do the same thing in your WebGuiClient.py  (which would be conceptually the same) .. I'm pretty sure you already have something similar to it - but probably named something else.

I did a very cursory look at mrlpy.  I can tell your Python-Fu is very great ...  my Python-Fu is very weak in comparison.  But I'm very excited by what you are doing.

Instead of a PythonProxy, I think perhaps it migh be best to have a generalized Proxy for "all" languages which do not completely serialize into the current language running in the current process.  

For this, the WebGui must intercept the "register" request - check to see if it's natively serializable - if not create a Proxy - then load the proxy with the incoming data and continue with the runtime registration.

In addition, there should be a more structured communication setup - even before registration.  That's the reason I created a Hello object.  Within the Hello object contains platform info, but with this discussion I can see that its too "Java-centric" and I'll start changing it to be more universal.  The Hello object contains the local Platform - which contains MrlVersion, and other details.

I'd like to get a post or example script of how to use mrlpy.  I'm very interested in having Mrl language agnostic, since the real power of Mrl is communication & messaging vs specific implementations.

I know this is a bit of a ramble, but its morning & pre coffee ... I'll have code, docs and diagrams which will make more sense .. but all this takes time.. :)

 

AutonomicPerfectionist's picture

Thanks for the ideas. I think

Thanks for the ideas. I think I know what to change, but unfortunately I can't work on it today (landscaping with my dad in 100 degree heat, fun), so I'll start making changes tomorrow. After that, though, I have to start studying for my ACT test on Saturday so it'll be a little bit before I can finish making the changes. Ill take a look at WebGuiClient and figure out how to translate it tonight. I'll also quickly spin up an example script tonight too and put it on pyrobotlab.
Thanks again!
GroG's picture

Ya ... sounds like fun :) A

Ya ... sounds like fun :)

A quick disclamer ...  WebGuiClient doesn't have much to copy at the moment .. still working on it,  
but it "will" be the one to copy from - as it will :

  • create a ws connection with webgui
  • use the message api
  • say "Hello" with platform information
  • register itself
  • use the pub/sub 
  • use the Proxy service
  • use blocking api

in process of making that stuff now

AutonomicPerfectionist's picture

Current WebGui having Problems

Just a heads-up, GroG, but the current WebGui is having problems with POST requests. For whatever reason, atmosphere is labeling the request's content-type as text/plain, and request.body().asString() returns null. Using Lamiak's WebGui works perfectly with no changes to mrlpy. I've looked at the headers before sending and they are correct, but atmosphere is still having problems. I've temporily fixed it in my local copy by transplanting Lamiak's WebGui over and modifying it just a tiny bit so that it would compile (changing createMessage() calls to Message.createMessage(), adding null NameTypeProvider as first argument to createMessage() calls).

As for mrlpy's examples, they are up in the examples folder on the github page. They have been excluded from PyPi so they will not be installed with the rest of mrlpy.

GroG's picture

Thanks AP, Not totally

Thanks AP,

Not totally suprised considering the messaging system pretty much got ripped out and now all of the messaging api is getting refactored and put in modular classes outside of Services.  Sorry for the disruption, the pieces should be back together shortly.

Messages needed to know how to create themselves, without Services.

I'm curious, does the webgui work for you currently ?  If not I should check-in more refactoring, because I test the WebGui angular client pretty regularly - and locally its workindg for me.

Thanks for the demo scripts and information for mrlpy !   I think it was really fortunate timing for us both to be working on this but from opposite sides at the same time.  Its like two teams making a tunnel and going about it blind for weeks and meeting one another...  :)

AutonomicPerfectionist's picture

Hey, sorry for not responding

Hey, sorry for not responding quickly (suddenly got really busy around the house).

Unfortunately, my webgui is not working with the latest commit. From what I see, it looks as if the body of the request is consumed elsewhere and does not get to the handle method of webgui. Also, the service API is broken for params encoded in a POST body. I've been working on that recently. It looks as if synchronous returns is not yet supported in the current commit.

This is the segment of the log that seems to show there is a problem:

00:03:23.395 [New I/O worker #6] INFO  class org.myrobotlab.service.WebGui - GET /api/messages
00:03:23.424 [New I/O worker #6] ERROR class org.myrobotlab.service.WebGui - msg is null null
 

Unfortunately, I will be out of town for a week starting tomorrow so I won't be able to add on or anything. Just a heads up, the proxy service is still called PythonProxy because I don't want to overwrite the Proxy class just yet and it currently supports executing native services, but only through the python command currently.

Thanks, GroG for the help! I will be back Tuesday and I will continue working on this project then.

 

P.S. Still having access problems, so code is pushed to my fork

juerg's picture

in what way will mrlpy be

in what way will mrlpy be different from the python service we already have in mrl?

AutonomicPerfectionist's picture

Hello juerg Python is an odd

Hello juerg

Python is an odd language that allows writing extensions to the interpreter, normally written in C. Some libraries, such as lxml, require these extensions in order to function.

In MRL, the Python service uses a Python interpreter written in Java called Jython. Jython supports all pure python ("Pythonic") code, but, for reasons of portability and simplicity, is incapable of loading Python modules that were written as C extensions. Instead, Jython allows importing Java classes as modules, but it still leaves a compatibility hole. Therefore, certain Python libraries will not work in Jython and can't be used in MRL. mrlpy is a library that runs in the standard CPython interpreter (default interpreter that is installed when installing Python), and therefore is fully compatible with Python modules that were written in C or in native code. Using mrlpy does make MRL more platform-dependent, but also opens the door for many more services to be written that take advantage of those native / C libraries.

CPython and therefore mrlpy are supported on all major platforms, but certain Python libraries, since they are written in C, will not work on certain platforms.

juerg's picture

thanks a lot for your

thanks a lot for your clarifications. Sounds like your mrlpy will add more options for us - great!

Mats's picture

ROS Bridge

I started to look at ROS and found that there is a Java implementation of ROS called RosJava.

These are the steps to install RosJava on Windows without installing ROS. The links are for Linux I guess, but the procedure is very similar.
 
Step 1: Download ros_core ( http://rosjava.github.io/rosjava_core/latest/installing.html )
 
git clone https://github.com/rosjava/rosjava_core
cd rosjava_core
( The installation says git checkout -b hydro origin/hydro, but that's an old release )
git checkout -b kinetic origin/kinetic
if it already exists
git checkout kinetic
 
Step 2. Build ros_code using Gradle ( http://rosjava.github.io/rosjava_core/latest/building.html )
 
cd rosjava_core
gradlew install
 
Step 3. Build the documentation ( A few steps failed when creating the docs in Windows, I didn't dig into the details ).
gradlew docs
 
Step 4. Run the tests ( 142 test completed, 8 failed. Same here. I didn't dig into the details of the 8 failing tests ).
gradlew test
 
Step 5. Generate Eclipse project files
gradlew eclipse
 
In the examples you can find Client.java and Server.java
and also
Listener.java and Talker.java
 
I guess they could be used as templates for building the bridge.
 
I have an instance of ROS runnnig on my Raspberry, but have only made very basic tests using the messaging system. Steep learning curve.
 
So I'm very far from using ROS, but I think it contains some good stuff, so a bridge would be a good thing to have in MRL.
AutonomicPerfectionist's picture

Hello Mats, and thanks for

Hello Mats, and thanks for the info.
I've used rosjava before (attempted a bridge awhile ago), and I did get it to communicate with Ros successfully. Unfortunately, rosjava is no longer under active development and is only compatible with, I believe, Ros Jade. Kinetic is the current release with lunar being released soon. There has been work with Rosjava 2.0 (newer, more distributed Ros version semi compatible with normal Ros), but it's still in early development. In addition, rosjava requires the generation of message types to be compatible with Ros networks, something that is not quite perfect. Rosjava would be perfect for a Ros bridge from mrl, as long as the network is Jade or older (hydro or groovy). I'll take a look at my old bridge code to see if I can make it work this time. Thanks for the info, mats!