Quick HTTPClient example

The following will dump all the data to the python tab (print it) from the default urban dictionary site.
You might want to find a site which delivers just simple text or parse out the joke or part of interest.

Make sure you have the HTTPClient service installed.

from java.lang import String

http = Runtime.createAndStart("http","HTTPClient")

print String(http.get("http://www.urbandictionary.com/"))

So the tough (and it should

So the tough (and it should not be very tough) part is parsing - so you get clean simple text to "speak"

given the following URL

http://www.ajokeaday.com/Clasificacion.asp?ID=64&Pagina=10

It comes back with all sorts of gobbly gook - but if you look closely - you'll find the interesting stuff always starts with

<div class="chiste">

Two strands of DNA were walking down the street. One says to the other, "Do these genes make me look fat?

</div>

I think with the following method

def find_between( s, first, last ):
    try:
        start = s.index( first ) + len( first )
        end = s.index( last, start )
        return s[start:end]
    except ValueError:
        return ""

might allow this to work...

speech.speak(find_between( String(http.get("http://www.ajokeaday.com/Clasificacion.asp?ID=64&Pagina=10")),"<div class=\"chiste\">", "</div>")

"random jokes - would just be random categories and pages... you might have to strip more html formatting out with replace functions"

Analyzation of an HTML-Website

This is an good working method to get the important data (the changed classes) out of theese websites (it'll only work with theese):

https://krupp-gymnasium.eu/idesk/infodisplay/mods/link.local.php/panelI…

It is a Java-Method and REALLY detailed and the opposite of an optimized code, but hey, it's working very good ...

Sorry, the code is partly in German:

public ArrayList<ArrayList<ArrayList<ArrayList<String>>>> analysieren(String komp) {
        ArrayList<ArrayList<ArrayList<ArrayList<String>>>> rückgabe = new ArrayList<ArrayList<ArrayList<ArrayList<String>>>>();
//        System.out.println("|" + komp + "|");

        String body = komp.substring(komp.indexOf("<body>") + 7, komp.indexOf("</body>") - 1);
//        System.out.println("|" + body + "|");

        String hauptinfo = body.substring(body.indexOf("<table") + 29, body.indexOf("</table>") - 1);
        hauptinfo = hauptinfo.substring(hauptinfo.indexOf("<tr>") + 13, hauptinfo.indexOf("</tr>") - 5);
        hauptinfo = hauptinfo.substring(hauptinfo.indexOf("</td>") + 14, hauptinfo.length());
        hauptinfo = hauptinfo.substring(hauptinfo.indexOf("</td>") + 14, hauptinfo.length());
        hauptinfo = hauptinfo.substring(hauptinfo.indexOf("<p>") + 3, hauptinfo.indexOf("</p>"));
//        System.out.println("|" + hauptinfo + "|");

        String stand = hauptinfo.substring(hauptinfo.lastIndexOf("</span>") + 15, hauptinfo.length());
//        System.out.println("|" + stand + "|");

        ArrayList<ArrayList<ArrayList<String>>> standrückgabe = new ArrayList<ArrayList<ArrayList<String>>>();
        if (true) {
            ArrayList<ArrayList<String>> temp = new ArrayList<ArrayList<String>>();
            ArrayList<String> temp2 = new ArrayList<String>();
            temp2.add(stand);
            temp.add(temp2);
            standrückgabe.add(temp);
        }
        rückgabe.add(standrückgabe);

        String hauptteil = body.substring(body.indexOf("</table>") + 10, body.length());
        hauptteil = hauptteil.substring(hauptteil.indexOf("<center>") + 9, hauptteil.indexOf("</center>"));
//        System.out.println("|" + hauptteil + "|");

        String tag = hauptteil.substring(hauptteil.indexOf("<div") + 23, hauptteil.indexOf("</div>"));
//        System.out.println("|" + tag + "|");

        ArrayList<ArrayList<ArrayList<String>>> tagrückgabe = new ArrayList<ArrayList<ArrayList<String>>>();
        if (true) {
            ArrayList<ArrayList<String>> temp = new ArrayList<ArrayList<String>>();
            ArrayList<String> temp2 = new ArrayList<String>();
            temp2.add(tag);
            temp.add(temp2);
            tagrückgabe.add(temp);
        }
        rückgabe.add(tagrückgabe);

        String taginfo = hauptteil.substring(hauptteil.indexOf("<table") + 22, hauptteil.indexOf("</table>") - 1);
//        System.out.println("|" + taginfo + "|");

        String taginfokopf = taginfo.substring(taginfo.indexOf("<tr") + 17, taginfo.indexOf("</tr>"));
//        System.out.println("|" + taginfokopf + "|");

        String taginfohaupt = taginfo.substring(taginfo.indexOf("</tr>") + 6, taginfo.length());
//        System.out.println("|" + taginfohaupt + "|");

        String hauptvert = hauptteil.substring(hauptteil.indexOf("<p>") + 4, hauptteil.length());
        hauptvert = hauptvert.substring(hauptvert.indexOf("<table") + 26, hauptvert.indexOf("</table>") - 1);
//        System.out.println("|" + hauptvert + "|");

        String hauptvertkopf = hauptvert.substring(hauptvert.indexOf("<tr") + 17, hauptvert.indexOf("</tr>"));
//        System.out.println("|" + hauptvertkopf + "|");

        String hauptverthaupt = hauptvert.substring(hauptvert.indexOf("</tr>") + 6, hauptvert.length());
//        System.out.println("|" + hauptverthaupt + "|");

        String schbea = hauptverthaupt + "\n";
        ArrayList<ArrayList<ArrayList<String>>> vertretung = new ArrayList<ArrayList<ArrayList<String>>>();
        while (!schbea.equals("")) {
            String einezeile = schbea.substring(schbea.indexOf("<tr"), schbea.indexOf("</tr>"));
            einezeile = einezeile.substring(einezeile.indexOf(">") + 1, einezeile.length());
//            System.out.println("|" + einezeile + "|");

            if (einezeile.contains("inline_header")) {
                String eineinfo = einezeile.substring(einezeile.indexOf("<td"), einezeile.indexOf("</td>"));
                eineinfo = eineinfo.substring(eineinfo.indexOf(">") + 1, eineinfo.length());
//                System.out.println("1|" + eineinfo + "|");

                ArrayList<ArrayList<String>> temp = new ArrayList<ArrayList<String>>();
                ArrayList<String> temp2 = new ArrayList<String>();
                temp2.add(eineinfo);
                temp.add(temp2);
                vertretung.add(temp);
            } else {
                ArrayList<ArrayList<String>> temp = vertretung.remove(vertretung.size() - 1);
                ArrayList<String> temp2 = new ArrayList<String>();

                String einezeileb = einezeile;
                for (int i = 0; i < 7; i++) {
                    String eineinfo = einezeileb.substring(einezeileb.indexOf("<td"), einezeileb.indexOf("</td>"));
                    eineinfo = eineinfo.substring(eineinfo.indexOf(">") + 1, eineinfo.length());
//                    System.out.println("2|" + eineinfo + "|");

                    einezeileb = einezeileb.substring(einezeileb.indexOf("</td>") + 5, einezeileb.length());
//                    System.out.println("rs|" + einezeileb + "|");

                    temp2.add(eineinfo);
                }

                temp.add(temp2);
                vertretung.add(temp);
            }

            schbea = schbea.substring(schbea.indexOf("</tr>") + 6, schbea.length());
//            System.out.println("r|" + schbea + "|");
        }

        rückgabe.add(vertretung);

//        for (ArrayList<ArrayList<String>> temp : vertretung) {
//            for (ArrayList<String> temp2 : temp) {
//                for (String temp3 : temp2) {
//                    System.out.print(temp3 + "|");
//                }
//                System.out.print("\n");
//            }
//        }
        return rückgabe;
    }

Ma vo, warum in deutsch? Ist

Ma vo, warum in deutsch? Ist english nicht besser? Dann können andere leute auch das begreifen? Oder hab ich es falsch?

Of course English would be

Of course English would be better, but I had written this a few mounths ago. A few variable-names are partly German or use shortcuts of German words.

Although I think, it is understand-able ...

Thanks for sharing Ma. Vo. I

Thanks for sharing Ma. Vo. I was curious if you could tell me what you use it for. It looks like its final output is to print something. What does it print? How does it help you? How do you use it?

I use it to get the data

I use it to get the data (changed classes) out of theese two websites (posted above the code). They are displaying which teacher are ill or rooms are blocked, and so on.

Then I display the data in a better (/easier) way. This script worked already in an servlet to provide a website with only the "data" for one class (mostly my ; )). Currently it's running in an Android-App, there I worked with it further, save/load it, customize it further ...

But the main thing is this function (and the method to load the HTML from the website, which isn't always so easy at our school-server ..., they break the configuration and certificates regulary).

The data that comes back, is only an big, encapsulated array.

The many prints had been only for testing and debugging, I hadn't removes them, because when somewhere is an error, I can quickly find and fix it (or can persuade me, that the error is'nt at this part).

I hope I had expressed myself clearly ... ; ),

Marvin

I just decided to upload the whole class, it's completly implemented for Android ...

https://drive.google.com/file/d/0B3Q0--r6Z_6VVXlKYjVQSEpXa1E/edit?usp=s…

And sorry (wvantoorn) the variable- and method- names are partly german and all comments, too ...

I don't if Google can translate it, but it should be relativly clear what it does ...

important methods:

vertretungspeichern(Context ctx) -> this is called when the stored data should be updated, it only gives back an boolean, wether this was succesful or not (e.g. no network-connection)

=>Reading from the website->Analyzing it, Generating a JSON, Saving it

vertretunglesen(Context ctx, String klasse, int tag) -> this function is called when the data from the website is needed (to display it), it takes the class (e.g. "9E" (5A, 5B, 5C, 5D, 6A-D, 7A-D, 8A-D, 9A-E, EP, Q1, (Q2,) VK)) and the day (today ot tomorrow).

=>Reading from the storage->Get data out of JSON

alright - I tested this one

alright - I tested this one it works ...

Some things you will need to be careful with - special characters coming back from the Joke site - the joke creates a mp3 file on the file system with the name of the file being the text of the joke - if there are characters the filesystem does not support - it will fail - this particular joke works.. although I thought it was "funny" she say "Euro" for the dash -

from java.lang import String

def find_between( s, first, last ):

try:

start = s.index( first ) + len( first )

end = s.index( last, start )

return s[start:end]

except ValueError:

return ""

speech = Runtime.createAndStart("speech","Speech")

http = Runtime.createAndStart("http","HTTPClient")

cleanText = String(http.get("http://www.ajokeaday.com/Clasificacion.asp?ID=64&Pagina=32"))

# print cleanText

joke = find_between(cleanText.toString(),"<div class=\"chiste\">", "</div>")

speech.speak(joke)

Heh.. All Joking aside.. A challenge

I don't usually do much Python, because I'm working most of the time on the Java framework..

but here's "potentially" this could be a Python joke function for InMoov .. there were several (many) issues..

Illegal characters.. I had problems stripping bad stuff out.. html <br> carriage return need stripping out.. i wanted random jokes..

ok .. so its "almost" ready .. the only problem is .. I've disovered google does not accept text over a relatively short length... Sooooo.. you'll need to split up the text into block and then submit them to speech !

And if you want to get "fancy" you should split them up were word count > max value OR puctuation exists .. whatever comes first ! Have Fun !

# put this stuff at the top

from java.lang import String

import string

import random

# don't use this speech use inmoov's mouth

speech = Runtime.createAndStart("speech","Speech")

http = Runtime.createAndStart("http","HTTPClient")

def say_joke():

ID = random.randint(1, 80)

Pagina = random.randint(1, 80)

url = "http://www.ajokeaday.com/Clasificacion.asp?ID=%s&Pagina=%s" %(ID, Pagina)

print url

data = String(http.get(url)).toString()

joke = find_between(data,"<div class=\"chiste\">", "</div>")

punchLine = joke.replace('<BR>','').replace('?','').replace('\"','').replace(',','').replace('<','').replace('>','').replace('/','').replace('\n','').replace('\r','')

print punchLine

speech.speakBlocking(punchLine)

speech.speakBlocking("I heard this story in the late seventies. Just prior to a manned launch an insect ")

speech.speakBlocking("was heard in the superstructure of the rocket. The risk analysis and decision had to ")

speech.speakBlocking("be made to launch or to delay the launch and purge the insect. The risk was ")

speech.speakBlocking("negligible so the rocket launch. The insect was dubbed the Gemini Cricket. ")

def find_between( s, first, last ):

try:

start = s.index( first ) + len( first )

end = s.index( last, start )

return s[start:end]

except ValueError:

return ""

# the command will be a single function - in this case say_joke() will tell a random joke

say_joke()

unknown service org.myrobotlab.service.HTTPClient

I have a problem with the launch of HTTPClient service.

Of course, the service at GUI runtime is look as installed, but not be found by script.

Can you help me?

mrl 1.0.1412

[python] [INFO] exec(String)

from java.lang import String

def find_between( s, first, last ):

try:

start = s.index( first ) + len( first )

end = s.index( last, start )

return s[start:end]

except ValueError:

return ""

#speech = Runtime.createAndStart("speech","Speech")

http = Runtime.createAndStart("http","HTTPClient")

cleanText = String(http.get("http://www.ajokeaday.com/Clasificacion.asp?ID=64&Pagina=32"))

# print cleanText

joke = find_between(cleanText.toString(),"<div class=\"chiste\">", "</div>")

#speech.speak(joke)

[python.interpreter.3] [INFO] Runtime.createService http

[python.interpreter.3] [ERROR] unknown service org.myrobotlab.service.HTTPClient

[python.interpreter.3] [ERROR] org.myrobotlab.service.HTTPClient is not installed

[python.interpreter.3] [INFO] installing org.myrobotlab.service.HTTPClient

[python.interpreter.3] [ERROR] org.myrobotlab.service.HTTPClient not found

[python.interpreter.3] [INFO] class org.myrobotlab.service.HTTPClient not found

[python.interpreter.3] [ERROR] runtime error createAndStart(http, HTTPClient) error

[python.interpreter.3] [ERROR] ------

java.lang.NullPointerException

at org.myrobotlab.service.Runtime.createAndStart(Runtime.java:266)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)

at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)

at org.python.core.PyObject.__call__(PyObject.java:478)

at org.python.core.PyObject.__call__(PyObject.java:482)

at org.python.pycode._pyx3.f$0(<string>:16)

at org.python.pycode._pyx3.call_function(<string>)

at org.python.core.PyTableCode.call(PyTableCode.java:167)

at org.python.core.PyCode.call(PyCode.java:18)

at org.python.core.Py.runCode(Py.java:1386)

at org.python.core.Py.exec(Py.java:1430)

at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:267)

at org.myrobotlab.service.Python$PIThread.run(Python.java:177)

Services are case sensitive

Hi Darek

Service names are case sensitive. You can start it like this from a script.

http = Runtime.createAndStart("http","HttpClient")

/Mats

Hi Mats,thanks for your

Hi Mats,

thanks for your help. Unfortunately, it still does not work:

mrl 1.0.1701

[python] [INFO] exec(String)

from java.lang import String

http = Runtime.createAndStart("http","HttpClient")

#http = Runtime.start("http","HttpClient")

print String(http.get("http://www.urbandictionary.com/"))

[python.interpreter.3] [INFO] Runtime.createService http

[python.interpreter.3] [ERROR] ------

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)

at java.lang.reflect.Constructor.newInstance(Unknown Source)

at org.myrobotlab.framework.Instantiator.getThrowableNewInstance(Instantiator.java:59)

at org.myrobotlab.framework.Instantiator.getNewInstance(Instantiator.java:20)

at org.myrobotlab.framework.Instantiator.getNewInstance(Instantiator.java:37)

at org.myrobotlab.service.Runtime.createService(Runtime.java:368)

at org.myrobotlab.service.Runtime.create(Runtime.java:254)

at org.myrobotlab.service.Runtime.createAndStart(Runtime.java:267)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)

at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:204)

at org.python.core.PyObject.__call__(PyObject.java:478)

at org.python.core.PyObject.__call__(PyObject.java:482)

at org.python.pycode._pyx3.f$0(<string>:4)

at org.python.pycode._pyx3.call_function(<string>)

at org.python.core.PyTableCode.call(PyTableCode.java:167)

at org.python.core.PyCode.call(PyCode.java:18)

at org.python.core.Py.runCode(Py.java:1386)

at org.python.core.Py.exec(Py.java:1430)

at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:267)

at org.myrobotlab.service.Python$PIThread.run(Python.java:161)

just in case I make "install

just in case I make "install all". Now... it looks much better, although it stiil not works.

13:42:55.552 [python] INFO c.m.s.Python [Python.java:527] exec(String)

from java.lang import String

http = Runtime.createAndStart("http","HttpClient")

#http = Runtime.start("http","HttpClient")

print String(http.get("http://www.urbandictionary.com/"))

13:42:55.579 [python.interpreter.3] INFO c.m.s.Runtime [Runtime.java:331] Runtime.createService http

13:42:55.586 [python.interpreter.3] INFO c.m.f.Service [Service.java:988] getMessageSet loading 156 non-sub-routable methods

13:42:55.596 [python.interpreter.3] WARN c.m.f.Service [Service.java:1348] no such method Runtime.registered(HttpClient) : - attempting upcasting

13:42:55.596 [python.interpreter.3] WARN c.m.f.Service [Service.java:1353] ouch! need to search through 249 methods

13:42:55.598 [gui] WARN c.m.f.Service [Service.java:1348] no such method GUIService.registered(HttpClient) : - attempting upcasting

13:42:55.599 [gui] WARN c.m.f.Service [Service.java:1353] ouch! need to search through 191 methods

13:42:55.602 [python] WARN c.m.f.Service [Service.java:1348] no such method Python.onRegistered(HttpClient) : - attempting upcasting

13:42:55.602 [python] WARN c.m.f.Service [Service.java:1353] ouch! need to search through 174 methods

13:42:55.602 [python] INFO c.m.s.Python [Python.java:527] exec(String)

from org.myrobotlab.service import HttpClient

http = Runtime.getService("http")

13:42:55.851 [python.interpreter.3] ERROR c.m.f.Service [Service.java:2131] python error PyException - null Traceback (most recent call last):

File "<string>", line 4, in <module>

at org.myrobotlab.service.HttpClient.processResponse(HttpClient.java:206)

at org.myrobotlab.service.HttpClient.get(HttpClient.java:140)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

java.lang.NoSuchMethodError: java.lang.NoSuchMethodError: org.apache.http.impl.client.DefaultHttpClient.execute(Lorg/apache/http/client/methods/HttpUriRequest;)Lorg/apache/http/client/methods/CloseableHttpResponse;

This has to do with

This has to do with conflicting HttpClient versions ..

Its such a popular library - that I'm betting more than one service included it as a dependency .. but they are with different version, the versions clash .. and BOOM

I'll figure out which is which ....

Wow ! .. That was a big mess.

Wow ! .. That was a big mess. Different versions of apach httpclient DO NOT PLAY WELL TOGETHER !!!

There was a evil mixture of 4.2.5 3.1 & 4.5.2 ... But kwatters & I got it all cleaned up.

Below was tested

http = Runtime.createAndStart("http","HttpClient")

print http.get("http://www.urbandictionary.com/")

FULL Worky version 1721 !

Some things to note :

run it and install services in a "CLEAN" directory - if the httpclient 4.2.5 jars get in they will fight
its HttpClient and its case sensitive
http.get(url) returns a String so you dont need to stick it in a new string - because .get returns what you would expect, which is typically a string
http.getBytes(url) will return a byte array if thats what you want

I'm glad that it was only a

I'm glad that it was only a minor scrimmage processes. And that I could in time to warn you.

I dread to think what would be happen (;

Regarding your suggestions to http.getBytes - I couldn't decode from byte array to json, so I bypassed by convert from string to json as below:

Thanks a lot!

RemoteAdapter dependecies

Hi Grog

To aviod having to "install all" and only install the services that I really use, I added dependencies to RemoteAdapter. I thought that it was the correct place to add them. Where should the dependencies for each service be defined for MRL to work properly ?

Currently I can't start RemoteAdapter, It fails with message:

22:41:12.536 [runtime] ERROR c.m.f.Service [Service.java:2131] runtime error NoClassDefFoundError org/atmosphere/wasync/Encoder

I tried to "install all" but still the same error.

It works from Eclipse.

/Mats

Hi Mats, you had the right

Hi Mats,

you had the right idea of adding nettosphere as a dependency.
wasync is a websocket client library from the same project which gives us the webgui. _goodtimes_

RemoteAdapter should learn to interface with the WebGui.

It seems like you add them, then commented them out, then added them again.

They should be added, although at the moment I'm not seeing the httpclient dependecy... might be there, but the wasyc is definately there...

Is it worky now ?

Hi GroG I added them when I

Hi GroG

I added them when I started to test on the RasPi, because it wouldn't install otherwise.

I will make at test to see if it's enough to have only nettosphere as a dependency in RemoteAdapter.

It installs and starts now.

/Mats

RemoteAdapter

Hi GroG

You were 100% correct. It was only nettosphere that was needed as a dependency in RemoteAdapter.

It really exciting to test out RemoteAdapter. I created two scripts. One in each of the PI's. The script in the head pi starts all services in the head + RemoteAdapter. As a last step it the sends an SSH comnand to start MRL and the body script. The body script starts a services for the body and it's own instance of RemoteAdapter. Since I only have a speaker in the head, the body script sends a command to the head to say that the initialization of ther body script has completed. And it's all worky. AWSOME.

I will continue testing. RemoteAdapter is working far better than I ever expected.

Really great job.

Same problem

Hello everyone sorry to dig out this old solved post!

I meet the same problem as you did, which is when trying to start RemoteAdapter i get the error:

NoClassDefFoundError org/atmosphere/wasync/Encoder.

I tried to install all and it did not work either. I understood that you solved this by adding nettosphere as a dependency to RemoteAdapter. In order to do that i must clone the git under eclipse and add it? Is there an easier way?

Thank you for your time, All the best,

Nicolas Taguet.

Hi The workaround is to go to

The workaround is to go to the runtime tab => system => install all.

What version are you using ?