I have start to implement a way to detect if the Serial communication between MRL and arduinos is still active. The goal is to put both MRL and arduinos in a stanby state and try to reestablish connection.

The principle is simple. Arduino service in MRL send msg at fixed time and expect to get a reply (ACK) from the arduino before the timeout expired, and arduino expect to get at least 1 msg before it's timeout expire.

Currently

  • the process need to be manually start until we are sure it's stable
    • Arduino.enabledHeartbeat() method will activate this process
    • still need to do a stopHeartbeat() method
  • When the Serial communication is cut (ie: USB wire unplugged, crash of the PC/Arduino, etc) the arduino will softReset() and the Arduino service in MRL will disconnect()
  • still need to find good number for the timeouts

Now that this is working, I want to discuss on what do we want things to happen when the communication is lost.

My view is

  • On Arduino side
    • stop the update() loop
    • add a virtual Device::stanbyBegin()
      • called when the communication is cut
      • used to set devices in stanby mode, stop motors etc
    • add a virtual Device::stanby()
      • same as update by run in the stanby mode
      • We may want to have some device to continue working while the communication is off... By example, the neopixel ring may warn that the communication is off
    • add a virtual Device::stanbyEnd()
      • called when the communication is re-establish
      • used to restart device
  • On MRL side
    • warn the services attached to the arduino service that there is no more communication with the arduino
    • try to reconnect to the arduino
    • when communication is working again, reput everything in working condition

Some things need to be fixed to have this working

  • when MRL connect to arduino, the arduino reset (at least on Window). This is not compatible with what I want to do. Grog said that can be change, but I still have no idea how
  • When connection is reestablish, MRL will need to know if the arduino have been reset or not

So let me know what you think of this

kwatters

8 years 3 months ago

Hey Calamity,

  I think the idea of a heartbeat is a very very good one.  In particular,  for safety, in that if the connction to MRL is broken between the arduino & mrl,  we should be able to tell the arduino to shutdown and detach all devices.  Anyway,, very cool..  

 

I just tried uploading the latest locally and it seems I've got a compilation error.  (maybe something was missed in the checkin..)

( think it's just a minor typo ... )

 

Uploaded Fri Aug 12 10:28:27 EDT 2016 Loading configuration...
Initializing packages...
Preparing boards...
Verifying and uploading...
sketch\MrlComm.cpp: In member function 'void MrlComm::processCommand(int)':

MrlComm.cpp:354: error: 'heartbeatEnable' was not declared in this scope

   heartbeatEnable = true;

   ^

exit status 1
'heartbeatEnable' was not declared in this scope

Exit Value : 1

GroG

8 years 3 months ago

I would suggest trying to maintain or re-establish state vs "shutting down" ..  

Dumping the device could get you into this situation :

  • connection is momentarily broken - motor is running full tilt
  • connection is broken - devices are removed - your motor is still running full tilt .. and
    now you have to go through the procedure of creating a new motor device to attempt to control it.   YIKES THE WALL IS COMING CLOSER !!!  LOOK OUT !!

We have two states which should always be trying to stay in sync  (Java State & MrlComm State) 
On connection I would propose that nothing is reset nor re-initialized .. but state information should be the first set of messages exchanged on 're-connect'

That way - if connection is lost .. then regained it might look like this...

  • connection is momentarily broken - motor is running full tilt
  • connection is re-established - both sides say what they have - they are in agreement
  • Java Arduino says STOP !!  .. wall avoided

First step of this I think is taking the softReset() out of setup()

calamity

8 years 3 months ago

I agree that maintain/re-establish state is the way to go over shutting down

the softReset() call is just temporary to allow for testing the heartbeat function

 

You example is good, but I think the arduino need  to enter  in a stanby or "not connected" state to avoid problem like you said.  Connection loss my be for only a few second, but it may take longer time to re-establish.

softReset() is already out of the setup() but it's called in the mrlcomm constructor so it's about the same. But what blocking me to go further is that the arduino reset  when MRL try to connect to it (at least on window). So until I can take care of that, the only viable solution is a shut down of the arduino.

 

 

Mats

8 years 3 months ago

Hi

I see two problems with MLRComm trying to keep the state in case of loss of communication.

1. There is nothing that says that MRL will reconnect.

2. The Arduino board will be reset when MRL connects, so there will not be any state in MRLComm to sync with.

In my opinion MRLComm should do an emergency shutdown of all known devices, and then a "hard reset" if the communication is lost.

MRLComm needs to do this quick, because when MRL loose contact with the Arduino board, Windows will reset the Arduino board shortly after. It takes just a few seconds. So after that MRLComm don't know anything about any device.

/Mats

 

calamity

8 years 3 months ago

yes, it's a typo and it's fixed now

I had some trouble with git yesterday and and need to retype some stuff and forgot to test that after

 

GroG

8 years 3 months ago

Ok .. I think I was never clear on 'reset' ...

Hardware reset - is power is cut - and it effectively reboots

I know Arduino can run a program without a serial connection for as long as it has power ... day .. months years ;)

BUT WHAT EXACTLY HAPPENS WHEN SERIAL IS DISCONNECTED AND RE-CONNECTED ???

If just setup and loop are 're-run' then no big deal ..

The real super important question is "Is the Heap re-initialized on serial re-connect"  ?? ...  

I never thought it was....

If (Heap reinitialized) .. Ya there is no point of trying to re-sync - the MrlComm enviorment has been completely destroyed...

If (Heap not reinitialized) - I dont see why we wouldn't want to try to preserve state - and resync on next connection

 

my 2 cents...

I think the Heap reinitialized. softReset() is now called from mrlComm constructor and it's been called when connecting. So it's not just a restart of setup().

What i'm looking for is if there is a way to prevent that reset to happen on connection.

And even if we can't prevent the heap to reinitialize, the MRLComm  environement can still be saved to EEPROM and rebuild after the reset

I couln't find any way to prevent the reset on connect on java. So it look that it will not be possible to reconnect without a reset.

I was exploring the possibility to save the arduino state to EEPROM and that look viable alternative. From the test I did I'm able to restart Status and heartbeat after a reset, but I have not try yet to save and recreate the devices and pins that way.

But I just realize that the Arduino have no handle on the Motor Control, or even on the pinMode. So without handle for that, it's won't be possible to put the arduino in a "safe" state when a deconnection happen. The only safe thing to do is to reset the arduino as Mats describe.

 

You all are way ahead of me :)

I finally did a test - although I think everyone here has done something similar ..

static long x;
 
void setup() {
  Serial.begin(9600);
}
 
void loop() {
  x = x + 1;
  Serial.print(x);
  Serial.print("\n");
}
 
When you connect the serial monitor you see the value increment, when you disconnect and re-connect it starts again at 0 :P - I have to admit I was surprised.
 
So all memory is re-initialized on a serial connection.
 
I guess it was a design choice made a long time ago, and short of modifying hardware you can't really disable this feature..
 
I would say serialization to EEPROM is not worth it - serialization can get buggy and messy and hard to support.
 
Ok .. So I agree with everyone :D
 
MrlDevices could implement an overloaded method to shut down safely and then all devices could be purged.  But this can only happen if the MrlComm side can detect a disconnect.  
 
So what would be the proposal of detecting disconnect on the MrlComm side ?   If you don't get the heartbeat in a certain amount of time ?
 
I'm beginning to feel, give the limitations of the serial connection and platform, that it may not be worth it ..
 

I think the proposal is that MLR should send a heartbeat message if no other messages are sent, at a heartbeat rate.

MRLComm should receive that heartbeat message within a timeout limit.

If  the limit is reached, MRLComm should loop thru all definded devices and execute a "shutdown" method and then do a hard reset. 

I also think that the setup() loop only needs to clear the serial buffer(s) and then wait for the first Magic number to occur.

When the first message occurs MRLComm should respond to this "first message" with a message to MRL to recreate the devices that were lost during the reset.

MRL then needs to resend the "first message" to the MRLComm, since several device creation messages may need to execute before MRLComm can exeute it. So much of the resyncronisation logic needs to be in the Arduino service.

/Mats

kwatters

8 years 3 months ago

Yup, very well documented behavior.  When you connect to the serial port, it will reset the arduino and run the setup method.

Some people have provided some hardware work arounds that include a few jumper wires so that the arduino doesn't know it was reset.

Mats,  I think you're exactly right about what needs to happen.  If MRLComm has lost connection to the java Arduino service, it needs to shutdown and detach all devices.  (in particular, motors, and servos..  otherwise that would be dangerous)

So, MRLComm needs to initiate the heartbeat and arduino-java needs to ack it back to mrlcomm.  If MRLComm doesn't get that ack, it needs to shutdown and remove all devices.

We should add "deviceDetach()" as one of the virtual methods in the device class.  Each device implementation needs to know how to un-attach.

MRLComm, in the event of a missed heartbeat would then iterate and detach all devices.

It would be much nicer if we could poll the Serial device to see if it's DTE / DTR toggle from high to low, but it would seem the Arduino doesn't really fully support all the control signals of the Serial port.

 

 

 

I guess this is a real design question:

who initiates the heartbeat.  MRLComm or Java-Arduino ?

As far as the duration of the heartbeat message,  I think something less than 1 second would be good, but this does beg the question..

If MRLComm initiates the heartbeat and Arduino-Java is doing a FullGC, Arduino Java might not be able to respond in time...

I seem to be going in circles on which way this heartbeat should go...

Also, I want to point out that if we add "detachDevice" as a virtual function on the device class, then  the whole conversation about "servo" attach vs "device" attach.. and  servo detach vs device detach actually become the same thing.  (So long as you provide the initial angle to the servo when you attach it...)

@Grog,  I know you have strong feelings about the difference of an "energized" servo vs. attaching the servo device.  But for me, the servo detach and device detach are converging to the same end behavior.

 

 

I think who initiate the heartbeat do no matters, as the heartbeat is only there to test the communication

I did code a first version of heartbeat and it's working

It's initiate by the MRL that will send a message every 800ms (that have to be test other timing). then MRL expect an message from MRLComm (the ack from the hearbeat message, or any message send by MRLComm will satisfy it. If it's more than 800ms without an answer, it disconnect the connection.

On MRLComm side, MRLComm expect to get at least 1 message each second, be the heartbeat message or any message. If it don't get it it start the reset procedure

So the heartbeat message is use only to insure that some communication is done inside a timeframe.

You can test it on the latest build with arduino.enabledHeartbeat() method

 

about servo_attach/device_attach. I like it the way it is right now as it allow MRLComm to remember some servo setting like speed, last position etc. so you can de-energize the servo and re-energize it while keeping the same setting. If we use device_attach to attach the servo, those info will be wipe out of MRLComm and need to be resent from MRL on each attach/detach

 

GroG

8 years 3 months ago

Is it worth it ?

I have to ask the question..  because with any new functionality there always is a cost.

There is the cost of more code.  More complexity, more maintenance, less memory, fewer resources...

I'm not saying I'm against it, I'm only saying these things need to be taken into consideration too.

For example - if the only effective way to check if the heartbeat has stopped, is with a seperate thread or setting a timer - this can potentially mess with Servos or anything else which is timer sensitive..

Again I'm not saying I'm against it, I'm just wondering if it cost is not worth the gain

 

 

the heartbeat can check on the MRLComm on the update() method using millis() as timing event. Not very precise, but it's good enough. and it use only 6 bytes of memory (a long and 2 bytes variable)

On MRL side, I use addTask and all is done inside a single method called by the task timer (and a variable setting in the processMessage(). So it's really quite simple stuff.

 

Ok sounds great as long as the timer does not modify any of the timer values for which servo and soft serial depend on ..

Or perhaps you are making a timer which simply checks the time diff between last heartbeat and is checked every loop