bugCERTI - Bugs: bug #34922, Zero-Lookahead Deadlocks

 
 

bug #34922: Zero-Lookahead Deadlocks

Submitter:  Steffen Strassburger <strassbu>
Submitted:  Mon 28 Nov 2011 10:42:17 AM UTC
   
 
Category:  CERTI Severity:  * 3 - Normal
Item Group:  Software error Status:  In Progress
Privacy:  Public Assigned to:  jb_ch
Open/Closed:  Open Release:  * 3.4.0
Reproducibility:  Every Time Fixed Release:  * Unknown
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 19 Dec 2017 12:05:16 PM UTC, comment #13: 

Taking over this one

JbCh <jb_ch>
Group administrator
Mon 18 Jun 2012 02:33:19 PM UTC, comment #12: 

OK Steffen,

I'll try to investigate the Michael problem and
come back to you in order to see if the two problem
may be linked or not.

Massimiliano,

Could elaborate a little more about your context?
Are you running on Windows as well?

I'm currently unable to reproduce the freeze on Linux...

Eric NOULARD <erk>
Group administrator
Mon 18 Jun 2012 09:22:34 AM UTC, comment #11: 

All, sorry that I did not follow up on this. I cannot tell if the suggested fix, i.e. using the "PRIME protocol" would help in my case.
Eric: I was not able to use the provided installers (wrong compiler version, I use Visual Studio 2005), and then I must admit that I simply lost track and did not have the time to follow up.

Steffen Strassburger <strassbu>
Wed 13 Jun 2012 09:30:26 AM UTC, comment #10: 

Hi guys,
any news on this topic? Apparently, I am experiencing the same issue, and switching to the PRIME protocol does not help...

Massimiliano D'Angelo <tuxwiller>
Mon 05 Dec 2011 01:32:06 PM UTC, comment #9: 


I just uploaded two CERTI Windows installers, produced by Martin
(Adelantado), those were built with MinGX on WinXP or Win7.

http://download.savannah.gnu.org/releases/certi/contrib/CERTI-3.4.2cvs-Win7-MinGW.exe
http://download.savannah.gnu.org/releases/certi/contrib/CERTI-3.4.2cvs-WinXP-MinGW.exe


Eric NOULARD <erk>
Group administrator
Mon 05 Dec 2011 09:10:11 AM UTC, comment #8: 

Ok then,

We will try provide you with a pre-compiled CERTI.
However providing binaries is painful because
we have to have several combination of systems & compilers.

We'd rather provide a "should-be-easy" procedure to compile
& install from source than providing binaries.
And this is more like the way open source project work.
That way we can interact more efficiently with our users.

You can give a try to current procedure:
http://www.nongnu.org/certi/certi_doc/Install/html/index.html
and we will help you if ever you encounter trouble.

Alternatively you may ask for help on the CERTI mailing list
if anyone has a similar platform & compiler setup as yours.
If a user provide us with a binary installer I may upload
it to CERTI download area for sharing it more easily with others.



Eric NOULARD <erk>
Group administrator
Mon 05 Dec 2011 08:28:43 AM UTC, comment #7: 

I agree that switching the default time management protocol to the PRIME protocol would probably be reasonable. I guess other RTIs use similar approaches, at least with the commercial pRTI that I also use time advancement even with zero lookahead "feels" to be very fast (they don't talk about their implementation, of course, as they are commercial).

About recompiling CERTI and testing the prime protocol - I currently do not have a visual studio project set up to compile CERTI. I am one of those users who use CERTI "out of the box". I can try to set it up, but it will take some time and I am away for a conference until the end of next week. I will give it a go then.

Steffen Strassburger <strassbu>
Fri 02 Dec 2011 12:03:46 PM UTC, comment #6: 

Hi Steffen,

Right for SLX I think you already told us that
I just forgot, sorry.

Thank you for the log files.
We are now able to reproduce the pb with our test federate.

I'll have to dive a little more into the time management
code (and reference article)
to understand whether if it's "normal" limitation
of the classical algorithm or an implementation mistake.
In fact it includes zero-lk handling but may be not for all
combination of TAR/TARA/NER/NERA.

In the meantime would you be able to recompile CERTI with
CERTI_USE_NULL_PRIME_MESSAGE_PROTOCOL set to ON.

It should definitely work with that option and we are thinking
of making it the default TM protocol so it would be nice to
have your feedback as well. Our local test are OK with this
new protocol.

Eric NOULARD <erk>
Group administrator
Tue 29 Nov 2011 03:25:08 PM UTC, comment #5: 

I have tried to get as much logging as possible and attaching it. The log for RTIA for Fed 1 contains some comments from me.

>Do you federates join the federation with a 0 lookahead or
>do they join with srictly positive lk and the modify_lookahead
>to 0?


The federates already join with zero lookahead.

I cannot easily create a small example code exhibiting the problem. Firstly, I am using SLX, a commercial discrete event simulator, which you would need, plus the SLX-HLA-Interface that I have developed is about 20000 lines of code. To reproduce the problem would mean starting a peace of code from scratch, which I unfortunately cannot do on short notice.

I hope the log files help to identify the problem.
Thank you for your support!

(file #24496, file #24497, file #24498)

Steffen Strassburger <strassbu>
Mon 28 Nov 2011 02:27:20 PM UTC, comment #4: 

Just to be clear.

> Fed1 - NERA(10), Fed2 - NERA(12) and have waited about 20 minutes and they still have no TAG


is not to be expected.
This may "freeze" for say a coule of 10th of second no more.
There must be a bug somewhere.

Eric NOULARD <erk>
Group administrator
Mon 28 Nov 2011 02:17:52 PM UTC, comment #3: 


> I have also read your paper. I am aware of the original CMB
> protocol and its time creep problem. I teach some of the
> basics of distributed simulation in one of my classes.


Ok I wasn't aware of that sorry.

> As I recall, the original CMB protocol does not work for zero
> lookahead, unless you implement some dead lock detection and
> avoidance scheme.


Yes that's right. When I speak about "classical CMB" it's the
one from CMB and Fujimoto's modification for zero-lookahead
handling which temporarily adds an epsilon value to the user provided lookahead in order to avoid the deadlock.
This is implemented in CERTI as well and this is necessary even
for TAR/TARA.
Our modification is different is computes a global maximum on rtig side in order to be able to make the best possible jump
into the future for all federate.

The printout you gave is from RTIG console.
You should have more information
in the RTIA/federate console (if you have some).
If you are running on Windows you may have to enable RTIA
console with "RTIA_CONSOLE_SHOW" cmake option.

Another way to get the trace of time management messages
is to set some environment var in RTIG console.
setting RTIG_MSG=D will trace every RTIA<->RTIG messages
and/or setting RTIG_NULLMSG=D will trace NULL message
related on the RTIG side.

You have similar var on RTIA side, RTIA_MSG and RTIA_NULLMSG.
On RTIA you also have RTIA_TM which trace even more information
about time management.

Do you federates join the federation with a 0 lookahead or
do they join with srictly positive lk and the modify_lookahead
to 0?

A fast try on my side with current CERTI CVS seems to
exhibit a problem if:
  - classical CMB is used (our modified CMB does not have the issue)
  and
  - federate join with non-zero lk and modify to zero
    before calling any TM service.

I won't have much time this week but may be you can attach
a small test code which exhibits the trouble and i'll try to look into it may be next week.




The RTIA trace looks like this:
RTIA: Statistics (processed messages)
List of federate initiated services
--------------------------------------------------
       1 Message::OPEN_CONNEXION (MSG#1)
       1 Message::CREATE_FEDERATION_EXECUTION (MSG#3)
       1 Message::JOIN_FEDERATION_EXECUTION (MSG#5)
       1 Message::PUBLISH_INTERACTION_CLASS (MSG#31)
       1 Message::SUBSCRIBE_INTERACTION_CLASS (MSG#35)
       1 Message::ENABLE_TIME_REGULATION (MSG#81)
       1 Message::ENABLE_TIME_CONSTRAINED (MSG#83)
       2 Message::QUERY_LBTS (MSG#85)
       1 Message::MODIFY_LOOKAHEAD (MSG#88)
       1 Message::QUERY_LOOKAHEAD (MSG#89)
       2 Message::NEXT_EVENT_REQUEST_AVAILABLE (MSG#95)
       2 Message::GET_INTERACTION_CLASS_HANDLE (MSG#117)
       6 Message::TICK_REQUEST (MSG#142)
       4 Message::TICK_REQUEST_NEXT (MSG#143)

List of RTI initiated services
--------------------------------------------------
       4 NetworkMessage::MESSAGE_NULL (MSG#2)
       3 NetworkMessage::SET_TIME_REGULATING (MSG#7)
       1 NetworkMessage::TIME_REGULATION_ENABLED (MSG#9)
       1 NetworkMessage::TIME_CONSTRAINED_ENABLED (MSG#10)

 Number of Federate messages : 25
 Number of RTIG messages : 9
 TCP Socket  3 : total =       489 Bytes sent
 TCP Socket  3 : total =       632 Bytes received
 UDP Socket  5 : total =         0 Bytes sent
 UDP Socket  5 : total =         0 Bytes received



Eric NOULARD <erk>
Group administrator
Mon 28 Nov 2011 01:22:55 PM UTC, comment #2: 

Eric,
I would say that they never receive the TAG, but how can I be sure? I have decreased the time gaps significantly (Fed1 - NERA(10), Fed2 - NERA(12) and have waited about 20 minutes and they still have no TAG. How do I determine the number of NULL messages - the printout in RTIG when I abort one federate is the following:

 TCP Socket 300 : total =      6450 Bytes sent
 TCP Socket 300 : total =       338 Bytes received
 UDP Socket 244 : total =         0 Bytes sent
 UDP Socket 244 : total =         0 Bytes received
 TCP Socket 288 : total =      6471 Bytes sent
 TCP Socket 288 : total =       303 Bytes received
 UDP Socket 244 : total =         0 Bytes sent
 UDP Socket 244 : total =         0 Bytes received

I have also read your paper. I am aware of the original CMB protocol and its time creep problem. I teach some of the basics of distributed simulation in one of my classes. As I recall, the original CMB protocol does not work for zero lookahead, unless you implement some dead lock detection and avoidance scheme.

Steffen Strassburger <strassbu>
Mon 28 Nov 2011 12:07:53 PM UTC, comment #1: 

Do they NEVER receive TAG or
does it take very long time before the first TAG?
When you abort the run how many NULL_MESSAGE count do you get?

With such "jump" in time and zero-lookahead if you
use the classical NULL message algorithm with only
event-driven (a.k.a. NERing) federate
you will face the time-creep problem.
See paragraphs ยง4.x of the paper:
http://download.savannah.gnu.org/releases/certi/papers/11S-SIW-045.pdf.

Knowing that, the time advance may take very long time but
after some time federate must receive TAG.
Very long time being the time to send/receive apprximately
4*150*1000 = 600 000 messages.


That said, with CERTI you can enable a conservative optimization of the NULL Message algorithm which should avoid the time-creep problem.
In order to use the new algorithm called (CERTI NULL PRIME Message) you should enable the "CERTI_USE_NULL_PRIME_MESSAGE_PROTOCOL" CMake option and recompile CERTI.




Eric NOULARD <erk>
Group administrator
Mon 28 Nov 2011 10:42:17 AM UTC, original submission:  

I am trying to use two zero lookahead federates, both time constrained, time regulating, both lookahead zero.

When I try to advance time using nextEventRequestAvailable in both federates, none will receive a time advance grant.

Example:
1) Both federates join, turn on time management and are both at time 0.000.
2) Federate 1 issues nextEventRequestAvailabe to 100.00
3) Federate 2 issues nextEventRequestAvailabe to 150.00

Both tick the RTI using tick(), but none of them receives a timeAdvanceGrant.

Kind regards,
Steffen Strassburger (strassbu@web.de)

Steffen Strassburger <strassbu>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #24496:  Log_RTIA_Fed1.txt added by strassbu (44KiB - text/plain)
file #24497:  Log_RTIA_Fed2.txt added by strassbu (72KiB - text/plain)
file #24498:  Log_RTIG.txt added by strassbu (19KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jb_ch (Posted a comment)
  • -email is unavailable- added by tuxwiller (Posted a comment)
  • -email is unavailable- added by erk (Posted a comment)
  • -email is unavailable- added by strassbu (Submitted the item)
  • -email is unavailable- added by strassbu
  •  

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-12-19 jb_ch Assigned toNone jb_ch
    2011-12-05 erk StatusNeed Info In Progress
    2011-11-29 strassbu Attached File- Added Log_RTIA_Fed1.txt, #24496
        Attached File- Added Log_RTIA_Fed2.txt, #24497
        Attached File- Added Log_RTIG.txt, #24498
    2011-11-28 erk StatusNone Need Info
    2011-11-28 strassbu Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code