Hello,

Happy new year 2021 to you as well and thanks for your registration to Savannah and CERTI!

To answer your questions:

--
1) which is the effect of requestObjectAttributeValueUpdate?

The requestObjectAttributeValueUpdate() are used in special cases when it may be desirable to explicitly request an update (i.e. for object attributes) for the benefit of a newly subscribed federate. On our side, we are never using this service and a federate will discover object instances whenever instances are registered. On our side, all federates are using the following procedure.

-> Create/Join
-> Get Handles (Class/Object/Interaction/Attribute etc...)
-> Publish/Subscribe
-> Register Object Instance
-> Initial Sync Point
-> Run with UAV/RAV SI/RI with or without time management.

--
2) did you ever had some issues with this call using multiple federates (number grater than 4) that discover all together the objects in the federation?

No never!
As I have said we have tested the CERTI for bench-marking time management algorithms (C++) and we have never observed any issue (up to 200 federates, fully distributed or local)


--
3) is the discover phase an hard phase for the RTI? I assume CERTI is tested with way more than 5/6 federates.

No it is not, as I have said we tried with a lot of federates without any issues.


Thus I have two assumptions for your issue.

A) The requestObjectAttributeValueUpdate might be inconsistent as it is not fully tested on our side currently because we are not using it. Why would you need this function call?

B) What you are calling the "main Python program" is generating federates as thread and not as process which might be an issue.

Best regards,
Jean-Baptiste

Hello,

May I ask you to create a savannah account and log in before updating the ticket? Because for now you are posting under "Anonymous" and this doesn't make much sense.

You are saying:
"I am still performing some tests and I discovered that if I run in pure threads form the same Linux shell, I have this error (as per attached log)"

It is very difficult to guess what you meant here but I won't be able to help much without being able to reproduce the error. On our side, we are running Certi with multiple C++ federates (for 10 to more than 200 for benchmarks) and we have never observed the behavior that you are describing here.

However, we are not often using multi-threaded federates in C++. I understood that this was your case with Python and I don't know how Python is handling shared variables and how you are handling all the complexity of the multi-threaded python run-time execution in one federate. This is definitely something that needs to be analyzed properly. How many threads can call HLA services within one federate?

Can you share empty Python Federates (not containing semantics but only HLA calls) that can reproduce your issue?

Best regards,
jb-ch

hello Jean-Baptiste
I thank you again for your answer and I apologize for the latency in my reply.
I really understand the difficulty in understanding the issue and I hope that the log from valgrind where showing something "parlant" to you, but it seems that the bug we are facing is new to you.

We kept on investigating and we realized that the problem happens when we have > 3 federates (so starting from 4). Each federate has inside a variable number of python objects so it seems that processing the information takes less or more time for each federate. This gap may produce the irregularity to the CERTI framework deletes the memory.

The issue can be linked somewhere in between the threads of the federates, the python wrapper and the RTI framework.

We tried to add some time.sleep to slow down the simulation and the passage of information between the threads, but it didn't work.

I will contact again you as soon as I will be able to reproduce the error with a sharable code, and of course I will provide this code for your investigation.

Thanks again and have a nice day
Alberto

Hi Jean-Baptiste
I really thank you for your feedback and for investing time on this.
You can find my answers below

1) As an official "CERTI 4.0.0" has been released (no tarball and no tag done) you may be using a version from the master branch, can you give me the revision number.
I took the last commit from the master branch
83a4aeb9469eb27f0c3770889c5dd0232791587e
from this page
https://git.savannah.nongnu.org/cgit/certi.git/commit/?id=83a4aeb9469eb27f0c3770889c5dd0232791587e

2) Same thing on the PyHLA side, which version are you using, if this is from a git clone please give me the revision number, if this is from a tarball just let me know which was this?
we are using the latest version 1.1.1 take from here
http://mirror.netcologne.de/savannah/certi/contrib/PyHLA/

3) Have you done any modification in the PyHLA library?
No, we use the library as it is

4) Do all your federates are Python entities plugged via the same PyHLA version?
Yes, all our federates are in Python

5) What is the kernel version (uname -a) of your ubuntu and what is the compiler (gcc, clang which version?)
Kernel: Linux NOTE-ALFE-ID 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
gcc: gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
clang: not used
build CERTI as per the README file using cmake

6) Can you share the code to reproduce?
I am asking internally what we can do on that. I really appreciate your support and I understand the difficulty on investigating this error without the code.
I attached to the previous message some output errors coming from Valgrind. They shows the location of the memory leak, I don't know if they could help.
The idea of our code is the following (all code in Python):
- a main federate creates the federation and calls the others to plug
- each federates contain some other python objects, but monitoring the memory occupation there is nothing bad happening
- the main federate register all the synchronization points and often (but unfortunately not always) as all the federates reached the first synchronization point, the system crashes

The error on the free() pointer, seems to happen as some federate is too fast and free a pointer that should not. We are using CERTI since some time but as our simulation scenario are becoming bigger, we found out that.

To give you the size, we are currently having 8 federates, containing on average 8/9 internal python objects. But finally, the clients to CERTI are the 8 federates with their own ambassadors.


Let me know,

Best regards,
Jean-Baptiste

hello CERTI team
I am sorry for the double message, I wrongly clicked on "quote" button.

As suggested by Janette Cardoso, I am adding more details on the error we are facing.
As mentioned, I work for GOMSpace Simulation Department and we are currently using CERTI for our satellite simulations.

OS: Ubuntu 20.04 from WSL2
CERTI version: CERTI RTIG 4.0.0

We developed models in Python, biding to CERTI via the library PyHLA.

The issue seems to be on CERTI as it happens during the initialization of our models.
They go through standard synchronization points such as "readyToPopulate", "readyToRun", "readyToResign".

In between "readyToPopulate" and "readyToRun", the models register on the RTI.

Normally we have the messages:

==========================
RESPONSES FROM ROOT OBJECT
==========================

stating that the object registers in the RTI.

The memory error occurs sometimes right after the "populate" phase, sometimes after the first rtia.tick() in the "run" phase.

The errors I am facing are:
  - free(): invalid next size (fast)
  - malloc(): invalid next size (unsorted)
  - munmap_chunk(): invalid pointer
  - corrupted size vs. prev_size Aborted (core dumped)

Those errors arrives in different part of the run and they kill the RTI apparently randomly.
The more frequent error is "free(): invalid next size (fast)"
"malloc(): invalid next size (unsorted)" normally arrives at the end, before "readyToResign"

In attachment I send you some output messages from our run in Valgrind.

I hope it will help

thanks a lot in advantage for your kind support
Best regards
Alberto



(file #50376)

Hello team
First of all thanks for your work, the platform is great!
Unfortunately I am facing a memory error and I am stacked since some days on that
"munmap_chunk() invalid pointer"
or
"free(): invalid next size (fast)"

I am updating some attributes as string but I don't see the issue on that as sometimes it works and sometimes it doesn't

The error arrives after a synchronization point, if the program passes it, it works.

Did you have already something similar?
Thanks in advance
Alberto