bugCERTI - Bugs: bug #59529, memory error as freed pointer

 
 

bug #59529: memory error as freed pointer

Submitted by:  None
Submitted on:  Tue 24 Nov 2020 08:53:44 AM UTC  
 
Category:  PyHLA Severity:  5 - Blocker
Item Group:  Non-software error Status:  In Progress
Privacy:  Public Assigned to:  JbCh <jb_ch>
Open/Closed:  Open Release:  GIT-master
Reproducibility:  Intermittent Fixed Release:  GIT-master

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Mon 04 Jan 2021 11:05:29 AM UTC, comment #11: 

Hello,

Just an update from what I investigated a little bit on the requestObjectAttributeValueUpdate:

The method requestObjectAttributeValueUpdate must be used in conjunction with requestClassAttributeValueUpdate() and it realizes the intent of the "Request Attribute Value Update" Object Management service as specified the HLA Interface Specification (§4.14 in version 1.1; §6.15 in version 1.3).

For the each subscriber using this "requestAttributeValueUpdate" service the publisher will receive an "provideAttributeValueUpdate" callback and then the publisher shall call "UpdateAttributeValue" service.

This is the developer responsibility to implement this request/provide handling properly the services/callbacks between the federates.

Best regards,
Jean-Baptiste

JbCh <jb_ch>
Project AdministratorIn charge of this item.
Mon 04 Jan 2021 10:29:13 AM UTC, comment #10: 

Hello,

Happy new year 2021 to you as well and thanks for your registration to Savannah and CERTI!

To answer your questions:

--
1) which is the effect of requestObjectAttributeValueUpdate?

The requestObjectAttributeValueUpdate() are used in special cases when it may be desirable to explicitly request an update (i.e. for object attributes) for the benefit of a newly subscribed federate. On our side, we are never using this service and a federate will discover object instances whenever instances are registered. On our side, all federates are using the following procedure.

-> Create/Join
-> Get Handles (Class/Object/Interaction/Attribute etc...)
-> Publish/Subscribe
-> Register Object Instance
-> Initial Sync Point
-> Run with UAV/RAV SI/RI with or without time management.

--
2) did you ever had some issues with this call using multiple federates (number grater than 4) that discover all together the objects in the federation?

No never!
As I have said we have tested the CERTI for bench-marking time management algorithms (C++) and we have never observed any issue (up to 200 federates, fully distributed or local)

--
3) is the discover phase an hard phase for the RTI? I assume CERTI is tested with way more than 5/6 federates.

No it is not, as I have said we tried with a lot of federates without any issues.

Thus I have two assumptions for your issue.

A) The requestObjectAttributeValueUpdate might be inconsistent as it is not fully tested on our side currently because we are not using it. Why would you need this function call?

B) What you are calling the "main Python program" is generating federates as thread and not as process which might be an issue.

Best regards,
Jean-Baptiste

JbCh <jb_ch>
Project AdministratorIn charge of this item.
Wed 30 Dec 2020 08:28:10 AM UTC, comment #9: 

Hi Jean-Baptiste
I hope my message will find you well after this Christmas period.

I write to you as I have some good news about the memory issue.
I try to describe the architecture of our implementation and my discovery.

As mentioned, all the federates are written in Python and they plug to RTIg using PyHLA library.

1) The main Python program launches the RTIg in a separate sub-process.

2) The main Python program creates the federate A.
   a) Federate A creates the FOM file
   b) Federate A creates RTI ambassador
   c) Federate A creates its own ambassador and joins the  federation
      i) getObjectClassHandle (get object Federate A from FOM)
      ii) getAttributeHandle  (get attributes of Federate A)
      iii) publishObjectClass (publish object-attributes)
      iv) registerObjectInstance

3) The main Python program creates the federate B.
   b) Federate B creates RTI ambassador
   c) Federate B creates its own ambassador and joins the  federation
      i) getObjectClassHandle (get object Federate B from FOM)
      ii) getAttributeHandle  (get attributes of Federate B)
      iii) publishObjectClass (publish object-attributes)
      iv) registerObjectInstance
   d) Federate B creates link to Federates A attributes
      i) getObjectClassHandle (get object Federate A)
      ii) getAttributeHandle  (get interesting attributes of Federate A)
      iii) subscribeObjectClassAttributes (subscribe object-attributes)

[... this process applies to many federates]

Reached the synchronization point of "_readyToRun", all federates have the first timeAdvanceRequest.

As they request to advance time, automatically from the RTI the federates receive information about the object they subscribed to with the discovery.

   4) all Federates discover the subscribed objects
      i) discoverObjectInstance (receive object and attributes)
      ii) requestObjectAttributeValueUpdate

Now every time an attribute is updated via the updateAttributeValues, RTI propagates the reflect callback

   5) all Federates reflect the updated values
      i) reflectAttributeValues

This process is quite straight forward and created following basically the example found online.

I realized that the issue is caused by the 4.ii process, so the requestObjectAttributeValueUpdate after the discover.
In this phase, the RTI is propagating all the objects to all the federates. I have the impression that as requestObjectAttributeValueUpdate  is called, some memory is already released and we have errors such as:

free(): invalid next size (fast)

So I removed the 4.ii: requestObjectAttributeValueUpdate.

I am able in any case to reflect the attribute updates. Our simulations and our results are not affected by the removal of requestObjectAttributeValueUpdate call after the discover.

Here my questions.

1) which is the effect of requestObjectAttributeValueUpdate?
2) did you ever had some issues with this call using multiple federates (number grater than 4) that discover all together the objects in the federation?
3) is the discover phase an hard phase for the RTI? I assume CERTI is tested with way more than 5/6 federates.

Thanks as always for your kind support.

Waiting to hear from you I wish you a happy new year!
Let's hope in this 2021
Alberto

Alberto Ferrero <alfe>
Thu 17 Dec 2020 07:47:16 PM UTC, comment #8: 

Hello,

May I ask you to create a savannah account and log in before updating the ticket? Because for now you are posting under "Anonymous" and this doesn't make much sense.

You are saying:
"I am still performing some tests and I discovered that if I run in pure threads form the same Linux shell, I have this error (as per attached log)"

It is very difficult to guess what you meant here but I won't be able to help much without being able to reproduce the error. On our side, we are running Certi with multiple C++ federates (for 10 to more than 200 for benchmarks) and we have never observed the behavior that you are describing here.

However, we are not often using multi-threaded federates in C++. I understood that this was your case with Python and I don't know how Python is handling shared variables and how you are handling all the complexity of the multi-threaded python run-time execution in one federate. This is definitely something that needs to be analyzed properly. How many threads can call HLA services within one federate?

Can you share empty Python Federates (not containing semantics but only HLA calls) that can reproduce your issue?

Best regards,
jb-ch

JbCh <jb_ch>
Project AdministratorIn charge of this item.
Thu 17 Dec 2020 05:12:58 PM UTC, comment #7: 

Hello,
I am still performing some tests and I discovered that if I run in pure threads form the same Linux shell, I have this error (as per attached log)
Is it something that speaks to you?
Thanks in advance
Alberto

(file #50505)

Anonymous
Fri 11 Dec 2020 08:32:50 AM UTC, comment #6: 

hello Jean-Baptiste
I thank you again for your answer and I apologize for the latency in my reply.
I really understand the difficulty in understanding the issue and I hope that the log from valgrind where showing something "parlant" to you, but it seems that the bug we are facing is new to you.

We kept on investigating and we realized that the problem happens when we have > 3 federates (so starting from 4). Each federate has inside a variable number of python objects so it seems that processing the information takes less or more time for each federate. This gap may produce the irregularity to the CERTI framework deletes the memory.

The issue can be linked somewhere in between the threads of the federates, the python wrapper and the RTI framework.

We tried to add some time.sleep to slow down the simulation and the passage of information between the threads, but it didn't work.

I will contact again you as soon as I will be able to reproduce the error with a sharable code, and of course I will provide this code for your investigation.

Thanks again and have a nice day
Alberto

Anonymous
Fri 04 Dec 2020 03:32:22 PM UTC, comment #5: 

Hello,

Thanks for the input, i have checked the version of pyhla that you are using compared to the git one prepared for CERTI 4.0.0:
https://git.savannah.nongnu.org/cgit/certi/applications.git/log/?h=br_jbch_4.0.0

But there is no major difference that can explain your problem.

I won't be able to help a lot without source code to reproduce the issue. We are using CERTI mainly with C++ code and we haven't observed such behavior.

Other questions:
1) Which version of Python are you using?

2) What are the option you are using with CERTI for compilation?

3) Are you running the entire simulation (CERTI+Federation) on the same host (i.e. localhost)

4) Have you been able to test it on another linux machine with different install (different kernel, different gcc, different python)?

Best regards,
Jean-Baptiste

JbCh <jb_ch>
Project AdministratorIn charge of this item.
Thu 03 Dec 2020 01:12:26 PM UTC, comment #4: 

Hi Jean-Baptiste
I really thank you for your feedback and for investing time on this.
You can find my answers below

1) As an official "CERTI 4.0.0" has been released (no tarball and no tag done) you may be using a version from the master branch, can you give me the revision number.
I took the last commit from the master branch
83a4aeb9469eb27f0c3770889c5dd0232791587e
from this page
https://git.savannah.nongnu.org/cgit/certi.git/commit/?id=83a4aeb9469eb27f0c3770889c5dd0232791587e

2) Same thing on the PyHLA side, which version are you using, if this is from a git clone please give me the revision number, if this is from a tarball just let me know which was this?
we are using the latest version 1.1.1 take from here
http://mirror.netcologne.de/savannah/certi/contrib/PyHLA/

3) Have you done any modification in the PyHLA library?
No, we use the library as it is

4) Do all your federates are Python entities plugged via the same PyHLA version?
Yes, all our federates are in Python

5) What is the kernel version (uname -a) of your ubuntu and what is the compiler (gcc, clang which version?)
Kernel: Linux NOTE-ALFE-ID 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
gcc: gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
clang: not used
build CERTI as per the README file using cmake

6) Can you share the code to reproduce?
I am asking internally what we can do on that. I really appreciate your support and I understand the difficulty on investigating this error without the code.
I attached to the previous message some output errors coming from Valgrind. They shows the location of the memory leak, I don't know if they could help.
The idea of our code is the following (all code in Python):
- a main federate creates the federation and calls the others to plug
- each federates contain some other python objects, but monitoring the memory occupation there is nothing bad happening
- the main federate register all the synchronization points and often (but unfortunately not always) as all the federates reached the first synchronization point, the system crashes

The error on the free() pointer, seems to happen as some federate is too fast and free a pointer that should not. We are using CERTI since some time but as our simulation scenario are becoming bigger, we found out that.

To give you the size, we are currently having 8 federates, containing on average 8/9 internal python objects. But finally, the clients to CERTI are the 8 federates with their own ambassadors.

Let me know,

Best regards,
Jean-Baptiste

Anonymous
Wed 02 Dec 2020 05:13:51 PM UTC, comment #3: 

Hello,

I can't help without further details here.

1) As an official "CERTI 4.0.0" has been released (no tarball and no tag done) you may be using a version from the master branch, can you give me the revision number.

2) Same thing on the PyHLA side, which version are you using, if this is from a git clone please give me the revision number, if this is from a tarball just let me know which was this?

3) Have you done any modification in the PyHLA library?

4) Do all your federates are Python entities plugged via the same PyHLA version?

5) What is the kernel version (uname -a) of your ubuntu and what is the compiler (gcc, clang which version?)

6) Can you share the code to reproduce?

Let me know,

Best regards,
Jean-Baptiste

JbCh <jb_ch>
Project AdministratorIn charge of this item.
Wed 02 Dec 2020 02:12:38 PM UTC, comment #2: 

hello CERTI team
I am sorry for the double message, I wrongly clicked on "quote" button.

As suggested by Janette Cardoso, I am adding more details on the error we are facing.
As mentioned, I work for GOMSpace Simulation Department and we are currently using CERTI for our satellite simulations.

OS: Ubuntu 20.04 from WSL2
CERTI version: CERTI RTIG 4.0.0

We developed models in Python, biding to CERTI via the library PyHLA.

The issue seems to be on CERTI as it happens during the initialization of our models.
They go through standard synchronization points such as "readyToPopulate", "readyToRun", "readyToResign".

In between "readyToPopulate" and "readyToRun", the models register on the RTI.

Normally we have the messages:

==========================
RESPONSES FROM ROOT OBJECT
==========================

stating that the object registers in the RTI.

The memory error occurs sometimes right after the "populate" phase, sometimes after the first rtia.tick() in the "run" phase.

The errors I am facing are:
  - free(): invalid next size (fast)
  - malloc(): invalid next size (unsorted)
  - munmap_chunk(): invalid pointer
  - corrupted size vs. prev_size Aborted (core dumped)

Those errors arrives in different part of the run and they kill the RTI apparently randomly.
The more frequent error is "free(): invalid next size (fast)"
"malloc(): invalid next size (unsorted)" normally arrives at the end, before "readyToResign"

In attachment I send you some output messages from our run in Valgrind.

I hope it will help

thanks a lot in advantage for your kind support
Best regards
Alberto

(file #50376)

Anonymous
Wed 02 Dec 2020 02:00:38 PM UTC, comment #1: 

original submission:

> Hello team
> First of all thanks for your work, the platform is great!
> Unfortunately I am facing a memory error and I am stacked since some days on that
> "munmap_chunk() invalid pointer"
> or
> "free(): invalid next size (fast)"
>
> I am updating some attributes as string but I don't see the issue on that as sometimes it works and sometimes it doesn't
>
> The error arrives after a synchronization point, if the program passes it, it works.
>
> Did you have already something similar?
> Thanks in advance
> Alberto
>

Anonymous
Tue 24 Nov 2020 08:53:44 AM UTC, original submission:  

Hello team
First of all thanks for your work, the platform is great!
Unfortunately I am facing a memory error and I am stacked since some days on that
"munmap_chunk() invalid pointer"
or
"free(): invalid next size (fast)"

I am updating some attributes as string but I don't see the issue on that as sometimes it works and sometimes it doesn't

The error arrives after a synchronization point, if the program passes it, it works.

Did you have already something similar?
Thanks in advance
Alberto

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50505:  RTI.log added by None (2KiB - application/octet-stream)
file #50376:  CERTI_valgrind_errors.txt added by None (7KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by edgo
  • -email is unavailable- added by alfe (Posted a comment)
  • -email is unavailable- added by jb_ch (Posted a comment)
  •  

     

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-01-04 jb_ch StatusNone => In Progress
    2021-01-04 edgo Carbon-Copy- => Added edgo@gomspace.com
    2020-12-17 None Attached File- => Added RTI.log, #50505
    2020-12-02 jb_ch Assigned toNone => jb_ch
    2020-12-02 None Attached File- => Added CERTI_valgrind_errors.txt, #50376

    Back to the top


    Powered by Savane 3.6