bugTransport Sample Protocol - Bugs: bug #12270, provider with huge number of...

 
 

bug #12270: provider with huge number of symbols implies high network load during request_infos

Submitted by:  Eric NOULARD <erk>
Submitted on:  Thu 10 Mar 2005 12:51:24 PM UTC  
 
Category: CoreSeverity: 4 - Important
Status: FixedPrivacy: Public
Assigned to: Eric NOULARD <erk>Open/Closed: Closed
Release: NoneFixed Release: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 23 Oct 2005 06:53:43 PM UTC, comment #3:

The tsp_request_filtered_infos just tested
and checked-in CVS.
Now a tsp consumer knowing what sample he wants
may issue

TSP_request_open
TSP_request_filtered_infos(...,'minimal',...)
TSP_request_sample

the 'minimal' version of TSP_request_filtered_infos
won't send any symbols list at all.

Note that the consumer should check answer_sample
for "unknown" symbols.

Eric NOULARD <erk>
Project AdministratorIn charge of this item.
Tue 04 Oct 2005 07:40:41 AM UTC, comment #2:

Adding a new tsp_request_filtered_informations which
won't return any symbols list if not ask.
Minimal informations will be:
- base frequency
- provider name
Minimal+ will be:
- Minimal +
- number of symbol
Other "filtered" request should be added when we are
all OK on how to implement it (xpath, SQL like...) :

see ML thread
http://lists.nongnu.org/archive/html/tsp-devel/2005-03/msg00034.html

Eric NOULARD <erk>
Project AdministratorIn charge of this item.
Tue 03 May 2005 08:40:00 PM UTC, comment #1:

I would like to add more info about this issue.
We discussed the possibility of using tree-like behaviour
for symbols browsing which is find for 'discovering symbols'
but which almost of no use for
validating symbols when receiving a tsp_sample_request
the nowadays implementation is doing a double loop
like:

for each symbol_requested
for each symbol_in_the_provider_list
strcmp()...

Then for each symbol requested we have to go through
a loop which is of the order of the number of symbols
the provider have.
If your favorite provider have 1 000 000 symbols
then you request_sample 100 of them, you'll
go for a
100 000 000 strcmp loop :))

I bet you will go for RPC timeout (or any asynchronous CMD
timeout) any time soon.

For huge provider symbols we need a efficient fast
lookup mecanism in order to validate sample_request
as fast as TSP light :))

We may either use tsp/src/util/libutil/hash.h from Bob
or have a look to in-memory DB like:
http://www.sqlite.org/
or berkeley DB (if it is possible to make in-memory DB?)
http://www.sleepycat.com/

Eric NOULARD <erk>
Project AdministratorIn charge of this item.
Thu 10 Mar 2005 12:51:24 PM UTC, original submission:

The tsp_consumer_request_info should not
automatically ask for ALL provider symbols since
it may implies high volume of data to go from
provider to consumer side even if the consumer only
wants to see a very small amount of symbols.

You may try with the bb_simu (create array of size 100000)
and bb_tsp_provider.

jsynoptic consumer will probably crash
and tsp_gdisp and tsp_gdisp+ may take several
minutes to start-up depending on the network
link efficiency between provider and consumer.

Eric NOULARD <erk>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

Do you think this task is very important?
If so, you can click here to add your encouragement to it.
This task has 0 encouragements so far.

Only logged-in users can vote.

 

Please enter the title of George Orwell's famous dystopian book (it's a date):

 

 

Follow 4 latest changes.

Date Changed By Updated Field Previous Value => Replaced By
Sun 23 Oct 2005 06:53:43 PM UTCerkStatusIn Progress=>Fixed
  Assigned toNone=>erk
  Open/ClosedOpen=>Closed
Tue 04 Oct 2005 07:40:40 AM UTCerkStatusNone=>In Progress

Back to the top


Powered by Savane 3.1-cleanup1