bugTransport Sample Protocol - Bugs: bug #19455, Connecting a client to a server...

 
 

bug #19455: Connecting a client to a server running on localhost sometimes fails

Submitter:  Robert de Vries <rhdv>
Submitted:  Fri 30 Mar 2007 03:10:35 PM UTC
   
 
Category:  Core Severity:  3 - Normal
Status:  In Progress Privacy:  Public
Assigned to:  rhdv Open/Closed:  Open
Release:  * 0.8.2 Fixed Release:  * Unknown
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 01 Apr 2007 08:49:02 PM UTC, comment #12: 

As a short intro, I think I found the silver bullet
we need for fixing the bug without incompatibility problem.
RPCLib has a "get_myaddress" function which does
not use gethostXXX function and returns the appropriate @IP
even if ethX is not UP.
I've just checked-in
tsp/src/util/libutil/tsp_check_host_and_ip.c

You may update and rebuild, then call
tsp_check_host_and_ip
with ethX down and you should get 127.0.0.1 with get_myaddress
whereas gethostbyname gives you down @IP.

You may look at the code for this function (OpenSolaris) here:
http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/ucblib/librpcsoc/get_myaddress.c

Glibc should contains similar code.

I think we could replace the buggy provider code
which tried to guess @IP with get_myaddress.

A longer answers follows :))

> Are there any important reasons why we cannot change the protocol at this moment?


At least because the compatibility question was raised
at the end of the last Workshop :((

The more important reason I see is that the bug
is not "critical" for most users since they really
need to have effectively networked config.
I do not minimize the annoyance for a standalone system
but I think you will understand my point.

Moreover I think we should "break" compatibility for
a bunch of reasons and not for fixing a single bug
unless it is critical for all users.

I think we may break on-the-wire when we will
introduce new/unimplemented features that need it.

> Are there already major projects using it?


Depending on what is considered "major" I would say
between 1 and 4.

> Or are there users that cannot relink their applications?


I think that all of them can do it but with a recompilation.
The fact is before the forthcoming release tsp 0.8.3 the unix
default build was static, so most of users as to go
from relink to recompile.

I soon as most users are used to dynamically compiled
TSP you are perfectly right and I made just this answer
to the "compatibility question":
We may break compatibility but we will try it hard
to only request relinking which would be easy if
you use dynamically compiled TSP.
 

Eric NOULARD <erk>
Group administrator
Sun 01 Apr 2007 07:58:04 PM UTC, comment #11: 

I admit that there is an on-the-wire incompatibility. But a relink should fix the problem (or if you use shared libraries not even that).
Maybe you need to fix the java/python/whatever bindings a bit, but the changes should not be user visible.

Are there any important reasons why we cannot change the protocol at this moment? Are there already major projects using it? Or are there users that cannot relink their applications?

If this is so we can just remove the change at the server side and ignore the IP address at the client side.

On a related note:
I have the impression that the java etc. versions are somewhat decoupled from the C version. Is that so?
If so are there any automated tests to verify that the implementations in the different languages are fully compatible?

Robert de Vries <rhdv>
Group Member
Sun 01 Apr 2007 04:38:35 PM UTC, comment #10: 

Just a last minute thought.

I think we should make the fix backward compatible
with the 0.8.x branch and may be upgrade TSP protocol
version.

The fact is if provider now send data_address
as ":<port>" instead of "<@IP>:<port>" all
current TSP consumers won't be able to handle
the answer_sample_init...

Such a shame :))

As Cesare asked during the Workshop I think
it's not the time to break compatibility
for a bug fix :((

The fix is good but it impacts compatibility.
So now I see two ways for fixing tghe bug:

1) change TSP_PROTOCOL_VERSION and handle compatibility
   on provider-side, i.e. send <@IP>:<port> if
   TSP_PROTOCOL_VERSION is less or equal 0x00010001
   (which is current value)

2) find a way to guess the "right" address or hostname
   on provider-side in the buggy case.

What is your opinion.
(Fred, Yves, Cesare opinion are welcomed too :)))

Eric NOULARD <erk>
Group administrator
Sun 01 Apr 2007 01:31:49 PM UTC, comment #9: 

Your patch is functionnally OK.
Nevertheless, I've just made some modifications to your patch
and attached mine to the tracker.

It is in an attempt to:

1) Make the change more "local".
   It modifies 4 files instead of 7.

2) Avoid adding parameters to
   data_receiver and stream_sender
   because I think its not as generic as we
   should aim at.

The idea is to keep the data_address I/F we had
before but make tsp_consumer.c handle the
"default missing parts" in our case add
the missing "hostname" to "data_address"
before calling TSP_data_receiver_create.

I remove the <sys/param> include for
MAXHOSTNAMELEN and use
TSP_MAXHOSTNAMELEN, the less we depend on
system includes the less we got in portability trouble.

We may further use CMake in order
to link TSP_MAXHOSTNAMELEN to
MAXHOSTNAMELEN if it is defined.

Would you try my patch in order to see if it suits
your needs?

Tests are OK on my side even if I need to test
it on real-networked configuration I do not have
at home.

When we are OK with the patch I think
we will (you/or me) check in CVS as soon as
other TSP consumer implementation
(java, ruby at least) are ready to handle the new
TSP data channel URL scheme.

I assigned the patch to you since you began this,
and you are TSP developer :))


PS: my patch is a directory diff because
    Savannah CVS seems offline, should be OK though...
    I hope :))

(file #12365)

Eric NOULARD <erk>
Group administrator
Sat 31 Mar 2007 12:54:24 PM UTC, comment #8: 

I'll propose a patch.

Robert de Vries <rhdv>
Group Member
Sat 31 Mar 2007 12:28:53 PM UTC, comment #7: 

I'm OK with your point
and I will just do that unless you
want to do it and provide a patch?

Eric NOULARD <erk>
Group administrator
Sat 31 Mar 2007 12:24:48 PM UTC, comment #6: 

You may have a point about allowing different communication mechanisms for the control channel (RPC/XMLRPC) vs. the data channel. So I would only slightly change the current behaviour. If you do not want to use a special (possibly high-speed) channel for the data you send back ":port number". If on the other hand you wanted to you use a special high speed fancy link you would specify  an URL. e.g. TCP://hostname:port
or SHM://whatever/ for a special shared memory interconnect (e.g. reflective memory, scramnet)

But for now I would just send the port number back.

Robert de Vries <rhdv>
Group Member
Sat 31 Mar 2007 12:16:00 PM UTC, comment #5: 

Thanks a lot
for this lenghtly reply and fine analysis.

And please excuse me for my doubt about network config,
I had made tsp_check_host_and_ip because most TSP
connection problem were a consequence of buggy/weird config.

The data_address contained in
answer_sample_init (which is the reply to request_sample_init)
effectively includes the @IP + port.

This was done on purpose because we wanted
to separate
command channel handling (ONC-RPC, XML-RPC, etc...)
from
data channel handling (currently socket opening on "data_address"

It may seems weird but we can imagine that even if
you use "tsp_command.yourlan.nl" for command channel
(sending TSP_request_xxx and receiving TSP_answer_xxx)
you "MAY" perfectly get the data channel stream from
"tsp_data.yourlan.nl" if the answer_sample_init answers
this way. A first use case may be if your TSP provider
box has several network I/F and you may want to accept
command on eth0 and send sample on eth1.

This is not possible with current TSP implementation
but I would like to keep this option open.
In fact the data_address of TSP data channel has been
thought as a kind of "url" which may be extended for
other kind of data channel communication medium
(shared memory, messgae queue, SSL socket etc...)

However you are perfectly right, since we don't use those
"fancy possible features" today we should not
pay for what we don't use.

I will try to modify the code as you told,
only send the port. Then use the same "name" we
use for command channel in data channel. This creates
a link between the two channels but it seems
reasonnable for our current usage.

As a side-note we did a gethostname followed by gethostbyname
on provider side on provider side since we had no mean
to know the @IP address that was used by consumer side
to reach us.

Eric NOULARD <erk>
Group administrator
Sat 31 Mar 2007 11:48:20 AM UTC, comment #4: 

I have analyzed a bit further.
The problem is caused by a reply from the TSP server.
The client first connects to the RPC part of the server using the localhost address. This works fine until the client wants to create the data connection.
To initiate the data connection the client calls TSP_request_sample_init(). This replies with a structure containing  a string called data_address. This holds the IP address and port number. The IP address is the IP address derived from the hostname.
This is the root of the problem. First of all you don't need to send the IP address back to the client. The port number is enough. If you use the same hostname for creating the RPC connection as for the data connection everything would work fine all of the time.

I would use the information in server_info in TSP_otsp_t (but probably in a bit more easy to use form) to create the TCP connection.

Robert de Vries <rhdv>
Group Member
Sat 31 Mar 2007 11:13:43 AM UTC, comment #3: 

Simple procedure to repeat bug:
start tsp_stub_server
start tsp_stdout_client (different terminal)
All ok.
sudo ifdown eth0/eth1/whatever network i/f you have
start tsp_stdout_client (different terminal)
#=========================================================#
# Launching <stdout_client> for printing symbols received #
#=========================================================#
/opt/tsp/bin/tsp_stdout_client: Using provider URL <localhost>
/opt/tsp/bin/tsp_stdout_client: Asking for:
         <5> samples (0 => INFINITE loop)
      of <1> TSP symbols
      at period <1>.
/opt/tsp/bin/tsp_stdout_client: Asking for 1 symbols
   symbol <0> is <t>
  ERROR||/home/rhdv/src/tsp-0.8.2-build/RPM/BUILD/tsp-0.8.2-Source/src/core/driver/tsp_stream_receiver.c##TSP_stream_receiver_create##184: pb connecting to socket
  ERROR||/home/rhdv/src/tsp-0.8.2-build/RPM/BUILD/tsp-0.8.2-Source/src/core/driver/tsp_data_receiver.c##TSP_data_receiver_create##140: TSP_stream_receiver_create failed
  ERROR||/home/rhdv/src/tsp-0.8.2-build/RPM/BUILD/tsp-0.8.2-Source/src/core/driver/tsp_consumer.c##TSP_consumer_request_sample_init##1153: Unable to create data receiver
tsp_stdout_client: /home/rhdv/src/tsp-0.8.2-build/RPM/BUILD/tsp-0.8.2-Source/src/core/driver/tsp_consumer.c:1232: TSP_consumer_read_sample: Assertion `otsp->sample_fifo!=0' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread -1208440128 (LWP 4496)]
0x0012e402 in __kernel_vsyscall ()
(gdb) where
#0  0x0012e402 in __kernel_vsyscall ()
#1  0x484ecd40 in raise () from /lib/libc.so.6
#2  0x484ee591 in abort () from /lib/libc.so.6
#3  0x484e638b in __assert_fail () from /lib/libc.so.6
#4  0x08052129 in TSP_consumer_read_sample (provider=0x93bc990, sample=0xbfc84094, new_sample=0xbfc840ac)
    at /home/rhdv/src/tsp-0.8.2-build/RPM/BUILD/tsp-0.8.2-Source/src/core/driver/tsp_consumer.c:1232
#5  0x0804c79a in main (argc=1, argv=0x93ba018)
    at /home/rhdv/src/tsp-0.8.2-build/RPM/BUILD/tsp-0.8.2-Source/src/consumers/stdout/client_stdout.c:332

Robert de Vries <rhdv>
Group Member
Sat 31 Mar 2007 11:08:50 AM UTC, comment #2: 

Output of tsp_check_host_and_ip:
hostname is  <calvin.lan> (as reported by gethostname)
@IP returned is <10.0.0.153> <addrtype= AF_INET> (as reported by gethostbyname)
hostname returned is <calvin.lan> for @IP <10.0.0.153> <addrtype= AF_INET> (as reported by gethostbyaddr)

Contents of /etc/hosts:
127.0.0.1       localhost.localdomain   localhost
10.0.0.153      calvin.lan      calvin

The output of tsp_check_host_and_ip is the same whether or not eth1  (my wireless lan) is active or not.
However if it is not active, I experience the bug.

So as you can see, is my network configuration not buggy at all, it is a bug in TSP. That is also why I posted a bug report here and not a bug report with myself/my sysadmin.

Robert de Vries <rhdv>
Group Member
Fri 30 Mar 2007 04:10:21 PM UTC, comment #1: 

Robert,

Could you please

1) execute tsp_check_host_and_ip
   and post the result here
   (when you are in the buggy config state)

2) post the excerpt of your /etc/hosts file
   containing the definition
   @IP name alias for localhost
   @IP name alias for the "real" hostname
   (the name you get with /bin/hostname

Some Fedora/Redhat distribution have the
bad idea to associate @IP w.x.y.z with localhost
instead of 127.0.0.1.

So that we "gethostbyname(localhost)" to get @IP and get
w.x.y.z instead of 127.0.0.1, then since
w.x.y.z you get an error which leads to the core dump.

> Of course first of all the client should not dump core.


You are right the core dump is really bad and we
must fix it.

regarding:

> Second of all the client should not attempt to connect to
> anything other than localhost.


You may be right and we should be currently
doing just that (unless you use -u TSP_URL)
Nevertheless, in the end we call
clnt_create("localhost" etc...)
which may be got trapped into the wrong gethostbyname behavior.

I should check whether if the core dump occurs
in the call or outside the call.


Eric NOULARD <erk>
Group administrator
Fri 30 Mar 2007 03:10:35 PM UTC, original submission:  

When you connect a client to a server running on localhost, it fails with a core dump when the hostname of the host does not point to a valid IP address.
Of course first of all the client should not dump core. Second of all the client should not attempt to connect to anything other than localhost.
To repeat this, disconnect your computer from the network and bring down your eth0, eth1 or whatever connects you to the outside world. Then start up a provider and try to connect a consumer. Result instant crash.

Robert de Vries <rhdv>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #12365:  localhost-v2-erk.patch added by erk (8KiB - text/x-patch - More local patch for localhost)
file #12357:  localhost.patch added by rhdv (9KiB - text/x-patch - Patch to allow running on non-networked system using localhost only)

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by erk (Posted a comment)
  • -email is unavailable- added by rhdv (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2007-10-25 erk Dependencies- bugs #21424 is dependent
    2007-05-14 erk Dependencies- bugs #17035 is dependent
    2007-04-01 erk StatusConfirmed In Progress
    2007-04-01 erk Attached File- Added localhost-v2-erk.patch, #12365
        Assigned toerk rhdv
    2007-03-31 rhdv Attached File- Added localhost.patch, #12357
    2007-03-31 erk StatusNone Confirmed
        Assigned toNone erk

    Back to the top

    Powered by Savane 3.14-f13d.
    Corresponding source code