From rogerpack2005 at gmail.com  Tue Oct 30 23:09:10 2007
From: rogerpack2005 at gmail.com (Roger Pack)
Date: Tue, 30 Oct 2007 20:09:10 -0700
Subject: [distribustream-talk] what algorithm?
Message-ID: <966599840710302009w5be8a237w9585630cbec0cd76@mail.gmail.com>

So my question is...based on what algorithm will distribustream work?
There are several competing ones, it seems.
Bullet (bad because of overhead)
some dht ones.
Hmm.
Maybe some application layer multicast ones.
Take care.
-Roger
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/distribustream-talk/attachments/20071030/085db698/attachment.html 

From tony at clickcaster.com  Wed Oct 31 00:03:04 2007
From: tony at clickcaster.com (Tony Arcieri)
Date: Tue, 30 Oct 2007 22:03:04 -0600
Subject: [distribustream-talk] what algorithm?
In-Reply-To: <966599840710302009w5be8a237w9585630cbec0cd76@mail.gmail.com>
References: <966599840710302009w5be8a237w9585630cbec0cd76@mail.gmail.com>
Message-ID: <c7e6b2b00710302103j716deb48jb3e56e91026562f6@mail.gmail.com>

On 10/30/07, Roger Pack <rogerpack2005 at gmail.com> wrote:
>
> So my question is...based on what algorithm will distribustream work?


DistribuStream presently uses a Bayesian trust model, established after
first connecting peers randomly then waiting for success/failure reports.
The algorithm could be expanded greatly to improve its ability to construct
robust P2P networks.

The server picks peers to connect for a given chunk transfer.  The main
constraints are:

1. Does the potential "giver" peer have chunks the "taker" peer is
requesting?
2. Are the peers in the same prefix? (at least the same /16, if not the same
/24)
3. Are the peers firewalled? (the server could do a connect test as soon as
a client registers their listen port)
4. What is the trust between peers? (the "random" connecting of untrusted
peers could be replaced by something that operates on the above two bits of
data)
5. Does the "giver" have enough bandwidth? (modeled over time)
6. Are total inbound and outbound links minimized? (to improve TCP
congestion)

-- 
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/distribustream-talk/attachments/20071030/1dcad619/attachment.html 

From rogerpack2005 at gmail.com  Wed Oct 31 09:38:32 2007
From: rogerpack2005 at gmail.com (Roger Pack)
Date: Wed, 31 Oct 2007 06:38:32 -0700
Subject: [distribustream-talk] what algorithm?
In-Reply-To: <c7e6b2b00710302103j716deb48jb3e56e91026562f6@mail.gmail.com>
References: <966599840710302009w5be8a237w9585630cbec0cd76@mail.gmail.com>
	<c7e6b2b00710302103j716deb48jb3e56e91026562f6@mail.gmail.com>
Message-ID: <966599840710310638x6394381fu70e24fa8b9efca9@mail.gmail.com>

How do the peer rendezvous?

On 10/30/07, Tony Arcieri <tony at clickcaster.com> wrote:
>
> On 10/30/07, Roger Pack <rogerpack2005 at gmail.com> wrote:
> >
> > So my question is...based on what algorithm will distribustream work?
>
>
> DistribuStream presently uses a Bayesian trust model, established after
> first connecting peers randomly then waiting for success/failure reports.
> The algorithm could be expanded greatly to improve its ability to construct
> robust P2P networks.
>
> The server picks peers to connect for a given chunk transfer.  The main
> constraints are:
>
> 1. Does the potential "giver" peer have chunks the "taker" peer is
> requesting?
> 2. Are the peers in the same prefix? (at least the same /16, if not the
> same /24)
> 3. Are the peers firewalled? (the server could do a connect test as soon
> as a client registers their listen port)
> 4. What is the trust between peers? (the "random" connecting of untrusted
> peers could be replaced by something that operates on the above two bits of
> data)
> 5. Does the "giver" have enough bandwidth? (modeled over time)
> 6. Are total inbound and outbound links minimized? (to improve TCP
> congestion)
>
> --
> Tony Arcieri
> ClickCaster, Inc.
> tony at clickcaster.com




-- 
-Roger Pack
I like belief. http://www.google.com/search?q=free+bible
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/distribustream-talk/attachments/20071031/b51cdad4/attachment.html 

From tony at clickcaster.com  Wed Oct 31 14:10:45 2007
From: tony at clickcaster.com (Tony Arcieri)
Date: Wed, 31 Oct 2007 12:10:45 -0600
Subject: [distribustream-talk] what algorithm?
In-Reply-To: <966599840710310638x6394381fu70e24fa8b9efca9@mail.gmail.com>
References: <966599840710302009w5be8a237w9585630cbec0cd76@mail.gmail.com>
	<c7e6b2b00710302103j716deb48jb3e56e91026562f6@mail.gmail.com>
	<966599840710310638x6394381fu70e24fa8b9efca9@mail.gmail.com>
Message-ID: <c7e6b2b00710311110l3f2278cs37403b1e36459e46@mail.gmail.com>

On 10/31/07, Roger Pack <rogerpack2005 at gmail.com> wrote:
>
> How do the peer rendezvous?


The server explicitly instructs peers to connect to each other and transfer
particular chunks of the file

-- 
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/distribustream-talk/attachments/20071031/1eb862a7/attachment.html 

From rogerpack2005 at gmail.com  Wed Oct 31 14:26:00 2007
From: rogerpack2005 at gmail.com (Roger Pack)
Date: Wed, 31 Oct 2007 11:26:00 -0700
Subject: [distribustream-talk] what algorithm?
In-Reply-To: <c7e6b2b00710311110l3f2278cs37403b1e36459e46@mail.gmail.com>
References: <966599840710302009w5be8a237w9585630cbec0cd76@mail.gmail.com>
	<c7e6b2b00710302103j716deb48jb3e56e91026562f6@mail.gmail.com>
	<966599840710310638x6394381fu70e24fa8b9efca9@mail.gmail.com>
	<c7e6b2b00710311110l3f2278cs37403b1e36459e46@mail.gmail.com>
Message-ID: <966599840710311126p54455e8dl46730da0746e1fdc@mail.gmail.com>

the comments I'd have are
1) doesn't this create a bottleneck at the server? Don't you want peers to
be able to self organize so it can scale more easily?
2) It is nice for these protocols to 'just' use HTTP instead of creating and
using a whole new transport protocol.  Then it's nicer on everyone :)

Cheers for now.
-Roger

On 10/31/07, Tony Arcieri <tony at clickcaster.com> wrote:
>
> On 10/31/07, Roger Pack <rogerpack2005 at gmail.com> wrote:
> >
> > How do the peer rendezvous?
>
>
> The server explicitly instructs peers to connect to each other and
> transfer particular chunks of the file
>
> --
> Tony Arcieri
> ClickCaster, Inc.
> tony at clickcaster.com
>



-- 
-Roger Pack
I like belief. http://www.google.com/search?q=free+bible
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/distribustream-talk/attachments/20071031/577204da/attachment.html 

From tony at clickcaster.com  Wed Oct 31 14:58:08 2007
From: tony at clickcaster.com (Tony Arcieri)
Date: Wed, 31 Oct 2007 12:58:08 -0600
Subject: [distribustream-talk] what algorithm?
In-Reply-To: <966599840710311126p54455e8dl46730da0746e1fdc@mail.gmail.com>
References: <966599840710302009w5be8a237w9585630cbec0cd76@mail.gmail.com>
	<c7e6b2b00710302103j716deb48jb3e56e91026562f6@mail.gmail.com>
	<966599840710310638x6394381fu70e24fa8b9efca9@mail.gmail.com>
	<c7e6b2b00710311110l3f2278cs37403b1e36459e46@mail.gmail.com>
	<966599840710311126p54455e8dl46730da0746e1fdc@mail.gmail.com>
Message-ID: <c7e6b2b00710311158m1a02980bwe566a6d87dd4b019@mail.gmail.com>

On 10/31/07, Roger Pack <rogerpack2005 at gmail.com> wrote:
>
> the comments I'd have are
> 1) doesn't this create a bottleneck at the server? Don't you want peers to
> be able to self organize so it can scale more easily?


Self-organization can work quite well, particularly in trusted
environments.  Link State Routing is a great example.  Thus far it's worked
fairly well for P2P protocols in untrusted environments as well.

DistribuStream uses a server-managed network for several reasons.  The first
is to dramatically simplify the client-side logic, making clients
lightweight and easy to implement.

This also affords algorithmic improvements implemented on the server side
only without changes to the clients.  All client/server and peer-to-peer
communication can be modeled in the form of state machines that clients must
obey if they wish for transfers to continue.  Since the clients are just
state machines, algorithmic improvements can also be built with expectations
of clients functioning in exactly the same way, preventing such improvements
from being mitigated by differences in client implementations.

In terms of this congesting the server, it's a potential problem.  The
client/server protocol uses lightweight JSON asynchronous messaging across a
persistent connection, so message processing is as simple.

DistribuStream is built on the Ruby/EventMachine library.  On Linux, this
library uses the epoll(4) system call for connection multiplexing, allowing
it to scale to thousands of concurrent connections.

2) It is nice for these protocols to 'just' use HTTP instead of creating and
> using a whole new transport protocol.  Then it's nicer on everyone :)


DistribuStream is built around existing standards as much as possible.
Using HTTP makes it effectively function as an ad hoc HTTP caching proxy
network.

-- 
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/distribustream-talk/attachments/20071031/64f45e38/attachment.html