From ibc at aliax.net  Tue Jan 20 18:46:26 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Wed, 21 Jan 2009 00:46:26 +0100
Subject: [Revactor-talk] Initial look to Revactor and some doubts
Message-ID: <200901210046.26358.ibc@aliax.net>

Hi, I'm interested in Revactor to build a SIP server. I'm right now started 
learning about EventMachine and Revactor, not sure which one would be more 
appropiate for my project.

For now I've started my SIP Server from scratch but I expect to migrate it to 
an efficient concurrent model.

SIP is a very complex protocol. For example: if I use a SIP proxy in front of 
my server then all the data will arrive to my server using the same TCP 
connection. An example of SIP request would 
be:

-------------------
MESSAGE sip:bob at domain.org SIP/2.0<CRLF>
From: sip:alice at domain.org;tag=qweqweqwe<CRLF>
To: sip:bob at domain.org<CRLF>
Via: xxxxxxxxx<CRLF>
Cseq: xxxxxxxxx<CRLF>
Content-Type: text/plain
Content-Length: 5
<CRLF>
hello
-------------------

As you can see, the parser must read the "Content-Length" header and read that 
ammount of bytes after the empty line (CRLF) which separes the header from 
the body. After that ammount of bytes, the next data is supposed to be a new 
SIP message.

Since a single TCP connection would be use (in my particular case) it means 
that two requests (or more) could be groupped (a MESSAGE and an INVITE):

-------------------
MESSAGE sip:bob at domain.org SIP/2.0<CRLF>
From: sip:alice at domain.org;tag=qweqweqwe<CRLF>
To: sip:bob at domain.org<CRLF>
Via: xxxxxxxxx<CRLF>
Cseq: xxxxxxxxx<CRLF>
Content-Type: text/plain<CRLF>
Content-Length: 5<CRLF>
<CRLF>
helloINVITE sip:carol at domain.org SIP/2.0<CRLF>
From: sip:pepe at domain.org;tag=qweqweqwsdsde<CRLF>
To: sip:carol at domain.org<CRLF>
Via: xxxxxxxxx<CRLF>
Cseq: xxxxxxxxx<CRLF>
Content-Length: 0<CRLF>
<CRLF>
-------------------

How could I use Revactor :: Filter class to achive it?
I've also read that Revactor :: Filter :: Line delimiter just allows one 
single char as separator, but SIP requires lines ending with CRLF. Would it 
be an issue?

Thanks a lot for any help.


-- 
I?aki Baz Castillo

From tony at medioh.com  Tue Jan 20 19:14:24 2009
From: tony at medioh.com (Tony Arcieri)
Date: Tue, 20 Jan 2009 17:14:24 -0700
Subject: [Revactor-talk] Initial look to Revactor and some doubts
In-Reply-To: <200901210046.26358.ibc@aliax.net>
References: <200901210046.26358.ibc@aliax.net>
Message-ID: <c7e6b2b00901201614v2fff4c21i44c06094bac8b8d5@mail.gmail.com>

On Tue, Jan 20, 2009 at 4:46 PM, I?aki Baz Castillo <ibc at aliax.net> wrote:

> Hi, I'm interested in Revactor to build a SIP server. I'm right now started
> learning about EventMachine and Revactor, not sure which one would be more
> appropiate for my project.
>

EventMachine is probably your best bet.  I am not actively maintaining
Revactor, and worse it's tied to Ruby 1.9 due to its use of Fibers.  1.9
continues its glacial development pace (we got a 1.9.1 RC for Christmas
rather than a final, more "stable" version) and due to its incompatibilities
and bugs it has not seen widespread adoption.

EventMachine is maintaned, fast, and works on Ruby 1.8.  The internals and
API are quite ugly, but when you get past that it's probably the best tool
for the job.


> For now I've started my SIP Server from scratch but I expect to migrate it
> to
> an efficient concurrent model.
>

Well, you are going to find yourself in asynchronous callback hell with
something like EventMachine trying to write a SIP server.  I'm curious why
you picked Ruby to write a SIP server in.

Are you familiar with Adhearsion?  It provides an extremely rich set of Ruby
bindings for Asterisk, an already-robust SIP server:

http://docs.adhearsion.com/display/adhearsion/Getting+Started


> SIP is a very complex protocol. For example: if I use a SIP proxy in front
> of
> my server then all the data will arrive to my server using the same TCP
> connection. An example of SIP request would
> be:
>
> -------------------
> MESSAGE sip:bob at domain.org <sip%3Abob at domain.org> SIP/2.0<CRLF>
> From: sip:alice at domain.org <sip%3Aalice at domain.org>;tag=qweqweqwe<CRLF>
> To: sip:bob at domain.org <sip%3Abob at domain.org><CRLF>
> Via: xxxxxxxxx<CRLF>
> Cseq: xxxxxxxxx<CRLF>
> Content-Type: text/plain
> Content-Length: 5
> <CRLF>
> hello
> -------------------
>
> As you can see, the parser must read the "Content-Length" header and read
> that
> ammount of bytes after the empty line (CRLF) which separes the header from
> the body. After that ammount of bytes, the next data is supposed to be a
> new
> SIP message.


Revactor really isn't going to help you here, although it could help you in
the higher level structure of your application.

My ideal advice for how to tackle this problem is the "Mongrel approach":
find the ABNF grammar for SIP headers and convert it into a Ragel
description:

http://www.complang.org/ragel/

...and building a small Ruby C extension which wraps the Ragel parser.  This
is a technique Zed Shaw pioneered (at least in the Ruby world) which works
extremely well.   Take a look at Mongrel as an example or ask on the Mongrel
mailing list if you'd like to pursue this approach.  The parser will be very
fast and guaranteed correct.  If anything, you will have to relax the
grammar to interoperate with broken SIP implementations.

Once you've done this, you feed any data you receive into the parser.  The
parser will build a structure (like a hash) containing the elements in the
header, and tell you when it has finished and what data has left over.  You
just spoonfeed all data to the parser and it's smart enough to know when
it's consumed the entire header and leaves you with the leftover data,
namely the message body (and potentially part of the next header).

So in the example you gave, the parser would magically consume everything up
to the last <CRLF>, build a structure representing the header (e.g. {:type
=> "MESSAGE", :protocol => "SIP/2.0", :from =>
"sip:alice at domain.org<sip%3Aalice at domain.org>;tag=qweqweqwe",
:content-length => 5, ...}, and leave you with a string containing the rest
of the message.

Since a single TCP connection would be use (in my particular case) it means
> that two requests (or more) could be groupped (a MESSAGE and an INVITE):
>

I saw your post on the EventMachine mailing list as well, and there I
suggested you use a simple, informal state machine to tackle the rest of the
message processing.  As an example, this code implements an HTTP client
using a Ragel-generated parser and uses a simple informal state machine to
control the message processing (so it can be asynchronous underneath and
work on things like EventMachine):

http://github.com/tarcieri/fasthttp/blob/c6d3cac049e81b2cc16e0379a0f5dff433f22994/lib/fasthttp.rb

Basically, your state machine would start out in the :header state, and
would use the Ragel parser to suck in the header and determine when it's
finished reading the header.  After the entire header has been read, it
would flip into the next state :body and feed it the remaining data.  In
this state the client would stream/buffer the data elsewhere (e.g. a
callback method), consuming everything it can until it's consumed the entire
Content-Length.  At this point, it would flip into the next state (:header
again, I assume) and pass any remaining data to the dispatcher.


> How could I use Revactor :: Filter class to achive it?
> I've also read that Revactor :: Filter :: Line delimiter just allows one
> single char as separator, but SIP requires lines ending with CRLF. Would it
> be an issue?
>

Yes, there's no delimiter-based filters available in Revactor which work on
multicharacter boundaries, unfortunately.  You're pretty much on your own
doing the parsing, regardless of what you end up using for the underlying
I/O layer.

-- 
Tony Arcieri
medioh.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/revactor-talk/attachments/20090120/14af7f8e/attachment.html>

From ibc at aliax.net  Tue Jan 20 19:51:53 2009
From: ibc at aliax.net (=?utf-8?q?I=C3=B1aki_Baz_Castillo?=)
Date: Wed, 21 Jan 2009 01:51:53 +0100
Subject: [Revactor-talk] Initial look to Revactor and some doubts
In-Reply-To: <c7e6b2b00901201614v2fff4c21i44c06094bac8b8d5@mail.gmail.com>
References: <200901210046.26358.ibc@aliax.net>
	<c7e6b2b00901201614v2fff4c21i44c06094bac8b8d5@mail.gmail.com>
Message-ID: <200901210151.53488.ibc@aliax.net>

El Mi?rcoles, 21 de Enero de 2009, Tony Arcieri escribi?:
> On Tue, Jan 20, 2009 at 4:46 PM, I?aki Baz Castillo <ibc at aliax.net> wrote:
> > Hi, I'm interested in Revactor to build a SIP server. I'm right now
> > started learning about EventMachine and Revactor, not sure which one
> > would be more appropiate for my project.
>
> EventMachine is probably your best bet.  I am not actively maintaining
> Revactor, and worse it's tied to Ruby 1.9 due to its use of Fibers.  1.9
> continues its glacial development pace (we got a 1.9.1 RC for Christmas
> rather than a final, more "stable" version) and due to its
> incompatibilities and bugs it has not seen widespread adoption.

That's a pity. Waiting for a stable release of Ruby 1.9 is not a problem for 
me since my project will take long time.
Anyway, if you say that Revactor is not actively mantained perhaps then I 
should choose EM.



> > For now I've started my SIP Server from scratch but I expect to migrate
> > it to
> > an efficient concurrent model.
>
> Well, you are going to find yourself in asynchronous callback hell with
> something like EventMachine trying to write a SIP server.

Well, I hope that using EM or Revactor would be what I need to achieve that 
hell, wouldn't it? :)



> I'm curious why you picked Ruby to write a SIP server in.

Just because I like Ruby language and nothing similar exists. There are some 
SIP servers written in Python language, I wonder why Ruby wouldn't be a 
choice. Perhaps you could detail a bit more your question please? I'm really 
interested on it.



> Are you familiar with Adhearsion?  It provides an extremely rich set of
> Ruby bindings for Asterisk, an already-robust SIP server:

Well... I know *very* well Asterisk and I can sure you that Asterisk is not 
a "good" SIP server. Perhaps it's a good and flexible multiprotocol PBX, but 
its SIP stack is really bad. It doesn't implement cool SIP features like IM 
or presence status, and many others. It implements SIP protocol in its "own" 
way, I'm really tired of it.
Under my understanding (and I work every day with Asterisk in my job) Asterisk 
is the best option for people with no real knowledge of SIP protocol. 
Asterisk provides a "easy" and complete solution. Just it. Ufortunatelly VoIP 
world is really bigger than Asterisk and I want to explore it with my own 
solution :)


> > As you can see, the parser must read the "Content-Length" header and read
> > that
> > ammount of bytes after the empty line (CRLF) which separes the header
> > from the body. After that ammount of bytes, the next data is supposed to
> > be a new
> > SIP message.
>
> Revactor really isn't going to help you here, although it could help you in
> the higher level structure of your application.

Do you mean that I could use EM for the listener layer (receiving and sending 
SIP messages) and Revactor for higher levels as dialog/PBX logic?


> My ideal advice for how to tackle this problem is the "Mongrel approach":
> find the ABNF grammar for SIP headers and convert it into a Ragel
> description:
>
> http://www.complang.org/ragel/
>
> ...and building a small Ruby C extension which wraps the Ragel parser.

For now I've used TreeTop Ruby parser and I've already implemented the whole 
SIP BNF grammar using it. I know it's slower than Ragel anyway, so in a 
future I should migrate to Ragel.


> This is a technique Zed Shaw pioneered (at least in the Ruby world) which
> works extremely well.   Take a look at Mongrel as an example or ask on the
> Mongrel mailing list if you'd like to pursue this approach.  The parser
> will be very fast and guaranteed correct.  If anything, you will have to
> relax the grammar to interoperate with broken SIP implementations.

(Un)fortunatelly I know very well the SIP world and I'm already aware of the 
real SIP implementations and their "funny" SIP grammar and behaviour :)



> Once you've done this, you feed any data you receive into the parser.  The
> parser will build a structure (like a hash) containing the elements in the
> header, and tell you when it has finished and what data has left over.  You
> just spoonfeed all data to the parser and it's smart enough to know when
> it's consumed the entire header and leaves you with the leftover data,
> namely the message body (and potentially part of the next header).

> So in the example you gave, the parser would magically consume everything
> up to the last <CRLF>, build a structure representing the header (e.g.
> {:type => "MESSAGE", :protocol => "SIP/2.0", :from =>
> "sip:alice at domain.org<sip%3Aalice at domain.org>;tag=qweqweqwe",
>
> :content-length => 5, ...}, and leave you with a string containing the rest
>
> of the message.

That's ok, but in my first approach I've already implemented it and I use a 
custom "Header" class to store each parsed header. Note for example that, 
like HTTP, SIP allows line folding (a header value takes a new line prefixed 
by some space or tab):

Subject; Hello
 world!

Do you think that using Ragel I could store each detected (and not fully 
parsed yet) header into an Array of Header class instances? What I do after 
getting that Array is a complete parsing of the required headers.



> I saw your post on the EventMachine mailing list as well, and there I
> suggested you use a simple, informal state machine to tackle the rest of
> the message processing.  As an example, this code implements an HTTP client
> using a Ragel-generated parser and uses a simple informal state machine to
> control the message processing (so it can be asynchronous underneath and
> work on things like EventMachine):
>
> http://github.com/tarcieri/fasthttp/blob/c6d3cac049e81b2cc16e0379a0f5dff433
>f22994/lib/fasthttp.rb

Yes, I already took a look to it and seems really interesting.


> Basically, your state machine would start out in the :header state, and
> would use the Ragel parser to suck in the header and determine when it's
> finished reading the header.  After the entire header has been read, it
> would flip into the next state :body and feed it the remaining data.  In
> this state the client would stream/buffer the data elsewhere (e.g. a
> callback method), consuming everything it can until it's consumed the
> entire Content-Length.  At this point, it would flip into the next state
> (:header again, I assume) and pass any remaining data to the dispatcher.

That's exactly the behaviour I've already implemented in my own listener (but 
just it, I've not implemented nothing about callbacks yet, this is why I'm 
interested in migrating to EM or Revactor).


> > How could I use Revactor :: Filter class to achive it?
> > I've also read that Revactor :: Filter :: Line delimiter just allows one
> > single char as separator, but SIP requires lines ending with CRLF. Would
> > it be an issue?
>
> Yes, there's no delimiter-based filters available in Revactor which work on
> multicharacter boundaries, unfortunately.  You're pretty much on your own
> doing the parsing, regardless of what you end up using for the underlying
> I/O layer.

Ok, really thanks a lot for your response. Let me just insist on a question:
If waiting for Ruby 1.9 is not a real issue for me, should/could I choose 
Revactor? or it's definitively not mantained so better if I use EM?


Thanks a lot and best regards.


-- 
I?aki Baz Castillo

From tony at medioh.com  Tue Jan 20 20:23:05 2009
From: tony at medioh.com (Tony Arcieri)
Date: Tue, 20 Jan 2009 18:23:05 -0700
Subject: [Revactor-talk] Initial look to Revactor and some doubts
In-Reply-To: <200901210151.53488.ibc@aliax.net>
References: <200901210046.26358.ibc@aliax.net>
	<c7e6b2b00901201614v2fff4c21i44c06094bac8b8d5@mail.gmail.com>
	<200901210151.53488.ibc@aliax.net>
Message-ID: <c7e6b2b00901201723l31fd5fe5y9a39bbbf1131a4f5@mail.gmail.com>

On Tue, Jan 20, 2009 at 5:51 PM, I?aki Baz Castillo <ibc at aliax.net> wrote:

> That's a pity. Waiting for a stable release of Ruby 1.9 is not a problem
> for
> me since my project will take long time.
> Anyway, if you say that Revactor is not actively mantained perhaps then I
> should choose EM.
>

I think if you try to write a SIP server in an event-driven manner with
something like EventMachine you are going to find yourself extremely
frustrated with how complex and hard to understand your program is.  I'm not
a fan of event driven programming (which is why I wrote Revactor in the
first place) and only use it where I absolutely must (I wrote an
event-driven library which underlies Revactor called Rev)

Furthermore, for something like a SIP server you're going to want to avoid
the MRI/YARV implementations of Ruby entirely.  These branches lack a
compacting garbage collector, so your server will experience unbounded
memory growth that can only be solved by restarting it.  That's probably bad
for a SIP implementation.

You might look at JRuby or Rubinius, both of which feature much better
garbage collection than MRI/YARV.  JRuby is mature and Rubinius is not,
however Rubinius does feature an Actor model implementation which is more or
less API compatible with Revactor.  The Actor model has proven remarkably
successful for the implementation of PBXes by Ericsson (in Erlang) and I
would certainly suggest it as an alternative to threads.  Event-driven
programming is another alternative to threads, however I wouldn't recommend
it for anything which requires complex interactions between various
connections.  If you're not too worried

There are other Actor implementations for Ruby available, like Omnibus (
http://rubyforge.org/projects/concurrent) and Dramatis.  The author of the
former (MenTaLguY) is committed to getting it running on JRuby.

EventMachine is available for JRuby but I wouldn't particularly recommend it
as the core JRuby I/O routines are already based around NIO2 to begin with
which eliminates most of the need for EventMachine to begin with.  You just
need to decide if you really want to go with an event-driven model versus a
Actor-based (or even threaded) one for concurrency.


> Well, I hope that using EM or Revactor would be what I need to achieve that
> hell, wouldn't it? :)
>

You should try to avoid it if you can.  Revactor (or Rubinius Actors) let
you build synchronous APIs on top of fundamentally asynchronous code which
makes them much, much simpler to work with.  Revactor is fundamentally
event-driven underneath but aims to free the programmer of the headaches
associated with an event-driven callback system by letting you wrap those
callbacks up in synchronous APIs that look like normal Ruby code.

For now I've used TreeTop Ruby parser and I've already implemented the whole
> SIP BNF grammar using it. I know it's slower than Ragel anyway, so in a
> future I should migrate to Ragel.
>

Well that's cool.  Since the SIP grammar is regular you don't really need a
full-blown PEG parser like TreeTop provides, but that's a good way to go
until you actually run into performance problems.

> I saw your post on the EventMachine mailing list as well, and there I
> > suggested you use a simple, informal state machine to tackle the rest of
> > the message processing.  As an example, this code implements an HTTP
> client
> > using a Ragel-generated parser and uses a simple informal state machine
> to
> > control the message processing (so it can be asynchronous underneath and
> > work on things like EventMachine):
> >
> >
> http://github.com/tarcieri/fasthttp/blob/c6d3cac049e81b2cc16e0379a0f5dff433
> >f22994/lib/fasthttp.rb
>
> Yes, I already took a look to it and seems really interesting.
>

Cool

-- 
Tony Arcieri
medioh.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/revactor-talk/attachments/20090120/2f5288c0/attachment.html>

From ibc at aliax.net  Wed Jan 21 04:51:31 2009
From: ibc at aliax.net (=?UTF-8?Q?I=C3=B1aki_Baz_Castillo?=)
Date: Wed, 21 Jan 2009 10:51:31 +0100
Subject: [Revactor-talk] Initial look to Revactor and some doubts
In-Reply-To: <c7e6b2b00901201723l31fd5fe5y9a39bbbf1131a4f5@mail.gmail.com>
References: <200901210046.26358.ibc@aliax.net>
	<c7e6b2b00901201614v2fff4c21i44c06094bac8b8d5@mail.gmail.com>
	<200901210151.53488.ibc@aliax.net>
	<c7e6b2b00901201723l31fd5fe5y9a39bbbf1131a4f5@mail.gmail.com>
Message-ID: <cc1f582e0901210151r281ffb90n21b27b9546432565@mail.gmail.com>

2009/1/21 Tony Arcieri <tony at medioh.com>:

> I think if you try to write a SIP server in an event-driven manner with
> something like EventMachine you are going to find yourself extremely
> frustrated with how complex and hard to understand your program is.  I'm not
> a fan of event driven programming (which is why I wrote Revactor in the
> first place) and only use it where I absolutely must (I wrote an
> event-driven library which underlies Revactor called Rev)

Yes, in fact what I like of Revactor is the sequential programming
style it allows, "hidding" the asynchronous callbacks when an event
occurs. For eaxmple, at SIP top layer I would need to implement a
"dialplan" when receiving an INVITE, something like:

---------------------
  # leg A is created when an INVITE arrives from an user.

  # After transport and transaction layer, the logic layer (dialplan) starts:

  # First we create a leg B to the solicited PSTN number and route it
to the PSTN gateway:
  pstn_call = Call("sip:" + request.ruri.username + "@" + PSTN_GATEWAY_IP:5060,
     { :timeout => 40, :stop_on_callee_hangup => false })

  # The above Call can fail due to reject response, busy response,
remote server failure:
  case pstn_call.final_response
  when /^3[0-9][0-9]$/
    puts "3XX Redirect response received, creating a new call..."
    new_call = Call(pstn_call.final_response.headers["Contact"])
    leg_a.hangup()

  when "486"
    puts "User busy"
    leg_a.hangup()

  when /^4[0-9][0-9]$/
    puts "Some 4XX response"
    leg_a.hangup()
  end

  # If we are here, the call was answered with 200 and now it's
finished (callee sent a BYE)

  # Now call to a recording machine to talk about our previous call
(opinion about it and so):
  am_call = Call("sip:answer-machine at media-services.domain.org")

  leg_a.hangup
-----------------------------

As you see, it seems a sequential and nonbloking code, but the fact is
the the Call function can take various minutes or hours (until caller
or callee sends a BYE or  an internal monitorization system detects a
failure in the signalling). After the Call function ends (and if the
caller didn't hangup) more dialplan logic is permorfed.

Yes, I wouldn't programm some logic in an event-model way...




> Furthermore, for something like a SIP server you're going to want to avoid
> the MRI/YARV implementations of Ruby entirely.  These branches lack a
> compacting garbage collector, so your server will experience unbounded
> memory growth that can only be solved by restarting it.  That's probably bad
> for a SIP implementation.

Ok.



> You might look at JRuby or Rubinius, both of which feature much better
> garbage collection than MRI/YARV.  JRuby is mature and Rubinius is not,
> however Rubinius does feature an Actor model implementation which is more or
> less API compatible with Revactor.  The Actor model has proven remarkably
> successful for the implementation of PBXes by Ericsson (in Erlang) and I
> would certainly suggest it as an alternative to threads.  Event-driven
> programming is another alternative to threads, however I wouldn't recommend
> it for anything which requires complex interactions between various
> connections.  If you're not too worried

Ok.



> There are other Actor implementations for Ruby available, like Omnibus
> (http://rubyforge.org/projects/concurrent) and Dramatis.  The author of the
> former (MenTaLguY) is committed to getting it running on JRuby.

mmm, so many options... I don't know how to choose the most appropiate :(
For example, you say that Rubinius implements an Actor model, but
Dramatis seems to work also in Rubinius (and JRuby, and 1.8 - 1.9).

If I choose JRuby, should I use Dramatis?
Note that Dramatis has no activity since June 2008, and I see no real
doc or web project about Omnibius...




> EventMachine is available for JRuby but I wouldn't particularly recommend it
> as the core JRuby I/O routines are already based around NIO2 to begin with
> which eliminates most of the need for EventMachine to begin with.  You just
> need to decide if you really want to go with an event-driven model versus a
> Actor-based (or even threaded) one for concurrency.

I like the Actor model concept, but sincerelly I have no idea of which
implementation to choose. Of course I need a mature and mantained
implementation. No more requeriments (I could use 1.8, 1.9, JRuby,
Rubinius...). Could you please help me to decide one?


>>
>> Well, I hope that using EM or Revactor would be what I need to achieve
>> that
>> hell, wouldn't it? :)
>
> You should try to avoid it if you can.  Revactor (or Rubinius Actors) let
> you build synchronous APIs on top of fundamentally asynchronous code which
> makes them much, much simpler to work with.  Revactor is fundamentally
> event-driven underneath but aims to free the programmer of the headaches
> associated with an event-driven callback system by letting you wrap those
> callbacks up in synchronous APIs that look like normal Ruby code.

:)


Again thanks a lot for your time solving my doubts. Best regards.


-- 
I?aki Baz Castillo
<ibc at aliax.net>