From nobody at rubyforge.org Mon Nov 13 08:44:30 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 08:44:30 -0500 (EST) Subject: [Archipelago-submits] [5] trunk/archipelago/TODO: test commit Message-ID: <20061113134430.DFE54A970007@rubyforge.org> An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/archipelago-submits/attachments/20061113/4cfd882c/attachment.html From nobody at rubyforge.org Mon Nov 13 09:04:49 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 09:04:49 -0500 (EST) Subject: [Archipelago-submits] [6] trunk/archipelago/TODO: test commit Message-ID: <20061113140449.9FFAEA97000A@rubyforge.org> An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/archipelago-submits/attachments/20061113/1eeebb61/attachment.html From nobody at rubyforge.org Mon Nov 13 09:32:53 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 09:32:53 -0500 (EST) Subject: [Archipelago-submits] [7] trunk/archipelago/TODO: test commit Message-ID: <20061113143253.23DF1A970007@rubyforge.org> An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/archipelago-submits/attachments/20061113/bcff8dcc/attachment.html From nobody at rubyforge.org Mon Nov 13 09:45:26 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 09:45:26 -0500 (EST) Subject: [Archipelago-submits] [8] trunk/archipelago/TODO: test commit Message-ID: <20061113144526.EB483A970007@rubyforge.org> An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/archipelago-submits/attachments/20061113/3e2ea9e7/attachment.html From nobody at rubyforge.org Mon Nov 13 16:05:01 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 16:05:01 -0500 (EST) Subject: [Archipelago-submits] [9] trunk/archipelago/TODO: test commit Message-ID: <20061113210501.5A5BFA97000A@rubyforge.org> Revision: 9 Author: zond Date: 2006-11-13 16:05:00 -0500 (Mon, 13 Nov 2006) Log Message: ----------- test commit Modified Paths: -------------- trunk/archipelago/TODO Modified: trunk/archipelago/TODO =================================================================== --- trunk/archipelago/TODO 2006-11-13 14:45:26 UTC (rev 8) +++ trunk/archipelago/TODO 2006-11-13 21:05:00 UTC (rev 9) @@ -7,4 +7,3 @@ the call. Preferably without incurring performance lossage. * Test the transaction recovery mechanism of Chest. - From nobody at rubyforge.org Mon Nov 13 17:52:23 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 17:52:23 -0500 (EST) Subject: [Archipelago-submits] [10] trunk/archipelago/Rakefile: removed old MTN stuff in Rakefile Message-ID: <20061113225223.A9FD452417CA@rubyforge.org> Revision: 10 Author: zond Date: 2006-11-13 17:52:23 -0500 (Mon, 13 Nov 2006) Log Message: ----------- removed old MTN stuff in Rakefile Modified Paths: -------------- trunk/archipelago/Rakefile Modified: trunk/archipelago/Rakefile =================================================================== --- trunk/archipelago/Rakefile 2006-11-13 21:05:00 UTC (rev 9) +++ trunk/archipelago/Rakefile 2006-11-13 22:52:23 UTC (rev 10) @@ -29,7 +29,6 @@ fl.include "#{dir}/**/*" end fl.include "Rakefile" - fl.exclude(/\b_MTN\b/) end Rake::GemPackageTask.new(spec) do |pkg| From nobody at rubyforge.org Mon Nov 13 18:04:27 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 18:04:27 -0500 (EST) Subject: [Archipelago-submits] [11] trunk/archipelago: intermediate commit damn you svn Message-ID: <20061113230427.D315652417CA@rubyforge.org> Revision: 11 Author: zond Date: 2006-11-13 18:04:26 -0500 (Mon, 13 Nov 2006) Log Message: ----------- intermediate commit damn you svn Modified Paths: -------------- trunk/archipelago/lib/archipelago.rb trunk/archipelago/lib/disco.rb trunk/archipelago/lib/hashish.rb trunk/archipelago/lib/pirate.rb trunk/archipelago/lib/tranny.rb trunk/archipelago/lib/treasure.rb trunk/archipelago/tests/current_benchmark.rb trunk/archipelago/tests/current_test.rb trunk/archipelago/tests/disco_test.rb trunk/archipelago/tests/pirate_test.rb trunk/archipelago/tests/test_helper.rb trunk/archipelago/tests/tranny_test.rb trunk/archipelago/tests/treasure_benchmark.rb trunk/archipelago/tests/treasure_test.rb Added Paths: ----------- trunk/archipelago/lib/archipelago/ trunk/archipelago/lib/archipelago/current.rb Removed Paths: ------------- trunk/archipelago/lib/current.rb Copied: trunk/archipelago/lib/archipelago/current.rb (from rev 1, trunk/archipelago/lib/current.rb) =================================================================== --- trunk/archipelago/lib/archipelago/current.rb (rev 0) +++ trunk/archipelago/lib/archipelago/current.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -0,0 +1,128 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'monitor' + +# +# Make threads dumpable for show. +# +class Thread + def _dump(l) + "" + end + def self._load(s) + nil + end +end + +module Archipelago + + module Current + + # + # A module that will allow any class to synchronize over any other + # object. + # + module Synchronized + include MonitorMixin + alias :lock :mon_enter + alias :unlock :mon_exit + # + # We dont care about lock ownership. + # + def mon_check_owner + end + # + # Get a lock for this +object+ + # + def lock_on(object) + Thread.exclusive do + this_lock = @archipelago_current_synchronized_lock_by_object[object] ||= Lock.new + this_lock.lock + end + end + # + # Release any lock on this +object+. + # + def unlock_on(object) + Thread.exclusive do + this_lock = @archipelago_current_synchronized_lock_by_object[object] ||= Lock.new + this_lock.unlock + if this_lock.mon_entering_queue.empty? + @archipelago_current_synchronized_lock_by_object.delete(object) + end + end + end + # + # Makes sure the given +block+ is only run once at a time + # for this instance and the given +object+ + # + # Optionally do NOT lock, if you dont +actually+ want to. + # + def synchronize_on(object, actually = true, &block) + lock_on(object) if actually + begin + return yield + ensure + unlock_on(object) if actually + end + end + + private + + # + # Initialize our instance variables + # when someone is extended with us. + # + def self.extend_object(obj) + super(obj) + obj.instance_eval do + sync_initialize + end + end + + # + # Do the actual initialization. + # + def sync_initialize + @archipelago_current_synchronized_lock_by_object = {} + mon_initialize + end + + # + # Initialize upon creation. + # + # Classes that include this module should call super() + # upon initialization. + # + def initialize(*args) + super() + sync_initialize + end + end + + # + # Just a convenience empty class with locking functionality. + # + class Lock + include Synchronized + attr_reader :mon_entering_queue + end + + end + +end Modified: trunk/archipelago/lib/archipelago.rb =================================================================== --- trunk/archipelago/lib/archipelago.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/lib/archipelago.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -17,8 +17,8 @@ $: << File.dirname(File.expand_path(__FILE__)) -require 'disco' -require 'current' -require 'tranny' -require 'treasure' -require 'pirate' +require 'archipelago/disco' +require 'archipelago/current' +require 'archipelago/tranny' +require 'archipelago/treasure' +require 'archipelago/pirate' Deleted: trunk/archipelago/lib/current.rb =================================================================== --- trunk/archipelago/lib/current.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/lib/current.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -1,128 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'monitor' - -# -# Make threads dumpable for show. -# -class Thread - def _dump(l) - "" - end - def self._load(s) - nil - end -end - -module Archipelago - - module Current - - # - # A module that will allow any class to synchronize over any other - # object. - # - module Synchronized - include MonitorMixin - alias :lock :mon_enter - alias :unlock :mon_exit - # - # We dont care about lock ownership. - # - def mon_check_owner - end - # - # Get a lock for this +object+ - # - def lock_on(object) - Thread.exclusive do - this_lock = @archipelago_current_synchronized_lock_by_object[object] ||= Lock.new - this_lock.lock - end - end - # - # Release any lock on this +object+. - # - def unlock_on(object) - Thread.exclusive do - this_lock = @archipelago_current_synchronized_lock_by_object[object] ||= Lock.new - this_lock.unlock - if this_lock.mon_entering_queue.empty? - @archipelago_current_synchronized_lock_by_object.delete(object) - end - end - end - # - # Makes sure the given +block+ is only run once at a time - # for this instance and the given +object+ - # - # Optionally do NOT lock, if you dont +actually+ want to. - # - def synchronize_on(object, actually = true, &block) - lock_on(object) if actually - begin - return yield - ensure - unlock_on(object) if actually - end - end - - private - - # - # Initialize our instance variables - # when someone is extended with us. - # - def self.extend_object(obj) - super(obj) - obj.instance_eval do - sync_initialize - end - end - - # - # Do the actual initialization. - # - def sync_initialize - @archipelago_current_synchronized_lock_by_object = {} - mon_initialize - end - - # - # Initialize upon creation. - # - # Classes that include this module should call super() - # upon initialization. - # - def initialize(*args) - super() - sync_initialize - end - end - - # - # Just a convenience empty class with locking functionality. - # - class Lock - include Synchronized - attr_reader :mon_entering_queue - end - - end - -end Modified: trunk/archipelago/lib/disco.rb =================================================================== --- trunk/archipelago/lib/disco.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/lib/disco.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -19,7 +19,7 @@ require 'thread' require 'ipaddr' require 'pp' -require 'current' +require 'archipelago/current' require 'drb' require 'set' Modified: trunk/archipelago/lib/hashish.rb =================================================================== --- trunk/archipelago/lib/hashish.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/lib/hashish.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -15,7 +15,7 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -require 'current' +require 'archipelago/current' require 'bdb' module Archipelago Modified: trunk/archipelago/lib/pirate.rb =================================================================== --- trunk/archipelago/lib/pirate.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/lib/pirate.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -15,9 +15,9 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -require 'disco' -require 'tranny' -require 'treasure' +require 'archipelago/disco' +require 'archipelago/tranny' +require 'archipelago/treasure' require 'pp' require 'drb' require 'digest/sha1' Modified: trunk/archipelago/lib/tranny.rb =================================================================== --- trunk/archipelago/lib/tranny.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/lib/tranny.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -18,9 +18,10 @@ require 'bdb' require 'pathname' require 'drb' -require 'current' +require 'archipelago/current' require 'digest/sha1' -require 'disco' +require 'archipelago/disco' +require 'archipelago/hashish' module Archipelago Modified: trunk/archipelago/lib/treasure.rb =================================================================== --- trunk/archipelago/lib/treasure.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/lib/treasure.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -16,14 +16,14 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. require 'drb' -require 'disco' +require 'archipelago/disco' require 'bdb' require 'pathname' require 'digest/sha1' require 'pp' require 'set' -require 'hashish' -require 'tranny' +require 'archipelago/hashish' +require 'archipelago/tranny' module Archipelago Modified: trunk/archipelago/tests/current_benchmark.rb =================================================================== --- trunk/archipelago/tests/current_benchmark.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/tests/current_benchmark.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -1,6 +1,5 @@ require File.join(File.dirname(__FILE__), 'test_helper') -require 'current' class CurrentBenchmark < Test::Unit::TestCase Modified: trunk/archipelago/tests/current_test.rb =================================================================== --- trunk/archipelago/tests/current_test.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/tests/current_test.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -1,6 +1,5 @@ require File.join(File.dirname(__FILE__), 'test_helper') -require 'current' class CurrentTest < Test::Unit::TestCase Modified: trunk/archipelago/tests/disco_test.rb =================================================================== --- trunk/archipelago/tests/disco_test.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/tests/disco_test.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -1,10 +1,5 @@ require File.join(File.dirname(__FILE__), 'test_helper') -require 'disco' -require 'drb' -require 'socket' -require 'ipaddr' -require 'thread' class RemoteValidator include DRb::DRbUndumped Modified: trunk/archipelago/tests/pirate_test.rb =================================================================== --- trunk/archipelago/tests/pirate_test.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/tests/pirate_test.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -1,10 +1,5 @@ require File.join(File.dirname(__FILE__), 'test_helper') -require 'treasure' -require 'drb' -require 'tranny' -require 'hashish' -require 'pirate' class PirateTest < Test::Unit::TestCase Modified: trunk/archipelago/tests/test_helper.rb =================================================================== --- trunk/archipelago/tests/test_helper.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/tests/test_helper.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -3,10 +3,13 @@ $: << File.join(home, "..", "lib") require 'pp' +require 'drb' require 'test/unit' -require 'tranny' -require 'treasure' +require 'archipelago' require 'benchmark' +require 'socket' +require 'ipaddr' +require 'thread' class TestTransaction def join(o) Modified: trunk/archipelago/tests/tranny_test.rb =================================================================== --- trunk/archipelago/tests/tranny_test.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/tests/tranny_test.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -1,7 +1,5 @@ require File.join(File.dirname(__FILE__), 'test_helper') -require 'tranny' -require 'drb' class TrannyTest < Test::Unit::TestCase Modified: trunk/archipelago/tests/treasure_benchmark.rb =================================================================== --- trunk/archipelago/tests/treasure_benchmark.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/tests/treasure_benchmark.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -1,8 +1,5 @@ require File.join(File.dirname(__FILE__), 'test_helper') -require 'treasure' -require 'drb' -require 'hashish' class TreasureBenchmark < Test::Unit::TestCase Modified: trunk/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-13 22:52:23 UTC (rev 10) +++ trunk/archipelago/tests/treasure_test.rb 2006-11-13 23:04:26 UTC (rev 11) @@ -1,9 +1,5 @@ require File.join(File.dirname(__FILE__), 'test_helper') -require 'treasure' -require 'drb' -require 'tranny' -require 'hashish' class TreasureTest < Test::Unit::TestCase From nobody at rubyforge.org Mon Nov 13 19:27:05 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 19:27:05 -0500 (EST) Subject: [Archipelago-submits] [13] trunk/archipelago: added :published_at as required field in Records. Message-ID: <20061114002705.71D8BA970014@rubyforge.org> Revision: 13 Author: zond Date: 2006-11-13 19:27:04 -0500 (Mon, 13 Nov 2006) Log Message: ----------- added :published_at as required field in Records. added evaluate! method to Chest to evaluate new classes remotely. made pirate send evaluate! calls to new chests. Modified Paths: -------------- trunk/archipelago/lib/archipelago/disco.rb trunk/archipelago/lib/archipelago/pirate.rb trunk/archipelago/lib/archipelago/treasure.rb trunk/archipelago/scripts/chest.rb trunk/archipelago/scripts/pirate.rb trunk/archipelago/scripts/tranny.rb trunk/archipelago/tests/disco_test.rb trunk/archipelago/tests/treasure_test.rb Modified: trunk/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-13 23:06:06 UTC (rev 12) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-14 00:27:04 UTC (rev 13) @@ -107,6 +107,7 @@ # def publish!(options = {}) @jockey ||= Archipelago::Disco::Jockey.new(@jockey_options.merge(options[:jockey_options] || {})) + @service_description.merge!({:published_at => Time.now}) @jockey.publish(Archipelago::Disco::Record.new(@service_description.merge(options[:service_description] || {}))) end @@ -203,8 +204,9 @@ # Initialize this Record with a hash that must contain an :service_id and a :validator. # def initialize(hash) - raise "Record must have an :service_id" unless hash.include?(:service_id) + raise "Record must have a :service_id" unless hash.include?(:service_id) raise "Record must have a :validator" unless hash.include?(:validator) + raise "Record must have a :published_at" unless hash.include?(:published_at) super(hash) end # Modified: trunk/archipelago/lib/archipelago/pirate.rb =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb 2006-11-13 23:06:06 UTC (rev 12) +++ trunk/archipelago/lib/archipelago/pirate.rb 2006-11-14 00:27:04 UTC (rev 13) @@ -74,6 +74,9 @@ # Will look for Archipelago::Treasure::Chests matching :chest_description or CHEST_DESCRIPTION and # Archipelago::Tranny::Managers matching :tranny_description or TRANNY_DESCRIPTION. # + # Will send off all :chest_eval_files to any chest found for possible evaluation to ensure existence + # of required classes and modules at the chest. + # def initialize(options = {}) @treasure_map = Archipelago::Disco::Jockey.new(options[:jockey_options] || {}) @@ -81,6 +84,10 @@ @tranny_description = TRANNY_DESCRIPTION.merge(options[:tranny_description] || {}) @jockey_options = options[:jockey_options] || {} + @chest_eval_files = options[:chest_eval_files] || [] + + @chests_having_evaluated = Set.new + start_service_updater(options[:initial_service_update_interval] || INITIAL_SERVICE_UPDATE_INTERVAL, options[:maximum_service_update_interval] || MAXIMUM_SERVICE_UPDATE_INTERVAL) @@ -200,11 +207,32 @@ end # + # Get all chests from the Archipelago::Disco::Jockey and + # send our @chest_eval_files to it if we have not already. + # + def get_chests + @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) + @chests.values.each do |chest| + unless @chests_having_evaluated.include?([chest[:service_id], chest[:published_at]]) + @chest_eval_files.each do |filename| + begin + chest[:service].evaluate!(filename, open(filename).read) + rescue Exception => e + puts e + pp e.backtrace + end + end + @chests_having_evaluated << [chest[:service_id], chest[:published_at]] + end + end + end + + # # Start a thread looking up existing chests between every # +initial+ and +maximum+ seconds. # def start_service_updater(initial, maximum) - @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) + get_chests @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) Thread.start do standoff = initial @@ -213,7 +241,7 @@ sleep(standoff) standoff *= 2 standoff = maximum if standoff > maximum - @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) + get_chests @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) rescue Exception => e puts e Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-13 23:06:06 UTC (rev 12) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-14 00:27:04 UTC (rev 13) @@ -16,7 +16,6 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. require 'drb' -require 'archipelago/disco' require 'bdb' require 'pathname' require 'digest/sha1' @@ -24,6 +23,7 @@ require 'set' require 'archipelago/hashish' require 'archipelago/tranny' +require 'archipelago/disco' module Archipelago @@ -214,6 +214,13 @@ @crashed = Set.new # + # [label,label...] + # To know what data we have already evaluated + # so that we dont overdo it. + # + @seen_data = Set.new + + # # The magical persistent map that defines how we actually # store our data. # @@ -231,6 +238,22 @@ end # + # Evaluate +data+ if we have not already seen +label+. + # + def evaluate!(label, data) + unless @seen_data.include?(label) + begin + Object.class_eval(data) + rescue Exception => e + puts e + pp e.backtrace + ensure + @seen_data << label + end + end + end + + # # Return the contents of this chest using a given +key+ and +transaction+. # def [](key, transaction = nil) Modified: trunk/archipelago/scripts/chest.rb =================================================================== --- trunk/archipelago/scripts/chest.rb 2006-11-13 23:06:06 UTC (rev 12) +++ trunk/archipelago/scripts/chest.rb 2006-11-14 00:27:04 UTC (rev 13) @@ -7,7 +7,7 @@ $: << File.join(File.dirname(__FILE__), "..", "lib") -require 'treasure' +require 'archipelago/treasure' if ARGV.size > 1 DRb.start_service(ARGV[1]) Modified: trunk/archipelago/scripts/pirate.rb =================================================================== --- trunk/archipelago/scripts/pirate.rb 2006-11-13 23:06:06 UTC (rev 12) +++ trunk/archipelago/scripts/pirate.rb 2006-11-14 00:27:04 UTC (rev 13) @@ -1,6 +1,6 @@ #!/usr/bin/env ruby -require 'pirate' +require 'archipelago/pirate' DRb.start_service("druby://localhost:#{rand(1000) + 5000}") @p = Archipelago::Pirate::Captain.new Modified: trunk/archipelago/scripts/tranny.rb =================================================================== --- trunk/archipelago/scripts/tranny.rb 2006-11-13 23:06:06 UTC (rev 12) +++ trunk/archipelago/scripts/tranny.rb 2006-11-14 00:27:04 UTC (rev 13) @@ -7,7 +7,7 @@ $: << File.join(File.dirname(__FILE__), "..", "lib") -require 'treasure' +require 'archipelago/treasure' if ARGV.size > 1 DRb.start_service(ARGV[1]) Modified: trunk/archipelago/tests/disco_test.rb =================================================================== --- trunk/archipelago/tests/disco_test.rb 2006-11-13 23:06:06 UTC (rev 12) +++ trunk/archipelago/tests/disco_test.rb 2006-11-14 00:27:04 UTC (rev 13) @@ -22,8 +22,9 @@ @d2 = TestJockey.new(:thrifty_publishing => false) @v1 = RemoteValidator.new(true) @p1 = Archipelago::Disco::Record.new(:service_id => 1, - :validator => DRbObject.new(@v1), - :epa => "blar") + :published_at => Time.now, + :validator => DRbObject.new(@v1), + :epa => "blar") @d1.publish(@p1) assert(!@d2.lookup(Archipelago::Disco::Query.new(:epa => "blar")).empty?) @@ -73,8 +74,9 @@ assert(empty) @d1.publish(Archipelago::Disco::Record.new(:service_id => 1, - :validator => Archipelago::Disco::MockValidator.new, - :epa => "blar2")) + :published_at => Time.now, + :validator => Archipelago::Disco::MockValidator.new, + :epa => "blar2")) assert_within(0.5) do !empty @@ -89,7 +91,10 @@ def test_thrifty_publishing @ltq.clear - @d1.publish(Archipelago::Disco::Record.new(:glada => "jaa", :validator => Archipelago::Disco::MockValidator.new, :service_id => 344)) + @d1.publish(Archipelago::Disco::Record.new(:glada => "jaa", + :published_at => Time.now, + :validator => Archipelago::Disco::MockValidator.new, + :service_id => 344)) sleep 0.1 assert(@ltq.any? do |d| o = Marshal.load(d) @@ -106,7 +111,10 @@ @ltq.clear c3 = Archipelago::Disco::Jockey.new(:thrifty_publishing => true) - c3.publish(Archipelago::Disco::Record.new(:glad => "ja", :validator => Archipelago::Disco::MockValidator.new, :service_id => 33)) + c3.publish(Archipelago::Disco::Record.new(:glad => "ja", + :published_at => Time.now, + :validator => Archipelago::Disco::MockValidator.new, + :service_id => 33)) sleep 0.1 assert(@ltq.empty?) @@ -118,7 +126,10 @@ end def test_thrifty_replying - @d1.publish(Archipelago::Disco::Record.new(:gladaa => "jaaa", :validator => Archipelago::Disco::MockValidator.new, :service_id => 3444)) + @d1.publish(Archipelago::Disco::Record.new(:gladaa => "jaaa", + :published_at => Time.now, + :validator => Archipelago::Disco::MockValidator.new, + :service_id => 3444)) @ltq.clear assert(!@d2.lookup(Archipelago::Disco::Query.new(:gladaa => "jaaa")).empty?) @@ -136,7 +147,10 @@ @d1.stop @d2.stop c3 = Archipelago::Disco::Jockey.new(:thrifty_replying => true, :thrifty_publishing => true) - c3.publish(Archipelago::Disco::Record.new(:glad2 => "ja2", :validator => Archipelago::Disco::MockValidator.new, :service_id => 34)) + c3.publish(Archipelago::Disco::Record.new(:glad2 => "ja2", + :published_at => Time.now, + :validator => Archipelago::Disco::MockValidator.new, + :service_id => 34)) @ltq.clear @@ -157,14 +171,20 @@ end def test_thrifty_caching - @d2.publish(Archipelago::Disco::Record.new(:bojkotta => "jag", :validator => Archipelago::Disco::MockValidator.new, :service_id => 411)) + @d2.publish(Archipelago::Disco::Record.new(:bojkotta => "jag", + :published_at => Time.now, + :validator => Archipelago::Disco::MockValidator.new, + :service_id => 411)) sleep 0.1 assert(@d1.remote_services.include?(411)) c1 = TestJockey.new(:thrifty_caching => true) assert(!c1.local_services.include?(41)) assert(!c1.remote_services.include?(41)) - @d1.publish(Archipelago::Disco::Record.new(:bojkott => "ja", :validator => Archipelago::Disco::MockValidator.new, :service_id => 41)) + @d1.publish(Archipelago::Disco::Record.new(:bojkott => "ja", + :published_at => Time.now, + :validator => Archipelago::Disco::MockValidator.new, + :service_id => 41)) sleep 0.1 assert(!c1.local_services.include?(41)) assert(!c1.remote_services.include?(41)) Modified: trunk/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-13 23:06:06 UTC (rev 12) +++ trunk/archipelago/tests/treasure_test.rb 2006-11-14 00:27:04 UTC (rev 13) @@ -17,6 +17,15 @@ DRb.stop_service end + def test_eval + @c.evaluate!("burk", "class Burk; end") + b = Burk.new + @c.evaluate!("burk", "class Bong; end") + assert_raise(NameError) do + b = Bong.new + end + end + def test_store_load_update s = "hehu" @c["oj"] = "hehu" From nobody at rubyforge.org Mon Nov 13 19:34:37 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 19:34:37 -0500 (EST) Subject: [Archipelago-submits] [14] trunk/archipelago: added a test for the automatic Chest#evaluate! call in Pirate::Captain Message-ID: <20061114003437.6298B5241951@rubyforge.org> Revision: 14 Author: zond Date: 2006-11-13 19:34:37 -0500 (Mon, 13 Nov 2006) Log Message: ----------- added a test for the automatic Chest#evaluate! call in Pirate::Captain Modified Paths: -------------- trunk/archipelago/lib/archipelago/disco.rb trunk/archipelago/lib/archipelago/pirate.rb trunk/archipelago/tests/disco_test.rb trunk/archipelago/tests/pirate_test.rb Added Paths: ----------- trunk/archipelago/tests/evaltest Modified: trunk/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-14 00:27:04 UTC (rev 13) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-14 00:34:37 UTC (rev 14) @@ -368,7 +368,7 @@ # # Stops all the threads in this instance. # - def stop + def stop! @listener_thread.kill @unilistener_thread.kill @shouter_thread.kill Modified: trunk/archipelago/lib/archipelago/pirate.rb =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb 2006-11-14 00:27:04 UTC (rev 13) +++ trunk/archipelago/lib/archipelago/pirate.rb 2006-11-14 00:34:37 UTC (rev 14) @@ -65,7 +65,7 @@ # Archipelago::Treasure:Dubloons in them. # class Captain - attr_reader :chests, :treasure_map, :trannies, :transaction + attr_reader :chests, :treasure_map, :trannies # # Initialize an instance using an Archipelago::Disco::Jockey with :jockey_options # if given, that looks for new services :initial_service_update_interval or INITIAL_SERVICE_UPDATE_INTERVAL, @@ -190,6 +190,13 @@ 'yar!' end + # + # Stops the service update thread for this Pirate. + # + def stop! + @service_update_thread.kill + end + private # @@ -234,7 +241,7 @@ def start_service_updater(initial, maximum) get_chests @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) - Thread.start do + @service_update_thread = Thread.start do standoff = initial loop do begin Modified: trunk/archipelago/tests/disco_test.rb =================================================================== --- trunk/archipelago/tests/disco_test.rb 2006-11-14 00:27:04 UTC (rev 13) +++ trunk/archipelago/tests/disco_test.rb 2006-11-14 00:34:37 UTC (rev 14) @@ -58,8 +58,8 @@ end def teardown - @d1.stop - @d2.stop + @d1.stop! + @d2.stop! DRb.stop_service @lt.kill @listener.close @@ -106,8 +106,8 @@ end end) - @d1.stop - @d2.stop + @d1.stop! + @d2.stop! @ltq.clear c3 = Archipelago::Disco::Jockey.new(:thrifty_publishing => true) @@ -121,8 +121,8 @@ c1 = Archipelago::Disco::Jockey.new(:thrifty_publishing => true) assert(!c1.lookup(Archipelago::Disco::Query.new(:glad => "ja")).empty?) - c1.stop - c3.stop + c1.stop! + c3.stop! end def test_thrifty_replying @@ -144,8 +144,8 @@ end) - @d1.stop - @d2.stop + @d1.stop! + @d2.stop! c3 = Archipelago::Disco::Jockey.new(:thrifty_replying => true, :thrifty_publishing => true) c3.publish(Archipelago::Disco::Record.new(:glad2 => "ja2", :published_at => Time.now, @@ -166,8 +166,8 @@ end end) - c1.stop - c3.stop + c1.stop! + c3.stop! end def test_thrifty_caching Added: trunk/archipelago/tests/evaltest =================================================================== --- trunk/archipelago/tests/evaltest (rev 0) +++ trunk/archipelago/tests/evaltest 2006-11-14 00:34:37 UTC (rev 14) @@ -0,0 +1,3 @@ +class Evaltest + +end Modified: trunk/archipelago/tests/pirate_test.rb =================================================================== --- trunk/archipelago/tests/pirate_test.rb 2006-11-14 00:27:04 UTC (rev 13) +++ trunk/archipelago/tests/pirate_test.rb 2006-11-14 00:34:37 UTC (rev 14) @@ -22,12 +22,30 @@ end def teardown + @p.stop! @c.persistence_provider.unlink @c2.persistence_provider.unlink @tm.persistence_provider.unlink DRb.stop_service end + def test_evaluate + assert_raise(NameError) do + e = Evaltest.new + end + + p2 = Archipelago::Pirate::Captain.new(:chest_description => {:class => "TestChest"}, + :tranny_description => {:class => "TestManager"}, + :chest_eval_files => File.join(File.dirname(__FILE__), 'evaltest')) + assert_within(10) do + !p2.chests.empty? + end + + e = Evaltest.new + + p2.stop! + end + def test_write_read 0.upto(100) do |n| @p["#{n}"] = "#{n}" From nobody at rubyforge.org Mon Nov 13 18:06:07 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 18:06:07 -0500 (EST) Subject: [Archipelago-submits] [12] trunk/archipelago/lib: moved all chunky source files into an archipelago subdir to reduce the require-namespace-problems Message-ID: <20061113230607.C5B7052417CA@rubyforge.org> Revision: 12 Author: zond Date: 2006-11-13 18:06:06 -0500 (Mon, 13 Nov 2006) Log Message: ----------- moved all chunky source files into an archipelago subdir to reduce the require-namespace-problems Added Paths: ----------- trunk/archipelago/lib/archipelago/disco.rb trunk/archipelago/lib/archipelago/hashish.rb trunk/archipelago/lib/archipelago/pirate.rb trunk/archipelago/lib/archipelago/tranny.rb trunk/archipelago/lib/archipelago/treasure.rb Removed Paths: ------------- trunk/archipelago/lib/disco.rb trunk/archipelago/lib/hashish.rb trunk/archipelago/lib/pirate.rb trunk/archipelago/lib/tranny.rb trunk/archipelago/lib/treasure.rb Copied: trunk/archipelago/lib/archipelago/disco.rb (from rev 11, trunk/archipelago/lib/disco.rb) =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb (rev 0) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-13 23:06:06 UTC (rev 12) @@ -0,0 +1,548 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'socket' +require 'thread' +require 'ipaddr' +require 'pp' +require 'archipelago/current' +require 'drb' +require 'set' + +module Archipelago + + module Disco + + # + # Default address to use. + # + ADDRESS = "234.2.4.2" + # + # Default port to use. + # + PORT = 25242 + # + # Default port range to use for unicast. + # + UNIPORTS = 25243..26243 + # + # Default lookup timeout. + # + LOOKUP_TIMEOUT = 10 + # + # Default initial pause between resending lookup queries. + # Will be doubled for each resend. + # + INITIAL_LOOKUP_STANDOFF = 0.1 + # + # Default pause between trying to validate all services we + # know about. + # + VALIDATION_INTERVAL = 60 + # + # Only save stuff that we KNOW we want. + # + THRIFTY_CACHING = true + # + # Only reply to the one actually asking about a service. + # + THRIFTY_REPLYING = true + # + # Dont send on publish, only on query. + # + THRIFTY_PUBLISHING = false + + # + # A module to simplify publishing services. + # + # If you include it you can use the publish! method + # at your convenience. + # + # If you want to customize the publishing related behaviour you can + # call initialize_publishable with a Hash of options. + # + # See Archipelago::Treasure::Chest or Archipelago::Tranny::Manager for examples. + # + # It will store the service_id of this service in a directory beside this + # file (publishable.rb) named as the class you include into unless you + # define @persistence_provider before you call initialize_publishable. + # + module Publishable + + # + # Will initialize this instance with @service_description and @jockey_options + # and merge these with the optionally given :service_description and + # :jockey_options. + # + def initialize_publishable(options = {}) + @service_description = { + :service_id => service_id, + :validator => DRbObject.new(self), + :service => DRbObject.new(self), + :class => self.class.name + }.merge(options[:service_description] || {}) + @jockey_options = options[:jockey_options] || {} + end + + # + # Create an Archipelago::Disco::Jockey for this instance using @jockey_options + # or optionally given :jockey_options. + # + # Will publish this service using @service_description or optionally given + # :service_description. + # + def publish!(options = {}) + @jockey ||= Archipelago::Disco::Jockey.new(@jockey_options.merge(options[:jockey_options] || {})) + @jockey.publish(Archipelago::Disco::Record.new(@service_description.merge(options[:service_description] || {}))) + end + + # + # We are always valid if we are able to reply. + # + def valid? + true + end + + # + # Returns our semi-unique id so that we can be found again. + # + def service_id + # + # The provider of happy magic persistent hashes of different kinds. + # + @persistence_provider ||= Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join(self.class.name + ".db")) + # + # Stuff that didnt fit in any of the other databases. + # + @metadata ||= @persistence_provider.get_hashish("metadata") + service_id = @metadata["service_id"] + unless service_id + host = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" + service_id = @metadata["service_id"] ||= Digest::SHA1.hexdigest("#{host}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") + end + return service_id + end + + end + + # + # A mock validator to be used for dumb systems that dont want + # to validate. + # + class MockValidator + def valid? + true + end + end + + # + # A Hash-like description of a service. + # + class ServiceDescription + IGNORABLE_ATTRIBUTES = Set[:unicast_reply] + attr_reader :attributes + # + # Initialize this service description with a hash + # that describes its attributes. + # + def initialize(hash = {}) + @attributes = hash + end + # + # Forwards as much as possible to our Hash. + # + def method_missing(meth, *args, &block) + if @attributes.respond_to?(meth) + if block + @attributes.send(meth, *args, &block) + else + @attributes.send(meth, *args) + end + else + super(*args) + end + end + # + # Returns whether this ServiceDescription matches the given +match+. + # + def matches?(match) + match.each do |key, value| + unless IGNORABLE_ATTRIBUTES.include?(key) + return false unless @attributes.include?(key) && (value.nil? || @attributes[key] == value) + end + end + true + end + end + + # + # A class used to query the Disco network for services. + # + class Query < ServiceDescription + end + + # + # A class used to define an existing service. + # + class Record < ServiceDescription + # + # Initialize this Record with a hash that must contain an :service_id and a :validator. + # + def initialize(hash) + raise "Record must have an :service_id" unless hash.include?(:service_id) + raise "Record must have a :validator" unless hash.include?(:validator) + super(hash) + end + # + # Returns whether this service is still valid. + # + def valid? + begin + self[:validator].valid? + rescue DRb::DRbError => e + false + end + end + end + + # + # A container of services. + # + class ServiceLocker + attr_reader :hash + include Archipelago::Current::Synchronized + def initialize(hash = nil) + super + @hash = hash || {} + end + # + # Merge this locker with another. + # + def merge(sd) + rval = @hash.clone + rval.merge!(sd.hash) + ServiceLocker.new(rval) + end + # + # Forwards as much as possible to our Hash. + # + def method_missing(meth, *args, &block) + if @hash.respond_to?(meth) + synchronize do + if block + @hash.send(meth, *args, &block) + else + @hash.send(meth, *args) + end + end + else + super(*args) + end + end + # + # Find all containing services matching +match+. + # + def get_services(match) + rval = ServiceLocker.new + self.each do |service_id, service_data| + rval[service_id] = service_data if service_data.matches?(match) && service_data.valid? + end + return rval + end + # + # Remove all non-valid services. + # + def validate! + self.clone.each do |service_id, service_data| + self.delete(service_id) unless service_data.valid? + end + end + end + + # + # The main discovery class used to both publish and lookup services. + # + class Jockey + + attr_reader :new_service_semaphore + + # + # Will create a Jockey service running on :address and :port or + # ADDRESS and PORT if none are given. + # + # Will the first available unicast port within :uniports or if not given UNIPORTS for receiving unicast messages. + # + # Will have a default :lookup_timeout of LOOKUP_TIMEOUT, a default + # :initial_lookup_standoff of INITIAL_LOOKUP_STANDOFF and a default + # :validation_interval of VALIDATION_INTERVAL. + # + # Will only cache (and validate, which saves network traffic) stuff + # that has been looked up before if :thrifty_caching, or THRIFTY_CACHING if not given. + # + # Will only reply to the one that sent out the query (and therefore save lots of network traffic) + # if :thrifty_replying, or THRIFTY_REPLYING if not given. + # + # Will send out a multicast when a new service is published unless :thrifty_publishing, or + # THRIFTY_PUBLISHING if not given. + # + # Will reply to all queries to which it has matching local services with a unicast message if :thrifty_replying, + # or if not given THRIFTY_REPLYING. Otherwise will reply with multicasts. + # + def initialize(options = {}) + @thrifty_caching = options.include?(:thrifty_caching) ? options[:thrifty_caching] : THRIFTY_CACHING + @thrifty_replying = options.include?(:thrifty_replying) ? options[:thrifty_replying] : THRIFTY_REPLYING + @thrifty_publishing = options.include?(:thrifty_publishing) ? options[:thrifty_publishing] : THRIFTY_PUBLISHING + @lookup_timeout = options[:lookup_timeout] || LOOKUP_TIMEOUT + @initial_lookup_standoff = options[:initial_lookup_standoff] || INITIAL_LOOKUP_STANDOFF + + @remote_services = ServiceLocker.new + @local_services = ServiceLocker.new + @subscribed_services = Set.new + + @incoming = Queue.new + @outgoing = Queue.new + + @new_service_semaphore = MonitorMixin::ConditionVariable.new(Archipelago::Current::Lock.new) + + @listener = UDPSocket.new + @unilistener = UDPSocket.new + + @listener.setsockopt(Socket::IPPROTO_IP, + Socket::IP_ADD_MEMBERSHIP, + IPAddr.new(options[:address] || ADDRESS).hton + Socket.gethostbyname("0.0.0.0")[3]) + + @listener.setsockopt(Socket::SOL_SOCKET, + Socket::SO_REUSEADDR, + true) + begin + @listener.setsockopt(Socket::SOL_SOCKET, + Socket::SO_REUSEPORT, + true) + rescue + # /moo + end + @listener.bind('', options[:port] || PORT) + + uniports = options[:uniports] || UNIPORTS + this_port = uniports.min + begin + @unilistener.bind('', this_port) + rescue Errno::EADDRINUSE => e + if this_port < uniports.max + this_port += 1 + retry + else + raise e + end + end + @unicast_address = "#{Socket::gethostbyname(Socket::gethostname)[0]}:#{this_port}" rescue "localhost:#{this_port}" + + @sender = UDPSocket.new + @sender.connect(options[:address] || ADDRESS, options[:port] || PORT) + + @unisender = UDPSocket.new + + start_listener + start_unilistener + start_shouter + start_picker + start_validator(options[:validation_interval] || VALIDATION_INTERVAL) + end + + # + # Stops all the threads in this instance. + # + def stop + @listener_thread.kill + @unilistener_thread.kill + @shouter_thread.kill + @picker_thread.kill + @validator_thread.kill + end + + # + # Lookup any services matching +match+, optionally with a +timeout+. + # + # Will immediately return if we know of matching and valid services, + # will otherwise send out regular Queries and return as soon as + # matching services are found, or when the +timeout+ runs out. + # + def lookup(match, timeout = @lookup_timeout) + match[:unicast_reply] = @unicast_address + @subscribed_services << match if @thrifty_caching + standoff = @initial_lookup_standoff + + @outgoing << [nil, match] + known_services = @remote_services.get_services(match).merge(@local_services.get_services(match)) + return known_services unless known_services.empty? + + @new_service_semaphore.wait(standoff) + standoff *= 2 + + t = Time.new + while Time.new < t + timeout + known_services = @remote_services.get_services(match).merge(@local_services.get_services(match)) + return known_services unless known_services.empty? + + @new_service_semaphore.wait(standoff) + standoff *= 2 + + @outgoing << [nil, match] + end + + ServiceLocker.new + end + + # + # Record the given +service+ and broadcast about it. + # + def publish(service) + if service.valid? + @local_services[service[:service_id]] = service + @new_service_semaphore.broadcast + unless @thrifty_publishing + @outgoing << [nil, service] + end + end + end + + private + + # + # Start the validating thread. + # + def start_validator(validation_interval) + @validator_thread = Thread.new do + loop do + begin + @local_services.validate! + @remote_services.validate! + sleep(validation_interval) + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Start the thread sending Records and Queries + # + def start_shouter + @shouter_thread = Thread.new do + loop do + begin + recipient, data = @outgoing.pop + if recipient + address, port = recipient.split(/:/) + @unisender.send(Marshal.dump(data), 0, address, port.to_i) + else + begin + @sender.write(Marshal.dump(data)) + rescue Errno::ECONNREFUSED => e + retry + end + end + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Start the thread receiving Records and Queries + # + def start_listener + @listener_thread = Thread.new do + loop do + begin + @incoming << Marshal.load(@listener.recv(1024)) + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Start the thread receiving Records and Queries + # on unicast. + # + def start_unilistener + @unilistener_thread = Thread.new do + loop do + begin + @incoming << Marshal.load(@unilistener.recv(1024)) + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Start the thread picking incoming Records and Queries and + # handling them properly + # + def start_picker + @picker_thread = Thread.new do + loop do + begin + data = @incoming.pop + if Archipelago::Disco::Query === data + @local_services.get_services(data).each do |service_id, service_data| + if @thrifty_replying + @outgoing << [data[:unicast_reply], service_data] + else + @outgoing << [nil, service_data] + end + end + elsif Archipelago::Disco::Record === data + if interesting?(data) && data.valid? + @remote_services[data[:service_id]] = data + @new_service_semaphore.broadcast + end + end + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Are we generous in our caching, or have we been + # asked about this type of +publish+ before? + # + def interesting?(publish) + @subscribed_services.each do |subscribed| + return true if publish.matches?(subscribed) + end + return !@thrifty_caching + end + + end + + end + +end Copied: trunk/archipelago/lib/archipelago/hashish.rb (from rev 11, trunk/archipelago/lib/hashish.rb) =================================================================== --- trunk/archipelago/lib/archipelago/hashish.rb (rev 0) +++ trunk/archipelago/lib/archipelago/hashish.rb 2006-11-13 23:06:06 UTC (rev 12) @@ -0,0 +1,199 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'archipelago/current' +require 'bdb' + +module Archipelago + + # + # The module containing the persistence default provider. + # + module Hashish + + # + # In essence a Berkeley Database backed Hash. + # + # Will cache all values having been written or read + # in a normal Hash cache for fast access. + # + # Will save the last update timestamp for all keys + # in a separate Hash cache AND a separate Berkeley Database. + # + class BerkeleyHashish + include Archipelago::Current::Synchronized + # + # Initialize an instance with the +name+ and BDB::Env +env+. + # + def initialize(name, env) + super() + @content_db = env.open_db(BDB::HASH, name, "content", BDB::CREATE) + @content = {} + @timestamps_db = env.open_db(BDB::HASH, name, "timestamps", BDB::CREATE | BDB::NOMMAP) + @timestamps = {} + @lock = Archipelago::Current::Lock.new + end + # + # Returns a deep ( Marshal.load(Marshal.dump(o)) ) clone + # of the object represented by +key+. + # + def get_deep_clone(key) + return Marshal.load(@content_db[Marshal.dump(key)]) + end + # + # Simply get the value for the +key+. + # + def [](key) + @lock.synchronize_on(key) do + + value = @content[key] + return value if value + + return get_from_db(key) + + end + end + # + # Insert +value+ under +key+. + # + def []=(key, value) + @lock.synchronize_on(key) do + + @content[key] = value + + write_to_db(key, Marshal.dump(key), Marshal.dump(value)) + + return value + + end + end + # + # Stores whatever is under +key+ if it is not the same as + # whats in the persistent db. + # + def store_if_changed(key) + @lock.synchronize_on(key) do + + serialized_key = Marshal.dump(key) + serialized_value = Marshal.dump(@content[key]) + + write_to_db(key, serialized_key, serialized_value) if @content_db[serialized_key] != serialized_value + + end + end + # + # Returns the last time the value under +key+ was changed. + # + def timestamp(key) + @lock.synchronize_on(key) do + + timestamp = @timestamps[key] + return timestamp if timestamp + + serialized_key = Marshal.dump(key) + serialized_timestamp = @timestamps_db[serialized_key] + return nil unless serialized_timestamp + + timestamp = Marshal.load(serialized_timestamp) + @timestamps[key] = timestamp + return timestamp + + end + end + # + # Delete +key+ and its value and timestamp. + # + def delete(key) + @lock.synchronize_on(key) do + + serialized_key = Marshal.dump(key) + + @content.delete(key) + @content_db[serialized_key] = nil + @timestamps.delete(key) + @timestamps_db[serialized_key] = nil + + end + end + + private + + # + # Write +key+, serialized as +serialized_key+ and + # +serialized_value+ to the db. + # + def write_to_db(key, serialized_key, serialized_value) + now = Time.now + + @content_db[serialized_key] = serialized_value + @timestamps_db[serialized_key] = Marshal.dump(now) + @timestamps[key] = now + end + + # + # Read +key+ from db and if it is found + # put it in the cache Hash. + # + def get_from_db(key) + serialized_key = Marshal.dump(key) + serialized_value = @content_db[serialized_key] + return nil unless serialized_value + + value = Marshal.load(serialized_value) + @content[key] = value + return value + end + end + + # + # A simple persistence provider backed by Berkeley db. + # + class BerkeleyHashishProvider + # + # Initialize an instance with the given + # +env_path+ to its database dir. + # + def initialize(env_path) + env_path.mkpath + @env = BDB::Env.open(env_path, BDB::CREATE | BDB::INIT_MPOOL) + end + # + # Returns a cleverly cached (but slightly inefficient) + # hash-like instance (see Archipelago::Hashish::BerkeleyHashish) + # using +name+. + # + def get_cached_hashish(name) + BerkeleyHashish.new(name, @env) + end + # + # Returns a normal hash-like instance using +name+. + # + def get_hashish(name) + @env.open_db(BDB::HASH, name, nil, BDB::CREATE | BDB::NOMMAP) + end + # + # Removes the persistent files of this instance. + # + def unlink + p = Pathname.new(@env.home) + p.rmtree if p.exist? + end + end + + end + +end Copied: trunk/archipelago/lib/archipelago/pirate.rb (from rev 11, trunk/archipelago/lib/pirate.rb) =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb (rev 0) +++ trunk/archipelago/lib/archipelago/pirate.rb 2006-11-13 23:06:06 UTC (rev 12) @@ -0,0 +1,229 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'archipelago/disco' +require 'archipelago/tranny' +require 'archipelago/treasure' +require 'pp' +require 'drb' +require 'digest/sha1' + +module Archipelago + + # + # The client library that knows about + # all remote databases and handles reads + # and write to them. + # + module Pirate + + # + # Raised when you try to begin a transaction but have no managers + # available. + # + class NoTransactionManagerAvailableException < RuntimeError + def initialize(pirate) + super("#{pirate} can not find any transaction manager for you") + end + end + + # + # Raised when you try to do stuff without any remote database + # available. + # + class NoRemoteDatabaseAvailableException < RuntimeError + def initialize(pirate) + super("#{pirate} can not find any remote database for you") + end + end + + INITIAL_SERVICE_UPDATE_INTERVAL = 1 + MAXIMUM_SERVICE_UPDATE_INTERVAL = 60 + CHEST_DESCRIPTION = { + :class => 'Archipelago::Treasure::Chest' + } + TRANNY_DESCRIPTION = { + :class => 'Archipelago::Tranny::Manager' + } + + # + # The class that actually keeps track of the Archipelago::Treasure:Chests and the + # Archipelago::Treasure:Dubloons in them. + # + class Captain + attr_reader :chests, :treasure_map, :trannies, :transaction + # + # Initialize an instance using an Archipelago::Disco::Jockey with :jockey_options + # if given, that looks for new services :initial_service_update_interval or INITIAL_SERVICE_UPDATE_INTERVAL, + # when it starts and never slower than every :maximum_service_update_interval or MAXIMUM_SERVICE_UPDATE_INTERVAL. + # + # Will look for Archipelago::Treasure::Chests matching :chest_description or CHEST_DESCRIPTION and + # Archipelago::Tranny::Managers matching :tranny_description or TRANNY_DESCRIPTION. + # + def initialize(options = {}) + @treasure_map = Archipelago::Disco::Jockey.new(options[:jockey_options] || {}) + + @chest_description = CHEST_DESCRIPTION.merge(options[:chest_description] || {}) + @tranny_description = TRANNY_DESCRIPTION.merge(options[:tranny_description] || {}) + @jockey_options = options[:jockey_options] || {} + + start_service_updater(options[:initial_service_update_interval] || INITIAL_SERVICE_UPDATE_INTERVAL, + options[:maximum_service_update_interval] || MAXIMUM_SERVICE_UPDATE_INTERVAL) + + @yar_counter = 0 + + @transaction = nil + end + + # + # Get a value from the distributed database network using a +key+, + # optionally within a +transaction+. + # + def [](key, transaction = nil) + responsible_chest(key)[:service][key, transaction || @transaction] + end + + # + # Write a value to the distributed database network, + # optionally inside a transaction. + # + # Usage: + # * p["my key", transaction] = value + # * p["my key"] = value + # + def []=(key, p1, p2 = nil) + if @transaction && p2.nil? + p2 = p1 + p1 = @transaction + end + responsible_chest(key)[:service][key, p1] = p2 + end + + # + # Delete a value from the distributed database network, + # optionally inside a transaction. + # + def delete(key, transaction = nil) + responsible_chest(key)[:service].delete(key, transaction || @transaction) + end + + # + # Return a clone of this instance bound to a newly created transaction. + # + def begin + raise NoTransactionManagerAvailableException.new(self) if @trannies.empty? + + rval = self.clone + rval.instance_eval do + @transaction = @trannies.values.first[:service].begin + end + + return rval + end + + # + # Execute +block+ within a transaction. + # + # Will commit! transaction after the block is finished unless + # the transaction is aborted or commited already. + # + # Will abort! the transaction if any exception is raised. + # + def transaction(&block) #:yields: transaction + raise NoTransactionManagerAvailableException.new(self) if @trannies.empty? + + @transaction = @trannies.values.first[:service].begin + begin + yield(@transaction) + @transaction.commit! if @transaction.state == :active + rescue Exception => e + @transaction.abort! unless @transaction.state == :aborted + puts e + pp e.backtrace + ensure + @transaction = nil + end + end + + # + # Commit the transaction we are a member of and forget about it. + # + def commit! + @transaction.commit! + @transaction = nil + end + + # + # Abort the transaction we are a member of and forget about it. + # + def abort! + @transaction.abort! + @transaction = nil + end + + # + # Yarrr! + # + def yar! + @yar_counter += 1 + 'yar!' + end + + private + + # + # Get the chest responsible for +key+. + # + def responsible_chest(key) + raise NoRemoteDatabaseAvailableException.new(self) if @chests.empty? + + key_id = Digest::SHA1.new(Marshal.dump(key)).to_s + sorted_chest_ids = @chests.keys.sort + sorted_chest_ids.each do |id| + return @chests[id] if id > key_id + end + return @chests[sorted_chest_ids.first] + end + + # + # Start a thread looking up existing chests between every + # +initial+ and +maximum+ seconds. + # + def start_service_updater(initial, maximum) + @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) + @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) + Thread.start do + standoff = initial + loop do + begin + sleep(standoff) + standoff *= 2 + standoff = maximum if standoff > maximum + @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) + @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + end + + end + +end Copied: trunk/archipelago/lib/archipelago/tranny.rb (from rev 11, trunk/archipelago/lib/tranny.rb) =================================================================== --- trunk/archipelago/lib/archipelago/tranny.rb (rev 0) +++ trunk/archipelago/lib/archipelago/tranny.rb 2006-11-13 23:06:06 UTC (rev 12) @@ -0,0 +1,651 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'bdb' +require 'pathname' +require 'drb' +require 'archipelago/current' +require 'digest/sha1' +require 'archipelago/disco' +require 'archipelago/hashish' + +module Archipelago + + module Tranny + + # + # Time before any Transaction will be automatically aborted. + # + TRANSACTION_TIMEOUT = 60 * 60 + + # + # If a member tries to join a transaction that it has allready joined + # it will receive this. + # + class JoinCountException < RuntimeError + def initialize(member, transaction) + super("#{member.inspect} has allready joined the transaction #{transaction.inspect}") + end + end + + # + # If a member tries to commit a transaction not in :active state + # or abort a transaction in :commited state, or any other grave + # state offence + # + class IllegalOperationException < RuntimeError + def initialize(operation, transaction) + super("#{operation.inspect} is illegal for #{transaction.inspect}") + end + end + + # + # If a member tries to get a transaction that the manager doesnt know about + # it gets this. + # + class UnknownTransactionException < RuntimeError + def initialize(transaction_id, manager) + super("#{manager.inspect} doesnt know about any transaction with id #{transaction_id.inspect}") + end + end + + # + # If a member tries to report its state and the manager doesnt know about the + # member, it gets this. + # + class UnknownMemberException < RuntimeError + def initialize(member, transaction) + super("#{transaction.inspect} doesnt know about any member like #{member.inspect}") + end + end + + # + # If a member misbehaves (or the Transaction is completely fucked up) this will be raised + # + class IllegalStateException < RuntimeError + def initialize(transaction) + super("#{transaction.inspect} is in a completely fucked up state") + end + end + + # + # The manager itself. + # + # This will be the drb exported object that participants talk to, + # either directly or through a TransactionProxy. + # + # See also the TransactionProxy and Transaction classes. + # + class Manager + + include DRb::DRbUndumped + include Archipelago::Disco::Publishable + + attr_accessor :error_logger + attr_accessor :transaction_timeout + + # + # Will use a BerkeleyHashishProvider using tranny_manager.db in the same dir to get its hashes + # if not :persistence_provider is given. + # + # Will create Transactions timing out after :transaction_timeout seconds or TRANSACTION_TIMEOUT + # if none is given. + # + # Will use Archipelago::Disco::Publishable by calling initialize_publishable with +options+. + # + def initialize(options = {}) + @persistence_provider = options[:persistence_provider] || Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join("tranny_manager.db")) + + initialize_publishable(options) + + @transaction_timeout = options[:transaction_timeout] || TRANSACTION_TIMEOUT + + @metadata = @persistence_provider.get_hashish("metadata") + @db = @persistence_provider.get_cached_hashish("db") + end + + # + # Returns a proxy to a newly created Transaction. + # + # The Transaction will timeout after :timeout seconds + # or @transaction_timeout if none is given. + # + def begin(options = {}) + options[:manager] = self + options[:timeout] ||= @transaction_timeout + return Transaction.new(options).proxy + end + + # + # Used by transactions to notify this manager + # about an error. Delegates the actual logging + # to @error_logger.call(exception). + # + # Set @error_logger or override this method + # if you want to log any errors. + # + def log_error(exception) + self.error_logger.call(exception) if self.error_logger + end + + # + # Used by a +transaction+ to store its state for future reference. + # + def store_transaction!(transaction) + @db[transaction.transaction_id] = transaction + end + + # + # Used by a +transaction+ to remove its state when finished. + # + def remove_transaction!(transaction) + @db.delete(transaction.transaction_id) + end + + # + # Used by a transaction proxy to run +meth+ with +args+ on its +transaction_id+. + # + def call_instance_method(transaction_id, meth, *args) + transaction = @db[transaction_id] + if transaction + return transaction.send(meth, *args) + else + raise UnknownTransactionException.new(transaction_id, self) + end + end + + end + + + + + # + # A proxy to a transaction managed by this manager. + # + # Used because standard DRbObjects only use object_id + # to identify objects, while we want the more unique transaction_id. + # + # Will also remember the state of the transaction when it detects + # methods that will terminate it (:commit | :abort). + # + class TransactionProxy + attr_accessor :transaction_id + def initialize(transaction) + @manager = transaction.manager + @transaction_id = transaction.transaction_id + @state = :unknown + end + # + # Return the cached state of the wrapped Archipelago::Tranny::Transaction + # if it is known, otherwise fetch it. + # + def state + if @state == :unknown + method_missing(:state) + else + @state + end + end + # + # Implemented to ensure that all TransactionProxies with the same + # transaction_id are hashwise equal. + # + def hash + @transaction_id.hash + end + # + # Implemented to ensure that all TransactionProxies with the same + # transaction_id are hashwise equal. + # + def eql?(o) + if TransactionProxy === o + o.transaction_id == @transaction_id + else + false + end + end + # + # Forwards everything to our Transaction and remembers + # returnvalue if necessary. + # + def method_missing(meth, *args) #:nodoc: + rval = @manager.call_instance_method(@transaction_id, meth, *args) + case meth + when :abort! + @state = :aborted + when :commit! + @state = rval + end + return rval + end + end + + + + + # + # A transaction managed by the manager. + # + # A transaction can have the following states: + # + # * :active - When it is first created. + # * This transaction can be commited or aborted. + # + # * :voting - When it has started the two-phase commit and the voting has begun. + # * This transaction can be aborted. + # + # * :commited - After everyone has voted and voted either :unchanged or :commit. + # * This transaction can not be changed. + # + # * :aborted - After someone has called abort! or voted :abort in a vote! + # * This transaction can not be changed. + # + # Anyone that wants to join the transaction must implement the following methods: + # + # * abort!(transaction): Abort the provided transaction. + # * commit!(transaction): Commit the provided transaction. + # * prepare!(transaction): Prepare for commiting the provided transaction. + # + # abort! and commit! should not return any values, but can raise exceptions + # if required. + # + # prepare! must return either :abort, :commit or :unchanged, depending on what + # the member is prepared to do. :unchanged is only when the member has not changed + # state during the transaction, and means that it does not require any further + # notification on the progress of the transaction. + # + # A member that has returned :commit on the prepare! must store the transaction + # proxy in a persistent manner to be able to connect to the manager + # and get a new copy of the transaction in case of communications failure. + # + # A member that reconnects to a crashed transaction manager should not abort! the + # transaction if the transaction is still in :active state, since the transaction will + # have been aborted on commit! anyway (since the manager will not be able to prepare! the + # disconnected member after either the manager or member having crashed) if needed. + # If the disconnect was just a temporary networking problem, the transaction will + # continue as planned. + # + # A member that reconnects to a crashed transaction manager where the transaction + # is in :voting state should just wait around and see if it gets prepare! called. + # In case of temporary network failure the transaction will continue as planned, otherwise + # it will abort! automatically. + # + # A member that reconnects to a disconnected transaction manager where the transaction + # is in :commited state should just commit its state. Then it must notify the transaction + # using report_commited! so that the transaction can disappear gracefully. + # + # A member that reconnects to a disconnected transaction manager that either doesnt know + # of the transaction or returns an :aborted transaction may safely abort the state change + # and forget about the transaction. + # + class Transaction + + include Archipelago::Current::Synchronized + + attr_reader :state, :transaction_id, :proxy, :manager + + # + # Create a transaction managed by the provided +manager+. + # + # Will have :manager as TransactionManager, and will timeout + # after :timeout seconds. + # + def initialize(options) + super() + # + # A hash where members are keys and their state the values. + # + @members = {} + @members.extend(Archipelago::Current::Synchronized) + # + # We are alive! + # + @state = :active + # + # We have a timeout! + # + @timeout = options[:timeout] + # + # We are unique! + # + @transaction_id = "#{options[:manager].service_id}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}" + # + # We have a birth time! + # + @created_at = Time.now + # + # We have a manager! + # + self.manager = options[:manager] + + store_us_for_future_reference! + + start_timeout_thread + end + + # + # Store this manager as ours and create a proxy that knows about it. + # + # Used by Archipelago::Tranny:Manager when restoring crashed + # Archipelago::Tranny:Transactions. + # + def manager=(manager) + # + # We have a manager! + # + @manager = DRbObject.new(manager) + # + # We have a proxy to send forth into the world! + # + @proxy = TransactionProxy.new(self) + nil + end + + # + # Special dump call to NOT dump our manager or proxy, + # since DRbObjects dont take kindly to being restored + # in an environment where they are are known to be invalid. + # + def _dump(l) + return Marshal.dump([ + @members, + @state, + @timeout, + @transaction_id, + @created_at + ]) + end + + def self._load(s) + members, state, timeout, transaction_id, created_at = Marshal.load(s) + rval = self.allocate + rval.instance_variable_set(:@members, members) + rval.instance_variable_set(:@state, state) + rval.instance_variable_set(:@timeout, timeout) + rval.instance_variable_set(:@transaction_id, transaction_id) + rval.instance_variable_set(:@created_at, created_at) + rval + end + + # + # Starts the thread that will abort! us automatically + # after we have lived @timeout + # + def start_timeout_thread + Thread.new do + now = Time.now + die_at = @created_at + @timeout + if die_at > now + sleep(die_at - now) + end + abort! + end + nil + end + + # + # What it sounds like. + # + def store_us_for_future_reference! + @manager.store_transaction!(self) + nil + end + + # + # Remove ourselves, we are redundant + # + def remove_us_we_are_redundant! + @manager.remove_transaction!(self) + nil + end + + # + # Used by members that failed during commit. + # + def report_commited!(member) + raise UnknownMemberException(member, self) unless @members.include?(member) + raise IllegalOperationException(:report_commited!, self) unless self.state == :commited + + @members[member] = :commited + + remove_us_if_all_are_commited! + nil + end + + # + # Abort the transaction, sending all participants the abort! message. + # + def abort! + synchronize do + raise IllegalOperationException.new(:abort!, self) if [:commited, :aborted].include?(self.state) + + # + # Set our state. + # + @state = :aborted + + store_us_for_future_reference! + + # + # Abort all members. + # + threads = [] + @members.clone.each do |member, state| + raise RuntimeException.new("This is not supposed to be possible, but we are in abort! with member " + + "#{member.inspect} in state #{state.inspect}") if state == :commited + if state != :aborted + threads << Thread.new do + begin + member.abort!(self.proxy) + @members.synchronize do + @members[member] = :aborted + end + rescue Exception => e + @manager.log_error(e) + # + # We must not let the other members stop just + # because one member failed. No more Mr Nice Guy! + # + end + end + end + end + + # + # Wait for all members to finish. + # + threads.each do |thread| + thread.join + end + + # + # No use having aborted transactions lying about. + # + # NB: This means that disconnected members that cant + # find their old transactions will have to presume they + # have been aborted. + # + remove_us_we_are_redundant! + end + nil + end + + # + # Commits the transaction, returning the new state (:aborted | :commited) + # + def commit! + synchronize do + raise IllegalOperationException.new(:commit!, self) unless self.state == :active + + # + # Vote for the outcome and act on it + # + case vote! + when :abort + abort! + when :commit + _commit! + when :unchanged + @state = :commited + remove_us_we_are_redundant! + end + + return @state + end + nil + end + + # + # Join a +member+ to this transaction. + # + # Will raise a JoinCountException if the given member has allready + # joined this transaction. + # + def join(member) + @members.synchronize do + raise IllegalOperationException.new(:join, self) unless self.state == :active + raise JoinCountException.new(member, self) if @members.include?(member) + + @members[member] = :active + end + store_us_for_future_reference! + nil + end + + private + + # + # Commit the transaction, sending all participants the commit message. + # + # The transaction is commited by all members having commit! called. + # + def _commit! + # + # Set our state. + # + @state = :commited + + store_us_for_future_reference! + + # + # Commit all members. + # + threads = [] + @members.clone.each do |member, state| + raise RuntimeException.new("This is not supposed to be possible, but we are in _commit with member " + + "#{member.inspect} in state #{state.inspect}") unless [:prepared, :commited].include?(state) + threads << Thread.new do + begin + member.commit!(self.proxy) + @members.synchronize do + @members[member] = :commited + end + rescue Exception => e + @manager.log_error(e) + # + # We must not let the other members stop just + # because one member failed. No more Mr Nice Guy! + # + end + end + end + + # + # Wait for all members to finish. + # + threads.each do |thread| + thread.join + end + + remove_us_if_all_are_commited! + end + + def remove_us_if_all_are_commited! + # + # Check to see if all members have been told about the decision. + # + all_have_commited = true + @members.each do |member, state| + all_have_commited = false unless state == :comitted + end + + # + # If they have, remove ourselves from the manager. + # + remove_us_we_are_redundant! if all_have_commited + end + + # + # Let the members of the transaction vote for its outcome. + # + # Members vote by having prepare! called. + # + # Valid returnvalues for the prepare! call are: + # :abort, if the member wants to abort the transaction + # :commit, if the member wants to commit the transaction + # :unchanged, if the member has not changed state during the transaction + # + def vote! + raise IllegalOperationException.new(:vote!, self) unless self.state == :active + + @state = :voting + + store_us_for_future_reference! + + return_value = :commit + + threads = [] + @members.clone.each do |member, state| + raise RuntimeException.new("This is not supposed to be possible, but we are in vote! with member " + + "#{member.inspect} in state #{state.inspect}") unless state == :active + threads << Thread.new do + this_result = nil + begin + this_result = member.prepare!(self.proxy) + rescue Exception => e + @manager.log_error(e) + this_result = :abort + end + @members.synchronize do + case this_result + when :abort + @members[member] = :aborted + return_value = :abort + when :unchanged + @members.delete(member) + else + @members[member] = :prepared + end + end + end + end + + threads.each do |thread| + thread.join + end + + if @members.empty? + return_value = :unchanged + end + + return_value + end + + end + end + +end Copied: trunk/archipelago/lib/archipelago/treasure.rb (from rev 11, trunk/archipelago/lib/treasure.rb) =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb (rev 0) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-13 23:06:06 UTC (rev 12) @@ -0,0 +1,672 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'drb' +require 'archipelago/disco' +require 'bdb' +require 'pathname' +require 'digest/sha1' +require 'pp' +require 'set' +require 'archipelago/hashish' +require 'archipelago/tranny' + +module Archipelago + + module Treasure + + # + # Minimum time between trying to recover + # crashed transactions. + # + TRANSACTION_RECOVERY_INTERVAL = 30 + + # + # Raised whenever the optimistic locking of the serializable transaction isolation level + # was proved wrong. + # + class RollbackException < RuntimeError + def initialize(chest, transaction) + super("#{chest} has been modified during #{transaction}") + end + end + + # + # Raised whenever a method is called on an object that we dont know about. + # + class UnknownObjectException < RuntimeError + def initialize(chest, key, transaction) + super("#{chest} does not contain #{key} under #{transaction}") + end + end + + # + # Raised if a Dubloon or Chest doesnt know what transaction you are talking about. + # + class UnknownTransactionException < RuntimeError + def initialize(source, transaction) + super("#{source} is not a part of #{transaction}") + end + end + + # + # Raised if anyone tries to commit a non-prepared transaction. + # + class IllegalCommitException < RuntimeError + def initialize(source, transaction) + super("#{transaction} is not prepared in #{source}") + end + end + + # + # Raised if someone tries to join us to a non-active transaction. + # + class IllegalJoinException < RuntimeError + def initialize(transaction) + super("#{transaction} is not active") + end + end + + # + # A proxy to something in the chest. + # + class Dubloon + # + # Remove all methods so that we look like our target. + # + instance_methods.each do |method| + undef_method method unless method =~ /^__/ + end + # + # Initialize us with knowledge of our +chest+, the +key+ to our + # target in the +chest+, the known +public_methods+ of our target + # and any +transaction+ we are associated with. + # + def initialize(key, chest, transaction, chest_id, public_methods) + @key = key + @chest = chest + @transaction = transaction + @chest_id = chest_id + @public_methods = public_methods + end + # + # The public_methods of our target. + # + def public_methods + return @public_methods.clone + end + # + # Return a clone of myself that is joined to + # the +transaction+. + # + def join(transaction) + @chest.join!(transaction) if transaction + return Dubloon.new(@key, @chest, transaction, @chest_id, @public_methods) + end + # + # Raises exception if the given +transaction+ + # is not the same as our own. + # + def assert_transaction(transaction) + raise UnknownTransactionException.new(self, transaction) unless transaction == @transaction + end + # + # The object_id of our chest-held target. + # + def object_id + id = "#{@chest_id}:#{@key}" + id << ":#{@transaction.transaction_id}" if @transaction + return id + end + # + # Does our target respond to +meth+? + # + def respond_to?(meth) + return @public_methods.include?(meth.to_sym) || @public_methods.include?(meth.to_s) + end + # + # Call +meth+ with +args+ and +block+ on our target if it responds to + # it. + # + def method_missing(meth, *args, &block) + if respond_to?(meth) + return @chest.call_instance_method(@key, meth, @transaction, *args, &block) + else + return super(meth, *args) + end + end + + end + + # + # A possibly remote database that only returns proxies to its + # contents, and thus runs all methods on its contents itself. + # + # Has support for optimistically locked distributed serializable transactions. + # + class Chest + + # + # The Chest never leaves its host. + # + include DRb::DRbUndumped + + # + # The Chest can be published. + # + include Archipelago::Disco::Publishable + + # + # Initialize a Chest + # + # Will use a BerkeleyHashishProvider using treasure_chest.db in the same dir to get its hashes + # if not :persistence_provider is given. + # + # Will try to recover crashed transaction every :transaction_recovery_interval seconds + # or TRANSACTION_RECOVERY_INTERVAL if none is given. + # + # Will use Archipelago::Disco::Publishable by calling initialize_publishable with +options+. + # + def initialize(options = {}) + # + # The provider of happy magic persistent hashes of different kinds. + # + @persistence_provider = options[:persistence_provider] || Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join("treasure_chest.db")) + + # + # Use the given options to initialize the publishable + # instance variables. + # + initialize_publishable(options) + + # + # [transaction => [key => instance]] + # To know what stuff is visible to a transaction. + # + @snapshot_by_transaction = {} + @snapshot_by_transaction.extend(Archipelago::Current::Synchronized) + + # + # [transaction => [key => when the key was read/updated/deleted] + # To know if a transaction is ok to prepare and commit. + # + @timestamp_by_key_by_transaction = {} + + # + # [transaction => [key => instance]] + # To know what transactions were prepared but not + # properly finished last run. + # + @crashed = Set.new + + # + # The magical persistent map that defines how we actually + # store our data. + # + @db = @persistence_provider.get_cached_hashish("db") + + initialize_prepared(options[:transaction_recovery_interval] || TRANSACTION_RECOVERY_INTERVAL) + + end + + # + # The transactions active in this Chest. + # + def active_transactions + @snapshot_by_transaction.keys.clone + end + + # + # Return the contents of this chest using a given +key+ and +transaction+. + # + def [](key, transaction = nil) + join!(transaction) + + instance = ensure_instance_with_transaction(key, transaction) + return nil unless instance + + if Dubloon === instance + return instance.join(transaction) + else + return Dubloon.new(key, DRbObject.new(self), transaction, self.service_id, instance.public_methods) + end + end + + # + # Delete the value of +key+ within +transaction+. + # + def delete(key, transaction = nil) + join!(transaction) + + rval = nil + + if transaction + # + # If we have a transaction we must note that it is deleted in a + # separate space for that transaction. + # + snapshot = @snapshot_by_transaction[transaction] + snapshot.synchronize do + + rval = snapshot[key] + + snapshot[key] = :deleted + # + # Make sure we remember when it was last changed according to our main db. + # + timestamps = @timestamp_by_key_by_transaction[transaction] + timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) + + end + else + # + # Otherwise just ask our persistence provider to delete it. + # + @db.delete(key) + end + + rval.freeze + + return rval + end + + # + # Put something into this chest with a given +key+, +value+ and + # +transaction+. + # + def []=(key, p1, p2 = nil) + if p2 + value = p2 + transaction = p1 + else + value = p1 + transaction = nil + end + + return set(key, value, transaction) + end + + # + # Call an instance +method+ on whatever this chest holds at +key+ + # with any +transaction+ and +args+. + # + def call_instance_method(key, method, transaction, *arguments, &block) + if transaction + return call_with_transaction(key, method, transaction, *arguments, &block) + else + return call_without_transaction(key, method, *arguments, &block) + end + end + + # + # Abort +transaction+ in this Chest. + # + def abort!(transaction) + assert_transaction(transaction) + + snapshot = @snapshot_by_transaction[transaction] + # + # Make sure nobody can modify this transaction while we + # are aborting it. + # + snapshot.synchronize do + serialized_transaction = Marshal.dump(transaction) + + # + # If this transaction was successfully prepared + # + if @snapshot_by_transaction_db.include?(serialized_transaction) + # + # Unlock the keys that are part of it. + # + snapshot.each do |key, value| + @db.unlock_on(key) + end + # + # And remove it from persistent storage. + # + @snapshot_by_transaction_db[serialized_transaction] = nil + # + # And remove its timestamps from persistent storage. + # + @timestamp_by_key_by_transaction_db[serialized_transaction] = nil + end + # + # Finally delete it from the snapshots. + # + @snapshot_by_transaction.delete(transaction) + # + # And from the timestamps. + # + @timestamp_by_key_by_transaction.delete(transaction) + end + end + + # + # Prepares +transaction+ in this Chest. + # + # NB: This will cause any update of the data within + # this transaction to block until it is either aborted + # or commited! + # + def prepare!(transaction) + assert_transaction(transaction) + + # + # If we dont know about this transaction then it can't very well + # affect us. + # + return :unchanged unless @snapshot_by_transaction.include?(transaction) + + snapshot = @snapshot_by_transaction[transaction] + # + # Make sure nobody can modify this transaction while we are + # preparing it. + # + snapshot.synchronize do + + # + # Remember what locks we acquire so that we can + # unlock them in case of failure. + # + locks = [] + timestamp_by_key = @timestamp_by_key_by_transaction[transaction] + # + # Acquire a lock on each key in the transaction + # + snapshot.each do |key, value| + if @db.timestamp(key) == timestamp_by_key[key] + @db.lock_on(key) + locks << key + else + locks.each do |key| + @db.unlock_on(key) + end + return :abort + end + end + serialized_transaction = Marshal.dump(transaction) + + # + # Dump its state to persistent storage. + # + @snapshot_by_transaction_db[serialized_transaction] = Marshal.dump(snapshot) + # + # And dump its timestamps to persistent storage + # + @timestamp_by_key_by_transaction_db[serialized_transaction] = Marshal.dump(timestamp_by_key) + return :prepared + end + end + + # + # Commits +transaction+ in this Chest. + # + # NB: Transaction must be prepared before commit is called. + # + def commit!(transaction) + assert_transaction(transaction) + raise IllegalCommitException.new(self, transaction) unless @snapshot_by_transaction_db.include?(Marshal.dump(transaction)) + + snapshot = @snapshot_by_transaction[transaction] + # + # Make sure nobody can modify this transaction while we are + # commiting it. + # + snapshot.synchronize do + + # + # Copy each key and value from our private space to the real space + # + snapshot.each do |key, value| + if value == :deleted + @db.delete(key) + else + @db[key] = value + end + end + + # + # Call abort! to clean up after the transaction. + # + abort!(transaction) + + end + end + + private + + # + # Allocates space for this +transaction+. + # + # Will also call +transaction+.join to make sure + # it is aware of us. + # + def join!(transaction) + if transaction + if transaction.state == :active + @snapshot_by_transaction.synchronize do + unless @snapshot_by_transaction.include?(transaction) + @snapshot_by_transaction[transaction] = {} + @snapshot_by_transaction[transaction].extend(Archipelago::Current::Synchronized) + @timestamp_by_key_by_transaction[transaction] = {} + transaction.join(DRbObject.new(self)) + end + end + else + raise IllegalJoinException.new(transaction) + end + end + end + + # + # Raises if we are not in this transaction. + # + def assert_transaction(transaction) + raise UnknownTransactionException.new(self, transaction) unless @snapshot_by_transaction.include?(transaction) + end + + # + # Call a method within a transaction. + # + def call_with_transaction(key, method, transaction, *arguments, &block) + assert_transaction(transaction) + + # + # Fetch our instance from the snapshot. + # + snapshot = @snapshot_by_transaction[transaction] + instance = snapshot[key] + instance = nil if instance == :deleted + + raise UnknownObjectException.new(self, key, transaction) unless instance + + begin + return execute(instance, method, *arguments, &block) + ensure + # + # Make sure we remember when this object was last changed according + # to the main db. + # + snapshot.synchronize do + timestamps = @timestamp_by_key_by_transaction[transaction] + timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) + end + end + end + + # + # Execute +m+ with arguments +a+ and block +b+ on +o+. + # + def execute(o, m, *a, &b) + if b + return o.send(m, *a, &b) + else + return o.send(m, *a) + end + end + + # + # Call a method outside any transaction (ie inside a transaction of its own). + # + def call_without_transaction(key, method, *arguments, &block) + instance = @db[key] + + raise UnknownObjectException(self, key, transaction) unless instance + + begin + return execute(instance, method, *arguments, &block) + ensure + @db.store_if_changed(key) + end + end + + # + # Initializes our storage of prepared transactions. + # + def initialize_prepared(transaction_recovery_interval) + # + # Load stored timestamps for our transaction from db. + # + @timestamp_by_key_by_transaction_db = @persistence_provider.get_hashish("prepared_timestamps") + @timestamp_by_key_by_transaction_db.each do |serialized_transaction, serialized_timestamps| + @timestamp_by_key_by_transaction[Marshal.load(serialized_transaction)] = Marshal.load(serialized_timestamps) + end + + # + # Load stored snapshots for our transaction from db. + # + @snapshot_by_transaction_db = @persistence_provider.get_hashish("prepared") + @snapshot_by_transaction_db.each do |serialized_transaction, serialized_snapshot| + transaction = Marshal.load(transaction) + + @crashed << transaction + @snapshot_by_transaction[transaction] = Marshal.load(serialized_snapshot) + end + start_recovery_thread(transaction_recovery_interval) + end + + # + # Starts the thread that will keep trying to recover + # our crashed transactions. + # + def start_recovery_thread(transaction_recovery_interval) + Thread.new do + loop do + begin + @crashed.clone.each do |transaction| + begin + case transaction.state + when :commited + commit!(transaction) + @crashed.delete(transaction) + when :aborted + abort!(transaction) + @crashed.delete(transaction) + end + rescue Archipelago::Tranny::UnknownTransactionException => e + abort!(transaction) + @crashed.delete(transaction) + end + end + sleep(transaction_recovery_interval) + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Insert +value+ under +key+ and +transaction+ + # in this chest. + # + def set(key, value, transaction) + join!(transaction) + value.assert_transaction(transaction) if Dubloon === value + + if transaction + snapshot = @snapshot_by_transaction[transaction] + + # + # If we have a transaction we must put it in a + # separate space for that transaction. + # + snapshot.synchronize do + + snapshot[key] = value + # + # Make sure we remember the last time this was changed according to + # our main db. + # + timestamps = @timestamp_by_key_by_transaction[transaction] + timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) + + end + else + @db[key] = value + end + + return value if Dubloon === value + + return Dubloon.new(key, DRbObject.new(self), transaction, service_id, value.public_methods) + end + + # + # Try to fetch the data of +key+ from the private space + # of +transaction+ and put it there if it was not there + # already. + # + def ensure_instance_with_transaction(key, transaction) + if transaction + snapshot = @snapshot_by_transaction[transaction] + snapshot.synchronize do + + # + # If we dont have this key in the snapshot. + # + unless snapshot.include?(key) + # + # Fetch the new value for the snapshot + # + new_value = @db.get_deep_clone(key) + # + # If it exists then copy it to the snapshot + # otherwise remove the transaction hash if it is empty. + # + if new_value + snapshot[key] = new_value + else + @snapshot_by_transaction.delete(transaction) if snapshot.empty? + end + end + + rval = snapshot[key] + return rval == :deleted ? nil : rval + + end + else + return @db[key] + end + end + + end + + end + +end Deleted: trunk/archipelago/lib/disco.rb =================================================================== --- trunk/archipelago/lib/disco.rb 2006-11-13 23:04:26 UTC (rev 11) +++ trunk/archipelago/lib/disco.rb 2006-11-13 23:06:06 UTC (rev 12) @@ -1,548 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'socket' -require 'thread' -require 'ipaddr' -require 'pp' -require 'archipelago/current' -require 'drb' -require 'set' - -module Archipelago - - module Disco - - # - # Default address to use. - # - ADDRESS = "234.2.4.2" - # - # Default port to use. - # - PORT = 25242 - # - # Default port range to use for unicast. - # - UNIPORTS = 25243..26243 - # - # Default lookup timeout. - # - LOOKUP_TIMEOUT = 10 - # - # Default initial pause between resending lookup queries. - # Will be doubled for each resend. - # - INITIAL_LOOKUP_STANDOFF = 0.1 - # - # Default pause between trying to validate all services we - # know about. - # - VALIDATION_INTERVAL = 60 - # - # Only save stuff that we KNOW we want. - # - THRIFTY_CACHING = true - # - # Only reply to the one actually asking about a service. - # - THRIFTY_REPLYING = true - # - # Dont send on publish, only on query. - # - THRIFTY_PUBLISHING = false - - # - # A module to simplify publishing services. - # - # If you include it you can use the publish! method - # at your convenience. - # - # If you want to customize the publishing related behaviour you can - # call initialize_publishable with a Hash of options. - # - # See Archipelago::Treasure::Chest or Archipelago::Tranny::Manager for examples. - # - # It will store the service_id of this service in a directory beside this - # file (publishable.rb) named as the class you include into unless you - # define @persistence_provider before you call initialize_publishable. - # - module Publishable - - # - # Will initialize this instance with @service_description and @jockey_options - # and merge these with the optionally given :service_description and - # :jockey_options. - # - def initialize_publishable(options = {}) - @service_description = { - :service_id => service_id, - :validator => DRbObject.new(self), - :service => DRbObject.new(self), - :class => self.class.name - }.merge(options[:service_description] || {}) - @jockey_options = options[:jockey_options] || {} - end - - # - # Create an Archipelago::Disco::Jockey for this instance using @jockey_options - # or optionally given :jockey_options. - # - # Will publish this service using @service_description or optionally given - # :service_description. - # - def publish!(options = {}) - @jockey ||= Archipelago::Disco::Jockey.new(@jockey_options.merge(options[:jockey_options] || {})) - @jockey.publish(Archipelago::Disco::Record.new(@service_description.merge(options[:service_description] || {}))) - end - - # - # We are always valid if we are able to reply. - # - def valid? - true - end - - # - # Returns our semi-unique id so that we can be found again. - # - def service_id - # - # The provider of happy magic persistent hashes of different kinds. - # - @persistence_provider ||= Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join(self.class.name + ".db")) - # - # Stuff that didnt fit in any of the other databases. - # - @metadata ||= @persistence_provider.get_hashish("metadata") - service_id = @metadata["service_id"] - unless service_id - host = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" - service_id = @metadata["service_id"] ||= Digest::SHA1.hexdigest("#{host}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") - end - return service_id - end - - end - - # - # A mock validator to be used for dumb systems that dont want - # to validate. - # - class MockValidator - def valid? - true - end - end - - # - # A Hash-like description of a service. - # - class ServiceDescription - IGNORABLE_ATTRIBUTES = Set[:unicast_reply] - attr_reader :attributes - # - # Initialize this service description with a hash - # that describes its attributes. - # - def initialize(hash = {}) - @attributes = hash - end - # - # Forwards as much as possible to our Hash. - # - def method_missing(meth, *args, &block) - if @attributes.respond_to?(meth) - if block - @attributes.send(meth, *args, &block) - else - @attributes.send(meth, *args) - end - else - super(*args) - end - end - # - # Returns whether this ServiceDescription matches the given +match+. - # - def matches?(match) - match.each do |key, value| - unless IGNORABLE_ATTRIBUTES.include?(key) - return false unless @attributes.include?(key) && (value.nil? || @attributes[key] == value) - end - end - true - end - end - - # - # A class used to query the Disco network for services. - # - class Query < ServiceDescription - end - - # - # A class used to define an existing service. - # - class Record < ServiceDescription - # - # Initialize this Record with a hash that must contain an :service_id and a :validator. - # - def initialize(hash) - raise "Record must have an :service_id" unless hash.include?(:service_id) - raise "Record must have a :validator" unless hash.include?(:validator) - super(hash) - end - # - # Returns whether this service is still valid. - # - def valid? - begin - self[:validator].valid? - rescue DRb::DRbError => e - false - end - end - end - - # - # A container of services. - # - class ServiceLocker - attr_reader :hash - include Archipelago::Current::Synchronized - def initialize(hash = nil) - super - @hash = hash || {} - end - # - # Merge this locker with another. - # - def merge(sd) - rval = @hash.clone - rval.merge!(sd.hash) - ServiceLocker.new(rval) - end - # - # Forwards as much as possible to our Hash. - # - def method_missing(meth, *args, &block) - if @hash.respond_to?(meth) - synchronize do - if block - @hash.send(meth, *args, &block) - else - @hash.send(meth, *args) - end - end - else - super(*args) - end - end - # - # Find all containing services matching +match+. - # - def get_services(match) - rval = ServiceLocker.new - self.each do |service_id, service_data| - rval[service_id] = service_data if service_data.matches?(match) && service_data.valid? - end - return rval - end - # - # Remove all non-valid services. - # - def validate! - self.clone.each do |service_id, service_data| - self.delete(service_id) unless service_data.valid? - end - end - end - - # - # The main discovery class used to both publish and lookup services. - # - class Jockey - - attr_reader :new_service_semaphore - - # - # Will create a Jockey service running on :address and :port or - # ADDRESS and PORT if none are given. - # - # Will the first available unicast port within :uniports or if not given UNIPORTS for receiving unicast messages. - # - # Will have a default :lookup_timeout of LOOKUP_TIMEOUT, a default - # :initial_lookup_standoff of INITIAL_LOOKUP_STANDOFF and a default - # :validation_interval of VALIDATION_INTERVAL. - # - # Will only cache (and validate, which saves network traffic) stuff - # that has been looked up before if :thrifty_caching, or THRIFTY_CACHING if not given. - # - # Will only reply to the one that sent out the query (and therefore save lots of network traffic) - # if :thrifty_replying, or THRIFTY_REPLYING if not given. - # - # Will send out a multicast when a new service is published unless :thrifty_publishing, or - # THRIFTY_PUBLISHING if not given. - # - # Will reply to all queries to which it has matching local services with a unicast message if :thrifty_replying, - # or if not given THRIFTY_REPLYING. Otherwise will reply with multicasts. - # - def initialize(options = {}) - @thrifty_caching = options.include?(:thrifty_caching) ? options[:thrifty_caching] : THRIFTY_CACHING - @thrifty_replying = options.include?(:thrifty_replying) ? options[:thrifty_replying] : THRIFTY_REPLYING - @thrifty_publishing = options.include?(:thrifty_publishing) ? options[:thrifty_publishing] : THRIFTY_PUBLISHING - @lookup_timeout = options[:lookup_timeout] || LOOKUP_TIMEOUT - @initial_lookup_standoff = options[:initial_lookup_standoff] || INITIAL_LOOKUP_STANDOFF - - @remote_services = ServiceLocker.new - @local_services = ServiceLocker.new - @subscribed_services = Set.new - - @incoming = Queue.new - @outgoing = Queue.new - - @new_service_semaphore = MonitorMixin::ConditionVariable.new(Archipelago::Current::Lock.new) - - @listener = UDPSocket.new - @unilistener = UDPSocket.new - - @listener.setsockopt(Socket::IPPROTO_IP, - Socket::IP_ADD_MEMBERSHIP, - IPAddr.new(options[:address] || ADDRESS).hton + Socket.gethostbyname("0.0.0.0")[3]) - - @listener.setsockopt(Socket::SOL_SOCKET, - Socket::SO_REUSEADDR, - true) - begin - @listener.setsockopt(Socket::SOL_SOCKET, - Socket::SO_REUSEPORT, - true) - rescue - # /moo - end - @listener.bind('', options[:port] || PORT) - - uniports = options[:uniports] || UNIPORTS - this_port = uniports.min - begin - @unilistener.bind('', this_port) - rescue Errno::EADDRINUSE => e - if this_port < uniports.max - this_port += 1 - retry - else - raise e - end - end - @unicast_address = "#{Socket::gethostbyname(Socket::gethostname)[0]}:#{this_port}" rescue "localhost:#{this_port}" - - @sender = UDPSocket.new - @sender.connect(options[:address] || ADDRESS, options[:port] || PORT) - - @unisender = UDPSocket.new - - start_listener - start_unilistener - start_shouter - start_picker - start_validator(options[:validation_interval] || VALIDATION_INTERVAL) - end - - # - # Stops all the threads in this instance. - # - def stop - @listener_thread.kill - @unilistener_thread.kill - @shouter_thread.kill - @picker_thread.kill - @validator_thread.kill - end - - # - # Lookup any services matching +match+, optionally with a +timeout+. - # - # Will immediately return if we know of matching and valid services, - # will otherwise send out regular Queries and return as soon as - # matching services are found, or when the +timeout+ runs out. - # - def lookup(match, timeout = @lookup_timeout) - match[:unicast_reply] = @unicast_address - @subscribed_services << match if @thrifty_caching - standoff = @initial_lookup_standoff - - @outgoing << [nil, match] - known_services = @remote_services.get_services(match).merge(@local_services.get_services(match)) - return known_services unless known_services.empty? - - @new_service_semaphore.wait(standoff) - standoff *= 2 - - t = Time.new - while Time.new < t + timeout - known_services = @remote_services.get_services(match).merge(@local_services.get_services(match)) - return known_services unless known_services.empty? - - @new_service_semaphore.wait(standoff) - standoff *= 2 - - @outgoing << [nil, match] - end - - ServiceLocker.new - end - - # - # Record the given +service+ and broadcast about it. - # - def publish(service) - if service.valid? - @local_services[service[:service_id]] = service - @new_service_semaphore.broadcast - unless @thrifty_publishing - @outgoing << [nil, service] - end - end - end - - private - - # - # Start the validating thread. - # - def start_validator(validation_interval) - @validator_thread = Thread.new do - loop do - begin - @local_services.validate! - @remote_services.validate! - sleep(validation_interval) - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Start the thread sending Records and Queries - # - def start_shouter - @shouter_thread = Thread.new do - loop do - begin - recipient, data = @outgoing.pop - if recipient - address, port = recipient.split(/:/) - @unisender.send(Marshal.dump(data), 0, address, port.to_i) - else - begin - @sender.write(Marshal.dump(data)) - rescue Errno::ECONNREFUSED => e - retry - end - end - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Start the thread receiving Records and Queries - # - def start_listener - @listener_thread = Thread.new do - loop do - begin - @incoming << Marshal.load(@listener.recv(1024)) - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Start the thread receiving Records and Queries - # on unicast. - # - def start_unilistener - @unilistener_thread = Thread.new do - loop do - begin - @incoming << Marshal.load(@unilistener.recv(1024)) - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Start the thread picking incoming Records and Queries and - # handling them properly - # - def start_picker - @picker_thread = Thread.new do - loop do - begin - data = @incoming.pop - if Archipelago::Disco::Query === data - @local_services.get_services(data).each do |service_id, service_data| - if @thrifty_replying - @outgoing << [data[:unicast_reply], service_data] - else - @outgoing << [nil, service_data] - end - end - elsif Archipelago::Disco::Record === data - if interesting?(data) && data.valid? - @remote_services[data[:service_id]] = data - @new_service_semaphore.broadcast - end - end - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Are we generous in our caching, or have we been - # asked about this type of +publish+ before? - # - def interesting?(publish) - @subscribed_services.each do |subscribed| - return true if publish.matches?(subscribed) - end - return !@thrifty_caching - end - - end - - end - -end Deleted: trunk/archipelago/lib/hashish.rb =================================================================== --- trunk/archipelago/lib/hashish.rb 2006-11-13 23:04:26 UTC (rev 11) +++ trunk/archipelago/lib/hashish.rb 2006-11-13 23:06:06 UTC (rev 12) @@ -1,199 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'archipelago/current' -require 'bdb' - -module Archipelago - - # - # The module containing the persistence default provider. - # - module Hashish - - # - # In essence a Berkeley Database backed Hash. - # - # Will cache all values having been written or read - # in a normal Hash cache for fast access. - # - # Will save the last update timestamp for all keys - # in a separate Hash cache AND a separate Berkeley Database. - # - class BerkeleyHashish - include Archipelago::Current::Synchronized - # - # Initialize an instance with the +name+ and BDB::Env +env+. - # - def initialize(name, env) - super() - @content_db = env.open_db(BDB::HASH, name, "content", BDB::CREATE) - @content = {} - @timestamps_db = env.open_db(BDB::HASH, name, "timestamps", BDB::CREATE | BDB::NOMMAP) - @timestamps = {} - @lock = Archipelago::Current::Lock.new - end - # - # Returns a deep ( Marshal.load(Marshal.dump(o)) ) clone - # of the object represented by +key+. - # - def get_deep_clone(key) - return Marshal.load(@content_db[Marshal.dump(key)]) - end - # - # Simply get the value for the +key+. - # - def [](key) - @lock.synchronize_on(key) do - - value = @content[key] - return value if value - - return get_from_db(key) - - end - end - # - # Insert +value+ under +key+. - # - def []=(key, value) - @lock.synchronize_on(key) do - - @content[key] = value - - write_to_db(key, Marshal.dump(key), Marshal.dump(value)) - - return value - - end - end - # - # Stores whatever is under +key+ if it is not the same as - # whats in the persistent db. - # - def store_if_changed(key) - @lock.synchronize_on(key) do - - serialized_key = Marshal.dump(key) - serialized_value = Marshal.dump(@content[key]) - - write_to_db(key, serialized_key, serialized_value) if @content_db[serialized_key] != serialized_value - - end - end - # - # Returns the last time the value under +key+ was changed. - # - def timestamp(key) - @lock.synchronize_on(key) do - - timestamp = @timestamps[key] - return timestamp if timestamp - - serialized_key = Marshal.dump(key) - serialized_timestamp = @timestamps_db[serialized_key] - return nil unless serialized_timestamp - - timestamp = Marshal.load(serialized_timestamp) - @timestamps[key] = timestamp - return timestamp - - end - end - # - # Delete +key+ and its value and timestamp. - # - def delete(key) - @lock.synchronize_on(key) do - - serialized_key = Marshal.dump(key) - - @content.delete(key) - @content_db[serialized_key] = nil - @timestamps.delete(key) - @timestamps_db[serialized_key] = nil - - end - end - - private - - # - # Write +key+, serialized as +serialized_key+ and - # +serialized_value+ to the db. - # - def write_to_db(key, serialized_key, serialized_value) - now = Time.now - - @content_db[serialized_key] = serialized_value - @timestamps_db[serialized_key] = Marshal.dump(now) - @timestamps[key] = now - end - - # - # Read +key+ from db and if it is found - # put it in the cache Hash. - # - def get_from_db(key) - serialized_key = Marshal.dump(key) - serialized_value = @content_db[serialized_key] - return nil unless serialized_value - - value = Marshal.load(serialized_value) - @content[key] = value - return value - end - end - - # - # A simple persistence provider backed by Berkeley db. - # - class BerkeleyHashishProvider - # - # Initialize an instance with the given - # +env_path+ to its database dir. - # - def initialize(env_path) - env_path.mkpath - @env = BDB::Env.open(env_path, BDB::CREATE | BDB::INIT_MPOOL) - end - # - # Returns a cleverly cached (but slightly inefficient) - # hash-like instance (see Archipelago::Hashish::BerkeleyHashish) - # using +name+. - # - def get_cached_hashish(name) - BerkeleyHashish.new(name, @env) - end - # - # Returns a normal hash-like instance using +name+. - # - def get_hashish(name) - @env.open_db(BDB::HASH, name, nil, BDB::CREATE | BDB::NOMMAP) - end - # - # Removes the persistent files of this instance. - # - def unlink - p = Pathname.new(@env.home) - p.rmtree if p.exist? - end - end - - end - -end Deleted: trunk/archipelago/lib/pirate.rb =================================================================== --- trunk/archipelago/lib/pirate.rb 2006-11-13 23:04:26 UTC (rev 11) +++ trunk/archipelago/lib/pirate.rb 2006-11-13 23:06:06 UTC (rev 12) @@ -1,229 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'archipelago/disco' -require 'archipelago/tranny' -require 'archipelago/treasure' -require 'pp' -require 'drb' -require 'digest/sha1' - -module Archipelago - - # - # The client library that knows about - # all remote databases and handles reads - # and write to them. - # - module Pirate - - # - # Raised when you try to begin a transaction but have no managers - # available. - # - class NoTransactionManagerAvailableException < RuntimeError - def initialize(pirate) - super("#{pirate} can not find any transaction manager for you") - end - end - - # - # Raised when you try to do stuff without any remote database - # available. - # - class NoRemoteDatabaseAvailableException < RuntimeError - def initialize(pirate) - super("#{pirate} can not find any remote database for you") - end - end - - INITIAL_SERVICE_UPDATE_INTERVAL = 1 - MAXIMUM_SERVICE_UPDATE_INTERVAL = 60 - CHEST_DESCRIPTION = { - :class => 'Archipelago::Treasure::Chest' - } - TRANNY_DESCRIPTION = { - :class => 'Archipelago::Tranny::Manager' - } - - # - # The class that actually keeps track of the Archipelago::Treasure:Chests and the - # Archipelago::Treasure:Dubloons in them. - # - class Captain - attr_reader :chests, :treasure_map, :trannies, :transaction - # - # Initialize an instance using an Archipelago::Disco::Jockey with :jockey_options - # if given, that looks for new services :initial_service_update_interval or INITIAL_SERVICE_UPDATE_INTERVAL, - # when it starts and never slower than every :maximum_service_update_interval or MAXIMUM_SERVICE_UPDATE_INTERVAL. - # - # Will look for Archipelago::Treasure::Chests matching :chest_description or CHEST_DESCRIPTION and - # Archipelago::Tranny::Managers matching :tranny_description or TRANNY_DESCRIPTION. - # - def initialize(options = {}) - @treasure_map = Archipelago::Disco::Jockey.new(options[:jockey_options] || {}) - - @chest_description = CHEST_DESCRIPTION.merge(options[:chest_description] || {}) - @tranny_description = TRANNY_DESCRIPTION.merge(options[:tranny_description] || {}) - @jockey_options = options[:jockey_options] || {} - - start_service_updater(options[:initial_service_update_interval] || INITIAL_SERVICE_UPDATE_INTERVAL, - options[:maximum_service_update_interval] || MAXIMUM_SERVICE_UPDATE_INTERVAL) - - @yar_counter = 0 - - @transaction = nil - end - - # - # Get a value from the distributed database network using a +key+, - # optionally within a +transaction+. - # - def [](key, transaction = nil) - responsible_chest(key)[:service][key, transaction || @transaction] - end - - # - # Write a value to the distributed database network, - # optionally inside a transaction. - # - # Usage: - # * p["my key", transaction] = value - # * p["my key"] = value - # - def []=(key, p1, p2 = nil) - if @transaction && p2.nil? - p2 = p1 - p1 = @transaction - end - responsible_chest(key)[:service][key, p1] = p2 - end - - # - # Delete a value from the distributed database network, - # optionally inside a transaction. - # - def delete(key, transaction = nil) - responsible_chest(key)[:service].delete(key, transaction || @transaction) - end - - # - # Return a clone of this instance bound to a newly created transaction. - # - def begin - raise NoTransactionManagerAvailableException.new(self) if @trannies.empty? - - rval = self.clone - rval.instance_eval do - @transaction = @trannies.values.first[:service].begin - end - - return rval - end - - # - # Execute +block+ within a transaction. - # - # Will commit! transaction after the block is finished unless - # the transaction is aborted or commited already. - # - # Will abort! the transaction if any exception is raised. - # - def transaction(&block) #:yields: transaction - raise NoTransactionManagerAvailableException.new(self) if @trannies.empty? - - @transaction = @trannies.values.first[:service].begin - begin - yield(@transaction) - @transaction.commit! if @transaction.state == :active - rescue Exception => e - @transaction.abort! unless @transaction.state == :aborted - puts e - pp e.backtrace - ensure - @transaction = nil - end - end - - # - # Commit the transaction we are a member of and forget about it. - # - def commit! - @transaction.commit! - @transaction = nil - end - - # - # Abort the transaction we are a member of and forget about it. - # - def abort! - @transaction.abort! - @transaction = nil - end - - # - # Yarrr! - # - def yar! - @yar_counter += 1 - 'yar!' - end - - private - - # - # Get the chest responsible for +key+. - # - def responsible_chest(key) - raise NoRemoteDatabaseAvailableException.new(self) if @chests.empty? - - key_id = Digest::SHA1.new(Marshal.dump(key)).to_s - sorted_chest_ids = @chests.keys.sort - sorted_chest_ids.each do |id| - return @chests[id] if id > key_id - end - return @chests[sorted_chest_ids.first] - end - - # - # Start a thread looking up existing chests between every - # +initial+ and +maximum+ seconds. - # - def start_service_updater(initial, maximum) - @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) - @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) - Thread.start do - standoff = initial - loop do - begin - sleep(standoff) - standoff *= 2 - standoff = maximum if standoff > maximum - @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) - @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - end - - end - -end Deleted: trunk/archipelago/lib/tranny.rb =================================================================== --- trunk/archipelago/lib/tranny.rb 2006-11-13 23:04:26 UTC (rev 11) +++ trunk/archipelago/lib/tranny.rb 2006-11-13 23:06:06 UTC (rev 12) @@ -1,651 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'bdb' -require 'pathname' -require 'drb' -require 'archipelago/current' -require 'digest/sha1' -require 'archipelago/disco' -require 'archipelago/hashish' - -module Archipelago - - module Tranny - - # - # Time before any Transaction will be automatically aborted. - # - TRANSACTION_TIMEOUT = 60 * 60 - - # - # If a member tries to join a transaction that it has allready joined - # it will receive this. - # - class JoinCountException < RuntimeError - def initialize(member, transaction) - super("#{member.inspect} has allready joined the transaction #{transaction.inspect}") - end - end - - # - # If a member tries to commit a transaction not in :active state - # or abort a transaction in :commited state, or any other grave - # state offence - # - class IllegalOperationException < RuntimeError - def initialize(operation, transaction) - super("#{operation.inspect} is illegal for #{transaction.inspect}") - end - end - - # - # If a member tries to get a transaction that the manager doesnt know about - # it gets this. - # - class UnknownTransactionException < RuntimeError - def initialize(transaction_id, manager) - super("#{manager.inspect} doesnt know about any transaction with id #{transaction_id.inspect}") - end - end - - # - # If a member tries to report its state and the manager doesnt know about the - # member, it gets this. - # - class UnknownMemberException < RuntimeError - def initialize(member, transaction) - super("#{transaction.inspect} doesnt know about any member like #{member.inspect}") - end - end - - # - # If a member misbehaves (or the Transaction is completely fucked up) this will be raised - # - class IllegalStateException < RuntimeError - def initialize(transaction) - super("#{transaction.inspect} is in a completely fucked up state") - end - end - - # - # The manager itself. - # - # This will be the drb exported object that participants talk to, - # either directly or through a TransactionProxy. - # - # See also the TransactionProxy and Transaction classes. - # - class Manager - - include DRb::DRbUndumped - include Archipelago::Disco::Publishable - - attr_accessor :error_logger - attr_accessor :transaction_timeout - - # - # Will use a BerkeleyHashishProvider using tranny_manager.db in the same dir to get its hashes - # if not :persistence_provider is given. - # - # Will create Transactions timing out after :transaction_timeout seconds or TRANSACTION_TIMEOUT - # if none is given. - # - # Will use Archipelago::Disco::Publishable by calling initialize_publishable with +options+. - # - def initialize(options = {}) - @persistence_provider = options[:persistence_provider] || Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join("tranny_manager.db")) - - initialize_publishable(options) - - @transaction_timeout = options[:transaction_timeout] || TRANSACTION_TIMEOUT - - @metadata = @persistence_provider.get_hashish("metadata") - @db = @persistence_provider.get_cached_hashish("db") - end - - # - # Returns a proxy to a newly created Transaction. - # - # The Transaction will timeout after :timeout seconds - # or @transaction_timeout if none is given. - # - def begin(options = {}) - options[:manager] = self - options[:timeout] ||= @transaction_timeout - return Transaction.new(options).proxy - end - - # - # Used by transactions to notify this manager - # about an error. Delegates the actual logging - # to @error_logger.call(exception). - # - # Set @error_logger or override this method - # if you want to log any errors. - # - def log_error(exception) - self.error_logger.call(exception) if self.error_logger - end - - # - # Used by a +transaction+ to store its state for future reference. - # - def store_transaction!(transaction) - @db[transaction.transaction_id] = transaction - end - - # - # Used by a +transaction+ to remove its state when finished. - # - def remove_transaction!(transaction) - @db.delete(transaction.transaction_id) - end - - # - # Used by a transaction proxy to run +meth+ with +args+ on its +transaction_id+. - # - def call_instance_method(transaction_id, meth, *args) - transaction = @db[transaction_id] - if transaction - return transaction.send(meth, *args) - else - raise UnknownTransactionException.new(transaction_id, self) - end - end - - end - - - - - # - # A proxy to a transaction managed by this manager. - # - # Used because standard DRbObjects only use object_id - # to identify objects, while we want the more unique transaction_id. - # - # Will also remember the state of the transaction when it detects - # methods that will terminate it (:commit | :abort). - # - class TransactionProxy - attr_accessor :transaction_id - def initialize(transaction) - @manager = transaction.manager - @transaction_id = transaction.transaction_id - @state = :unknown - end - # - # Return the cached state of the wrapped Archipelago::Tranny::Transaction - # if it is known, otherwise fetch it. - # - def state - if @state == :unknown - method_missing(:state) - else - @state - end - end - # - # Implemented to ensure that all TransactionProxies with the same - # transaction_id are hashwise equal. - # - def hash - @transaction_id.hash - end - # - # Implemented to ensure that all TransactionProxies with the same - # transaction_id are hashwise equal. - # - def eql?(o) - if TransactionProxy === o - o.transaction_id == @transaction_id - else - false - end - end - # - # Forwards everything to our Transaction and remembers - # returnvalue if necessary. - # - def method_missing(meth, *args) #:nodoc: - rval = @manager.call_instance_method(@transaction_id, meth, *args) - case meth - when :abort! - @state = :aborted - when :commit! - @state = rval - end - return rval - end - end - - - - - # - # A transaction managed by the manager. - # - # A transaction can have the following states: - # - # * :active - When it is first created. - # * This transaction can be commited or aborted. - # - # * :voting - When it has started the two-phase commit and the voting has begun. - # * This transaction can be aborted. - # - # * :commited - After everyone has voted and voted either :unchanged or :commit. - # * This transaction can not be changed. - # - # * :aborted - After someone has called abort! or voted :abort in a vote! - # * This transaction can not be changed. - # - # Anyone that wants to join the transaction must implement the following methods: - # - # * abort!(transaction): Abort the provided transaction. - # * commit!(transaction): Commit the provided transaction. - # * prepare!(transaction): Prepare for commiting the provided transaction. - # - # abort! and commit! should not return any values, but can raise exceptions - # if required. - # - # prepare! must return either :abort, :commit or :unchanged, depending on what - # the member is prepared to do. :unchanged is only when the member has not changed - # state during the transaction, and means that it does not require any further - # notification on the progress of the transaction. - # - # A member that has returned :commit on the prepare! must store the transaction - # proxy in a persistent manner to be able to connect to the manager - # and get a new copy of the transaction in case of communications failure. - # - # A member that reconnects to a crashed transaction manager should not abort! the - # transaction if the transaction is still in :active state, since the transaction will - # have been aborted on commit! anyway (since the manager will not be able to prepare! the - # disconnected member after either the manager or member having crashed) if needed. - # If the disconnect was just a temporary networking problem, the transaction will - # continue as planned. - # - # A member that reconnects to a crashed transaction manager where the transaction - # is in :voting state should just wait around and see if it gets prepare! called. - # In case of temporary network failure the transaction will continue as planned, otherwise - # it will abort! automatically. - # - # A member that reconnects to a disconnected transaction manager where the transaction - # is in :commited state should just commit its state. Then it must notify the transaction - # using report_commited! so that the transaction can disappear gracefully. - # - # A member that reconnects to a disconnected transaction manager that either doesnt know - # of the transaction or returns an :aborted transaction may safely abort the state change - # and forget about the transaction. - # - class Transaction - - include Archipelago::Current::Synchronized - - attr_reader :state, :transaction_id, :proxy, :manager - - # - # Create a transaction managed by the provided +manager+. - # - # Will have :manager as TransactionManager, and will timeout - # after :timeout seconds. - # - def initialize(options) - super() - # - # A hash where members are keys and their state the values. - # - @members = {} - @members.extend(Archipelago::Current::Synchronized) - # - # We are alive! - # - @state = :active - # - # We have a timeout! - # - @timeout = options[:timeout] - # - # We are unique! - # - @transaction_id = "#{options[:manager].service_id}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}" - # - # We have a birth time! - # - @created_at = Time.now - # - # We have a manager! - # - self.manager = options[:manager] - - store_us_for_future_reference! - - start_timeout_thread - end - - # - # Store this manager as ours and create a proxy that knows about it. - # - # Used by Archipelago::Tranny:Manager when restoring crashed - # Archipelago::Tranny:Transactions. - # - def manager=(manager) - # - # We have a manager! - # - @manager = DRbObject.new(manager) - # - # We have a proxy to send forth into the world! - # - @proxy = TransactionProxy.new(self) - nil - end - - # - # Special dump call to NOT dump our manager or proxy, - # since DRbObjects dont take kindly to being restored - # in an environment where they are are known to be invalid. - # - def _dump(l) - return Marshal.dump([ - @members, - @state, - @timeout, - @transaction_id, - @created_at - ]) - end - - def self._load(s) - members, state, timeout, transaction_id, created_at = Marshal.load(s) - rval = self.allocate - rval.instance_variable_set(:@members, members) - rval.instance_variable_set(:@state, state) - rval.instance_variable_set(:@timeout, timeout) - rval.instance_variable_set(:@transaction_id, transaction_id) - rval.instance_variable_set(:@created_at, created_at) - rval - end - - # - # Starts the thread that will abort! us automatically - # after we have lived @timeout - # - def start_timeout_thread - Thread.new do - now = Time.now - die_at = @created_at + @timeout - if die_at > now - sleep(die_at - now) - end - abort! - end - nil - end - - # - # What it sounds like. - # - def store_us_for_future_reference! - @manager.store_transaction!(self) - nil - end - - # - # Remove ourselves, we are redundant - # - def remove_us_we_are_redundant! - @manager.remove_transaction!(self) - nil - end - - # - # Used by members that failed during commit. - # - def report_commited!(member) - raise UnknownMemberException(member, self) unless @members.include?(member) - raise IllegalOperationException(:report_commited!, self) unless self.state == :commited - - @members[member] = :commited - - remove_us_if_all_are_commited! - nil - end - - # - # Abort the transaction, sending all participants the abort! message. - # - def abort! - synchronize do - raise IllegalOperationException.new(:abort!, self) if [:commited, :aborted].include?(self.state) - - # - # Set our state. - # - @state = :aborted - - store_us_for_future_reference! - - # - # Abort all members. - # - threads = [] - @members.clone.each do |member, state| - raise RuntimeException.new("This is not supposed to be possible, but we are in abort! with member " + - "#{member.inspect} in state #{state.inspect}") if state == :commited - if state != :aborted - threads << Thread.new do - begin - member.abort!(self.proxy) - @members.synchronize do - @members[member] = :aborted - end - rescue Exception => e - @manager.log_error(e) - # - # We must not let the other members stop just - # because one member failed. No more Mr Nice Guy! - # - end - end - end - end - - # - # Wait for all members to finish. - # - threads.each do |thread| - thread.join - end - - # - # No use having aborted transactions lying about. - # - # NB: This means that disconnected members that cant - # find their old transactions will have to presume they - # have been aborted. - # - remove_us_we_are_redundant! - end - nil - end - - # - # Commits the transaction, returning the new state (:aborted | :commited) - # - def commit! - synchronize do - raise IllegalOperationException.new(:commit!, self) unless self.state == :active - - # - # Vote for the outcome and act on it - # - case vote! - when :abort - abort! - when :commit - _commit! - when :unchanged - @state = :commited - remove_us_we_are_redundant! - end - - return @state - end - nil - end - - # - # Join a +member+ to this transaction. - # - # Will raise a JoinCountException if the given member has allready - # joined this transaction. - # - def join(member) - @members.synchronize do - raise IllegalOperationException.new(:join, self) unless self.state == :active - raise JoinCountException.new(member, self) if @members.include?(member) - - @members[member] = :active - end - store_us_for_future_reference! - nil - end - - private - - # - # Commit the transaction, sending all participants the commit message. - # - # The transaction is commited by all members having commit! called. - # - def _commit! - # - # Set our state. - # - @state = :commited - - store_us_for_future_reference! - - # - # Commit all members. - # - threads = [] - @members.clone.each do |member, state| - raise RuntimeException.new("This is not supposed to be possible, but we are in _commit with member " + - "#{member.inspect} in state #{state.inspect}") unless [:prepared, :commited].include?(state) - threads << Thread.new do - begin - member.commit!(self.proxy) - @members.synchronize do - @members[member] = :commited - end - rescue Exception => e - @manager.log_error(e) - # - # We must not let the other members stop just - # because one member failed. No more Mr Nice Guy! - # - end - end - end - - # - # Wait for all members to finish. - # - threads.each do |thread| - thread.join - end - - remove_us_if_all_are_commited! - end - - def remove_us_if_all_are_commited! - # - # Check to see if all members have been told about the decision. - # - all_have_commited = true - @members.each do |member, state| - all_have_commited = false unless state == :comitted - end - - # - # If they have, remove ourselves from the manager. - # - remove_us_we_are_redundant! if all_have_commited - end - - # - # Let the members of the transaction vote for its outcome. - # - # Members vote by having prepare! called. - # - # Valid returnvalues for the prepare! call are: - # :abort, if the member wants to abort the transaction - # :commit, if the member wants to commit the transaction - # :unchanged, if the member has not changed state during the transaction - # - def vote! - raise IllegalOperationException.new(:vote!, self) unless self.state == :active - - @state = :voting - - store_us_for_future_reference! - - return_value = :commit - - threads = [] - @members.clone.each do |member, state| - raise RuntimeException.new("This is not supposed to be possible, but we are in vote! with member " + - "#{member.inspect} in state #{state.inspect}") unless state == :active - threads << Thread.new do - this_result = nil - begin - this_result = member.prepare!(self.proxy) - rescue Exception => e - @manager.log_error(e) - this_result = :abort - end - @members.synchronize do - case this_result - when :abort - @members[member] = :aborted - return_value = :abort - when :unchanged - @members.delete(member) - else - @members[member] = :prepared - end - end - end - end - - threads.each do |thread| - thread.join - end - - if @members.empty? - return_value = :unchanged - end - - return_value - end - - end - end - -end Deleted: trunk/archipelago/lib/treasure.rb =================================================================== --- trunk/archipelago/lib/treasure.rb 2006-11-13 23:04:26 UTC (rev 11) +++ trunk/archipelago/lib/treasure.rb 2006-11-13 23:06:06 UTC (rev 12) @@ -1,672 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'drb' -require 'archipelago/disco' -require 'bdb' -require 'pathname' -require 'digest/sha1' -require 'pp' -require 'set' -require 'archipelago/hashish' -require 'archipelago/tranny' - -module Archipelago - - module Treasure - - # - # Minimum time between trying to recover - # crashed transactions. - # - TRANSACTION_RECOVERY_INTERVAL = 30 - - # - # Raised whenever the optimistic locking of the serializable transaction isolation level - # was proved wrong. - # - class RollbackException < RuntimeError - def initialize(chest, transaction) - super("#{chest} has been modified during #{transaction}") - end - end - - # - # Raised whenever a method is called on an object that we dont know about. - # - class UnknownObjectException < RuntimeError - def initialize(chest, key, transaction) - super("#{chest} does not contain #{key} under #{transaction}") - end - end - - # - # Raised if a Dubloon or Chest doesnt know what transaction you are talking about. - # - class UnknownTransactionException < RuntimeError - def initialize(source, transaction) - super("#{source} is not a part of #{transaction}") - end - end - - # - # Raised if anyone tries to commit a non-prepared transaction. - # - class IllegalCommitException < RuntimeError - def initialize(source, transaction) - super("#{transaction} is not prepared in #{source}") - end - end - - # - # Raised if someone tries to join us to a non-active transaction. - # - class IllegalJoinException < RuntimeError - def initialize(transaction) - super("#{transaction} is not active") - end - end - - # - # A proxy to something in the chest. - # - class Dubloon - # - # Remove all methods so that we look like our target. - # - instance_methods.each do |method| - undef_method method unless method =~ /^__/ - end - # - # Initialize us with knowledge of our +chest+, the +key+ to our - # target in the +chest+, the known +public_methods+ of our target - # and any +transaction+ we are associated with. - # - def initialize(key, chest, transaction, chest_id, public_methods) - @key = key - @chest = chest - @transaction = transaction - @chest_id = chest_id - @public_methods = public_methods - end - # - # The public_methods of our target. - # - def public_methods - return @public_methods.clone - end - # - # Return a clone of myself that is joined to - # the +transaction+. - # - def join(transaction) - @chest.join!(transaction) if transaction - return Dubloon.new(@key, @chest, transaction, @chest_id, @public_methods) - end - # - # Raises exception if the given +transaction+ - # is not the same as our own. - # - def assert_transaction(transaction) - raise UnknownTransactionException.new(self, transaction) unless transaction == @transaction - end - # - # The object_id of our chest-held target. - # - def object_id - id = "#{@chest_id}:#{@key}" - id << ":#{@transaction.transaction_id}" if @transaction - return id - end - # - # Does our target respond to +meth+? - # - def respond_to?(meth) - return @public_methods.include?(meth.to_sym) || @public_methods.include?(meth.to_s) - end - # - # Call +meth+ with +args+ and +block+ on our target if it responds to - # it. - # - def method_missing(meth, *args, &block) - if respond_to?(meth) - return @chest.call_instance_method(@key, meth, @transaction, *args, &block) - else - return super(meth, *args) - end - end - - end - - # - # A possibly remote database that only returns proxies to its - # contents, and thus runs all methods on its contents itself. - # - # Has support for optimistically locked distributed serializable transactions. - # - class Chest - - # - # The Chest never leaves its host. - # - include DRb::DRbUndumped - - # - # The Chest can be published. - # - include Archipelago::Disco::Publishable - - # - # Initialize a Chest - # - # Will use a BerkeleyHashishProvider using treasure_chest.db in the same dir to get its hashes - # if not :persistence_provider is given. - # - # Will try to recover crashed transaction every :transaction_recovery_interval seconds - # or TRANSACTION_RECOVERY_INTERVAL if none is given. - # - # Will use Archipelago::Disco::Publishable by calling initialize_publishable with +options+. - # - def initialize(options = {}) - # - # The provider of happy magic persistent hashes of different kinds. - # - @persistence_provider = options[:persistence_provider] || Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join("treasure_chest.db")) - - # - # Use the given options to initialize the publishable - # instance variables. - # - initialize_publishable(options) - - # - # [transaction => [key => instance]] - # To know what stuff is visible to a transaction. - # - @snapshot_by_transaction = {} - @snapshot_by_transaction.extend(Archipelago::Current::Synchronized) - - # - # [transaction => [key => when the key was read/updated/deleted] - # To know if a transaction is ok to prepare and commit. - # - @timestamp_by_key_by_transaction = {} - - # - # [transaction => [key => instance]] - # To know what transactions were prepared but not - # properly finished last run. - # - @crashed = Set.new - - # - # The magical persistent map that defines how we actually - # store our data. - # - @db = @persistence_provider.get_cached_hashish("db") - - initialize_prepared(options[:transaction_recovery_interval] || TRANSACTION_RECOVERY_INTERVAL) - - end - - # - # The transactions active in this Chest. - # - def active_transactions - @snapshot_by_transaction.keys.clone - end - - # - # Return the contents of this chest using a given +key+ and +transaction+. - # - def [](key, transaction = nil) - join!(transaction) - - instance = ensure_instance_with_transaction(key, transaction) - return nil unless instance - - if Dubloon === instance - return instance.join(transaction) - else - return Dubloon.new(key, DRbObject.new(self), transaction, self.service_id, instance.public_methods) - end - end - - # - # Delete the value of +key+ within +transaction+. - # - def delete(key, transaction = nil) - join!(transaction) - - rval = nil - - if transaction - # - # If we have a transaction we must note that it is deleted in a - # separate space for that transaction. - # - snapshot = @snapshot_by_transaction[transaction] - snapshot.synchronize do - - rval = snapshot[key] - - snapshot[key] = :deleted - # - # Make sure we remember when it was last changed according to our main db. - # - timestamps = @timestamp_by_key_by_transaction[transaction] - timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) - - end - else - # - # Otherwise just ask our persistence provider to delete it. - # - @db.delete(key) - end - - rval.freeze - - return rval - end - - # - # Put something into this chest with a given +key+, +value+ and - # +transaction+. - # - def []=(key, p1, p2 = nil) - if p2 - value = p2 - transaction = p1 - else - value = p1 - transaction = nil - end - - return set(key, value, transaction) - end - - # - # Call an instance +method+ on whatever this chest holds at +key+ - # with any +transaction+ and +args+. - # - def call_instance_method(key, method, transaction, *arguments, &block) - if transaction - return call_with_transaction(key, method, transaction, *arguments, &block) - else - return call_without_transaction(key, method, *arguments, &block) - end - end - - # - # Abort +transaction+ in this Chest. - # - def abort!(transaction) - assert_transaction(transaction) - - snapshot = @snapshot_by_transaction[transaction] - # - # Make sure nobody can modify this transaction while we - # are aborting it. - # - snapshot.synchronize do - serialized_transaction = Marshal.dump(transaction) - - # - # If this transaction was successfully prepared - # - if @snapshot_by_transaction_db.include?(serialized_transaction) - # - # Unlock the keys that are part of it. - # - snapshot.each do |key, value| - @db.unlock_on(key) - end - # - # And remove it from persistent storage. - # - @snapshot_by_transaction_db[serialized_transaction] = nil - # - # And remove its timestamps from persistent storage. - # - @timestamp_by_key_by_transaction_db[serialized_transaction] = nil - end - # - # Finally delete it from the snapshots. - # - @snapshot_by_transaction.delete(transaction) - # - # And from the timestamps. - # - @timestamp_by_key_by_transaction.delete(transaction) - end - end - - # - # Prepares +transaction+ in this Chest. - # - # NB: This will cause any update of the data within - # this transaction to block until it is either aborted - # or commited! - # - def prepare!(transaction) - assert_transaction(transaction) - - # - # If we dont know about this transaction then it can't very well - # affect us. - # - return :unchanged unless @snapshot_by_transaction.include?(transaction) - - snapshot = @snapshot_by_transaction[transaction] - # - # Make sure nobody can modify this transaction while we are - # preparing it. - # - snapshot.synchronize do - - # - # Remember what locks we acquire so that we can - # unlock them in case of failure. - # - locks = [] - timestamp_by_key = @timestamp_by_key_by_transaction[transaction] - # - # Acquire a lock on each key in the transaction - # - snapshot.each do |key, value| - if @db.timestamp(key) == timestamp_by_key[key] - @db.lock_on(key) - locks << key - else - locks.each do |key| - @db.unlock_on(key) - end - return :abort - end - end - serialized_transaction = Marshal.dump(transaction) - - # - # Dump its state to persistent storage. - # - @snapshot_by_transaction_db[serialized_transaction] = Marshal.dump(snapshot) - # - # And dump its timestamps to persistent storage - # - @timestamp_by_key_by_transaction_db[serialized_transaction] = Marshal.dump(timestamp_by_key) - return :prepared - end - end - - # - # Commits +transaction+ in this Chest. - # - # NB: Transaction must be prepared before commit is called. - # - def commit!(transaction) - assert_transaction(transaction) - raise IllegalCommitException.new(self, transaction) unless @snapshot_by_transaction_db.include?(Marshal.dump(transaction)) - - snapshot = @snapshot_by_transaction[transaction] - # - # Make sure nobody can modify this transaction while we are - # commiting it. - # - snapshot.synchronize do - - # - # Copy each key and value from our private space to the real space - # - snapshot.each do |key, value| - if value == :deleted - @db.delete(key) - else - @db[key] = value - end - end - - # - # Call abort! to clean up after the transaction. - # - abort!(transaction) - - end - end - - private - - # - # Allocates space for this +transaction+. - # - # Will also call +transaction+.join to make sure - # it is aware of us. - # - def join!(transaction) - if transaction - if transaction.state == :active - @snapshot_by_transaction.synchronize do - unless @snapshot_by_transaction.include?(transaction) - @snapshot_by_transaction[transaction] = {} - @snapshot_by_transaction[transaction].extend(Archipelago::Current::Synchronized) - @timestamp_by_key_by_transaction[transaction] = {} - transaction.join(DRbObject.new(self)) - end - end - else - raise IllegalJoinException.new(transaction) - end - end - end - - # - # Raises if we are not in this transaction. - # - def assert_transaction(transaction) - raise UnknownTransactionException.new(self, transaction) unless @snapshot_by_transaction.include?(transaction) - end - - # - # Call a method within a transaction. - # - def call_with_transaction(key, method, transaction, *arguments, &block) - assert_transaction(transaction) - - # - # Fetch our instance from the snapshot. - # - snapshot = @snapshot_by_transaction[transaction] - instance = snapshot[key] - instance = nil if instance == :deleted - - raise UnknownObjectException.new(self, key, transaction) unless instance - - begin - return execute(instance, method, *arguments, &block) - ensure - # - # Make sure we remember when this object was last changed according - # to the main db. - # - snapshot.synchronize do - timestamps = @timestamp_by_key_by_transaction[transaction] - timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) - end - end - end - - # - # Execute +m+ with arguments +a+ and block +b+ on +o+. - # - def execute(o, m, *a, &b) - if b - return o.send(m, *a, &b) - else - return o.send(m, *a) - end - end - - # - # Call a method outside any transaction (ie inside a transaction of its own). - # - def call_without_transaction(key, method, *arguments, &block) - instance = @db[key] - - raise UnknownObjectException(self, key, transaction) unless instance - - begin - return execute(instance, method, *arguments, &block) - ensure - @db.store_if_changed(key) - end - end - - # - # Initializes our storage of prepared transactions. - # - def initialize_prepared(transaction_recovery_interval) - # - # Load stored timestamps for our transaction from db. - # - @timestamp_by_key_by_transaction_db = @persistence_provider.get_hashish("prepared_timestamps") - @timestamp_by_key_by_transaction_db.each do |serialized_transaction, serialized_timestamps| - @timestamp_by_key_by_transaction[Marshal.load(serialized_transaction)] = Marshal.load(serialized_timestamps) - end - - # - # Load stored snapshots for our transaction from db. - # - @snapshot_by_transaction_db = @persistence_provider.get_hashish("prepared") - @snapshot_by_transaction_db.each do |serialized_transaction, serialized_snapshot| - transaction = Marshal.load(transaction) - - @crashed << transaction - @snapshot_by_transaction[transaction] = Marshal.load(serialized_snapshot) - end - start_recovery_thread(transaction_recovery_interval) - end - - # - # Starts the thread that will keep trying to recover - # our crashed transactions. - # - def start_recovery_thread(transaction_recovery_interval) - Thread.new do - loop do - begin - @crashed.clone.each do |transaction| - begin - case transaction.state - when :commited - commit!(transaction) - @crashed.delete(transaction) - when :aborted - abort!(transaction) - @crashed.delete(transaction) - end - rescue Archipelago::Tranny::UnknownTransactionException => e - abort!(transaction) - @crashed.delete(transaction) - end - end - sleep(transaction_recovery_interval) - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Insert +value+ under +key+ and +transaction+ - # in this chest. - # - def set(key, value, transaction) - join!(transaction) - value.assert_transaction(transaction) if Dubloon === value - - if transaction - snapshot = @snapshot_by_transaction[transaction] - - # - # If we have a transaction we must put it in a - # separate space for that transaction. - # - snapshot.synchronize do - - snapshot[key] = value - # - # Make sure we remember the last time this was changed according to - # our main db. - # - timestamps = @timestamp_by_key_by_transaction[transaction] - timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) - - end - else - @db[key] = value - end - - return value if Dubloon === value - - return Dubloon.new(key, DRbObject.new(self), transaction, service_id, value.public_methods) - end - - # - # Try to fetch the data of +key+ from the private space - # of +transaction+ and put it there if it was not there - # already. - # - def ensure_instance_with_transaction(key, transaction) - if transaction - snapshot = @snapshot_by_transaction[transaction] - snapshot.synchronize do - - # - # If we dont have this key in the snapshot. - # - unless snapshot.include?(key) - # - # Fetch the new value for the snapshot - # - new_value = @db.get_deep_clone(key) - # - # If it exists then copy it to the snapshot - # otherwise remove the transaction hash if it is empty. - # - if new_value - snapshot[key] = new_value - else - @snapshot_by_transaction.delete(transaction) if snapshot.empty? - end - end - - rval = snapshot[key] - return rval == :deleted ? nil : rval - - end - else - return @db[key] - end - end - - end - - end - -end From nobody at rubyforge.org Mon Nov 13 19:42:25 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 19:42:25 -0500 (EST) Subject: [Archipelago-submits] [15] trunk/archipelago/lib/archipelago/treasure.rb: made evaluated code in Chest persistent Message-ID: <20061114004225.C68BBA970014@rubyforge.org> Revision: 15 Author: zond Date: 2006-11-13 19:42:25 -0500 (Mon, 13 Nov 2006) Log Message: ----------- made evaluated code in Chest persistent Modified Paths: -------------- trunk/archipelago/lib/archipelago/treasure.rb Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-14 00:34:37 UTC (rev 14) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-14 00:42:25 UTC (rev 15) @@ -213,12 +213,7 @@ # @crashed = Set.new - # - # [label,label...] - # To know what data we have already evaluated - # so that we dont overdo it. - # - @seen_data = Set.new + initialize_seen_data # # The magical persistent map that defines how we actually @@ -248,7 +243,7 @@ puts e pp e.backtrace ensure - @seen_data << label + @seen_data[label] = data end end end @@ -474,6 +469,25 @@ private # + # Evaluates all data we have been told to evaluate in earlier runs. + # + def initialize_seen_data + # + # [label => data] + # To remember what we have evaluated and be able to evaluate it again. + # + @seen_data = @persistence_provider.get_hashish("seen_data") + @seen_data.each do |label, data| + begin + Object.class_eval(data) + rescue Exception => e + puts e + pp e.backtrace + end + end + end + + # # Allocates space for this +transaction+. # # Will also call +transaction+.join to make sure From nobody at rubyforge.org Mon Nov 13 20:25:01 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 20:25:01 -0500 (EST) Subject: [Archipelago-submits] [16] trunk/archipelago: made pirate able to evaluate in chests on demand. Message-ID: <20061114012501.356B05241961@rubyforge.org> Revision: 16 Author: zond Date: 2006-11-13 20:24:54 -0500 (Mon, 13 Nov 2006) Log Message: ----------- made pirate able to evaluate in chests on demand. added more tests. fixed bugs. Modified Paths: -------------- trunk/archipelago/lib/archipelago/pirate.rb trunk/archipelago/lib/archipelago/tranny.rb trunk/archipelago/lib/archipelago/treasure.rb trunk/archipelago/tests/pirate_test.rb trunk/archipelago/tests/treasure_test.rb Added Paths: ----------- trunk/archipelago/tests/evaltestmore Modified: trunk/archipelago/lib/archipelago/pirate.rb =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb 2006-11-14 00:42:25 UTC (rev 15) +++ trunk/archipelago/lib/archipelago/pirate.rb 2006-11-14 01:24:54 UTC (rev 16) @@ -86,7 +86,7 @@ @chest_eval_files = options[:chest_eval_files] || [] - @chests_having_evaluated = Set.new + @chests_having_evaluated = {} start_service_updater(options[:initial_service_update_interval] || INITIAL_SERVICE_UPDATE_INTERVAL, options[:maximum_service_update_interval] || MAXIMUM_SERVICE_UPDATE_INTERVAL) @@ -167,6 +167,14 @@ end # + # Evaluate this file in all known chests. + # + def evaluate!(filename) + @chest_eval_files << filename + evaluate_in_chests + end + + # # Commit the transaction we are a member of and forget about it. # def commit! @@ -214,22 +222,46 @@ end # - # Get all chests from the Archipelago::Disco::Jockey and - # send our @chest_eval_files to it if we have not already. + # Make sure all our known chests have evaluated + # all files we need them to. # - def get_chests - @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) + def evaluate_in_chests + # + # For all chests + # @chests.values.each do |chest| - unless @chests_having_evaluated.include?([chest[:service_id], chest[:published_at]]) - @chest_eval_files.each do |filename| + # + # Ensure that this chest has a Set of evaluated files + # + @chests_having_evaluated[ + [ + chest[:service_id], + chest[:published_at] + ] + ] ||= Set.new + @chest_eval_files.each do |filename| + unless @chests_having_evaluated[ + [ + chest[:service_id], + chest[:published_at] + ] + ].include?(filename) begin - chest[:service].evaluate!(filename, open(filename).read) + chest[:service].evaluate!(filename, + File.ctime(filename), + open(filename).read) rescue Exception => e puts e pp e.backtrace + ensure + @chests_having_evaluated[ + [ + chest[:service_id], + chest[:published_at] + ] + ] << filename end end - @chests_having_evaluated << [chest[:service_id], chest[:published_at]] end end end @@ -239,7 +271,8 @@ # +initial+ and +maximum+ seconds. # def start_service_updater(initial, maximum) - get_chests + @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) + evaluate_in_chests @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) @service_update_thread = Thread.start do standoff = initial @@ -248,7 +281,8 @@ sleep(standoff) standoff *= 2 standoff = maximum if standoff > maximum - get_chests + @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) + evaluate_in_chests @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) rescue Exception => e puts e Modified: trunk/archipelago/lib/archipelago/tranny.rb =================================================================== --- trunk/archipelago/lib/archipelago/tranny.rb 2006-11-14 00:42:25 UTC (rev 15) +++ trunk/archipelago/lib/archipelago/tranny.rb 2006-11-14 01:24:54 UTC (rev 16) @@ -114,7 +114,6 @@ @transaction_timeout = options[:transaction_timeout] || TRANSACTION_TIMEOUT - @metadata = @persistence_provider.get_hashish("metadata") @db = @persistence_provider.get_cached_hashish("db") end Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-14 00:42:25 UTC (rev 15) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-14 01:24:54 UTC (rev 16) @@ -233,17 +233,27 @@ end # - # Evaluate +data+ if we have not already seen +label+. + # Evaluate +data+ if we have not already seen +label+ or if we have an earlier +timestamp+ than the one given. # - def evaluate!(label, data) - unless @seen_data.include?(label) + def evaluate!(label, timestamp, data) + serialized_label = Marshal.dump(label) + go = false + + if @seen_data.include?(serialized_label) + last_timestamp, last_data = Marshal.load(@seen_data[serialized_label]) + go = true if timestamp > last_timestamp + else + go = true + end + + if go begin Object.class_eval(data) + @seen_data[serialized_label] = Marshal.dump([timestamp, data]) rescue Exception => e + @seen_data[serialized_label] = Marshal.load([timestamp, nil]) puts e pp e.backtrace - ensure - @seen_data[label] = data end end end @@ -473,16 +483,20 @@ # def initialize_seen_data # - # [label => data] - # To remember what we have evaluated and be able to evaluate it again. + # [label => [timestamp, data]] + # To remember what and when we have evaluated and be able to evaluate it again. # @seen_data = @persistence_provider.get_hashish("seen_data") - @seen_data.each do |label, data| - begin - Object.class_eval(data) - rescue Exception => e - puts e - pp e.backtrace + @seen_data.each do |serialized_label, serialized_pair| + timestamp, data = Marshal.load(serialized_pair) + + if data + begin + Object.class_eval(data) + rescue Exception => e + puts e + pp e.backtrace + end end end end Added: trunk/archipelago/tests/evaltestmore =================================================================== --- trunk/archipelago/tests/evaltestmore (rev 0) +++ trunk/archipelago/tests/evaltestmore 2006-11-14 01:24:54 UTC (rev 16) @@ -0,0 +1,3 @@ +class Evaltestmore + +end Modified: trunk/archipelago/tests/pirate_test.rb =================================================================== --- trunk/archipelago/tests/pirate_test.rb 2006-11-14 00:42:25 UTC (rev 15) +++ trunk/archipelago/tests/pirate_test.rb 2006-11-14 01:24:54 UTC (rev 16) @@ -36,13 +36,22 @@ p2 = Archipelago::Pirate::Captain.new(:chest_description => {:class => "TestChest"}, :tranny_description => {:class => "TestManager"}, - :chest_eval_files => File.join(File.dirname(__FILE__), 'evaltest')) + :chest_eval_files => [File.join(File.dirname(__FILE__), 'evaltest')]) + assert_within(10) do !p2.chests.empty? end e = Evaltest.new + assert_raise(NameError) do + e = Evaltestmore.new + end + + p2.evaluate!(File.join(File.dirname(__FILE__), "evaltestmore")) + + e = Evaltestmore.new + p2.stop! end Modified: trunk/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-14 00:42:25 UTC (rev 15) +++ trunk/archipelago/tests/treasure_test.rb 2006-11-14 01:24:54 UTC (rev 16) @@ -18,12 +18,15 @@ end def test_eval - @c.evaluate!("burk", "class Burk; end") + t = Time.now + @c.evaluate!("burk", t, "class Burk; end") b = Burk.new - @c.evaluate!("burk", "class Bong; end") + @c.evaluate!("burk", t, "class Bong; end") assert_raise(NameError) do b = Bong.new end + @c.evaluate!("burk", Time.now + 10, "class Bong; end") + b = Bong.new end def test_store_load_update From nobody at rubyforge.org Mon Nov 13 21:08:22 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 21:08:22 -0500 (EST) Subject: [Archipelago-submits] [17] trunk/archipelago: repackaged the gem according to the new file structure. Message-ID: <20061114020822.46E56A970029@rubyforge.org> Revision: 17 Author: zond Date: 2006-11-13 21:08:20 -0500 (Mon, 13 Nov 2006) Log Message: ----------- repackaged the gem according to the new file structure. added to TODO. made DRb::DRbUndumped give better error messages Modified Paths: -------------- trunk/archipelago/Rakefile trunk/archipelago/TODO trunk/archipelago/lib/archipelago/treasure.rb Modified: trunk/archipelago/Rakefile =================================================================== --- trunk/archipelago/Rakefile 2006-11-14 01:24:54 UTC (rev 16) +++ trunk/archipelago/Rakefile 2006-11-14 02:08:20 UTC (rev 17) @@ -13,7 +13,7 @@ s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A set of tools for distributed computing in ruby." - s.files = FileList['lib/*.rb', 'test/*', 'scripts/*', 'GPL-2', 'TODO', 'profiles/*'].to_a + s.files = FileList['lib/archipelago.rb', 'lib/archipelago/*.rb', 'test/*', 'scripts/*', 'GPL-2', 'TODO', 'profiles/*'].to_a s.require_path = "lib" s.autorequire = "archipelago" s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') Modified: trunk/archipelago/TODO =================================================================== --- trunk/archipelago/TODO 2006-11-14 01:24:54 UTC (rev 16) +++ trunk/archipelago/TODO 2006-11-14 02:08:20 UTC (rev 17) @@ -7,3 +7,7 @@ the call. Preferably without incurring performance lossage. * Test the transaction recovery mechanism of Chest. + + * Make Archipelago::Treasure::Dubloons work after an Archipelago::Treasure::Chest + has rebooted. For example: demand that the chest always run the same host+port + or make Dubloons able to lookup their home Chest by service_id. Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-14 01:24:54 UTC (rev 16) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-14 02:08:20 UTC (rev 17) @@ -25,6 +25,17 @@ require 'archipelago/tranny' require 'archipelago/disco' +# +# A bit more descriptive error messages. +# +module DRb + module DRbUndumped + def _dump(dummy) + raise TypeError, "can't dump #{self.class}" + end + end +end + module Archipelago module Treasure From nobody at rubyforge.org Mon Nov 13 21:09:04 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 13 Nov 2006 21:09:04 -0500 (EST) Subject: [Archipelago-submits] [18] trunk: added a non-working ActiveRecord::Base replacement Message-ID: <20061114020904.E1056A970029@rubyforge.org> Revision: 18 Author: zond Date: 2006-11-13 21:09:04 -0500 (Mon, 13 Nov 2006) Log Message: ----------- added a non-working ActiveRecord::Base replacement Added Paths: ----------- trunk/hyperactive/ trunk/hyperactive/GPL-2 trunk/hyperactive/README trunk/hyperactive/Rakefile trunk/hyperactive/lib/ trunk/hyperactive/lib/hyperactive/ trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive.rb trunk/hyperactive/test/ trunk/hyperactive/test/test_helper.rb Added: trunk/hyperactive/GPL-2 =================================================================== --- trunk/hyperactive/GPL-2 (rev 0) +++ trunk/hyperactive/GPL-2 2006-11-14 02:09:04 UTC (rev 18) @@ -0,0 +1,339 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + , 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. Added: trunk/hyperactive/README =================================================================== --- trunk/hyperactive/README (rev 0) +++ trunk/hyperactive/README 2006-11-14 02:09:04 UTC (rev 18) @@ -0,0 +1,4 @@ += This is HyperactiveRecord, a not-at-all drop-in replacement for ActiveRecord::Base + +It uses archipelago for persistence, and is meaningful only in an environment where the server process doesnt restart at each request. This means that cgi environment is not really an option. + Added: trunk/hyperactive/Rakefile =================================================================== --- trunk/hyperactive/Rakefile (rev 0) +++ trunk/hyperactive/Rakefile 2006-11-14 02:09:04 UTC (rev 18) @@ -0,0 +1,59 @@ + +require 'rake' +require 'rake/testtask' +require 'rubygems' +Gem::manage_gems +require 'rake/gempackagetask' + + +spec = Gem::Specification.new do |s| + s.platform = Gem::Platform::RUBY + s.name = "hyperactive_record" + s.version = "0.1.0" + s.author = "Martin Kihlgren" + s.email = "zond at troja dot ath dot cx" + s.summary = "A base class for Ruby on Rails models that uses archipelago for persistence." + s.files = FileList['lib/*.rb', 'test/*', 'GPL-2', 'TODO'].to_a + s.require_path = "lib" + s.autorequire = "hyperactive_record" + s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') + s.has_rdoc = true + s.rdoc_options << '--line-numbers' + s.rdoc_options << '--inline-source' + s.extra_rdoc_files = ["README"] +end + + +SOURCE_FILES = FileList.new do |fl| + [ "lib", "tests" ].each do |dir| + fl.include "#{dir}/**/*" + end + fl.include "Rakefile" +end + +Rake::GemPackageTask.new(spec) do |pkg| + pkg.need_tar = true +end + +task :default => [:units] do +end + +desc "Run all tests" +Rake::TestTask.new(:units) do |t| + t.pattern = 'tests/*_test.rb' + t.verbose = true + t.warning = true +end + +desc "Run all benchmarks" +Rake::TestTask.new(:bench) do |t| + t.pattern = 'tests/*_benchmark.rb' + t.verbose = true + t.warning = true +end + +desc "Package a gem from the source" +task :gem => "pkg/#{spec.name}-#{spec.version}.gem" do + puts "generated latest version" +end + Added: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb (rev 0) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-14 02:09:04 UTC (rev 18) @@ -0,0 +1,61 @@ + +require 'rubygems' +require 'archipelago' + +module Hyperactive + + class Record + + @@pirate = Archipelago::Pirate::Captain.new + + def self.setup(options = {}) + @@options = options + @@pirate = Archipelago::Pirate::Captain.new(options[:pirate_options]) + end + + def self.select(&block) + if block + self.select.select do |record| + yield(record) + end + else + @@pirate[self.all_key] + end + end + + def self.reject(&block) + self.select.reject do |record| + yield(record) + end + end + + def self.delete(instance) + @@pirate.delete(instance.record_id) + end + + def self.transaction(&block) + @@pirate.transaction(&block) + end + + attr_reader :record_id + + def initialize + host = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" + @record_id = Digest::SHA1.hexdigest("#{host}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") + @@pirate.evaluate!(File.expand_path(__FILE__)) + @@pirate[@record_id] = self + @@pirate[self.class.all_key] ||= Set.new + d = @@pirate[@record_id] + puts Marshal.dump(d) + @@pirate[self.class.all_key] << @@pirate[@record_id] + end + + private + + def self.all_key + "HyperactiveRecord:#{self.name}:all" + end + + end + +end Added: trunk/hyperactive/lib/hyperactive.rb =================================================================== --- trunk/hyperactive/lib/hyperactive.rb (rev 0) +++ trunk/hyperactive/lib/hyperactive.rb 2006-11-14 02:09:04 UTC (rev 18) @@ -0,0 +1,4 @@ + +$: << File.dirname(__FILE__) + +require 'hyperactive/record' Added: trunk/hyperactive/test/test_helper.rb =================================================================== --- trunk/hyperactive/test/test_helper.rb (rev 0) +++ trunk/hyperactive/test/test_helper.rb 2006-11-14 02:09:04 UTC (rev 18) @@ -0,0 +1,34 @@ + +home = File.expand_path(File.dirname(__FILE__)) +$: << File.join(home, "..", "lib") + +require 'pp' +require 'test/unit' +require 'benchmark' + +class Test::Unit::TestCase + + def bm(label, options = {}) + n = options[:n] || 1000 + width = options[:width] || 50 + Benchmark.benchmark(" " * width + Benchmark::Tms::CAPTION, width, Benchmark::Tms::FMTSTR, "ms/call") do |b| + times = b.report("#{n}x#{label}") do + n.times do + yield + end + end + [times * 1000 / n.to_f] + end + end + + def assert_within(timeout, &block) + t = Time.new + rval = yield + while !rval && t > Time.new - timeout + rval = yield + sleep(0.05) + end + assert(rval) + end + +end From nobody at rubyforge.org Tue Nov 14 10:45:49 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 14 Nov 2006 10:45:49 -0500 (EST) Subject: [Archipelago-submits] [19] trunk: made Publishable items 'undumpable', or rather always dumpable but producing a dumped DRbObject instead of themselves dumped. Message-ID: <20061114154549.23D0DA970012@rubyforge.org> Revision: 19 Author: zond Date: 2006-11-14 10:45:48 -0500 (Tue, 14 Nov 2006) Log Message: ----------- made Publishable items 'undumpable', or rather always dumpable but producing a dumped DRbObject instead of themselves dumped. Modified Paths: -------------- trunk/archipelago/lib/archipelago/disco.rb trunk/archipelago/lib/archipelago/tranny.rb trunk/archipelago/lib/archipelago/treasure.rb trunk/hyperactive/lib/hyperactive/record.rb Modified: trunk/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-14 02:09:04 UTC (rev 18) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-14 15:45:48 UTC (rev 19) @@ -84,6 +84,30 @@ module Publishable # + # Also add the ClassMethods to +base+. + # + def self.append_features(base) + super + base.extend(ClassMethods) + end + + module ClassMethods + # + # Just load whatever we have in +s+. + # + def _load(s) + DRbObject._load(s) + end + end + + # + # Dump a DRbObject refering to us. + # + def _dump(dummy_param) + DRbObject.new(self)._dump(dummy_param) + end + + # # Will initialize this instance with @service_description and @jockey_options # and merge these with the optionally given :service_description and # :jockey_options. Modified: trunk/archipelago/lib/archipelago/tranny.rb =================================================================== --- trunk/archipelago/lib/archipelago/tranny.rb 2006-11-14 02:09:04 UTC (rev 18) +++ trunk/archipelago/lib/archipelago/tranny.rb 2006-11-14 15:45:48 UTC (rev 19) @@ -92,7 +92,6 @@ # class Manager - include DRb::DRbUndumped include Archipelago::Disco::Publishable attr_accessor :error_logger Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-14 02:09:04 UTC (rev 18) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-14 15:45:48 UTC (rev 19) @@ -25,17 +25,6 @@ require 'archipelago/tranny' require 'archipelago/disco' -# -# A bit more descriptive error messages. -# -module DRb - module DRbUndumped - def _dump(dummy) - raise TypeError, "can't dump #{self.class}" - end - end -end - module Archipelago module Treasure @@ -172,11 +161,6 @@ class Chest # - # The Chest never leaves its host. - # - include DRb::DRbUndumped - - # # The Chest can be published. # include Archipelago::Disco::Publishable Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-14 02:09:04 UTC (rev 18) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-14 15:45:48 UTC (rev 19) @@ -45,8 +45,6 @@ @@pirate.evaluate!(File.expand_path(__FILE__)) @@pirate[@record_id] = self @@pirate[self.class.all_key] ||= Set.new - d = @@pirate[@record_id] - puts Marshal.dump(d) @@pirate[self.class.all_key] << @@pirate[@record_id] end From nobody at rubyforge.org Tue Nov 14 13:11:34 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 14 Nov 2006 13:11:34 -0500 (EST) Subject: [Archipelago-submits] [20] trunk: made Record.reject more failsafe. Message-ID: <20061114181134.D9261A970011@rubyforge.org> Revision: 20 Author: zond Date: 2006-11-14 13:11:33 -0500 (Tue, 14 Nov 2006) Log Message: ----------- made Record.reject more failsafe. added Archipelago::Disco::MC as a default Jockey. made :published_at automatic instead of required. added a Disco benchmark. made everyone except tests use MC instead of their own paultry Jockey. made Dubloons and TransactionProxies find missing services when needed. made everyone including Publish actually use the _dump and _load methods in Publish. added a nice debug method to scripts/pirate.rb Modified Paths: -------------- trunk/archipelago/lib/archipelago/disco.rb trunk/archipelago/lib/archipelago/pirate.rb trunk/archipelago/lib/archipelago/tranny.rb trunk/archipelago/lib/archipelago/treasure.rb trunk/archipelago/scripts/pirate.rb trunk/archipelago/tests/disco_test.rb trunk/archipelago/tests/test_helper.rb trunk/hyperactive/lib/hyperactive/record.rb Added Paths: ----------- trunk/archipelago/tests/disco_benchmark.rb Modified: trunk/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-14 15:45:48 UTC (rev 19) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-14 18:11:33 UTC (rev 20) @@ -115,8 +115,8 @@ def initialize_publishable(options = {}) @service_description = { :service_id => service_id, - :validator => DRbObject.new(self), - :service => DRbObject.new(self), + :validator => self, + :service => self, :class => self.class.name }.merge(options[:service_description] || {}) @jockey_options = options[:jockey_options] || {} @@ -130,8 +130,7 @@ # :service_description. # def publish!(options = {}) - @jockey ||= Archipelago::Disco::Jockey.new(@jockey_options.merge(options[:jockey_options] || {})) - @service_description.merge!({:published_at => Time.now}) + @jockey ||= defined?(Archipelago::Disco::MC) ? Archipelago::Disco::MC : Archipelago::Disco::Jockey.new(@jockey_options.merge(options[:jockey_options] || {})) @jockey.publish(Archipelago::Disco::Record.new(@service_description.merge(options[:service_description] || {}))) end @@ -230,7 +229,6 @@ def initialize(hash) raise "Record must have a :service_id" unless hash.include?(:service_id) raise "Record must have a :validator" unless hash.include?(:validator) - raise "Record must have a :published_at" unless hash.include?(:published_at) super(hash) end # @@ -304,8 +302,6 @@ # class Jockey - attr_reader :new_service_semaphore - # # Will create a Jockey service running on :address and :port or # ADDRESS and PORT if none are given. @@ -438,6 +434,7 @@ # def publish(service) if service.valid? + service[:published_at] = Time.now @local_services[service[:service_id]] = service @new_service_semaphore.broadcast unless @thrifty_publishing @@ -569,6 +566,8 @@ end + MC = Jockey.new(defined?(MC_OPTIONS) ? MC_OPTIONS : {}) unless defined?(MC_DISABLED) && MC_DISABLED + end end Modified: trunk/archipelago/lib/archipelago/pirate.rb =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb 2006-11-14 15:45:48 UTC (rev 19) +++ trunk/archipelago/lib/archipelago/pirate.rb 2006-11-14 18:11:33 UTC (rev 20) @@ -78,7 +78,7 @@ # of required classes and modules at the chest. # def initialize(options = {}) - @treasure_map = Archipelago::Disco::Jockey.new(options[:jockey_options] || {}) + @treasure_map = defined?(Archipelago::Disco::MC) ? Archipelago::Disco::MC : Archipelago::Disco::Jockey.new(options[:jockey_options] || {}) @chest_description = CHEST_DESCRIPTION.merge(options[:chest_description] || {}) @tranny_description = TRANNY_DESCRIPTION.merge(options[:tranny_description] || {}) Modified: trunk/archipelago/lib/archipelago/tranny.rb =================================================================== --- trunk/archipelago/lib/archipelago/tranny.rb 2006-11-14 15:45:48 UTC (rev 19) +++ trunk/archipelago/lib/archipelago/tranny.rb 2006-11-14 18:11:33 UTC (rev 20) @@ -184,6 +184,7 @@ attr_accessor :transaction_id def initialize(transaction) @manager = transaction.manager + @manager_id = transaction.manager.service_id @transaction_id = transaction.transaction_id @state = :unknown end @@ -221,7 +222,21 @@ # returnvalue if necessary. # def method_missing(meth, *args) #:nodoc: - rval = @manager.call_instance_method(@transaction_id, meth, *args) + rval = nil + begin + rval = @manager.call_instance_method(@transaction_id, meth, *args) + rescue DRb::DRbConnError => e + if defined?(Archipelago::Disco::MC) + possible_replacements = Archipelago::Disco::MC.lookup(Archipelago::Disco::Query.new({:service_id => @manager_id})) + raise e if possible_replacements.empty? + + @manager = possible_replacements[@manager_id][:service] + + retry + else + raise e + end + end case meth when :abort! @state = :aborted @@ -345,7 +360,7 @@ # # We have a manager! # - @manager = DRbObject.new(manager) + @manager = manager # # We have a proxy to send forth into the world! # Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-14 15:45:48 UTC (rev 19) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-14 18:11:33 UTC (rev 20) @@ -36,6 +36,12 @@ TRANSACTION_RECOVERY_INTERVAL = 30 # + # The known chests by service id to speed up + # recovery by Dubloons having lost their Chest. + # + CHESTS_BY_SERVICE_ID = {} + + # # Raised whenever the optimistic locking of the serializable transaction isolation level # was proved wrong. # @@ -104,6 +110,36 @@ @public_methods = public_methods end # + # A more or less normal dump of all our instance variables. + # + def _dump(dummy_levels) + Marshal.dump([ + @key, + @chest, + @transaction, + @chest_id, + @public_methods + ]) + end + # + # Load some instance variables and replace @chest if we know that + # it actually is not correct. + # + def self._load(s) + key, chest, transaction, chest_id, public_methods = Marshal.load(s) + instance = self.allocate + instance.instance_variable_set(:@key, key) + instance.instance_variable_set(:@transaction, transaction) + instance.instance_variable_set(:@chest_id, chest_id) + instance.instance_variable_set(:@public_methods, public_methods) + if CHEST_BY_SERVICE_ID.include?(chest_id) + instance.instance_variable_set(:@chest, CHEST_BY_SERVICE_ID[chest_id]) + else + instance.instance_variable_set(:@chest, chest) + end + return instance + end + # # The public_methods of our target. # def public_methods @@ -144,7 +180,21 @@ # def method_missing(meth, *args, &block) if respond_to?(meth) - return @chest.call_instance_method(@key, meth, @transaction, *args, &block) + begin + return @chest.call_instance_method(@key, meth, @transaction, *args, &block) + rescue DRb::DRbConnError => e + if defined?(Archipelago::Disco::MC) + possible_replacements = Archipelago::Disco::MC.lookup(Archipelago::Disco::Query.new({:service_id => @chest_id})) + raise e if possible_replacements.empty? + + @chest = possible_replacements[@chest_id][:service] + CHESTS_BY_SERVICE_ID[@chest_id] = @chest + + retry + else + raise e + end + end else return super(meth, *args) end @@ -218,6 +268,7 @@ initialize_prepared(options[:transaction_recovery_interval] || TRANSACTION_RECOVERY_INTERVAL) + CHESTS_BY_SERVICE_ID[self.service_id] = self end # @@ -265,7 +316,7 @@ if Dubloon === instance return instance.join(transaction) else - return Dubloon.new(key, DRbObject.new(self), transaction, self.service_id, instance.public_methods) + return Dubloon.new(key, self, transaction, self.service_id, instance.public_methods) end end @@ -510,7 +561,7 @@ @snapshot_by_transaction[transaction] = {} @snapshot_by_transaction[transaction].extend(Archipelago::Current::Synchronized) @timestamp_by_key_by_transaction[transaction] = {} - transaction.join(DRbObject.new(self)) + transaction.join(self) end end else @@ -670,7 +721,7 @@ return value if Dubloon === value - return Dubloon.new(key, DRbObject.new(self), transaction, service_id, value.public_methods) + return Dubloon.new(key, self, transaction, service_id, value.public_methods) end # Modified: trunk/archipelago/scripts/pirate.rb =================================================================== --- trunk/archipelago/scripts/pirate.rb 2006-11-14 15:45:48 UTC (rev 19) +++ trunk/archipelago/scripts/pirate.rb 2006-11-14 18:11:33 UTC (rev 20) @@ -2,5 +2,18 @@ require 'archipelago/pirate' +# +# An overload of Archipelago::Hashish::BerkeleyHashish +# just to get a nice debugging method. +# +class Archipelago::Hashish::BerkeleyHashish + # + # Get the serialized value for +key+. + # + def get_serialized_value(key) + @content_db[Marshal.dump(key)] + end +end + DRb.start_service("druby://localhost:#{rand(1000) + 5000}") @p = Archipelago::Pirate::Captain.new Added: trunk/archipelago/tests/disco_benchmark.rb =================================================================== --- trunk/archipelago/tests/disco_benchmark.rb (rev 0) +++ trunk/archipelago/tests/disco_benchmark.rb 2006-11-14 18:11:33 UTC (rev 20) @@ -0,0 +1,21 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class DiscoBenchmark < Test::Unit::TestCase + + def test_lookup + dj1 = TestJockey.new + dj2 = TestJockey.new + x = 0 + bm("Jockey#publish/lookup", :n => 10) do + dj1.publish(Archipelago::Disco::Record.new({:service_id => x, :validator => Archipelago::Disco::MockValidator.new})) + assert(!dj2.lookup(Archipelago::Disco::Query.new({:service_id => x})).empty?) + x += 1 + end + dj1.publish(Archipelago::Disco::Record.new({:service_id => "brappa", :validator => Archipelago::Disco::MockValidator.new})) + bm("Jockey#lookup", :n => 10) do + assert(!dj2.lookup(Archipelago::Disco::Query.new({:service_id => "brappa"})).empty?) + end + end + +end Modified: trunk/archipelago/tests/disco_test.rb =================================================================== --- trunk/archipelago/tests/disco_test.rb 2006-11-14 15:45:48 UTC (rev 19) +++ trunk/archipelago/tests/disco_test.rb 2006-11-14 18:11:33 UTC (rev 20) @@ -22,7 +22,6 @@ @d2 = TestJockey.new(:thrifty_publishing => false) @v1 = RemoteValidator.new(true) @p1 = Archipelago::Disco::Record.new(:service_id => 1, - :published_at => Time.now, :validator => DRbObject.new(@v1), :epa => "blar") @d1.publish(@p1) @@ -74,7 +73,6 @@ assert(empty) @d1.publish(Archipelago::Disco::Record.new(:service_id => 1, - :published_at => Time.now, :validator => Archipelago::Disco::MockValidator.new, :epa => "blar2")) @@ -92,7 +90,6 @@ @ltq.clear @d1.publish(Archipelago::Disco::Record.new(:glada => "jaa", - :published_at => Time.now, :validator => Archipelago::Disco::MockValidator.new, :service_id => 344)) sleep 0.1 @@ -112,7 +109,6 @@ @ltq.clear c3 = Archipelago::Disco::Jockey.new(:thrifty_publishing => true) c3.publish(Archipelago::Disco::Record.new(:glad => "ja", - :published_at => Time.now, :validator => Archipelago::Disco::MockValidator.new, :service_id => 33)) sleep 0.1 @@ -127,7 +123,6 @@ def test_thrifty_replying @d1.publish(Archipelago::Disco::Record.new(:gladaa => "jaaa", - :published_at => Time.now, :validator => Archipelago::Disco::MockValidator.new, :service_id => 3444)) @@ -148,7 +143,6 @@ @d2.stop! c3 = Archipelago::Disco::Jockey.new(:thrifty_replying => true, :thrifty_publishing => true) c3.publish(Archipelago::Disco::Record.new(:glad2 => "ja2", - :published_at => Time.now, :validator => Archipelago::Disco::MockValidator.new, :service_id => 34)) @@ -172,7 +166,6 @@ def test_thrifty_caching @d2.publish(Archipelago::Disco::Record.new(:bojkotta => "jag", - :published_at => Time.now, :validator => Archipelago::Disco::MockValidator.new, :service_id => 411)) sleep 0.1 @@ -182,7 +175,6 @@ assert(!c1.local_services.include?(41)) assert(!c1.remote_services.include?(41)) @d1.publish(Archipelago::Disco::Record.new(:bojkott => "ja", - :published_at => Time.now, :validator => Archipelago::Disco::MockValidator.new, :service_id => 41)) sleep 0.1 Modified: trunk/archipelago/tests/test_helper.rb =================================================================== --- trunk/archipelago/tests/test_helper.rb 2006-11-14 15:45:48 UTC (rev 19) +++ trunk/archipelago/tests/test_helper.rb 2006-11-14 18:11:33 UTC (rev 20) @@ -2,6 +2,8 @@ home = File.expand_path(File.dirname(__FILE__)) $: << File.join(home, "..", "lib") +MC_DISABLED = true + require 'pp' require 'drb' require 'test/unit' Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-14 15:45:48 UTC (rev 19) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-14 18:11:33 UTC (rev 20) @@ -24,8 +24,12 @@ end def self.reject(&block) - self.select.reject do |record| - yield(record) + if block + self.select.reject do |record| + yield(record) + end + else + @@pirate[self.all_key] end end From nobody at rubyforge.org Tue Nov 14 13:14:37 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 14 Nov 2006 13:14:37 -0500 (EST) Subject: [Archipelago-submits] [21] trunk/archipelago/Rakefile: new release Message-ID: <20061114181437.2008EA97002D@rubyforge.org> Revision: 21 Author: zond Date: 2006-11-14 13:14:36 -0500 (Tue, 14 Nov 2006) Log Message: ----------- new release Modified Paths: -------------- trunk/archipelago/Rakefile Modified: trunk/archipelago/Rakefile =================================================================== --- trunk/archipelago/Rakefile 2006-11-14 18:11:33 UTC (rev 20) +++ trunk/archipelago/Rakefile 2006-11-14 18:14:36 UTC (rev 21) @@ -9,7 +9,7 @@ spec = Gem::Specification.new do |s| s.platform = Gem::Platform::RUBY s.name = "archipelago" - s.version = "0.1.0" + s.version = "0.1.1" s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A set of tools for distributed computing in ruby." From nobody at rubyforge.org Tue Nov 14 13:18:50 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 14 Nov 2006 13:18:50 -0500 (EST) Subject: [Archipelago-submits] [22] tags/: added tags Message-ID: <20061114181850.EDFCBA970011@rubyforge.org> Revision: 22 Author: zond Date: 2006-11-14 13:18:50 -0500 (Tue, 14 Nov 2006) Log Message: ----------- added tags Added Paths: ----------- tags/ From nobody at rubyforge.org Tue Nov 14 13:18:58 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 14 Nov 2006 13:18:58 -0500 (EST) Subject: [Archipelago-submits] [23] tags/release_0_1_1/: release 0.1.1 Message-ID: <20061114181858.656D8A970019@rubyforge.org> Revision: 23 Author: zond Date: 2006-11-14 13:18:58 -0500 (Tue, 14 Nov 2006) Log Message: ----------- release 0.1.1 Added Paths: ----------- tags/release_0_1_1/ Copied: tags/release_0_1_1 (from rev 22, trunk) From nobody at rubyforge.org Tue Nov 14 13:24:58 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 14 Nov 2006 13:24:58 -0500 (EST) Subject: [Archipelago-submits] [24] trunk/archipelago/scripts: cleaned up some scripts Message-ID: <20061114182458.14928A970011@rubyforge.org> Revision: 24 Author: zond Date: 2006-11-14 13:24:57 -0500 (Tue, 14 Nov 2006) Log Message: ----------- cleaned up some scripts Modified Paths: -------------- trunk/archipelago/scripts/chest.rb trunk/archipelago/scripts/pirate.rb trunk/archipelago/scripts/tranny.rb Modified: trunk/archipelago/scripts/chest.rb =================================================================== --- trunk/archipelago/scripts/chest.rb 2006-11-14 18:18:58 UTC (rev 23) +++ trunk/archipelago/scripts/chest.rb 2006-11-14 18:24:57 UTC (rev 24) @@ -1,7 +1,7 @@ #!/usr/bin/env ruby if ARGV.size < 1 - puts "Usage: #{$0} DB-PATH [DRB-URI]" + puts "Usage: #{$0} DB-PATH" exit 1 end @@ -9,11 +9,6 @@ require 'archipelago/treasure' -if ARGV.size > 1 - DRb.start_service(ARGV[1]) -else - DRb.start_service -end c = Archipelago::Treasure::Chest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(ARGV[0]))) c.publish! Modified: trunk/archipelago/scripts/pirate.rb =================================================================== --- trunk/archipelago/scripts/pirate.rb 2006-11-14 18:18:58 UTC (rev 23) +++ trunk/archipelago/scripts/pirate.rb 2006-11-14 18:24:57 UTC (rev 24) @@ -15,5 +15,5 @@ end end -DRb.start_service("druby://localhost:#{rand(1000) + 5000}") +DRb.start_service @p = Archipelago::Pirate::Captain.new Modified: trunk/archipelago/scripts/tranny.rb =================================================================== --- trunk/archipelago/scripts/tranny.rb 2006-11-14 18:18:58 UTC (rev 23) +++ trunk/archipelago/scripts/tranny.rb 2006-11-14 18:24:57 UTC (rev 24) @@ -1,7 +1,7 @@ #!/usr/bin/env ruby if ARGV.size < 1 - puts "Usage: #{$0} DB-PATH DRB-URI" + puts "Usage: #{$0} DB-PATH" exit 1 end @@ -9,11 +9,6 @@ require 'archipelago/treasure' -if ARGV.size > 1 - DRb.start_service(ARGV[1]) -else - DRb.start_service -end t = Archipelago::Tranny::Manager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(ARGV[0]))) t.publish! From nobody at rubyforge.org Tue Nov 14 13:32:11 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 14 Nov 2006 13:32:11 -0500 (EST) Subject: [Archipelago-submits] [25] trunk/archipelago/lib/archipelago/disco.rb: added comments to MC Message-ID: <20061114183211.EAEECA970011@rubyforge.org> Revision: 25 Author: zond Date: 2006-11-14 13:32:11 -0500 (Tue, 14 Nov 2006) Log Message: ----------- added comments to MC Modified Paths: -------------- trunk/archipelago/lib/archipelago/disco.rb Modified: trunk/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-14 18:24:57 UTC (rev 24) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-14 18:32:11 UTC (rev 25) @@ -566,6 +566,12 @@ end + # + # The default Archipelago::Disco::Jockey that is always available for lookups is Archipelago::Disco::MC. + # + # If you really need to you can customize it by defining MC_OPTIONS before loading disco.rb, and if you REALLY + # need to you can disable it completely by setting MC_DISABLED to true. + # MC = Jockey.new(defined?(MC_OPTIONS) ? MC_OPTIONS : {}) unless defined?(MC_DISABLED) && MC_DISABLED end From nobody at rubyforge.org Tue Nov 14 14:50:11 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 14 Nov 2006 14:50:11 -0500 (EST) Subject: [Archipelago-submits] [26] trunk: added lots of comments and methods to Hyperactive::Record (not tested so far). Message-ID: <20061114195011.86FCB524166D@rubyforge.org> Revision: 26 Author: zond Date: 2006-11-14 14:50:10 -0500 (Tue, 14 Nov 2006) Log Message: ----------- added lots of comments and methods to Hyperactive::Record (not tested so far). added overloaded debug methods to the pirate/console script. added DRb.start_service to tranny.rb and chest.rb scripts. Modified Paths: -------------- trunk/archipelago/scripts/chest.rb trunk/archipelago/scripts/pirate.rb trunk/archipelago/scripts/tranny.rb trunk/hyperactive/lib/hyperactive/record.rb Added Paths: ----------- trunk/archipelago/scripts/overloads.rb Modified: trunk/archipelago/scripts/chest.rb =================================================================== --- trunk/archipelago/scripts/chest.rb 2006-11-14 18:32:11 UTC (rev 25) +++ trunk/archipelago/scripts/chest.rb 2006-11-14 19:50:10 UTC (rev 26) @@ -9,6 +9,8 @@ require 'archipelago/treasure' +DRb.start_service + c = Archipelago::Treasure::Chest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(ARGV[0]))) c.publish! Added: trunk/archipelago/scripts/overloads.rb =================================================================== --- trunk/archipelago/scripts/overloads.rb (rev 0) +++ trunk/archipelago/scripts/overloads.rb 2006-11-14 19:50:10 UTC (rev 26) @@ -0,0 +1,61 @@ + +# +# An overload of Archipelago::Hashish::BerkeleyHashish +# just to get a few nice debugging methods. +# +class Archipelago::Hashish::BerkeleyHashish + attr_reader :content + # + # Get the serialized value for +key+. + # + def get_serialized_value(key) + @content_db[Marshal.dump(key)] + end + # + # Yield to +block+ once for every key/value inside this BerkeleyHashish. + # + def each(&block) + @content_db.each do |serialized_key, serialized_value| + key = Marshal.load(serialized_key) + yield(key, self[key]) + end + end + # + # Empties the BerkeleyHashish. + # + def clear + @content_db.clear + @content.clear + @timestamps_db.clear + @timestamps.clear + end +end + +# +# An overload of Archipelago::Treasure::Chest just to +# get a few nice debugging methods. +# +class Archipelago::Treasure::Chest + # + # Yield to +block+ once for every key/value inside this Chest. + # + def each(&block) + @db.each do |key, value| + yield(key, value) + end + end + # + # Returns a Hash containing what we contain. + # + def all + @db.each do |key, value| + end + @db.content + end + # + # Empties the Chest. + # + def clear + @db.clear + end +end Modified: trunk/archipelago/scripts/pirate.rb =================================================================== --- trunk/archipelago/scripts/pirate.rb 2006-11-14 18:32:11 UTC (rev 25) +++ trunk/archipelago/scripts/pirate.rb 2006-11-14 19:50:10 UTC (rev 26) @@ -2,18 +2,6 @@ require 'archipelago/pirate' -# -# An overload of Archipelago::Hashish::BerkeleyHashish -# just to get a nice debugging method. -# -class Archipelago::Hashish::BerkeleyHashish - # - # Get the serialized value for +key+. - # - def get_serialized_value(key) - @content_db[Marshal.dump(key)] - end -end - DRb.start_service @p = Archipelago::Pirate::Captain.new + at p.evaluate!(File.join(File.dirname(__FILE__), 'overloads.rb')) Modified: trunk/archipelago/scripts/tranny.rb =================================================================== --- trunk/archipelago/scripts/tranny.rb 2006-11-14 18:32:11 UTC (rev 25) +++ trunk/archipelago/scripts/tranny.rb 2006-11-14 19:50:10 UTC (rev 26) @@ -9,6 +9,8 @@ require 'archipelago/treasure' +DRb.start_service + t = Archipelago::Tranny::Manager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(ARGV[0]))) t.publish! Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-14 18:32:11 UTC (rev 25) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-14 19:50:10 UTC (rev 26) @@ -1,59 +1,190 @@ require 'rubygems' require 'archipelago' +require 'set' +# +# A utility module to provide the functionality required for example +# if you want to use archipelago in a Ruby on Rails project *hint hint*. +# module Hyperactive + # + # Just like a normal Set except that you can send it matcher objects + # instead of blocks for select/reject-like functionality. This due + # to the fact that a block will cause lots and lots of extra DRb traffic. + # + class MatchSet < Set + # + # Return all members of this MatchSet that return true on +matcher+.matches? + # + def match(matcher) + self.select do |e| + matcher.matches?(e) + end + end + # + # Return all members of this MatchSet except those that return true on +matcher+.matches? + # + def antimatch(matcher) + self.reject do |e| + matcher.matches?(e) + end + end + # + # Remove all entries in this MatchSet from the database and from ourselves. + # + def clear + self.each do |e| + e.destroy + end + super + end + end + + # + # A convenient base class to inherit when you want the basic utility methods + # provided by for example ActiveRecord::Base *hint hint*. + # + # NB: After an instance is created, it will actually return a copy within your local machine + # which is not what is usually the case. Every other time you fetch it using a select or other + # method you will instead receive a proxy object to the database. This means that nothing you + # do to it at that point will be persistent or even necessarily have a defined result. + # Therefore: do not use MyRecordSubclass.new to MyRecordSubclass#initialize objects, + # instead use MyRecordSubclass.get_instance, since it will return a fresh proxy object + # instead of the devious original: + # my_instance = MyRecordSubclass.get_instance(*any_old_arguments) + # class Record + # + # The default database connector. + # @@pirate = Archipelago::Pirate::Captain.new + # + # Call this if you want to change the default database connector + # to something else. + # def self.setup(options = {}) - @@options = options @@pirate = Archipelago::Pirate::Captain.new(options[:pirate_options]) end + # + # Will return all instances of this class that when sent to +matcher+.matches? + # return true. + # + def self.match(matcher) + self.all.match(matcher) + end + + # + # Will return all instances except those that when sent to +matcher+.matches? + # return true. + # + def self.antimatch(matcher) + self.all.antimatch(matcher) + end + + # + # Will return all instances of this class returning true when yielded to +block+. + # + # Will return all instances if no +block+ is given. + # def self.select(&block) if block self.select.select do |record| yield(record) end else - @@pirate[self.all_key] + self.all || MatchSet.new end end + # + # Will return all instances of this class except those returning true when yielded to +block+. + # + # Will return no instances if no +block+ is given. + # def self.reject(&block) if block self.select.reject do |record| yield(record) end else - @@pirate[self.all_key] + MatchSet.new end end + # + # Return the record with +record_id+. + # + def self.find(record_id) + @@pirate[record_id] + end + + # + # Will delete this instance from the database and + # from the set of known instances. + # def self.delete(instance) + @@pirate[self.all_key].delete(instance) @@pirate.delete(instance.record_id) end + # + # Will execute +block+ within a transaction. + # def self.transaction(&block) @@pirate.transaction(&block) end - attr_reader :record_id + # + # Will return all known instances of this class. + # + def self.all + @@pirate[self.all_key] + end - def initialize - host = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" - @record_id = Digest::SHA1.hexdigest("#{host}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") + # + # Use this method to get new instances of this class, since it will actually + # make sure it both resides in the database and return a proxy to the remote + # object. + # + def self.get_instance(*arguments) + instance = self.new(*arguments) + instance.instance_eval do + host = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" + @record_id = Digest::SHA1.hexdigest("#{host}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") + end + @@pirate.evaluate!(File.expand_path(__FILE__)) - @@pirate[@record_id] = self - @@pirate[self.class.all_key] ||= Set.new - @@pirate[self.class.all_key] << @@pirate[@record_id] + @@pirate[instance.record_id] = instance + + proxy = @@pirate[instance.record_id] + + @@pirate[self.all_key] ||= MatchSet.new + @@pirate[self.all_key] << proxy + return proxy end + # + # Our semi-unique id. + # + attr_reader :record_id + + # + # + # + def destroy + self.class.delete(self) + end + private + # + # The key used to store the MatchSet containing proxies to all our instances. + # def self.all_key "HyperactiveRecord:#{self.name}:all" end From nobody at rubyforge.org Fri Nov 17 08:55:04 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Fri, 17 Nov 2006 08:55:04 -0500 (EST) Subject: [Archipelago-submits] [27] trunk/archipelago: added possibility to setup Jockey and Captain after initialization Message-ID: <20061117135504.963B5A970002@rubyforge.org> Revision: 27 Author: zond Date: 2006-11-17 08:55:03 -0500 (Fri, 17 Nov 2006) Log Message: ----------- added possibility to setup Jockey and Captain after initialization Modified Paths: -------------- trunk/archipelago/lib/archipelago/disco.rb trunk/archipelago/lib/archipelago/pirate.rb Added Paths: ----------- trunk/archipelago/scripts/services.rb Modified: trunk/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-14 19:50:10 UTC (rev 26) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-17 13:55:03 UTC (rev 27) @@ -325,21 +325,34 @@ # or if not given THRIFTY_REPLYING. Otherwise will reply with multicasts. # def initialize(options = {}) - @thrifty_caching = options.include?(:thrifty_caching) ? options[:thrifty_caching] : THRIFTY_CACHING - @thrifty_replying = options.include?(:thrifty_replying) ? options[:thrifty_replying] : THRIFTY_REPLYING - @thrifty_publishing = options.include?(:thrifty_publishing) ? options[:thrifty_publishing] : THRIFTY_PUBLISHING - @lookup_timeout = options[:lookup_timeout] || LOOKUP_TIMEOUT - @initial_lookup_standoff = options[:initial_lookup_standoff] || INITIAL_LOOKUP_STANDOFF - @remote_services = ServiceLocker.new @local_services = ServiceLocker.new @subscribed_services = Set.new @incoming = Queue.new @outgoing = Queue.new - + @new_service_semaphore = MonitorMixin::ConditionVariable.new(Archipelago::Current::Lock.new) - + + setup(options) + + start_listener + start_unilistener + start_shouter + start_picker + start_validator(options[:validation_interval] || VALIDATION_INTERVAL) + end + + # + # Sets up this instance according to the given +options+. + # + def setup(options = {}) + @thrifty_caching = options.include?(:thrifty_caching) ? options[:thrifty_caching] : THRIFTY_CACHING + @thrifty_replying = options.include?(:thrifty_replying) ? options[:thrifty_replying] : THRIFTY_REPLYING + @thrifty_publishing = options.include?(:thrifty_publishing) ? options[:thrifty_publishing] : THRIFTY_PUBLISHING + @lookup_timeout = options[:lookup_timeout] || LOOKUP_TIMEOUT + @initial_lookup_standoff = options[:initial_lookup_standoff] || INITIAL_LOOKUP_STANDOFF + @listener = UDPSocket.new @unilistener = UDPSocket.new @@ -377,12 +390,6 @@ @sender.connect(options[:address] || ADDRESS, options[:port] || PORT) @unisender = UDPSocket.new - - start_listener - start_unilistener - start_shouter - start_picker - start_validator(options[:validation_interval] || VALIDATION_INTERVAL) end # Modified: trunk/archipelago/lib/archipelago/pirate.rb =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb 2006-11-14 19:50:10 UTC (rev 26) +++ trunk/archipelago/lib/archipelago/pirate.rb 2006-11-17 13:55:03 UTC (rev 27) @@ -78,22 +78,37 @@ # of required classes and modules at the chest. # def initialize(options = {}) - @treasure_map = defined?(Archipelago::Disco::MC) ? Archipelago::Disco::MC : Archipelago::Disco::Jockey.new(options[:jockey_options] || {}) + setup(options) + @transaction = nil + + start_service_updater(options[:initial_service_update_interval] || INITIAL_SERVICE_UPDATE_INTERVAL, + options[:maximum_service_update_interval] || MAXIMUM_SERVICE_UPDATE_INTERVAL) + + end + + # + # Sets up this instance with the given +options+. + # + def setup(options = {}) + @treasure_map ||= nil + if defined?(Archipelago::Disco::MC) + @treasure_map.stop! if @treasure_map && @treasure_map != Archipelago::Disco::MC + @treasure_map = options.include?(:jockey_options) ? Archipelago::Disco::Jockey.new(options[:jockey_options]) : Archipelago::Disco::MC + else + @treasure_map.stop! if @treasure_map + @treasure_map = Archipelago::Disco::Jockey.new(options[:jockey_options] || {}) + end + @chest_description = CHEST_DESCRIPTION.merge(options[:chest_description] || {}) @tranny_description = TRANNY_DESCRIPTION.merge(options[:tranny_description] || {}) - @jockey_options = options[:jockey_options] || {} - @chest_eval_files = options[:chest_eval_files] || [] + @chest_eval_files ||= [] + @chest_eval_files += options[:chest_eval_files] || [] - @chests_having_evaluated = {} + @chests_having_evaluated ||= {} - start_service_updater(options[:initial_service_update_interval] || INITIAL_SERVICE_UPDATE_INTERVAL, - options[:maximum_service_update_interval] || MAXIMUM_SERVICE_UPDATE_INTERVAL) - @yar_counter = 0 - - @transaction = nil end # Added: trunk/archipelago/scripts/services.rb =================================================================== --- trunk/archipelago/scripts/services.rb (rev 0) +++ trunk/archipelago/scripts/services.rb 2006-11-17 13:55:03 UTC (rev 27) @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +if ARGV.size < 1 + puts "Usage: #{$0} DB-PATH" + exit 1 +end + +$: << File.join(File.dirname(__FILE__), "..", "lib") + +require 'archipelago/treasure' +require 'archipelago/tranny' + +DRb.start_service + +t = Archipelago::Tranny::Manager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.join(ARGV[0], "tranny")))) +t.publish! +c = Archipelago::Treasure::Chest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.join(ARGV[0], "chest")))) +c.publish! + +DRb.thread.join Property changes on: trunk/archipelago/scripts/services.rb ___________________________________________________________________ Name: svn:executable + * From nobody at rubyforge.org Sun Nov 19 06:19:35 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sun, 19 Nov 2006 06:19:35 -0500 (EST) Subject: [Archipelago-submits] [28] trunk/archipelago: made bm a method on Object instead of TestCase in test_helper.rb. Message-ID: <20061119111935.E43E55240CA5@rubyforge.org> Revision: 28 Author: zond Date: 2006-11-19 06:19:35 -0500 (Sun, 19 Nov 2006) Log Message: ----------- made bm a method on Object instead of TestCase in test_helper.rb. added threaded collection methods in current. and tests. created a services script that starts a chest and a tranny. made all scripts able to load extra libs. Modified Paths: -------------- trunk/archipelago/lib/archipelago/current.rb trunk/archipelago/scripts/chest.rb trunk/archipelago/scripts/services.rb trunk/archipelago/scripts/tranny.rb trunk/archipelago/tests/current_test.rb trunk/archipelago/tests/test_helper.rb Modified: trunk/archipelago/lib/archipelago/current.rb =================================================================== --- trunk/archipelago/lib/archipelago/current.rb 2006-11-17 13:55:03 UTC (rev 27) +++ trunk/archipelago/lib/archipelago/current.rb 2006-11-19 11:19:35 UTC (rev 28) @@ -34,6 +34,73 @@ module Current # + # Adds a few threaded methods to the normal ruby collections. + # + module ThreadedCollection + + # + # Like each, except calls +block+ within a new thread. + # + def t_each(&block) + threads = [] + self.each do |*args| + threads << Thread.new do + yield(*args) + end + end + threads.each do |thread| + thread.join + end + end + + # + # Like collect, except calls +block+ within a new thread. + # + def t_collect(&block) + result = [] + result.extend(Synchronized) + self.t_each do |*args| + result.synchronize do + result << yield(*args) + end + end + return result + end + + # + # Like select, except calls +block+ within a new thread. + # + def t_select(&block) + result = [] + result.extend(Synchronized) + self.t_each do |*args| + matches = yield(*args) + result.synchronize do + result << args.first + end if matches + end + return result + end + + # + # Like reject, except calls +block+ within a new thread. + # + def t_reject(&block) + result = [] + result.extend(Synchronized) + self.t_each do |*args| + matches = yield(*args) + result.synchronize do + result << args.first + end unless matches + end + return result + end + + end + + + # # A module that will allow any class to synchronize over any other # object. # Modified: trunk/archipelago/scripts/chest.rb =================================================================== --- trunk/archipelago/scripts/chest.rb 2006-11-17 13:55:03 UTC (rev 27) +++ trunk/archipelago/scripts/chest.rb 2006-11-19 11:19:35 UTC (rev 28) @@ -1,7 +1,7 @@ #!/usr/bin/env ruby if ARGV.size < 1 - puts "Usage: #{$0} DB-PATH" + puts "Usage: #{$0} DB-PATH [EXTRA-LIBS]" exit 1 end @@ -9,6 +9,10 @@ require 'archipelago/treasure' +ARGV[1..-1].each do |lib| + load lib +end + DRb.start_service c = Archipelago::Treasure::Chest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(ARGV[0]))) Modified: trunk/archipelago/scripts/services.rb =================================================================== --- trunk/archipelago/scripts/services.rb 2006-11-17 13:55:03 UTC (rev 27) +++ trunk/archipelago/scripts/services.rb 2006-11-19 11:19:35 UTC (rev 28) @@ -1,7 +1,7 @@ #!/usr/bin/env ruby if ARGV.size < 1 - puts "Usage: #{$0} DB-PATH" + puts "Usage: #{$0} DB-PATH [EXTRA-LIBS]" exit 1 end @@ -10,6 +10,10 @@ require 'archipelago/treasure' require 'archipelago/tranny' +ARGV[1..-1].each do |lib| + load lib +end + DRb.start_service t = Archipelago::Tranny::Manager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.join(ARGV[0], "tranny")))) Modified: trunk/archipelago/scripts/tranny.rb =================================================================== --- trunk/archipelago/scripts/tranny.rb 2006-11-17 13:55:03 UTC (rev 27) +++ trunk/archipelago/scripts/tranny.rb 2006-11-19 11:19:35 UTC (rev 28) @@ -1,7 +1,7 @@ #!/usr/bin/env ruby if ARGV.size < 1 - puts "Usage: #{$0} DB-PATH" + puts "Usage: #{$0} DB-PATH [EXTRA-LIBS]" exit 1 end @@ -9,6 +9,10 @@ require 'archipelago/treasure' +ARGV[1..-1].each do |lib| + load lib +end + DRb.start_service t = Archipelago::Tranny::Manager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(ARGV[0]))) Modified: trunk/archipelago/tests/current_test.rb =================================================================== --- trunk/archipelago/tests/current_test.rb 2006-11-17 13:55:03 UTC (rev 27) +++ trunk/archipelago/tests/current_test.rb 2006-11-19 11:19:35 UTC (rev 28) @@ -24,4 +24,19 @@ end end + def test_threaded_collection + a = Array(10) + a.extend(Archipelago::Current::ThreadedCollection) + + assert_equal(a.select do |e| e == nil end, + a.t_select do |e| e == nil end) + assert_equal(a.reject do |e| e == nil end, + a.t_reject do |e| e == nil end) + assert_equal(a.collect do |e| e == nil end, + a.t_collect do |e| e == nil end) + b = [] + a.each do |e| b << e end + assert_equal(b, a) + end + end Modified: trunk/archipelago/tests/test_helper.rb =================================================================== --- trunk/archipelago/tests/test_helper.rb 2006-11-17 13:55:03 UTC (rev 27) +++ trunk/archipelago/tests/test_helper.rb 2006-11-19 11:19:35 UTC (rev 28) @@ -37,21 +37,21 @@ attr_reader :persistence_provider end -class Test::Unit::TestCase - - def bm(label, options = {}) - n = options[:n] || 1000 - width = options[:width] || 50 - Benchmark.benchmark(" " * width + Benchmark::Tms::CAPTION, width, Benchmark::Tms::FMTSTR, "ms/call") do |b| - times = b.report("#{n}x#{label}") do - n.times do - yield - end +def bm(label = "", options = {}) + n = options[:n] || 1000 + width = options[:width] || 50 + Benchmark.benchmark(" " * width + Benchmark::Tms::CAPTION, width, Benchmark::Tms::FMTSTR, "ms/call") do |b| + times = b.report("#{n}x#{label}") do + n.times do + yield end - [times * 1000 / n.to_f] end + [times * 1000 / n.to_f] end +end +class Test::Unit::TestCase + def assert_within(timeout, &block) t = Time.new rval = yield From nobody at rubyforge.org Sun Nov 19 06:20:38 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sun, 19 Nov 2006 06:20:38 -0500 (EST) Subject: [Archipelago-submits] [29] trunk/archipelago: moved scripts to script Message-ID: <20061119112038.350F45240CA5@rubyforge.org> Revision: 29 Author: zond Date: 2006-11-19 06:20:37 -0500 (Sun, 19 Nov 2006) Log Message: ----------- moved scripts to script Modified Paths: -------------- trunk/archipelago/README Added Paths: ----------- trunk/archipelago/script/ trunk/archipelago/script/chest.rb trunk/archipelago/script/overloads.rb trunk/archipelago/script/pirate.rb trunk/archipelago/script/services.rb trunk/archipelago/script/tranny.rb Removed Paths: ------------- trunk/archipelago/script/chest.rb trunk/archipelago/script/pirate.rb trunk/archipelago/script/tranny.rb trunk/archipelago/scripts/ Modified: trunk/archipelago/README =================================================================== --- trunk/archipelago/README 2006-11-19 11:19:35 UTC (rev 28) +++ trunk/archipelago/README 2006-11-19 11:20:37 UTC (rev 29) @@ -20,15 +20,15 @@ To set up an Archipelago::Tranny::Manager do the following (from scripts/tranny.rb): - :include:scripts/tranny.rb + :include:script/tranny.rb To set up an Archipelago::Treasure::Chest do the following (from scripts/chest.rb): - :include:scripts/chest.rb + :include:script/chest.rb To set up an Archipelago::Pirate::Captain do the following (from scripts/pirate.rb): - :include:scripts/pirate.rb + :include:script/pirate.rb To set up a test environment to play around with, run the following commands in a few terminals: Copied: trunk/archipelago/script (from rev 23, trunk/archipelago/scripts) Deleted: trunk/archipelago/script/chest.rb =================================================================== --- trunk/archipelago/scripts/chest.rb 2006-11-14 18:18:58 UTC (rev 23) +++ trunk/archipelago/script/chest.rb 2006-11-19 11:20:37 UTC (rev 29) @@ -1,20 +0,0 @@ -#!/usr/bin/env ruby - -if ARGV.size < 1 - puts "Usage: #{$0} DB-PATH [DRB-URI]" - exit 1 -end - -$: << File.join(File.dirname(__FILE__), "..", "lib") - -require 'archipelago/treasure' - -if ARGV.size > 1 - DRb.start_service(ARGV[1]) -else - DRb.start_service -end -c = Archipelago::Treasure::Chest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(ARGV[0]))) -c.publish! - -DRb.thread.join Copied: trunk/archipelago/script/chest.rb (from rev 28, trunk/archipelago/scripts/chest.rb) =================================================================== --- trunk/archipelago/script/chest.rb (rev 0) +++ trunk/archipelago/script/chest.rb 2006-11-19 11:20:37 UTC (rev 29) @@ -0,0 +1,21 @@ +#!/usr/bin/env ruby + +if ARGV.size < 1 + puts "Usage: #{$0} DB-PATH [EXTRA-LIBS]" + exit 1 +end + +$: << File.join(File.dirname(__FILE__), "..", "lib") + +require 'archipelago/treasure' + +ARGV[1..-1].each do |lib| + load lib +end + +DRb.start_service + +c = Archipelago::Treasure::Chest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(ARGV[0]))) +c.publish! + +DRb.thread.join Copied: trunk/archipelago/script/overloads.rb (from rev 28, trunk/archipelago/scripts/overloads.rb) =================================================================== --- trunk/archipelago/script/overloads.rb (rev 0) +++ trunk/archipelago/script/overloads.rb 2006-11-19 11:20:37 UTC (rev 29) @@ -0,0 +1,61 @@ + +# +# An overload of Archipelago::Hashish::BerkeleyHashish +# just to get a few nice debugging methods. +# +class Archipelago::Hashish::BerkeleyHashish + attr_reader :content + # + # Get the serialized value for +key+. + # + def get_serialized_value(key) + @content_db[Marshal.dump(key)] + end + # + # Yield to +block+ once for every key/value inside this BerkeleyHashish. + # + def each(&block) + @content_db.each do |serialized_key, serialized_value| + key = Marshal.load(serialized_key) + yield(key, self[key]) + end + end + # + # Empties the BerkeleyHashish. + # + def clear + @content_db.clear + @content.clear + @timestamps_db.clear + @timestamps.clear + end +end + +# +# An overload of Archipelago::Treasure::Chest just to +# get a few nice debugging methods. +# +class Archipelago::Treasure::Chest + # + # Yield to +block+ once for every key/value inside this Chest. + # + def each(&block) + @db.each do |key, value| + yield(key, value) + end + end + # + # Returns a Hash containing what we contain. + # + def all + @db.each do |key, value| + end + @db.content + end + # + # Empties the Chest. + # + def clear + @db.clear + end +end Deleted: trunk/archipelago/script/pirate.rb =================================================================== --- trunk/archipelago/scripts/pirate.rb 2006-11-14 18:18:58 UTC (rev 23) +++ trunk/archipelago/script/pirate.rb 2006-11-19 11:20:37 UTC (rev 29) @@ -1,19 +0,0 @@ -#!/usr/bin/env ruby - -require 'archipelago/pirate' - -# -# An overload of Archipelago::Hashish::BerkeleyHashish -# just to get a nice debugging method. -# -class Archipelago::Hashish::BerkeleyHashish - # - # Get the serialized value for +key+. - # - def get_serialized_value(key) - @content_db[Marshal.dump(key)] - end -end - -DRb.start_service("druby://localhost:#{rand(1000) + 5000}") - at p = Archipelago::Pirate::Captain.new Copied: trunk/archipelago/script/pirate.rb (from rev 28, trunk/archipelago/scripts/pirate.rb) =================================================================== --- trunk/archipelago/script/pirate.rb (rev 0) +++ trunk/archipelago/script/pirate.rb 2006-11-19 11:20:37 UTC (rev 29) @@ -0,0 +1,7 @@ +#!/usr/bin/env ruby + +require 'archipelago/pirate' + +DRb.start_service + at p = Archipelago::Pirate::Captain.new + at p.evaluate!(File.join(File.dirname(__FILE__), 'overloads.rb')) Copied: trunk/archipelago/script/services.rb (from rev 28, trunk/archipelago/scripts/services.rb) =================================================================== --- trunk/archipelago/script/services.rb (rev 0) +++ trunk/archipelago/script/services.rb 2006-11-19 11:20:37 UTC (rev 29) @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby + +if ARGV.size < 1 + puts "Usage: #{$0} DB-PATH [EXTRA-LIBS]" + exit 1 +end + +$: << File.join(File.dirname(__FILE__), "..", "lib") + +require 'archipelago/treasure' +require 'archipelago/tranny' + +ARGV[1..-1].each do |lib| + load lib +end + +DRb.start_service + +t = Archipelago::Tranny::Manager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.join(ARGV[0], "tranny")))) +t.publish! +c = Archipelago::Treasure::Chest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.join(ARGV[0], "chest")))) +c.publish! + +DRb.thread.join Deleted: trunk/archipelago/script/tranny.rb =================================================================== --- trunk/archipelago/scripts/tranny.rb 2006-11-14 18:18:58 UTC (rev 23) +++ trunk/archipelago/script/tranny.rb 2006-11-19 11:20:37 UTC (rev 29) @@ -1,20 +0,0 @@ -#!/usr/bin/env ruby - -if ARGV.size < 1 - puts "Usage: #{$0} DB-PATH DRB-URI" - exit 1 -end - -$: << File.join(File.dirname(__FILE__), "..", "lib") - -require 'archipelago/treasure' - -if ARGV.size > 1 - DRb.start_service(ARGV[1]) -else - DRb.start_service -end -t = Archipelago::Tranny::Manager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(ARGV[0]))) -t.publish! - -DRb.thread.join Copied: trunk/archipelago/script/tranny.rb (from rev 28, trunk/archipelago/scripts/tranny.rb) =================================================================== --- trunk/archipelago/script/tranny.rb (rev 0) +++ trunk/archipelago/script/tranny.rb 2006-11-19 11:20:37 UTC (rev 29) @@ -0,0 +1,21 @@ +#!/usr/bin/env ruby + +if ARGV.size < 1 + puts "Usage: #{$0} DB-PATH [EXTRA-LIBS]" + exit 1 +end + +$: << File.join(File.dirname(__FILE__), "..", "lib") + +require 'archipelago/treasure' + +ARGV[1..-1].each do |lib| + load lib +end + +DRb.start_service + +t = Archipelago::Tranny::Manager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(ARGV[0]))) +t.publish! + +DRb.thread.join From nobody at rubyforge.org Sun Nov 19 13:55:18 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sun, 19 Nov 2006 13:55:18 -0500 (EST) Subject: [Archipelago-submits] [30] trunk: created a file of its own for the hash class. Message-ID: <20061119185518.9EAFA5240989@rubyforge.org> Revision: 30 Author: zond Date: 2006-11-19 13:55:17 -0500 (Sun, 19 Nov 2006) Log Message: ----------- created a file of its own for the hash class. renamed it tree. fixed bugs in archipelago. Modified Paths: -------------- trunk/archipelago/Rakefile trunk/archipelago/lib/archipelago/current.rb trunk/archipelago/tests/current_test.rb trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive.rb Added Paths: ----------- trunk/hyperactive/lib/hyperactive/tree.rb trunk/hyperactive/test/tree_test.rb Modified: trunk/archipelago/Rakefile =================================================================== --- trunk/archipelago/Rakefile 2006-11-19 11:20:37 UTC (rev 29) +++ trunk/archipelago/Rakefile 2006-11-19 18:55:17 UTC (rev 30) @@ -13,7 +13,7 @@ s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A set of tools for distributed computing in ruby." - s.files = FileList['lib/archipelago.rb', 'lib/archipelago/*.rb', 'test/*', 'scripts/*', 'GPL-2', 'TODO', 'profiles/*'].to_a + s.files = FileList['lib/archipelago.rb', 'lib/archipelago/*.rb', 'test/*', 'script/*', 'GPL-2', 'TODO', 'profiles/*'].to_a s.require_path = "lib" s.autorequire = "archipelago" s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') Modified: trunk/archipelago/lib/archipelago/current.rb =================================================================== --- trunk/archipelago/lib/archipelago/current.rb 2006-11-19 11:20:37 UTC (rev 29) +++ trunk/archipelago/lib/archipelago/current.rb 2006-11-19 18:55:17 UTC (rev 30) @@ -36,6 +36,8 @@ # # Adds a few threaded methods to the normal ruby collections. # + # NB: Will work slightly different than the unthreaded ones in certain circumstances. + # module ThreadedCollection # @@ -43,9 +45,9 @@ # def t_each(&block) threads = [] - self.each do |*args| + self.each do |args| threads << Thread.new do - yield(*args) + yield(args) end end threads.each do |thread| @@ -59,9 +61,9 @@ def t_collect(&block) result = [] result.extend(Synchronized) - self.t_each do |*args| + self.t_each do |args| result.synchronize do - result << yield(*args) + result << yield(args) end end return result @@ -73,10 +75,10 @@ def t_select(&block) result = [] result.extend(Synchronized) - self.t_each do |*args| - matches = yield(*args) + self.t_each do |args| + matches = yield(args) result.synchronize do - result << args.first + result << args end if matches end return result @@ -88,10 +90,10 @@ def t_reject(&block) result = [] result.extend(Synchronized) - self.t_each do |*args| - matches = yield(*args) + self.t_each do |args| + matches = yield(args) result.synchronize do - result << args.first + result << args end unless matches end return result Modified: trunk/archipelago/tests/current_test.rb =================================================================== --- trunk/archipelago/tests/current_test.rb 2006-11-19 11:20:37 UTC (rev 29) +++ trunk/archipelago/tests/current_test.rb 2006-11-19 18:55:17 UTC (rev 30) @@ -35,8 +35,22 @@ assert_equal(a.collect do |e| e == nil end, a.t_collect do |e| e == nil end) b = [] - a.each do |e| b << e end + a.t_each do |e| b << e end assert_equal(b, a) + + a = {1 => 1, 2 => 2, 3 => 3, 4 => 4, 5 => 5} + a.extend(Archipelago::Current::ThreadedCollection) + + assert_equal(a.select do |k,v| k == nil end, + a.t_select do |k,v| k == nil end) + assert_equal(a.to_a.reject do |k,v| k == nil end, + a.t_reject do |k,v| k == nil end) + assert_equal(a.collect do |k,v| k == nil end, + a.t_collect do |k,c| k == nil end) + + b = {} + a.t_each do |k,v| b[k] = v end + assert_equal(b,a) end end Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-19 11:20:37 UTC (rev 29) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-19 18:55:17 UTC (rev 30) @@ -1,7 +1,6 @@ require 'rubygems' require 'archipelago' -require 'set' # # A utility module to provide the functionality required for example @@ -10,37 +9,9 @@ module Hyperactive # - # Just like a normal Set except that you can send it matcher objects - # instead of blocks for select/reject-like functionality. This due - # to the fact that a block will cause lots and lots of extra DRb traffic. + # The default database connector. # - class MatchSet < Set - # - # Return all members of this MatchSet that return true on +matcher+.matches? - # - def match(matcher) - self.select do |e| - matcher.matches?(e) - end - end - # - # Return all members of this MatchSet except those that return true on +matcher+.matches? - # - def antimatch(matcher) - self.reject do |e| - matcher.matches?(e) - end - end - # - # Remove all entries in this MatchSet from the database and from ourselves. - # - def clear - self.each do |e| - e.destroy - end - super - end - end + CAPTAIN = Archipelago::Pirate::Captain.new # # A convenient base class to inherit when you want the basic utility methods @@ -53,100 +24,38 @@ # Therefore: do not use MyRecordSubclass.new to MyRecordSubclass#initialize objects, # instead use MyRecordSubclass.get_instance, since it will return a fresh proxy object # instead of the devious original: - # my_instance = MyRecordSubclass.get_instance(*any_old_arguments) + # my_instance = MyRecordSubclass.get_instance(*the_same_arguments_as_to_initialize) # class Record - # - # The default database connector. - # - @@pirate = Archipelago::Pirate::Captain.new + @@pre_create_hooks = [] + @@post_create_hooks = [] + @@pre_destroy_hooks = [] + @@post_destroy_hooks = [] # # Call this if you want to change the default database connector # to something else. # def self.setup(options = {}) - @@pirate = Archipelago::Pirate::Captain.new(options[:pirate_options]) + CAPTAIN.setup(options[:pirate_options]) end # - # Will return all instances of this class that when sent to +matcher+.matches? - # return true. - # - def self.match(matcher) - self.all.match(matcher) - end - - # - # Will return all instances except those that when sent to +matcher+.matches? - # return true. - # - def self.antimatch(matcher) - self.all.antimatch(matcher) - end - - # - # Will return all instances of this class returning true when yielded to +block+. - # - # Will return all instances if no +block+ is given. - # - def self.select(&block) - if block - self.select.select do |record| - yield(record) - end - else - self.all || MatchSet.new - end - end - - # - # Will return all instances of this class except those returning true when yielded to +block+. - # - # Will return no instances if no +block+ is given. - # - def self.reject(&block) - if block - self.select.reject do |record| - yield(record) - end - else - MatchSet.new - end - end - - # # Return the record with +record_id+. # def self.find(record_id) - @@pirate[record_id] + CAPTAIN[record_id] end # - # Will delete this instance from the database and - # from the set of known instances. - # - def self.delete(instance) - @@pirate[self.all_key].delete(instance) - @@pirate.delete(instance.record_id) - end - - # # Will execute +block+ within a transaction. # def self.transaction(&block) - @@pirate.transaction(&block) + CAPTAIN.transaction(&block) end # - # Will return all known instances of this class. - # - def self.all - @@pirate[self.all_key] - end - - # # Use this method to get new instances of this class, since it will actually # make sure it both resides in the database and return a proxy to the remote # object. @@ -158,13 +67,20 @@ @record_id = Digest::SHA1.hexdigest("#{host}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") end - @@pirate.evaluate!(File.expand_path(__FILE__)) - @@pirate[instance.record_id] = instance + @@pre_create_hooks.each do |hook| + return false if hook.call(instance) == false + end + CAPTAIN[instance.record_id] = instance + @@post_create_hooks.each do |hook| + begin + hook.call(instance) + rescue + # /moo + end + end - proxy = @@pirate[instance.record_id] + proxy = CAPTAIN[instance.record_id] - @@pirate[self.all_key] ||= MatchSet.new - @@pirate[self.all_key] << proxy return proxy end @@ -174,10 +90,29 @@ attr_reader :record_id # + # Remove this instance from the database calling all the right hooks. # + # Freezes this instance after having deleted it. # + # Returns false without destroying anything if any of the @@pre_destroy_hooks + # returns false. + # + # Returns true otherwise. + # def destroy - self.class.delete(self) + @@pre_destroy_hooks.each do |hook| + return false if hook.call(self) == false + end + CAPTAIN.delete(@record_id) + self.freeze + @@post_destroy_hooks.each do |hook| + begin + hook.call(self) + rescue + # /moo + end + end + return true end private @@ -190,5 +125,4 @@ end end - end Added: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb (rev 0) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-19 18:55:17 UTC (rev 30) @@ -0,0 +1,150 @@ + +require 'rubygems' +require 'archipelago' +require 'rbtree' + +module Hyperactive + + class Tree < Record + + attr_accessor :elements, :subtrees + + WIDTH = 1 << 4 + + # + # Dont call this! Call Tree.get_instance(options) instead! + # + def initialize(options = {}) + @width = options[:width] || WIDTH + @elements = RBTree.new + @subtrees = nil + end + + def size + if @elements + @elements.size + else + @subtrees.t_collect do |key, tree| + tree.size + end.inject(0) do |sum, size| + sum + size + end + end + end + + def select(callable) + if @elements + @elements.values.select do |e| + callable.call(e) + end + else + @subsets.t_collect do |key, tree| + tree.match(callable) + end.inject([]) do |sum, match| + sum + match + end + end + end + + def reject(callable) + if @elements + @elements.values.reject do |e| + callable.call(e) + end + else + @subsets.t_collect do |key, tree| + tree.antimatch(callable) + end.inject([]) do |sum, match| + sum + match + end + end + end + + def []=(key, value) + if @elements + if @elements.size < @width + @elements[key] = value + else + split! + self[key] = value + end + else + @subtrees.each do |this_key, tree| + if this_key > key + tree[key] = value + return value + end + end + @subtrees[@subtrees.keys.first][key] = value + return value + end + end + + def [](key) + if @elements + return @elements[key] + else + @subtrees.each do |this_key, tree| + if this_key > key + return tree[key] + end + end + return @subtrees[@subtrees.keys.first][key] + end + end + + def collect(callable) + rval = [] + self.each do |e| + rval << callable.call(e) + end + return rval + end + + def each(callable) + if @elements + @elements.each do |key, value| + callable.call(key, value) + end + else + @subtrees.t_each do |key, tree| + tree.each(callable) + end + nil + end + end + + def clear + if @elements + @elements.each do |key, value| + value.destroy if value.respond_to?(:destroy) + end + @elements = RBTree.new + else + @subtrees.each do |key, tree| + tree.clear + tree.destroy + end + @elements = RBTree.new + @subtrees = nil + end + end + + private + + def split! + raise "Cant split twice!" unless @elements + + @subtrees = RBTree.new + @subtrees.extend(Archipelago::Current::ThreadedCollection) + @elements.each do |key, value| + new_tree = Tree.get_instance(:width => @width) + new_tree[key] = value + @subtrees[key] = new_tree + end + @elements = nil + end + + end + +end Modified: trunk/hyperactive/lib/hyperactive.rb =================================================================== --- trunk/hyperactive/lib/hyperactive.rb 2006-11-19 11:20:37 UTC (rev 29) +++ trunk/hyperactive/lib/hyperactive.rb 2006-11-19 18:55:17 UTC (rev 30) @@ -2,3 +2,4 @@ $: << File.dirname(__FILE__) require 'hyperactive/record' +require 'hyperactive/tree' Added: trunk/hyperactive/test/tree_test.rb =================================================================== --- trunk/hyperactive/test/tree_test.rb (rev 0) +++ trunk/hyperactive/test/tree_test.rb 2006-11-19 18:55:17 UTC (rev 30) @@ -0,0 +1,10 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class TreeTest < Test::Unit::TestCase + + def test_true + true + end + +end From nobody at rubyforge.org Sun Nov 19 17:57:57 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sun, 19 Nov 2006 17:57:57 -0500 (EST) Subject: [Archipelago-submits] [31] trunk/hyperactive: blargh Message-ID: <20061119225757.F24C65240EAA@rubyforge.org> Revision: 31 Author: zond Date: 2006-11-19 17:57:57 -0500 (Sun, 19 Nov 2006) Log Message: ----------- blargh Added Paths: ----------- trunk/hyperactive/tests/ trunk/hyperactive/tests/tree_test.rb Removed Paths: ------------- trunk/hyperactive/test/ Copied: trunk/hyperactive/tests (from rev 23, trunk/hyperactive/test) Copied: trunk/hyperactive/tests/tree_test.rb (from rev 30, trunk/hyperactive/test/tree_test.rb) =================================================================== --- trunk/hyperactive/tests/tree_test.rb (rev 0) +++ trunk/hyperactive/tests/tree_test.rb 2006-11-19 22:57:57 UTC (rev 31) @@ -0,0 +1,20 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class TreeTest < Test::Unit::TestCase + + def setup + DRb.start_service + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @tm = TestManager.new + end + + def teardown + end + + def test_true + true + end + +end From nobody at rubyforge.org Sun Nov 19 18:36:44 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sun, 19 Nov 2006 18:36:44 -0500 (EST) Subject: [Archipelago-submits] [32] trunk/archipelago/tests/treasure_test.rb: better tests Message-ID: <20061119233644.6C2C15240F6C@rubyforge.org> Revision: 32 Author: zond Date: 2006-11-19 18:36:43 -0500 (Sun, 19 Nov 2006) Log Message: ----------- better tests Modified Paths: -------------- trunk/archipelago/tests/treasure_test.rb Modified: trunk/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-19 22:57:57 UTC (rev 31) +++ trunk/archipelago/tests/treasure_test.rb 2006-11-19 23:36:43 UTC (rev 32) @@ -7,7 +7,7 @@ DRb.start_service @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) - @tm = TestManager.new + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) end def teardown From nobody at rubyforge.org Sun Nov 19 18:52:43 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sun, 19 Nov 2006 18:52:43 -0500 (EST) Subject: [Archipelago-submits] [33] trunk: added some broken tests to Tree Message-ID: <20061119235243.DF6645240F80@rubyforge.org> Revision: 33 Author: zond Date: 2006-11-19 18:52:42 -0500 (Sun, 19 Nov 2006) Log Message: ----------- added some broken tests to Tree Modified Paths: -------------- trunk/archipelago/tests/test_helper.rb trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/tests/test_helper.rb trunk/hyperactive/tests/tree_test.rb Modified: trunk/archipelago/tests/test_helper.rb =================================================================== --- trunk/archipelago/tests/test_helper.rb 2006-11-19 23:36:43 UTC (rev 32) +++ trunk/archipelago/tests/test_helper.rb 2006-11-19 23:52:42 UTC (rev 33) @@ -2,7 +2,7 @@ home = File.expand_path(File.dirname(__FILE__)) $: << File.join(home, "..", "lib") -MC_DISABLED = true +MC_DISABLED = true unless defined?(MC_ENABLED) && MC_ENABLED require 'pp' require 'drb' Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-19 23:36:43 UTC (rev 32) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-19 23:52:42 UTC (rev 33) @@ -89,6 +89,18 @@ # attr_reader :record_id + def hash + @record_id.hash + end + + def eql?(o) + if Record === o + o.record_id == @record_id + else + false + end + end + # # Remove this instance from the database calling all the right hooks. # Modified: trunk/hyperactive/tests/test_helper.rb =================================================================== --- trunk/hyperactive/tests/test_helper.rb 2006-11-19 23:36:43 UTC (rev 32) +++ trunk/hyperactive/tests/test_helper.rb 2006-11-19 23:52:42 UTC (rev 33) @@ -2,33 +2,12 @@ home = File.expand_path(File.dirname(__FILE__)) $: << File.join(home, "..", "lib") +require 'hyperactive' + +DRb.start_service +MC_ENABLED = true + require 'pp' require 'test/unit' require 'benchmark' - -class Test::Unit::TestCase - - def bm(label, options = {}) - n = options[:n] || 1000 - width = options[:width] || 50 - Benchmark.benchmark(" " * width + Benchmark::Tms::CAPTION, width, Benchmark::Tms::FMTSTR, "ms/call") do |b| - times = b.report("#{n}x#{label}") do - n.times do - yield - end - end - [times * 1000 / n.to_f] - end - end - - def assert_within(timeout, &block) - t = Time.new - rval = yield - while !rval && t > Time.new - timeout - rval = yield - sleep(0.05) - end - assert(rval) - end - -end +require '../archipelago/tests/test_helper' Modified: trunk/hyperactive/tests/tree_test.rb =================================================================== --- trunk/hyperactive/tests/tree_test.rb 2006-11-19 23:36:43 UTC (rev 32) +++ trunk/hyperactive/tests/tree_test.rb 2006-11-19 23:52:42 UTC (rev 33) @@ -4,17 +4,41 @@ class TreeTest < Test::Unit::TestCase def setup - DRb.start_service @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) - @tm = TestManager.new + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + assert_within(10) do + !Hyperactive::CAPTAIN.chests.empty? + end + assert_within(10) do + !Hyperactive::CAPTAIN.trannies.empty? + end end def teardown + @c.persistence_provider.unlink + @c2.persistence_provider.unlink + @tm.persistence_provider.unlink end - def test_true - true + def test_set_get + h = Hyperactive::Tree.get_instance + h2 = {} + 50.times do + r = Hyperactive::Record.get_instance + h[r.record_id] = r + h2[r.record_id] = r + end + + h2.each do |k,v| + assert_equal(v, h[k]) + assert_equal(v, Hyperactive::CAPTAIN[h.record_id][k]) + end end end From nobody at rubyforge.org Mon Nov 20 08:33:21 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 20 Nov 2006 08:33:21 -0500 (EST) Subject: [Archipelago-submits] [34] trunk/hyperactive/lib/hyperactive: made tree pass its tests. Message-ID: <20061120133321.8063552417F0@rubyforge.org> Revision: 34 Author: zond Date: 2006-11-20 08:33:20 -0500 (Mon, 20 Nov 2006) Log Message: ----------- made tree pass its tests. made tree better in balancing itself. Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive/tree.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-19 23:52:42 UTC (rev 33) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-20 13:33:20 UTC (rev 34) @@ -94,7 +94,11 @@ end def eql?(o) - if Record === o + self.==(o) + end + + def ==(o) + if o.is_a?(Record) o.record_id == @record_id else false Modified: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-19 23:52:42 UTC (rev 33) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-20 13:33:20 UTC (rev 34) @@ -16,7 +16,7 @@ # def initialize(options = {}) @width = options[:width] || WIDTH - @elements = RBTree.new + @elements = {} @subtrees = nil end @@ -24,7 +24,7 @@ if @elements @elements.size else - @subtrees.t_collect do |key, tree| + @subtrees.t_collect do |tree| tree.size end.inject(0) do |sum, size| sum + size @@ -34,12 +34,12 @@ def select(callable) if @elements - @elements.values.select do |e| - callable.call(e) + @elements.select do |k,v| + callable.call(k,v) end else - @subsets.t_collect do |key, tree| - tree.match(callable) + @subsets.t_collect do |tree| + tree.select(callable) end.inject([]) do |sum, match| sum + match end @@ -48,12 +48,12 @@ def reject(callable) if @elements - @elements.values.reject do |e| - callable.call(e) + @elements.reject do |k,v| + callable.call(k,v) end else - @subsets.t_collect do |key, tree| - tree.antimatch(callable) + @subsets.t_collect do |tree| + tree.reject(callable) end.inject([]) do |sum, match| sum + match end @@ -69,27 +69,16 @@ self[key] = value end else - @subtrees.each do |this_key, tree| - if this_key > key - tree[key] = value - return value - end - end - @subtrees[@subtrees.keys.first][key] = value - return value + insert_into_subtree(key, value) end + return value end def [](key) if @elements return @elements[key] else - @subtrees.each do |this_key, tree| - if this_key > key - return tree[key] - end - end - return @subtrees[@subtrees.keys.first][key] + return @subtrees[key.hash % @width][key] end end @@ -107,7 +96,7 @@ callable.call(key, value) end else - @subtrees.t_each do |key, tree| + @subtrees.t_each do |tree| tree.each(callable) end nil @@ -121,26 +110,31 @@ end @elements = RBTree.new else - @subtrees.each do |key, tree| + @subtrees.each do |tree| tree.clear tree.destroy end - @elements = RBTree.new + @elements = {} @subtrees = nil end end private + def insert_into_subtree(key, value) + @subtrees[key.hash % @width][key] = value + end + def split! raise "Cant split twice!" unless @elements - @subtrees = RBTree.new + @subtrees = [] @subtrees.extend(Archipelago::Current::ThreadedCollection) + 0.upto(@width - 1) do + @subtrees << Tree.get_instance(:width => @width) + end @elements.each do |key, value| - new_tree = Tree.get_instance(:width => @width) - new_tree[key] = value - @subtrees[key] = new_tree + insert_into_subtree(key, value) end @elements = nil end From nobody at rubyforge.org Mon Nov 20 08:57:27 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 20 Nov 2006 08:57:27 -0500 (EST) Subject: [Archipelago-submits] [35] trunk/archipelago/lib/archipelago/treasure.rb: optimized certain calls on dubloons Message-ID: <20061120135727.AA02752417DE@rubyforge.org> Revision: 35 Author: zond Date: 2006-11-20 08:57:26 -0500 (Mon, 20 Nov 2006) Log Message: ----------- optimized certain calls on dubloons Modified Paths: -------------- trunk/archipelago/lib/archipelago/treasure.rb Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-20 13:33:20 UTC (rev 34) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-20 13:57:26 UTC (rev 35) @@ -98,6 +98,30 @@ undef_method method unless method =~ /^__/ end # + # This Dubloon will always have the same hash, based on + # object_id. + # + def hash + self.object_id.hash + end + # + # Defers to Dubloon#==. + # + def eql?(o) + self.==(o) + end + # + # If o is a Dubloon, will return true if it has the same Dubloon#object_id. + # + # Otherwise will defer to Dubloon#method_missing. + # + def ==(o) + if Dubloon === o + return true if self.object_id == o.object_id + end + return self.method_missing(:==, o) + end + # # Initialize us with knowledge of our +chest+, the +key+ to our # target in the +chest+, the known +public_methods+ of our target # and any +transaction+ we are associated with. @@ -161,7 +185,8 @@ raise UnknownTransactionException.new(self, transaction) unless transaction == @transaction end # - # The object_id of our chest-held target. + # This Dubloon will always have the same object_id, based on + # @chest_id and @key and possibly @transaction. # def object_id id = "#{@chest_id}:#{@key}" From nobody at rubyforge.org Mon Nov 20 09:04:05 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 20 Nov 2006 09:04:05 -0500 (EST) Subject: [Archipelago-submits] [36] trunk/hyperactive/lib/hyperactive/record.rb: made record use dubloon for base methods Message-ID: <20061120140405.8D88A52417FE@rubyforge.org> Revision: 36 Author: zond Date: 2006-11-20 09:04:05 -0500 (Mon, 20 Nov 2006) Log Message: ----------- made record use dubloon for base methods Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-20 13:57:26 UTC (rev 35) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-20 14:04:05 UTC (rev 36) @@ -89,22 +89,6 @@ # attr_reader :record_id - def hash - @record_id.hash - end - - def eql?(o) - self.==(o) - end - - def ==(o) - if o.is_a?(Record) - o.record_id == @record_id - else - false - end - end - # # Remove this instance from the database calling all the right hooks. # From nobody at rubyforge.org Mon Nov 20 09:20:58 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 20 Nov 2006 09:20:58 -0500 (EST) Subject: [Archipelago-submits] [37] trunk/archipelago/lib/archipelago/disco.rb: made host lookups happen more seldom Message-ID: <20061120142058.D26E452417F2@rubyforge.org> Revision: 37 Author: zond Date: 2006-11-20 09:20:56 -0500 (Mon, 20 Nov 2006) Log Message: ----------- made host lookups happen more seldom Modified Paths: -------------- trunk/archipelago/lib/archipelago/disco.rb Modified: trunk/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-20 14:04:05 UTC (rev 36) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-20 14:20:56 UTC (rev 37) @@ -67,6 +67,11 @@ THRIFTY_PUBLISHING = false # + # The host we are running on. + # + HOST = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" + + # # A module to simplify publishing services. # # If you include it you can use the publish! method @@ -155,8 +160,7 @@ @metadata ||= @persistence_provider.get_hashish("metadata") service_id = @metadata["service_id"] unless service_id - host = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" - service_id = @metadata["service_id"] ||= Digest::SHA1.hexdigest("#{host}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") + service_id = @metadata["service_id"] ||= Digest::SHA1.hexdigest("#{HOST}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") end return service_id end @@ -384,7 +388,7 @@ raise e end end - @unicast_address = "#{Socket::gethostbyname(Socket::gethostname)[0]}:#{this_port}" rescue "localhost:#{this_port}" + @unicast_address = "#{HOST}:#{this_port}" @sender = UDPSocket.new @sender.connect(options[:address] || ADDRESS, options[:port] || PORT) From nobody at rubyforge.org Tue Nov 21 04:26:10 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 21 Nov 2006 04:26:10 -0500 (EST) Subject: [Archipelago-submits] [38] trunk/hyperactive: added benchmark for tree. Message-ID: <20061121092610.36E965240A6E@rubyforge.org> Revision: 38 Author: zond Date: 2006-11-21 04:26:07 -0500 (Tue, 21 Nov 2006) Log Message: ----------- added benchmark for tree. fixed bugs in tree. Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive/tree.rb trunk/hyperactive/tests/test_helper.rb trunk/hyperactive/tests/tree_test.rb Added Paths: ----------- trunk/hyperactive/tests/tree_benchmark.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-20 14:20:56 UTC (rev 37) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-21 09:26:07 UTC (rev 38) @@ -34,6 +34,11 @@ @@post_destroy_hooks = [] # + # The host we are running on. + # + HOST = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" + + # # Call this if you want to change the default database connector # to something else. # @@ -63,8 +68,7 @@ def self.get_instance(*arguments) instance = self.new(*arguments) instance.instance_eval do - host = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" - @record_id = Digest::SHA1.hexdigest("#{host}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") + @record_id = Digest::SHA1.hexdigest("#{HOST}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") end @@pre_create_hooks.each do |hook| Modified: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-20 14:20:56 UTC (rev 37) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-21 09:26:07 UTC (rev 38) @@ -20,11 +20,19 @@ @subtrees = nil end + def delete(key) + if @elements + @elements.delete(key) + else + subtree_for(key).delete(key) + end + end + def size if @elements @elements.size else - @subtrees.t_collect do |tree| + @subtrees.t_collect do |key, tree| tree.size end.inject(0) do |sum, size| sum + size @@ -69,7 +77,7 @@ self[key] = value end else - insert_into_subtree(key, value) + subtree_for(key)[key] = value end return value end @@ -78,7 +86,7 @@ if @elements return @elements[key] else - return @subtrees[key.hash % @width][key] + return subtree_for(key)[key] end end @@ -96,7 +104,7 @@ callable.call(key, value) end else - @subtrees.t_each do |tree| + @subtrees.t_each do |key, tree| tree.each(callable) end nil @@ -110,7 +118,7 @@ end @elements = RBTree.new else - @subtrees.each do |tree| + @subtrees.each do |key, tree| tree.clear tree.destroy end @@ -121,20 +129,24 @@ private - def insert_into_subtree(key, value) - @subtrees[key.hash % @width][key] = value + def subtree_for(key) + key_id = Digest::SHA1.new("#{key.hash}").to_s + @subtrees.each do |tree_id, tree| + return tree if tree_id > key_id + end + return @subtrees.values.first end - + def split! raise "Cant split twice!" unless @elements - @subtrees = [] + @subtrees = RBTree.new @subtrees.extend(Archipelago::Current::ThreadedCollection) 0.upto(@width - 1) do - @subtrees << Tree.get_instance(:width => @width) + @subtrees[Digest::SHA1.new("#{rand(1 << 32)}").to_s] = Tree.get_instance(:width => @width) end @elements.each do |key, value| - insert_into_subtree(key, value) + subtree_for(key)[key] = value end @elements = nil end Modified: trunk/hyperactive/tests/test_helper.rb =================================================================== --- trunk/hyperactive/tests/test_helper.rb 2006-11-20 14:20:56 UTC (rev 37) +++ trunk/hyperactive/tests/test_helper.rb 2006-11-21 09:26:07 UTC (rev 38) @@ -1,8 +1,11 @@ +SCRIPT_LINES__ = {} + home = File.expand_path(File.dirname(__FILE__)) $: << File.join(home, "..", "lib") require 'hyperactive' +require 'ruby-debug' DRb.start_service MC_ENABLED = true Added: trunk/hyperactive/tests/tree_benchmark.rb =================================================================== --- trunk/hyperactive/tests/tree_benchmark.rb (rev 0) +++ trunk/hyperactive/tests/tree_benchmark.rb 2006-11-21 09:26:07 UTC (rev 38) @@ -0,0 +1,50 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class TreeBenchmark < Test::Unit::TestCase + + def setup + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + assert_within(10) do + !Hyperactive::CAPTAIN.chests.empty? + end + assert_within(10) do + !Hyperactive::CAPTAIN.trannies.empty? + end + end + + def teardown + @c.persistence_provider.unlink + @c2.persistence_provider.unlink + @tm.persistence_provider.unlink + end + + def test_set_get + h = Hyperactive::Tree.get_instance + r = Hyperactive::Record.get_instance + bm("Tree#set/get/delete", :n => 100) do + f = rand(1 << 32) + h[f] = r + x = h[f] + h.delete(f) + end + bm("Tree#set", :n => 1000) do + f = rand(1 << 32) + h[f] = r + end + bm("Tree#set/get/delete (big)", :n => 100) do + f = rand(1 << 32) + h[f] = r + x = h[f] + h.delete(f) + end + end + +end Modified: trunk/hyperactive/tests/tree_test.rb =================================================================== --- trunk/hyperactive/tests/tree_test.rb 2006-11-20 14:20:56 UTC (rev 37) +++ trunk/hyperactive/tests/tree_test.rb 2006-11-21 09:26:07 UTC (rev 38) @@ -29,7 +29,7 @@ def test_set_get h = Hyperactive::Tree.get_instance h2 = {} - 50.times do + 10.times do r = Hyperactive::Record.get_instance h[r.record_id] = r h2[r.record_id] = r From nobody at rubyforge.org Tue Nov 21 04:34:50 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 21 Nov 2006 04:34:50 -0500 (EST) Subject: [Archipelago-submits] [39] trunk/hyperactive/lib/hyperactive/tree.rb: more bugfixes in tree Message-ID: <20061121093450.B8F1B5240A7E@rubyforge.org> Revision: 39 Author: zond Date: 2006-11-21 04:34:50 -0500 (Tue, 21 Nov 2006) Log Message: ----------- more bugfixes in tree Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/tree.rb Modified: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-21 09:26:07 UTC (rev 38) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-21 09:34:50 UTC (rev 39) @@ -32,7 +32,7 @@ if @elements @elements.size else - @subtrees.t_collect do |key, tree| + @subtrees.t_collect do |tree_id, tree| tree.size end.inject(0) do |sum, size| sum + size @@ -46,7 +46,7 @@ callable.call(k,v) end else - @subsets.t_collect do |tree| + @subtrees.t_collect do |tree_id, tree| tree.select(callable) end.inject([]) do |sum, match| sum + match @@ -60,7 +60,7 @@ callable.call(k,v) end else - @subsets.t_collect do |tree| + @subtrees.t_collect do |tree_id, tree| tree.reject(callable) end.inject([]) do |sum, match| sum + match @@ -74,7 +74,7 @@ @elements[key] = value else split! - self[key] = value + subtree_for(key)[key] = value end else subtree_for(key)[key] = value @@ -104,31 +104,52 @@ callable.call(key, value) end else - @subtrees.t_each do |key, tree| + @subtrees.t_each do |tree_id, tree| tree.each(callable) end nil end end - def clear + # + # Clear everything from this Tree and destroy it. + # + def destroy if @elements @elements.each do |key, value| value.destroy if value.respond_to?(:destroy) end - @elements = RBTree.new else - @subtrees.each do |key, tree| - tree.clear + @subtrees.each do |tree_id, tree| tree.destroy end - @elements = {} + end + freeze + super + end + + # + # Clear everything from this Tree. + # + def clear + unless @elements + @subtrees.each do |tree_id, tree| + tree.clear + end @subtrees = nil end + @elements = {} end private + # + # Finds the subtree responsible for +key+. + # + # Does it in this ugly way cause the nice way of just doing modulo gave + # really odd results since all hashes seem to give some modulo values + # a lot more often than expected. + # def subtree_for(key) key_id = Digest::SHA1.new("#{key.hash}").to_s @subtrees.each do |tree_id, tree| @@ -136,7 +157,11 @@ end return @subtrees.values.first end - + + # + # Split this Tree by creating @subtrees, + # then putting all @elements in them and then emptying @elements. + # def split! raise "Cant split twice!" unless @elements From nobody at rubyforge.org Thu Nov 23 07:57:54 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Thu, 23 Nov 2006 07:57:54 -0500 (EST) Subject: [Archipelago-submits] [40] trunk/archipelago/lib/archipelago/treasure.rb: optimized away the cached public_methods of dubloon proxy targets Message-ID: <20061123125754.9AE525240D70@rubyforge.org> Revision: 40 Author: zond Date: 2006-11-23 07:57:53 -0500 (Thu, 23 Nov 2006) Log Message: ----------- optimized away the cached public_methods of dubloon proxy targets Modified Paths: -------------- trunk/archipelago/lib/archipelago/treasure.rb Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-21 09:34:50 UTC (rev 39) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-23 12:57:53 UTC (rev 40) @@ -95,7 +95,7 @@ # Remove all methods so that we look like our target. # instance_methods.each do |method| - undef_method method unless method =~ /^__/ + undef_method method unless (method =~ /^__/ || method == "respond_to?") end # # This Dubloon will always have the same hash, based on @@ -126,12 +126,11 @@ # target in the +chest+, the known +public_methods+ of our target # and any +transaction+ we are associated with. # - def initialize(key, chest, transaction, chest_id, public_methods) + def initialize(key, chest, transaction, chest_id) @key = key @chest = chest @transaction = transaction @chest_id = chest_id - @public_methods = public_methods end # # A more or less normal dump of all our instance variables. @@ -141,8 +140,7 @@ @key, @chest, @transaction, - @chest_id, - @public_methods + @chest_id ]) end # @@ -150,12 +148,11 @@ # it actually is not correct. # def self._load(s) - key, chest, transaction, chest_id, public_methods = Marshal.load(s) + key, chest, transaction, chest_id = Marshal.load(s) instance = self.allocate instance.instance_variable_set(:@key, key) instance.instance_variable_set(:@transaction, transaction) instance.instance_variable_set(:@chest_id, chest_id) - instance.instance_variable_set(:@public_methods, public_methods) if CHEST_BY_SERVICE_ID.include?(chest_id) instance.instance_variable_set(:@chest, CHEST_BY_SERVICE_ID[chest_id]) else @@ -164,18 +161,12 @@ return instance end # - # The public_methods of our target. - # - def public_methods - return @public_methods.clone - end - # # Return a clone of myself that is joined to # the +transaction+. # def join(transaction) @chest.join!(transaction) if transaction - return Dubloon.new(@key, @chest, transaction, @chest_id, @public_methods) + return Dubloon.new(@key, @chest, transaction, @chest_id) end # # Raises exception if the given +transaction+ @@ -196,32 +187,29 @@ # # Does our target respond to +meth+? # + alias :dubloon_redirected_respond_to? :respond_to? def respond_to?(meth) - return @public_methods.include?(meth.to_sym) || @public_methods.include?(meth.to_s) + return dubloon_redirected_respond_to?(meth) || self.method_missing(:respond_to?, meth) end # # Call +meth+ with +args+ and +block+ on our target if it responds to # it. # def method_missing(meth, *args, &block) - if respond_to?(meth) - begin - return @chest.call_instance_method(@key, meth, @transaction, *args, &block) - rescue DRb::DRbConnError => e - if defined?(Archipelago::Disco::MC) - possible_replacements = Archipelago::Disco::MC.lookup(Archipelago::Disco::Query.new({:service_id => @chest_id})) - raise e if possible_replacements.empty? - - @chest = possible_replacements[@chest_id][:service] - CHESTS_BY_SERVICE_ID[@chest_id] = @chest - - retry - else - raise e - end + begin + return @chest.call_instance_method(@key, meth, @transaction, *args, &block) + rescue DRb::DRbConnError => e + if defined?(Archipelago::Disco::MC) + possible_replacements = Archipelago::Disco::MC.lookup(Archipelago::Disco::Query.new({:service_id => @chest_id})) + raise e if possible_replacements.empty? + + @chest = possible_replacements[@chest_id][:service] + CHESTS_BY_SERVICE_ID[@chest_id] = @chest + + retry + else + raise e end - else - return super(meth, *args) end end @@ -341,7 +329,7 @@ if Dubloon === instance return instance.join(transaction) else - return Dubloon.new(key, self, transaction, self.service_id, instance.public_methods) + return Dubloon.new(key, self, transaction, self.service_id) end end @@ -746,7 +734,7 @@ return value if Dubloon === value - return Dubloon.new(key, self, transaction, service_id, value.public_methods) + return Dubloon.new(key, self, transaction, service_id) end # From nobody at rubyforge.org Thu Nov 23 08:47:49 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Thu, 23 Nov 2006 08:47:49 -0500 (EST) Subject: [Archipelago-submits] [41] trunk/hyperactive: optimized tree a lot. Message-ID: <20061123134749.2346F5240E5E@rubyforge.org> Revision: 41 Author: zond Date: 2006-11-23 08:47:48 -0500 (Thu, 23 Nov 2006) Log Message: ----------- optimized tree a lot. documented stuff Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive/tree.rb trunk/hyperactive/tests/tree_benchmark.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-23 12:57:53 UTC (rev 40) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-23 13:47:48 UTC (rev 41) @@ -119,14 +119,5 @@ return true end - private - - # - # The key used to store the MatchSet containing proxies to all our instances. - # - def self.all_key - "HyperactiveRecord:#{self.name}:all" - end - end end Modified: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-23 12:57:53 UTC (rev 40) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-23 13:47:48 UTC (rev 41) @@ -5,11 +5,21 @@ module Hyperactive + # + # A class suitable for storing large and often-changing datasets in + # an Archipelago environment. + # + # Is constructed like a set of nested Hashes that automatically create + # new children on demand, and will thusly only have to check the path from + # the root node to the leaf for changes when method calls return (see Archipelago::Treasure::Chest) + # and will only have to actually store into database the leaf itself if it + # has changed. + # class Tree < Record attr_accessor :elements, :subtrees - WIDTH = 1 << 4 + WIDTH = 1 << 2 # # Dont call this! Call Tree.get_instance(options) instead! @@ -19,7 +29,32 @@ @elements = {} @subtrees = nil end + + # + # Returns how balanced this tree is + # + def balance + return 1 if @elements + + smallest = nil + largest = nil + avg = 0 + count = 0 + @subtrees.t_each do |tree_id, tree| + size = tree.size + avg += size + count += 1 + smallest = size if smallest.nil? || smallest > size + largest = size if largest.nil? || largest < size + end + avg /= count.to_f + return ((smallest.to_f / avg) + (avg / largest.to_f)) / 2.0 + end + + # + # Deletes +key+ in this Tree. + # def delete(key) if @elements @elements.delete(key) @@ -28,6 +63,9 @@ end end + # + # Returns the size of this Tree. + # def size if @elements @elements.size @@ -40,6 +78,9 @@ end end + # + # Returns all keys and values returning true for +callable+.call(key, value) in this Tree. + # def select(callable) if @elements @elements.select do |k,v| @@ -54,6 +95,9 @@ end end + # + # Returns all keys and values returning false for +callable+.call(key, value) in this Tree. + # def reject(callable) if @elements @elements.reject do |k,v| @@ -68,6 +112,9 @@ end end + # + # Puts +value+ under +key+ in this Tree. + # def []=(key, value) if @elements if @elements.size < @width @@ -82,6 +129,9 @@ return value end + # + # Returns the value for +key+ in this Tree. + # def [](key) if @elements return @elements[key] @@ -90,6 +140,9 @@ end end + # + # Returns an Array containing +callable+.call(key, value) from all values in this Tree. + # def collect(callable) rval = [] self.each do |e| @@ -98,6 +151,9 @@ return rval end + # + # Does +callable+.call(key, value) on all values in this Tree. + # def each(callable) if @elements @elements.each do |key, value| @@ -151,7 +207,7 @@ # a lot more often than expected. # def subtree_for(key) - key_id = Digest::SHA1.new("#{key.hash}").to_s + key_id = Digest::SHA1.new("#{key.hash}#{self.record_id}").to_s @subtrees.each do |tree_id, tree| return tree if tree_id > key_id end @@ -167,8 +223,9 @@ @subtrees = RBTree.new @subtrees.extend(Archipelago::Current::ThreadedCollection) - 0.upto(@width - 1) do - @subtrees[Digest::SHA1.new("#{rand(1 << 32)}").to_s] = Tree.get_instance(:width => @width) + step = (1 << 160) / @width + 0.upto(@width - 1) do |n| + @subtrees["%x" % (n * step)] = Tree.get_instance(:width => @width) end @elements.each do |key, value| subtree_for(key)[key] = value Modified: trunk/hyperactive/tests/tree_benchmark.rb =================================================================== --- trunk/hyperactive/tests/tree_benchmark.rb 2006-11-23 12:57:53 UTC (rev 40) +++ trunk/hyperactive/tests/tree_benchmark.rb 2006-11-23 13:47:48 UTC (rev 41) @@ -29,17 +29,32 @@ def test_set_get h = Hyperactive::Tree.get_instance r = Hyperactive::Record.get_instance - bm("Tree#set/get/delete", :n => 100) do + hash_test("Tree", h, r, 100) + end + + def test_regular_hash_set_get + Hyperactive::CAPTAIN["h"] = {} + h = Hyperactive::CAPTAIN["h"] + r = Hyperactive::Record.get_instance + hash_test("Hash", h, r, 100) + end + + private + + def hash_test(label, h, r, c) + bm("#{label}#set/get/delete", :n => c * 1) do f = rand(1 << 32) h[f] = r x = h[f] h.delete(f) end - bm("Tree#set", :n => 1000) do + bm("#{label}#set", :n => c * 10) do f = rand(1 << 32) h[f] = r end - bm("Tree#set/get/delete (big)", :n => 100) do + puts "balance: #{h.balance}" if h.respond_to?(:balance) + puts "sizes: #{h.subtrees.collect do |k,t| t.size end.inspect}" if h.respond_to?(:subtrees) + bm("#{label}#set/get/delete (big)", :n => c * 1) do f = rand(1 << 32) h[f] = r x = h[f] From nobody at rubyforge.org Fri Nov 24 06:44:14 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Fri, 24 Nov 2006 06:44:14 -0500 (EST) Subject: [Archipelago-submits] [42] trunk/archipelago: added support for #around_save(&block) for any value stored in BerkeleyHashish. Message-ID: <20061124114414.65AC85240B5F@rubyforge.org> Revision: 42 Author: zond Date: 2006-11-24 06:44:13 -0500 (Fri, 24 Nov 2006) Log Message: ----------- added support for #around_save(&block) for any value stored in BerkeleyHashish. added test for it. Modified Paths: -------------- trunk/archipelago/lib/archipelago/hashish.rb trunk/archipelago/tests/treasure_test.rb Modified: trunk/archipelago/lib/archipelago/hashish.rb =================================================================== --- trunk/archipelago/lib/archipelago/hashish.rb 2006-11-23 13:47:48 UTC (rev 41) +++ trunk/archipelago/lib/archipelago/hashish.rb 2006-11-24 11:44:13 UTC (rev 42) @@ -70,12 +70,16 @@ # # Insert +value+ under +key+. # + # Will call value.around_save and send + # it a block that does the actual saving if + # value.respond_to?(:around_save). + # def []=(key, value) @lock.synchronize_on(key) do @content[key] = value - write_to_db(key, Marshal.dump(key), Marshal.dump(value)) + write_to_db(key, Marshal.dump(key), Marshal.dump(value), value) return value @@ -85,13 +89,19 @@ # Stores whatever is under +key+ if it is not the same as # whats in the persistent db. # + # Will call value.around_save and send + # it a block that does the actual saving if + # value.respond_to?(:around_save) and + # a save is actually performed. + # def store_if_changed(key) @lock.synchronize_on(key) do serialized_key = Marshal.dump(key) - serialized_value = Marshal.dump(@content[key]) + value = @content[key] + serialized_value = Marshal.dump(value) - write_to_db(key, serialized_key, serialized_value) if @content_db[serialized_key] != serialized_value + write_to_db(key, serialized_key, serialized_value, value) if @content_db[serialized_key] != serialized_value end end @@ -136,7 +146,25 @@ # Write +key+, serialized as +serialized_key+ and # +serialized_value+ to the db. # - def write_to_db(key, serialized_key, serialized_value) + # Will call value.around_save and send + # it a block that does the actual saving if + # value.respond_to?(:around_save). + # + def write_to_db(key, serialized_key, serialized_value, value) + if value.respond_to?(:around_save) + value.around_save do + do_write_to_db(key, serialized_key, serialized_value) + end + else + do_write_to_db(key, serialized_key, serialized_value) + end + end + + # + # Actually writes +key+ serialized as +serialized_key+ an + # +serialized_value+ to the db. Used by write_to_db. + # + def do_write_to_db(key, serialized_key, serialized_value) now = Time.now @content_db[serialized_key] = serialized_value Modified: trunk/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-23 13:47:48 UTC (rev 41) +++ trunk/archipelago/tests/treasure_test.rb 2006-11-24 11:44:13 UTC (rev 42) @@ -1,6 +1,14 @@ require File.join(File.dirname(__FILE__), 'test_helper') +class A < String + def around_save(&block) + $BURKMAT += 1 + yield + $BURKMAT2 += 1 + end +end + class TreasureTest < Test::Unit::TestCase def setup @@ -8,6 +16,8 @@ @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + $BURKMAT = 0 + $BURKMAT2 = 0 end def teardown @@ -17,6 +27,16 @@ DRb.stop_service end + def test_around_save + s = A.new("hehu") + @c["oj"] = s + assert_equal(1, $BURKMAT) + assert_equal(1, $BURKMAT2) + @c["oj"].upcase! + assert_equal(2, $BURKMAT) + assert_equal(2, $BURKMAT2) + end + def test_eval t = Time.now @c.evaluate!("burk", t, "class Burk; end") From nobody at rubyforge.org Sat Nov 25 07:27:40 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sat, 25 Nov 2006 07:27:40 -0500 (EST) Subject: [Archipelago-submits] [43] trunk/archipelago: renamed around_save save_hook Message-ID: <20061125122740.D6ED65240CC0@rubyforge.org> Revision: 43 Author: zond Date: 2006-11-25 07:27:39 -0500 (Sat, 25 Nov 2006) Log Message: ----------- renamed around_save save_hook Modified Paths: -------------- trunk/archipelago/lib/archipelago/hashish.rb trunk/archipelago/tests/treasure_test.rb Modified: trunk/archipelago/lib/archipelago/hashish.rb =================================================================== --- trunk/archipelago/lib/archipelago/hashish.rb 2006-11-24 11:44:13 UTC (rev 42) +++ trunk/archipelago/lib/archipelago/hashish.rb 2006-11-25 12:27:39 UTC (rev 43) @@ -70,9 +70,9 @@ # # Insert +value+ under +key+. # - # Will call value.around_save and send + # Will call value.save_hook and send # it a block that does the actual saving if - # value.respond_to?(:around_save). + # value.respond_to?(:save_hook). # def []=(key, value) @lock.synchronize_on(key) do @@ -89,9 +89,9 @@ # Stores whatever is under +key+ if it is not the same as # whats in the persistent db. # - # Will call value.around_save and send + # Will call value.save_hook and send # it a block that does the actual saving if - # value.respond_to?(:around_save) and + # value.respond_to?(:save_hook) and # a save is actually performed. # def store_if_changed(key) @@ -146,13 +146,13 @@ # Write +key+, serialized as +serialized_key+ and # +serialized_value+ to the db. # - # Will call value.around_save and send + # Will call value.save_hook and send # it a block that does the actual saving if - # value.respond_to?(:around_save). + # value.respond_to?(:save_hook). # def write_to_db(key, serialized_key, serialized_value, value) - if value.respond_to?(:around_save) - value.around_save do + if value.respond_to?(:save_hook) + value.save_hook do do_write_to_db(key, serialized_key, serialized_value) end else Modified: trunk/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-24 11:44:13 UTC (rev 42) +++ trunk/archipelago/tests/treasure_test.rb 2006-11-25 12:27:39 UTC (rev 43) @@ -2,7 +2,7 @@ require File.join(File.dirname(__FILE__), 'test_helper') class A < String - def around_save(&block) + def save_hook(&block) $BURKMAT += 1 yield $BURKMAT2 += 1 From nobody at rubyforge.org Sat Nov 25 08:06:46 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sat, 25 Nov 2006 08:06:46 -0500 (EST) Subject: [Archipelago-submits] [44] trunk: added around hooks for creating, saving and destroying records Message-ID: <20061125130646.B5AA15240D0B@rubyforge.org> Revision: 44 Author: zond Date: 2006-11-25 08:06:45 -0500 (Sat, 25 Nov 2006) Log Message: ----------- added around hooks for creating, saving and destroying records Modified Paths: -------------- trunk/archipelago/lib/archipelago/treasure.rb trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive.rb Added Paths: ----------- trunk/hyperactive/lib/hyperactive/hooker.rb trunk/hyperactive/tests/record_test.rb Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-25 12:27:39 UTC (rev 43) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-25 13:06:45 UTC (rev 44) @@ -55,8 +55,12 @@ # Raised whenever a method is called on an object that we dont know about. # class UnknownObjectException < RuntimeError - def initialize(chest, key, transaction) - super("#{chest} does not contain #{key} under #{transaction}") + def initialize(chest, key, transaction = nil) + if transaction + super("#{chest} does not contain #{key} under #{transaction}") + else + super("#{chest} does not contain #{key}") + end end end @@ -636,7 +640,7 @@ def call_without_transaction(key, method, *arguments, &block) instance = @db[key] - raise UnknownObjectException(self, key, transaction) unless instance + raise UnknownObjectException(self, key) unless instance begin return execute(instance, method, *arguments, &block) Added: trunk/hyperactive/lib/hyperactive/hooker.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/hooker.rb (rev 0) +++ trunk/hyperactive/lib/hyperactive/hooker.rb 2006-11-25 13:06:45 UTC (rev 44) @@ -0,0 +1,63 @@ + +module Hyperactive + + # + # A utility method to simplify using around-hooks for any block. + # + module Hooker + + # + # Will call each hook in +hooks+ with a block that + # calls the rest of the +hooks+ with +object+ as argument + # and after that yields to the given +block+. + # + # This will allow you wrap any method call in a dynamic set + # of other methods. + # + # For more examples, see Hyperactive::Record. + # + # Example: + # class ImportantStuff + # attr_accessor :timestamp + # def is_valid? + # # do nifty stuff + # end + # def notify_someone_that_i_am_changed!(message) + # # do even more nifty stuff + # end + # def validate_and_notify(message) + # raise "i am invalid!" unless is_valid? + # yield + # notify_someone_that_i_am_changed!(message) + # end + # def do_save + # # save me in persistent storage + # end + # end + # class ImportantHandler + # def update_timestamp(important_stuff_instance) + # important_stuff_instance.timestamp = Time.now + # Hyperactive::Hooker.call_with_hooks("changed at #{Time.now}", + # important_stuff_instance.method(:validate_and_notify)) do + # important_stuff_instance.do_save + # end + # end + # end + # + def call_with_hooks(object, *hooks, &block) + if hooks.empty? + yield + else + first = hooks.first + rest = hooks[1..-1] + first.call(object) do + call_with_hooks(object, *rest, &block) + end + end + end + + module_function :call_with_hooks + + end + +end Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-25 12:27:39 UTC (rev 43) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-25 13:06:45 UTC (rev 44) @@ -28,10 +28,9 @@ # class Record - @@pre_create_hooks = [] - @@post_create_hooks = [] - @@pre_destroy_hooks = [] - @@post_destroy_hooks = [] + @@create_hooks_by_class = {} + @@destroy_hooks_by_class = {} + @@save_hooks_by_class = {} # # The host we are running on. @@ -47,6 +46,57 @@ end # + # Return our create_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent the instance about to be created (initial + # insertion into the database system) that take a block argument. + # + # The block argument will be a Proc that actually injects the + # instance into the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon creation. + # + def self.create_hooks + self.get_array_by_class(@@create_hooks_by_class) + end + + # + # Return our destroy_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent the instance about to be destroyed (removal + # from the database system) that take a block argument. + # + # The block argument will be a Proc that actually removes the + # instance from the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon destruction. + # + def self.destroy_hooks + self.get_array_by_class(@@destroy_hooks_by_class) + end + + # + # Return our save_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent the instance about to be saved (storage + # into the database system) that take a block argument. + # + # The block argument will be a Proc that actually saves the + # instance into the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon saving. + # + def self.save_hooks + self.get_array_by_class(@@save_hooks_by_class) + end + + # # Return the record with +record_id+. # def self.find(record_id) @@ -71,17 +121,9 @@ @record_id = Digest::SHA1.hexdigest("#{HOST}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") end - @@pre_create_hooks.each do |hook| - return false if hook.call(instance) == false + Hyperactive::Hooker.call_with_hooks(instance, *self.create_hooks) do + CAPTAIN[instance.record_id] = instance end - CAPTAIN[instance.record_id] = instance - @@post_create_hooks.each do |hook| - begin - hook.call(instance) - rescue - # /moo - end - end proxy = CAPTAIN[instance.record_id] @@ -94,6 +136,18 @@ attr_reader :record_id # + # This will allow us to wrap any write of us to persistent storage + # in the @@save_hooks as long as the Archipelago::Hashish provider + # supports it. See Archipelago::Hashish::BerkeleyHashish for an example + # of Hashish providers that do this. + # + def save_hook(&block) + Hyperactive::Hooker.call_with_hooks(self, *self.class.save_hooks) do + yield + end + end + + # # Remove this instance from the database calling all the right hooks. # # Freezes this instance after having deleted it. @@ -104,19 +158,30 @@ # Returns true otherwise. # def destroy - @@pre_destroy_hooks.each do |hook| - return false if hook.call(self) == false + Hyperactive::Hooker.call_with_hooks(self, *self.class.destroy_hooks) do + CAPTAIN.delete(@record_id) + self.freeze end - CAPTAIN.delete(@record_id) - self.freeze - @@post_destroy_hooks.each do |hook| - begin - hook.call(self) - rescue - # /moo - end + end + + private + + # + # Get an Array from +hash+ using self as + # key. If self doesnt exist in the +hash+ + # it will recurse by calling the same method in the + # superclass until it has been called in Hyperactive::Record. + # + def self.get_array_by_class(hash) + return hash[self] if hash.include?(self) + + if self == Record + hash[self] = [] + return self.get_array_by_class(hash) + else + hash[self] = self.superclass.get_array_by_class(hash).clone + return self.get_array_by_class(hash) end - return true end end Modified: trunk/hyperactive/lib/hyperactive.rb =================================================================== --- trunk/hyperactive/lib/hyperactive.rb 2006-11-25 12:27:39 UTC (rev 43) +++ trunk/hyperactive/lib/hyperactive.rb 2006-11-25 13:06:45 UTC (rev 44) @@ -1,5 +1,6 @@ $: << File.dirname(__FILE__) +require 'hyperactive/hooker' require 'hyperactive/record' require 'hyperactive/tree' Added: trunk/hyperactive/tests/record_test.rb =================================================================== --- trunk/hyperactive/tests/record_test.rb (rev 0) +++ trunk/hyperactive/tests/record_test.rb 2006-11-25 13:06:45 UTC (rev 44) @@ -0,0 +1,80 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class MyRecord < Hyperactive::Record + attr_accessor :bajs + + def self.save_hook(instance, &block) + $BEFORE_SAVE += 1 + yield + $AFTER_SAVE += 1 + end + save_hooks << self.method(:save_hook) + + def self.create_hook(instance, &block) + $BEFORE_CREATE += 1 + yield + $AFTER_CREATE += 1 + end + create_hooks << self.method(:create_hook) + + def self.destroy_hook(instance, &block) + $BEFORE_DESTROY += 1 + yield + $AFTER_DESTROY += 1 + end + destroy_hooks << self.method(:destroy_hook) +end + +class RecordTest < Test::Unit::TestCase + + def setup + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + assert_within(10) do + !Hyperactive::CAPTAIN.chests.empty? + end + assert_within(10) do + !Hyperactive::CAPTAIN.trannies.empty? + end + $BEFORE_SAVE = 0 + $AFTER_SAVE = 0 + $BEFORE_DESTROY = 0 + $AFTER_DESTROY = 0 + $BEFORE_CREATE = 0 + $AFTER_CREATE = 0 + end + + def teardown + @c.persistence_provider.unlink + @c2.persistence_provider.unlink + @tm.persistence_provider.unlink + end + + def test_create_update_destroy + r = MyRecord.get_instance + assert(Archipelago::Treasure::Dubloon === r) + assert_equal(1, $BEFORE_SAVE) + assert_equal(1, $AFTER_SAVE) + assert_equal(1, $BEFORE_CREATE) + assert_equal(1, $AFTER_CREATE) + r.bajs = "brunt" + assert_equal(2, $BEFORE_SAVE) + assert_equal(2, $AFTER_SAVE) + assert_equal("brunt", r.bajs) + assert_equal("brunt", Hyperactive::CAPTAIN[r.record_id].bajs) + i = r.record_id + assert_equal(r, Hyperactive::CAPTAIN[i]) + r.destroy + assert_equal(1, $BEFORE_DESTROY) + assert_equal(1, $AFTER_DESTROY) + assert_equal(nil, Hyperactive::CAPTAIN[i]) + end + +end From nobody at rubyforge.org Sat Nov 25 19:04:29 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sat, 25 Nov 2006 19:04:29 -0500 (EST) Subject: [Archipelago-submits] [45] trunk/archipelago/profiles: removed profiles Message-ID: <20061126000429.46AFC524145A@rubyforge.org> Revision: 45 Author: zond Date: 2006-11-25 19:04:28 -0500 (Sat, 25 Nov 2006) Log Message: ----------- removed profiles Removed Paths: ------------- trunk/archipelago/profiles/1000xChest#join!-prepare!-commit!.rb trunk/archipelago/profiles/1000xDubloon#[]=(t).rb trunk/archipelago/profiles/1000xDubloon#method_missing(t).rb trunk/archipelago/profiles/README trunk/archipelago/profiles/profile_helper.rb Deleted: trunk/archipelago/profiles/1000xChest#join!-prepare!-commit!.rb =================================================================== --- trunk/archipelago/profiles/1000xChest#join!-prepare!-commit!.rb 2006-11-25 13:06:45 UTC (rev 44) +++ trunk/archipelago/profiles/1000xChest#join!-prepare!-commit!.rb 2006-11-26 00:04:28 UTC (rev 45) @@ -1,19 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'profile_helper') -require 'treasure' -require 'drb' - -DRb.start_service - at c = TestChest.new - -k = "hej" -v = "oj" -t = TestTransaction.new -1000.times do |n| - @c[k, t] = v - @c.prepare!(t) - @c.commit!(t) -end - - at c.persistence_provider.unlink -DRb.stop_service Deleted: trunk/archipelago/profiles/1000xDubloon#[]=(t).rb =================================================================== --- trunk/archipelago/profiles/1000xDubloon#[]=(t).rb 2006-11-25 13:06:45 UTC (rev 44) +++ trunk/archipelago/profiles/1000xDubloon#[]=(t).rb 2006-11-26 00:04:28 UTC (rev 45) @@ -1,19 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'profile_helper') -require 'treasure' -require 'drb' - -DRb.start_service - at c = TestChest.new - at tm = TestManager.new - -tr = @tm.begin -k = "hej" -v = "oj" -1000.times do |n| - @c[k,tr] = v -end - - at c.persistence_provider.unlink -File.unlink(@tm.db.filename) -DRb.stop_service Deleted: trunk/archipelago/profiles/1000xDubloon#method_missing(t).rb =================================================================== --- trunk/archipelago/profiles/1000xDubloon#method_missing(t).rb 2006-11-25 13:06:45 UTC (rev 44) +++ trunk/archipelago/profiles/1000xDubloon#method_missing(t).rb 2006-11-26 00:04:28 UTC (rev 45) @@ -1,21 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'profile_helper') -require 'treasure' -require 'drb' - -DRb.start_service - at c = TestChest.new - at tm = TestManager.new - -tr = @tm.begin -k = "hej" -v = "oj" - at c[k,tr] = v -v = @c[k,tr] -1000.times do |n| - t = v.upcase -end - - at c.persistence_provider.unlink -File.unlink(@tm.db.filename) -DRb.stop_service Deleted: trunk/archipelago/profiles/README =================================================================== --- trunk/archipelago/profiles/README 2006-11-25 13:06:45 UTC (rev 44) +++ trunk/archipelago/profiles/README 2006-11-26 00:04:28 UTC (rev 45) @@ -1,3 +0,0 @@ -These are just simple scripts to profile certain parts of the system. - -Use ruby-prof to run them and get good profiling. Deleted: trunk/archipelago/profiles/profile_helper.rb =================================================================== --- trunk/archipelago/profiles/profile_helper.rb 2006-11-25 13:06:45 UTC (rev 44) +++ trunk/archipelago/profiles/profile_helper.rb 2006-11-26 00:04:28 UTC (rev 45) @@ -1,25 +0,0 @@ - -home = File.expand_path(File.dirname(__FILE__)) -$: << File.join(home, "..", "lib") - -require 'pp' -require 'tranny' -require 'treasure' - -class TestManager < Archipelago::Tranny::Manager - attr_reader :db - def log_error(e) - puts e - pp e.backtrace - end -end - -class TestChest < Archipelago::Treasure::Chest - attr_reader :persistence_provider -end - -class TestTransaction - def join(o) - end -end - From nobody at rubyforge.org Sat Nov 25 19:04:48 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sat, 25 Nov 2006 19:04:48 -0500 (EST) Subject: [Archipelago-submits] [46] trunk/archipelago/profiles/: removed profiles Message-ID: <20061126000448.BC823524145E@rubyforge.org> Revision: 46 Author: zond Date: 2006-11-25 19:04:48 -0500 (Sat, 25 Nov 2006) Log Message: ----------- removed profiles Removed Paths: ------------- trunk/archipelago/profiles/ From nobody at rubyforge.org Sat Nov 25 19:26:27 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sat, 25 Nov 2006 19:26:27 -0500 (EST) Subject: [Archipelago-submits] [47] trunk/hyperactive: cleaned up tests a bit Message-ID: <20061126002627.E9869524145F@rubyforge.org> Revision: 47 Author: zond Date: 2006-11-25 19:26:27 -0500 (Sat, 25 Nov 2006) Log Message: ----------- cleaned up tests a bit Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/tree.rb trunk/hyperactive/tests/tree_benchmark.rb Modified: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-26 00:04:48 UTC (rev 46) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-26 00:26:27 UTC (rev 47) @@ -31,28 +31,6 @@ end # - # Returns how balanced this tree is - # - def balance - return 1 if @elements - - smallest = nil - largest = nil - avg = 0 - count = 0 - @subtrees.t_each do |tree_id, tree| - size = tree.size - avg += size - count += 1 - smallest = size if smallest.nil? || smallest > size - largest = size if largest.nil? || largest < size - end - avg /= count.to_f - - return ((smallest.to_f / avg) + (avg / largest.to_f)) / 2.0 - end - - # # Deletes +key+ in this Tree. # def delete(key) Modified: trunk/hyperactive/tests/tree_benchmark.rb =================================================================== --- trunk/hyperactive/tests/tree_benchmark.rb 2006-11-26 00:04:48 UTC (rev 46) +++ trunk/hyperactive/tests/tree_benchmark.rb 2006-11-26 00:26:27 UTC (rev 47) @@ -52,8 +52,6 @@ f = rand(1 << 32) h[f] = r end - puts "balance: #{h.balance}" if h.respond_to?(:balance) - puts "sizes: #{h.subtrees.collect do |k,t| t.size end.inspect}" if h.respond_to?(:subtrees) bm("#{label}#set/get/delete (big)", :n => c * 1) do f = rand(1 << 32) h[f] = r From nobody at rubyforge.org Sat Nov 25 19:37:01 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sat, 25 Nov 2006 19:37:01 -0500 (EST) Subject: [Archipelago-submits] [48] trunk/hyperactive/tests/tree_test.rb: added select/reject tests as well Message-ID: <20061126003701.E6F765241463@rubyforge.org> Revision: 48 Author: zond Date: 2006-11-25 19:37:01 -0500 (Sat, 25 Nov 2006) Log Message: ----------- added select/reject tests as well Modified Paths: -------------- trunk/hyperactive/tests/tree_test.rb Modified: trunk/hyperactive/tests/tree_test.rb =================================================================== --- trunk/hyperactive/tests/tree_test.rb 2006-11-26 00:26:27 UTC (rev 47) +++ trunk/hyperactive/tests/tree_test.rb 2006-11-26 00:37:01 UTC (rev 48) @@ -1,6 +1,15 @@ require File.join(File.dirname(__FILE__), 'test_helper') +class RecordMatcher + def initialize(i) + @i = i + end + def call(k,v) + k == @i + end +end + class TreeTest < Test::Unit::TestCase def setup @@ -25,6 +34,16 @@ @c2.persistence_provider.unlink @tm.persistence_provider.unlink end + + def test_select_reject + h = Hyperactive::Tree.get_instance + r1 = Hyperactive::Record.get_instance + r2 = Hyperactive::Record.get_instance + h[r1.record_id] = r1 + h[r2.record_id] = r2 + assert_equal(r1.record_id, h.select(RecordMatcher.new(r1.record_id)).first.first) + assert_equal(r1.record_id, h.reject(RecordMatcher.new(r2.record_id)).keys.first) + end def test_set_get h = Hyperactive::Tree.get_instance From nobody at rubyforge.org Sat Nov 25 20:04:54 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sat, 25 Nov 2006 20:04:54 -0500 (EST) Subject: [Archipelago-submits] [49] trunk/hyperactive/lib/hyperactive/record.rb: created select and reject class method for Record. Message-ID: <20061126010454.9B81D524147C@rubyforge.org> Revision: 49 Author: zond Date: 2006-11-25 20:04:54 -0500 (Sat, 25 Nov 2006) Log Message: ----------- created select and reject class method for Record. completely untested :O Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-26 00:37:01 UTC (rev 48) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-26 01:04:54 UTC (rev 49) @@ -14,6 +14,51 @@ CAPTAIN = Archipelago::Pirate::Captain.new # + # A tiny callable class that saves stuff inside containers + # if they match certain criteria. + # + class MatchSaver + # + # Initialize this MatchSaver with a +key+, a callable +matcher+ + # and a +mode+ (:select, :reject, :delete_if_match or :delete_unless_match). + # + def initialize(key, matcher, mode) + @key = key + @matcher = matcher + @mode = mode + end + # + # Depending on @mode and return value of @matcher.call + # may save record in the Hash-like container named @key in + # the main database. + # + def call(record) + case @mode + when :select + if @matcher.call(record) + CAPTAIN[@key][record.record_id] = record + else + CAPTAIN[@key].delete(record.record_id) + end + when :reject + if @matcher.call(record) + CAPTAIN[@key].delete(record.record_id) + else + CAPTAIN[@key][record.record_id] = record unless @matcher.call(record) + end + when :delete_if_match + if @matcher.call(record) + CAPTAIN[@key].delete(record.record_id) + end + when :delete_unless_match + unless @matcher.call(record) + CAPTAIN[@key].delete(record.record_id) + end + end + end + end + + # # A convenient base class to inherit when you want the basic utility methods # provided by for example ActiveRecord::Base *hint hint*. # @@ -59,7 +104,7 @@ # instances upon creation. # def self.create_hooks - self.get_array_by_class(@@create_hooks_by_class) + self.get_hook_array_by_class(@@create_hooks_by_class) end # @@ -76,7 +121,7 @@ # instances upon destruction. # def self.destroy_hooks - self.get_array_by_class(@@destroy_hooks_by_class) + self.get_hook_array_by_class(@@destroy_hooks_by_class) end # @@ -93,10 +138,44 @@ # instances upon saving. # def self.save_hooks - self.get_array_by_class(@@save_hooks_by_class) + self.get_hook_array_by_class(@@save_hooks_by_class) end + + # + # Will define a method called +name+ that will include all + # existing instances of this class that when sent to +matcher+.call + # return true. Will only return instances saved after this selector + # is defined. + # + def self.select(name, matcher) + key = collection_key(name) + self.class_eval < Revision: 50 Author: zond Date: 2006-11-26 20:14:19 -0500 (Sun, 26 Nov 2006) Log Message: ----------- added tests for Record.reject and Record.select, and fixed some bugs for it Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive/tree.rb trunk/hyperactive/tests/record_test.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-26 01:04:54 UTC (rev 49) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 01:14:19 UTC (rev 50) @@ -30,9 +30,10 @@ # # Depending on @mode and return value of @matcher.call # may save record in the Hash-like container named @key in - # the main database. + # the main database after having yielded to +block+. # - def call(record) + def call(record, &block) + yield case @mode when :select if @matcher.call(record) @@ -44,7 +45,7 @@ if @matcher.call(record) CAPTAIN[@key].delete(record.record_id) else - CAPTAIN[@key][record.record_id] = record unless @matcher.call(record) + CAPTAIN[@key][record.record_id] = record end when :delete_if_match if @matcher.call(record) @@ -148,10 +149,11 @@ # is defined. # def self.select(name, matcher) - key = collection_key(name) + key = self.collection_key(name) + CAPTAIN[key] ||= Tree.get_instance self.class_eval <self as # key. If self doesnt exist in the +hash+ # it will recurse by calling the same method in the Modified: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-26 01:04:54 UTC (rev 49) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-27 01:14:19 UTC (rev 50) @@ -123,9 +123,9 @@ # def collect(callable) rval = [] - self.each do |e| - rval << callable.call(e) - end + self.each(Proc.new do |k,v| + rval << callable.call(k,v) + end) return rval end Modified: trunk/hyperactive/tests/record_test.rb =================================================================== --- trunk/hyperactive/tests/record_test.rb 2006-11-26 01:04:54 UTC (rev 49) +++ trunk/hyperactive/tests/record_test.rb 2006-11-27 01:14:19 UTC (rev 50) @@ -56,6 +56,44 @@ @c2.persistence_provider.unlink @tm.persistence_provider.unlink end + + def test_select_reject + MyRecord.class_eval do + select(:brunt, Proc.new do |r| + r.bajs == "brunt" + end) + reject(:not_brunt, Proc.new do |r| + r.bajs == "brunt" + end) + end + r1 = MyRecord.get_instance + r1.bajs = "brunt" + r2 = MyRecord.get_instance + r2.bajs = "brunt" + r3 = MyRecord.get_instance + r3.bajs = "gult" + r4 = MyRecord.get_instance + r4.bajs = "beige" + + assert_equal(Set.new([r2.record_id,r1.record_id]), Set.new(MyRecord.brunt.collect(Proc.new do |k,v| + v.record_id + end))) + assert_equal(Set.new([r3.record_id,r4.record_id]), Set.new(MyRecord.not_brunt.collect(Proc.new do |k,v| + v.record_id + end))) + + r1.destroy + r2.destroy + r3.destroy + r4.destroy + + assert_equal(Set.new, Set.new(MyRecord.brunt.collect(Proc.new do |k,v| + v.record_id + end))) + assert_equal(Set.new, Set.new(MyRecord.not_brunt.collect(Proc.new do |k,v| + v.record_id + end))) + end def test_create_update_destroy r = MyRecord.get_instance From nobody at rubyforge.org Sun Nov 26 21:04:49 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sun, 26 Nov 2006 21:04:49 -0500 (EST) Subject: [Archipelago-submits] [51] trunk/archipelago: added Publishable#stop! that stops the Jockey or unpublishes the Publishable. Message-ID: <20061127020449.5651F5241841@rubyforge.org> Revision: 51 Author: zond Date: 2006-11-26 21:04:48 -0500 (Sun, 26 Nov 2006) Log Message: ----------- added Publishable#stop! that stops the Jockey or unpublishes the Publishable. used it in tests. added assertions that it is the RIGHT chests the pirate has in pirate test. added BerkeleyHashish#each, Chest#each and Pirate#each and tests for them. simplified Publishable#service_id. made BerkeleyHashishProvider close the databases before it unlinks. Modified Paths: -------------- trunk/archipelago/lib/archipelago/disco.rb trunk/archipelago/lib/archipelago/hashish.rb trunk/archipelago/lib/archipelago/pirate.rb trunk/archipelago/lib/archipelago/treasure.rb trunk/archipelago/tests/pirate_test.rb trunk/archipelago/tests/treasure_test.rb Modified: trunk/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-27 01:14:19 UTC (rev 50) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-27 02:04:48 UTC (rev 51) @@ -145,6 +145,17 @@ def valid? true end + + # + # Stops the publishing of this Publishable. + # + def stop! + if defined?(Archipelago::Disco::MC) && @jockey == Archipelago::Disco::MC + @jockey.unpublish(self.service_id) + else + @jockey.stop! + end + end # # Returns our semi-unique id so that we can be found again. @@ -158,11 +169,7 @@ # Stuff that didnt fit in any of the other databases. # @metadata ||= @persistence_provider.get_hashish("metadata") - service_id = @metadata["service_id"] - unless service_id - service_id = @metadata["service_id"] ||= Digest::SHA1.hexdigest("#{HOST}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") - end - return service_id + return @metadata["service_id"] ||= Digest::SHA1.hexdigest("#{HOST}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") end end @@ -224,6 +231,16 @@ end # + # A class used to defined removed services. + # + class UnPublish + attr_reader :service_id + def initialize(service_id) + @service_id = service_id + end + end + + # # A class used to define an existing service. # class Record < ServiceDescription @@ -253,6 +270,7 @@ class ServiceLocker attr_reader :hash include Archipelago::Current::Synchronized + include Archipelago::Current::ThreadedCollection def initialize(hash = nil) super @hash = hash || {} @@ -278,7 +296,7 @@ end end else - super(*args) + super(meth, *args, &block) end end # @@ -454,6 +472,17 @@ end end + # + # Removes the service with given +service_id+ from the published services. + # + def unpublish(service_id) + @local_services.delete(service_id) + @new_service_semaphore.broadcast + unless @thrifty_publishing + @outgoing << [nil, UnPublish.new(service_id)] + end + end + private # @@ -534,8 +563,8 @@ end # - # Start the thread picking incoming Records and Queries and - # handling them properly + # Start the thread picking incoming Records, Queries and UnPublishes and + # handling them properly. # def start_picker @picker_thread = Thread.new do @@ -555,6 +584,8 @@ @remote_services[data[:service_id]] = data @new_service_semaphore.broadcast end + elsif Archipelago::Disco::UnPublish === data + @remote_services.delete(data.service_id) end rescue Exception => e puts e Modified: trunk/archipelago/lib/archipelago/hashish.rb =================================================================== --- trunk/archipelago/lib/archipelago/hashish.rb 2006-11-27 01:14:19 UTC (rev 50) +++ trunk/archipelago/lib/archipelago/hashish.rb 2006-11-27 02:04:48 UTC (rev 51) @@ -48,6 +48,13 @@ @lock = Archipelago::Current::Lock.new end # + # Close the @content_db and @timestamps_db behind this BerkeleyHashish + # + def close! + @content_db.close + @timestamps_db.close + end + # # Returns a deep ( Marshal.load(Marshal.dump(o)) ) clone # of the object represented by +key+. # @@ -125,6 +132,19 @@ end end # + # Will do +callable+.call(key, value) for each + # key-and-value pair in this Hashish. + # + # NB: This is totaly thread-unsafe, only do this + # for management or rescue! + # + def each(callable) + @content_db.each do |serialized_key, serialized_value| + key = Marshal.load(serialized_key) + callable.call(key, self.[](key)) + end + end + # # Delete +key+ and its value and timestamp. # def delete(key) @@ -198,6 +218,8 @@ def initialize(env_path) env_path.mkpath @env = BDB::Env.open(env_path, BDB::CREATE | BDB::INIT_MPOOL) + @berkeley_hashishes = [] + @bdb_dbs = [] end # # Returns a cleverly cached (but slightly inefficient) @@ -205,20 +227,31 @@ # using +name+. # def get_cached_hashish(name) - BerkeleyHashish.new(name, @env) + hashish = BerkeleyHashish.new(name, @env) + @berkeley_hashishes << hashish + return hashish end # # Returns a normal hash-like instance using +name+. # def get_hashish(name) - @env.open_db(BDB::HASH, name, nil, BDB::CREATE | BDB::NOMMAP) + db = @env.open_db(BDB::HASH, name, nil, BDB::CREATE | BDB::NOMMAP) + @bdb_dbs << db + return db end # - # Removes the persistent files of this instance. + # Closes databases opened by this instance and removes the persistent files. # def unlink - p = Pathname.new(@env.home) - p.rmtree if p.exist? + @berkeley_hashishes.each do |h| + h.close! + end + @bdb_dbs.each do |d| + d.close + end + home = Pathname.new(@env.home) + @env.close + home.rmtree if home.exist? end end Modified: trunk/archipelago/lib/archipelago/pirate.rb =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb 2006-11-27 01:14:19 UTC (rev 50) +++ trunk/archipelago/lib/archipelago/pirate.rb 2006-11-27 02:04:48 UTC (rev 51) @@ -220,6 +220,19 @@ @service_update_thread.kill end + # + # Will do +callable+.call(key, value) + # for each key-and-value pair in this database network. + # + # NB: This is totaly thread-unsafe, only do this + # for management or rescue! + # + def each(callable) + @chests.t_each do |service_id, chest| + chest[:service].each(callable) + end + end + private # Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-27 01:14:19 UTC (rev 50) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-27 02:04:48 UTC (rev 51) @@ -296,6 +296,17 @@ end # + # Will do +callable+.call(key, value) + # for each key-and-value pair in this Chest. + # + # NB: This is totaly thread-unsafe, only do this + # for management or rescue! + # + def each(callable) + @db.each(callable) + end + + # # Evaluate +data+ if we have not already seen +label+ or if we have an earlier +timestamp+ than the one given. # def evaluate!(label, timestamp, data) Modified: trunk/archipelago/tests/pirate_test.rb =================================================================== --- trunk/archipelago/tests/pirate_test.rb 2006-11-27 01:14:19 UTC (rev 50) +++ trunk/archipelago/tests/pirate_test.rb 2006-11-27 02:04:48 UTC (rev 51) @@ -11,24 +11,37 @@ @c.publish! @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) @c2.publish! - @tm = TestManager.new + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny.db"))) @tm.publish! - assert_within(10) do - !@p.chests.empty? + assert_within(2) do + Set.new(@p.chests.keys) == Set.new([@c.service_id, @c2.service_id]) end - assert_within(10) do - !@p.trannies.empty? + assert_within(2) do + @p.trannies.keys == [@tm.service_id] end end def teardown @p.stop! + @c.stop! @c.persistence_provider.unlink + @c2.stop! @c2.persistence_provider.unlink + @tm.stop! @tm.persistence_provider.unlink DRb.stop_service end + def test_each + @p["oj"] = "bla" + @p["brunt"] = "ja" + h = {} + @p.each(Proc.new do |k,v| + h[k] = v + end) + assert_equal({"oj" => "bla", "brunt" => "ja"}, h) + end + def test_evaluate assert_raise(NameError) do e = Evaltest.new Modified: trunk/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-27 01:14:19 UTC (rev 50) +++ trunk/archipelago/tests/treasure_test.rb 2006-11-27 02:04:48 UTC (rev 51) @@ -27,6 +27,16 @@ DRb.stop_service end + def test_each + @c["oj"] = "bla" + @c["brunt"] = "ja" + h = {} + @c.each(Proc.new do |k,v| + h[k] = v + end) + assert_equal({"oj" => "bla", "brunt" => "ja"}, h) + end + def test_around_save s = A.new("hehu") @c["oj"] = s From nobody at rubyforge.org Sun Nov 26 22:00:59 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sun, 26 Nov 2006 22:00:59 -0500 (EST) Subject: [Archipelago-submits] [52] trunk/archipelago: improved comments. Message-ID: <20061127030059.6F232A970002@rubyforge.org> Revision: 52 Author: zond Date: 2006-11-26 22:00:58 -0500 (Sun, 26 Nov 2006) Log Message: ----------- improved comments. made save_hook take old value as argument. fixed typos. Modified Paths: -------------- trunk/archipelago/lib/archipelago/hashish.rb trunk/archipelago/lib/archipelago/treasure.rb trunk/archipelago/tests/treasure_test.rb Modified: trunk/archipelago/lib/archipelago/hashish.rb =================================================================== --- trunk/archipelago/lib/archipelago/hashish.rb 2006-11-27 02:04:48 UTC (rev 51) +++ trunk/archipelago/lib/archipelago/hashish.rb 2006-11-27 03:00:58 UTC (rev 52) @@ -77,7 +77,7 @@ # # Insert +value+ under +key+. # - # Will call value.save_hook and send + # Will call value.save_hook(old_value) and send # it a block that does the actual saving if # value.respond_to?(:save_hook). # @@ -96,7 +96,7 @@ # Stores whatever is under +key+ if it is not the same as # whats in the persistent db. # - # Will call value.save_hook and send + # Will call value.save_hook(old_value) and send # it a block that does the actual saving if # value.respond_to?(:save_hook) and # a save is actually performed. @@ -166,13 +166,15 @@ # Write +key+, serialized as +serialized_key+ and # +serialized_value+ to the db. # - # Will call value.save_hook and send + # Will call value.save_hook(old_value) and send # it a block that does the actual saving if # value.respond_to?(:save_hook). # def write_to_db(key, serialized_key, serialized_value, value) if value.respond_to?(:save_hook) - value.save_hook do + old_serialized_value = @content_db[serialized_key] + old_value = old_serialized_value ? Marshal.load(old_serialized_value) : nil + value.save_hook(old_value) do do_write_to_db(key, serialized_key, serialized_value) end else Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-27 02:04:48 UTC (rev 51) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-27 03:00:58 UTC (rev 52) @@ -39,7 +39,7 @@ # The known chests by service id to speed up # recovery by Dubloons having lost their Chest. # - CHESTS_BY_SERVICE_ID = {} + CHEST_BY_SERVICE_ID = {} # # Raised whenever the optimistic locking of the serializable transaction isolation level @@ -153,16 +153,11 @@ # def self._load(s) key, chest, transaction, chest_id = Marshal.load(s) - instance = self.allocate - instance.instance_variable_set(:@key, key) - instance.instance_variable_set(:@transaction, transaction) - instance.instance_variable_set(:@chest_id, chest_id) if CHEST_BY_SERVICE_ID.include?(chest_id) - instance.instance_variable_set(:@chest, CHEST_BY_SERVICE_ID[chest_id]) - else - instance.instance_variable_set(:@chest, chest) + chest = CHEST_BY_SERVICE_ID[chest_id] end - return instance + + return self.new(key, chest, transaction, chest_id) end # # Return a clone of myself that is joined to @@ -208,7 +203,7 @@ raise e if possible_replacements.empty? @chest = possible_replacements[@chest_id][:service] - CHESTS_BY_SERVICE_ID[@chest_id] = @chest + CHEST_BY_SERVICE_ID[@chest_id] = @chest retry else @@ -285,7 +280,7 @@ initialize_prepared(options[:transaction_recovery_interval] || TRANSACTION_RECOVERY_INTERVAL) - CHESTS_BY_SERVICE_ID[self.service_id] = self + CHEST_BY_SERVICE_ID[self.service_id] = self end # Modified: trunk/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-27 02:04:48 UTC (rev 51) +++ trunk/archipelago/tests/treasure_test.rb 2006-11-27 03:00:58 UTC (rev 52) @@ -2,7 +2,7 @@ require File.join(File.dirname(__FILE__), 'test_helper') class A < String - def save_hook(&block) + def save_hook(old_value, &block) $BURKMAT += 1 yield $BURKMAT2 += 1 From nobody at rubyforge.org Sun Nov 26 22:02:33 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sun, 26 Nov 2006 22:02:33 -0500 (EST) Subject: [Archipelago-submits] [53] trunk/hyperactive: improved tests. Message-ID: <20061127030233.D4BBEA970002@rubyforge.org> Revision: 53 Author: zond Date: 2006-11-26 22:02:33 -0500 (Sun, 26 Nov 2006) Log Message: ----------- improved tests. added Record.index_by and Record.find_by (completely untested!). improved comments. Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/hooker.rb trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/tests/record_test.rb trunk/hyperactive/tests/tree_test.rb Modified: trunk/hyperactive/lib/hyperactive/hooker.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/hooker.rb 2006-11-27 03:00:58 UTC (rev 52) +++ trunk/hyperactive/lib/hyperactive/hooker.rb 2006-11-27 03:02:33 UTC (rev 53) @@ -8,7 +8,7 @@ # # Will call each hook in +hooks+ with a block that - # calls the rest of the +hooks+ with +object+ as argument + # calls the rest of the +hooks+ with +argument+ as argument # and after that yields to the given +block+. # # This will allow you wrap any method call in a dynamic set @@ -44,14 +44,14 @@ # end # end # - def call_with_hooks(object, *hooks, &block) + def call_with_hooks(argument, *hooks, &block) if hooks.empty? yield else first = hooks.first rest = hooks[1..-1] - first.call(object) do - call_with_hooks(object, *rest, &block) + first.call(argument) do + call_with_hooks(argument, *rest, &block) end end end Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 03:00:58 UTC (rev 52) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 03:02:33 UTC (rev 53) @@ -14,6 +14,69 @@ CAPTAIN = Archipelago::Pirate::Captain.new # + # A tiny callable class that saves stuff in + # indexes depending on certain attributes. + # + class IndexBuilder + # + # Get the first part of the key, that depends on the +attributes+. + # + def self.get_attribute_key_part(attributes) + "Hyperactive::IndexBuilder::#{attributes.join(",")}" + end + # + # Get the last part of the key, that depends on the +values+. + # + def self.get_value_key_part(values) + "#{values.join(",")}" + end + # + # Initialize an IndexBuilder giving it an array of +attributes+ + # that will be indexed. + # + def initialize(attributes) + @attributes = attributes + end + # + # Get the Tree for the given +record+. + # + def get_tree_for(record) + values = attributes.collect do |att| + if record.respond_to?(att) + record.send(att) + else + nil + end + end + key = "#{self.class.get_attribute_key_part(attributes)}::#{self.class.get_value_key_part(values)}" + return CAPTAIN[key] ||= Tree.get_instance + end + # + # Call this IndexBuilder and pass it a +block+. + # + # If the +argument+ is an Array then we know we are a save hook, + # otherwise we are a destroy hook. + # + def call(argument, &block) + yield + + # + # If the argument is an Array (of old value, new value) + # then we are a save hook, otherwise a destroy hook. + # + if Array === argument + record = argument.last + old_record = argument.first + get_tree_for(old_record).delete(record.record_id) + get_tree_for(record)[record.record_id] = record + else + record = argument + get_tree_for(record).delete(record.record_id) + end + end + end + + # # A tiny callable class that saves stuff inside containers # if they match certain criteria. # @@ -32,8 +95,12 @@ # may save record in the Hash-like container named @key in # the main database after having yielded to +block+. # - def call(record, &block) + def call(argument, &block) yield + + record = argument + record = argument.last if Array === argument + case @mode when :select if @matcher.call(record) @@ -129,8 +196,9 @@ # Return our save_hooks, which can then be treated as any old Array. # # These must be callable objects with an arity of 1 - # that will be sent the instance about to be saved (storage - # into the database system) that take a block argument. + # that will be sent [the old version, the new version] of the + # instance about to be saved (storage into the database system) + # along with a block argument. # # The block argument will be a Proc that actually saves the # instance into the database system. @@ -141,6 +209,17 @@ def self.save_hooks self.get_hook_array_by_class(@@save_hooks_by_class) end + + def self.index_by(*attributes) + attribute_key_part = IndexBuilder.get_attribute_key_part(attributes) + self.class_eval < {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) assert_within(10) do - !Hyperactive::CAPTAIN.chests.empty? + Set.new(Hyperactive::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) end assert_within(10) do - !Hyperactive::CAPTAIN.trannies.empty? + Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] end $BEFORE_SAVE = 0 $AFTER_SAVE = 0 @@ -52,8 +52,11 @@ end def teardown + @c.stop! @c.persistence_provider.unlink + @c2.stop! @c2.persistence_provider.unlink + @tm.stop! @tm.persistence_provider.unlink end Modified: trunk/hyperactive/tests/tree_test.rb =================================================================== --- trunk/hyperactive/tests/tree_test.rb 2006-11-27 03:00:58 UTC (rev 52) +++ trunk/hyperactive/tests/tree_test.rb 2006-11-27 03:02:33 UTC (rev 53) @@ -22,16 +22,19 @@ Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) assert_within(10) do - !Hyperactive::CAPTAIN.chests.empty? + Set.new(Hyperactive::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) end assert_within(10) do - !Hyperactive::CAPTAIN.trannies.empty? + Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] end end def teardown + @c.stop! @c.persistence_provider.unlink + @c2.stop! @c2.persistence_provider.unlink + @tm.stop! @tm.persistence_provider.unlink end From nobody at rubyforge.org Sun Nov 26 22:27:34 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Sun, 26 Nov 2006 22:27:34 -0500 (EST) Subject: [Archipelago-submits] [54] trunk/hyperactive: packaged first version of hyperactive_record. Message-ID: <20061127032734.D57A7A970003@rubyforge.org> Revision: 54 Author: zond Date: 2006-11-26 22:27:33 -0500 (Sun, 26 Nov 2006) Log Message: ----------- packaged first version of hyperactive_record. added tests for index_by and find_by. made tree test more failsafe in setup. fixed bugs in record. Modified Paths: -------------- trunk/hyperactive/Rakefile trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/tests/record_test.rb trunk/hyperactive/tests/tree_test.rb Property Changed: ---------------- trunk/hyperactive/ Property changes on: trunk/hyperactive ___________________________________________________________________ Name: svn:ignore + pkg Name: svn::ignore + Modified: trunk/hyperactive/Rakefile =================================================================== --- trunk/hyperactive/Rakefile 2006-11-27 03:02:33 UTC (rev 53) +++ trunk/hyperactive/Rakefile 2006-11-27 03:27:33 UTC (rev 54) @@ -12,8 +12,8 @@ s.version = "0.1.0" s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" - s.summary = "A base class for Ruby on Rails models that uses archipelago for persistence." - s.files = FileList['lib/*.rb', 'test/*', 'GPL-2', 'TODO'].to_a + s.summary = "A base class for persistent objects that uses archipelago for persistence. Useful for Ruby on Rails models for example." + s.files = FileList['lib/*.rb', 'test/*', 'GPL-2'].to_a s.require_path = "lib" s.autorequire = "hyperactive_record" s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 03:02:33 UTC (rev 53) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 03:27:33 UTC (rev 54) @@ -41,14 +41,14 @@ # Get the Tree for the given +record+. # def get_tree_for(record) - values = attributes.collect do |att| + values = @attributes.collect do |att| if record.respond_to?(att) record.send(att) else nil end end - key = "#{self.class.get_attribute_key_part(attributes)}::#{self.class.get_value_key_part(values)}" + key = "#{self.class.get_attribute_key_part(@attributes)}::#{self.class.get_value_key_part(values)}" return CAPTAIN[key] ||= Tree.get_instance end # @@ -214,11 +214,13 @@ attribute_key_part = IndexBuilder.get_attribute_key_part(attributes) self.class_eval < {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) - assert_within(10) do + assert_within(20) do Set.new(Hyperactive::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) end - assert_within(10) do + assert_within(20) do Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] end end From nobody at rubyforge.org Mon Nov 27 04:29:04 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 04:29:04 -0500 (EST) Subject: [Archipelago-submits] [55] trunk/archipelago: upped version to 0.2.0. Message-ID: <20061127092904.7CD655241549@rubyforge.org> Revision: 55 Author: zond Date: 2006-11-27 04:29:03 -0500 (Mon, 27 Nov 2006) Log Message: ----------- upped version to 0.2.0. made Dubloons always answer false to respond_to?(:marshal_dump) Modified Paths: -------------- trunk/archipelago/Rakefile trunk/archipelago/lib/archipelago/treasure.rb Modified: trunk/archipelago/Rakefile =================================================================== --- trunk/archipelago/Rakefile 2006-11-27 03:27:33 UTC (rev 54) +++ trunk/archipelago/Rakefile 2006-11-27 09:29:03 UTC (rev 55) @@ -9,7 +9,7 @@ spec = Gem::Specification.new do |s| s.platform = Gem::Platform::RUBY s.name = "archipelago" - s.version = "0.1.1" + s.version = "0.2.0" s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A set of tools for distributed computing in ruby." Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-27 03:27:33 UTC (rev 54) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-27 09:29:03 UTC (rev 55) @@ -186,9 +186,10 @@ # # Does our target respond to +meth+? # - alias :dubloon_redirected_respond_to? :respond_to? def respond_to?(meth) - return dubloon_redirected_respond_to?(meth) || self.method_missing(:respond_to?, meth) + # This one will be called far too often, and it seems safe to ignore it. + return false if meth == :marshal_dump + return super(meth) || self.method_missing(:respond_to?, meth) end # # Call +meth+ with +args+ and +block+ on our target if it responds to From nobody at rubyforge.org Mon Nov 27 04:37:22 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 04:37:22 -0500 (EST) Subject: [Archipelago-submits] [56] trunk/hyperactive: improved setup for tree benchmark. Message-ID: <20061127093722.5207C5241554@rubyforge.org> Revision: 56 Author: zond Date: 2006-11-27 04:37:18 -0500 (Mon, 27 Nov 2006) Log Message: ----------- improved setup for tree benchmark. made trees slightly wider. Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/tree.rb trunk/hyperactive/tests/tree_benchmark.rb Modified: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-27 09:29:03 UTC (rev 55) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-27 09:37:18 UTC (rev 56) @@ -19,7 +19,7 @@ attr_accessor :elements, :subtrees - WIDTH = 1 << 2 + WIDTH = 1 << 3 # # Dont call this! Call Tree.get_instance(options) instead! Modified: trunk/hyperactive/tests/tree_benchmark.rb =================================================================== --- trunk/hyperactive/tests/tree_benchmark.rb 2006-11-27 09:29:03 UTC (rev 55) +++ trunk/hyperactive/tests/tree_benchmark.rb 2006-11-27 09:37:18 UTC (rev 56) @@ -12,24 +12,27 @@ @tm.publish! Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) - assert_within(10) do - !Hyperactive::CAPTAIN.chests.empty? + assert_within(20) do + Set.new(Hyperactive::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) end - assert_within(10) do - !Hyperactive::CAPTAIN.trannies.empty? + assert_within(20) do + Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] end end def teardown + @c.stop! @c.persistence_provider.unlink + @c2.stop! @c2.persistence_provider.unlink + @tm.stop! @tm.persistence_provider.unlink end def test_set_get h = Hyperactive::Tree.get_instance r = Hyperactive::Record.get_instance - hash_test("Tree", h, r, 100) + hash_test("Tree", h, r, 1000) end def test_regular_hash_set_get From nobody at rubyforge.org Mon Nov 27 04:41:32 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 04:41:32 -0500 (EST) Subject: [Archipelago-submits] [57] trunk/archipelago/Rakefile: removed profiles from gem Message-ID: <20061127094132.3CCD05241554@rubyforge.org> Revision: 57 Author: zond Date: 2006-11-27 04:41:31 -0500 (Mon, 27 Nov 2006) Log Message: ----------- removed profiles from gem Modified Paths: -------------- trunk/archipelago/Rakefile Modified: trunk/archipelago/Rakefile =================================================================== --- trunk/archipelago/Rakefile 2006-11-27 09:37:18 UTC (rev 56) +++ trunk/archipelago/Rakefile 2006-11-27 09:41:31 UTC (rev 57) @@ -13,7 +13,7 @@ s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A set of tools for distributed computing in ruby." - s.files = FileList['lib/archipelago.rb', 'lib/archipelago/*.rb', 'test/*', 'script/*', 'GPL-2', 'TODO', 'profiles/*'].to_a + s.files = FileList['lib/archipelago.rb', 'lib/archipelago/*.rb', 'tests/*', 'script/*', 'GPL-2', 'TODO'].to_a s.require_path = "lib" s.autorequire = "archipelago" s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') From nobody at rubyforge.org Mon Nov 27 04:52:54 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 04:52:54 -0500 (EST) Subject: [Archipelago-submits] [58] trunk/archipelago/Rakefile: updated Rakefile Message-ID: <20061127095254.F12EB524154A@rubyforge.org> Revision: 58 Author: zond Date: 2006-11-27 04:52:54 -0500 (Mon, 27 Nov 2006) Log Message: ----------- updated Rakefile Modified Paths: -------------- trunk/archipelago/Rakefile Modified: trunk/archipelago/Rakefile =================================================================== --- trunk/archipelago/Rakefile 2006-11-27 09:41:31 UTC (rev 57) +++ trunk/archipelago/Rakefile 2006-11-27 09:52:54 UTC (rev 58) @@ -13,7 +13,7 @@ s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A set of tools for distributed computing in ruby." - s.files = FileList['lib/archipelago.rb', 'lib/archipelago/*.rb', 'tests/*', 'script/*', 'GPL-2', 'TODO'].to_a + s.files = FileList['lib/**/*.rb', 'tests/*', 'script/*', 'GPL-2', 'TODO'].to_a s.require_path = "lib" s.autorequire = "archipelago" s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') From nobody at rubyforge.org Mon Nov 27 05:10:18 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 05:10:18 -0500 (EST) Subject: [Archipelago-submits] [59] trunk/archipelago/lib/archipelago/disco.rb: made service_id into real strings. Message-ID: <20061127101018.8A7E75241567@rubyforge.org> Revision: 59 Author: zond Date: 2006-11-27 05:10:14 -0500 (Mon, 27 Nov 2006) Log Message: ----------- made service_id into real strings. improved Disco::Jockey#stop! Modified Paths: -------------- trunk/archipelago/lib/archipelago/disco.rb Modified: trunk/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-27 09:52:54 UTC (rev 58) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-27 10:10:14 UTC (rev 59) @@ -169,7 +169,7 @@ # Stuff that didnt fit in any of the other databases. # @metadata ||= @persistence_provider.get_hashish("metadata") - return @metadata["service_id"] ||= Digest::SHA1.hexdigest("#{HOST}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}") + return @metadata["service_id"] ||= Digest::SHA1.hexdigest("#{HOST}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}").to_s end end @@ -418,11 +418,17 @@ # Stops all the threads in this instance. # def stop! + @local_services.each do |service_id, service_description| + self.unpublish(service_id) + end @listener_thread.kill @unilistener_thread.kill + @validator_thread.kill + @picker_thread.kill + until @outgoing.empty? + sleep(0.01) + end @shouter_thread.kill - @picker_thread.kill - @validator_thread.kill end # From nobody at rubyforge.org Mon Nov 27 05:24:03 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 05:24:03 -0500 (EST) Subject: [Archipelago-submits] [60] trunk/archipelago/lib/archipelago/pirate.rb: made Captain#stop! also stop its Jockey Message-ID: <20061127102403.0C766524154D@rubyforge.org> Revision: 60 Author: zond Date: 2006-11-27 05:24:02 -0500 (Mon, 27 Nov 2006) Log Message: ----------- made Captain#stop! also stop its Jockey Modified Paths: -------------- trunk/archipelago/lib/archipelago/pirate.rb Modified: trunk/archipelago/lib/archipelago/pirate.rb =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb 2006-11-27 10:10:14 UTC (rev 59) +++ trunk/archipelago/lib/archipelago/pirate.rb 2006-11-27 10:24:02 UTC (rev 60) @@ -214,10 +214,14 @@ end # - # Stops the service update thread for this Pirate. + # Stops the service update thread for this Pirate and also unpublishes and/or + # stops the Jockey. # def stop! @service_update_thread.kill + unless defined?(Archipelago::Disco::MC) && @treasure_map && @treasure_map == Archipelago::Disco::MC + @treasure_map.stop! + end end # From nobody at rubyforge.org Mon Nov 27 06:08:35 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 06:08:35 -0500 (EST) Subject: [Archipelago-submits] [61] trunk: updated gem spec for hyperactive. Message-ID: <20061127110835.69D13524151D@rubyforge.org> Revision: 61 Author: zond Date: 2006-11-27 06:08:34 -0500 (Mon, 27 Nov 2006) Log Message: ----------- updated gem spec for hyperactive. made lots of tests more failsafe. stopped loading test_helper of archipelago in hyperactive. made hyperactive tree_benchmark a bit faster again. filled out the README of hyperactive. made Jockey#stop! and Publishable#stop! more failsafe. added a clear! method to Jockey. added Captain#update_services! to immediately update services from Jockey. Modified Paths: -------------- trunk/archipelago/lib/archipelago/disco.rb trunk/archipelago/lib/archipelago/pirate.rb trunk/archipelago/tests/pirate_test.rb trunk/hyperactive/README trunk/hyperactive/Rakefile trunk/hyperactive/tests/record_test.rb trunk/hyperactive/tests/test_helper.rb trunk/hyperactive/tests/tree_benchmark.rb trunk/hyperactive/tests/tree_test.rb Modified: trunk/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-27 10:24:02 UTC (rev 60) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-27 11:08:34 UTC (rev 61) @@ -143,17 +143,24 @@ # We are always valid if we are able to reply. # def valid? - true + if defined?(@valid) + @valid + else + @valid = true + end end # # Stops the publishing of this Publishable. # def stop! - if defined?(Archipelago::Disco::MC) && @jockey == Archipelago::Disco::MC - @jockey.unpublish(self.service_id) - else - @jockey.stop! + if valid? + @valid = false + if defined?(Archipelago::Disco::MC) && @jockey == Archipelago::Disco::MC + @jockey.unpublish(self.service_id) + else + @jockey.stop! + end end end @@ -347,6 +354,7 @@ # or if not given THRIFTY_REPLYING. Otherwise will reply with multicasts. # def initialize(options = {}) + @valid = true @remote_services = ServiceLocker.new @local_services = ServiceLocker.new @subscribed_services = Set.new @@ -415,20 +423,31 @@ end # + # Clears our local and remote services. + # + def clear! + @local_services = ServiceLocker.new + @remote_services = ServiceLocker.new + end + + # # Stops all the threads in this instance. # def stop! - @local_services.each do |service_id, service_description| - self.unpublish(service_id) + if @valid + @valid = false + @local_services.each do |service_id, service_description| + self.unpublish(service_id) + end + @listener_thread.kill + @unilistener_thread.kill + @validator_thread.kill + @picker_thread.kill + until @outgoing.empty? + sleep(0.01) + end + @shouter_thread.kill end - @listener_thread.kill - @unilistener_thread.kill - @validator_thread.kill - @picker_thread.kill - until @outgoing.empty? - sleep(0.01) - end - @shouter_thread.kill end # Modified: trunk/archipelago/lib/archipelago/pirate.rb =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb 2006-11-27 10:24:02 UTC (rev 60) +++ trunk/archipelago/lib/archipelago/pirate.rb 2006-11-27 11:08:34 UTC (rev 61) @@ -237,6 +237,15 @@ end end + # + # Does an immediate update of our service lists. + # + def update_services! + @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) + evaluate_in_chests + @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) + end + private # @@ -303,9 +312,7 @@ # +initial+ and +maximum+ seconds. # def start_service_updater(initial, maximum) - @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) - evaluate_in_chests - @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) + update_services! @service_update_thread = Thread.start do standoff = initial loop do @@ -313,9 +320,7 @@ sleep(standoff) standoff *= 2 standoff = maximum if standoff > maximum - @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) - evaluate_in_chests - @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) + update_services! rescue Exception => e puts e pp e.backtrace @@ -323,6 +328,7 @@ end end end + end end Modified: trunk/archipelago/tests/pirate_test.rb =================================================================== --- trunk/archipelago/tests/pirate_test.rb 2006-11-27 10:24:02 UTC (rev 60) +++ trunk/archipelago/tests/pirate_test.rb 2006-11-27 11:08:34 UTC (rev 61) @@ -13,10 +13,11 @@ @c2.publish! @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny.db"))) @tm.publish! - assert_within(2) do - Set.new(@p.chests.keys) == Set.new([@c.service_id, @c2.service_id]) + @p.update_services! + assert_within(10) do + @p.chests.keys.sort == [@c.service_id, @c2.service_id].sort end - assert_within(2) do + assert_within(10) do @p.trannies.keys == [@tm.service_id] end end Modified: trunk/hyperactive/README =================================================================== --- trunk/hyperactive/README 2006-11-27 10:24:02 UTC (rev 60) +++ trunk/hyperactive/README 2006-11-27 11:08:34 UTC (rev 61) @@ -2,3 +2,32 @@ It uses archipelago for persistence, and is meaningful only in an environment where the server process doesnt restart at each request. This means that cgi environment is not really an option. +== Dependencies: + +Hyperactive::Tree:: RBTree: http://raa.ruby-lang.org/project/ruby-rbtree + +== Sub packages: + +Hyperactive::Record:: The base class itself, providing you with cached selectors and rejectors, along with cached finders for any number of attributes. +Hyperactive::Tree:: A collection class that contains any number of Hyperactive::Records in a tree structure. Scales logarithmically, so use with care. + +== Examples: + +To define a Record subclass that has the properties @active and @city that are indexed in two ways: + + class MyClass < Hyperactive::Record + select(:active_records, Proc.new do |record| + record.active == true + end) + index_by(:city) + end + +This will allow you to find all active MyClass instances by: + + MyClass.active_records + +Or finding all MyClass instances connected to Stockholm by: + + MyClass.find_by_city("Stockholm") + + Modified: trunk/hyperactive/Rakefile =================================================================== --- trunk/hyperactive/Rakefile 2006-11-27 10:24:02 UTC (rev 60) +++ trunk/hyperactive/Rakefile 2006-11-27 11:08:34 UTC (rev 61) @@ -8,19 +8,20 @@ spec = Gem::Specification.new do |s| s.platform = Gem::Platform::RUBY - s.name = "hyperactive_record" + s.name = "hyperactive" s.version = "0.1.0" s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A base class for persistent objects that uses archipelago for persistence. Useful for Ruby on Rails models for example." - s.files = FileList['lib/*.rb', 'test/*', 'GPL-2'].to_a + s.files = FileList['lib/hyperactive.rb', 'lib/hyperactive/*.rb', 'tests/*', 'GPL-2'].to_a s.require_path = "lib" - s.autorequire = "hyperactive_record" + s.autorequire = "hyperactive" s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') s.has_rdoc = true s.rdoc_options << '--line-numbers' s.rdoc_options << '--inline-source' s.extra_rdoc_files = ["README"] + s.add_dependency('archipelago', '>= 0.2.0') end Modified: trunk/hyperactive/tests/record_test.rb =================================================================== --- trunk/hyperactive/tests/record_test.rb 2006-11-27 10:24:02 UTC (rev 60) +++ trunk/hyperactive/tests/record_test.rb 2006-11-27 11:08:34 UTC (rev 61) @@ -37,8 +37,9 @@ @tm.publish! Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) + Hyperactive::CAPTAIN.update_services! assert_within(10) do - Set.new(Hyperactive::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) + Hyperactive::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort end assert_within(10) do Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] @@ -58,6 +59,14 @@ @c2.persistence_provider.unlink @tm.stop! @tm.persistence_provider.unlink + Archipelago::Disco::MC.clear! + Hyperactive::CAPTAIN.update_services! + assert_within(10) do + Hyperactive::CAPTAIN.chests.empty? + end + assert_within(10) do + Hyperactive::CAPTAIN.trannies.empty? + end end def test_index_find Modified: trunk/hyperactive/tests/test_helper.rb =================================================================== --- trunk/hyperactive/tests/test_helper.rb 2006-11-27 10:24:02 UTC (rev 60) +++ trunk/hyperactive/tests/test_helper.rb 2006-11-27 11:08:34 UTC (rev 61) @@ -1,16 +1,66 @@ -SCRIPT_LINES__ = {} - home = File.expand_path(File.dirname(__FILE__)) $: << File.join(home, "..", "lib") require 'hyperactive' -require 'ruby-debug' +require 'pp' +require 'test/unit' +require 'benchmark' +require 'drb' +require 'archipelago' +require 'socket' +require 'ipaddr' +require 'thread' DRb.start_service -MC_ENABLED = true -require 'pp' -require 'test/unit' -require 'benchmark' -require '../archipelago/tests/test_helper' +class TestTransaction + def join(o) + end + def state + :active + end +end + +class TestManager < Archipelago::Tranny::Manager + attr_reader :persistence_provider + def log_error(e) + puts e + pp e.backtrace + end +end + +class TestJockey < Archipelago::Disco::Jockey + attr_reader :remote_services, :local_services +end + +class TestChest < Archipelago::Treasure::Chest + attr_reader :persistence_provider +end + +def bm(label = "", options = {}) + n = options[:n] || 1000 + width = options[:width] || 50 + Benchmark.benchmark(" " * width + Benchmark::Tms::CAPTION, width, Benchmark::Tms::FMTSTR, "ms/call") do |b| + times = b.report("#{n}x#{label}") do + n.times do + yield + end + end + [times * 1000 / n.to_f] + end +end + +class Test::Unit::TestCase + + def assert_within(timeout, &block) + t = Time.new + rval = yield + while !rval && t > Time.new - timeout + rval = yield + sleep(0.05) + end + assert(rval) + end + +end Modified: trunk/hyperactive/tests/tree_benchmark.rb =================================================================== --- trunk/hyperactive/tests/tree_benchmark.rb 2006-11-27 10:24:02 UTC (rev 60) +++ trunk/hyperactive/tests/tree_benchmark.rb 2006-11-27 11:08:34 UTC (rev 61) @@ -32,7 +32,7 @@ def test_set_get h = Hyperactive::Tree.get_instance r = Hyperactive::Record.get_instance - hash_test("Tree", h, r, 1000) + hash_test("Tree", h, r, 100) end def test_regular_hash_set_get Modified: trunk/hyperactive/tests/tree_test.rb =================================================================== --- trunk/hyperactive/tests/tree_test.rb 2006-11-27 10:24:02 UTC (rev 60) +++ trunk/hyperactive/tests/tree_test.rb 2006-11-27 11:08:34 UTC (rev 61) @@ -11,7 +11,7 @@ end class TreeTest < Test::Unit::TestCase - + def setup @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) @c.publish! @@ -21,10 +21,11 @@ @tm.publish! Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) - assert_within(20) do - Set.new(Hyperactive::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) + Hyperactive::CAPTAIN.update_services! + assert_within(10) do + Hyperactive::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort end - assert_within(20) do + assert_within(10) do Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] end end @@ -36,6 +37,14 @@ @c2.persistence_provider.unlink @tm.stop! @tm.persistence_provider.unlink + Archipelago::Disco::MC.clear! + Hyperactive::CAPTAIN.update_services! + assert_within(10) do + Hyperactive::CAPTAIN.chests.empty? + end + assert_within(10) do + Hyperactive::CAPTAIN.trannies.empty? + end end def test_select_reject From nobody at rubyforge.org Mon Nov 27 06:09:29 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 06:09:29 -0500 (EST) Subject: [Archipelago-submits] [62] tags/release_0_2_0/: created release_0_2_0 Message-ID: <20061127110929.9321A524151D@rubyforge.org> Revision: 62 Author: zond Date: 2006-11-27 06:09:29 -0500 (Mon, 27 Nov 2006) Log Message: ----------- created release_0_2_0 Added Paths: ----------- tags/release_0_2_0/ Copied: tags/release_0_2_0 (from rev 61, trunk) From nobody at rubyforge.org Mon Nov 27 06:24:17 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 06:24:17 -0500 (EST) Subject: [Archipelago-submits] [63] trunk/hyperactive/lib: added license Message-ID: <20061127112417.962B6524156C@rubyforge.org> Revision: 63 Author: zond Date: 2006-11-27 06:24:17 -0500 (Mon, 27 Nov 2006) Log Message: ----------- added license Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/hooker.rb trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive/tree.rb trunk/hyperactive/lib/hyperactive.rb Modified: trunk/hyperactive/lib/hyperactive/hooker.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/hooker.rb 2006-11-27 11:09:29 UTC (rev 62) +++ trunk/hyperactive/lib/hyperactive/hooker.rb 2006-11-27 11:24:17 UTC (rev 63) @@ -1,4 +1,21 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + module Hyperactive # Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 11:09:29 UTC (rev 62) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 11:24:17 UTC (rev 63) @@ -1,4 +1,21 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + require 'rubygems' require 'archipelago' Modified: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-27 11:09:29 UTC (rev 62) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-27 11:24:17 UTC (rev 63) @@ -1,4 +1,21 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + require 'rubygems' require 'archipelago' require 'rbtree' Modified: trunk/hyperactive/lib/hyperactive.rb =================================================================== --- trunk/hyperactive/lib/hyperactive.rb 2006-11-27 11:09:29 UTC (rev 62) +++ trunk/hyperactive/lib/hyperactive.rb 2006-11-27 11:24:17 UTC (rev 63) @@ -1,3 +1,19 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. $: << File.dirname(__FILE__) From nobody at rubyforge.org Mon Nov 27 06:24:55 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 06:24:55 -0500 (EST) Subject: [Archipelago-submits] [64] trunk/hyperactive/Rakefile: bumped the version Message-ID: <20061127112455.30DB75241581@rubyforge.org> Revision: 64 Author: zond Date: 2006-11-27 06:24:54 -0500 (Mon, 27 Nov 2006) Log Message: ----------- bumped the version Modified Paths: -------------- trunk/hyperactive/Rakefile Modified: trunk/hyperactive/Rakefile =================================================================== --- trunk/hyperactive/Rakefile 2006-11-27 11:24:17 UTC (rev 63) +++ trunk/hyperactive/Rakefile 2006-11-27 11:24:54 UTC (rev 64) @@ -9,7 +9,7 @@ spec = Gem::Specification.new do |s| s.platform = Gem::Platform::RUBY s.name = "hyperactive" - s.version = "0.1.0" + s.version = "0.1.1" s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A base class for persistent objects that uses archipelago for persistence. Useful for Ruby on Rails models for example." From nobody at rubyforge.org Mon Nov 27 06:42:22 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 06:42:22 -0500 (EST) Subject: [Archipelago-submits] [66] trunk/hyperactive/lib/hyperactive/record.rb: added comment for index_by Message-ID: <20061127114223.00F475241587@rubyforge.org> Revision: 66 Author: zond Date: 2006-11-27 06:42:22 -0500 (Mon, 27 Nov 2006) Log Message: ----------- added comment for index_by Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 11:27:22 UTC (rev 65) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 11:42:22 UTC (rev 66) @@ -227,6 +227,12 @@ self.get_hook_array_by_class(@@save_hooks_by_class) end + # + # Create an index for this class. + # + # Will create a method find_by_#{attributes.join("_and_")} for this + # class that will return what you expect. + # def self.index_by(*attributes) attribute_key_part = IndexBuilder.get_attribute_key_part(attributes) self.class_eval < Revision: 67 Author: zond Date: 2006-11-27 06:46:06 -0500 (Mon, 27 Nov 2006) Log Message: ----------- more README Modified Paths: -------------- trunk/hyperactive/README Modified: trunk/hyperactive/README =================================================================== --- trunk/hyperactive/README 2006-11-27 11:42:22 UTC (rev 66) +++ trunk/hyperactive/README 2006-11-27 11:46:06 UTC (rev 67) @@ -16,6 +16,7 @@ To define a Record subclass that has the properties @active and @city that are indexed in two ways: class MyClass < Hyperactive::Record + attr_accessor :active, :city select(:active_records, Proc.new do |record| record.active == true end) From nobody at rubyforge.org Mon Nov 27 06:27:23 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 06:27:23 -0500 (EST) Subject: [Archipelago-submits] [65] tags: made release_0_2_1 Message-ID: <20061127112723.5818C524158A@rubyforge.org> Revision: 65 Author: zond Date: 2006-11-27 06:27:22 -0500 (Mon, 27 Nov 2006) Log Message: ----------- made release_0_2_1 Added Paths: ----------- tags/release_0_2_1/ tags/release_0_2_1/hyperactive/Rakefile tags/release_0_2_1/hyperactive/lib/hyperactive/hooker.rb tags/release_0_2_1/hyperactive/lib/hyperactive/record.rb tags/release_0_2_1/hyperactive/lib/hyperactive/tree.rb tags/release_0_2_1/hyperactive/lib/hyperactive.rb Removed Paths: ------------- tags/release_0_2_1/hyperactive/Rakefile tags/release_0_2_1/hyperactive/lib/hyperactive/hooker.rb tags/release_0_2_1/hyperactive/lib/hyperactive/record.rb tags/release_0_2_1/hyperactive/lib/hyperactive/tree.rb tags/release_0_2_1/hyperactive/lib/hyperactive.rb Copied: tags/release_0_2_1 (from rev 61, trunk) Deleted: tags/release_0_2_1/hyperactive/Rakefile =================================================================== --- trunk/hyperactive/Rakefile 2006-11-27 11:08:34 UTC (rev 61) +++ tags/release_0_2_1/hyperactive/Rakefile 2006-11-27 11:27:22 UTC (rev 65) @@ -1,60 +0,0 @@ - -require 'rake' -require 'rake/testtask' -require 'rubygems' -Gem::manage_gems -require 'rake/gempackagetask' - - -spec = Gem::Specification.new do |s| - s.platform = Gem::Platform::RUBY - s.name = "hyperactive" - s.version = "0.1.0" - s.author = "Martin Kihlgren" - s.email = "zond at troja dot ath dot cx" - s.summary = "A base class for persistent objects that uses archipelago for persistence. Useful for Ruby on Rails models for example." - s.files = FileList['lib/hyperactive.rb', 'lib/hyperactive/*.rb', 'tests/*', 'GPL-2'].to_a - s.require_path = "lib" - s.autorequire = "hyperactive" - s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') - s.has_rdoc = true - s.rdoc_options << '--line-numbers' - s.rdoc_options << '--inline-source' - s.extra_rdoc_files = ["README"] - s.add_dependency('archipelago', '>= 0.2.0') -end - - -SOURCE_FILES = FileList.new do |fl| - [ "lib", "tests" ].each do |dir| - fl.include "#{dir}/**/*" - end - fl.include "Rakefile" -end - -Rake::GemPackageTask.new(spec) do |pkg| - pkg.need_tar = true -end - -task :default => [:units] do -end - -desc "Run all tests" -Rake::TestTask.new(:units) do |t| - t.pattern = 'tests/*_test.rb' - t.verbose = true - t.warning = true -end - -desc "Run all benchmarks" -Rake::TestTask.new(:bench) do |t| - t.pattern = 'tests/*_benchmark.rb' - t.verbose = true - t.warning = true -end - -desc "Package a gem from the source" -task :gem => "pkg/#{spec.name}-#{spec.version}.gem" do - puts "generated latest version" -end - Copied: tags/release_0_2_1/hyperactive/Rakefile (from rev 64, trunk/hyperactive/Rakefile) =================================================================== --- tags/release_0_2_1/hyperactive/Rakefile (rev 0) +++ tags/release_0_2_1/hyperactive/Rakefile 2006-11-27 11:27:22 UTC (rev 65) @@ -0,0 +1,60 @@ + +require 'rake' +require 'rake/testtask' +require 'rubygems' +Gem::manage_gems +require 'rake/gempackagetask' + + +spec = Gem::Specification.new do |s| + s.platform = Gem::Platform::RUBY + s.name = "hyperactive" + s.version = "0.1.1" + s.author = "Martin Kihlgren" + s.email = "zond at troja dot ath dot cx" + s.summary = "A base class for persistent objects that uses archipelago for persistence. Useful for Ruby on Rails models for example." + s.files = FileList['lib/hyperactive.rb', 'lib/hyperactive/*.rb', 'tests/*', 'GPL-2'].to_a + s.require_path = "lib" + s.autorequire = "hyperactive" + s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') + s.has_rdoc = true + s.rdoc_options << '--line-numbers' + s.rdoc_options << '--inline-source' + s.extra_rdoc_files = ["README"] + s.add_dependency('archipelago', '>= 0.2.0') +end + + +SOURCE_FILES = FileList.new do |fl| + [ "lib", "tests" ].each do |dir| + fl.include "#{dir}/**/*" + end + fl.include "Rakefile" +end + +Rake::GemPackageTask.new(spec) do |pkg| + pkg.need_tar = true +end + +task :default => [:units] do +end + +desc "Run all tests" +Rake::TestTask.new(:units) do |t| + t.pattern = 'tests/*_test.rb' + t.verbose = true + t.warning = true +end + +desc "Run all benchmarks" +Rake::TestTask.new(:bench) do |t| + t.pattern = 'tests/*_benchmark.rb' + t.verbose = true + t.warning = true +end + +desc "Package a gem from the source" +task :gem => "pkg/#{spec.name}-#{spec.version}.gem" do + puts "generated latest version" +end + Deleted: tags/release_0_2_1/hyperactive/lib/hyperactive/hooker.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/hooker.rb 2006-11-27 11:08:34 UTC (rev 61) +++ tags/release_0_2_1/hyperactive/lib/hyperactive/hooker.rb 2006-11-27 11:27:22 UTC (rev 65) @@ -1,63 +0,0 @@ - -module Hyperactive - - # - # A utility method to simplify using around-hooks for any block. - # - module Hooker - - # - # Will call each hook in +hooks+ with a block that - # calls the rest of the +hooks+ with +argument+ as argument - # and after that yields to the given +block+. - # - # This will allow you wrap any method call in a dynamic set - # of other methods. - # - # For more examples, see Hyperactive::Record. - # - # Example: - # class ImportantStuff - # attr_accessor :timestamp - # def is_valid? - # # do nifty stuff - # end - # def notify_someone_that_i_am_changed!(message) - # # do even more nifty stuff - # end - # def validate_and_notify(message) - # raise "i am invalid!" unless is_valid? - # yield - # notify_someone_that_i_am_changed!(message) - # end - # def do_save - # # save me in persistent storage - # end - # end - # class ImportantHandler - # def update_timestamp(important_stuff_instance) - # important_stuff_instance.timestamp = Time.now - # Hyperactive::Hooker.call_with_hooks("changed at #{Time.now}", - # important_stuff_instance.method(:validate_and_notify)) do - # important_stuff_instance.do_save - # end - # end - # end - # - def call_with_hooks(argument, *hooks, &block) - if hooks.empty? - yield - else - first = hooks.first - rest = hooks[1..-1] - first.call(argument) do - call_with_hooks(argument, *rest, &block) - end - end - end - - module_function :call_with_hooks - - end - -end Copied: tags/release_0_2_1/hyperactive/lib/hyperactive/hooker.rb (from rev 63, trunk/hyperactive/lib/hyperactive/hooker.rb) =================================================================== --- tags/release_0_2_1/hyperactive/lib/hyperactive/hooker.rb (rev 0) +++ tags/release_0_2_1/hyperactive/lib/hyperactive/hooker.rb 2006-11-27 11:27:22 UTC (rev 65) @@ -0,0 +1,80 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + + +module Hyperactive + + # + # A utility method to simplify using around-hooks for any block. + # + module Hooker + + # + # Will call each hook in +hooks+ with a block that + # calls the rest of the +hooks+ with +argument+ as argument + # and after that yields to the given +block+. + # + # This will allow you wrap any method call in a dynamic set + # of other methods. + # + # For more examples, see Hyperactive::Record. + # + # Example: + # class ImportantStuff + # attr_accessor :timestamp + # def is_valid? + # # do nifty stuff + # end + # def notify_someone_that_i_am_changed!(message) + # # do even more nifty stuff + # end + # def validate_and_notify(message) + # raise "i am invalid!" unless is_valid? + # yield + # notify_someone_that_i_am_changed!(message) + # end + # def do_save + # # save me in persistent storage + # end + # end + # class ImportantHandler + # def update_timestamp(important_stuff_instance) + # important_stuff_instance.timestamp = Time.now + # Hyperactive::Hooker.call_with_hooks("changed at #{Time.now}", + # important_stuff_instance.method(:validate_and_notify)) do + # important_stuff_instance.do_save + # end + # end + # end + # + def call_with_hooks(argument, *hooks, &block) + if hooks.empty? + yield + else + first = hooks.first + rest = hooks[1..-1] + first.call(argument) do + call_with_hooks(argument, *rest, &block) + end + end + end + + module_function :call_with_hooks + + end + +end Deleted: tags/release_0_2_1/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 11:08:34 UTC (rev 61) +++ tags/release_0_2_1/hyperactive/lib/hyperactive/record.rb 2006-11-27 11:27:22 UTC (rev 65) @@ -1,358 +0,0 @@ - -require 'rubygems' -require 'archipelago' - -# -# A utility module to provide the functionality required for example -# if you want to use archipelago in a Ruby on Rails project *hint hint*. -# -module Hyperactive - - # - # The default database connector. - # - CAPTAIN = Archipelago::Pirate::Captain.new - - # - # A tiny callable class that saves stuff in - # indexes depending on certain attributes. - # - class IndexBuilder - # - # Get the first part of the key, that depends on the +attributes+. - # - def self.get_attribute_key_part(attributes) - "Hyperactive::IndexBuilder::#{attributes.join(",")}" - end - # - # Get the last part of the key, that depends on the +values+. - # - def self.get_value_key_part(values) - "#{values.join(",")}" - end - # - # Initialize an IndexBuilder giving it an array of +attributes+ - # that will be indexed. - # - def initialize(attributes) - @attributes = attributes - end - # - # Get the Tree for the given +record+. - # - def get_tree_for(record) - values = @attributes.collect do |att| - if record.respond_to?(att) - record.send(att) - else - nil - end - end - key = "#{self.class.get_attribute_key_part(@attributes)}::#{self.class.get_value_key_part(values)}" - return CAPTAIN[key] ||= Tree.get_instance - end - # - # Call this IndexBuilder and pass it a +block+. - # - # If the +argument+ is an Array then we know we are a save hook, - # otherwise we are a destroy hook. - # - def call(argument, &block) - yield - - # - # If the argument is an Array (of old value, new value) - # then we are a save hook, otherwise a destroy hook. - # - if Array === argument - record = argument.last - old_record = argument.first - get_tree_for(old_record).delete(record.record_id) - get_tree_for(record)[record.record_id] = record - else - record = argument - get_tree_for(record).delete(record.record_id) - end - end - end - - # - # A tiny callable class that saves stuff inside containers - # if they match certain criteria. - # - class MatchSaver - # - # Initialize this MatchSaver with a +key+, a callable +matcher+ - # and a +mode+ (:select, :reject, :delete_if_match or :delete_unless_match). - # - def initialize(key, matcher, mode) - @key = key - @matcher = matcher - @mode = mode - end - # - # Depending on @mode and return value of @matcher.call - # may save record in the Hash-like container named @key in - # the main database after having yielded to +block+. - # - def call(argument, &block) - yield - - record = argument - record = argument.last if Array === argument - - case @mode - when :select - if @matcher.call(record) - CAPTAIN[@key][record.record_id] = record - else - CAPTAIN[@key].delete(record.record_id) - end - when :reject - if @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) - else - CAPTAIN[@key][record.record_id] = record - end - when :delete_if_match - if @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) - end - when :delete_unless_match - unless @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) - end - end - end - end - - # - # A convenient base class to inherit when you want the basic utility methods - # provided by for example ActiveRecord::Base *hint hint*. - # - # NB: After an instance is created, it will actually return a copy within your local machine - # which is not what is usually the case. Every other time you fetch it using a select or other - # method you will instead receive a proxy object to the database. This means that nothing you - # do to it at that point will be persistent or even necessarily have a defined result. - # Therefore: do not use MyRecordSubclass.new to MyRecordSubclass#initialize objects, - # instead use MyRecordSubclass.get_instance, since it will return a fresh proxy object - # instead of the devious original: - # my_instance = MyRecordSubclass.get_instance(*the_same_arguments_as_to_initialize) - # - class Record - - @@create_hooks_by_class = {} - @@destroy_hooks_by_class = {} - @@save_hooks_by_class = {} - - # - # The host we are running on. - # - HOST = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" - - # - # Call this if you want to change the default database connector - # to something else. - # - def self.setup(options = {}) - CAPTAIN.setup(options[:pirate_options]) - end - - # - # Return our create_hooks, which can then be treated as any old Array. - # - # These must be callable objects with an arity of 1 - # that will be sent the instance about to be created (initial - # insertion into the database system) that take a block argument. - # - # The block argument will be a Proc that actually injects the - # instance into the database system. - # - # Use this to preprocess, validate and/or postprocess your - # instances upon creation. - # - def self.create_hooks - self.get_hook_array_by_class(@@create_hooks_by_class) - end - - # - # Return our destroy_hooks, which can then be treated as any old Array. - # - # These must be callable objects with an arity of 1 - # that will be sent the instance about to be destroyed (removal - # from the database system) that take a block argument. - # - # The block argument will be a Proc that actually removes the - # instance from the database system. - # - # Use this to preprocess, validate and/or postprocess your - # instances upon destruction. - # - def self.destroy_hooks - self.get_hook_array_by_class(@@destroy_hooks_by_class) - end - - # - # Return our save_hooks, which can then be treated as any old Array. - # - # These must be callable objects with an arity of 1 - # that will be sent [the old version, the new version] of the - # instance about to be saved (storage into the database system) - # along with a block argument. - # - # The block argument will be a Proc that actually saves the - # instance into the database system. - # - # Use this to preprocess, validate and/or postprocess your - # instances upon saving. - # - def self.save_hooks - self.get_hook_array_by_class(@@save_hooks_by_class) - end - - def self.index_by(*attributes) - attribute_key_part = IndexBuilder.get_attribute_key_part(attributes) - self.class_eval <self as - # key. If self doesnt exist in the +hash+ - # it will recurse by calling the same method in the - # superclass until it has been called in Hyperactive::Record. - # - def self.get_hook_array_by_class(hash) - return hash[self] if hash.include?(self) - - if self == Record - hash[self] = [] - return self.get_hook_array_by_class(hash) - else - hash[self] = self.superclass.get_hook_array_by_class(hash).clone - return self.get_hook_array_by_class(hash) - end - end - - end -end Copied: tags/release_0_2_1/hyperactive/lib/hyperactive/record.rb (from rev 63, trunk/hyperactive/lib/hyperactive/record.rb) =================================================================== --- tags/release_0_2_1/hyperactive/lib/hyperactive/record.rb (rev 0) +++ tags/release_0_2_1/hyperactive/lib/hyperactive/record.rb 2006-11-27 11:27:22 UTC (rev 65) @@ -0,0 +1,375 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + + +require 'rubygems' +require 'archipelago' + +# +# A utility module to provide the functionality required for example +# if you want to use archipelago in a Ruby on Rails project *hint hint*. +# +module Hyperactive + + # + # The default database connector. + # + CAPTAIN = Archipelago::Pirate::Captain.new + + # + # A tiny callable class that saves stuff in + # indexes depending on certain attributes. + # + class IndexBuilder + # + # Get the first part of the key, that depends on the +attributes+. + # + def self.get_attribute_key_part(attributes) + "Hyperactive::IndexBuilder::#{attributes.join(",")}" + end + # + # Get the last part of the key, that depends on the +values+. + # + def self.get_value_key_part(values) + "#{values.join(",")}" + end + # + # Initialize an IndexBuilder giving it an array of +attributes+ + # that will be indexed. + # + def initialize(attributes) + @attributes = attributes + end + # + # Get the Tree for the given +record+. + # + def get_tree_for(record) + values = @attributes.collect do |att| + if record.respond_to?(att) + record.send(att) + else + nil + end + end + key = "#{self.class.get_attribute_key_part(@attributes)}::#{self.class.get_value_key_part(values)}" + return CAPTAIN[key] ||= Tree.get_instance + end + # + # Call this IndexBuilder and pass it a +block+. + # + # If the +argument+ is an Array then we know we are a save hook, + # otherwise we are a destroy hook. + # + def call(argument, &block) + yield + + # + # If the argument is an Array (of old value, new value) + # then we are a save hook, otherwise a destroy hook. + # + if Array === argument + record = argument.last + old_record = argument.first + get_tree_for(old_record).delete(record.record_id) + get_tree_for(record)[record.record_id] = record + else + record = argument + get_tree_for(record).delete(record.record_id) + end + end + end + + # + # A tiny callable class that saves stuff inside containers + # if they match certain criteria. + # + class MatchSaver + # + # Initialize this MatchSaver with a +key+, a callable +matcher+ + # and a +mode+ (:select, :reject, :delete_if_match or :delete_unless_match). + # + def initialize(key, matcher, mode) + @key = key + @matcher = matcher + @mode = mode + end + # + # Depending on @mode and return value of @matcher.call + # may save record in the Hash-like container named @key in + # the main database after having yielded to +block+. + # + def call(argument, &block) + yield + + record = argument + record = argument.last if Array === argument + + case @mode + when :select + if @matcher.call(record) + CAPTAIN[@key][record.record_id] = record + else + CAPTAIN[@key].delete(record.record_id) + end + when :reject + if @matcher.call(record) + CAPTAIN[@key].delete(record.record_id) + else + CAPTAIN[@key][record.record_id] = record + end + when :delete_if_match + if @matcher.call(record) + CAPTAIN[@key].delete(record.record_id) + end + when :delete_unless_match + unless @matcher.call(record) + CAPTAIN[@key].delete(record.record_id) + end + end + end + end + + # + # A convenient base class to inherit when you want the basic utility methods + # provided by for example ActiveRecord::Base *hint hint*. + # + # NB: After an instance is created, it will actually return a copy within your local machine + # which is not what is usually the case. Every other time you fetch it using a select or other + # method you will instead receive a proxy object to the database. This means that nothing you + # do to it at that point will be persistent or even necessarily have a defined result. + # Therefore: do not use MyRecordSubclass.new to MyRecordSubclass#initialize objects, + # instead use MyRecordSubclass.get_instance, since it will return a fresh proxy object + # instead of the devious original: + # my_instance = MyRecordSubclass.get_instance(*the_same_arguments_as_to_initialize) + # + class Record + + @@create_hooks_by_class = {} + @@destroy_hooks_by_class = {} + @@save_hooks_by_class = {} + + # + # The host we are running on. + # + HOST = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" + + # + # Call this if you want to change the default database connector + # to something else. + # + def self.setup(options = {}) + CAPTAIN.setup(options[:pirate_options]) + end + + # + # Return our create_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent the instance about to be created (initial + # insertion into the database system) that take a block argument. + # + # The block argument will be a Proc that actually injects the + # instance into the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon creation. + # + def self.create_hooks + self.get_hook_array_by_class(@@create_hooks_by_class) + end + + # + # Return our destroy_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent the instance about to be destroyed (removal + # from the database system) that take a block argument. + # + # The block argument will be a Proc that actually removes the + # instance from the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon destruction. + # + def self.destroy_hooks + self.get_hook_array_by_class(@@destroy_hooks_by_class) + end + + # + # Return our save_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent [the old version, the new version] of the + # instance about to be saved (storage into the database system) + # along with a block argument. + # + # The block argument will be a Proc that actually saves the + # instance into the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon saving. + # + def self.save_hooks + self.get_hook_array_by_class(@@save_hooks_by_class) + end + + def self.index_by(*attributes) + attribute_key_part = IndexBuilder.get_attribute_key_part(attributes) + self.class_eval <self as + # key. If self doesnt exist in the +hash+ + # it will recurse by calling the same method in the + # superclass until it has been called in Hyperactive::Record. + # + def self.get_hook_array_by_class(hash) + return hash[self] if hash.include?(self) + + if self == Record + hash[self] = [] + return self.get_hook_array_by_class(hash) + else + hash[self] = self.superclass.get_hook_array_by_class(hash).clone + return self.get_hook_array_by_class(hash) + end + end + + end +end Deleted: tags/release_0_2_1/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-27 11:08:34 UTC (rev 61) +++ tags/release_0_2_1/hyperactive/lib/hyperactive/tree.rb 2006-11-27 11:27:22 UTC (rev 65) @@ -1,216 +0,0 @@ - -require 'rubygems' -require 'archipelago' -require 'rbtree' - -module Hyperactive - - # - # A class suitable for storing large and often-changing datasets in - # an Archipelago environment. - # - # Is constructed like a set of nested Hashes that automatically create - # new children on demand, and will thusly only have to check the path from - # the root node to the leaf for changes when method calls return (see Archipelago::Treasure::Chest) - # and will only have to actually store into database the leaf itself if it - # has changed. - # - class Tree < Record - - attr_accessor :elements, :subtrees - - WIDTH = 1 << 3 - - # - # Dont call this! Call Tree.get_instance(options) instead! - # - def initialize(options = {}) - @width = options[:width] || WIDTH - @elements = {} - @subtrees = nil - end - - # - # Deletes +key+ in this Tree. - # - def delete(key) - if @elements - @elements.delete(key) - else - subtree_for(key).delete(key) - end - end - - # - # Returns the size of this Tree. - # - def size - if @elements - @elements.size - else - @subtrees.t_collect do |tree_id, tree| - tree.size - end.inject(0) do |sum, size| - sum + size - end - end - end - - # - # Returns all keys and values returning true for +callable+.call(key, value) in this Tree. - # - def select(callable) - if @elements - @elements.select do |k,v| - callable.call(k,v) - end - else - @subtrees.t_collect do |tree_id, tree| - tree.select(callable) - end.inject([]) do |sum, match| - sum + match - end - end - end - - # - # Returns all keys and values returning false for +callable+.call(key, value) in this Tree. - # - def reject(callable) - if @elements - @elements.reject do |k,v| - callable.call(k,v) - end - else - @subtrees.t_collect do |tree_id, tree| - tree.reject(callable) - end.inject([]) do |sum, match| - sum + match - end - end - end - - # - # Puts +value+ under +key+ in this Tree. - # - def []=(key, value) - if @elements - if @elements.size < @width - @elements[key] = value - else - split! - subtree_for(key)[key] = value - end - else - subtree_for(key)[key] = value - end - return value - end - - # - # Returns the value for +key+ in this Tree. - # - def [](key) - if @elements - return @elements[key] - else - return subtree_for(key)[key] - end - end - - # - # Returns an Array containing +callable+.call(key, value) from all values in this Tree. - # - def collect(callable) - rval = [] - self.each(Proc.new do |k,v| - rval << callable.call(k,v) - end) - return rval - end - - # - # Does +callable+.call(key, value) on all values in this Tree. - # - def each(callable) - if @elements - @elements.each do |key, value| - callable.call(key, value) - end - else - @subtrees.t_each do |tree_id, tree| - tree.each(callable) - end - nil - end - end - - # - # Clear everything from this Tree and destroy it. - # - def destroy - if @elements - @elements.each do |key, value| - value.destroy if value.respond_to?(:destroy) - end - else - @subtrees.each do |tree_id, tree| - tree.destroy - end - end - freeze - super - end - - # - # Clear everything from this Tree. - # - def clear - unless @elements - @subtrees.each do |tree_id, tree| - tree.clear - end - @subtrees = nil - end - @elements = {} - end - - private - - # - # Finds the subtree responsible for +key+. - # - # Does it in this ugly way cause the nice way of just doing modulo gave - # really odd results since all hashes seem to give some modulo values - # a lot more often than expected. - # - def subtree_for(key) - key_id = Digest::SHA1.new("#{key.hash}#{self.record_id}").to_s - @subtrees.each do |tree_id, tree| - return tree if tree_id > key_id - end - return @subtrees.values.first - end - - # - # Split this Tree by creating @subtrees, - # then putting all @elements in them and then emptying @elements. - # - def split! - raise "Cant split twice!" unless @elements - - @subtrees = RBTree.new - @subtrees.extend(Archipelago::Current::ThreadedCollection) - step = (1 << 160) / @width - 0.upto(@width - 1) do |n| - @subtrees["%x" % (n * step)] = Tree.get_instance(:width => @width) - end - @elements.each do |key, value| - subtree_for(key)[key] = value - end - @elements = nil - end - - end - -end Copied: tags/release_0_2_1/hyperactive/lib/hyperactive/tree.rb (from rev 63, trunk/hyperactive/lib/hyperactive/tree.rb) =================================================================== --- tags/release_0_2_1/hyperactive/lib/hyperactive/tree.rb (rev 0) +++ tags/release_0_2_1/hyperactive/lib/hyperactive/tree.rb 2006-11-27 11:27:22 UTC (rev 65) @@ -0,0 +1,233 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + + +require 'rubygems' +require 'archipelago' +require 'rbtree' + +module Hyperactive + + # + # A class suitable for storing large and often-changing datasets in + # an Archipelago environment. + # + # Is constructed like a set of nested Hashes that automatically create + # new children on demand, and will thusly only have to check the path from + # the root node to the leaf for changes when method calls return (see Archipelago::Treasure::Chest) + # and will only have to actually store into database the leaf itself if it + # has changed. + # + class Tree < Record + + attr_accessor :elements, :subtrees + + WIDTH = 1 << 3 + + # + # Dont call this! Call Tree.get_instance(options) instead! + # + def initialize(options = {}) + @width = options[:width] || WIDTH + @elements = {} + @subtrees = nil + end + + # + # Deletes +key+ in this Tree. + # + def delete(key) + if @elements + @elements.delete(key) + else + subtree_for(key).delete(key) + end + end + + # + # Returns the size of this Tree. + # + def size + if @elements + @elements.size + else + @subtrees.t_collect do |tree_id, tree| + tree.size + end.inject(0) do |sum, size| + sum + size + end + end + end + + # + # Returns all keys and values returning true for +callable+.call(key, value) in this Tree. + # + def select(callable) + if @elements + @elements.select do |k,v| + callable.call(k,v) + end + else + @subtrees.t_collect do |tree_id, tree| + tree.select(callable) + end.inject([]) do |sum, match| + sum + match + end + end + end + + # + # Returns all keys and values returning false for +callable+.call(key, value) in this Tree. + # + def reject(callable) + if @elements + @elements.reject do |k,v| + callable.call(k,v) + end + else + @subtrees.t_collect do |tree_id, tree| + tree.reject(callable) + end.inject([]) do |sum, match| + sum + match + end + end + end + + # + # Puts +value+ under +key+ in this Tree. + # + def []=(key, value) + if @elements + if @elements.size < @width + @elements[key] = value + else + split! + subtree_for(key)[key] = value + end + else + subtree_for(key)[key] = value + end + return value + end + + # + # Returns the value for +key+ in this Tree. + # + def [](key) + if @elements + return @elements[key] + else + return subtree_for(key)[key] + end + end + + # + # Returns an Array containing +callable+.call(key, value) from all values in this Tree. + # + def collect(callable) + rval = [] + self.each(Proc.new do |k,v| + rval << callable.call(k,v) + end) + return rval + end + + # + # Does +callable+.call(key, value) on all values in this Tree. + # + def each(callable) + if @elements + @elements.each do |key, value| + callable.call(key, value) + end + else + @subtrees.t_each do |tree_id, tree| + tree.each(callable) + end + nil + end + end + + # + # Clear everything from this Tree and destroy it. + # + def destroy + if @elements + @elements.each do |key, value| + value.destroy if value.respond_to?(:destroy) + end + else + @subtrees.each do |tree_id, tree| + tree.destroy + end + end + freeze + super + end + + # + # Clear everything from this Tree. + # + def clear + unless @elements + @subtrees.each do |tree_id, tree| + tree.clear + end + @subtrees = nil + end + @elements = {} + end + + private + + # + # Finds the subtree responsible for +key+. + # + # Does it in this ugly way cause the nice way of just doing modulo gave + # really odd results since all hashes seem to give some modulo values + # a lot more often than expected. + # + def subtree_for(key) + key_id = Digest::SHA1.new("#{key.hash}#{self.record_id}").to_s + @subtrees.each do |tree_id, tree| + return tree if tree_id > key_id + end + return @subtrees.values.first + end + + # + # Split this Tree by creating @subtrees, + # then putting all @elements in them and then emptying @elements. + # + def split! + raise "Cant split twice!" unless @elements + + @subtrees = RBTree.new + @subtrees.extend(Archipelago::Current::ThreadedCollection) + step = (1 << 160) / @width + 0.upto(@width - 1) do |n| + @subtrees["%x" % (n * step)] = Tree.get_instance(:width => @width) + end + @elements.each do |key, value| + subtree_for(key)[key] = value + end + @elements = nil + end + + end + +end Deleted: tags/release_0_2_1/hyperactive/lib/hyperactive.rb =================================================================== --- trunk/hyperactive/lib/hyperactive.rb 2006-11-27 11:08:34 UTC (rev 61) +++ tags/release_0_2_1/hyperactive/lib/hyperactive.rb 2006-11-27 11:27:22 UTC (rev 65) @@ -1,6 +0,0 @@ - -$: << File.dirname(__FILE__) - -require 'hyperactive/hooker' -require 'hyperactive/record' -require 'hyperactive/tree' Copied: tags/release_0_2_1/hyperactive/lib/hyperactive.rb (from rev 63, trunk/hyperactive/lib/hyperactive.rb) =================================================================== --- tags/release_0_2_1/hyperactive/lib/hyperactive.rb (rev 0) +++ tags/release_0_2_1/hyperactive/lib/hyperactive.rb 2006-11-27 11:27:22 UTC (rev 65) @@ -0,0 +1,22 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +$: << File.dirname(__FILE__) + +require 'hyperactive/hooker' +require 'hyperactive/record' +require 'hyperactive/tree' From nobody at rubyforge.org Mon Nov 27 07:43:41 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 07:43:41 -0500 (EST) Subject: [Archipelago-submits] [68] trunk/hyperactive: made tree a module of its own Message-ID: <20061127124341.979655241589@rubyforge.org> Revision: 68 Author: zond Date: 2006-11-27 07:43:40 -0500 (Mon, 27 Nov 2006) Log Message: ----------- made tree a module of its own Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive/tree.rb trunk/hyperactive/tests/tree_benchmark.rb trunk/hyperactive/tests/tree_test.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 11:46:06 UTC (rev 67) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 12:43:40 UTC (rev 68) @@ -66,7 +66,7 @@ end end key = "#{self.class.get_attribute_key_part(@attributes)}::#{self.class.get_value_key_part(values)}" - return CAPTAIN[key] ||= Tree.get_instance + return CAPTAIN[key] ||= Hyperactive::Tree::Root.get_instance end # # Call this IndexBuilder and pass it a +block+. @@ -254,7 +254,7 @@ # def self.select(name, matcher) key = self.collection_key(name) - CAPTAIN[key] ||= Tree.get_instance + CAPTAIN[key] ||= Hyperactive::Tree::Root.get_instance self.class_eval <Tree.get_instance(options) instead! + # A class suitable for storing large and often-changing datasets in + # an Archipelago environment. # - def initialize(options = {}) - @width = options[:width] || WIDTH - @elements = {} - @subtrees = nil - end - + # Is constructed like a set of nested Hashes that automatically create + # new children on demand, and will thusly only have to check the path from + # the root node to the leaf for changes when method calls return (see Archipelago::Treasure::Chest) + # and will only have to actually store into database the leaf itself if it + # has changed. # - # Deletes +key+ in this Tree. - # - def delete(key) - if @elements - @elements.delete(key) - else - subtree_for(key).delete(key) + class Root < Record + + attr_accessor :elements, :subtrees + + WIDTH = 1 << 3 + + # + # Dont call this! Call Root.get_instance(options) instead! + # + def initialize(options = {}) + @width = options[:width] || WIDTH + @elements = {} + @subtrees = nil end - end + + # + # Deletes +key+ in this Root. + # + def delete(key) + if @elements + @elements.delete(key) + else + subtree_for(key).delete(key) + end + end - # - # Returns the size of this Tree. - # - def size - if @elements - @elements.size - else - @subtrees.t_collect do |tree_id, tree| - tree.size - end.inject(0) do |sum, size| - sum + size + # + # Returns the size of this Root. + # + def size + if @elements + @elements.size + else + @subtrees.t_collect do |tree_id, tree| + tree.size + end.inject(0) do |sum, size| + sum + size + end end end - end - - # - # Returns all keys and values returning true for +callable+.call(key, value) in this Tree. - # - def select(callable) - if @elements - @elements.select do |k,v| - callable.call(k,v) + + # + # Returns all keys and values returning true for +callable+.call(key, value) in this Root. + # + def select(callable) + if @elements + @elements.select do |k,v| + callable.call(k,v) + end + else + @subtrees.t_collect do |tree_id, tree| + tree.select(callable) + end.inject([]) do |sum, match| + sum + match + end end - else - @subtrees.t_collect do |tree_id, tree| - tree.select(callable) - end.inject([]) do |sum, match| - sum + match - end end - end - # - # Returns all keys and values returning false for +callable+.call(key, value) in this Tree. - # - def reject(callable) - if @elements - @elements.reject do |k,v| - callable.call(k,v) + # + # Returns all keys and values returning false for +callable+.call(key, value) in this Root. + # + def reject(callable) + if @elements + @elements.reject do |k,v| + callable.call(k,v) + end + else + @subtrees.t_collect do |tree_id, tree| + tree.reject(callable) + end.inject([]) do |sum, match| + sum + match + end end - else - @subtrees.t_collect do |tree_id, tree| - tree.reject(callable) - end.inject([]) do |sum, match| - sum + match - end end - end - # - # Puts +value+ under +key+ in this Tree. - # - def []=(key, value) - if @elements - if @elements.size < @width - @elements[key] = value + # + # Puts +value+ under +key+ in this Root. + # + def []=(key, value) + if @elements + if @elements.size < @width + @elements[key] = value + else + split! + subtree_for(key)[key] = value + end else - split! subtree_for(key)[key] = value end - else - subtree_for(key)[key] = value + return value end - return value - end - # - # Returns the value for +key+ in this Tree. - # - def [](key) - if @elements - return @elements[key] - else - return subtree_for(key)[key] + # + # Returns the value for +key+ in this Root. + # + def [](key) + if @elements + return @elements[key] + else + return subtree_for(key)[key] + end end - end - # - # Returns an Array containing +callable+.call(key, value) from all values in this Tree. - # - def collect(callable) - rval = [] - self.each(Proc.new do |k,v| - rval << callable.call(k,v) - end) - return rval - end + # + # Returns an Array containing +callable+.call(key, value) from all values in this Root. + # + def collect(callable) + rval = [] + self.each(Proc.new do |k,v| + rval << callable.call(k,v) + end) + return rval + end - # - # Does +callable+.call(key, value) on all values in this Tree. - # - def each(callable) - if @elements - @elements.each do |key, value| - callable.call(key, value) + # + # Does +callable+.call(key, value) on all values in this Root. + # + def each(callable) + if @elements + @elements.each do |key, value| + callable.call(key, value) + end + else + @subtrees.t_each do |tree_id, tree| + tree.each(callable) + end + nil end - else - @subtrees.t_each do |tree_id, tree| - tree.each(callable) + end + + # + # Clear everything from this Tree and destroy it. + # + def destroy + if @elements + @elements.each do |key, value| + value.destroy if value.respond_to?(:destroy) + end + else + @subtrees.each do |tree_id, tree| + tree.destroy + end end - nil + freeze + super end - end - # - # Clear everything from this Tree and destroy it. - # - def destroy - if @elements - @elements.each do |key, value| - value.destroy if value.respond_to?(:destroy) + # + # Clear everything from this Root. + # + def clear + unless @elements + @subtrees.each do |tree_id, tree| + tree.clear + end + @subtrees = nil end - else - @subtrees.each do |tree_id, tree| - tree.destroy - end + @elements = {} end - freeze - super - end - # - # Clear everything from this Tree. - # - def clear - unless @elements + private + + # + # Finds the subtree responsible for +key+. + # + # Does it in this ugly way cause the nice way of just doing modulo gave + # really odd results since all hashes seem to give some modulo values + # a lot more often than expected. + # + def subtree_for(key) + key_id = Digest::SHA1.new("#{key.hash}#{self.record_id}").to_s @subtrees.each do |tree_id, tree| - tree.clear + return tree if tree_id > key_id end - @subtrees = nil + return @subtrees.values.first end - @elements = {} - end - private + # + # Split this Tree by creating @subtrees, + # then putting all @elements in them and then emptying @elements. + # + def split! + raise "Cant split twice!" unless @elements - # - # Finds the subtree responsible for +key+. - # - # Does it in this ugly way cause the nice way of just doing modulo gave - # really odd results since all hashes seem to give some modulo values - # a lot more often than expected. - # - def subtree_for(key) - key_id = Digest::SHA1.new("#{key.hash}#{self.record_id}").to_s - @subtrees.each do |tree_id, tree| - return tree if tree_id > key_id + @subtrees = RBTree.new + @subtrees.extend(Archipelago::Current::ThreadedCollection) + step = (1 << 160) / @width + 0.upto(@width - 1) do |n| + @subtrees["%x" % (n * step)] = Root.get_instance(:width => @width) + end + @elements.each do |key, value| + subtree_for(key)[key] = value + end + @elements = nil end - return @subtrees.values.first - end - # - # Split this Tree by creating @subtrees, - # then putting all @elements in them and then emptying @elements. - # - def split! - raise "Cant split twice!" unless @elements - - @subtrees = RBTree.new - @subtrees.extend(Archipelago::Current::ThreadedCollection) - step = (1 << 160) / @width - 0.upto(@width - 1) do |n| - @subtrees["%x" % (n * step)] = Tree.get_instance(:width => @width) - end - @elements.each do |key, value| - subtree_for(key)[key] = value - end - @elements = nil end end Modified: trunk/hyperactive/tests/tree_benchmark.rb =================================================================== --- trunk/hyperactive/tests/tree_benchmark.rb 2006-11-27 11:46:06 UTC (rev 67) +++ trunk/hyperactive/tests/tree_benchmark.rb 2006-11-27 12:43:40 UTC (rev 68) @@ -30,7 +30,7 @@ end def test_set_get - h = Hyperactive::Tree.get_instance + h = Hyperactive::Tree::Root.get_instance r = Hyperactive::Record.get_instance hash_test("Tree", h, r, 100) end Modified: trunk/hyperactive/tests/tree_test.rb =================================================================== --- trunk/hyperactive/tests/tree_test.rb 2006-11-27 11:46:06 UTC (rev 67) +++ trunk/hyperactive/tests/tree_test.rb 2006-11-27 12:43:40 UTC (rev 68) @@ -48,7 +48,7 @@ end def test_select_reject - h = Hyperactive::Tree.get_instance + h = Hyperactive::Tree::Root.get_instance r1 = Hyperactive::Record.get_instance r2 = Hyperactive::Record.get_instance h[r1.record_id] = r1 @@ -58,7 +58,7 @@ end def test_set_get - h = Hyperactive::Tree.get_instance + h = Hyperactive::Tree::Root.get_instance h2 = {} 10.times do r = Hyperactive::Record.get_instance From nobody at rubyforge.org Mon Nov 27 07:53:51 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 07:53:51 -0500 (EST) Subject: [Archipelago-submits] [69] trunk/hyperactive: made Record into a module of its own Message-ID: <20061127125351.F1CE4524156C@rubyforge.org> Revision: 69 Author: zond Date: 2006-11-27 07:53:50 -0500 (Mon, 27 Nov 2006) Log Message: ----------- made Record into a module of its own Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive/tree.rb trunk/hyperactive/tests/record_test.rb trunk/hyperactive/tests/tree_benchmark.rb trunk/hyperactive/tests/tree_test.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 12:43:40 UTC (rev 68) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 12:53:50 UTC (rev 69) @@ -24,358 +24,362 @@ # if you want to use archipelago in a Ruby on Rails project *hint hint*. # module Hyperactive - - # - # The default database connector. - # - CAPTAIN = Archipelago::Pirate::Captain.new - - # - # A tiny callable class that saves stuff in - # indexes depending on certain attributes. - # - class IndexBuilder + + module Record + # - # Get the first part of the key, that depends on the +attributes+. + # The default database connector. # - def self.get_attribute_key_part(attributes) - "Hyperactive::IndexBuilder::#{attributes.join(",")}" - end + CAPTAIN = Archipelago::Pirate::Captain.new + # - # Get the last part of the key, that depends on the +values+. + # A tiny callable class that saves stuff in + # indexes depending on certain attributes. # - def self.get_value_key_part(values) - "#{values.join(",")}" - end - # - # Initialize an IndexBuilder giving it an array of +attributes+ - # that will be indexed. - # - def initialize(attributes) - @attributes = attributes - end - # - # Get the Tree for the given +record+. - # - def get_tree_for(record) - values = @attributes.collect do |att| - if record.respond_to?(att) - record.send(att) + class IndexBuilder + # + # Get the first part of the key, that depends on the +attributes+. + # + def self.get_attribute_key_part(attributes) + "Hyperactive::IndexBuilder::#{attributes.join(",")}" + end + # + # Get the last part of the key, that depends on the +values+. + # + def self.get_value_key_part(values) + "#{values.join(",")}" + end + # + # Initialize an IndexBuilder giving it an array of +attributes+ + # that will be indexed. + # + def initialize(attributes) + @attributes = attributes + end + # + # Get the Tree for the given +record+. + # + def get_tree_for(record) + values = @attributes.collect do |att| + if record.respond_to?(att) + record.send(att) + else + nil + end + end + key = "#{self.class.get_attribute_key_part(@attributes)}::#{self.class.get_value_key_part(values)}" + return CAPTAIN[key] ||= Hyperactive::Tree::Root.get_instance + end + # + # Call this IndexBuilder and pass it a +block+. + # + # If the +argument+ is an Array then we know we are a save hook, + # otherwise we are a destroy hook. + # + def call(argument, &block) + yield + + # + # If the argument is an Array (of old value, new value) + # then we are a save hook, otherwise a destroy hook. + # + if Array === argument + record = argument.last + old_record = argument.first + get_tree_for(old_record).delete(record.record_id) + get_tree_for(record)[record.record_id] = record else - nil + record = argument + get_tree_for(record).delete(record.record_id) end end - key = "#{self.class.get_attribute_key_part(@attributes)}::#{self.class.get_value_key_part(values)}" - return CAPTAIN[key] ||= Hyperactive::Tree::Root.get_instance end + # - # Call this IndexBuilder and pass it a +block+. + # A tiny callable class that saves stuff inside containers + # if they match certain criteria. # - # If the +argument+ is an Array then we know we are a save hook, - # otherwise we are a destroy hook. - # - def call(argument, &block) - yield - + class MatchSaver # - # If the argument is an Array (of old value, new value) - # then we are a save hook, otherwise a destroy hook. + # Initialize this MatchSaver with a +key+, a callable +matcher+ + # and a +mode+ (:select, :reject, :delete_if_match or :delete_unless_match). # - if Array === argument - record = argument.last - old_record = argument.first - get_tree_for(old_record).delete(record.record_id) - get_tree_for(record)[record.record_id] = record - else + def initialize(key, matcher, mode) + @key = key + @matcher = matcher + @mode = mode + end + # + # Depending on @mode and return value of @matcher.call + # may save record in the Hash-like container named @key in + # the main database after having yielded to +block+. + # + def call(argument, &block) + yield + record = argument - get_tree_for(record).delete(record.record_id) + record = argument.last if Array === argument + + case @mode + when :select + if @matcher.call(record) + CAPTAIN[@key][record.record_id] = record + else + CAPTAIN[@key].delete(record.record_id) + end + when :reject + if @matcher.call(record) + CAPTAIN[@key].delete(record.record_id) + else + CAPTAIN[@key][record.record_id] = record + end + when :delete_if_match + if @matcher.call(record) + CAPTAIN[@key].delete(record.record_id) + end + when :delete_unless_match + unless @matcher.call(record) + CAPTAIN[@key].delete(record.record_id) + end + end end end - end - # - # A tiny callable class that saves stuff inside containers - # if they match certain criteria. - # - class MatchSaver # - # Initialize this MatchSaver with a +key+, a callable +matcher+ - # and a +mode+ (:select, :reject, :delete_if_match or :delete_unless_match). + # A convenient base class to inherit when you want the basic utility methods + # provided by for example ActiveRecord::Base *hint hint*. # - def initialize(key, matcher, mode) - @key = key - @matcher = matcher - @mode = mode - end + # NB: After an instance is created, it will actually return a copy within your local machine + # which is not what is usually the case. Every other time you fetch it using a select or other + # method you will instead receive a proxy object to the database. This means that nothing you + # do to it at that point will be persistent or even necessarily have a defined result. + # Therefore: do not use MyRecordSubclass.new to MyRecordSubclass#initialize objects, + # instead use MyRecordSubclass.get_instance, since it will return a fresh proxy object + # instead of the devious original: + # my_instance = MyRecordSubclass.get_instance(*the_same_arguments_as_to_initialize) # - # Depending on @mode and return value of @matcher.call - # may save record in the Hash-like container named @key in - # the main database after having yielded to +block+. - # - def call(argument, &block) - yield + class Bass + + @@create_hooks_by_class = {} + @@destroy_hooks_by_class = {} + @@save_hooks_by_class = {} - record = argument - record = argument.last if Array === argument + # + # The host we are running on. + # + HOST = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" - case @mode - when :select - if @matcher.call(record) - CAPTAIN[@key][record.record_id] = record - else - CAPTAIN[@key].delete(record.record_id) - end - when :reject - if @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) - else - CAPTAIN[@key][record.record_id] = record - end - when :delete_if_match - if @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) - end - when :delete_unless_match - unless @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) - end + # + # Call this if you want to change the default database connector + # to something else. + # + def self.setup(options = {}) + CAPTAIN.setup(options[:pirate_options]) end - end - end - # - # A convenient base class to inherit when you want the basic utility methods - # provided by for example ActiveRecord::Base *hint hint*. - # - # NB: After an instance is created, it will actually return a copy within your local machine - # which is not what is usually the case. Every other time you fetch it using a select or other - # method you will instead receive a proxy object to the database. This means that nothing you - # do to it at that point will be persistent or even necessarily have a defined result. - # Therefore: do not use MyRecordSubclass.new to MyRecordSubclass#initialize objects, - # instead use MyRecordSubclass.get_instance, since it will return a fresh proxy object - # instead of the devious original: - # my_instance = MyRecordSubclass.get_instance(*the_same_arguments_as_to_initialize) - # - class Record + # + # Return our create_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent the instance about to be created (initial + # insertion into the database system) that take a block argument. + # + # The block argument will be a Proc that actually injects the + # instance into the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon creation. + # + def self.create_hooks + self.get_hook_array_by_class(@@create_hooks_by_class) + end - @@create_hooks_by_class = {} - @@destroy_hooks_by_class = {} - @@save_hooks_by_class = {} + # + # Return our destroy_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent the instance about to be destroyed (removal + # from the database system) that take a block argument. + # + # The block argument will be a Proc that actually removes the + # instance from the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon destruction. + # + def self.destroy_hooks + self.get_hook_array_by_class(@@destroy_hooks_by_class) + end - # - # The host we are running on. - # - HOST = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" + # + # Return our save_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent [the old version, the new version] of the + # instance about to be saved (storage into the database system) + # along with a block argument. + # + # The block argument will be a Proc that actually saves the + # instance into the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon saving. + # + def self.save_hooks + self.get_hook_array_by_class(@@save_hooks_by_class) + end - # - # Call this if you want to change the default database connector - # to something else. - # - def self.setup(options = {}) - CAPTAIN.setup(options[:pirate_options]) - end - - # - # Return our create_hooks, which can then be treated as any old Array. - # - # These must be callable objects with an arity of 1 - # that will be sent the instance about to be created (initial - # insertion into the database system) that take a block argument. - # - # The block argument will be a Proc that actually injects the - # instance into the database system. - # - # Use this to preprocess, validate and/or postprocess your - # instances upon creation. - # - def self.create_hooks - self.get_hook_array_by_class(@@create_hooks_by_class) - end - - # - # Return our destroy_hooks, which can then be treated as any old Array. - # - # These must be callable objects with an arity of 1 - # that will be sent the instance about to be destroyed (removal - # from the database system) that take a block argument. - # - # The block argument will be a Proc that actually removes the - # instance from the database system. - # - # Use this to preprocess, validate and/or postprocess your - # instances upon destruction. - # - def self.destroy_hooks - self.get_hook_array_by_class(@@destroy_hooks_by_class) - end - - # - # Return our save_hooks, which can then be treated as any old Array. - # - # These must be callable objects with an arity of 1 - # that will be sent [the old version, the new version] of the - # instance about to be saved (storage into the database system) - # along with a block argument. - # - # The block argument will be a Proc that actually saves the - # instance into the database system. - # - # Use this to preprocess, validate and/or postprocess your - # instances upon saving. - # - def self.save_hooks - self.get_hook_array_by_class(@@save_hooks_by_class) - end - - # - # Create an index for this class. - # - # Will create a method find_by_#{attributes.join("_and_")} for this - # class that will return what you expect. - # - def self.index_by(*attributes) - attribute_key_part = IndexBuilder.get_attribute_key_part(attributes) - self.class_eval <self as - # key. If self doesnt exist in the +hash+ - # it will recurse by calling the same method in the - # superclass until it has been called in Hyperactive::Record. - # - def self.get_hook_array_by_class(hash) - return hash[self] if hash.include?(self) - - if self == Record - hash[self] = [] - return self.get_hook_array_by_class(hash) - else - hash[self] = self.superclass.get_hook_array_by_class(hash).clone - return self.get_hook_array_by_class(hash) + + private + + # + # The key used to store the collection with the given +sym+ as name. + # + def self.collection_key(sym) + "Hyperactive::Record::Bass::collection_key::#{sym}" end + + # + # Get an Array from +hash+ using self as + # key. If self doesnt exist in the +hash+ + # it will recurse by calling the same method in the + # superclass until it has been called in Hyperactive::Record::Base. + # + def self.get_hook_array_by_class(hash) + return hash[self] if hash.include?(self) + + if self == Bass + hash[self] = [] + return self.get_hook_array_by_class(hash) + else + hash[self] = self.superclass.get_hook_array_by_class(hash).clone + return self.get_hook_array_by_class(hash) + end + end + end - end end + Modified: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-27 12:43:40 UTC (rev 68) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-27 12:53:50 UTC (rev 69) @@ -34,7 +34,7 @@ # and will only have to actually store into database the leaf itself if it # has changed. # - class Root < Record + class Root < Hyperactive::Record::Bass attr_accessor :elements, :subtrees Modified: trunk/hyperactive/tests/record_test.rb =================================================================== --- trunk/hyperactive/tests/record_test.rb 2006-11-27 12:43:40 UTC (rev 68) +++ trunk/hyperactive/tests/record_test.rb 2006-11-27 12:53:50 UTC (rev 69) @@ -1,7 +1,7 @@ require File.join(File.dirname(__FILE__), 'test_helper') -class MyRecord < Hyperactive::Record +class MyRecord < Hyperactive::Record::Bass attr_accessor :bajs def self.save_hook(instance, &block) @@ -35,14 +35,14 @@ @c2.publish! @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) @tm.publish! - Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) - Hyperactive::CAPTAIN.update_services! + Hyperactive::Record::CAPTAIN.update_services! assert_within(10) do - Hyperactive::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort + Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort end assert_within(10) do - Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] end $BEFORE_SAVE = 0 $AFTER_SAVE = 0 @@ -60,12 +60,12 @@ @tm.stop! @tm.persistence_provider.unlink Archipelago::Disco::MC.clear! - Hyperactive::CAPTAIN.update_services! + Hyperactive::Record::CAPTAIN.update_services! assert_within(10) do - Hyperactive::CAPTAIN.chests.empty? + Hyperactive::Record::CAPTAIN.chests.empty? end assert_within(10) do - Hyperactive::CAPTAIN.trannies.empty? + Hyperactive::Record::CAPTAIN.trannies.empty? end end @@ -146,13 +146,13 @@ assert_equal(2, $BEFORE_SAVE) assert_equal(2, $AFTER_SAVE) assert_equal("brunt", r.bajs) - assert_equal("brunt", Hyperactive::CAPTAIN[r.record_id].bajs) + assert_equal("brunt", Hyperactive::Record::CAPTAIN[r.record_id].bajs) i = r.record_id - assert_equal(r, Hyperactive::CAPTAIN[i]) + assert_equal(r, Hyperactive::Record::CAPTAIN[i]) r.destroy assert_equal(1, $BEFORE_DESTROY) assert_equal(1, $AFTER_DESTROY) - assert_equal(nil, Hyperactive::CAPTAIN[i]) + assert_equal(nil, Hyperactive::Record::CAPTAIN[i]) end end Modified: trunk/hyperactive/tests/tree_benchmark.rb =================================================================== --- trunk/hyperactive/tests/tree_benchmark.rb 2006-11-27 12:43:40 UTC (rev 68) +++ trunk/hyperactive/tests/tree_benchmark.rb 2006-11-27 12:53:50 UTC (rev 69) @@ -10,13 +10,13 @@ @c2.publish! @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) @tm.publish! - Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) assert_within(20) do - Set.new(Hyperactive::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) + Set.new(Hyperactive::Record::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) end assert_within(20) do - Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] end end @@ -31,14 +31,14 @@ def test_set_get h = Hyperactive::Tree::Root.get_instance - r = Hyperactive::Record.get_instance + r = Hyperactive::Record::Bass.get_instance hash_test("Tree", h, r, 100) end def test_regular_hash_set_get - Hyperactive::CAPTAIN["h"] = {} - h = Hyperactive::CAPTAIN["h"] - r = Hyperactive::Record.get_instance + Hyperactive::Record::CAPTAIN["h"] = {} + h = Hyperactive::Record::CAPTAIN["h"] + r = Hyperactive::Record::Bass.get_instance hash_test("Hash", h, r, 100) end Modified: trunk/hyperactive/tests/tree_test.rb =================================================================== --- trunk/hyperactive/tests/tree_test.rb 2006-11-27 12:43:40 UTC (rev 68) +++ trunk/hyperactive/tests/tree_test.rb 2006-11-27 12:53:50 UTC (rev 69) @@ -19,14 +19,14 @@ @c2.publish! @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) @tm.publish! - Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) - Hyperactive::CAPTAIN.update_services! + Hyperactive::Record::CAPTAIN.update_services! assert_within(10) do - Hyperactive::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort + Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort end assert_within(10) do - Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] end end @@ -38,19 +38,19 @@ @tm.stop! @tm.persistence_provider.unlink Archipelago::Disco::MC.clear! - Hyperactive::CAPTAIN.update_services! + Hyperactive::Record::CAPTAIN.update_services! assert_within(10) do - Hyperactive::CAPTAIN.chests.empty? + Hyperactive::Record::CAPTAIN.chests.empty? end assert_within(10) do - Hyperactive::CAPTAIN.trannies.empty? + Hyperactive::Record::CAPTAIN.trannies.empty? end end def test_select_reject h = Hyperactive::Tree::Root.get_instance - r1 = Hyperactive::Record.get_instance - r2 = Hyperactive::Record.get_instance + r1 = Hyperactive::Record::Bass.get_instance + r2 = Hyperactive::Record::Bass.get_instance h[r1.record_id] = r1 h[r2.record_id] = r2 assert_equal(r1.record_id, h.select(RecordMatcher.new(r1.record_id)).first.first) @@ -61,14 +61,14 @@ h = Hyperactive::Tree::Root.get_instance h2 = {} 10.times do - r = Hyperactive::Record.get_instance + r = Hyperactive::Record::Bass.get_instance h[r.record_id] = r h2[r.record_id] = r end h2.each do |k,v| assert_equal(v, h[k]) - assert_equal(v, Hyperactive::CAPTAIN[h.record_id][k]) + assert_equal(v, Hyperactive::Record::CAPTAIN[h.record_id][k]) end end From nobody at rubyforge.org Mon Nov 27 08:42:17 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 08:42:17 -0500 (EST) Subject: [Archipelago-submits] [70] trunk/hyperactive/lib: added a list class that provides linked list backend Message-ID: <20061127134217.F0BA652415A4@rubyforge.org> Revision: 70 Author: zond Date: 2006-11-27 08:42:17 -0500 (Mon, 27 Nov 2006) Log Message: ----------- added a list class that provides linked list backend Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive/tree.rb trunk/hyperactive/lib/hyperactive.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 12:53:50 UTC (rev 69) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 13:42:17 UTC (rev 70) @@ -24,7 +24,11 @@ # if you want to use archipelago in a Ruby on Rails project *hint hint*. # module Hyperactive - + + # + # The package containing the base class Bass that simplifies + # using archipelago for generic database stuff. + # module Record # @@ -59,7 +63,7 @@ # # Get the Tree for the given +record+. # - def get_tree_for(record) + def get_key_for(record) values = @attributes.collect do |att| if record.respond_to?(att) record.send(att) @@ -68,7 +72,6 @@ end end key = "#{self.class.get_attribute_key_part(@attributes)}::#{self.class.get_value_key_part(values)}" - return CAPTAIN[key] ||= Hyperactive::Tree::Root.get_instance end # # Call this IndexBuilder and pass it a +block+. @@ -86,11 +89,15 @@ if Array === argument record = argument.last old_record = argument.first - get_tree_for(old_record).delete(record.record_id) - get_tree_for(record)[record.record_id] = record + old_key = get_key_for(old_record) + new_key = get_key_for(record) + if old_key != new_key + (CAPTAIN[old_key] ||= Hyperactive::Tree::Root.get_instance).delete(record.record_id) + (CAPTAIN[new_key] ||= Hyperactive::Tree::Root.get_instance)[record.record_id] = record + end else record = argument - get_tree_for(record).delete(record.record_id) + (CAPTAIN[get_key_for(record)] ||= Hyperactive::Tree::Root.get_instance).delete(record.record_id) end end end Modified: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-27 12:53:50 UTC (rev 69) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-27 13:42:17 UTC (rev 70) @@ -22,6 +22,10 @@ module Hyperactive + # + # The package containing the Hash-like Tree class that provides any + # kind of index for your Hyperactive classes. + # module Tree # Modified: trunk/hyperactive/lib/hyperactive.rb =================================================================== --- trunk/hyperactive/lib/hyperactive.rb 2006-11-27 12:53:50 UTC (rev 69) +++ trunk/hyperactive/lib/hyperactive.rb 2006-11-27 13:42:17 UTC (rev 70) @@ -20,3 +20,4 @@ require 'hyperactive/hooker' require 'hyperactive/record' require 'hyperactive/tree' +require 'hyperactive/list' From nobody at rubyforge.org Mon Nov 27 08:43:00 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 08:43:00 -0500 (EST) Subject: [Archipelago-submits] [71] trunk/hyperactive: added a list class that provides linked list backend Message-ID: <20061127134300.17F9152415A4@rubyforge.org> Revision: 71 Author: zond Date: 2006-11-27 08:42:59 -0500 (Mon, 27 Nov 2006) Log Message: ----------- added a list class that provides linked list backend Added Paths: ----------- trunk/hyperactive/lib/hyperactive/list.rb trunk/hyperactive/tests/list_test.rb Added: trunk/hyperactive/lib/hyperactive/list.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/list.rb (rev 0) +++ trunk/hyperactive/lib/hyperactive/list.rb 2006-11-27 13:42:59 UTC (rev 71) @@ -0,0 +1,154 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + + +require 'rubygems' +require 'archipelago' + +module Hyperactive + + # + # A package that simplifies putting dumb linked lists in the distributed database. + # + module List + + # + # A List element. + # + class Element < Hyperactive::Record::Bass + attr_accessor :previous, :next, :value + end + + # + # A List head. + # + class Head < Hyperactive::Record::Bass + attr_reader :first_element, :last_element, :size + + # + # Create a Head. + # + # NB: As usual, dont call this. Use Head.get_instance instead. + # + def initialize + @size = 0 + @first_element = @last_element = nil + end + + # + # Return the first value of the list. + # + def first + @first_element.value + end + + # + # Return the last value of the list. + # + def last + @last_element.value + end + + # + # Push +v+ onto the end of this list. + # + def <<(v) + if @first_element + new_element = Element.get_instance + new_element.value = v + new_element.previous = @last_element + @last_element.next = new_element + @last_element = new_element + else + start(v) + end + @size += 1 + return v + end + + # + # Push +v+ onto the beginning of this list. + # + def unshift(v) + if @first_element + new_element = Element.get_instance + new_element.value = v + new_element.next = @first_element + @first_element.previous = new_element + @first_element = new_element + else + start(v) + end + @size += 1 + return v + end + + # + # Remove the last value from this list and return it. + # + def pop + v = nil + if size > 1 + element = @last_element + @last_element = element.previous + @last_element.next = nil + v = element.value + element.destroy + else + v = @first_element.value + @first_element.destroy + @first_element = @last_element = nil + end + @size -= 1 + return v + end + + # + # Remove the first value from this list and return it. + # + def shift + v = nil + if size > 1 + element = @first_element + @first_element = element.next + @first_element.previous = nil + v = element.value + element.destroy + else + v = @first_element.value + @first_element.destroy + @first_element = @last_element = nil + end + @size -= 1 + return v + end + + private + + # + # Start this list with +v+ when it has no other values. + # + def start(v) + @first_element = @last_element = Element.get_instance + @first_element.value = v + end + + end + + end + +end Added: trunk/hyperactive/tests/list_test.rb =================================================================== --- trunk/hyperactive/tests/list_test.rb (rev 0) +++ trunk/hyperactive/tests/list_test.rb 2006-11-27 13:42:59 UTC (rev 71) @@ -0,0 +1,76 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class ListTest < Test::Unit::TestCase + + def setup + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + Hyperactive::Record::CAPTAIN.update_services! + assert_within(10) do + Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort + end + assert_within(10) do + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + end + end + + def teardown + @c.stop! + @c.persistence_provider.unlink + @c2.stop! + @c2.persistence_provider.unlink + @tm.stop! + @tm.persistence_provider.unlink + Archipelago::Disco::MC.clear! + Hyperactive::Record::CAPTAIN.update_services! + assert_within(10) do + Hyperactive::Record::CAPTAIN.chests.empty? + end + assert_within(10) do + Hyperactive::Record::CAPTAIN.trannies.empty? + end + end + + def test_push_unshift_pop_shift + l = Hyperactive::List::Head.get_instance + assert_equal(0, l.size) + l << (r = Hyperactive::Record::Bass.get_instance) + assert_equal(1, l.size) + assert_equal(l.first, l.last) + assert_equal(r, l.first) + l << (r2 = Hyperactive::Record::Bass.get_instance) + assert_equal(2, l.size) + assert_equal(r2, l.last) + assert_equal(r, l.first) + assert_equal(r2, l.first_element.next.value) + l.unshift(r3 = Hyperactive::Record::Bass.get_instance) + assert_equal(3, l.size) + assert_equal(r3, l.first) + assert_equal(r2, l.last) + assert_equal(r, l.first_element.next.value) + assert_equal(r2, l.first_element.next.next.value) + assert_equal(r3, l.last_element.previous.previous.value) + r4 = l.shift + assert_equal(r3, r4) + assert_equal(2, l.size) + assert_equal(r, l.first) + assert_equal(r2, l.last) + r5 = l.pop + assert_equal(1, l.size) + assert_equal(r, l.first) + assert_equal(r, l.last) + assert_equal(r2, r5) + r6 = l.pop + assert_equal(0, l.size) + assert_equal(nil, l.first_element) + assert_equal(nil, l.last_element) + end + +end From nobody at rubyforge.org Mon Nov 27 13:23:56 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Mon, 27 Nov 2006 13:23:56 -0500 (EST) Subject: [Archipelago-submits] [72] trunk/hyperactive: made Record.select/reject take blocks instead of callable objects Message-ID: <20061127182356.690B6524155C@rubyforge.org> Revision: 72 Author: zond Date: 2006-11-27 13:23:55 -0500 (Mon, 27 Nov 2006) Log Message: ----------- made Record.select/reject take blocks instead of callable objects Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/tests/record_test.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 13:42:59 UTC (rev 71) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-27 18:23:55 UTC (rev 72) @@ -261,7 +261,7 @@ # return true. Will only return instances saved after this selector # is defined. # - def self.select(name, matcher) + def self.select(name, &block) key = self.collection_key(name) CAPTAIN[key] ||= Hyperactive::Tree::Root.get_instance self.class_eval < Revision: 73 Author: zond Date: 2006-11-28 08:40:30 -0500 (Tue, 28 Nov 2006) Log Message: ----------- created release_0_2_2 Modified Paths: -------------- tags/release_0_2_2/archipelago/Rakefile tags/release_0_2_2/hyperactive/Rakefile trunk/archipelago/Rakefile trunk/hyperactive/Rakefile Added Paths: ----------- tags/release_0_2_2/ Copied: tags/release_0_2_2 (from rev 72, trunk) Modified: tags/release_0_2_2/archipelago/Rakefile =================================================================== --- trunk/archipelago/Rakefile 2006-11-27 18:23:55 UTC (rev 72) +++ tags/release_0_2_2/archipelago/Rakefile 2006-11-28 13:40:30 UTC (rev 73) @@ -9,7 +9,7 @@ spec = Gem::Specification.new do |s| s.platform = Gem::Platform::RUBY s.name = "archipelago" - s.version = "0.2.0" + s.version = "0.2.2" s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A set of tools for distributed computing in ruby." Modified: tags/release_0_2_2/hyperactive/Rakefile =================================================================== --- trunk/hyperactive/Rakefile 2006-11-27 18:23:55 UTC (rev 72) +++ tags/release_0_2_2/hyperactive/Rakefile 2006-11-28 13:40:30 UTC (rev 73) @@ -9,7 +9,7 @@ spec = Gem::Specification.new do |s| s.platform = Gem::Platform::RUBY s.name = "hyperactive" - s.version = "0.1.1" + s.version = "0.2.2" s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A base class for persistent objects that uses archipelago for persistence. Useful for Ruby on Rails models for example." Modified: trunk/archipelago/Rakefile =================================================================== --- trunk/archipelago/Rakefile 2006-11-27 18:23:55 UTC (rev 72) +++ trunk/archipelago/Rakefile 2006-11-28 13:40:30 UTC (rev 73) @@ -9,7 +9,7 @@ spec = Gem::Specification.new do |s| s.platform = Gem::Platform::RUBY s.name = "archipelago" - s.version = "0.2.0" + s.version = "0.2.2" s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A set of tools for distributed computing in ruby." Modified: trunk/hyperactive/Rakefile =================================================================== --- trunk/hyperactive/Rakefile 2006-11-27 18:23:55 UTC (rev 72) +++ trunk/hyperactive/Rakefile 2006-11-28 13:40:30 UTC (rev 73) @@ -9,7 +9,7 @@ spec = Gem::Specification.new do |s| s.platform = Gem::Platform::RUBY s.name = "hyperactive" - s.version = "0.1.1" + s.version = "0.2.2" s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A base class for persistent objects that uses archipelago for persistence. Useful for Ruby on Rails models for example." From nobody at rubyforge.org Tue Nov 28 18:38:25 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 28 Nov 2006 18:38:25 -0500 (EST) Subject: [Archipelago-submits] [74] trunk: improved README in both projects Message-ID: <20061128233825.70B8D5241B39@rubyforge.org> Revision: 74 Author: zond Date: 2006-11-28 18:38:23 -0500 (Tue, 28 Nov 2006) Log Message: ----------- improved README in both projects Modified Paths: -------------- trunk/archipelago/README trunk/hyperactive/README Modified: trunk/archipelago/README =================================================================== --- trunk/archipelago/README 2006-11-28 13:40:30 UTC (rev 73) +++ trunk/archipelago/README 2006-11-28 23:38:23 UTC (rev 74) @@ -13,27 +13,33 @@ Archipelago::Treasure:: A distributed object database where the objects never leave the database, instead you do your operations upon references to the objects. It has support for serializably isolated transactions with optimistic concurrency control using Archipelago::Tranny or any transaction manager with similar semantics. Archipelago::Pirate:: A client tool to allocate Archipelago::Treasure::Chests for different keys and act like an almost normal local Hash for providing distributed and scaleable object database facilities to any ruby application. -== Examples: +== Usage: -To build a ruby gem from these sources do 'rake gem'. The gem will be -placed within the pkg/ directory. +To use archipelago in the simplest and most obvious way, just run the script/services.rb script. -To set up an Archipelago::Tranny::Manager do the following (from scripts/tranny.rb): +You can run this on any number of machines in your local network, and they will all communicate through their own +discovery services. - :include:script/tranny.rb +Then you instantiate an Archipelago::Pirate::Captain anywhere in the network. This instance can be used basically as +a normal Hash - most everything will be transparently hidden from you. -To set up an Archipelago::Treasure::Chest do the following (from scripts/chest.rb): +Everything you put in this instance will be persistently stored in the network. - :include:script/chest.rb +== Examples: -To set up an Archipelago::Pirate::Captain do the following (from scripts/pirate.rb): +To run a transaction manager and a database server, just do the following (from script/services.rb): + :include:script/services.rb + +To set up an Archipelago::Pirate::Captain do the following (from script/pirate.rb): + :include:script/pirate.rb -To set up a test environment to play around with, run the following +Or you can run script/console, which starts an irb and loads script/pirate.rb. + +So, to set up a test environment to play around with in a few simple steps, run the following commands in a few terminals: -* script/tranny.rb /tmp/tranny -* script/chest.rb /tmp/chest1 -* script/chest.rb /tmp/chest2 -* script/console + script/services.rb /tmp/services1 + script/services.rb /tmp/services2 + script/console Modified: trunk/hyperactive/README =================================================================== --- trunk/hyperactive/README 2006-11-28 13:40:30 UTC (rev 73) +++ trunk/hyperactive/README 2006-11-28 23:38:23 UTC (rev 74) @@ -8,9 +8,19 @@ == Sub packages: -Hyperactive::Record:: The base class itself, providing you with cached selectors and rejectors, along with cached finders for any number of attributes. +Hyperactive::Record:: The base class package itself, providing you with cached selectors and rejectors, along with cached finders for any number of attributes. Hyperactive::Tree:: A collection class that contains any number of Hyperactive::Records in a tree structure. Scales logarithmically, so use with care. +== Usage: + +To use Hyperactive in the simplest way, just run the script/services.rb script from the Archipelago +distribution to get a few services running, then subclass Hyperactive::Record::Bass and create +objects with MyBassSubclass.get_instance (not new!). They will be automagically +stored in the network, and fetchable via their instance.record_id. + +By loading Hyperactive::Record you will automatically define Hyperactive::Record::CAPTAIN which will +be an Archipelago::Pirate::Captain that is your interface to the distributed database. + == Examples: To define a Record subclass that has the properties @active and @city that are indexed in two ways: From nobody at rubyforge.org Tue Nov 28 18:59:24 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 28 Nov 2006 18:59:24 -0500 (EST) Subject: [Archipelago-submits] [75] trunk/archipelago: added include? to Captain, Chest and BerkeleyHashish, and tests Message-ID: <20061128235924.9B0BC5241B22@rubyforge.org> Revision: 75 Author: zond Date: 2006-11-28 18:59:24 -0500 (Tue, 28 Nov 2006) Log Message: ----------- added include? to Captain, Chest and BerkeleyHashish, and tests Modified Paths: -------------- trunk/archipelago/lib/archipelago/hashish.rb trunk/archipelago/lib/archipelago/pirate.rb trunk/archipelago/lib/archipelago/treasure.rb trunk/archipelago/tests/pirate_test.rb trunk/archipelago/tests/treasure_test.rb Modified: trunk/archipelago/lib/archipelago/hashish.rb =================================================================== --- trunk/archipelago/lib/archipelago/hashish.rb 2006-11-28 23:38:23 UTC (rev 74) +++ trunk/archipelago/lib/archipelago/hashish.rb 2006-11-28 23:59:24 UTC (rev 75) @@ -62,6 +62,12 @@ return Marshal.load(@content_db[Marshal.dump(key)]) end # + # Returns true if this BerkeleyHashish include +key+. + # + def include?(key) + @content.include?(key) || @content_db.include?(Marshal.dump(key)) + end + # # Simply get the value for the +key+. # def [](key) Modified: trunk/archipelago/lib/archipelago/pirate.rb =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb 2006-11-28 23:38:23 UTC (rev 74) +++ trunk/archipelago/lib/archipelago/pirate.rb 2006-11-28 23:59:24 UTC (rev 75) @@ -112,6 +112,13 @@ end # + # Returns true if this Captain includes the given +key+, optionally within a +transaction+. + # + def include?(key, transaction = nil) + responsible_chest(key)[:service].include?(key, transaction || @transaction) + end + + # # Get a value from the distributed database network using a +key+, # optionally within a +transaction+. # @@ -254,7 +261,7 @@ def responsible_chest(key) raise NoRemoteDatabaseAvailableException.new(self) if @chests.empty? - key_id = Digest::SHA1.new(Marshal.dump(key)).to_s + key_id = Digest::SHA1.hexdigest(Marshal.dump(key)) sorted_chest_ids = @chests.keys.sort sorted_chest_ids.each do |id| return @chests[id] if id > key_id Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-28 23:38:23 UTC (rev 74) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-28 23:59:24 UTC (rev 75) @@ -329,6 +329,20 @@ end # + # Returns true if this Chest includes the given +key+, optionally within a +transaction+. + # + def include?(key, transaction = nil) + join!(transaction) + + if transaction + return @snapshot_by_transaction[transaction].include?(key) + else + return @db.include?(key) + end + + end + + # # Return the contents of this chest using a given +key+ and +transaction+. # def [](key, transaction = nil) Modified: trunk/archipelago/tests/pirate_test.rb =================================================================== --- trunk/archipelago/tests/pirate_test.rb 2006-11-28 23:38:23 UTC (rev 74) +++ trunk/archipelago/tests/pirate_test.rb 2006-11-28 23:59:24 UTC (rev 75) @@ -33,6 +33,28 @@ DRb.stop_service end + def test_include + assert(!@p.include?("blabla")) + @p["blabla"] = "kissobjas" + assert(@p.include?("blabla")) + t = @tm.begin + assert(!@p.include?("hehu")) + assert(!@p.include?("hehu", t)) + @p["hehu", t] = "brunte" + assert(!@p.include?("hehu")) + assert(@p.include?("hehu", t)) + t.commit! + assert(@p.include?("hehu")) + t = @tm.begin + assert(!@p.include?("hehu2")) + assert(!@p.include?("hehu2", t)) + @p["hehu2", t] = "brunte" + assert(!@p.include?("hehu2")) + assert(@p.include?("hehu2", t)) + t.abort! + assert(!@p.include?("hehu2")) + end + def test_each @p["oj"] = "bla" @p["brunt"] = "ja" Modified: trunk/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-28 23:38:23 UTC (rev 74) +++ trunk/archipelago/tests/treasure_test.rb 2006-11-28 23:59:24 UTC (rev 75) @@ -37,6 +37,28 @@ assert_equal({"oj" => "bla", "brunt" => "ja"}, h) end + def test_include + assert(!@c.include?("blabla")) + @c["blabla"] = "kissobjas" + assert(@c.include?("blabla")) + t = @tm.begin + assert(!@c.include?("hehu")) + assert(!@c.include?("hehu", t)) + @c["hehu", t] = "brunte" + assert(!@c.include?("hehu")) + assert(@c.include?("hehu", t)) + t.commit! + assert(@c.include?("hehu")) + t = @tm.begin + assert(!@c.include?("hehu2")) + assert(!@c.include?("hehu2", t)) + @c["hehu2", t] = "brunte" + assert(!@c.include?("hehu2")) + assert(@c.include?("hehu2", t)) + t.abort! + assert(!@c.include?("hehu2")) + end + def test_around_save s = A.new("hehu") @c["oj"] = s @@ -61,7 +83,9 @@ def test_store_load_update s = "hehu" + assert(!@c.include?("oj")) @c["oj"] = "hehu" + assert(@c.include?("oj")) oj1 = @c["oj"] assert_equal(oj1, @c["oj"]) assert_equal("hehu", oj1) From nobody at rubyforge.org Tue Nov 28 19:11:24 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 28 Nov 2006 19:11:24 -0500 (EST) Subject: [Archipelago-submits] [76] trunk/archipelago: added load_hook support for stuff stored in the Chest, and tests for it Message-ID: <20061129001124.BA6195241B4B@rubyforge.org> Revision: 76 Author: zond Date: 2006-11-28 19:11:24 -0500 (Tue, 28 Nov 2006) Log Message: ----------- added load_hook support for stuff stored in the Chest, and tests for it Modified Paths: -------------- trunk/archipelago/lib/archipelago/hashish.rb trunk/archipelago/tests/pirate_test.rb trunk/archipelago/tests/tranny_test.rb trunk/archipelago/tests/treasure_benchmark.rb trunk/archipelago/tests/treasure_test.rb Modified: trunk/archipelago/lib/archipelago/hashish.rb =================================================================== --- trunk/archipelago/lib/archipelago/hashish.rb 2006-11-28 23:59:24 UTC (rev 75) +++ trunk/archipelago/lib/archipelago/hashish.rb 2006-11-29 00:11:24 UTC (rev 76) @@ -70,6 +70,11 @@ # # Simply get the value for the +key+. # + # Will call value.load_hook and send it + # a block that does the actuall insertion of the value + # into the live hash if value.respond_to?(:load_hook) + # if the value didnt exist in the live hash yet. + # def [](key) @lock.synchronize_on(key) do @@ -204,13 +209,23 @@ # Read +key+ from db and if it is found # put it in the cache Hash. # + # Will call value.load_hook and send it + # a block that does the actuall insertion of the value + # into the live hash if value.respond_to?(:load_hook). + # def get_from_db(key) serialized_key = Marshal.dump(key) serialized_value = @content_db[serialized_key] return nil unless serialized_value value = Marshal.load(serialized_value) - @content[key] = value + if value.respond_to?(:load_hook) + value.load_hook do + @content[key] = value + end + else + @content[key] = value + end return value end end @@ -248,15 +263,21 @@ return db end # - # Closes databases opened by this instance and removes the persistent files. + # Closes databases opened by this instance. # - def unlink + def close! @berkeley_hashishes.each do |h| h.close! end @bdb_dbs.each do |d| d.close end + end + # + # Closes databases opened by this instance and removes the persistent files. + # + def unlink! + close! home = Pathname.new(@env.home) @env.close home.rmtree if home.exist? Modified: trunk/archipelago/tests/pirate_test.rb =================================================================== --- trunk/archipelago/tests/pirate_test.rb 2006-11-28 23:59:24 UTC (rev 75) +++ trunk/archipelago/tests/pirate_test.rb 2006-11-29 00:11:24 UTC (rev 76) @@ -25,11 +25,11 @@ def teardown @p.stop! @c.stop! - @c.persistence_provider.unlink + @c.persistence_provider.unlink! @c2.stop! - @c2.persistence_provider.unlink + @c2.persistence_provider.unlink! @tm.stop! - @tm.persistence_provider.unlink + @tm.persistence_provider.unlink! DRb.stop_service end Modified: trunk/archipelago/tests/tranny_test.rb =================================================================== --- trunk/archipelago/tests/tranny_test.rb 2006-11-28 23:59:24 UTC (rev 75) +++ trunk/archipelago/tests/tranny_test.rb 2006-11-29 00:11:24 UTC (rev 76) @@ -9,7 +9,7 @@ end def teardown - @tm.persistence_provider.unlink + @tm.persistence_provider.unlink! DRb.stop_service end Modified: trunk/archipelago/tests/treasure_benchmark.rb =================================================================== --- trunk/archipelago/tests/treasure_benchmark.rb 2006-11-28 23:59:24 UTC (rev 75) +++ trunk/archipelago/tests/treasure_benchmark.rb 2006-11-29 00:11:24 UTC (rev 76) @@ -9,7 +9,7 @@ end def teardown - @c.persistence_provider.unlink + @c.persistence_provider.unlink! DRb.stop_service end Modified: trunk/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-28 23:59:24 UTC (rev 75) +++ trunk/archipelago/tests/treasure_test.rb 2006-11-29 00:11:24 UTC (rev 76) @@ -7,6 +7,11 @@ yield $BURKMAT2 += 1 end + def load_hook(&block) + $BURKMAT3 += 1 + yield + $BURKMAT4 += 1 + end end class TreasureTest < Test::Unit::TestCase @@ -18,12 +23,14 @@ @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) $BURKMAT = 0 $BURKMAT2 = 0 + $BURKMAT3 = 0 + $BURKMAT4 = 0 end def teardown - @c.persistence_provider.unlink - @c2.persistence_provider.unlink - @tm.persistence_provider.unlink + @c.persistence_provider.unlink! + @c2.persistence_provider.unlink! + @tm.persistence_provider.unlink! DRb.stop_service end @@ -59,6 +66,15 @@ assert(!@c.include?("hehu2")) end + def test_around_load + @c["brunis"] = A.new("hirr") + @c.persistence_provider.close! + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + a = @c["brunis"] + assert_equal(1, $BURKMAT3) + assert_equal(1, $BURKMAT4) + end + def test_around_save s = A.new("hehu") @c["oj"] = s From nobody at rubyforge.org Tue Nov 28 20:06:36 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 28 Nov 2006 20:06:36 -0500 (EST) Subject: [Archipelago-submits] [77] trunk/archipelago: added test for Captain#transaction and added a few utility methods Message-ID: <20061129010636.CA31D5241B56@rubyforge.org> Revision: 77 Author: zond Date: 2006-11-28 20:06:36 -0500 (Tue, 28 Nov 2006) Log Message: ----------- added test for Captain#transaction and added a few utility methods Modified Paths: -------------- trunk/archipelago/lib/archipelago/pirate.rb trunk/archipelago/lib/archipelago/tranny.rb trunk/archipelago/tests/pirate_test.rb Modified: trunk/archipelago/lib/archipelago/pirate.rb =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb 2006-11-29 00:11:24 UTC (rev 76) +++ trunk/archipelago/lib/archipelago/pirate.rb 2006-11-29 01:06:36 UTC (rev 77) @@ -51,6 +51,15 @@ end end + # + # Raised when the transaction block failed to commit the transaction in the end. + # + class CommitFailedException < RuntimeError + def initialize(pirate, transaction) + super("#{pirate} failed to commit #{transaction}") + end + end + INITIAL_SERVICE_UPDATE_INTERVAL = 1 MAXIMUM_SERVICE_UPDATE_INTERVAL = 60 CHEST_DESCRIPTION = { @@ -165,6 +174,13 @@ end # + # Returns our active transaction, if any. + # + def active_transaction + @transaction + end + + # # Execute +block+ within a transaction. # # Will commit! transaction after the block is finished unless @@ -178,11 +194,10 @@ @transaction = @trannies.values.first[:service].begin begin yield(@transaction) - @transaction.commit! if @transaction.state == :active + raise CommitFailedException.new(self, @transaction) unless @transaction.commit! == :commited rescue Exception => e @transaction.abort! unless @transaction.state == :aborted - puts e - pp e.backtrace + raise e ensure @transaction = nil end Modified: trunk/archipelago/lib/archipelago/tranny.rb =================================================================== --- trunk/archipelago/lib/archipelago/tranny.rb 2006-11-29 00:11:24 UTC (rev 76) +++ trunk/archipelago/lib/archipelago/tranny.rb 2006-11-29 01:06:36 UTC (rev 77) @@ -218,6 +218,12 @@ end end # + # Implemented to allow comparison between proxies. + # + def ==(o) + eql?(o) + end + # # Forwards everything to our Transaction and remembers # returnvalue if necessary. # Modified: trunk/archipelago/tests/pirate_test.rb =================================================================== --- trunk/archipelago/tests/pirate_test.rb 2006-11-29 00:11:24 UTC (rev 76) +++ trunk/archipelago/tests/pirate_test.rb 2006-11-29 01:06:36 UTC (rev 77) @@ -111,6 +111,40 @@ assert_equal(s1, s2) end + def test_transaction + p2 = Archipelago::Pirate::Captain.new(:chest_description => {:class => "TestChest"}, + :tranny_description => {:class => "TestManager"}) + assert_within(10) do + p2.chests.keys.sort == [@c.service_id, @c2.service_id].sort + end + assert_within(10) do + p2.trannies.keys == [@tm.service_id] + end + trans = nil + assert_raise(Archipelago::Pirate::CommitFailedException) do + @p.transaction do |trans| + assert_equal(trans.transaction_id, @p.active_transaction.transaction_id) + @p["hehu"] = "haha" + assert(!p2.include?("hehu")) + assert(!p2.include?("hehu", nil)) + assert_equal(nil, p2.active_transaction) + trans2 = nil + p2.transaction do |trans2| + assert_equal(trans2.transaction_id, p2.active_transaction.transaction_id) + p2["hehu"] = "hoj" + assert_equal("haha", @p["hehu"]) + assert_equal("hoj", @p["hehu", trans2]) + end + assert_equal(:commited, trans2.state) + assert_equal(:active, trans.state) + assert_equal("hoj", p2["hehu"]) + assert_equal("haha", p2["hehu", trans]) + end + end + assert_equal(:aborted, trans.state) + assert_equal("hoj", @p["hehu"]) + end + def test_write_read_transaction $T = true @p["hej"] = "haha" From nobody at rubyforge.org Tue Nov 28 20:19:04 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 28 Nov 2006 20:19:04 -0500 (EST) Subject: [Archipelago-submits] [78] trunk/archipelago: added support for the with_transaction method for objects proxied in Chests Message-ID: <20061129011904.730FA5241B58@rubyforge.org> Revision: 78 Author: zond Date: 2006-11-28 20:19:04 -0500 (Tue, 28 Nov 2006) Log Message: ----------- added support for the with_transaction method for objects proxied in Chests Modified Paths: -------------- trunk/archipelago/lib/archipelago/treasure.rb trunk/archipelago/tests/treasure_test.rb Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-29 01:06:36 UTC (rev 77) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-29 01:19:04 UTC (rev 78) @@ -94,6 +94,19 @@ # # A proxy to something in the chest. # + # This will do a very efficient masquerade as the object it proxies. When asked + # for its class or any other attribute it will forward the query to the proxied object. + # + # It can also be a part of a transaction, either because it was fetched from the chest + # within a transaction, or because it is the return value of a Dubloon#join call. + # + # In this case all forwarded methods will also be within the same transaction, and + # any change to the proxied object will be inside that transaction. + # + # If the proxied object itself needs to handle transaction semantics it can implement + # the with_transaction(transaction, &block) method, which will wrap the + # method call itself within the home Chest of the proxied object. + # class Dubloon # # Remove all methods so that we look like our target. @@ -416,6 +429,10 @@ # Call an instance +method+ on whatever this chest holds at +key+ # with any +transaction+ and +args+. # + # If a +transaction+ is provided and the value for the +key+ + # respond_to?(:with_transaction) then the actual method call + # will be wrapped within the block sent to with_transaction(transaction, &block). + # def call_instance_method(key, method, transaction, *arguments, &block) if transaction return call_with_transaction(key, method, transaction, *arguments, &block) @@ -616,8 +633,11 @@ end # - # Call a method within a transaction. + # Call a +method+ within a +transaction+. # + # If the object we want to run the +method+ on respond_to?(:with_transaction) + # then we will execute the actual +method+ within a block sent to the with_transaction method. + # def call_with_transaction(key, method, transaction, *arguments, &block) assert_transaction(transaction) @@ -631,7 +651,15 @@ raise UnknownObjectException.new(self, key, transaction) unless instance begin - return execute(instance, method, *arguments, &block) + if instance.respond_to?(:with_transaction) + return_value = nil + instance.with_transaction(transaction) do + return_value = execute(instance, method, *arguments, &block) + end + return return_value + else + return execute(instance, method, *arguments, &block) + end ensure # # Make sure we remember when this object was last changed according Modified: trunk/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-29 01:06:36 UTC (rev 77) +++ trunk/archipelago/tests/treasure_test.rb 2006-11-29 01:19:04 UTC (rev 78) @@ -12,6 +12,12 @@ yield $BURKMAT4 += 1 end + def with_transaction(transaction, &block) + $BURKMAT7 = transaction + $BURKMAT5 += 1 + yield + $BURKMAT6 += 1 + end end class TreasureTest < Test::Unit::TestCase @@ -25,6 +31,9 @@ $BURKMAT2 = 0 $BURKMAT3 = 0 $BURKMAT4 = 0 + $BURKMAT5 = 0 + $BURKMAT6 = 0 + $BURKMAT7 = nil end def teardown @@ -34,6 +43,15 @@ DRb.stop_service end + def test_with_transaction + t = @tm.begin + @c["hehu"] = A.new("kiss") + @c["hehu", t].upcase! + assert_equal(1, $BURKMAT5) + assert_equal(1, $BURKMAT6) + assert(!$BURKMAT7.nil?) + end + def test_each @c["oj"] = "bla" @c["brunt"] = "ja" From nobody at rubyforge.org Tue Nov 28 21:54:23 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 28 Nov 2006 21:54:23 -0500 (EST) Subject: [Archipelago-submits] [79] trunk/archipelago: added to TODO. Message-ID: <20061129025423.EFA645241B00@rubyforge.org> Revision: 79 Author: zond Date: 2006-11-28 21:54:23 -0500 (Tue, 28 Nov 2006) Log Message: ----------- added to TODO. made Current::ThreadedCollection take callables as well as blocks. fixed broken include? in BerkeleyHashish. fixed so that store_if_changed doesnt restore removed objects. Modified Paths: -------------- trunk/archipelago/TODO trunk/archipelago/lib/archipelago/current.rb trunk/archipelago/lib/archipelago/hashish.rb Modified: trunk/archipelago/TODO =================================================================== --- trunk/archipelago/TODO 2006-11-29 01:19:04 UTC (rev 78) +++ trunk/archipelago/TODO 2006-11-29 02:54:23 UTC (rev 79) @@ -6,6 +6,9 @@ check whether the instance before the call differs from the instance after the call. Preferably without incurring performance lossage. + * Make Dubloon proxy targets able to provide a dirty_state method that decides + whether they are to be considered dirty, clean or auto-select. + * Test the transaction recovery mechanism of Chest. * Make Archipelago::Treasure::Dubloons work after an Archipelago::Treasure::Chest Modified: trunk/archipelago/lib/archipelago/current.rb =================================================================== --- trunk/archipelago/lib/archipelago/current.rb 2006-11-29 01:19:04 UTC (rev 78) +++ trunk/archipelago/lib/archipelago/current.rb 2006-11-29 02:54:23 UTC (rev 79) @@ -36,47 +36,69 @@ # # Adds a few threaded methods to the normal ruby collections. # + # The only method your class has to implement to use this module is each(&block). + # # NB: Will work slightly different than the unthreaded ones in certain circumstances. # module ThreadedCollection # - # Like each, except calls +block+ within a new thread. + # Like each, except calls +block+ or +callable+ within a new thread. # - def t_each(&block) + def t_each(callable = nil, &block) + raise "You have to provide either callable or block" if callable.nil? && block.nil? + threads = [] + self.each do |args| threads << Thread.new do - yield(args) + if callable + callable.call(args) + else + yield(args) + end end end + threads.each do |thread| thread.join end end # - # Like collect, except calls +block+ within a new thread. + # Like collect, except calls +block+ or +callable+ within a new thread. # - def t_collect(&block) + def t_collect(callable = nil, &block) + raise "You have to provide either callable or block" if callable.nil? && block.nil? + result = [] result.extend(Synchronized) self.t_each do |args| result.synchronize do - result << yield(args) + if callable + result << callable.call(args) + else + result << yield(args) + end end end return result end # - # Like select, except calls +block+ within a new thread. + # Like select, except calls +block+ or +callable+ within a new thread. # - def t_select(&block) + def t_select(callable = nil, &block) + raise "You have to provide either callable or block" if callable.nil? && block.nil? + result = [] result.extend(Synchronized) self.t_each do |args| - matches = yield(args) + if callable + matches = callable.call(args) + else + matches = yield(args) + end result.synchronize do result << args end if matches @@ -85,13 +107,19 @@ end # - # Like reject, except calls +block+ within a new thread. + # Like reject, except calls +block+ or +callable+ within a new thread. # - def t_reject(&block) + def t_reject(callable = nil, &block) + raise "You have to provide either callable or block" if callable.nil? && block.nil? + result = [] result.extend(Synchronized) self.t_each do |args| - matches = yield(args) + if callable + matches = callable.call(args) + else + matches = yield(args) + end result.synchronize do result << args end unless matches Modified: trunk/archipelago/lib/archipelago/hashish.rb =================================================================== --- trunk/archipelago/lib/archipelago/hashish.rb 2006-11-29 01:19:04 UTC (rev 78) +++ trunk/archipelago/lib/archipelago/hashish.rb 2006-11-29 02:54:23 UTC (rev 79) @@ -65,7 +65,7 @@ # Returns true if this BerkeleyHashish include +key+. # def include?(key) - @content.include?(key) || @content_db.include?(Marshal.dump(key)) + @content.include?(key) || !@content_db[Marshal.dump(key)].nil? end # # Simply get the value for the +key+. @@ -105,7 +105,8 @@ end # # Stores whatever is under +key+ if it is not the same as - # whats in the persistent db. + # whats in the persistent db - if the +key+ actually has + # a representation in the db. # # Will call value.save_hook(old_value) and send # it a block that does the actual saving if @@ -116,11 +117,13 @@ @lock.synchronize_on(key) do serialized_key = Marshal.dump(key) + value = @content[key] serialized_value = Marshal.dump(value) + old_serialized_value = @content_db[serialized_key] + + write_to_db(key, serialized_key, serialized_value, value) if old_serialized_value && old_serialized_value != serialized_value - write_to_db(key, serialized_key, serialized_value, value) if @content_db[serialized_key] != serialized_value - end end # From nobody at rubyforge.org Tue Nov 28 22:02:32 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 28 Nov 2006 22:02:32 -0500 (EST) Subject: [Archipelago-submits] [80] trunk/archipelago/lib/archipelago/current.rb: made callables in ThreadedCollection take splatted arguments Message-ID: <20061129030232.2F77F5241B72@rubyforge.org> Revision: 80 Author: zond Date: 2006-11-28 22:02:31 -0500 (Tue, 28 Nov 2006) Log Message: ----------- made callables in ThreadedCollection take splatted arguments Modified Paths: -------------- trunk/archipelago/lib/archipelago/current.rb Modified: trunk/archipelago/lib/archipelago/current.rb =================================================================== --- trunk/archipelago/lib/archipelago/current.rb 2006-11-29 02:54:23 UTC (rev 79) +++ trunk/archipelago/lib/archipelago/current.rb 2006-11-29 03:02:31 UTC (rev 80) @@ -53,7 +53,7 @@ self.each do |args| threads << Thread.new do if callable - callable.call(args) + callable.call(*args) else yield(args) end @@ -76,7 +76,7 @@ self.t_each do |args| result.synchronize do if callable - result << callable.call(args) + result << callable.call(*args) else result << yield(args) end @@ -95,7 +95,7 @@ result.extend(Synchronized) self.t_each do |args| if callable - matches = callable.call(args) + matches = callable.call(*args) else matches = yield(args) end @@ -116,7 +116,7 @@ result.extend(Synchronized) self.t_each do |args| if callable - matches = callable.call(args) + matches = callable.call(*args) else matches = yield(args) end From nobody at rubyforge.org Tue Nov 28 22:05:49 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Tue, 28 Nov 2006 22:05:49 -0500 (EST) Subject: [Archipelago-submits] [81] trunk/archipelago/lib/archipelago: removed redundant code Message-ID: <20061129030549.AF16B5241B74@rubyforge.org> Revision: 81 Author: zond Date: 2006-11-28 22:05:49 -0500 (Tue, 28 Nov 2006) Log Message: ----------- removed redundant code Modified Paths: -------------- trunk/archipelago/lib/archipelago/disco.rb trunk/archipelago/lib/archipelago/treasure.rb Modified: trunk/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-29 03:02:31 UTC (rev 80) +++ trunk/archipelago/lib/archipelago/disco.rb 2006-11-29 03:05:49 UTC (rev 81) @@ -209,11 +209,7 @@ # def method_missing(meth, *args, &block) if @attributes.respond_to?(meth) - if block - @attributes.send(meth, *args, &block) - else - @attributes.send(meth, *args) - end + @attributes.send(meth, *args, &block) else super(*args) end @@ -296,11 +292,7 @@ def method_missing(meth, *args, &block) if @hash.respond_to?(meth) synchronize do - if block - @hash.send(meth, *args, &block) - else - @hash.send(meth, *args) - end + @hash.send(meth, *args, &block) end else super(meth, *args, &block) Modified: trunk/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-29 03:02:31 UTC (rev 80) +++ trunk/archipelago/lib/archipelago/treasure.rb 2006-11-29 03:05:49 UTC (rev 81) @@ -654,11 +654,11 @@ if instance.respond_to?(:with_transaction) return_value = nil instance.with_transaction(transaction) do - return_value = execute(instance, method, *arguments, &block) + return_value = instance.send(method, *arguments, &block) end return return_value else - return execute(instance, method, *arguments, &block) + return instance.send(method, *arguments, &block) end ensure # @@ -673,17 +673,6 @@ end # - # Execute +m+ with arguments +a+ and block +b+ on +o+. - # - def execute(o, m, *a, &b) - if b - return o.send(m, *a, &b) - else - return o.send(m, *a) - end - end - - # # Call a method outside any transaction (ie inside a transaction of its own). # def call_without_transaction(key, method, *arguments, &block) @@ -692,7 +681,7 @@ raise UnknownObjectException(self, key) unless instance begin - return execute(instance, method, *arguments, &block) + return instance.send(method, *arguments, &block) ensure @db.store_if_changed(key) end From nobody at rubyforge.org Wed Nov 29 00:37:04 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Wed, 29 Nov 2006 00:37:04 -0500 (EST) Subject: [Archipelago-submits] [82] trunk/archipelago: fixed more tests for Current::ThreadedCollection and cleaned it up a lot Message-ID: <20061129053704.B10DE52409A5@rubyforge.org> Revision: 82 Author: zond Date: 2006-11-29 00:37:03 -0500 (Wed, 29 Nov 2006) Log Message: ----------- fixed more tests for Current::ThreadedCollection and cleaned it up a lot Modified Paths: -------------- trunk/archipelago/lib/archipelago/current.rb trunk/archipelago/tests/current_test.rb Modified: trunk/archipelago/lib/archipelago/current.rb =================================================================== --- trunk/archipelago/lib/archipelago/current.rb 2006-11-29 03:05:49 UTC (rev 81) +++ trunk/archipelago/lib/archipelago/current.rb 2006-11-29 05:37:03 UTC (rev 82) @@ -52,11 +52,7 @@ self.each do |args| threads << Thread.new do - if callable - callable.call(*args) - else - yield(args) - end + call_helper(callable, args, &block) end end @@ -74,12 +70,9 @@ result = [] result.extend(Synchronized) self.t_each do |args| + new_value = call_helper(callable, args, &block) result.synchronize do - if callable - result << callable.call(*args) - else - result << yield(args) - end + result << new_value end end return result @@ -94,11 +87,7 @@ result = [] result.extend(Synchronized) self.t_each do |args| - if callable - matches = callable.call(*args) - else - matches = yield(args) - end + matches = call_helper(callable, args, &block) result.synchronize do result << args end if matches @@ -115,11 +104,7 @@ result = [] result.extend(Synchronized) self.t_each do |args| - if callable - matches = callable.call(*args) - else - matches = yield(args) - end + matches = call_helper(callable, args, &block) result.synchronize do result << args end unless matches @@ -127,6 +112,20 @@ return result end + private + + def call_helper(o = nil, args = nil, &block) + if o + if Array === args + return o.call(*args) + else + return o.call(args) + end + else + return yield(args) + end + end + end Modified: trunk/archipelago/tests/current_test.rb =================================================================== --- trunk/archipelago/tests/current_test.rb 2006-11-29 03:05:49 UTC (rev 81) +++ trunk/archipelago/tests/current_test.rb 2006-11-29 05:37:03 UTC (rev 82) @@ -1,6 +1,22 @@ require File.join(File.dirname(__FILE__), 'test_helper') +class HashCollector + attr_accessor :epa + def call(k,v) + @epa ||= {} + @epa[k] = v + end +end + +class ArrayCollector + attr_accessor :epa + def call(e) + @epa ||= [] + @epa << e + end +end + class CurrentTest < Test::Unit::TestCase def test_synchronized @@ -24,6 +40,24 @@ end end + def test_lock_on + t = true + + a = "hej" + a.extend(Archipelago::Current::Synchronized) + a.lock_on("epa") + Thread.new do + a.lock_on("epa") + t = false + end + Thread.pass + assert(t) + a.unlock_on("epa") + assert_within(0.5) do + !t + end + end + def test_threaded_collection a = Array(10) a.extend(Archipelago::Current::ThreadedCollection) @@ -51,6 +85,16 @@ b = {} a.t_each do |k,v| b[k] = v end assert_equal(b,a) + + c = HashCollector.new + a.t_each(c) + assert_equal(a, c.epa) + + a = [1,2,3,4,5,6,7] + a.extend(Archipelago::Current::ThreadedCollection) + c = ArrayCollector.new + a.t_each(c) + assert_equal(a.sort, c.epa.sort) end end From nobody at rubyforge.org Wed Nov 29 00:42:35 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Wed, 29 Nov 2006 00:42:35 -0500 (EST) Subject: [Archipelago-submits] [83] trunk/hyperactive: totally revamped the Tree and renamed it Hash. Message-ID: <20061129054235.4BF3D524099E@rubyforge.org> Revision: 83 Author: zond Date: 2006-11-29 00:42:34 -0500 (Wed, 29 Nov 2006) Log Message: ----------- totally revamped the Tree and renamed it Hash. gained theoretical constant time complexity. added lots of tests improved README Modified Paths: -------------- trunk/hyperactive/README trunk/hyperactive/lib/hyperactive/list.rb trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive.rb trunk/hyperactive/tests/list_test.rb trunk/hyperactive/tests/record_test.rb Added Paths: ----------- trunk/hyperactive/lib/hyperactive/hash.rb trunk/hyperactive/tests/hash_benchmark.rb trunk/hyperactive/tests/hash_test.rb trunk/hyperactive/tests/list_benchmark.rb Removed Paths: ------------- trunk/hyperactive/lib/hyperactive/tree.rb trunk/hyperactive/tests/tree_benchmark.rb trunk/hyperactive/tests/tree_test.rb Modified: trunk/hyperactive/README =================================================================== --- trunk/hyperactive/README 2006-11-29 05:37:03 UTC (rev 82) +++ trunk/hyperactive/README 2006-11-29 05:42:34 UTC (rev 83) @@ -2,14 +2,11 @@ It uses archipelago for persistence, and is meaningful only in an environment where the server process doesnt restart at each request. This means that cgi environment is not really an option. -== Dependencies: - -Hyperactive::Tree:: RBTree: http://raa.ruby-lang.org/project/ruby-rbtree - == Sub packages: Hyperactive::Record:: The base class package itself, providing you with cached selectors and rejectors, along with cached finders for any number of attributes. -Hyperactive::Tree:: A collection class that contains any number of Hyperactive::Records in a tree structure. Scales logarithmically, so use with care. +Hyperactive::Hash:: A collection class that contains any number of Hyperactive::Records in a hash like structure. +Hyperactive::List:: A collection class that contains any number of Hyperactive::Records in a list like structure. == Usage: Copied: trunk/hyperactive/lib/hyperactive/hash.rb (from rev 73, trunk/hyperactive/lib/hyperactive/tree.rb) =================================================================== --- trunk/hyperactive/lib/hyperactive/hash.rb (rev 0) +++ trunk/hyperactive/lib/hyperactive/hash.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -0,0 +1,172 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + + +require 'rubygems' +require 'archipelago' +require 'digest/sha1' + +module Hyperactive + + # + # The package containing the Hash class that provides any + # kind of index for your Hyperactive classes. + # + # Is supposed to be constantly scaling, but preliminary benchmarks show some + # problems with that assumption? + # + module Hash + + # + # A wrapper class that knows what key, value and list_element belong together. + # + class Element < Hyperactive::Record::Bass + attr_accessor :key, :value, :list_element + # + # Initialize a new Hash::Element with given +key+, +value+ and +list_element+. + # + def initialize(key, value, list_element) + self.key = key + self.value = value + self.list_element = list_element + end + end + + # + # A class suitable for storing large and often-changing datasets in + # an Archipelago environment. + # + class Head < Hyperactive::Record::Bass + + attr_accessor :list + + include Archipelago::Current::ThreadedCollection + + # + # Initialize a Hash::Head. + # + # NB: Remember to call create on the new instance or use Head.get_instance to get that done automatically. + # + def initialize + self.list = nil + end + + # + # Return the size of this Hash. + # + def size + self.list.size + end + + # + # Return the value for +key+. + # + def [](key) + element = Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction] + if element + return element.value + else + return nil + end + end + + # + # Returns whether +key+ is included in this Hash. + # + def include?(key) + Hyperactive::Record::CAPTAIN.include?(my_key_for(key), @transaction) + end + + # + # Insert +value+ under +key+ in this Hash. + # + def []=(key, value) + self.list = Hyperactive::List::Head.get_instance_with_transaction(@transaction) unless self.list + + if (element = Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction]) + element.value = value + else + element = Element.get_instance_with_transaction(@transaction, key, value, nil) + self.list << element + element.list_element = self.list.last_element + Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction] = element + end + end + + # + # Delete +key+ from this Hash. + # + def delete(key) + self.list = Hyperactive::List::Head.get_instance_with_transaction(@transaction) unless self.list + + return_value = nil + + element = Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction] + if element + Hyperactive::Record::CAPTAIN.delete(my_key_for(key), @transaction) + self.list.unlink!(element.list_element) + return_value = element.value + element.destroy! + end + return return_value + end + + # + # Will yield to +block+ once for each key/value pair in this Hash. + # + def each(&block) + self.list = Hyperactive::List::Head.get_instance_with_transaction(@transaction) unless self.list + + self.list.each do |element| + yield([element.key, element.value]) + end + end + + # + # Remove all key/value pairs from this Hash. + # + def clear! + self.list = Hyperactive::List::Head.get_instance_with_transaction(@transaction) unless self.list + + self.list.t_each do |element| + element.destroy! + end + self.list.clear! + end + + # + # Clear everything from this Tree and destroy it. + # + def destroy! + self.clear! + super + end + + private + + # + # Get my private key for a given +key+. + # + def my_key_for(key) + Digest::SHA1.hexdigest("#{Marshal.dump(key)}#{self.record_id}") + end + + end + + end + +end Modified: trunk/hyperactive/lib/hyperactive/list.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/list.rb 2006-11-29 05:37:03 UTC (rev 82) +++ trunk/hyperactive/lib/hyperactive/list.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -27,73 +27,148 @@ module List # - # A List element. + # A wrapper class that knows its previous and next List::Elements as well as its value and the id of its list. # class Element < Hyperactive::Record::Bass - attr_accessor :previous, :next, :value + attr_accessor :previous, :next, :value, :list_id + + include Archipelago::Current::ThreadedCollection + + # + # Initialize this List::Element with given +previous_element+, +next_element+, +value+ and +list_id+. + # + def initialize(previous_element, next_element, value, list_id) + self.previous = previous_element + self.next = next_element + self.value = value + self.list_id = list_id + end + + # + # Yield to +block+ once for this and each following element. + # + def each(&block) + element = self + while element + yield(element) + element = self.next + end + end end # # A List head. # class Head < Hyperactive::Record::Bass - attr_reader :first_element, :last_element, :size + attr_accessor :first_element, :last_element, :size + include Archipelago::Current::ThreadedCollection + # - # Create a Head. + # Create a List::Head. This is in essence the linked list. # - # NB: As usual, dont call this. Use Head.get_instance instead. + # NB: Remember to call create on the new instance or use Head.get_instance to get that done automatically. # def initialize - @size = 0 - @first_element = @last_element = nil + self.size = 0 + self.first_element = self.last_element = nil end # + # Unlinks the given +element+ and reconnects the List around it. + # + def unlink!(element) + raise "#{element} is not a part of #{self}" unless element.list_id == @record_id + + if size > 1 + if element == self.first_element + self.first_element = element.next + self.first_element.previous = nil + elsif element == self.last_element + self.last_element = element.previous + self.last_element.next = nil + else + element.previous.next = element.next + element.next.previous = element.previous + end + else + if element == self.first_element + self.first_element = self.last_element = nil + else + raise "#{element} is not a part of #{self} even though it claims to be" + end + end + self.size -= 1 + element.destroy! + end + + # + # Remove all elements from this List::Head. + # + def clear! + self.first_element.t_each do |element| + element.destroy! + end + self.first_element = self.last_element = nil + self.size = 0 + end + + # + # Destroys this list and all its elements. + # + def destroy! + self.clear! + super + end + + # # Return the first value of the list. # def first - @first_element.value + if self.first_element + self.first_element.value + else + nil + end end - + # # Return the last value of the list. # def last - @last_element.value + if self.last_element + self.last_element.value + else + end end # # Push +v+ onto the end of this list. # def <<(v) - if @first_element - new_element = Element.get_instance - new_element.value = v - new_element.previous = @last_element - @last_element.next = new_element - @last_element = new_element + if self.first_element + new_element = Element.get_instance_with_transaction(@transaction, self.last_element, nil, v, @record_id) + self.last_element.next = new_element + self.last_element = new_element else start(v) end - @size += 1 + self.size += 1 return v end - + # # Push +v+ onto the beginning of this list. # def unshift(v) - if @first_element - new_element = Element.get_instance - new_element.value = v - new_element.next = @first_element - @first_element.previous = new_element - @first_element = new_element + if self.first_element + new_element = Element.get_instance_with_transaction(@transaction, nil, self.first_element, v, @record_id) + self.first_element.previous = new_element + self.first_element = new_element else start(v) end - @size += 1 + self.size += 1 return v end @@ -103,37 +178,48 @@ def pop v = nil if size > 1 - element = @last_element - @last_element = element.previous - @last_element.next = nil + element = self.last_element + self.last_element = element.previous + self.last_element.next = nil v = element.value - element.destroy + element.destroy! else - v = @first_element.value - @first_element.destroy - @first_element = @last_element = nil + v = self.first_element.value + self.first_element.destroy! + self.first_element = self.last_element = nil end - @size -= 1 + self.size -= 1 return v end # + # Yield to +block+ once for each value in this list. + # + def each(&block) + element = self.first_element + while element + yield(element.value) + element = element.next + end + end + + # # Remove the first value from this list and return it. # def shift v = nil if size > 1 - element = @first_element - @first_element = element.next - @first_element.previous = nil + element = self.first_element + self.first_element = element.next + self.first_element.previous = nil v = element.value - element.destroy + element.destroy! else - v = @first_element.value - @first_element.destroy - @first_element = @last_element = nil + v = self.first_element.value + self.first_element.destroy! + self.first_element = self.last_element = nil end - @size -= 1 + self.size -= 1 return v end @@ -143,8 +229,7 @@ # Start this list with +v+ when it has no other values. # def start(v) - @first_element = @last_element = Element.get_instance - @first_element.value = v + self.first_element = self.last_element = Element.get_instance_with_transaction(@transaction, nil, nil, v, @record_id) end end Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-29 05:37:03 UTC (rev 82) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -61,7 +61,7 @@ @attributes = attributes end # - # Get the Tree for the given +record+. + # Get the key for the given +record+. # def get_key_for(record) values = @attributes.collect do |att| @@ -92,12 +92,12 @@ old_key = get_key_for(old_record) new_key = get_key_for(record) if old_key != new_key - (CAPTAIN[old_key] ||= Hyperactive::Tree::Root.get_instance).delete(record.record_id) - (CAPTAIN[new_key] ||= Hyperactive::Tree::Root.get_instance)[record.record_id] = record + (CAPTAIN[old_key] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction)).delete(record.record_id) + (CAPTAIN[new_key] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction))[record.record_id] = record end else record = argument - (CAPTAIN[get_key_for(record)] ||= Hyperactive::Tree::Root.get_instance).delete(record.record_id) + (CAPTAIN[get_key_for(record)] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction)).delete(record.record_id) end end end @@ -130,23 +130,23 @@ case @mode when :select if @matcher.call(record) - CAPTAIN[@key][record.record_id] = record + CAPTAIN[@key, record.transaction][record.record_id] = record else - CAPTAIN[@key].delete(record.record_id) + CAPTAIN[@key, record.transaction].delete(record.record_id) end when :reject if @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) + CAPTAIN[@key, record.transaction].delete(record.record_id) else - CAPTAIN[@key][record.record_id] = record + CAPTAIN[@key, record.transaction][record.record_id] = record end when :delete_if_match if @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) + CAPTAIN[@key, record.transaction].delete(record.record_id) end when :delete_unless_match unless @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) + CAPTAIN[@key, record.transaction].delete(record.record_id) end end end @@ -156,20 +156,19 @@ # A convenient base class to inherit when you want the basic utility methods # provided by for example ActiveRecord::Base *hint hint*. # - # NB: After an instance is created, it will actually return a copy within your local machine - # which is not what is usually the case. Every other time you fetch it using a select or other + # NB: When an instance is created you will actually have a copy within your local machine + # which is not what you usually want. Every other time you fetch it using a select or other # method you will instead receive a proxy object to the database. This means that nothing you # do to it at that point will be persistent or even necessarily have a defined result. - # Therefore: do not use MyRecordSubclass.new to MyRecordSubclass#initialize objects, - # instead use MyRecordSubclass.get_instance, since it will return a fresh proxy object - # instead of the devious original: - # my_instance = MyRecordSubclass.get_instance(*the_same_arguments_as_to_initialize) + # Therefore: do not use the instantiated object, instead call my_instance.save + # to get a proxy to the object stored into the database. # class Bass @@create_hooks_by_class = {} @@destroy_hooks_by_class = {} @@save_hooks_by_class = {} + @@load_hooks_by_class = {} # # The host we are running on. @@ -185,6 +184,50 @@ end # + # Works like normal attr_reader but with transactional awareness. + # + def self.attr_reader(*attributes) + attributes.each do |attribute| + define_method(attribute) do + value = instance_variable_get("@#{attribute}") + if Archipelago::Treasure::Dubloon === value + if @transaction ||= nil + return value.join(@transaction) + else + return value + end + else + return value + end + end + end + end + + # + # Works like normal attr_writer but with transactional awareness. + # + def self.attr_writer(*attributes) + attributes.each do |attribute| + define_method("#{attribute}=") do |new_value| + if Archipelago::Treasure::Dubloon === new_value + new_value.assert_transaction(@transaction) if @transaction ||= nil + instance_variable_set("@#{attribute}", new_value) + else + instance_variable_set("@#{attribute}", new_value) + end + end + end + end + + # + # Works like normal attr_accessor but with transactional awareness. + # + def self.attr_accessor(*attributes) + attr_reader(*attributes) + attr_writer(*attributes) + end + + # # Return our create_hooks, which can then be treated as any old Array. # # These must be callable objects with an arity of 1 @@ -219,6 +262,23 @@ end # + # Return our load_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent the instance about to be loaded (insertion + # into the live hash of the database in question) that take a block argument. + # + # The block argument will be a Proc that actually puts the + # instance into the live hash. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon loading. + # + def self.load_hooks + self.get_hook_array_by_class(@@load_hooks_by_class) + end + + # # Return our save_hooks, which can then be treated as any old Array. # # These must be callable objects with an arity of 1 @@ -263,7 +323,7 @@ # def self.select(name, &block) key = self.collection_key(name) - CAPTAIN[key] ||= Hyperactive::Tree::Root.get_instance + CAPTAIN[key] ||= Hyperactive::Hash::Head.get_instance self.class_eval <(o) + if Record === o + @record_id <=> o.record_id + else + 0 end + end + + # + # Save this Record instance into the distributed database and return a proxy to the saved object. + # + # This will also wrap the actual insertion within the create_hooks you have defined for this class. + # + def create + @record_id ||= Digest::SHA1.hexdigest("#{HOST}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}").to_s + @transaction ||= nil - Hyperactive::Hooker.call_with_hooks(instance, *self.create_hooks) do - CAPTAIN[instance.record_id] = instance + Hyperactive::Hooker.call_with_hooks(self, *self.class.create_hooks) do + CAPTAIN[self.record_id, @transaction] = self end - proxy = CAPTAIN[instance.record_id] + proxy = CAPTAIN[@record_id, @transaction] return proxy end - + # - # Our semi-unique id. + # Will execute +block+ within a transaction. # - attr_reader :record_id - + # What it does is just set the @transaction instance variable + # before calling the block, and unsetting it after. # + # This means that any classes that want to be transaction sensitive + # need to take heed regarding the @transaction instance variable. + # + # For example, when creating new Record instances you may want to use + # get_instance_with_transaction(@transaction, *args) to ensure that the + # new instance exists within the same transaction as yourself. + # + # See Hyperactive::List::Head and Hyperactive::Hash::Head for examples of this behaviour. + # + def with_transaction(transaction, &block) + @transaction = transaction + begin + yield + ensure + @transaction = nil + end + end + + # # This will allow us to wrap any write of us to persistent storage # in the @@save_hooks as long as the Archipelago::Hashish provider # supports it. See Archipelago::Hashish::BerkeleyHashish for an example @@ -351,6 +457,18 @@ end # + # This will allow us to wrap any load of us from persistent storage + # in the @@load_hooks as long as the Archipelago::Hashish provider + # supports it. See Archipelago::Hashish::BerkeleyHashish for an example + # of Hashish providers that do this. + # + def load_hook(&block) + Hyperactive::Hooker.call_with_hooks(self, *self.class.load_hooks) do + yield + end + end + + # # Remove this instance from the database calling all the right hooks. # # Freezes this instance after having deleted it. @@ -360,9 +478,9 @@ # # Returns true otherwise. # - def destroy + def destroy! Hyperactive::Hooker.call_with_hooks(self, *self.class.destroy_hooks) do - CAPTAIN.delete(@record_id) + CAPTAIN.delete(@record_id, @transaction) self.freeze end end Deleted: trunk/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-29 05:37:03 UTC (rev 82) +++ trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -1,241 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - - -require 'rubygems' -require 'archipelago' -require 'rbtree' - -module Hyperactive - - # - # The package containing the Hash-like Tree class that provides any - # kind of index for your Hyperactive classes. - # - module Tree - - # - # A class suitable for storing large and often-changing datasets in - # an Archipelago environment. - # - # Is constructed like a set of nested Hashes that automatically create - # new children on demand, and will thusly only have to check the path from - # the root node to the leaf for changes when method calls return (see Archipelago::Treasure::Chest) - # and will only have to actually store into database the leaf itself if it - # has changed. - # - class Root < Hyperactive::Record::Bass - - attr_accessor :elements, :subtrees - - WIDTH = 1 << 3 - - # - # Dont call this! Call Root.get_instance(options) instead! - # - def initialize(options = {}) - @width = options[:width] || WIDTH - @elements = {} - @subtrees = nil - end - - # - # Deletes +key+ in this Root. - # - def delete(key) - if @elements - @elements.delete(key) - else - subtree_for(key).delete(key) - end - end - - # - # Returns the size of this Root. - # - def size - if @elements - @elements.size - else - @subtrees.t_collect do |tree_id, tree| - tree.size - end.inject(0) do |sum, size| - sum + size - end - end - end - - # - # Returns all keys and values returning true for +callable+.call(key, value) in this Root. - # - def select(callable) - if @elements - @elements.select do |k,v| - callable.call(k,v) - end - else - @subtrees.t_collect do |tree_id, tree| - tree.select(callable) - end.inject([]) do |sum, match| - sum + match - end - end - end - - # - # Returns all keys and values returning false for +callable+.call(key, value) in this Root. - # - def reject(callable) - if @elements - @elements.reject do |k,v| - callable.call(k,v) - end - else - @subtrees.t_collect do |tree_id, tree| - tree.reject(callable) - end.inject([]) do |sum, match| - sum + match - end - end - end - - # - # Puts +value+ under +key+ in this Root. - # - def []=(key, value) - if @elements - if @elements.size < @width - @elements[key] = value - else - split! - subtree_for(key)[key] = value - end - else - subtree_for(key)[key] = value - end - return value - end - - # - # Returns the value for +key+ in this Root. - # - def [](key) - if @elements - return @elements[key] - else - return subtree_for(key)[key] - end - end - - # - # Returns an Array containing +callable+.call(key, value) from all values in this Root. - # - def collect(callable) - rval = [] - self.each(Proc.new do |k,v| - rval << callable.call(k,v) - end) - return rval - end - - # - # Does +callable+.call(key, value) on all values in this Root. - # - def each(callable) - if @elements - @elements.each do |key, value| - callable.call(key, value) - end - else - @subtrees.t_each do |tree_id, tree| - tree.each(callable) - end - nil - end - end - - # - # Clear everything from this Tree and destroy it. - # - def destroy - if @elements - @elements.each do |key, value| - value.destroy if value.respond_to?(:destroy) - end - else - @subtrees.each do |tree_id, tree| - tree.destroy - end - end - freeze - super - end - - # - # Clear everything from this Root. - # - def clear - unless @elements - @subtrees.each do |tree_id, tree| - tree.clear - end - @subtrees = nil - end - @elements = {} - end - - private - - # - # Finds the subtree responsible for +key+. - # - # Does it in this ugly way cause the nice way of just doing modulo gave - # really odd results since all hashes seem to give some modulo values - # a lot more often than expected. - # - def subtree_for(key) - key_id = Digest::SHA1.new("#{key.hash}#{self.record_id}").to_s - @subtrees.each do |tree_id, tree| - return tree if tree_id > key_id - end - return @subtrees.values.first - end - - # - # Split this Tree by creating @subtrees, - # then putting all @elements in them and then emptying @elements. - # - def split! - raise "Cant split twice!" unless @elements - - @subtrees = RBTree.new - @subtrees.extend(Archipelago::Current::ThreadedCollection) - step = (1 << 160) / @width - 0.upto(@width - 1) do |n| - @subtrees["%x" % (n * step)] = Root.get_instance(:width => @width) - end - @elements.each do |key, value| - subtree_for(key)[key] = value - end - @elements = nil - end - - end - - end - -end Modified: trunk/hyperactive/lib/hyperactive.rb =================================================================== --- trunk/hyperactive/lib/hyperactive.rb 2006-11-29 05:37:03 UTC (rev 82) +++ trunk/hyperactive/lib/hyperactive.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -19,5 +19,5 @@ require 'hyperactive/hooker' require 'hyperactive/record' -require 'hyperactive/tree' +require 'hyperactive/hash' require 'hyperactive/list' Copied: trunk/hyperactive/tests/hash_benchmark.rb (from rev 73, trunk/hyperactive/tests/tree_benchmark.rb) =================================================================== --- trunk/hyperactive/tests/hash_benchmark.rb (rev 0) +++ trunk/hyperactive/tests/hash_benchmark.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -0,0 +1,66 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class HashBenchmark < Test::Unit::TestCase + + def setup + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + assert_within(20) do + Set.new(Hyperactive::Record::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) + end + assert_within(20) do + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + end + end + + def teardown + @c.stop! + @c.persistence_provider.unlink! + @c2.stop! + @c2.persistence_provider.unlink! + @tm.stop! + @tm.persistence_provider.unlink! + end + + def test_set_get + h = Hyperactive::Hash::Head.get_instance + r = Hyperactive::Record::Bass.get_instance + hash_test("Hyperactive::Hash", h, r, 100) + end + + def test_regular_hash_set_get + Hyperactive::Record::CAPTAIN["h"] = {} + h = Hyperactive::Record::CAPTAIN["h"] + r = Hyperactive::Record::Bass.get_instance + hash_test("Hash", h, r, 100) + end + + private + + def hash_test(label, h, r, c) + bm("#{label}#set/get/delete", :n => c * 1) do + f = rand(1 << 32) + h[f] = r + x = h[f] + h.delete(f) + end + bm("#{label}#set", :n => c * 10) do + f = rand(1 << 32) + h[f] = r + end + bm("#{label}#set/get/delete (big)", :n => c * 1) do + f = rand(1 << 32) + h[f] = r + x = h[f] + h.delete(f) + end + end + +end Copied: trunk/hyperactive/tests/hash_test.rb (from rev 73, trunk/hyperactive/tests/tree_test.rb) =================================================================== --- trunk/hyperactive/tests/hash_test.rb (rev 0) +++ trunk/hyperactive/tests/hash_test.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -0,0 +1,76 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class RecordMatcher + def initialize(i) + @i = i + end + def call(k,v) + k == @i + end +end + +class HashTest < Test::Unit::TestCase + + def setup + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + Hyperactive::Record::CAPTAIN.update_services! + assert_within(10) do + Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort + end + assert_within(10) do + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + end + end + + def teardown + @c.stop! + @c.persistence_provider.unlink! + @c2.stop! + @c2.persistence_provider.unlink! + @tm.stop! + @tm.persistence_provider.unlink! + end + + def test_select_reject + h = Hyperactive::Hash::Head.get_instance + r1 = Hyperactive::Record::Bass.get_instance + r2 = Hyperactive::Record::Bass.get_instance + h[r1.record_id] = r1 + h[r2.record_id] = r2 + assert_equal(r1.record_id, h.t_select(RecordMatcher.new(r1.record_id)).first.first) + assert_equal(r1.record_id, h.t_reject(RecordMatcher.new(r2.record_id)).first.first) + end + + def test_delete + h = Hyperactive::Hash::Head.get_instance + r = Hyperactive::Record::Bass.get_instance + h[r.record_id] = r + assert(h.include?(r.record_id)) + h.delete(r.record_id) + assert(!h.include?(r.record_id)) + end + + def test_set_get + h = Hyperactive::Hash::Head.get_instance + h2 = {} + 10.times do + r = Hyperactive::Record::Bass.get_instance + h[r.record_id] = r + h2[r.record_id] = r + end + + h2.each do |k,v| + assert_equal(v, h[k]) + assert_equal(v, Hyperactive::Record::CAPTAIN[h.record_id][k]) + end + end + +end Added: trunk/hyperactive/tests/list_benchmark.rb =================================================================== --- trunk/hyperactive/tests/list_benchmark.rb (rev 0) +++ trunk/hyperactive/tests/list_benchmark.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -0,0 +1,48 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class ListBenchmark < Test::Unit::TestCase + + def setup + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + assert_within(20) do + Set.new(Hyperactive::Record::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) + end + assert_within(20) do + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + end + end + + def teardown + @c.stop! + @c.persistence_provider.unlink! + @c2.stop! + @c2.persistence_provider.unlink! + @tm.stop! + @tm.persistence_provider.unlink! + end + + def test_set_get + l = Hyperactive::List::Head.get_instance + r = Hyperactive::Record::Bass.get_instance + bm("List::Head#push/pop", :n => 1000) do + l << r + l.pop + end + bm("List::Head#push", :n => 1000) do + l << r + end + bm("List::Head#push/pop (big)", :n => 1000) do + l << r + l.pop + end + end + +end Modified: trunk/hyperactive/tests/list_test.rb =================================================================== --- trunk/hyperactive/tests/list_test.rb 2006-11-29 05:37:03 UTC (rev 82) +++ trunk/hyperactive/tests/list_test.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -1,6 +1,14 @@ require File.join(File.dirname(__FILE__), 'test_helper') +class Collector + attr_accessor :es + def call(e) + @es ||= [] + @es << e + end +end + class ListTest < Test::Unit::TestCase def setup @@ -23,21 +31,71 @@ def teardown @c.stop! - @c.persistence_provider.unlink + @c.persistence_provider.unlink! @c2.stop! - @c2.persistence_provider.unlink + @c2.persistence_provider.unlink! @tm.stop! - @tm.persistence_provider.unlink - Archipelago::Disco::MC.clear! - Hyperactive::Record::CAPTAIN.update_services! - assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.empty? - end - assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.empty? - end + @tm.persistence_provider.unlink! end + def test_clear + l = Hyperactive::List::Head.get_instance + l << (r = Hyperactive::Record::Bass.get_instance) + assert_equal(1, l.size) + assert_equal(r, l.first) + assert_equal(r, l.last) + l.clear! + assert_equal(0, l.size) + assert_equal(nil, l.first) + assert_equal(nil, l.last) + end + + def test_destroy + l = Hyperactive::List::Head.get_instance + l << (r = Hyperactive::Record::Bass.get_instance) + r1 = l.first_element.record_id + r2 = l.record_id + l.destroy! + assert(!Hyperactive::Record::CAPTAIN.include?(r1)) + assert(!Hyperactive::Record::CAPTAIN.include?(r2)) + end + + def test_each + l = Hyperactive::List::Head.get_instance + l << (r = Hyperactive::Record::Bass.get_instance) + l << (r2 = Hyperactive::Record::Bass.get_instance) + l << (r3 = Hyperactive::Record::Bass.get_instance) + c = Collector.new + l.t_each(c) + assert_equal(3, c.es.size) + assert(c.es.include?(r)) + assert(c.es.include?(r2)) + assert(c.es.include?(r3)) + end + + def test_unlink + l = Hyperactive::List::Head.get_instance + l << (r = Hyperactive::Record::Bass.get_instance) + l << (r2 = Hyperactive::Record::Bass.get_instance) + e = l.last_element + l << (r3 = Hyperactive::Record::Bass.get_instance) + assert_equal([r, r2, r3].sort, l.t_collect do |ec| ec end.sort) + l.unlink!(e) + assert_equal(2, l.size) + assert_equal(r, l.first) + assert_equal(r3, l.last) + assert_equal(r3, l.first_element.next.value) + l.unlink!(l.last_element) + assert_equal(1, l.size) + assert_equal(r, l.first) + assert_equal(r, l.last) + l.unlink!(l.first_element) + assert_equal(0, l.size) + assert_equal(nil, l.first) + assert_equal(nil, l.last) + assert_equal([], l.t_collect do |e| e end) + end + def test_push_unshift_pop_shift l = Hyperactive::List::Head.get_instance assert_equal(0, l.size) Modified: trunk/hyperactive/tests/record_test.rb =================================================================== --- trunk/hyperactive/tests/record_test.rb 2006-11-29 05:37:03 UTC (rev 82) +++ trunk/hyperactive/tests/record_test.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -3,7 +3,9 @@ class MyRecord < Hyperactive::Record::Bass attr_accessor :bajs - + def initialize + @bajs = nil + end def self.save_hook(instance, &block) $BEFORE_SAVE += 1 yield @@ -54,19 +56,11 @@ def teardown @c.stop! - @c.persistence_provider.unlink + @c.persistence_provider.unlink! @c2.stop! - @c2.persistence_provider.unlink + @c2.persistence_provider.unlink! @tm.stop! - @tm.persistence_provider.unlink - Archipelago::Disco::MC.clear! - Hyperactive::Record::CAPTAIN.update_services! - assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.empty? - end - assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.empty? - end + @tm.persistence_provider.unlink! end def test_index_find @@ -82,18 +76,18 @@ r4 = MyRecord.get_instance r4.bajs = "beige" - assert_equal(Set.new([r2.record_id,r1.record_id]), Set.new(MyRecord.find_by_bajs("brunt").collect(Proc.new do |k,v| - v.record_id - end))) - assert_equal([r3.record_id], MyRecord.find_by_bajs("gult").collect(Proc.new do |k,v| - v.record_id - end)) + assert_equal(Set.new([r2.record_id,r1.record_id]), Set.new(MyRecord.find_by_bajs("brunt").t_collect(Proc.new do |k,v| + v.record_id + end))) + assert_equal([r3.record_id], MyRecord.find_by_bajs("gult").t_collect(Proc.new do |k,v| + v.record_id + end)) - r1.destroy + r1.destroy! - assert_equal([r2.record_id], MyRecord.find_by_bajs("brunt").collect(Proc.new do |k,v| - v.record_id - end)) + assert_equal([r2.record_id], MyRecord.find_by_bajs("brunt").t_collect(Proc.new do |k,v| + v.record_id + end)) end @@ -115,22 +109,22 @@ r4 = MyRecord.get_instance r4.bajs = "beige" - assert_equal(Set.new([r2.record_id,r1.record_id]), Set.new(MyRecord.brunt.collect(Proc.new do |k,v| + assert_equal(Set.new([r2.record_id,r1.record_id]), Set.new(MyRecord.brunt.t_collect(Proc.new do |k,v| v.record_id end))) - assert_equal(Set.new([r3.record_id,r4.record_id]), Set.new(MyRecord.not_brunt.collect(Proc.new do |k,v| + assert_equal(Set.new([r3.record_id,r4.record_id]), Set.new(MyRecord.not_brunt.t_collect(Proc.new do |k,v| v.record_id end))) - r1.destroy - r2.destroy - r3.destroy - r4.destroy + r1.destroy! + r2.destroy! + r3.destroy! + r4.destroy! - assert_equal(Set.new, Set.new(MyRecord.brunt.collect(Proc.new do |k,v| + assert_equal(Set.new, Set.new(MyRecord.brunt.t_collect(Proc.new do |k,v| v.record_id end))) - assert_equal(Set.new, Set.new(MyRecord.not_brunt.collect(Proc.new do |k,v| + assert_equal(Set.new, Set.new(MyRecord.not_brunt.t_collect(Proc.new do |k,v| v.record_id end))) end @@ -149,7 +143,7 @@ assert_equal("brunt", Hyperactive::Record::CAPTAIN[r.record_id].bajs) i = r.record_id assert_equal(r, Hyperactive::Record::CAPTAIN[i]) - r.destroy + r.destroy! assert_equal(1, $BEFORE_DESTROY) assert_equal(1, $AFTER_DESTROY) assert_equal(nil, Hyperactive::Record::CAPTAIN[i]) Deleted: trunk/hyperactive/tests/tree_benchmark.rb =================================================================== --- trunk/hyperactive/tests/tree_benchmark.rb 2006-11-29 05:37:03 UTC (rev 82) +++ trunk/hyperactive/tests/tree_benchmark.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -1,66 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'test_helper') - -class TreeBenchmark < Test::Unit::TestCase - - def setup - @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) - @c.publish! - @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) - @c2.publish! - @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) - @tm.publish! - Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, - :tranny_description => {:class => 'TestManager'}) - assert_within(20) do - Set.new(Hyperactive::Record::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) - end - assert_within(20) do - Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] - end - end - - def teardown - @c.stop! - @c.persistence_provider.unlink - @c2.stop! - @c2.persistence_provider.unlink - @tm.stop! - @tm.persistence_provider.unlink - end - - def test_set_get - h = Hyperactive::Tree::Root.get_instance - r = Hyperactive::Record::Bass.get_instance - hash_test("Tree", h, r, 100) - end - - def test_regular_hash_set_get - Hyperactive::Record::CAPTAIN["h"] = {} - h = Hyperactive::Record::CAPTAIN["h"] - r = Hyperactive::Record::Bass.get_instance - hash_test("Hash", h, r, 100) - end - - private - - def hash_test(label, h, r, c) - bm("#{label}#set/get/delete", :n => c * 1) do - f = rand(1 << 32) - h[f] = r - x = h[f] - h.delete(f) - end - bm("#{label}#set", :n => c * 10) do - f = rand(1 << 32) - h[f] = r - end - bm("#{label}#set/get/delete (big)", :n => c * 1) do - f = rand(1 << 32) - h[f] = r - x = h[f] - h.delete(f) - end - end - -end Deleted: trunk/hyperactive/tests/tree_test.rb =================================================================== --- trunk/hyperactive/tests/tree_test.rb 2006-11-29 05:37:03 UTC (rev 82) +++ trunk/hyperactive/tests/tree_test.rb 2006-11-29 05:42:34 UTC (rev 83) @@ -1,75 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'test_helper') - -class RecordMatcher - def initialize(i) - @i = i - end - def call(k,v) - k == @i - end -end - -class TreeTest < Test::Unit::TestCase - - def setup - @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) - @c.publish! - @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) - @c2.publish! - @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) - @tm.publish! - Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, - :tranny_description => {:class => 'TestManager'}) - Hyperactive::Record::CAPTAIN.update_services! - assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort - end - assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] - end - end - - def teardown - @c.stop! - @c.persistence_provider.unlink - @c2.stop! - @c2.persistence_provider.unlink - @tm.stop! - @tm.persistence_provider.unlink - Archipelago::Disco::MC.clear! - Hyperactive::Record::CAPTAIN.update_services! - assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.empty? - end - assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.empty? - end - end - - def test_select_reject - h = Hyperactive::Tree::Root.get_instance - r1 = Hyperactive::Record::Bass.get_instance - r2 = Hyperactive::Record::Bass.get_instance - h[r1.record_id] = r1 - h[r2.record_id] = r2 - assert_equal(r1.record_id, h.select(RecordMatcher.new(r1.record_id)).first.first) - assert_equal(r1.record_id, h.reject(RecordMatcher.new(r2.record_id)).keys.first) - end - - def test_set_get - h = Hyperactive::Tree::Root.get_instance - h2 = {} - 10.times do - r = Hyperactive::Record::Bass.get_instance - h[r.record_id] = r - h2[r.record_id] = r - end - - h2.each do |k,v| - assert_equal(v, h[k]) - assert_equal(v, Hyperactive::Record::CAPTAIN[h.record_id][k]) - end - end - -end From nobody at rubyforge.org Wed Nov 29 00:43:14 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Wed, 29 Nov 2006 00:43:14 -0500 (EST) Subject: [Archipelago-submits] [84] trunk: created release 0.2.3 Message-ID: <20061129054314.51F98524097C@rubyforge.org> Revision: 84 Author: zond Date: 2006-11-29 00:43:14 -0500 (Wed, 29 Nov 2006) Log Message: ----------- created release 0.2.3 Modified Paths: -------------- trunk/archipelago/Rakefile trunk/hyperactive/Rakefile Modified: trunk/archipelago/Rakefile =================================================================== --- trunk/archipelago/Rakefile 2006-11-29 05:42:34 UTC (rev 83) +++ trunk/archipelago/Rakefile 2006-11-29 05:43:14 UTC (rev 84) @@ -9,7 +9,7 @@ spec = Gem::Specification.new do |s| s.platform = Gem::Platform::RUBY s.name = "archipelago" - s.version = "0.2.2" + s.version = "0.2.3" s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A set of tools for distributed computing in ruby." Modified: trunk/hyperactive/Rakefile =================================================================== --- trunk/hyperactive/Rakefile 2006-11-29 05:42:34 UTC (rev 83) +++ trunk/hyperactive/Rakefile 2006-11-29 05:43:14 UTC (rev 84) @@ -9,7 +9,7 @@ spec = Gem::Specification.new do |s| s.platform = Gem::Platform::RUBY s.name = "hyperactive" - s.version = "0.2.2" + s.version = "0.2.3" s.author = "Martin Kihlgren" s.email = "zond at troja dot ath dot cx" s.summary = "A base class for persistent objects that uses archipelago for persistence. Useful for Ruby on Rails models for example." From nobody at rubyforge.org Wed Nov 29 00:45:13 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Wed, 29 Nov 2006 00:45:13 -0500 (EST) Subject: [Archipelago-submits] [85] tags: created release 0.2.3 Message-ID: <20061129054514.402C5524097C@rubyforge.org> Revision: 85 Author: zond Date: 2006-11-29 00:45:13 -0500 (Wed, 29 Nov 2006) Log Message: ----------- created release 0.2.3 Added Paths: ----------- tags/release_0_2_3/ tags/release_0_2_3/archipelago/README tags/release_0_2_3/archipelago/Rakefile tags/release_0_2_3/archipelago/TODO tags/release_0_2_3/archipelago/lib/archipelago/current.rb tags/release_0_2_3/archipelago/lib/archipelago/disco.rb tags/release_0_2_3/archipelago/lib/archipelago/hashish.rb tags/release_0_2_3/archipelago/lib/archipelago/pirate.rb tags/release_0_2_3/archipelago/lib/archipelago/tranny.rb tags/release_0_2_3/archipelago/lib/archipelago/treasure.rb tags/release_0_2_3/archipelago/tests/current_test.rb tags/release_0_2_3/archipelago/tests/pirate_test.rb tags/release_0_2_3/archipelago/tests/tranny_test.rb tags/release_0_2_3/archipelago/tests/treasure_benchmark.rb tags/release_0_2_3/archipelago/tests/treasure_test.rb tags/release_0_2_3/hyperactive/README tags/release_0_2_3/hyperactive/Rakefile tags/release_0_2_3/hyperactive/lib/hyperactive/hash.rb tags/release_0_2_3/hyperactive/lib/hyperactive/list.rb tags/release_0_2_3/hyperactive/lib/hyperactive/record.rb tags/release_0_2_3/hyperactive/lib/hyperactive.rb tags/release_0_2_3/hyperactive/tests/hash_benchmark.rb tags/release_0_2_3/hyperactive/tests/hash_test.rb tags/release_0_2_3/hyperactive/tests/list_benchmark.rb tags/release_0_2_3/hyperactive/tests/list_test.rb tags/release_0_2_3/hyperactive/tests/record_test.rb Removed Paths: ------------- tags/release_0_2_3/archipelago/README tags/release_0_2_3/archipelago/Rakefile tags/release_0_2_3/archipelago/TODO tags/release_0_2_3/archipelago/lib/archipelago/current.rb tags/release_0_2_3/archipelago/lib/archipelago/disco.rb tags/release_0_2_3/archipelago/lib/archipelago/hashish.rb tags/release_0_2_3/archipelago/lib/archipelago/pirate.rb tags/release_0_2_3/archipelago/lib/archipelago/tranny.rb tags/release_0_2_3/archipelago/lib/archipelago/treasure.rb tags/release_0_2_3/archipelago/tests/current_test.rb tags/release_0_2_3/archipelago/tests/pirate_test.rb tags/release_0_2_3/archipelago/tests/tranny_test.rb tags/release_0_2_3/archipelago/tests/treasure_benchmark.rb tags/release_0_2_3/archipelago/tests/treasure_test.rb tags/release_0_2_3/hyperactive/README tags/release_0_2_3/hyperactive/Rakefile tags/release_0_2_3/hyperactive/lib/hyperactive/list.rb tags/release_0_2_3/hyperactive/lib/hyperactive/record.rb tags/release_0_2_3/hyperactive/lib/hyperactive/tree.rb tags/release_0_2_3/hyperactive/lib/hyperactive.rb tags/release_0_2_3/hyperactive/tests/list_test.rb tags/release_0_2_3/hyperactive/tests/record_test.rb tags/release_0_2_3/hyperactive/tests/tree_benchmark.rb tags/release_0_2_3/hyperactive/tests/tree_test.rb Copied: tags/release_0_2_3 (from rev 73, trunk) Deleted: tags/release_0_2_3/archipelago/README =================================================================== --- trunk/archipelago/README 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/README 2006-11-29 05:45:13 UTC (rev 85) @@ -1,39 +0,0 @@ -= This is Archipelago, a distributed computing toolkit for ruby. - -It consists of several different parts, that can be used standalone or in conjunction. - -== Dependencies: -Archipelago::Hashish::BerkeleyHashishProvider:: ruby bdb: http://moulon.inra.fr/ruby/bdb.html - -== Sub packages: -Archipelago::Disco:: A UDP multicast discovery service useful to find services in your network with a minimum of configuration. -Archipelago::Tranny:: A distributed transaction manager inspired by the jini service mahalo. -Archipelago::Current:: A tiny concurrency toolkit used in various parts of Archipelago -Archipelago::Hashish:: A hash-like tool that provides transparent persistence. -Archipelago::Treasure:: A distributed object database where the objects never leave the database, instead you do your operations upon references to the objects. It has support for serializably isolated transactions with optimistic concurrency control using Archipelago::Tranny or any transaction manager with similar semantics. -Archipelago::Pirate:: A client tool to allocate Archipelago::Treasure::Chests for different keys and act like an almost normal local Hash for providing distributed and scaleable object database facilities to any ruby application. - -== Examples: - -To build a ruby gem from these sources do 'rake gem'. The gem will be -placed within the pkg/ directory. - -To set up an Archipelago::Tranny::Manager do the following (from scripts/tranny.rb): - - :include:script/tranny.rb - -To set up an Archipelago::Treasure::Chest do the following (from scripts/chest.rb): - - :include:script/chest.rb - -To set up an Archipelago::Pirate::Captain do the following (from scripts/pirate.rb): - - :include:script/pirate.rb - -To set up a test environment to play around with, run the following -commands in a few terminals: - -* script/tranny.rb /tmp/tranny -* script/chest.rb /tmp/chest1 -* script/chest.rb /tmp/chest2 -* script/console Copied: tags/release_0_2_3/archipelago/README (from rev 74, trunk/archipelago/README) =================================================================== --- tags/release_0_2_3/archipelago/README (rev 0) +++ tags/release_0_2_3/archipelago/README 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,45 @@ += This is Archipelago, a distributed computing toolkit for ruby. + +It consists of several different parts, that can be used standalone or in conjunction. + +== Dependencies: +Archipelago::Hashish::BerkeleyHashishProvider:: ruby bdb: http://moulon.inra.fr/ruby/bdb.html + +== Sub packages: +Archipelago::Disco:: A UDP multicast discovery service useful to find services in your network with a minimum of configuration. +Archipelago::Tranny:: A distributed transaction manager inspired by the jini service mahalo. +Archipelago::Current:: A tiny concurrency toolkit used in various parts of Archipelago +Archipelago::Hashish:: A hash-like tool that provides transparent persistence. +Archipelago::Treasure:: A distributed object database where the objects never leave the database, instead you do your operations upon references to the objects. It has support for serializably isolated transactions with optimistic concurrency control using Archipelago::Tranny or any transaction manager with similar semantics. +Archipelago::Pirate:: A client tool to allocate Archipelago::Treasure::Chests for different keys and act like an almost normal local Hash for providing distributed and scaleable object database facilities to any ruby application. + +== Usage: + +To use archipelago in the simplest and most obvious way, just run the script/services.rb script. + +You can run this on any number of machines in your local network, and they will all communicate through their own +discovery services. + +Then you instantiate an Archipelago::Pirate::Captain anywhere in the network. This instance can be used basically as +a normal Hash - most everything will be transparently hidden from you. + +Everything you put in this instance will be persistently stored in the network. + +== Examples: + +To run a transaction manager and a database server, just do the following (from script/services.rb): + + :include:script/services.rb + +To set up an Archipelago::Pirate::Captain do the following (from script/pirate.rb): + + :include:script/pirate.rb + +Or you can run script/console, which starts an irb and loads script/pirate.rb. + +So, to set up a test environment to play around with in a few simple steps, run the following +commands in a few terminals: + + script/services.rb /tmp/services1 + script/services.rb /tmp/services2 + script/console Deleted: tags/release_0_2_3/archipelago/Rakefile =================================================================== --- trunk/archipelago/Rakefile 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/Rakefile 2006-11-29 05:45:13 UTC (rev 85) @@ -1,59 +0,0 @@ - -require 'rake' -require 'rake/testtask' -require 'rubygems' -Gem::manage_gems -require 'rake/gempackagetask' - - -spec = Gem::Specification.new do |s| - s.platform = Gem::Platform::RUBY - s.name = "archipelago" - s.version = "0.2.2" - s.author = "Martin Kihlgren" - s.email = "zond at troja dot ath dot cx" - s.summary = "A set of tools for distributed computing in ruby." - s.files = FileList['lib/**/*.rb', 'tests/*', 'script/*', 'GPL-2', 'TODO'].to_a - s.require_path = "lib" - s.autorequire = "archipelago" - s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') - s.has_rdoc = true - s.rdoc_options << '--line-numbers' - s.rdoc_options << '--inline-source' - s.extra_rdoc_files = ["README"] -end - - -SOURCE_FILES = FileList.new do |fl| - [ "lib", "tests" ].each do |dir| - fl.include "#{dir}/**/*" - end - fl.include "Rakefile" -end - -Rake::GemPackageTask.new(spec) do |pkg| - pkg.need_tar = true -end - -task :default => [:units] do -end - -desc "Run all tests" -Rake::TestTask.new(:units) do |t| - t.pattern = 'tests/*_test.rb' - t.verbose = true - t.warning = true -end - -desc "Run all benchmarks" -Rake::TestTask.new(:bench) do |t| - t.pattern = 'tests/*_benchmark.rb' - t.verbose = true - t.warning = true -end - -desc "Package a gem from the source" -task :gem => "pkg/#{spec.name}-#{spec.version}.gem" do - puts "generated latest version" -end - Copied: tags/release_0_2_3/archipelago/Rakefile (from rev 84, trunk/archipelago/Rakefile) =================================================================== --- tags/release_0_2_3/archipelago/Rakefile (rev 0) +++ tags/release_0_2_3/archipelago/Rakefile 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,59 @@ + +require 'rake' +require 'rake/testtask' +require 'rubygems' +Gem::manage_gems +require 'rake/gempackagetask' + + +spec = Gem::Specification.new do |s| + s.platform = Gem::Platform::RUBY + s.name = "archipelago" + s.version = "0.2.3" + s.author = "Martin Kihlgren" + s.email = "zond at troja dot ath dot cx" + s.summary = "A set of tools for distributed computing in ruby." + s.files = FileList['lib/**/*.rb', 'tests/*', 'script/*', 'GPL-2', 'TODO'].to_a + s.require_path = "lib" + s.autorequire = "archipelago" + s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') + s.has_rdoc = true + s.rdoc_options << '--line-numbers' + s.rdoc_options << '--inline-source' + s.extra_rdoc_files = ["README"] +end + + +SOURCE_FILES = FileList.new do |fl| + [ "lib", "tests" ].each do |dir| + fl.include "#{dir}/**/*" + end + fl.include "Rakefile" +end + +Rake::GemPackageTask.new(spec) do |pkg| + pkg.need_tar = true +end + +task :default => [:units] do +end + +desc "Run all tests" +Rake::TestTask.new(:units) do |t| + t.pattern = 'tests/*_test.rb' + t.verbose = true + t.warning = true +end + +desc "Run all benchmarks" +Rake::TestTask.new(:bench) do |t| + t.pattern = 'tests/*_benchmark.rb' + t.verbose = true + t.warning = true +end + +desc "Package a gem from the source" +task :gem => "pkg/#{spec.name}-#{spec.version}.gem" do + puts "generated latest version" +end + Deleted: tags/release_0_2_3/archipelago/TODO =================================================================== --- trunk/archipelago/TODO 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/TODO 2006-11-29 05:45:13 UTC (rev 85) @@ -1,13 +0,0 @@ - - * Create a new HashishProvider with built in redundancy, - for example using the Chord project: http://pdos.csail.mit.edu/chord/ - - * Make Chest aware about whether transactions have affected it 'for real' ie - check whether the instance before the call differs from the instance after - the call. Preferably without incurring performance lossage. - - * Test the transaction recovery mechanism of Chest. - - * Make Archipelago::Treasure::Dubloons work after an Archipelago::Treasure::Chest - has rebooted. For example: demand that the chest always run the same host+port - or make Dubloons able to lookup their home Chest by service_id. Copied: tags/release_0_2_3/archipelago/TODO (from rev 79, trunk/archipelago/TODO) =================================================================== --- tags/release_0_2_3/archipelago/TODO (rev 0) +++ tags/release_0_2_3/archipelago/TODO 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,16 @@ + + * Create a new HashishProvider with built in redundancy, + for example using the Chord project: http://pdos.csail.mit.edu/chord/ + + * Make Chest aware about whether transactions have affected it 'for real' ie + check whether the instance before the call differs from the instance after + the call. Preferably without incurring performance lossage. + + * Make Dubloon proxy targets able to provide a dirty_state method that decides + whether they are to be considered dirty, clean or auto-select. + + * Test the transaction recovery mechanism of Chest. + + * Make Archipelago::Treasure::Dubloons work after an Archipelago::Treasure::Chest + has rebooted. For example: demand that the chest always run the same host+port + or make Dubloons able to lookup their home Chest by service_id. Deleted: tags/release_0_2_3/archipelago/lib/archipelago/current.rb =================================================================== --- trunk/archipelago/lib/archipelago/current.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/lib/archipelago/current.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,197 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'monitor' - -# -# Make threads dumpable for show. -# -class Thread - def _dump(l) - "" - end - def self._load(s) - nil - end -end - -module Archipelago - - module Current - - # - # Adds a few threaded methods to the normal ruby collections. - # - # NB: Will work slightly different than the unthreaded ones in certain circumstances. - # - module ThreadedCollection - - # - # Like each, except calls +block+ within a new thread. - # - def t_each(&block) - threads = [] - self.each do |args| - threads << Thread.new do - yield(args) - end - end - threads.each do |thread| - thread.join - end - end - - # - # Like collect, except calls +block+ within a new thread. - # - def t_collect(&block) - result = [] - result.extend(Synchronized) - self.t_each do |args| - result.synchronize do - result << yield(args) - end - end - return result - end - - # - # Like select, except calls +block+ within a new thread. - # - def t_select(&block) - result = [] - result.extend(Synchronized) - self.t_each do |args| - matches = yield(args) - result.synchronize do - result << args - end if matches - end - return result - end - - # - # Like reject, except calls +block+ within a new thread. - # - def t_reject(&block) - result = [] - result.extend(Synchronized) - self.t_each do |args| - matches = yield(args) - result.synchronize do - result << args - end unless matches - end - return result - end - - end - - - # - # A module that will allow any class to synchronize over any other - # object. - # - module Synchronized - include MonitorMixin - alias :lock :mon_enter - alias :unlock :mon_exit - # - # We dont care about lock ownership. - # - def mon_check_owner - end - # - # Get a lock for this +object+ - # - def lock_on(object) - Thread.exclusive do - this_lock = @archipelago_current_synchronized_lock_by_object[object] ||= Lock.new - this_lock.lock - end - end - # - # Release any lock on this +object+. - # - def unlock_on(object) - Thread.exclusive do - this_lock = @archipelago_current_synchronized_lock_by_object[object] ||= Lock.new - this_lock.unlock - if this_lock.mon_entering_queue.empty? - @archipelago_current_synchronized_lock_by_object.delete(object) - end - end - end - # - # Makes sure the given +block+ is only run once at a time - # for this instance and the given +object+ - # - # Optionally do NOT lock, if you dont +actually+ want to. - # - def synchronize_on(object, actually = true, &block) - lock_on(object) if actually - begin - return yield - ensure - unlock_on(object) if actually - end - end - - private - - # - # Initialize our instance variables - # when someone is extended with us. - # - def self.extend_object(obj) - super(obj) - obj.instance_eval do - sync_initialize - end - end - - # - # Do the actual initialization. - # - def sync_initialize - @archipelago_current_synchronized_lock_by_object = {} - mon_initialize - end - - # - # Initialize upon creation. - # - # Classes that include this module should call super() - # upon initialization. - # - def initialize(*args) - super() - sync_initialize - end - end - - # - # Just a convenience empty class with locking functionality. - # - class Lock - include Synchronized - attr_reader :mon_entering_queue - end - - end - -end Copied: tags/release_0_2_3/archipelago/lib/archipelago/current.rb (from rev 82, trunk/archipelago/lib/archipelago/current.rb) =================================================================== --- tags/release_0_2_3/archipelago/lib/archipelago/current.rb (rev 0) +++ tags/release_0_2_3/archipelago/lib/archipelago/current.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,224 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'monitor' + +# +# Make threads dumpable for show. +# +class Thread + def _dump(l) + "" + end + def self._load(s) + nil + end +end + +module Archipelago + + module Current + + # + # Adds a few threaded methods to the normal ruby collections. + # + # The only method your class has to implement to use this module is each(&block). + # + # NB: Will work slightly different than the unthreaded ones in certain circumstances. + # + module ThreadedCollection + + # + # Like each, except calls +block+ or +callable+ within a new thread. + # + def t_each(callable = nil, &block) + raise "You have to provide either callable or block" if callable.nil? && block.nil? + + threads = [] + + self.each do |args| + threads << Thread.new do + call_helper(callable, args, &block) + end + end + + threads.each do |thread| + thread.join + end + end + + # + # Like collect, except calls +block+ or +callable+ within a new thread. + # + def t_collect(callable = nil, &block) + raise "You have to provide either callable or block" if callable.nil? && block.nil? + + result = [] + result.extend(Synchronized) + self.t_each do |args| + new_value = call_helper(callable, args, &block) + result.synchronize do + result << new_value + end + end + return result + end + + # + # Like select, except calls +block+ or +callable+ within a new thread. + # + def t_select(callable = nil, &block) + raise "You have to provide either callable or block" if callable.nil? && block.nil? + + result = [] + result.extend(Synchronized) + self.t_each do |args| + matches = call_helper(callable, args, &block) + result.synchronize do + result << args + end if matches + end + return result + end + + # + # Like reject, except calls +block+ or +callable+ within a new thread. + # + def t_reject(callable = nil, &block) + raise "You have to provide either callable or block" if callable.nil? && block.nil? + + result = [] + result.extend(Synchronized) + self.t_each do |args| + matches = call_helper(callable, args, &block) + result.synchronize do + result << args + end unless matches + end + return result + end + + private + + def call_helper(o = nil, args = nil, &block) + if o + if Array === args + return o.call(*args) + else + return o.call(args) + end + else + return yield(args) + end + end + + end + + + # + # A module that will allow any class to synchronize over any other + # object. + # + module Synchronized + include MonitorMixin + alias :lock :mon_enter + alias :unlock :mon_exit + # + # We dont care about lock ownership. + # + def mon_check_owner + end + # + # Get a lock for this +object+ + # + def lock_on(object) + Thread.exclusive do + this_lock = @archipelago_current_synchronized_lock_by_object[object] ||= Lock.new + this_lock.lock + end + end + # + # Release any lock on this +object+. + # + def unlock_on(object) + Thread.exclusive do + this_lock = @archipelago_current_synchronized_lock_by_object[object] ||= Lock.new + this_lock.unlock + if this_lock.mon_entering_queue.empty? + @archipelago_current_synchronized_lock_by_object.delete(object) + end + end + end + # + # Makes sure the given +block+ is only run once at a time + # for this instance and the given +object+ + # + # Optionally do NOT lock, if you dont +actually+ want to. + # + def synchronize_on(object, actually = true, &block) + lock_on(object) if actually + begin + return yield + ensure + unlock_on(object) if actually + end + end + + private + + # + # Initialize our instance variables + # when someone is extended with us. + # + def self.extend_object(obj) + super(obj) + obj.instance_eval do + sync_initialize + end + end + + # + # Do the actual initialization. + # + def sync_initialize + @archipelago_current_synchronized_lock_by_object = {} + mon_initialize + end + + # + # Initialize upon creation. + # + # Classes that include this module should call super() + # upon initialization. + # + def initialize(*args) + super() + sync_initialize + end + end + + # + # Just a convenience empty class with locking functionality. + # + class Lock + include Synchronized + attr_reader :mon_entering_queue + end + + end + +end Deleted: tags/release_0_2_3/archipelago/lib/archipelago/disco.rb =================================================================== --- trunk/archipelago/lib/archipelago/disco.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/lib/archipelago/disco.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,646 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'socket' -require 'thread' -require 'ipaddr' -require 'pp' -require 'archipelago/current' -require 'drb' -require 'set' - -module Archipelago - - module Disco - - # - # Default address to use. - # - ADDRESS = "234.2.4.2" - # - # Default port to use. - # - PORT = 25242 - # - # Default port range to use for unicast. - # - UNIPORTS = 25243..26243 - # - # Default lookup timeout. - # - LOOKUP_TIMEOUT = 10 - # - # Default initial pause between resending lookup queries. - # Will be doubled for each resend. - # - INITIAL_LOOKUP_STANDOFF = 0.1 - # - # Default pause between trying to validate all services we - # know about. - # - VALIDATION_INTERVAL = 60 - # - # Only save stuff that we KNOW we want. - # - THRIFTY_CACHING = true - # - # Only reply to the one actually asking about a service. - # - THRIFTY_REPLYING = true - # - # Dont send on publish, only on query. - # - THRIFTY_PUBLISHING = false - - # - # The host we are running on. - # - HOST = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" - - # - # A module to simplify publishing services. - # - # If you include it you can use the publish! method - # at your convenience. - # - # If you want to customize the publishing related behaviour you can - # call initialize_publishable with a Hash of options. - # - # See Archipelago::Treasure::Chest or Archipelago::Tranny::Manager for examples. - # - # It will store the service_id of this service in a directory beside this - # file (publishable.rb) named as the class you include into unless you - # define @persistence_provider before you call initialize_publishable. - # - module Publishable - - # - # Also add the ClassMethods to +base+. - # - def self.append_features(base) - super - base.extend(ClassMethods) - end - - module ClassMethods - # - # Just load whatever we have in +s+. - # - def _load(s) - DRbObject._load(s) - end - end - - # - # Dump a DRbObject refering to us. - # - def _dump(dummy_param) - DRbObject.new(self)._dump(dummy_param) - end - - # - # Will initialize this instance with @service_description and @jockey_options - # and merge these with the optionally given :service_description and - # :jockey_options. - # - def initialize_publishable(options = {}) - @service_description = { - :service_id => service_id, - :validator => self, - :service => self, - :class => self.class.name - }.merge(options[:service_description] || {}) - @jockey_options = options[:jockey_options] || {} - end - - # - # Create an Archipelago::Disco::Jockey for this instance using @jockey_options - # or optionally given :jockey_options. - # - # Will publish this service using @service_description or optionally given - # :service_description. - # - def publish!(options = {}) - @jockey ||= defined?(Archipelago::Disco::MC) ? Archipelago::Disco::MC : Archipelago::Disco::Jockey.new(@jockey_options.merge(options[:jockey_options] || {})) - @jockey.publish(Archipelago::Disco::Record.new(@service_description.merge(options[:service_description] || {}))) - end - - # - # We are always valid if we are able to reply. - # - def valid? - if defined?(@valid) - @valid - else - @valid = true - end - end - - # - # Stops the publishing of this Publishable. - # - def stop! - if valid? - @valid = false - if defined?(Archipelago::Disco::MC) && @jockey == Archipelago::Disco::MC - @jockey.unpublish(self.service_id) - else - @jockey.stop! - end - end - end - - # - # Returns our semi-unique id so that we can be found again. - # - def service_id - # - # The provider of happy magic persistent hashes of different kinds. - # - @persistence_provider ||= Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join(self.class.name + ".db")) - # - # Stuff that didnt fit in any of the other databases. - # - @metadata ||= @persistence_provider.get_hashish("metadata") - return @metadata["service_id"] ||= Digest::SHA1.hexdigest("#{HOST}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}").to_s - end - - end - - # - # A mock validator to be used for dumb systems that dont want - # to validate. - # - class MockValidator - def valid? - true - end - end - - # - # A Hash-like description of a service. - # - class ServiceDescription - IGNORABLE_ATTRIBUTES = Set[:unicast_reply] - attr_reader :attributes - # - # Initialize this service description with a hash - # that describes its attributes. - # - def initialize(hash = {}) - @attributes = hash - end - # - # Forwards as much as possible to our Hash. - # - def method_missing(meth, *args, &block) - if @attributes.respond_to?(meth) - if block - @attributes.send(meth, *args, &block) - else - @attributes.send(meth, *args) - end - else - super(*args) - end - end - # - # Returns whether this ServiceDescription matches the given +match+. - # - def matches?(match) - match.each do |key, value| - unless IGNORABLE_ATTRIBUTES.include?(key) - return false unless @attributes.include?(key) && (value.nil? || @attributes[key] == value) - end - end - true - end - end - - # - # A class used to query the Disco network for services. - # - class Query < ServiceDescription - end - - # - # A class used to defined removed services. - # - class UnPublish - attr_reader :service_id - def initialize(service_id) - @service_id = service_id - end - end - - # - # A class used to define an existing service. - # - class Record < ServiceDescription - # - # Initialize this Record with a hash that must contain an :service_id and a :validator. - # - def initialize(hash) - raise "Record must have a :service_id" unless hash.include?(:service_id) - raise "Record must have a :validator" unless hash.include?(:validator) - super(hash) - end - # - # Returns whether this service is still valid. - # - def valid? - begin - self[:validator].valid? - rescue DRb::DRbError => e - false - end - end - end - - # - # A container of services. - # - class ServiceLocker - attr_reader :hash - include Archipelago::Current::Synchronized - include Archipelago::Current::ThreadedCollection - def initialize(hash = nil) - super - @hash = hash || {} - end - # - # Merge this locker with another. - # - def merge(sd) - rval = @hash.clone - rval.merge!(sd.hash) - ServiceLocker.new(rval) - end - # - # Forwards as much as possible to our Hash. - # - def method_missing(meth, *args, &block) - if @hash.respond_to?(meth) - synchronize do - if block - @hash.send(meth, *args, &block) - else - @hash.send(meth, *args) - end - end - else - super(meth, *args, &block) - end - end - # - # Find all containing services matching +match+. - # - def get_services(match) - rval = ServiceLocker.new - self.each do |service_id, service_data| - rval[service_id] = service_data if service_data.matches?(match) && service_data.valid? - end - return rval - end - # - # Remove all non-valid services. - # - def validate! - self.clone.each do |service_id, service_data| - self.delete(service_id) unless service_data.valid? - end - end - end - - # - # The main discovery class used to both publish and lookup services. - # - class Jockey - - # - # Will create a Jockey service running on :address and :port or - # ADDRESS and PORT if none are given. - # - # Will the first available unicast port within :uniports or if not given UNIPORTS for receiving unicast messages. - # - # Will have a default :lookup_timeout of LOOKUP_TIMEOUT, a default - # :initial_lookup_standoff of INITIAL_LOOKUP_STANDOFF and a default - # :validation_interval of VALIDATION_INTERVAL. - # - # Will only cache (and validate, which saves network traffic) stuff - # that has been looked up before if :thrifty_caching, or THRIFTY_CACHING if not given. - # - # Will only reply to the one that sent out the query (and therefore save lots of network traffic) - # if :thrifty_replying, or THRIFTY_REPLYING if not given. - # - # Will send out a multicast when a new service is published unless :thrifty_publishing, or - # THRIFTY_PUBLISHING if not given. - # - # Will reply to all queries to which it has matching local services with a unicast message if :thrifty_replying, - # or if not given THRIFTY_REPLYING. Otherwise will reply with multicasts. - # - def initialize(options = {}) - @valid = true - @remote_services = ServiceLocker.new - @local_services = ServiceLocker.new - @subscribed_services = Set.new - - @incoming = Queue.new - @outgoing = Queue.new - - @new_service_semaphore = MonitorMixin::ConditionVariable.new(Archipelago::Current::Lock.new) - - setup(options) - - start_listener - start_unilistener - start_shouter - start_picker - start_validator(options[:validation_interval] || VALIDATION_INTERVAL) - end - - # - # Sets up this instance according to the given +options+. - # - def setup(options = {}) - @thrifty_caching = options.include?(:thrifty_caching) ? options[:thrifty_caching] : THRIFTY_CACHING - @thrifty_replying = options.include?(:thrifty_replying) ? options[:thrifty_replying] : THRIFTY_REPLYING - @thrifty_publishing = options.include?(:thrifty_publishing) ? options[:thrifty_publishing] : THRIFTY_PUBLISHING - @lookup_timeout = options[:lookup_timeout] || LOOKUP_TIMEOUT - @initial_lookup_standoff = options[:initial_lookup_standoff] || INITIAL_LOOKUP_STANDOFF - - @listener = UDPSocket.new - @unilistener = UDPSocket.new - - @listener.setsockopt(Socket::IPPROTO_IP, - Socket::IP_ADD_MEMBERSHIP, - IPAddr.new(options[:address] || ADDRESS).hton + Socket.gethostbyname("0.0.0.0")[3]) - - @listener.setsockopt(Socket::SOL_SOCKET, - Socket::SO_REUSEADDR, - true) - begin - @listener.setsockopt(Socket::SOL_SOCKET, - Socket::SO_REUSEPORT, - true) - rescue - # /moo - end - @listener.bind('', options[:port] || PORT) - - uniports = options[:uniports] || UNIPORTS - this_port = uniports.min - begin - @unilistener.bind('', this_port) - rescue Errno::EADDRINUSE => e - if this_port < uniports.max - this_port += 1 - retry - else - raise e - end - end - @unicast_address = "#{HOST}:#{this_port}" - - @sender = UDPSocket.new - @sender.connect(options[:address] || ADDRESS, options[:port] || PORT) - - @unisender = UDPSocket.new - end - - # - # Clears our local and remote services. - # - def clear! - @local_services = ServiceLocker.new - @remote_services = ServiceLocker.new - end - - # - # Stops all the threads in this instance. - # - def stop! - if @valid - @valid = false - @local_services.each do |service_id, service_description| - self.unpublish(service_id) - end - @listener_thread.kill - @unilistener_thread.kill - @validator_thread.kill - @picker_thread.kill - until @outgoing.empty? - sleep(0.01) - end - @shouter_thread.kill - end - end - - # - # Lookup any services matching +match+, optionally with a +timeout+. - # - # Will immediately return if we know of matching and valid services, - # will otherwise send out regular Queries and return as soon as - # matching services are found, or when the +timeout+ runs out. - # - def lookup(match, timeout = @lookup_timeout) - match[:unicast_reply] = @unicast_address - @subscribed_services << match if @thrifty_caching - standoff = @initial_lookup_standoff - - @outgoing << [nil, match] - known_services = @remote_services.get_services(match).merge(@local_services.get_services(match)) - return known_services unless known_services.empty? - - @new_service_semaphore.wait(standoff) - standoff *= 2 - - t = Time.new - while Time.new < t + timeout - known_services = @remote_services.get_services(match).merge(@local_services.get_services(match)) - return known_services unless known_services.empty? - - @new_service_semaphore.wait(standoff) - standoff *= 2 - - @outgoing << [nil, match] - end - - ServiceLocker.new - end - - # - # Record the given +service+ and broadcast about it. - # - def publish(service) - if service.valid? - service[:published_at] = Time.now - @local_services[service[:service_id]] = service - @new_service_semaphore.broadcast - unless @thrifty_publishing - @outgoing << [nil, service] - end - end - end - - # - # Removes the service with given +service_id+ from the published services. - # - def unpublish(service_id) - @local_services.delete(service_id) - @new_service_semaphore.broadcast - unless @thrifty_publishing - @outgoing << [nil, UnPublish.new(service_id)] - end - end - - private - - # - # Start the validating thread. - # - def start_validator(validation_interval) - @validator_thread = Thread.new do - loop do - begin - @local_services.validate! - @remote_services.validate! - sleep(validation_interval) - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Start the thread sending Records and Queries - # - def start_shouter - @shouter_thread = Thread.new do - loop do - begin - recipient, data = @outgoing.pop - if recipient - address, port = recipient.split(/:/) - @unisender.send(Marshal.dump(data), 0, address, port.to_i) - else - begin - @sender.write(Marshal.dump(data)) - rescue Errno::ECONNREFUSED => e - retry - end - end - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Start the thread receiving Records and Queries - # - def start_listener - @listener_thread = Thread.new do - loop do - begin - @incoming << Marshal.load(@listener.recv(1024)) - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Start the thread receiving Records and Queries - # on unicast. - # - def start_unilistener - @unilistener_thread = Thread.new do - loop do - begin - @incoming << Marshal.load(@unilistener.recv(1024)) - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Start the thread picking incoming Records, Queries and UnPublishes and - # handling them properly. - # - def start_picker - @picker_thread = Thread.new do - loop do - begin - data = @incoming.pop - if Archipelago::Disco::Query === data - @local_services.get_services(data).each do |service_id, service_data| - if @thrifty_replying - @outgoing << [data[:unicast_reply], service_data] - else - @outgoing << [nil, service_data] - end - end - elsif Archipelago::Disco::Record === data - if interesting?(data) && data.valid? - @remote_services[data[:service_id]] = data - @new_service_semaphore.broadcast - end - elsif Archipelago::Disco::UnPublish === data - @remote_services.delete(data.service_id) - end - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Are we generous in our caching, or have we been - # asked about this type of +publish+ before? - # - def interesting?(publish) - @subscribed_services.each do |subscribed| - return true if publish.matches?(subscribed) - end - return !@thrifty_caching - end - - end - - # - # The default Archipelago::Disco::Jockey that is always available for lookups is Archipelago::Disco::MC. - # - # If you really need to you can customize it by defining MC_OPTIONS before loading disco.rb, and if you REALLY - # need to you can disable it completely by setting MC_DISABLED to true. - # - MC = Jockey.new(defined?(MC_OPTIONS) ? MC_OPTIONS : {}) unless defined?(MC_DISABLED) && MC_DISABLED - - end - -end Copied: tags/release_0_2_3/archipelago/lib/archipelago/disco.rb (from rev 81, trunk/archipelago/lib/archipelago/disco.rb) =================================================================== --- tags/release_0_2_3/archipelago/lib/archipelago/disco.rb (rev 0) +++ tags/release_0_2_3/archipelago/lib/archipelago/disco.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,638 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'socket' +require 'thread' +require 'ipaddr' +require 'pp' +require 'archipelago/current' +require 'drb' +require 'set' + +module Archipelago + + module Disco + + # + # Default address to use. + # + ADDRESS = "234.2.4.2" + # + # Default port to use. + # + PORT = 25242 + # + # Default port range to use for unicast. + # + UNIPORTS = 25243..26243 + # + # Default lookup timeout. + # + LOOKUP_TIMEOUT = 10 + # + # Default initial pause between resending lookup queries. + # Will be doubled for each resend. + # + INITIAL_LOOKUP_STANDOFF = 0.1 + # + # Default pause between trying to validate all services we + # know about. + # + VALIDATION_INTERVAL = 60 + # + # Only save stuff that we KNOW we want. + # + THRIFTY_CACHING = true + # + # Only reply to the one actually asking about a service. + # + THRIFTY_REPLYING = true + # + # Dont send on publish, only on query. + # + THRIFTY_PUBLISHING = false + + # + # The host we are running on. + # + HOST = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" + + # + # A module to simplify publishing services. + # + # If you include it you can use the publish! method + # at your convenience. + # + # If you want to customize the publishing related behaviour you can + # call initialize_publishable with a Hash of options. + # + # See Archipelago::Treasure::Chest or Archipelago::Tranny::Manager for examples. + # + # It will store the service_id of this service in a directory beside this + # file (publishable.rb) named as the class you include into unless you + # define @persistence_provider before you call initialize_publishable. + # + module Publishable + + # + # Also add the ClassMethods to +base+. + # + def self.append_features(base) + super + base.extend(ClassMethods) + end + + module ClassMethods + # + # Just load whatever we have in +s+. + # + def _load(s) + DRbObject._load(s) + end + end + + # + # Dump a DRbObject refering to us. + # + def _dump(dummy_param) + DRbObject.new(self)._dump(dummy_param) + end + + # + # Will initialize this instance with @service_description and @jockey_options + # and merge these with the optionally given :service_description and + # :jockey_options. + # + def initialize_publishable(options = {}) + @service_description = { + :service_id => service_id, + :validator => self, + :service => self, + :class => self.class.name + }.merge(options[:service_description] || {}) + @jockey_options = options[:jockey_options] || {} + end + + # + # Create an Archipelago::Disco::Jockey for this instance using @jockey_options + # or optionally given :jockey_options. + # + # Will publish this service using @service_description or optionally given + # :service_description. + # + def publish!(options = {}) + @jockey ||= defined?(Archipelago::Disco::MC) ? Archipelago::Disco::MC : Archipelago::Disco::Jockey.new(@jockey_options.merge(options[:jockey_options] || {})) + @jockey.publish(Archipelago::Disco::Record.new(@service_description.merge(options[:service_description] || {}))) + end + + # + # We are always valid if we are able to reply. + # + def valid? + if defined?(@valid) + @valid + else + @valid = true + end + end + + # + # Stops the publishing of this Publishable. + # + def stop! + if valid? + @valid = false + if defined?(Archipelago::Disco::MC) && @jockey == Archipelago::Disco::MC + @jockey.unpublish(self.service_id) + else + @jockey.stop! + end + end + end + + # + # Returns our semi-unique id so that we can be found again. + # + def service_id + # + # The provider of happy magic persistent hashes of different kinds. + # + @persistence_provider ||= Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join(self.class.name + ".db")) + # + # Stuff that didnt fit in any of the other databases. + # + @metadata ||= @persistence_provider.get_hashish("metadata") + return @metadata["service_id"] ||= Digest::SHA1.hexdigest("#{HOST}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}").to_s + end + + end + + # + # A mock validator to be used for dumb systems that dont want + # to validate. + # + class MockValidator + def valid? + true + end + end + + # + # A Hash-like description of a service. + # + class ServiceDescription + IGNORABLE_ATTRIBUTES = Set[:unicast_reply] + attr_reader :attributes + # + # Initialize this service description with a hash + # that describes its attributes. + # + def initialize(hash = {}) + @attributes = hash + end + # + # Forwards as much as possible to our Hash. + # + def method_missing(meth, *args, &block) + if @attributes.respond_to?(meth) + @attributes.send(meth, *args, &block) + else + super(*args) + end + end + # + # Returns whether this ServiceDescription matches the given +match+. + # + def matches?(match) + match.each do |key, value| + unless IGNORABLE_ATTRIBUTES.include?(key) + return false unless @attributes.include?(key) && (value.nil? || @attributes[key] == value) + end + end + true + end + end + + # + # A class used to query the Disco network for services. + # + class Query < ServiceDescription + end + + # + # A class used to defined removed services. + # + class UnPublish + attr_reader :service_id + def initialize(service_id) + @service_id = service_id + end + end + + # + # A class used to define an existing service. + # + class Record < ServiceDescription + # + # Initialize this Record with a hash that must contain an :service_id and a :validator. + # + def initialize(hash) + raise "Record must have a :service_id" unless hash.include?(:service_id) + raise "Record must have a :validator" unless hash.include?(:validator) + super(hash) + end + # + # Returns whether this service is still valid. + # + def valid? + begin + self[:validator].valid? + rescue DRb::DRbError => e + false + end + end + end + + # + # A container of services. + # + class ServiceLocker + attr_reader :hash + include Archipelago::Current::Synchronized + include Archipelago::Current::ThreadedCollection + def initialize(hash = nil) + super + @hash = hash || {} + end + # + # Merge this locker with another. + # + def merge(sd) + rval = @hash.clone + rval.merge!(sd.hash) + ServiceLocker.new(rval) + end + # + # Forwards as much as possible to our Hash. + # + def method_missing(meth, *args, &block) + if @hash.respond_to?(meth) + synchronize do + @hash.send(meth, *args, &block) + end + else + super(meth, *args, &block) + end + end + # + # Find all containing services matching +match+. + # + def get_services(match) + rval = ServiceLocker.new + self.each do |service_id, service_data| + rval[service_id] = service_data if service_data.matches?(match) && service_data.valid? + end + return rval + end + # + # Remove all non-valid services. + # + def validate! + self.clone.each do |service_id, service_data| + self.delete(service_id) unless service_data.valid? + end + end + end + + # + # The main discovery class used to both publish and lookup services. + # + class Jockey + + # + # Will create a Jockey service running on :address and :port or + # ADDRESS and PORT if none are given. + # + # Will the first available unicast port within :uniports or if not given UNIPORTS for receiving unicast messages. + # + # Will have a default :lookup_timeout of LOOKUP_TIMEOUT, a default + # :initial_lookup_standoff of INITIAL_LOOKUP_STANDOFF and a default + # :validation_interval of VALIDATION_INTERVAL. + # + # Will only cache (and validate, which saves network traffic) stuff + # that has been looked up before if :thrifty_caching, or THRIFTY_CACHING if not given. + # + # Will only reply to the one that sent out the query (and therefore save lots of network traffic) + # if :thrifty_replying, or THRIFTY_REPLYING if not given. + # + # Will send out a multicast when a new service is published unless :thrifty_publishing, or + # THRIFTY_PUBLISHING if not given. + # + # Will reply to all queries to which it has matching local services with a unicast message if :thrifty_replying, + # or if not given THRIFTY_REPLYING. Otherwise will reply with multicasts. + # + def initialize(options = {}) + @valid = true + @remote_services = ServiceLocker.new + @local_services = ServiceLocker.new + @subscribed_services = Set.new + + @incoming = Queue.new + @outgoing = Queue.new + + @new_service_semaphore = MonitorMixin::ConditionVariable.new(Archipelago::Current::Lock.new) + + setup(options) + + start_listener + start_unilistener + start_shouter + start_picker + start_validator(options[:validation_interval] || VALIDATION_INTERVAL) + end + + # + # Sets up this instance according to the given +options+. + # + def setup(options = {}) + @thrifty_caching = options.include?(:thrifty_caching) ? options[:thrifty_caching] : THRIFTY_CACHING + @thrifty_replying = options.include?(:thrifty_replying) ? options[:thrifty_replying] : THRIFTY_REPLYING + @thrifty_publishing = options.include?(:thrifty_publishing) ? options[:thrifty_publishing] : THRIFTY_PUBLISHING + @lookup_timeout = options[:lookup_timeout] || LOOKUP_TIMEOUT + @initial_lookup_standoff = options[:initial_lookup_standoff] || INITIAL_LOOKUP_STANDOFF + + @listener = UDPSocket.new + @unilistener = UDPSocket.new + + @listener.setsockopt(Socket::IPPROTO_IP, + Socket::IP_ADD_MEMBERSHIP, + IPAddr.new(options[:address] || ADDRESS).hton + Socket.gethostbyname("0.0.0.0")[3]) + + @listener.setsockopt(Socket::SOL_SOCKET, + Socket::SO_REUSEADDR, + true) + begin + @listener.setsockopt(Socket::SOL_SOCKET, + Socket::SO_REUSEPORT, + true) + rescue + # /moo + end + @listener.bind('', options[:port] || PORT) + + uniports = options[:uniports] || UNIPORTS + this_port = uniports.min + begin + @unilistener.bind('', this_port) + rescue Errno::EADDRINUSE => e + if this_port < uniports.max + this_port += 1 + retry + else + raise e + end + end + @unicast_address = "#{HOST}:#{this_port}" + + @sender = UDPSocket.new + @sender.connect(options[:address] || ADDRESS, options[:port] || PORT) + + @unisender = UDPSocket.new + end + + # + # Clears our local and remote services. + # + def clear! + @local_services = ServiceLocker.new + @remote_services = ServiceLocker.new + end + + # + # Stops all the threads in this instance. + # + def stop! + if @valid + @valid = false + @local_services.each do |service_id, service_description| + self.unpublish(service_id) + end + @listener_thread.kill + @unilistener_thread.kill + @validator_thread.kill + @picker_thread.kill + until @outgoing.empty? + sleep(0.01) + end + @shouter_thread.kill + end + end + + # + # Lookup any services matching +match+, optionally with a +timeout+. + # + # Will immediately return if we know of matching and valid services, + # will otherwise send out regular Queries and return as soon as + # matching services are found, or when the +timeout+ runs out. + # + def lookup(match, timeout = @lookup_timeout) + match[:unicast_reply] = @unicast_address + @subscribed_services << match if @thrifty_caching + standoff = @initial_lookup_standoff + + @outgoing << [nil, match] + known_services = @remote_services.get_services(match).merge(@local_services.get_services(match)) + return known_services unless known_services.empty? + + @new_service_semaphore.wait(standoff) + standoff *= 2 + + t = Time.new + while Time.new < t + timeout + known_services = @remote_services.get_services(match).merge(@local_services.get_services(match)) + return known_services unless known_services.empty? + + @new_service_semaphore.wait(standoff) + standoff *= 2 + + @outgoing << [nil, match] + end + + ServiceLocker.new + end + + # + # Record the given +service+ and broadcast about it. + # + def publish(service) + if service.valid? + service[:published_at] = Time.now + @local_services[service[:service_id]] = service + @new_service_semaphore.broadcast + unless @thrifty_publishing + @outgoing << [nil, service] + end + end + end + + # + # Removes the service with given +service_id+ from the published services. + # + def unpublish(service_id) + @local_services.delete(service_id) + @new_service_semaphore.broadcast + unless @thrifty_publishing + @outgoing << [nil, UnPublish.new(service_id)] + end + end + + private + + # + # Start the validating thread. + # + def start_validator(validation_interval) + @validator_thread = Thread.new do + loop do + begin + @local_services.validate! + @remote_services.validate! + sleep(validation_interval) + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Start the thread sending Records and Queries + # + def start_shouter + @shouter_thread = Thread.new do + loop do + begin + recipient, data = @outgoing.pop + if recipient + address, port = recipient.split(/:/) + @unisender.send(Marshal.dump(data), 0, address, port.to_i) + else + begin + @sender.write(Marshal.dump(data)) + rescue Errno::ECONNREFUSED => e + retry + end + end + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Start the thread receiving Records and Queries + # + def start_listener + @listener_thread = Thread.new do + loop do + begin + @incoming << Marshal.load(@listener.recv(1024)) + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Start the thread receiving Records and Queries + # on unicast. + # + def start_unilistener + @unilistener_thread = Thread.new do + loop do + begin + @incoming << Marshal.load(@unilistener.recv(1024)) + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Start the thread picking incoming Records, Queries and UnPublishes and + # handling them properly. + # + def start_picker + @picker_thread = Thread.new do + loop do + begin + data = @incoming.pop + if Archipelago::Disco::Query === data + @local_services.get_services(data).each do |service_id, service_data| + if @thrifty_replying + @outgoing << [data[:unicast_reply], service_data] + else + @outgoing << [nil, service_data] + end + end + elsif Archipelago::Disco::Record === data + if interesting?(data) && data.valid? + @remote_services[data[:service_id]] = data + @new_service_semaphore.broadcast + end + elsif Archipelago::Disco::UnPublish === data + @remote_services.delete(data.service_id) + end + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Are we generous in our caching, or have we been + # asked about this type of +publish+ before? + # + def interesting?(publish) + @subscribed_services.each do |subscribed| + return true if publish.matches?(subscribed) + end + return !@thrifty_caching + end + + end + + # + # The default Archipelago::Disco::Jockey that is always available for lookups is Archipelago::Disco::MC. + # + # If you really need to you can customize it by defining MC_OPTIONS before loading disco.rb, and if you REALLY + # need to you can disable it completely by setting MC_DISABLED to true. + # + MC = Jockey.new(defined?(MC_OPTIONS) ? MC_OPTIONS : {}) unless defined?(MC_DISABLED) && MC_DISABLED + + end + +end Deleted: tags/release_0_2_3/archipelago/lib/archipelago/hashish.rb =================================================================== --- trunk/archipelago/lib/archipelago/hashish.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/lib/archipelago/hashish.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,262 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'archipelago/current' -require 'bdb' - -module Archipelago - - # - # The module containing the persistence default provider. - # - module Hashish - - # - # In essence a Berkeley Database backed Hash. - # - # Will cache all values having been written or read - # in a normal Hash cache for fast access. - # - # Will save the last update timestamp for all keys - # in a separate Hash cache AND a separate Berkeley Database. - # - class BerkeleyHashish - include Archipelago::Current::Synchronized - # - # Initialize an instance with the +name+ and BDB::Env +env+. - # - def initialize(name, env) - super() - @content_db = env.open_db(BDB::HASH, name, "content", BDB::CREATE) - @content = {} - @timestamps_db = env.open_db(BDB::HASH, name, "timestamps", BDB::CREATE | BDB::NOMMAP) - @timestamps = {} - @lock = Archipelago::Current::Lock.new - end - # - # Close the @content_db and @timestamps_db behind this BerkeleyHashish - # - def close! - @content_db.close - @timestamps_db.close - end - # - # Returns a deep ( Marshal.load(Marshal.dump(o)) ) clone - # of the object represented by +key+. - # - def get_deep_clone(key) - return Marshal.load(@content_db[Marshal.dump(key)]) - end - # - # Simply get the value for the +key+. - # - def [](key) - @lock.synchronize_on(key) do - - value = @content[key] - return value if value - - return get_from_db(key) - - end - end - # - # Insert +value+ under +key+. - # - # Will call value.save_hook(old_value) and send - # it a block that does the actual saving if - # value.respond_to?(:save_hook). - # - def []=(key, value) - @lock.synchronize_on(key) do - - @content[key] = value - - write_to_db(key, Marshal.dump(key), Marshal.dump(value), value) - - return value - - end - end - # - # Stores whatever is under +key+ if it is not the same as - # whats in the persistent db. - # - # Will call value.save_hook(old_value) and send - # it a block that does the actual saving if - # value.respond_to?(:save_hook) and - # a save is actually performed. - # - def store_if_changed(key) - @lock.synchronize_on(key) do - - serialized_key = Marshal.dump(key) - value = @content[key] - serialized_value = Marshal.dump(value) - - write_to_db(key, serialized_key, serialized_value, value) if @content_db[serialized_key] != serialized_value - - end - end - # - # Returns the last time the value under +key+ was changed. - # - def timestamp(key) - @lock.synchronize_on(key) do - - timestamp = @timestamps[key] - return timestamp if timestamp - - serialized_key = Marshal.dump(key) - serialized_timestamp = @timestamps_db[serialized_key] - return nil unless serialized_timestamp - - timestamp = Marshal.load(serialized_timestamp) - @timestamps[key] = timestamp - return timestamp - - end - end - # - # Will do +callable+.call(key, value) for each - # key-and-value pair in this Hashish. - # - # NB: This is totaly thread-unsafe, only do this - # for management or rescue! - # - def each(callable) - @content_db.each do |serialized_key, serialized_value| - key = Marshal.load(serialized_key) - callable.call(key, self.[](key)) - end - end - # - # Delete +key+ and its value and timestamp. - # - def delete(key) - @lock.synchronize_on(key) do - - serialized_key = Marshal.dump(key) - - @content.delete(key) - @content_db[serialized_key] = nil - @timestamps.delete(key) - @timestamps_db[serialized_key] = nil - - end - end - - private - - # - # Write +key+, serialized as +serialized_key+ and - # +serialized_value+ to the db. - # - # Will call value.save_hook(old_value) and send - # it a block that does the actual saving if - # value.respond_to?(:save_hook). - # - def write_to_db(key, serialized_key, serialized_value, value) - if value.respond_to?(:save_hook) - old_serialized_value = @content_db[serialized_key] - old_value = old_serialized_value ? Marshal.load(old_serialized_value) : nil - value.save_hook(old_value) do - do_write_to_db(key, serialized_key, serialized_value) - end - else - do_write_to_db(key, serialized_key, serialized_value) - end - end - - # - # Actually writes +key+ serialized as +serialized_key+ an - # +serialized_value+ to the db. Used by write_to_db. - # - def do_write_to_db(key, serialized_key, serialized_value) - now = Time.now - - @content_db[serialized_key] = serialized_value - @timestamps_db[serialized_key] = Marshal.dump(now) - @timestamps[key] = now - end - - # - # Read +key+ from db and if it is found - # put it in the cache Hash. - # - def get_from_db(key) - serialized_key = Marshal.dump(key) - serialized_value = @content_db[serialized_key] - return nil unless serialized_value - - value = Marshal.load(serialized_value) - @content[key] = value - return value - end - end - - # - # A simple persistence provider backed by Berkeley db. - # - class BerkeleyHashishProvider - # - # Initialize an instance with the given - # +env_path+ to its database dir. - # - def initialize(env_path) - env_path.mkpath - @env = BDB::Env.open(env_path, BDB::CREATE | BDB::INIT_MPOOL) - @berkeley_hashishes = [] - @bdb_dbs = [] - end - # - # Returns a cleverly cached (but slightly inefficient) - # hash-like instance (see Archipelago::Hashish::BerkeleyHashish) - # using +name+. - # - def get_cached_hashish(name) - hashish = BerkeleyHashish.new(name, @env) - @berkeley_hashishes << hashish - return hashish - end - # - # Returns a normal hash-like instance using +name+. - # - def get_hashish(name) - db = @env.open_db(BDB::HASH, name, nil, BDB::CREATE | BDB::NOMMAP) - @bdb_dbs << db - return db - end - # - # Closes databases opened by this instance and removes the persistent files. - # - def unlink - @berkeley_hashishes.each do |h| - h.close! - end - @bdb_dbs.each do |d| - d.close - end - home = Pathname.new(@env.home) - @env.close - home.rmtree if home.exist? - end - end - - end - -end Copied: tags/release_0_2_3/archipelago/lib/archipelago/hashish.rb (from rev 79, trunk/archipelago/lib/archipelago/hashish.rb) =================================================================== --- tags/release_0_2_3/archipelago/lib/archipelago/hashish.rb (rev 0) +++ tags/release_0_2_3/archipelago/lib/archipelago/hashish.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,292 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'archipelago/current' +require 'bdb' + +module Archipelago + + # + # The module containing the persistence default provider. + # + module Hashish + + # + # In essence a Berkeley Database backed Hash. + # + # Will cache all values having been written or read + # in a normal Hash cache for fast access. + # + # Will save the last update timestamp for all keys + # in a separate Hash cache AND a separate Berkeley Database. + # + class BerkeleyHashish + include Archipelago::Current::Synchronized + # + # Initialize an instance with the +name+ and BDB::Env +env+. + # + def initialize(name, env) + super() + @content_db = env.open_db(BDB::HASH, name, "content", BDB::CREATE) + @content = {} + @timestamps_db = env.open_db(BDB::HASH, name, "timestamps", BDB::CREATE | BDB::NOMMAP) + @timestamps = {} + @lock = Archipelago::Current::Lock.new + end + # + # Close the @content_db and @timestamps_db behind this BerkeleyHashish + # + def close! + @content_db.close + @timestamps_db.close + end + # + # Returns a deep ( Marshal.load(Marshal.dump(o)) ) clone + # of the object represented by +key+. + # + def get_deep_clone(key) + return Marshal.load(@content_db[Marshal.dump(key)]) + end + # + # Returns true if this BerkeleyHashish include +key+. + # + def include?(key) + @content.include?(key) || !@content_db[Marshal.dump(key)].nil? + end + # + # Simply get the value for the +key+. + # + # Will call value.load_hook and send it + # a block that does the actuall insertion of the value + # into the live hash if value.respond_to?(:load_hook) + # if the value didnt exist in the live hash yet. + # + def [](key) + @lock.synchronize_on(key) do + + value = @content[key] + return value if value + + return get_from_db(key) + + end + end + # + # Insert +value+ under +key+. + # + # Will call value.save_hook(old_value) and send + # it a block that does the actual saving if + # value.respond_to?(:save_hook). + # + def []=(key, value) + @lock.synchronize_on(key) do + + @content[key] = value + + write_to_db(key, Marshal.dump(key), Marshal.dump(value), value) + + return value + + end + end + # + # Stores whatever is under +key+ if it is not the same as + # whats in the persistent db - if the +key+ actually has + # a representation in the db. + # + # Will call value.save_hook(old_value) and send + # it a block that does the actual saving if + # value.respond_to?(:save_hook) and + # a save is actually performed. + # + def store_if_changed(key) + @lock.synchronize_on(key) do + + serialized_key = Marshal.dump(key) + + value = @content[key] + serialized_value = Marshal.dump(value) + old_serialized_value = @content_db[serialized_key] + + write_to_db(key, serialized_key, serialized_value, value) if old_serialized_value && old_serialized_value != serialized_value + + end + end + # + # Returns the last time the value under +key+ was changed. + # + def timestamp(key) + @lock.synchronize_on(key) do + + timestamp = @timestamps[key] + return timestamp if timestamp + + serialized_key = Marshal.dump(key) + serialized_timestamp = @timestamps_db[serialized_key] + return nil unless serialized_timestamp + + timestamp = Marshal.load(serialized_timestamp) + @timestamps[key] = timestamp + return timestamp + + end + end + # + # Will do +callable+.call(key, value) for each + # key-and-value pair in this Hashish. + # + # NB: This is totaly thread-unsafe, only do this + # for management or rescue! + # + def each(callable) + @content_db.each do |serialized_key, serialized_value| + key = Marshal.load(serialized_key) + callable.call(key, self.[](key)) + end + end + # + # Delete +key+ and its value and timestamp. + # + def delete(key) + @lock.synchronize_on(key) do + + serialized_key = Marshal.dump(key) + + @content.delete(key) + @content_db[serialized_key] = nil + @timestamps.delete(key) + @timestamps_db[serialized_key] = nil + + end + end + + private + + # + # Write +key+, serialized as +serialized_key+ and + # +serialized_value+ to the db. + # + # Will call value.save_hook(old_value) and send + # it a block that does the actual saving if + # value.respond_to?(:save_hook). + # + def write_to_db(key, serialized_key, serialized_value, value) + if value.respond_to?(:save_hook) + old_serialized_value = @content_db[serialized_key] + old_value = old_serialized_value ? Marshal.load(old_serialized_value) : nil + value.save_hook(old_value) do + do_write_to_db(key, serialized_key, serialized_value) + end + else + do_write_to_db(key, serialized_key, serialized_value) + end + end + + # + # Actually writes +key+ serialized as +serialized_key+ an + # +serialized_value+ to the db. Used by write_to_db. + # + def do_write_to_db(key, serialized_key, serialized_value) + now = Time.now + + @content_db[serialized_key] = serialized_value + @timestamps_db[serialized_key] = Marshal.dump(now) + @timestamps[key] = now + end + + # + # Read +key+ from db and if it is found + # put it in the cache Hash. + # + # Will call value.load_hook and send it + # a block that does the actuall insertion of the value + # into the live hash if value.respond_to?(:load_hook). + # + def get_from_db(key) + serialized_key = Marshal.dump(key) + serialized_value = @content_db[serialized_key] + return nil unless serialized_value + + value = Marshal.load(serialized_value) + if value.respond_to?(:load_hook) + value.load_hook do + @content[key] = value + end + else + @content[key] = value + end + return value + end + end + + # + # A simple persistence provider backed by Berkeley db. + # + class BerkeleyHashishProvider + # + # Initialize an instance with the given + # +env_path+ to its database dir. + # + def initialize(env_path) + env_path.mkpath + @env = BDB::Env.open(env_path, BDB::CREATE | BDB::INIT_MPOOL) + @berkeley_hashishes = [] + @bdb_dbs = [] + end + # + # Returns a cleverly cached (but slightly inefficient) + # hash-like instance (see Archipelago::Hashish::BerkeleyHashish) + # using +name+. + # + def get_cached_hashish(name) + hashish = BerkeleyHashish.new(name, @env) + @berkeley_hashishes << hashish + return hashish + end + # + # Returns a normal hash-like instance using +name+. + # + def get_hashish(name) + db = @env.open_db(BDB::HASH, name, nil, BDB::CREATE | BDB::NOMMAP) + @bdb_dbs << db + return db + end + # + # Closes databases opened by this instance. + # + def close! + @berkeley_hashishes.each do |h| + h.close! + end + @bdb_dbs.each do |d| + d.close + end + end + # + # Closes databases opened by this instance and removes the persistent files. + # + def unlink! + close! + home = Pathname.new(@env.home) + @env.close + home.rmtree if home.exist? + end + end + + end + +end Deleted: tags/release_0_2_3/archipelago/lib/archipelago/pirate.rb =================================================================== --- trunk/archipelago/lib/archipelago/pirate.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/lib/archipelago/pirate.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,336 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'archipelago/disco' -require 'archipelago/tranny' -require 'archipelago/treasure' -require 'pp' -require 'drb' -require 'digest/sha1' - -module Archipelago - - # - # The client library that knows about - # all remote databases and handles reads - # and write to them. - # - module Pirate - - # - # Raised when you try to begin a transaction but have no managers - # available. - # - class NoTransactionManagerAvailableException < RuntimeError - def initialize(pirate) - super("#{pirate} can not find any transaction manager for you") - end - end - - # - # Raised when you try to do stuff without any remote database - # available. - # - class NoRemoteDatabaseAvailableException < RuntimeError - def initialize(pirate) - super("#{pirate} can not find any remote database for you") - end - end - - INITIAL_SERVICE_UPDATE_INTERVAL = 1 - MAXIMUM_SERVICE_UPDATE_INTERVAL = 60 - CHEST_DESCRIPTION = { - :class => 'Archipelago::Treasure::Chest' - } - TRANNY_DESCRIPTION = { - :class => 'Archipelago::Tranny::Manager' - } - - # - # The class that actually keeps track of the Archipelago::Treasure:Chests and the - # Archipelago::Treasure:Dubloons in them. - # - class Captain - attr_reader :chests, :treasure_map, :trannies - # - # Initialize an instance using an Archipelago::Disco::Jockey with :jockey_options - # if given, that looks for new services :initial_service_update_interval or INITIAL_SERVICE_UPDATE_INTERVAL, - # when it starts and never slower than every :maximum_service_update_interval or MAXIMUM_SERVICE_UPDATE_INTERVAL. - # - # Will look for Archipelago::Treasure::Chests matching :chest_description or CHEST_DESCRIPTION and - # Archipelago::Tranny::Managers matching :tranny_description or TRANNY_DESCRIPTION. - # - # Will send off all :chest_eval_files to any chest found for possible evaluation to ensure existence - # of required classes and modules at the chest. - # - def initialize(options = {}) - setup(options) - - @transaction = nil - - start_service_updater(options[:initial_service_update_interval] || INITIAL_SERVICE_UPDATE_INTERVAL, - options[:maximum_service_update_interval] || MAXIMUM_SERVICE_UPDATE_INTERVAL) - - end - - # - # Sets up this instance with the given +options+. - # - def setup(options = {}) - @treasure_map ||= nil - if defined?(Archipelago::Disco::MC) - @treasure_map.stop! if @treasure_map && @treasure_map != Archipelago::Disco::MC - @treasure_map = options.include?(:jockey_options) ? Archipelago::Disco::Jockey.new(options[:jockey_options]) : Archipelago::Disco::MC - else - @treasure_map.stop! if @treasure_map - @treasure_map = Archipelago::Disco::Jockey.new(options[:jockey_options] || {}) - end - - @chest_description = CHEST_DESCRIPTION.merge(options[:chest_description] || {}) - @tranny_description = TRANNY_DESCRIPTION.merge(options[:tranny_description] || {}) - - @chest_eval_files ||= [] - @chest_eval_files += options[:chest_eval_files] || [] - - @chests_having_evaluated ||= {} - - @yar_counter = 0 - end - - # - # Get a value from the distributed database network using a +key+, - # optionally within a +transaction+. - # - def [](key, transaction = nil) - responsible_chest(key)[:service][key, transaction || @transaction] - end - - # - # Write a value to the distributed database network, - # optionally inside a transaction. - # - # Usage: - # * p["my key", transaction] = value - # * p["my key"] = value - # - def []=(key, p1, p2 = nil) - if @transaction && p2.nil? - p2 = p1 - p1 = @transaction - end - responsible_chest(key)[:service][key, p1] = p2 - end - - # - # Delete a value from the distributed database network, - # optionally inside a transaction. - # - def delete(key, transaction = nil) - responsible_chest(key)[:service].delete(key, transaction || @transaction) - end - - # - # Return a clone of this instance bound to a newly created transaction. - # - def begin - raise NoTransactionManagerAvailableException.new(self) if @trannies.empty? - - rval = self.clone - rval.instance_eval do - @transaction = @trannies.values.first[:service].begin - end - - return rval - end - - # - # Execute +block+ within a transaction. - # - # Will commit! transaction after the block is finished unless - # the transaction is aborted or commited already. - # - # Will abort! the transaction if any exception is raised. - # - def transaction(&block) #:yields: transaction - raise NoTransactionManagerAvailableException.new(self) if @trannies.empty? - - @transaction = @trannies.values.first[:service].begin - begin - yield(@transaction) - @transaction.commit! if @transaction.state == :active - rescue Exception => e - @transaction.abort! unless @transaction.state == :aborted - puts e - pp e.backtrace - ensure - @transaction = nil - end - end - - # - # Evaluate this file in all known chests. - # - def evaluate!(filename) - @chest_eval_files << filename - evaluate_in_chests - end - - # - # Commit the transaction we are a member of and forget about it. - # - def commit! - @transaction.commit! - @transaction = nil - end - - # - # Abort the transaction we are a member of and forget about it. - # - def abort! - @transaction.abort! - @transaction = nil - end - - # - # Yarrr! - # - def yar! - @yar_counter += 1 - 'yar!' - end - - # - # Stops the service update thread for this Pirate and also unpublishes and/or - # stops the Jockey. - # - def stop! - @service_update_thread.kill - unless defined?(Archipelago::Disco::MC) && @treasure_map && @treasure_map == Archipelago::Disco::MC - @treasure_map.stop! - end - end - - # - # Will do +callable+.call(key, value) - # for each key-and-value pair in this database network. - # - # NB: This is totaly thread-unsafe, only do this - # for management or rescue! - # - def each(callable) - @chests.t_each do |service_id, chest| - chest[:service].each(callable) - end - end - - # - # Does an immediate update of our service lists. - # - def update_services! - @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) - evaluate_in_chests - @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) - end - - private - - # - # Get the chest responsible for +key+. - # - def responsible_chest(key) - raise NoRemoteDatabaseAvailableException.new(self) if @chests.empty? - - key_id = Digest::SHA1.new(Marshal.dump(key)).to_s - sorted_chest_ids = @chests.keys.sort - sorted_chest_ids.each do |id| - return @chests[id] if id > key_id - end - return @chests[sorted_chest_ids.first] - end - - # - # Make sure all our known chests have evaluated - # all files we need them to. - # - def evaluate_in_chests - # - # For all chests - # - @chests.values.each do |chest| - # - # Ensure that this chest has a Set of evaluated files - # - @chests_having_evaluated[ - [ - chest[:service_id], - chest[:published_at] - ] - ] ||= Set.new - @chest_eval_files.each do |filename| - unless @chests_having_evaluated[ - [ - chest[:service_id], - chest[:published_at] - ] - ].include?(filename) - begin - chest[:service].evaluate!(filename, - File.ctime(filename), - open(filename).read) - rescue Exception => e - puts e - pp e.backtrace - ensure - @chests_having_evaluated[ - [ - chest[:service_id], - chest[:published_at] - ] - ] << filename - end - end - end - end - end - - # - # Start a thread looking up existing chests between every - # +initial+ and +maximum+ seconds. - # - def start_service_updater(initial, maximum) - update_services! - @service_update_thread = Thread.start do - standoff = initial - loop do - begin - sleep(standoff) - standoff *= 2 - standoff = maximum if standoff > maximum - update_services! - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - end - - end - -end Copied: tags/release_0_2_3/archipelago/lib/archipelago/pirate.rb (from rev 77, trunk/archipelago/lib/archipelago/pirate.rb) =================================================================== --- tags/release_0_2_3/archipelago/lib/archipelago/pirate.rb (rev 0) +++ tags/release_0_2_3/archipelago/lib/archipelago/pirate.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,358 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'archipelago/disco' +require 'archipelago/tranny' +require 'archipelago/treasure' +require 'pp' +require 'drb' +require 'digest/sha1' + +module Archipelago + + # + # The client library that knows about + # all remote databases and handles reads + # and write to them. + # + module Pirate + + # + # Raised when you try to begin a transaction but have no managers + # available. + # + class NoTransactionManagerAvailableException < RuntimeError + def initialize(pirate) + super("#{pirate} can not find any transaction manager for you") + end + end + + # + # Raised when you try to do stuff without any remote database + # available. + # + class NoRemoteDatabaseAvailableException < RuntimeError + def initialize(pirate) + super("#{pirate} can not find any remote database for you") + end + end + + # + # Raised when the transaction block failed to commit the transaction in the end. + # + class CommitFailedException < RuntimeError + def initialize(pirate, transaction) + super("#{pirate} failed to commit #{transaction}") + end + end + + INITIAL_SERVICE_UPDATE_INTERVAL = 1 + MAXIMUM_SERVICE_UPDATE_INTERVAL = 60 + CHEST_DESCRIPTION = { + :class => 'Archipelago::Treasure::Chest' + } + TRANNY_DESCRIPTION = { + :class => 'Archipelago::Tranny::Manager' + } + + # + # The class that actually keeps track of the Archipelago::Treasure:Chests and the + # Archipelago::Treasure:Dubloons in them. + # + class Captain + attr_reader :chests, :treasure_map, :trannies + # + # Initialize an instance using an Archipelago::Disco::Jockey with :jockey_options + # if given, that looks for new services :initial_service_update_interval or INITIAL_SERVICE_UPDATE_INTERVAL, + # when it starts and never slower than every :maximum_service_update_interval or MAXIMUM_SERVICE_UPDATE_INTERVAL. + # + # Will look for Archipelago::Treasure::Chests matching :chest_description or CHEST_DESCRIPTION and + # Archipelago::Tranny::Managers matching :tranny_description or TRANNY_DESCRIPTION. + # + # Will send off all :chest_eval_files to any chest found for possible evaluation to ensure existence + # of required classes and modules at the chest. + # + def initialize(options = {}) + setup(options) + + @transaction = nil + + start_service_updater(options[:initial_service_update_interval] || INITIAL_SERVICE_UPDATE_INTERVAL, + options[:maximum_service_update_interval] || MAXIMUM_SERVICE_UPDATE_INTERVAL) + + end + + # + # Sets up this instance with the given +options+. + # + def setup(options = {}) + @treasure_map ||= nil + if defined?(Archipelago::Disco::MC) + @treasure_map.stop! if @treasure_map && @treasure_map != Archipelago::Disco::MC + @treasure_map = options.include?(:jockey_options) ? Archipelago::Disco::Jockey.new(options[:jockey_options]) : Archipelago::Disco::MC + else + @treasure_map.stop! if @treasure_map + @treasure_map = Archipelago::Disco::Jockey.new(options[:jockey_options] || {}) + end + + @chest_description = CHEST_DESCRIPTION.merge(options[:chest_description] || {}) + @tranny_description = TRANNY_DESCRIPTION.merge(options[:tranny_description] || {}) + + @chest_eval_files ||= [] + @chest_eval_files += options[:chest_eval_files] || [] + + @chests_having_evaluated ||= {} + + @yar_counter = 0 + end + + # + # Returns true if this Captain includes the given +key+, optionally within a +transaction+. + # + def include?(key, transaction = nil) + responsible_chest(key)[:service].include?(key, transaction || @transaction) + end + + # + # Get a value from the distributed database network using a +key+, + # optionally within a +transaction+. + # + def [](key, transaction = nil) + responsible_chest(key)[:service][key, transaction || @transaction] + end + + # + # Write a value to the distributed database network, + # optionally inside a transaction. + # + # Usage: + # * p["my key", transaction] = value + # * p["my key"] = value + # + def []=(key, p1, p2 = nil) + if @transaction && p2.nil? + p2 = p1 + p1 = @transaction + end + responsible_chest(key)[:service][key, p1] = p2 + end + + # + # Delete a value from the distributed database network, + # optionally inside a transaction. + # + def delete(key, transaction = nil) + responsible_chest(key)[:service].delete(key, transaction || @transaction) + end + + # + # Return a clone of this instance bound to a newly created transaction. + # + def begin + raise NoTransactionManagerAvailableException.new(self) if @trannies.empty? + + rval = self.clone + rval.instance_eval do + @transaction = @trannies.values.first[:service].begin + end + + return rval + end + + # + # Returns our active transaction, if any. + # + def active_transaction + @transaction + end + + # + # Execute +block+ within a transaction. + # + # Will commit! transaction after the block is finished unless + # the transaction is aborted or commited already. + # + # Will abort! the transaction if any exception is raised. + # + def transaction(&block) #:yields: transaction + raise NoTransactionManagerAvailableException.new(self) if @trannies.empty? + + @transaction = @trannies.values.first[:service].begin + begin + yield(@transaction) + raise CommitFailedException.new(self, @transaction) unless @transaction.commit! == :commited + rescue Exception => e + @transaction.abort! unless @transaction.state == :aborted + raise e + ensure + @transaction = nil + end + end + + # + # Evaluate this file in all known chests. + # + def evaluate!(filename) + @chest_eval_files << filename + evaluate_in_chests + end + + # + # Commit the transaction we are a member of and forget about it. + # + def commit! + @transaction.commit! + @transaction = nil + end + + # + # Abort the transaction we are a member of and forget about it. + # + def abort! + @transaction.abort! + @transaction = nil + end + + # + # Yarrr! + # + def yar! + @yar_counter += 1 + 'yar!' + end + + # + # Stops the service update thread for this Pirate and also unpublishes and/or + # stops the Jockey. + # + def stop! + @service_update_thread.kill + unless defined?(Archipelago::Disco::MC) && @treasure_map && @treasure_map == Archipelago::Disco::MC + @treasure_map.stop! + end + end + + # + # Will do +callable+.call(key, value) + # for each key-and-value pair in this database network. + # + # NB: This is totaly thread-unsafe, only do this + # for management or rescue! + # + def each(callable) + @chests.t_each do |service_id, chest| + chest[:service].each(callable) + end + end + + # + # Does an immediate update of our service lists. + # + def update_services! + @chests = @treasure_map.lookup(Archipelago::Disco::Query.new(@chest_description), 0) + evaluate_in_chests + @trannies = @treasure_map.lookup(Archipelago::Disco::Query.new(@tranny_description), 0) + end + + private + + # + # Get the chest responsible for +key+. + # + def responsible_chest(key) + raise NoRemoteDatabaseAvailableException.new(self) if @chests.empty? + + key_id = Digest::SHA1.hexdigest(Marshal.dump(key)) + sorted_chest_ids = @chests.keys.sort + sorted_chest_ids.each do |id| + return @chests[id] if id > key_id + end + return @chests[sorted_chest_ids.first] + end + + # + # Make sure all our known chests have evaluated + # all files we need them to. + # + def evaluate_in_chests + # + # For all chests + # + @chests.values.each do |chest| + # + # Ensure that this chest has a Set of evaluated files + # + @chests_having_evaluated[ + [ + chest[:service_id], + chest[:published_at] + ] + ] ||= Set.new + @chest_eval_files.each do |filename| + unless @chests_having_evaluated[ + [ + chest[:service_id], + chest[:published_at] + ] + ].include?(filename) + begin + chest[:service].evaluate!(filename, + File.ctime(filename), + open(filename).read) + rescue Exception => e + puts e + pp e.backtrace + ensure + @chests_having_evaluated[ + [ + chest[:service_id], + chest[:published_at] + ] + ] << filename + end + end + end + end + end + + # + # Start a thread looking up existing chests between every + # +initial+ and +maximum+ seconds. + # + def start_service_updater(initial, maximum) + update_services! + @service_update_thread = Thread.start do + standoff = initial + loop do + begin + sleep(standoff) + standoff *= 2 + standoff = maximum if standoff > maximum + update_services! + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + end + + end + +end Deleted: tags/release_0_2_3/archipelago/lib/archipelago/tranny.rb =================================================================== --- trunk/archipelago/lib/archipelago/tranny.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/lib/archipelago/tranny.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,664 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'bdb' -require 'pathname' -require 'drb' -require 'archipelago/current' -require 'digest/sha1' -require 'archipelago/disco' -require 'archipelago/hashish' - -module Archipelago - - module Tranny - - # - # Time before any Transaction will be automatically aborted. - # - TRANSACTION_TIMEOUT = 60 * 60 - - # - # If a member tries to join a transaction that it has allready joined - # it will receive this. - # - class JoinCountException < RuntimeError - def initialize(member, transaction) - super("#{member.inspect} has allready joined the transaction #{transaction.inspect}") - end - end - - # - # If a member tries to commit a transaction not in :active state - # or abort a transaction in :commited state, or any other grave - # state offence - # - class IllegalOperationException < RuntimeError - def initialize(operation, transaction) - super("#{operation.inspect} is illegal for #{transaction.inspect}") - end - end - - # - # If a member tries to get a transaction that the manager doesnt know about - # it gets this. - # - class UnknownTransactionException < RuntimeError - def initialize(transaction_id, manager) - super("#{manager.inspect} doesnt know about any transaction with id #{transaction_id.inspect}") - end - end - - # - # If a member tries to report its state and the manager doesnt know about the - # member, it gets this. - # - class UnknownMemberException < RuntimeError - def initialize(member, transaction) - super("#{transaction.inspect} doesnt know about any member like #{member.inspect}") - end - end - - # - # If a member misbehaves (or the Transaction is completely fucked up) this will be raised - # - class IllegalStateException < RuntimeError - def initialize(transaction) - super("#{transaction.inspect} is in a completely fucked up state") - end - end - - # - # The manager itself. - # - # This will be the drb exported object that participants talk to, - # either directly or through a TransactionProxy. - # - # See also the TransactionProxy and Transaction classes. - # - class Manager - - include Archipelago::Disco::Publishable - - attr_accessor :error_logger - attr_accessor :transaction_timeout - - # - # Will use a BerkeleyHashishProvider using tranny_manager.db in the same dir to get its hashes - # if not :persistence_provider is given. - # - # Will create Transactions timing out after :transaction_timeout seconds or TRANSACTION_TIMEOUT - # if none is given. - # - # Will use Archipelago::Disco::Publishable by calling initialize_publishable with +options+. - # - def initialize(options = {}) - @persistence_provider = options[:persistence_provider] || Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join("tranny_manager.db")) - - initialize_publishable(options) - - @transaction_timeout = options[:transaction_timeout] || TRANSACTION_TIMEOUT - - @db = @persistence_provider.get_cached_hashish("db") - end - - # - # Returns a proxy to a newly created Transaction. - # - # The Transaction will timeout after :timeout seconds - # or @transaction_timeout if none is given. - # - def begin(options = {}) - options[:manager] = self - options[:timeout] ||= @transaction_timeout - return Transaction.new(options).proxy - end - - # - # Used by transactions to notify this manager - # about an error. Delegates the actual logging - # to @error_logger.call(exception). - # - # Set @error_logger or override this method - # if you want to log any errors. - # - def log_error(exception) - self.error_logger.call(exception) if self.error_logger - end - - # - # Used by a +transaction+ to store its state for future reference. - # - def store_transaction!(transaction) - @db[transaction.transaction_id] = transaction - end - - # - # Used by a +transaction+ to remove its state when finished. - # - def remove_transaction!(transaction) - @db.delete(transaction.transaction_id) - end - - # - # Used by a transaction proxy to run +meth+ with +args+ on its +transaction_id+. - # - def call_instance_method(transaction_id, meth, *args) - transaction = @db[transaction_id] - if transaction - return transaction.send(meth, *args) - else - raise UnknownTransactionException.new(transaction_id, self) - end - end - - end - - - - - # - # A proxy to a transaction managed by this manager. - # - # Used because standard DRbObjects only use object_id - # to identify objects, while we want the more unique transaction_id. - # - # Will also remember the state of the transaction when it detects - # methods that will terminate it (:commit | :abort). - # - class TransactionProxy - attr_accessor :transaction_id - def initialize(transaction) - @manager = transaction.manager - @manager_id = transaction.manager.service_id - @transaction_id = transaction.transaction_id - @state = :unknown - end - # - # Return the cached state of the wrapped Archipelago::Tranny::Transaction - # if it is known, otherwise fetch it. - # - def state - if @state == :unknown - method_missing(:state) - else - @state - end - end - # - # Implemented to ensure that all TransactionProxies with the same - # transaction_id are hashwise equal. - # - def hash - @transaction_id.hash - end - # - # Implemented to ensure that all TransactionProxies with the same - # transaction_id are hashwise equal. - # - def eql?(o) - if TransactionProxy === o - o.transaction_id == @transaction_id - else - false - end - end - # - # Forwards everything to our Transaction and remembers - # returnvalue if necessary. - # - def method_missing(meth, *args) #:nodoc: - rval = nil - begin - rval = @manager.call_instance_method(@transaction_id, meth, *args) - rescue DRb::DRbConnError => e - if defined?(Archipelago::Disco::MC) - possible_replacements = Archipelago::Disco::MC.lookup(Archipelago::Disco::Query.new({:service_id => @manager_id})) - raise e if possible_replacements.empty? - - @manager = possible_replacements[@manager_id][:service] - - retry - else - raise e - end - end - case meth - when :abort! - @state = :aborted - when :commit! - @state = rval - end - return rval - end - end - - - - - # - # A transaction managed by the manager. - # - # A transaction can have the following states: - # - # * :active - When it is first created. - # * This transaction can be commited or aborted. - # - # * :voting - When it has started the two-phase commit and the voting has begun. - # * This transaction can be aborted. - # - # * :commited - After everyone has voted and voted either :unchanged or :commit. - # * This transaction can not be changed. - # - # * :aborted - After someone has called abort! or voted :abort in a vote! - # * This transaction can not be changed. - # - # Anyone that wants to join the transaction must implement the following methods: - # - # * abort!(transaction): Abort the provided transaction. - # * commit!(transaction): Commit the provided transaction. - # * prepare!(transaction): Prepare for commiting the provided transaction. - # - # abort! and commit! should not return any values, but can raise exceptions - # if required. - # - # prepare! must return either :abort, :commit or :unchanged, depending on what - # the member is prepared to do. :unchanged is only when the member has not changed - # state during the transaction, and means that it does not require any further - # notification on the progress of the transaction. - # - # A member that has returned :commit on the prepare! must store the transaction - # proxy in a persistent manner to be able to connect to the manager - # and get a new copy of the transaction in case of communications failure. - # - # A member that reconnects to a crashed transaction manager should not abort! the - # transaction if the transaction is still in :active state, since the transaction will - # have been aborted on commit! anyway (since the manager will not be able to prepare! the - # disconnected member after either the manager or member having crashed) if needed. - # If the disconnect was just a temporary networking problem, the transaction will - # continue as planned. - # - # A member that reconnects to a crashed transaction manager where the transaction - # is in :voting state should just wait around and see if it gets prepare! called. - # In case of temporary network failure the transaction will continue as planned, otherwise - # it will abort! automatically. - # - # A member that reconnects to a disconnected transaction manager where the transaction - # is in :commited state should just commit its state. Then it must notify the transaction - # using report_commited! so that the transaction can disappear gracefully. - # - # A member that reconnects to a disconnected transaction manager that either doesnt know - # of the transaction or returns an :aborted transaction may safely abort the state change - # and forget about the transaction. - # - class Transaction - - include Archipelago::Current::Synchronized - - attr_reader :state, :transaction_id, :proxy, :manager - - # - # Create a transaction managed by the provided +manager+. - # - # Will have :manager as TransactionManager, and will timeout - # after :timeout seconds. - # - def initialize(options) - super() - # - # A hash where members are keys and their state the values. - # - @members = {} - @members.extend(Archipelago::Current::Synchronized) - # - # We are alive! - # - @state = :active - # - # We have a timeout! - # - @timeout = options[:timeout] - # - # We are unique! - # - @transaction_id = "#{options[:manager].service_id}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}" - # - # We have a birth time! - # - @created_at = Time.now - # - # We have a manager! - # - self.manager = options[:manager] - - store_us_for_future_reference! - - start_timeout_thread - end - - # - # Store this manager as ours and create a proxy that knows about it. - # - # Used by Archipelago::Tranny:Manager when restoring crashed - # Archipelago::Tranny:Transactions. - # - def manager=(manager) - # - # We have a manager! - # - @manager = manager - # - # We have a proxy to send forth into the world! - # - @proxy = TransactionProxy.new(self) - nil - end - - # - # Special dump call to NOT dump our manager or proxy, - # since DRbObjects dont take kindly to being restored - # in an environment where they are are known to be invalid. - # - def _dump(l) - return Marshal.dump([ - @members, - @state, - @timeout, - @transaction_id, - @created_at - ]) - end - - def self._load(s) - members, state, timeout, transaction_id, created_at = Marshal.load(s) - rval = self.allocate - rval.instance_variable_set(:@members, members) - rval.instance_variable_set(:@state, state) - rval.instance_variable_set(:@timeout, timeout) - rval.instance_variable_set(:@transaction_id, transaction_id) - rval.instance_variable_set(:@created_at, created_at) - rval - end - - # - # Starts the thread that will abort! us automatically - # after we have lived @timeout - # - def start_timeout_thread - Thread.new do - now = Time.now - die_at = @created_at + @timeout - if die_at > now - sleep(die_at - now) - end - abort! - end - nil - end - - # - # What it sounds like. - # - def store_us_for_future_reference! - @manager.store_transaction!(self) - nil - end - - # - # Remove ourselves, we are redundant - # - def remove_us_we_are_redundant! - @manager.remove_transaction!(self) - nil - end - - # - # Used by members that failed during commit. - # - def report_commited!(member) - raise UnknownMemberException(member, self) unless @members.include?(member) - raise IllegalOperationException(:report_commited!, self) unless self.state == :commited - - @members[member] = :commited - - remove_us_if_all_are_commited! - nil - end - - # - # Abort the transaction, sending all participants the abort! message. - # - def abort! - synchronize do - raise IllegalOperationException.new(:abort!, self) if [:commited, :aborted].include?(self.state) - - # - # Set our state. - # - @state = :aborted - - store_us_for_future_reference! - - # - # Abort all members. - # - threads = [] - @members.clone.each do |member, state| - raise RuntimeException.new("This is not supposed to be possible, but we are in abort! with member " + - "#{member.inspect} in state #{state.inspect}") if state == :commited - if state != :aborted - threads << Thread.new do - begin - member.abort!(self.proxy) - @members.synchronize do - @members[member] = :aborted - end - rescue Exception => e - @manager.log_error(e) - # - # We must not let the other members stop just - # because one member failed. No more Mr Nice Guy! - # - end - end - end - end - - # - # Wait for all members to finish. - # - threads.each do |thread| - thread.join - end - - # - # No use having aborted transactions lying about. - # - # NB: This means that disconnected members that cant - # find their old transactions will have to presume they - # have been aborted. - # - remove_us_we_are_redundant! - end - nil - end - - # - # Commits the transaction, returning the new state (:aborted | :commited) - # - def commit! - synchronize do - raise IllegalOperationException.new(:commit!, self) unless self.state == :active - - # - # Vote for the outcome and act on it - # - case vote! - when :abort - abort! - when :commit - _commit! - when :unchanged - @state = :commited - remove_us_we_are_redundant! - end - - return @state - end - nil - end - - # - # Join a +member+ to this transaction. - # - # Will raise a JoinCountException if the given member has allready - # joined this transaction. - # - def join(member) - @members.synchronize do - raise IllegalOperationException.new(:join, self) unless self.state == :active - raise JoinCountException.new(member, self) if @members.include?(member) - - @members[member] = :active - end - store_us_for_future_reference! - nil - end - - private - - # - # Commit the transaction, sending all participants the commit message. - # - # The transaction is commited by all members having commit! called. - # - def _commit! - # - # Set our state. - # - @state = :commited - - store_us_for_future_reference! - - # - # Commit all members. - # - threads = [] - @members.clone.each do |member, state| - raise RuntimeException.new("This is not supposed to be possible, but we are in _commit with member " + - "#{member.inspect} in state #{state.inspect}") unless [:prepared, :commited].include?(state) - threads << Thread.new do - begin - member.commit!(self.proxy) - @members.synchronize do - @members[member] = :commited - end - rescue Exception => e - @manager.log_error(e) - # - # We must not let the other members stop just - # because one member failed. No more Mr Nice Guy! - # - end - end - end - - # - # Wait for all members to finish. - # - threads.each do |thread| - thread.join - end - - remove_us_if_all_are_commited! - end - - def remove_us_if_all_are_commited! - # - # Check to see if all members have been told about the decision. - # - all_have_commited = true - @members.each do |member, state| - all_have_commited = false unless state == :comitted - end - - # - # If they have, remove ourselves from the manager. - # - remove_us_we_are_redundant! if all_have_commited - end - - # - # Let the members of the transaction vote for its outcome. - # - # Members vote by having prepare! called. - # - # Valid returnvalues for the prepare! call are: - # :abort, if the member wants to abort the transaction - # :commit, if the member wants to commit the transaction - # :unchanged, if the member has not changed state during the transaction - # - def vote! - raise IllegalOperationException.new(:vote!, self) unless self.state == :active - - @state = :voting - - store_us_for_future_reference! - - return_value = :commit - - threads = [] - @members.clone.each do |member, state| - raise RuntimeException.new("This is not supposed to be possible, but we are in vote! with member " + - "#{member.inspect} in state #{state.inspect}") unless state == :active - threads << Thread.new do - this_result = nil - begin - this_result = member.prepare!(self.proxy) - rescue Exception => e - @manager.log_error(e) - this_result = :abort - end - @members.synchronize do - case this_result - when :abort - @members[member] = :aborted - return_value = :abort - when :unchanged - @members.delete(member) - else - @members[member] = :prepared - end - end - end - end - - threads.each do |thread| - thread.join - end - - if @members.empty? - return_value = :unchanged - end - - return_value - end - - end - end - -end Copied: tags/release_0_2_3/archipelago/lib/archipelago/tranny.rb (from rev 77, trunk/archipelago/lib/archipelago/tranny.rb) =================================================================== --- tags/release_0_2_3/archipelago/lib/archipelago/tranny.rb (rev 0) +++ tags/release_0_2_3/archipelago/lib/archipelago/tranny.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,670 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'bdb' +require 'pathname' +require 'drb' +require 'archipelago/current' +require 'digest/sha1' +require 'archipelago/disco' +require 'archipelago/hashish' + +module Archipelago + + module Tranny + + # + # Time before any Transaction will be automatically aborted. + # + TRANSACTION_TIMEOUT = 60 * 60 + + # + # If a member tries to join a transaction that it has allready joined + # it will receive this. + # + class JoinCountException < RuntimeError + def initialize(member, transaction) + super("#{member.inspect} has allready joined the transaction #{transaction.inspect}") + end + end + + # + # If a member tries to commit a transaction not in :active state + # or abort a transaction in :commited state, or any other grave + # state offence + # + class IllegalOperationException < RuntimeError + def initialize(operation, transaction) + super("#{operation.inspect} is illegal for #{transaction.inspect}") + end + end + + # + # If a member tries to get a transaction that the manager doesnt know about + # it gets this. + # + class UnknownTransactionException < RuntimeError + def initialize(transaction_id, manager) + super("#{manager.inspect} doesnt know about any transaction with id #{transaction_id.inspect}") + end + end + + # + # If a member tries to report its state and the manager doesnt know about the + # member, it gets this. + # + class UnknownMemberException < RuntimeError + def initialize(member, transaction) + super("#{transaction.inspect} doesnt know about any member like #{member.inspect}") + end + end + + # + # If a member misbehaves (or the Transaction is completely fucked up) this will be raised + # + class IllegalStateException < RuntimeError + def initialize(transaction) + super("#{transaction.inspect} is in a completely fucked up state") + end + end + + # + # The manager itself. + # + # This will be the drb exported object that participants talk to, + # either directly or through a TransactionProxy. + # + # See also the TransactionProxy and Transaction classes. + # + class Manager + + include Archipelago::Disco::Publishable + + attr_accessor :error_logger + attr_accessor :transaction_timeout + + # + # Will use a BerkeleyHashishProvider using tranny_manager.db in the same dir to get its hashes + # if not :persistence_provider is given. + # + # Will create Transactions timing out after :transaction_timeout seconds or TRANSACTION_TIMEOUT + # if none is given. + # + # Will use Archipelago::Disco::Publishable by calling initialize_publishable with +options+. + # + def initialize(options = {}) + @persistence_provider = options[:persistence_provider] || Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join("tranny_manager.db")) + + initialize_publishable(options) + + @transaction_timeout = options[:transaction_timeout] || TRANSACTION_TIMEOUT + + @db = @persistence_provider.get_cached_hashish("db") + end + + # + # Returns a proxy to a newly created Transaction. + # + # The Transaction will timeout after :timeout seconds + # or @transaction_timeout if none is given. + # + def begin(options = {}) + options[:manager] = self + options[:timeout] ||= @transaction_timeout + return Transaction.new(options).proxy + end + + # + # Used by transactions to notify this manager + # about an error. Delegates the actual logging + # to @error_logger.call(exception). + # + # Set @error_logger or override this method + # if you want to log any errors. + # + def log_error(exception) + self.error_logger.call(exception) if self.error_logger + end + + # + # Used by a +transaction+ to store its state for future reference. + # + def store_transaction!(transaction) + @db[transaction.transaction_id] = transaction + end + + # + # Used by a +transaction+ to remove its state when finished. + # + def remove_transaction!(transaction) + @db.delete(transaction.transaction_id) + end + + # + # Used by a transaction proxy to run +meth+ with +args+ on its +transaction_id+. + # + def call_instance_method(transaction_id, meth, *args) + transaction = @db[transaction_id] + if transaction + return transaction.send(meth, *args) + else + raise UnknownTransactionException.new(transaction_id, self) + end + end + + end + + + + + # + # A proxy to a transaction managed by this manager. + # + # Used because standard DRbObjects only use object_id + # to identify objects, while we want the more unique transaction_id. + # + # Will also remember the state of the transaction when it detects + # methods that will terminate it (:commit | :abort). + # + class TransactionProxy + attr_accessor :transaction_id + def initialize(transaction) + @manager = transaction.manager + @manager_id = transaction.manager.service_id + @transaction_id = transaction.transaction_id + @state = :unknown + end + # + # Return the cached state of the wrapped Archipelago::Tranny::Transaction + # if it is known, otherwise fetch it. + # + def state + if @state == :unknown + method_missing(:state) + else + @state + end + end + # + # Implemented to ensure that all TransactionProxies with the same + # transaction_id are hashwise equal. + # + def hash + @transaction_id.hash + end + # + # Implemented to ensure that all TransactionProxies with the same + # transaction_id are hashwise equal. + # + def eql?(o) + if TransactionProxy === o + o.transaction_id == @transaction_id + else + false + end + end + # + # Implemented to allow comparison between proxies. + # + def ==(o) + eql?(o) + end + # + # Forwards everything to our Transaction and remembers + # returnvalue if necessary. + # + def method_missing(meth, *args) #:nodoc: + rval = nil + begin + rval = @manager.call_instance_method(@transaction_id, meth, *args) + rescue DRb::DRbConnError => e + if defined?(Archipelago::Disco::MC) + possible_replacements = Archipelago::Disco::MC.lookup(Archipelago::Disco::Query.new({:service_id => @manager_id})) + raise e if possible_replacements.empty? + + @manager = possible_replacements[@manager_id][:service] + + retry + else + raise e + end + end + case meth + when :abort! + @state = :aborted + when :commit! + @state = rval + end + return rval + end + end + + + + + # + # A transaction managed by the manager. + # + # A transaction can have the following states: + # + # * :active - When it is first created. + # * This transaction can be commited or aborted. + # + # * :voting - When it has started the two-phase commit and the voting has begun. + # * This transaction can be aborted. + # + # * :commited - After everyone has voted and voted either :unchanged or :commit. + # * This transaction can not be changed. + # + # * :aborted - After someone has called abort! or voted :abort in a vote! + # * This transaction can not be changed. + # + # Anyone that wants to join the transaction must implement the following methods: + # + # * abort!(transaction): Abort the provided transaction. + # * commit!(transaction): Commit the provided transaction. + # * prepare!(transaction): Prepare for commiting the provided transaction. + # + # abort! and commit! should not return any values, but can raise exceptions + # if required. + # + # prepare! must return either :abort, :commit or :unchanged, depending on what + # the member is prepared to do. :unchanged is only when the member has not changed + # state during the transaction, and means that it does not require any further + # notification on the progress of the transaction. + # + # A member that has returned :commit on the prepare! must store the transaction + # proxy in a persistent manner to be able to connect to the manager + # and get a new copy of the transaction in case of communications failure. + # + # A member that reconnects to a crashed transaction manager should not abort! the + # transaction if the transaction is still in :active state, since the transaction will + # have been aborted on commit! anyway (since the manager will not be able to prepare! the + # disconnected member after either the manager or member having crashed) if needed. + # If the disconnect was just a temporary networking problem, the transaction will + # continue as planned. + # + # A member that reconnects to a crashed transaction manager where the transaction + # is in :voting state should just wait around and see if it gets prepare! called. + # In case of temporary network failure the transaction will continue as planned, otherwise + # it will abort! automatically. + # + # A member that reconnects to a disconnected transaction manager where the transaction + # is in :commited state should just commit its state. Then it must notify the transaction + # using report_commited! so that the transaction can disappear gracefully. + # + # A member that reconnects to a disconnected transaction manager that either doesnt know + # of the transaction or returns an :aborted transaction may safely abort the state change + # and forget about the transaction. + # + class Transaction + + include Archipelago::Current::Synchronized + + attr_reader :state, :transaction_id, :proxy, :manager + + # + # Create a transaction managed by the provided +manager+. + # + # Will have :manager as TransactionManager, and will timeout + # after :timeout seconds. + # + def initialize(options) + super() + # + # A hash where members are keys and their state the values. + # + @members = {} + @members.extend(Archipelago::Current::Synchronized) + # + # We are alive! + # + @state = :active + # + # We have a timeout! + # + @timeout = options[:timeout] + # + # We are unique! + # + @transaction_id = "#{options[:manager].service_id}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}" + # + # We have a birth time! + # + @created_at = Time.now + # + # We have a manager! + # + self.manager = options[:manager] + + store_us_for_future_reference! + + start_timeout_thread + end + + # + # Store this manager as ours and create a proxy that knows about it. + # + # Used by Archipelago::Tranny:Manager when restoring crashed + # Archipelago::Tranny:Transactions. + # + def manager=(manager) + # + # We have a manager! + # + @manager = manager + # + # We have a proxy to send forth into the world! + # + @proxy = TransactionProxy.new(self) + nil + end + + # + # Special dump call to NOT dump our manager or proxy, + # since DRbObjects dont take kindly to being restored + # in an environment where they are are known to be invalid. + # + def _dump(l) + return Marshal.dump([ + @members, + @state, + @timeout, + @transaction_id, + @created_at + ]) + end + + def self._load(s) + members, state, timeout, transaction_id, created_at = Marshal.load(s) + rval = self.allocate + rval.instance_variable_set(:@members, members) + rval.instance_variable_set(:@state, state) + rval.instance_variable_set(:@timeout, timeout) + rval.instance_variable_set(:@transaction_id, transaction_id) + rval.instance_variable_set(:@created_at, created_at) + rval + end + + # + # Starts the thread that will abort! us automatically + # after we have lived @timeout + # + def start_timeout_thread + Thread.new do + now = Time.now + die_at = @created_at + @timeout + if die_at > now + sleep(die_at - now) + end + abort! + end + nil + end + + # + # What it sounds like. + # + def store_us_for_future_reference! + @manager.store_transaction!(self) + nil + end + + # + # Remove ourselves, we are redundant + # + def remove_us_we_are_redundant! + @manager.remove_transaction!(self) + nil + end + + # + # Used by members that failed during commit. + # + def report_commited!(member) + raise UnknownMemberException(member, self) unless @members.include?(member) + raise IllegalOperationException(:report_commited!, self) unless self.state == :commited + + @members[member] = :commited + + remove_us_if_all_are_commited! + nil + end + + # + # Abort the transaction, sending all participants the abort! message. + # + def abort! + synchronize do + raise IllegalOperationException.new(:abort!, self) if [:commited, :aborted].include?(self.state) + + # + # Set our state. + # + @state = :aborted + + store_us_for_future_reference! + + # + # Abort all members. + # + threads = [] + @members.clone.each do |member, state| + raise RuntimeException.new("This is not supposed to be possible, but we are in abort! with member " + + "#{member.inspect} in state #{state.inspect}") if state == :commited + if state != :aborted + threads << Thread.new do + begin + member.abort!(self.proxy) + @members.synchronize do + @members[member] = :aborted + end + rescue Exception => e + @manager.log_error(e) + # + # We must not let the other members stop just + # because one member failed. No more Mr Nice Guy! + # + end + end + end + end + + # + # Wait for all members to finish. + # + threads.each do |thread| + thread.join + end + + # + # No use having aborted transactions lying about. + # + # NB: This means that disconnected members that cant + # find their old transactions will have to presume they + # have been aborted. + # + remove_us_we_are_redundant! + end + nil + end + + # + # Commits the transaction, returning the new state (:aborted | :commited) + # + def commit! + synchronize do + raise IllegalOperationException.new(:commit!, self) unless self.state == :active + + # + # Vote for the outcome and act on it + # + case vote! + when :abort + abort! + when :commit + _commit! + when :unchanged + @state = :commited + remove_us_we_are_redundant! + end + + return @state + end + nil + end + + # + # Join a +member+ to this transaction. + # + # Will raise a JoinCountException if the given member has allready + # joined this transaction. + # + def join(member) + @members.synchronize do + raise IllegalOperationException.new(:join, self) unless self.state == :active + raise JoinCountException.new(member, self) if @members.include?(member) + + @members[member] = :active + end + store_us_for_future_reference! + nil + end + + private + + # + # Commit the transaction, sending all participants the commit message. + # + # The transaction is commited by all members having commit! called. + # + def _commit! + # + # Set our state. + # + @state = :commited + + store_us_for_future_reference! + + # + # Commit all members. + # + threads = [] + @members.clone.each do |member, state| + raise RuntimeException.new("This is not supposed to be possible, but we are in _commit with member " + + "#{member.inspect} in state #{state.inspect}") unless [:prepared, :commited].include?(state) + threads << Thread.new do + begin + member.commit!(self.proxy) + @members.synchronize do + @members[member] = :commited + end + rescue Exception => e + @manager.log_error(e) + # + # We must not let the other members stop just + # because one member failed. No more Mr Nice Guy! + # + end + end + end + + # + # Wait for all members to finish. + # + threads.each do |thread| + thread.join + end + + remove_us_if_all_are_commited! + end + + def remove_us_if_all_are_commited! + # + # Check to see if all members have been told about the decision. + # + all_have_commited = true + @members.each do |member, state| + all_have_commited = false unless state == :comitted + end + + # + # If they have, remove ourselves from the manager. + # + remove_us_we_are_redundant! if all_have_commited + end + + # + # Let the members of the transaction vote for its outcome. + # + # Members vote by having prepare! called. + # + # Valid returnvalues for the prepare! call are: + # :abort, if the member wants to abort the transaction + # :commit, if the member wants to commit the transaction + # :unchanged, if the member has not changed state during the transaction + # + def vote! + raise IllegalOperationException.new(:vote!, self) unless self.state == :active + + @state = :voting + + store_us_for_future_reference! + + return_value = :commit + + threads = [] + @members.clone.each do |member, state| + raise RuntimeException.new("This is not supposed to be possible, but we are in vote! with member " + + "#{member.inspect} in state #{state.inspect}") unless state == :active + threads << Thread.new do + this_result = nil + begin + this_result = member.prepare!(self.proxy) + rescue Exception => e + @manager.log_error(e) + this_result = :abort + end + @members.synchronize do + case this_result + when :abort + @members[member] = :aborted + return_value = :abort + when :unchanged + @members.delete(member) + else + @members[member] = :prepared + end + end + end + end + + threads.each do |thread| + thread.join + end + + if @members.empty? + return_value = :unchanged + end + + return_value + end + + end + end + +end Deleted: tags/release_0_2_3/archipelago/lib/archipelago/treasure.rb =================================================================== --- trunk/archipelago/lib/archipelago/treasure.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/lib/archipelago/treasure.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,793 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -require 'drb' -require 'bdb' -require 'pathname' -require 'digest/sha1' -require 'pp' -require 'set' -require 'archipelago/hashish' -require 'archipelago/tranny' -require 'archipelago/disco' - -module Archipelago - - module Treasure - - # - # Minimum time between trying to recover - # crashed transactions. - # - TRANSACTION_RECOVERY_INTERVAL = 30 - - # - # The known chests by service id to speed up - # recovery by Dubloons having lost their Chest. - # - CHEST_BY_SERVICE_ID = {} - - # - # Raised whenever the optimistic locking of the serializable transaction isolation level - # was proved wrong. - # - class RollbackException < RuntimeError - def initialize(chest, transaction) - super("#{chest} has been modified during #{transaction}") - end - end - - # - # Raised whenever a method is called on an object that we dont know about. - # - class UnknownObjectException < RuntimeError - def initialize(chest, key, transaction = nil) - if transaction - super("#{chest} does not contain #{key} under #{transaction}") - else - super("#{chest} does not contain #{key}") - end - end - end - - # - # Raised if a Dubloon or Chest doesnt know what transaction you are talking about. - # - class UnknownTransactionException < RuntimeError - def initialize(source, transaction) - super("#{source} is not a part of #{transaction}") - end - end - - # - # Raised if anyone tries to commit a non-prepared transaction. - # - class IllegalCommitException < RuntimeError - def initialize(source, transaction) - super("#{transaction} is not prepared in #{source}") - end - end - - # - # Raised if someone tries to join us to a non-active transaction. - # - class IllegalJoinException < RuntimeError - def initialize(transaction) - super("#{transaction} is not active") - end - end - - # - # A proxy to something in the chest. - # - class Dubloon - # - # Remove all methods so that we look like our target. - # - instance_methods.each do |method| - undef_method method unless (method =~ /^__/ || method == "respond_to?") - end - # - # This Dubloon will always have the same hash, based on - # object_id. - # - def hash - self.object_id.hash - end - # - # Defers to Dubloon#==. - # - def eql?(o) - self.==(o) - end - # - # If o is a Dubloon, will return true if it has the same Dubloon#object_id. - # - # Otherwise will defer to Dubloon#method_missing. - # - def ==(o) - if Dubloon === o - return true if self.object_id == o.object_id - end - return self.method_missing(:==, o) - end - # - # Initialize us with knowledge of our +chest+, the +key+ to our - # target in the +chest+, the known +public_methods+ of our target - # and any +transaction+ we are associated with. - # - def initialize(key, chest, transaction, chest_id) - @key = key - @chest = chest - @transaction = transaction - @chest_id = chest_id - end - # - # A more or less normal dump of all our instance variables. - # - def _dump(dummy_levels) - Marshal.dump([ - @key, - @chest, - @transaction, - @chest_id - ]) - end - # - # Load some instance variables and replace @chest if we know that - # it actually is not correct. - # - def self._load(s) - key, chest, transaction, chest_id = Marshal.load(s) - if CHEST_BY_SERVICE_ID.include?(chest_id) - chest = CHEST_BY_SERVICE_ID[chest_id] - end - - return self.new(key, chest, transaction, chest_id) - end - # - # Return a clone of myself that is joined to - # the +transaction+. - # - def join(transaction) - @chest.join!(transaction) if transaction - return Dubloon.new(@key, @chest, transaction, @chest_id) - end - # - # Raises exception if the given +transaction+ - # is not the same as our own. - # - def assert_transaction(transaction) - raise UnknownTransactionException.new(self, transaction) unless transaction == @transaction - end - # - # This Dubloon will always have the same object_id, based on - # @chest_id and @key and possibly @transaction. - # - def object_id - id = "#{@chest_id}:#{@key}" - id << ":#{@transaction.transaction_id}" if @transaction - return id - end - # - # Does our target respond to +meth+? - # - def respond_to?(meth) - # This one will be called far too often, and it seems safe to ignore it. - return false if meth == :marshal_dump - return super(meth) || self.method_missing(:respond_to?, meth) - end - # - # Call +meth+ with +args+ and +block+ on our target if it responds to - # it. - # - def method_missing(meth, *args, &block) - begin - return @chest.call_instance_method(@key, meth, @transaction, *args, &block) - rescue DRb::DRbConnError => e - if defined?(Archipelago::Disco::MC) - possible_replacements = Archipelago::Disco::MC.lookup(Archipelago::Disco::Query.new({:service_id => @chest_id})) - raise e if possible_replacements.empty? - - @chest = possible_replacements[@chest_id][:service] - CHEST_BY_SERVICE_ID[@chest_id] = @chest - - retry - else - raise e - end - end - end - - end - - # - # A possibly remote database that only returns proxies to its - # contents, and thus runs all methods on its contents itself. - # - # Has support for optimistically locked distributed serializable transactions. - # - class Chest - - # - # The Chest can be published. - # - include Archipelago::Disco::Publishable - - # - # Initialize a Chest - # - # Will use a BerkeleyHashishProvider using treasure_chest.db in the same dir to get its hashes - # if not :persistence_provider is given. - # - # Will try to recover crashed transaction every :transaction_recovery_interval seconds - # or TRANSACTION_RECOVERY_INTERVAL if none is given. - # - # Will use Archipelago::Disco::Publishable by calling initialize_publishable with +options+. - # - def initialize(options = {}) - # - # The provider of happy magic persistent hashes of different kinds. - # - @persistence_provider = options[:persistence_provider] || Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join("treasure_chest.db")) - - # - # Use the given options to initialize the publishable - # instance variables. - # - initialize_publishable(options) - - # - # [transaction => [key => instance]] - # To know what stuff is visible to a transaction. - # - @snapshot_by_transaction = {} - @snapshot_by_transaction.extend(Archipelago::Current::Synchronized) - - # - # [transaction => [key => when the key was read/updated/deleted] - # To know if a transaction is ok to prepare and commit. - # - @timestamp_by_key_by_transaction = {} - - # - # [transaction => [key => instance]] - # To know what transactions were prepared but not - # properly finished last run. - # - @crashed = Set.new - - initialize_seen_data - - # - # The magical persistent map that defines how we actually - # store our data. - # - @db = @persistence_provider.get_cached_hashish("db") - - initialize_prepared(options[:transaction_recovery_interval] || TRANSACTION_RECOVERY_INTERVAL) - - CHEST_BY_SERVICE_ID[self.service_id] = self - end - - # - # The transactions active in this Chest. - # - def active_transactions - @snapshot_by_transaction.keys.clone - end - - # - # Will do +callable+.call(key, value) - # for each key-and-value pair in this Chest. - # - # NB: This is totaly thread-unsafe, only do this - # for management or rescue! - # - def each(callable) - @db.each(callable) - end - - # - # Evaluate +data+ if we have not already seen +label+ or if we have an earlier +timestamp+ than the one given. - # - def evaluate!(label, timestamp, data) - serialized_label = Marshal.dump(label) - go = false - - if @seen_data.include?(serialized_label) - last_timestamp, last_data = Marshal.load(@seen_data[serialized_label]) - go = true if timestamp > last_timestamp - else - go = true - end - - if go - begin - Object.class_eval(data) - @seen_data[serialized_label] = Marshal.dump([timestamp, data]) - rescue Exception => e - @seen_data[serialized_label] = Marshal.load([timestamp, nil]) - puts e - pp e.backtrace - end - end - end - - # - # Return the contents of this chest using a given +key+ and +transaction+. - # - def [](key, transaction = nil) - join!(transaction) - - instance = ensure_instance_with_transaction(key, transaction) - return nil unless instance - - if Dubloon === instance - return instance.join(transaction) - else - return Dubloon.new(key, self, transaction, self.service_id) - end - end - - # - # Delete the value of +key+ within +transaction+. - # - def delete(key, transaction = nil) - join!(transaction) - - rval = nil - - if transaction - # - # If we have a transaction we must note that it is deleted in a - # separate space for that transaction. - # - snapshot = @snapshot_by_transaction[transaction] - snapshot.synchronize do - - rval = snapshot[key] - - snapshot[key] = :deleted - # - # Make sure we remember when it was last changed according to our main db. - # - timestamps = @timestamp_by_key_by_transaction[transaction] - timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) - - end - else - # - # Otherwise just ask our persistence provider to delete it. - # - @db.delete(key) - end - - rval.freeze - - return rval - end - - # - # Put something into this chest with a given +key+, +value+ and - # +transaction+. - # - def []=(key, p1, p2 = nil) - if p2 - value = p2 - transaction = p1 - else - value = p1 - transaction = nil - end - - return set(key, value, transaction) - end - - # - # Call an instance +method+ on whatever this chest holds at +key+ - # with any +transaction+ and +args+. - # - def call_instance_method(key, method, transaction, *arguments, &block) - if transaction - return call_with_transaction(key, method, transaction, *arguments, &block) - else - return call_without_transaction(key, method, *arguments, &block) - end - end - - # - # Abort +transaction+ in this Chest. - # - def abort!(transaction) - assert_transaction(transaction) - - snapshot = @snapshot_by_transaction[transaction] - # - # Make sure nobody can modify this transaction while we - # are aborting it. - # - snapshot.synchronize do - serialized_transaction = Marshal.dump(transaction) - - # - # If this transaction was successfully prepared - # - if @snapshot_by_transaction_db.include?(serialized_transaction) - # - # Unlock the keys that are part of it. - # - snapshot.each do |key, value| - @db.unlock_on(key) - end - # - # And remove it from persistent storage. - # - @snapshot_by_transaction_db[serialized_transaction] = nil - # - # And remove its timestamps from persistent storage. - # - @timestamp_by_key_by_transaction_db[serialized_transaction] = nil - end - # - # Finally delete it from the snapshots. - # - @snapshot_by_transaction.delete(transaction) - # - # And from the timestamps. - # - @timestamp_by_key_by_transaction.delete(transaction) - end - end - - # - # Prepares +transaction+ in this Chest. - # - # NB: This will cause any update of the data within - # this transaction to block until it is either aborted - # or commited! - # - def prepare!(transaction) - assert_transaction(transaction) - - # - # If we dont know about this transaction then it can't very well - # affect us. - # - return :unchanged unless @snapshot_by_transaction.include?(transaction) - - snapshot = @snapshot_by_transaction[transaction] - # - # Make sure nobody can modify this transaction while we are - # preparing it. - # - snapshot.synchronize do - - # - # Remember what locks we acquire so that we can - # unlock them in case of failure. - # - locks = [] - timestamp_by_key = @timestamp_by_key_by_transaction[transaction] - # - # Acquire a lock on each key in the transaction - # - snapshot.each do |key, value| - if @db.timestamp(key) == timestamp_by_key[key] - @db.lock_on(key) - locks << key - else - locks.each do |key| - @db.unlock_on(key) - end - return :abort - end - end - serialized_transaction = Marshal.dump(transaction) - - # - # Dump its state to persistent storage. - # - @snapshot_by_transaction_db[serialized_transaction] = Marshal.dump(snapshot) - # - # And dump its timestamps to persistent storage - # - @timestamp_by_key_by_transaction_db[serialized_transaction] = Marshal.dump(timestamp_by_key) - return :prepared - end - end - - # - # Commits +transaction+ in this Chest. - # - # NB: Transaction must be prepared before commit is called. - # - def commit!(transaction) - assert_transaction(transaction) - raise IllegalCommitException.new(self, transaction) unless @snapshot_by_transaction_db.include?(Marshal.dump(transaction)) - - snapshot = @snapshot_by_transaction[transaction] - # - # Make sure nobody can modify this transaction while we are - # commiting it. - # - snapshot.synchronize do - - # - # Copy each key and value from our private space to the real space - # - snapshot.each do |key, value| - if value == :deleted - @db.delete(key) - else - @db[key] = value - end - end - - # - # Call abort! to clean up after the transaction. - # - abort!(transaction) - - end - end - - private - - # - # Evaluates all data we have been told to evaluate in earlier runs. - # - def initialize_seen_data - # - # [label => [timestamp, data]] - # To remember what and when we have evaluated and be able to evaluate it again. - # - @seen_data = @persistence_provider.get_hashish("seen_data") - @seen_data.each do |serialized_label, serialized_pair| - timestamp, data = Marshal.load(serialized_pair) - - if data - begin - Object.class_eval(data) - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Allocates space for this +transaction+. - # - # Will also call +transaction+.join to make sure - # it is aware of us. - # - def join!(transaction) - if transaction - if transaction.state == :active - @snapshot_by_transaction.synchronize do - unless @snapshot_by_transaction.include?(transaction) - @snapshot_by_transaction[transaction] = {} - @snapshot_by_transaction[transaction].extend(Archipelago::Current::Synchronized) - @timestamp_by_key_by_transaction[transaction] = {} - transaction.join(self) - end - end - else - raise IllegalJoinException.new(transaction) - end - end - end - - # - # Raises if we are not in this transaction. - # - def assert_transaction(transaction) - raise UnknownTransactionException.new(self, transaction) unless @snapshot_by_transaction.include?(transaction) - end - - # - # Call a method within a transaction. - # - def call_with_transaction(key, method, transaction, *arguments, &block) - assert_transaction(transaction) - - # - # Fetch our instance from the snapshot. - # - snapshot = @snapshot_by_transaction[transaction] - instance = snapshot[key] - instance = nil if instance == :deleted - - raise UnknownObjectException.new(self, key, transaction) unless instance - - begin - return execute(instance, method, *arguments, &block) - ensure - # - # Make sure we remember when this object was last changed according - # to the main db. - # - snapshot.synchronize do - timestamps = @timestamp_by_key_by_transaction[transaction] - timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) - end - end - end - - # - # Execute +m+ with arguments +a+ and block +b+ on +o+. - # - def execute(o, m, *a, &b) - if b - return o.send(m, *a, &b) - else - return o.send(m, *a) - end - end - - # - # Call a method outside any transaction (ie inside a transaction of its own). - # - def call_without_transaction(key, method, *arguments, &block) - instance = @db[key] - - raise UnknownObjectException(self, key) unless instance - - begin - return execute(instance, method, *arguments, &block) - ensure - @db.store_if_changed(key) - end - end - - # - # Initializes our storage of prepared transactions. - # - def initialize_prepared(transaction_recovery_interval) - # - # Load stored timestamps for our transaction from db. - # - @timestamp_by_key_by_transaction_db = @persistence_provider.get_hashish("prepared_timestamps") - @timestamp_by_key_by_transaction_db.each do |serialized_transaction, serialized_timestamps| - @timestamp_by_key_by_transaction[Marshal.load(serialized_transaction)] = Marshal.load(serialized_timestamps) - end - - # - # Load stored snapshots for our transaction from db. - # - @snapshot_by_transaction_db = @persistence_provider.get_hashish("prepared") - @snapshot_by_transaction_db.each do |serialized_transaction, serialized_snapshot| - transaction = Marshal.load(transaction) - - @crashed << transaction - @snapshot_by_transaction[transaction] = Marshal.load(serialized_snapshot) - end - start_recovery_thread(transaction_recovery_interval) - end - - # - # Starts the thread that will keep trying to recover - # our crashed transactions. - # - def start_recovery_thread(transaction_recovery_interval) - Thread.new do - loop do - begin - @crashed.clone.each do |transaction| - begin - case transaction.state - when :commited - commit!(transaction) - @crashed.delete(transaction) - when :aborted - abort!(transaction) - @crashed.delete(transaction) - end - rescue Archipelago::Tranny::UnknownTransactionException => e - abort!(transaction) - @crashed.delete(transaction) - end - end - sleep(transaction_recovery_interval) - rescue Exception => e - puts e - pp e.backtrace - end - end - end - end - - # - # Insert +value+ under +key+ and +transaction+ - # in this chest. - # - def set(key, value, transaction) - join!(transaction) - value.assert_transaction(transaction) if Dubloon === value - - if transaction - snapshot = @snapshot_by_transaction[transaction] - - # - # If we have a transaction we must put it in a - # separate space for that transaction. - # - snapshot.synchronize do - - snapshot[key] = value - # - # Make sure we remember the last time this was changed according to - # our main db. - # - timestamps = @timestamp_by_key_by_transaction[transaction] - timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) - - end - else - @db[key] = value - end - - return value if Dubloon === value - - return Dubloon.new(key, self, transaction, service_id) - end - - # - # Try to fetch the data of +key+ from the private space - # of +transaction+ and put it there if it was not there - # already. - # - def ensure_instance_with_transaction(key, transaction) - if transaction - snapshot = @snapshot_by_transaction[transaction] - snapshot.synchronize do - - # - # If we dont have this key in the snapshot. - # - unless snapshot.include?(key) - # - # Fetch the new value for the snapshot - # - new_value = @db.get_deep_clone(key) - # - # If it exists then copy it to the snapshot - # otherwise remove the transaction hash if it is empty. - # - if new_value - snapshot[key] = new_value - else - @snapshot_by_transaction.delete(transaction) if snapshot.empty? - end - end - - rval = snapshot[key] - return rval == :deleted ? nil : rval - - end - else - return @db[key] - end - end - - end - - end - -end Copied: tags/release_0_2_3/archipelago/lib/archipelago/treasure.rb (from rev 81, trunk/archipelago/lib/archipelago/treasure.rb) =================================================================== --- tags/release_0_2_3/archipelago/lib/archipelago/treasure.rb (rev 0) +++ tags/release_0_2_3/archipelago/lib/archipelago/treasure.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,824 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +require 'drb' +require 'bdb' +require 'pathname' +require 'digest/sha1' +require 'pp' +require 'set' +require 'archipelago/hashish' +require 'archipelago/tranny' +require 'archipelago/disco' + +module Archipelago + + module Treasure + + # + # Minimum time between trying to recover + # crashed transactions. + # + TRANSACTION_RECOVERY_INTERVAL = 30 + + # + # The known chests by service id to speed up + # recovery by Dubloons having lost their Chest. + # + CHEST_BY_SERVICE_ID = {} + + # + # Raised whenever the optimistic locking of the serializable transaction isolation level + # was proved wrong. + # + class RollbackException < RuntimeError + def initialize(chest, transaction) + super("#{chest} has been modified during #{transaction}") + end + end + + # + # Raised whenever a method is called on an object that we dont know about. + # + class UnknownObjectException < RuntimeError + def initialize(chest, key, transaction = nil) + if transaction + super("#{chest} does not contain #{key} under #{transaction}") + else + super("#{chest} does not contain #{key}") + end + end + end + + # + # Raised if a Dubloon or Chest doesnt know what transaction you are talking about. + # + class UnknownTransactionException < RuntimeError + def initialize(source, transaction) + super("#{source} is not a part of #{transaction}") + end + end + + # + # Raised if anyone tries to commit a non-prepared transaction. + # + class IllegalCommitException < RuntimeError + def initialize(source, transaction) + super("#{transaction} is not prepared in #{source}") + end + end + + # + # Raised if someone tries to join us to a non-active transaction. + # + class IllegalJoinException < RuntimeError + def initialize(transaction) + super("#{transaction} is not active") + end + end + + # + # A proxy to something in the chest. + # + # This will do a very efficient masquerade as the object it proxies. When asked + # for its class or any other attribute it will forward the query to the proxied object. + # + # It can also be a part of a transaction, either because it was fetched from the chest + # within a transaction, or because it is the return value of a Dubloon#join call. + # + # In this case all forwarded methods will also be within the same transaction, and + # any change to the proxied object will be inside that transaction. + # + # If the proxied object itself needs to handle transaction semantics it can implement + # the with_transaction(transaction, &block) method, which will wrap the + # method call itself within the home Chest of the proxied object. + # + class Dubloon + # + # Remove all methods so that we look like our target. + # + instance_methods.each do |method| + undef_method method unless (method =~ /^__/ || method == "respond_to?") + end + # + # This Dubloon will always have the same hash, based on + # object_id. + # + def hash + self.object_id.hash + end + # + # Defers to Dubloon#==. + # + def eql?(o) + self.==(o) + end + # + # If o is a Dubloon, will return true if it has the same Dubloon#object_id. + # + # Otherwise will defer to Dubloon#method_missing. + # + def ==(o) + if Dubloon === o + return true if self.object_id == o.object_id + end + return self.method_missing(:==, o) + end + # + # Initialize us with knowledge of our +chest+, the +key+ to our + # target in the +chest+, the known +public_methods+ of our target + # and any +transaction+ we are associated with. + # + def initialize(key, chest, transaction, chest_id) + @key = key + @chest = chest + @transaction = transaction + @chest_id = chest_id + end + # + # A more or less normal dump of all our instance variables. + # + def _dump(dummy_levels) + Marshal.dump([ + @key, + @chest, + @transaction, + @chest_id + ]) + end + # + # Load some instance variables and replace @chest if we know that + # it actually is not correct. + # + def self._load(s) + key, chest, transaction, chest_id = Marshal.load(s) + if CHEST_BY_SERVICE_ID.include?(chest_id) + chest = CHEST_BY_SERVICE_ID[chest_id] + end + + return self.new(key, chest, transaction, chest_id) + end + # + # Return a clone of myself that is joined to + # the +transaction+. + # + def join(transaction) + @chest.join!(transaction) if transaction + return Dubloon.new(@key, @chest, transaction, @chest_id) + end + # + # Raises exception if the given +transaction+ + # is not the same as our own. + # + def assert_transaction(transaction) + raise UnknownTransactionException.new(self, transaction) unless transaction == @transaction + end + # + # This Dubloon will always have the same object_id, based on + # @chest_id and @key and possibly @transaction. + # + def object_id + id = "#{@chest_id}:#{@key}" + id << ":#{@transaction.transaction_id}" if @transaction + return id + end + # + # Does our target respond to +meth+? + # + def respond_to?(meth) + # This one will be called far too often, and it seems safe to ignore it. + return false if meth == :marshal_dump + return super(meth) || self.method_missing(:respond_to?, meth) + end + # + # Call +meth+ with +args+ and +block+ on our target if it responds to + # it. + # + def method_missing(meth, *args, &block) + begin + return @chest.call_instance_method(@key, meth, @transaction, *args, &block) + rescue DRb::DRbConnError => e + if defined?(Archipelago::Disco::MC) + possible_replacements = Archipelago::Disco::MC.lookup(Archipelago::Disco::Query.new({:service_id => @chest_id})) + raise e if possible_replacements.empty? + + @chest = possible_replacements[@chest_id][:service] + CHEST_BY_SERVICE_ID[@chest_id] = @chest + + retry + else + raise e + end + end + end + + end + + # + # A possibly remote database that only returns proxies to its + # contents, and thus runs all methods on its contents itself. + # + # Has support for optimistically locked distributed serializable transactions. + # + class Chest + + # + # The Chest can be published. + # + include Archipelago::Disco::Publishable + + # + # Initialize a Chest + # + # Will use a BerkeleyHashishProvider using treasure_chest.db in the same dir to get its hashes + # if not :persistence_provider is given. + # + # Will try to recover crashed transaction every :transaction_recovery_interval seconds + # or TRANSACTION_RECOVERY_INTERVAL if none is given. + # + # Will use Archipelago::Disco::Publishable by calling initialize_publishable with +options+. + # + def initialize(options = {}) + # + # The provider of happy magic persistent hashes of different kinds. + # + @persistence_provider = options[:persistence_provider] || Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(File.expand_path(__FILE__)).parent.join("treasure_chest.db")) + + # + # Use the given options to initialize the publishable + # instance variables. + # + initialize_publishable(options) + + # + # [transaction => [key => instance]] + # To know what stuff is visible to a transaction. + # + @snapshot_by_transaction = {} + @snapshot_by_transaction.extend(Archipelago::Current::Synchronized) + + # + # [transaction => [key => when the key was read/updated/deleted] + # To know if a transaction is ok to prepare and commit. + # + @timestamp_by_key_by_transaction = {} + + # + # [transaction => [key => instance]] + # To know what transactions were prepared but not + # properly finished last run. + # + @crashed = Set.new + + initialize_seen_data + + # + # The magical persistent map that defines how we actually + # store our data. + # + @db = @persistence_provider.get_cached_hashish("db") + + initialize_prepared(options[:transaction_recovery_interval] || TRANSACTION_RECOVERY_INTERVAL) + + CHEST_BY_SERVICE_ID[self.service_id] = self + end + + # + # The transactions active in this Chest. + # + def active_transactions + @snapshot_by_transaction.keys.clone + end + + # + # Will do +callable+.call(key, value) + # for each key-and-value pair in this Chest. + # + # NB: This is totaly thread-unsafe, only do this + # for management or rescue! + # + def each(callable) + @db.each(callable) + end + + # + # Evaluate +data+ if we have not already seen +label+ or if we have an earlier +timestamp+ than the one given. + # + def evaluate!(label, timestamp, data) + serialized_label = Marshal.dump(label) + go = false + + if @seen_data.include?(serialized_label) + last_timestamp, last_data = Marshal.load(@seen_data[serialized_label]) + go = true if timestamp > last_timestamp + else + go = true + end + + if go + begin + Object.class_eval(data) + @seen_data[serialized_label] = Marshal.dump([timestamp, data]) + rescue Exception => e + @seen_data[serialized_label] = Marshal.load([timestamp, nil]) + puts e + pp e.backtrace + end + end + end + + # + # Returns true if this Chest includes the given +key+, optionally within a +transaction+. + # + def include?(key, transaction = nil) + join!(transaction) + + if transaction + return @snapshot_by_transaction[transaction].include?(key) + else + return @db.include?(key) + end + + end + + # + # Return the contents of this chest using a given +key+ and +transaction+. + # + def [](key, transaction = nil) + join!(transaction) + + instance = ensure_instance_with_transaction(key, transaction) + return nil unless instance + + if Dubloon === instance + return instance.join(transaction) + else + return Dubloon.new(key, self, transaction, self.service_id) + end + end + + # + # Delete the value of +key+ within +transaction+. + # + def delete(key, transaction = nil) + join!(transaction) + + rval = nil + + if transaction + # + # If we have a transaction we must note that it is deleted in a + # separate space for that transaction. + # + snapshot = @snapshot_by_transaction[transaction] + snapshot.synchronize do + + rval = snapshot[key] + + snapshot[key] = :deleted + # + # Make sure we remember when it was last changed according to our main db. + # + timestamps = @timestamp_by_key_by_transaction[transaction] + timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) + + end + else + # + # Otherwise just ask our persistence provider to delete it. + # + @db.delete(key) + end + + rval.freeze + + return rval + end + + # + # Put something into this chest with a given +key+, +value+ and + # +transaction+. + # + def []=(key, p1, p2 = nil) + if p2 + value = p2 + transaction = p1 + else + value = p1 + transaction = nil + end + + return set(key, value, transaction) + end + + # + # Call an instance +method+ on whatever this chest holds at +key+ + # with any +transaction+ and +args+. + # + # If a +transaction+ is provided and the value for the +key+ + # respond_to?(:with_transaction) then the actual method call + # will be wrapped within the block sent to with_transaction(transaction, &block). + # + def call_instance_method(key, method, transaction, *arguments, &block) + if transaction + return call_with_transaction(key, method, transaction, *arguments, &block) + else + return call_without_transaction(key, method, *arguments, &block) + end + end + + # + # Abort +transaction+ in this Chest. + # + def abort!(transaction) + assert_transaction(transaction) + + snapshot = @snapshot_by_transaction[transaction] + # + # Make sure nobody can modify this transaction while we + # are aborting it. + # + snapshot.synchronize do + serialized_transaction = Marshal.dump(transaction) + + # + # If this transaction was successfully prepared + # + if @snapshot_by_transaction_db.include?(serialized_transaction) + # + # Unlock the keys that are part of it. + # + snapshot.each do |key, value| + @db.unlock_on(key) + end + # + # And remove it from persistent storage. + # + @snapshot_by_transaction_db[serialized_transaction] = nil + # + # And remove its timestamps from persistent storage. + # + @timestamp_by_key_by_transaction_db[serialized_transaction] = nil + end + # + # Finally delete it from the snapshots. + # + @snapshot_by_transaction.delete(transaction) + # + # And from the timestamps. + # + @timestamp_by_key_by_transaction.delete(transaction) + end + end + + # + # Prepares +transaction+ in this Chest. + # + # NB: This will cause any update of the data within + # this transaction to block until it is either aborted + # or commited! + # + def prepare!(transaction) + assert_transaction(transaction) + + # + # If we dont know about this transaction then it can't very well + # affect us. + # + return :unchanged unless @snapshot_by_transaction.include?(transaction) + + snapshot = @snapshot_by_transaction[transaction] + # + # Make sure nobody can modify this transaction while we are + # preparing it. + # + snapshot.synchronize do + + # + # Remember what locks we acquire so that we can + # unlock them in case of failure. + # + locks = [] + timestamp_by_key = @timestamp_by_key_by_transaction[transaction] + # + # Acquire a lock on each key in the transaction + # + snapshot.each do |key, value| + if @db.timestamp(key) == timestamp_by_key[key] + @db.lock_on(key) + locks << key + else + locks.each do |key| + @db.unlock_on(key) + end + return :abort + end + end + serialized_transaction = Marshal.dump(transaction) + + # + # Dump its state to persistent storage. + # + @snapshot_by_transaction_db[serialized_transaction] = Marshal.dump(snapshot) + # + # And dump its timestamps to persistent storage + # + @timestamp_by_key_by_transaction_db[serialized_transaction] = Marshal.dump(timestamp_by_key) + return :prepared + end + end + + # + # Commits +transaction+ in this Chest. + # + # NB: Transaction must be prepared before commit is called. + # + def commit!(transaction) + assert_transaction(transaction) + raise IllegalCommitException.new(self, transaction) unless @snapshot_by_transaction_db.include?(Marshal.dump(transaction)) + + snapshot = @snapshot_by_transaction[transaction] + # + # Make sure nobody can modify this transaction while we are + # commiting it. + # + snapshot.synchronize do + + # + # Copy each key and value from our private space to the real space + # + snapshot.each do |key, value| + if value == :deleted + @db.delete(key) + else + @db[key] = value + end + end + + # + # Call abort! to clean up after the transaction. + # + abort!(transaction) + + end + end + + private + + # + # Evaluates all data we have been told to evaluate in earlier runs. + # + def initialize_seen_data + # + # [label => [timestamp, data]] + # To remember what and when we have evaluated and be able to evaluate it again. + # + @seen_data = @persistence_provider.get_hashish("seen_data") + @seen_data.each do |serialized_label, serialized_pair| + timestamp, data = Marshal.load(serialized_pair) + + if data + begin + Object.class_eval(data) + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Allocates space for this +transaction+. + # + # Will also call +transaction+.join to make sure + # it is aware of us. + # + def join!(transaction) + if transaction + if transaction.state == :active + @snapshot_by_transaction.synchronize do + unless @snapshot_by_transaction.include?(transaction) + @snapshot_by_transaction[transaction] = {} + @snapshot_by_transaction[transaction].extend(Archipelago::Current::Synchronized) + @timestamp_by_key_by_transaction[transaction] = {} + transaction.join(self) + end + end + else + raise IllegalJoinException.new(transaction) + end + end + end + + # + # Raises if we are not in this transaction. + # + def assert_transaction(transaction) + raise UnknownTransactionException.new(self, transaction) unless @snapshot_by_transaction.include?(transaction) + end + + # + # Call a +method+ within a +transaction+. + # + # If the object we want to run the +method+ on respond_to?(:with_transaction) + # then we will execute the actual +method+ within a block sent to the with_transaction method. + # + def call_with_transaction(key, method, transaction, *arguments, &block) + assert_transaction(transaction) + + # + # Fetch our instance from the snapshot. + # + snapshot = @snapshot_by_transaction[transaction] + instance = snapshot[key] + instance = nil if instance == :deleted + + raise UnknownObjectException.new(self, key, transaction) unless instance + + begin + if instance.respond_to?(:with_transaction) + return_value = nil + instance.with_transaction(transaction) do + return_value = instance.send(method, *arguments, &block) + end + return return_value + else + return instance.send(method, *arguments, &block) + end + ensure + # + # Make sure we remember when this object was last changed according + # to the main db. + # + snapshot.synchronize do + timestamps = @timestamp_by_key_by_transaction[transaction] + timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) + end + end + end + + # + # Call a method outside any transaction (ie inside a transaction of its own). + # + def call_without_transaction(key, method, *arguments, &block) + instance = @db[key] + + raise UnknownObjectException(self, key) unless instance + + begin + return instance.send(method, *arguments, &block) + ensure + @db.store_if_changed(key) + end + end + + # + # Initializes our storage of prepared transactions. + # + def initialize_prepared(transaction_recovery_interval) + # + # Load stored timestamps for our transaction from db. + # + @timestamp_by_key_by_transaction_db = @persistence_provider.get_hashish("prepared_timestamps") + @timestamp_by_key_by_transaction_db.each do |serialized_transaction, serialized_timestamps| + @timestamp_by_key_by_transaction[Marshal.load(serialized_transaction)] = Marshal.load(serialized_timestamps) + end + + # + # Load stored snapshots for our transaction from db. + # + @snapshot_by_transaction_db = @persistence_provider.get_hashish("prepared") + @snapshot_by_transaction_db.each do |serialized_transaction, serialized_snapshot| + transaction = Marshal.load(transaction) + + @crashed << transaction + @snapshot_by_transaction[transaction] = Marshal.load(serialized_snapshot) + end + start_recovery_thread(transaction_recovery_interval) + end + + # + # Starts the thread that will keep trying to recover + # our crashed transactions. + # + def start_recovery_thread(transaction_recovery_interval) + Thread.new do + loop do + begin + @crashed.clone.each do |transaction| + begin + case transaction.state + when :commited + commit!(transaction) + @crashed.delete(transaction) + when :aborted + abort!(transaction) + @crashed.delete(transaction) + end + rescue Archipelago::Tranny::UnknownTransactionException => e + abort!(transaction) + @crashed.delete(transaction) + end + end + sleep(transaction_recovery_interval) + rescue Exception => e + puts e + pp e.backtrace + end + end + end + end + + # + # Insert +value+ under +key+ and +transaction+ + # in this chest. + # + def set(key, value, transaction) + join!(transaction) + value.assert_transaction(transaction) if Dubloon === value + + if transaction + snapshot = @snapshot_by_transaction[transaction] + + # + # If we have a transaction we must put it in a + # separate space for that transaction. + # + snapshot.synchronize do + + snapshot[key] = value + # + # Make sure we remember the last time this was changed according to + # our main db. + # + timestamps = @timestamp_by_key_by_transaction[transaction] + timestamps[key] = @db.timestamp(key) unless timestamps.include?(key) + + end + else + @db[key] = value + end + + return value if Dubloon === value + + return Dubloon.new(key, self, transaction, service_id) + end + + # + # Try to fetch the data of +key+ from the private space + # of +transaction+ and put it there if it was not there + # already. + # + def ensure_instance_with_transaction(key, transaction) + if transaction + snapshot = @snapshot_by_transaction[transaction] + snapshot.synchronize do + + # + # If we dont have this key in the snapshot. + # + unless snapshot.include?(key) + # + # Fetch the new value for the snapshot + # + new_value = @db.get_deep_clone(key) + # + # If it exists then copy it to the snapshot + # otherwise remove the transaction hash if it is empty. + # + if new_value + snapshot[key] = new_value + else + @snapshot_by_transaction.delete(transaction) if snapshot.empty? + end + end + + rval = snapshot[key] + return rval == :deleted ? nil : rval + + end + else + return @db[key] + end + end + + end + + end + +end Deleted: tags/release_0_2_3/archipelago/tests/current_test.rb =================================================================== --- trunk/archipelago/tests/current_test.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/tests/current_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,56 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'test_helper') - -class CurrentTest < Test::Unit::TestCase - - def test_synchronized - t = true - - a = "hej" - a.extend(Archipelago::Current::Synchronized) - - o = "gnu" - a.synchronize_on(o) do - Thread.new do - a.synchronize_on(o) do - t = false - end - end - Thread.pass - assert(t) - end - assert_within(0.5) do - !t - end - end - - def test_threaded_collection - a = Array(10) - a.extend(Archipelago::Current::ThreadedCollection) - - assert_equal(a.select do |e| e == nil end, - a.t_select do |e| e == nil end) - assert_equal(a.reject do |e| e == nil end, - a.t_reject do |e| e == nil end) - assert_equal(a.collect do |e| e == nil end, - a.t_collect do |e| e == nil end) - b = [] - a.t_each do |e| b << e end - assert_equal(b, a) - - a = {1 => 1, 2 => 2, 3 => 3, 4 => 4, 5 => 5} - a.extend(Archipelago::Current::ThreadedCollection) - - assert_equal(a.select do |k,v| k == nil end, - a.t_select do |k,v| k == nil end) - assert_equal(a.to_a.reject do |k,v| k == nil end, - a.t_reject do |k,v| k == nil end) - assert_equal(a.collect do |k,v| k == nil end, - a.t_collect do |k,c| k == nil end) - - b = {} - a.t_each do |k,v| b[k] = v end - assert_equal(b,a) - end - -end Copied: tags/release_0_2_3/archipelago/tests/current_test.rb (from rev 82, trunk/archipelago/tests/current_test.rb) =================================================================== --- tags/release_0_2_3/archipelago/tests/current_test.rb (rev 0) +++ tags/release_0_2_3/archipelago/tests/current_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,100 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class HashCollector + attr_accessor :epa + def call(k,v) + @epa ||= {} + @epa[k] = v + end +end + +class ArrayCollector + attr_accessor :epa + def call(e) + @epa ||= [] + @epa << e + end +end + +class CurrentTest < Test::Unit::TestCase + + def test_synchronized + t = true + + a = "hej" + a.extend(Archipelago::Current::Synchronized) + + o = "gnu" + a.synchronize_on(o) do + Thread.new do + a.synchronize_on(o) do + t = false + end + end + Thread.pass + assert(t) + end + assert_within(0.5) do + !t + end + end + + def test_lock_on + t = true + + a = "hej" + a.extend(Archipelago::Current::Synchronized) + a.lock_on("epa") + Thread.new do + a.lock_on("epa") + t = false + end + Thread.pass + assert(t) + a.unlock_on("epa") + assert_within(0.5) do + !t + end + end + + def test_threaded_collection + a = Array(10) + a.extend(Archipelago::Current::ThreadedCollection) + + assert_equal(a.select do |e| e == nil end, + a.t_select do |e| e == nil end) + assert_equal(a.reject do |e| e == nil end, + a.t_reject do |e| e == nil end) + assert_equal(a.collect do |e| e == nil end, + a.t_collect do |e| e == nil end) + b = [] + a.t_each do |e| b << e end + assert_equal(b, a) + + a = {1 => 1, 2 => 2, 3 => 3, 4 => 4, 5 => 5} + a.extend(Archipelago::Current::ThreadedCollection) + + assert_equal(a.select do |k,v| k == nil end, + a.t_select do |k,v| k == nil end) + assert_equal(a.to_a.reject do |k,v| k == nil end, + a.t_reject do |k,v| k == nil end) + assert_equal(a.collect do |k,v| k == nil end, + a.t_collect do |k,c| k == nil end) + + b = {} + a.t_each do |k,v| b[k] = v end + assert_equal(b,a) + + c = HashCollector.new + a.t_each(c) + assert_equal(a, c.epa) + + a = [1,2,3,4,5,6,7] + a.extend(Archipelago::Current::ThreadedCollection) + c = ArrayCollector.new + a.t_each(c) + assert_equal(a.sort, c.epa.sort) + end + +end Deleted: tags/release_0_2_3/archipelago/tests/pirate_test.rb =================================================================== --- trunk/archipelago/tests/pirate_test.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/tests/pirate_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,111 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'test_helper') - -class PirateTest < Test::Unit::TestCase - - def setup - DRb.start_service("druby://localhost:#{rand(1000) + 5000}") - @p = Archipelago::Pirate::Captain.new(:chest_description => {:class => "TestChest"}, - :tranny_description => {:class => "TestManager"}) - @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) - @c.publish! - @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) - @c2.publish! - @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny.db"))) - @tm.publish! - @p.update_services! - assert_within(10) do - @p.chests.keys.sort == [@c.service_id, @c2.service_id].sort - end - assert_within(10) do - @p.trannies.keys == [@tm.service_id] - end - end - - def teardown - @p.stop! - @c.stop! - @c.persistence_provider.unlink - @c2.stop! - @c2.persistence_provider.unlink - @tm.stop! - @tm.persistence_provider.unlink - DRb.stop_service - end - - def test_each - @p["oj"] = "bla" - @p["brunt"] = "ja" - h = {} - @p.each(Proc.new do |k,v| - h[k] = v - end) - assert_equal({"oj" => "bla", "brunt" => "ja"}, h) - end - - def test_evaluate - assert_raise(NameError) do - e = Evaltest.new - end - - p2 = Archipelago::Pirate::Captain.new(:chest_description => {:class => "TestChest"}, - :tranny_description => {:class => "TestManager"}, - :chest_eval_files => [File.join(File.dirname(__FILE__), 'evaltest')]) - - assert_within(10) do - !p2.chests.empty? - end - - e = Evaltest.new - - assert_raise(NameError) do - e = Evaltestmore.new - end - - p2.evaluate!(File.join(File.dirname(__FILE__), "evaltestmore")) - - e = Evaltestmore.new - - p2.stop! - end - - def test_write_read - 0.upto(100) do |n| - @p["#{n}"] = "#{n}" - end - chest_ids = Set.new - 0.upto(100) do |n| - chest_ids << @c.service_id if @c["#{n}"] - chest_ids << @c2.service_id if @c2["#{n}"] - end - - assert(chest_ids.include?(@c.service_id), "This may fail due to bad luck, try again") - assert(chest_ids.include?(@c2.service_id), "This may fail due to bad luck, try again") - - @p["brupp"] = "jojo" - assert_equal("jojo", @p["brupp"]) - s1 = @p["brupp"] - s2 = @p["brupp"] - assert_equal(s1, s2) - end - - def test_write_read_transaction - $T = true - @p["hej"] = "haha" - p2 = @p.begin - assert_equal(p2["hej"], @p["hej"]) - p2["hej"] = "bums" - assert(p2["hej"] != @p["hej"]) - p2.commit! - assert_equal("bums", p2["hej"]) - assert_equal(p2["hej"], @p["hej"]) - - p2 = @p.begin - p2["hej"] = "glass" - assert(p2["hej"] != @p["hej"]) - p2.abort! - assert("bums", @p["hej"]) - $T = false - end - -end Copied: tags/release_0_2_3/archipelago/tests/pirate_test.rb (from rev 77, trunk/archipelago/tests/pirate_test.rb) =================================================================== --- tags/release_0_2_3/archipelago/tests/pirate_test.rb (rev 0) +++ tags/release_0_2_3/archipelago/tests/pirate_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,167 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class PirateTest < Test::Unit::TestCase + + def setup + DRb.start_service("druby://localhost:#{rand(1000) + 5000}") + @p = Archipelago::Pirate::Captain.new(:chest_description => {:class => "TestChest"}, + :tranny_description => {:class => "TestManager"}) + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny.db"))) + @tm.publish! + @p.update_services! + assert_within(10) do + @p.chests.keys.sort == [@c.service_id, @c2.service_id].sort + end + assert_within(10) do + @p.trannies.keys == [@tm.service_id] + end + end + + def teardown + @p.stop! + @c.stop! + @c.persistence_provider.unlink! + @c2.stop! + @c2.persistence_provider.unlink! + @tm.stop! + @tm.persistence_provider.unlink! + DRb.stop_service + end + + def test_include + assert(!@p.include?("blabla")) + @p["blabla"] = "kissobjas" + assert(@p.include?("blabla")) + t = @tm.begin + assert(!@p.include?("hehu")) + assert(!@p.include?("hehu", t)) + @p["hehu", t] = "brunte" + assert(!@p.include?("hehu")) + assert(@p.include?("hehu", t)) + t.commit! + assert(@p.include?("hehu")) + t = @tm.begin + assert(!@p.include?("hehu2")) + assert(!@p.include?("hehu2", t)) + @p["hehu2", t] = "brunte" + assert(!@p.include?("hehu2")) + assert(@p.include?("hehu2", t)) + t.abort! + assert(!@p.include?("hehu2")) + end + + def test_each + @p["oj"] = "bla" + @p["brunt"] = "ja" + h = {} + @p.each(Proc.new do |k,v| + h[k] = v + end) + assert_equal({"oj" => "bla", "brunt" => "ja"}, h) + end + + def test_evaluate + assert_raise(NameError) do + e = Evaltest.new + end + + p2 = Archipelago::Pirate::Captain.new(:chest_description => {:class => "TestChest"}, + :tranny_description => {:class => "TestManager"}, + :chest_eval_files => [File.join(File.dirname(__FILE__), 'evaltest')]) + + assert_within(10) do + !p2.chests.empty? + end + + e = Evaltest.new + + assert_raise(NameError) do + e = Evaltestmore.new + end + + p2.evaluate!(File.join(File.dirname(__FILE__), "evaltestmore")) + + e = Evaltestmore.new + + p2.stop! + end + + def test_write_read + 0.upto(100) do |n| + @p["#{n}"] = "#{n}" + end + chest_ids = Set.new + 0.upto(100) do |n| + chest_ids << @c.service_id if @c["#{n}"] + chest_ids << @c2.service_id if @c2["#{n}"] + end + + assert(chest_ids.include?(@c.service_id), "This may fail due to bad luck, try again") + assert(chest_ids.include?(@c2.service_id), "This may fail due to bad luck, try again") + + @p["brupp"] = "jojo" + assert_equal("jojo", @p["brupp"]) + s1 = @p["brupp"] + s2 = @p["brupp"] + assert_equal(s1, s2) + end + + def test_transaction + p2 = Archipelago::Pirate::Captain.new(:chest_description => {:class => "TestChest"}, + :tranny_description => {:class => "TestManager"}) + assert_within(10) do + p2.chests.keys.sort == [@c.service_id, @c2.service_id].sort + end + assert_within(10) do + p2.trannies.keys == [@tm.service_id] + end + trans = nil + assert_raise(Archipelago::Pirate::CommitFailedException) do + @p.transaction do |trans| + assert_equal(trans.transaction_id, @p.active_transaction.transaction_id) + @p["hehu"] = "haha" + assert(!p2.include?("hehu")) + assert(!p2.include?("hehu", nil)) + assert_equal(nil, p2.active_transaction) + trans2 = nil + p2.transaction do |trans2| + assert_equal(trans2.transaction_id, p2.active_transaction.transaction_id) + p2["hehu"] = "hoj" + assert_equal("haha", @p["hehu"]) + assert_equal("hoj", @p["hehu", trans2]) + end + assert_equal(:commited, trans2.state) + assert_equal(:active, trans.state) + assert_equal("hoj", p2["hehu"]) + assert_equal("haha", p2["hehu", trans]) + end + end + assert_equal(:aborted, trans.state) + assert_equal("hoj", @p["hehu"]) + end + + def test_write_read_transaction + $T = true + @p["hej"] = "haha" + p2 = @p.begin + assert_equal(p2["hej"], @p["hej"]) + p2["hej"] = "bums" + assert(p2["hej"] != @p["hej"]) + p2.commit! + assert_equal("bums", p2["hej"]) + assert_equal(p2["hej"], @p["hej"]) + + p2 = @p.begin + p2["hej"] = "glass" + assert(p2["hej"] != @p["hej"]) + p2.abort! + assert("bums", @p["hej"]) + $T = false + end + +end Deleted: tags/release_0_2_3/archipelago/tests/tranny_test.rb =================================================================== --- trunk/archipelago/tests/tranny_test.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/tests/tranny_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,68 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'test_helper') - -class TrannyTest < Test::Unit::TestCase - - def setup - DRb.start_service - @tm = TestManager.new(:db_path => Pathname.new(__FILE__).parent.join("tranny.db")) - end - - def teardown - @tm.persistence_provider.unlink - DRb.stop_service - end - - class Participant - include DRb::DRbUndumped - - attr_accessor :state - attr_accessor :mode - - def assert_equal(a,b) - raise "#{a} != #{b}" unless a == b - end - def initialize(mode) - @mode = mode - @state = :none - end - def abort!(t) - @state = :aborted - end - def prepare!(t) - assert_equal(:voting, t.state) - @mode - end - def commit!(t) - assert_equal(:commited, t.state) - @state = :commited - end - end - - def test_successful_commit - p1 = Participant.new(:commit) - p2 = Participant.new(:commit) - trans = @tm.begin - assert_equal(:active, trans.state) - trans.join(DRbObject.new(p1)) - trans.join(DRbObject.new(p2)) - assert_equal(:commited, trans.commit!) - assert_equal(:commited, trans.state) - assert_equal(:commited, p1.state) - assert_equal(:commited, p2.state) - end - - def test_aborted_commit - p1 = Participant.new(:commit) - p2 = Participant.new(:abort) - trans = @tm.begin - assert_equal(:active, trans.state) - trans.join(DRbObject.new(p1)) - trans.join(DRbObject.new(p2)) - assert_equal(:aborted, trans.commit!) - assert_equal(:aborted, trans.state) - assert_equal(:aborted, p1.state) - assert_equal(:none, p2.state) - end - -end Copied: tags/release_0_2_3/archipelago/tests/tranny_test.rb (from rev 76, trunk/archipelago/tests/tranny_test.rb) =================================================================== --- tags/release_0_2_3/archipelago/tests/tranny_test.rb (rev 0) +++ tags/release_0_2_3/archipelago/tests/tranny_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,68 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class TrannyTest < Test::Unit::TestCase + + def setup + DRb.start_service + @tm = TestManager.new(:db_path => Pathname.new(__FILE__).parent.join("tranny.db")) + end + + def teardown + @tm.persistence_provider.unlink! + DRb.stop_service + end + + class Participant + include DRb::DRbUndumped + + attr_accessor :state + attr_accessor :mode + + def assert_equal(a,b) + raise "#{a} != #{b}" unless a == b + end + def initialize(mode) + @mode = mode + @state = :none + end + def abort!(t) + @state = :aborted + end + def prepare!(t) + assert_equal(:voting, t.state) + @mode + end + def commit!(t) + assert_equal(:commited, t.state) + @state = :commited + end + end + + def test_successful_commit + p1 = Participant.new(:commit) + p2 = Participant.new(:commit) + trans = @tm.begin + assert_equal(:active, trans.state) + trans.join(DRbObject.new(p1)) + trans.join(DRbObject.new(p2)) + assert_equal(:commited, trans.commit!) + assert_equal(:commited, trans.state) + assert_equal(:commited, p1.state) + assert_equal(:commited, p2.state) + end + + def test_aborted_commit + p1 = Participant.new(:commit) + p2 = Participant.new(:abort) + trans = @tm.begin + assert_equal(:active, trans.state) + trans.join(DRbObject.new(p1)) + trans.join(DRbObject.new(p2)) + assert_equal(:aborted, trans.commit!) + assert_equal(:aborted, trans.state) + assert_equal(:aborted, p1.state) + assert_equal(:none, p2.state) + end + +end Deleted: tags/release_0_2_3/archipelago/tests/treasure_benchmark.rb =================================================================== --- trunk/archipelago/tests/treasure_benchmark.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/tests/treasure_benchmark.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,85 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'test_helper') - -class TreasureBenchmark < Test::Unit::TestCase - - def setup - DRb.start_service - @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) - end - - def teardown - @c.persistence_provider.unlink - DRb.stop_service - end - - def test_outside_transaction - k = "hehu" - v = "ojoj" - bm("Chest#[]=", :n => 1000) do - @c[k] = v - k = k.next - end - k = "hehu" - bm("Chest#[]", :n => 1000) do - t = @c[k] - k = k.next - end - k = "hehu" - bm("Chest#delete", :n => 1000) do - t = @c.delete(k) - k = k.next - end - @c[k] = v - v = @c[k] - bm("Dubloon#method_missing") do - t = v.size - end - end - - def test_inside_transaction - k = "hehu" - v = "ojoj" - tr = TestTransaction.new - bm("Chest#[]= (t)", :n => 1000) do - @c[k,tr] = v - k = k.next - end - k = "hehu" - bm("Chest#[] (t)", :n => 1000) do - t = @c[k,tr] - k = k.next - end - k = "hehu" - bm("Chest#delete (t)", :n => 1000) do - t = @c.delete(k,tr) - k = k.next - end - @c[k,tr] = v - v = @c[k,tr] - bm("Dubloon#method_missing (t)") do - t = v.upcase - end - end - - def test_transaction_overhead - k = "hehu" - v = "ojoj" - tr = TestTransaction.new - bm("Chest#join!/[]=/prepare!/commit!", :n => 1000) do - @c[k, tr] = v - @c.prepare!(tr) - @c.commit!(tr) - end - bm("Chest#join!/[]=/abort!", :n => 1000) do - @c[k, tr] = v - @c.abort!(tr) - end - bm("Chest#join!/[]=/prepare!/abort!", :n => 1000) do - @c[k, tr] = v - @c.prepare!(tr) - @c.abort!(tr) - end - end - -end Copied: tags/release_0_2_3/archipelago/tests/treasure_benchmark.rb (from rev 76, trunk/archipelago/tests/treasure_benchmark.rb) =================================================================== --- tags/release_0_2_3/archipelago/tests/treasure_benchmark.rb (rev 0) +++ tags/release_0_2_3/archipelago/tests/treasure_benchmark.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,85 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class TreasureBenchmark < Test::Unit::TestCase + + def setup + DRb.start_service + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + end + + def teardown + @c.persistence_provider.unlink! + DRb.stop_service + end + + def test_outside_transaction + k = "hehu" + v = "ojoj" + bm("Chest#[]=", :n => 1000) do + @c[k] = v + k = k.next + end + k = "hehu" + bm("Chest#[]", :n => 1000) do + t = @c[k] + k = k.next + end + k = "hehu" + bm("Chest#delete", :n => 1000) do + t = @c.delete(k) + k = k.next + end + @c[k] = v + v = @c[k] + bm("Dubloon#method_missing") do + t = v.size + end + end + + def test_inside_transaction + k = "hehu" + v = "ojoj" + tr = TestTransaction.new + bm("Chest#[]= (t)", :n => 1000) do + @c[k,tr] = v + k = k.next + end + k = "hehu" + bm("Chest#[] (t)", :n => 1000) do + t = @c[k,tr] + k = k.next + end + k = "hehu" + bm("Chest#delete (t)", :n => 1000) do + t = @c.delete(k,tr) + k = k.next + end + @c[k,tr] = v + v = @c[k,tr] + bm("Dubloon#method_missing (t)") do + t = v.upcase + end + end + + def test_transaction_overhead + k = "hehu" + v = "ojoj" + tr = TestTransaction.new + bm("Chest#join!/[]=/prepare!/commit!", :n => 1000) do + @c[k, tr] = v + @c.prepare!(tr) + @c.commit!(tr) + end + bm("Chest#join!/[]=/abort!", :n => 1000) do + @c[k, tr] = v + @c.abort!(tr) + end + bm("Chest#join!/[]=/prepare!/abort!", :n => 1000) do + @c[k, tr] = v + @c.prepare!(tr) + @c.abort!(tr) + end + end + +end Deleted: tags/release_0_2_3/archipelago/tests/treasure_test.rb =================================================================== --- trunk/archipelago/tests/treasure_test.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/archipelago/tests/treasure_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,295 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'test_helper') - -class A < String - def save_hook(old_value, &block) - $BURKMAT += 1 - yield - $BURKMAT2 += 1 - end -end - -class TreasureTest < Test::Unit::TestCase - - def setup - DRb.start_service - @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) - @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) - @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) - $BURKMAT = 0 - $BURKMAT2 = 0 - end - - def teardown - @c.persistence_provider.unlink - @c2.persistence_provider.unlink - @tm.persistence_provider.unlink - DRb.stop_service - end - - def test_each - @c["oj"] = "bla" - @c["brunt"] = "ja" - h = {} - @c.each(Proc.new do |k,v| - h[k] = v - end) - assert_equal({"oj" => "bla", "brunt" => "ja"}, h) - end - - def test_around_save - s = A.new("hehu") - @c["oj"] = s - assert_equal(1, $BURKMAT) - assert_equal(1, $BURKMAT2) - @c["oj"].upcase! - assert_equal(2, $BURKMAT) - assert_equal(2, $BURKMAT2) - end - - def test_eval - t = Time.now - @c.evaluate!("burk", t, "class Burk; end") - b = Burk.new - @c.evaluate!("burk", t, "class Bong; end") - assert_raise(NameError) do - b = Bong.new - end - @c.evaluate!("burk", Time.now + 10, "class Bong; end") - b = Bong.new - end - - def test_store_load_update - s = "hehu" - @c["oj"] = "hehu" - oj1 = @c["oj"] - assert_equal(oj1, @c["oj"]) - assert_equal("hehu", oj1) - - s2 = @c["oj"] - assert_equal(s, s2) - assert_equal(s.upcase, s2.upcase) - assert(s.object_id != s2.object_id) - - s2 << "kissekotte" - s << "kissekotte" - assert_equal(s, s2) - - s3 = @c["oj"] - assert_equal(s2, s3) - assert_equal(s2.object_id, s3.object_id) - end - - def test_multi_transaction - @c["oj"] = "hehu" - - t = @tm.begin - t2 = @tm.begin - - @c["oj", t] = "kiss" - @c["oj", t].upcase! - @c["oj", t2].upcase! - - assert_equal(:commited, t.commit!) - assert_equal(:aborted, t2.commit!) - assert_equal("KISS", @c["oj"]) - end - - def test_multi_chest - @c["oj"] = "hehu" - @c2["bla"] = "hirr" - - t = @tm.begin - - @c["oj", t].upcase! - @c2["bla", t].upcase! - - t.abort! - - assert_equal("hehu", @c["oj"]) - assert_equal("hirr", @c2["bla"]) - - t = @tm.begin - - @c["oj", t].upcase! - @c2["bla", t].upcase! - - assert_equal(:commited, t.commit!) - assert_equal("HEHU", @c["oj"]) - assert_equal("HIRR", @c2["bla"]) - - t = @tm.begin - - @c["oj", t] = "bark" - @c2["bla", t] = "boll" - @c2["bla"] = "burk" - - assert_equal(:aborted, t.commit!) - assert_equal("HEHU", @c["oj"]) - assert_equal("burk", @c2["bla"]) - - t = @tm.begin - t2 = @tm.begin - - s = @c["oj", t] - s2 = @c["oj", t2] - s3 = @c2["bla", t] - - s.swapcase! - s2.swapcase! - s3.swapcase! - - assert_equal(:commited, t2.commit!) - assert_equal(:aborted, t.commit!) - assert_equal("hehu", @c["oj"]) - assert_equal("burk", @c2["bla"]) - end - - def test_chained_linking - @c["oj"] = "hehu" - s = @c["oj"] - @c2["oj"] = s - s2 = @c2["oj"] - s2.swapcase! - assert_equal("HEHU", @c["oj"]) - assert_equal("HEHU", s2) - assert_equal("HEHU", @c2["oj"]) - end - - def test_transaction_store_load_update_commit - @c["oj"] = "hehu" - oj = @c["oj"] - - t = @tm.begin - - @c["oj", t] = "hirr" - oj2 = @c["oj", t] - - assert(oj != oj2) - assert_equal("hehu", oj) - assert_equal("hirr", oj2) - - assert_equal(:commited, t.commit!) - assert_raise(Archipelago::Treasure::UnknownTransactionException) do - oj2.size - end - assert_raise(Archipelago::Treasure::IllegalJoinException) do - @c["oj", t] - end - - assert_equal("hirr", oj) - assert_equal("hirr", @c["oj"]) - - t = @tm.begin - @c["oj", t] = "blar" - @c["oj"] = "kuk" - assert_equal(:aborted, t.commit!) - assert_equal("kuk", @c["oj"]) - - t = @tm.begin - @c["oj", t] = "bungk" - @c.delete("oj") - assert_equal(:aborted, t.commit!) - assert_equal(nil, @c["oj"]) - - t = @tm.begin - @c["oj"] = "baba" - @c.delete("oj", t) - assert_equal(:commited, t.commit!) - assert_equal(nil, @c["oj"]) - - t = @tm.begin - @c["oj"] = "bagi" - @c.delete("oj", t) - @c["oj"] = "ooqoq" - assert_equal(:aborted, t.commit!) - assert_equal("ooqoq", @c["oj"]) - - @c.delete("oj") - t = @tm.begin - @c["oj", t] = "ghgh" - assert_equal(nil, @c["oj"]) - assert_equal(:commited, t.commit!) - assert_equal("ghgh", @c["oj"]) - end - - def test_transaction_store_load_update_abort - @c["oj"] = "hehu" - oj1 = @c["oj"] - - assert(@c.active_transactions.empty?) - - t = @tm.begin - oj2 = @c["oj", t] - assert_equal(oj1, oj2) - - @c["oj", t] = "hehu2" - - assert_equal([t], @c.active_transactions) - - assert_equal("hehu2", @c["oj", t]) - assert_equal("hehu2", oj2) - assert("hehu2" != @c["oj"]) - assert("hehu2" != oj1) - assert(@c["oj"] != @c["oj", t]) - assert(oj1 != oj2) - assert_equal(@c["oj", t], @c["oj", t]) - assert_equal(oj2, @c["oj", t]) - - t2 = @tm.begin - assert_equal(@c["oj"], @c["oj", t2]) - oj3 = @c["oj", t2] - assert_equal(oj1, oj3) - assert(oj2 != oj3) - - @c["oj", t2] = "bar" - assert_equal(oj3, @c["oj", t2]) - assert(@c["oj", t] != @c["oj", t2]) - assert(oj2 != oj3) - - assert_equal("hehu", @c["oj"]) - assert_equal("hehu", oj1) - - assert_equal(tranny_sort([t, t2]), - tranny_sort(@c.active_transactions)) - - oj1.replace("brumma") - oj2.replace("bajs") - - assert_equal("brumma", @c["oj"]) - assert_equal("bar", @c["oj", t2]) - assert_equal("bajs", @c["oj", t]) - - @c.delete("oj") - @c.delete("oj", t) - - assert_equal("bar", @c["oj", t2]) - assert_equal(nil, @c["oj", t]) - - t.abort! - - assert_equal([t2], @c.active_transactions) - assert_raise(Archipelago::Treasure::IllegalJoinException) do - @c["oj", t] - end - assert_raise(Archipelago::Treasure::UnknownTransactionException) do - "hej" == oj2 - end - - t2.abort! - - assert_raise(Archipelago::Treasure::UnknownTransactionException) do - oj3.size - end - assert(@c.active_transactions.empty?) - end - - private - - def tranny_sort(a) - a.sort do |x,y| - x.transaction_id <=> y.transaction_id - end - end - -end Copied: tags/release_0_2_3/archipelago/tests/treasure_test.rb (from rev 78, trunk/archipelago/tests/treasure_test.rb) =================================================================== --- tags/release_0_2_3/archipelago/tests/treasure_test.rb (rev 0) +++ tags/release_0_2_3/archipelago/tests/treasure_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,353 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class A < String + def save_hook(old_value, &block) + $BURKMAT += 1 + yield + $BURKMAT2 += 1 + end + def load_hook(&block) + $BURKMAT3 += 1 + yield + $BURKMAT4 += 1 + end + def with_transaction(transaction, &block) + $BURKMAT7 = transaction + $BURKMAT5 += 1 + yield + $BURKMAT6 += 1 + end +end + +class TreasureTest < Test::Unit::TestCase + + def setup + DRb.start_service + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + $BURKMAT = 0 + $BURKMAT2 = 0 + $BURKMAT3 = 0 + $BURKMAT4 = 0 + $BURKMAT5 = 0 + $BURKMAT6 = 0 + $BURKMAT7 = nil + end + + def teardown + @c.persistence_provider.unlink! + @c2.persistence_provider.unlink! + @tm.persistence_provider.unlink! + DRb.stop_service + end + + def test_with_transaction + t = @tm.begin + @c["hehu"] = A.new("kiss") + @c["hehu", t].upcase! + assert_equal(1, $BURKMAT5) + assert_equal(1, $BURKMAT6) + assert(!$BURKMAT7.nil?) + end + + def test_each + @c["oj"] = "bla" + @c["brunt"] = "ja" + h = {} + @c.each(Proc.new do |k,v| + h[k] = v + end) + assert_equal({"oj" => "bla", "brunt" => "ja"}, h) + end + + def test_include + assert(!@c.include?("blabla")) + @c["blabla"] = "kissobjas" + assert(@c.include?("blabla")) + t = @tm.begin + assert(!@c.include?("hehu")) + assert(!@c.include?("hehu", t)) + @c["hehu", t] = "brunte" + assert(!@c.include?("hehu")) + assert(@c.include?("hehu", t)) + t.commit! + assert(@c.include?("hehu")) + t = @tm.begin + assert(!@c.include?("hehu2")) + assert(!@c.include?("hehu2", t)) + @c["hehu2", t] = "brunte" + assert(!@c.include?("hehu2")) + assert(@c.include?("hehu2", t)) + t.abort! + assert(!@c.include?("hehu2")) + end + + def test_around_load + @c["brunis"] = A.new("hirr") + @c.persistence_provider.close! + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + a = @c["brunis"] + assert_equal(1, $BURKMAT3) + assert_equal(1, $BURKMAT4) + end + + def test_around_save + s = A.new("hehu") + @c["oj"] = s + assert_equal(1, $BURKMAT) + assert_equal(1, $BURKMAT2) + @c["oj"].upcase! + assert_equal(2, $BURKMAT) + assert_equal(2, $BURKMAT2) + end + + def test_eval + t = Time.now + @c.evaluate!("burk", t, "class Burk; end") + b = Burk.new + @c.evaluate!("burk", t, "class Bong; end") + assert_raise(NameError) do + b = Bong.new + end + @c.evaluate!("burk", Time.now + 10, "class Bong; end") + b = Bong.new + end + + def test_store_load_update + s = "hehu" + assert(!@c.include?("oj")) + @c["oj"] = "hehu" + assert(@c.include?("oj")) + oj1 = @c["oj"] + assert_equal(oj1, @c["oj"]) + assert_equal("hehu", oj1) + + s2 = @c["oj"] + assert_equal(s, s2) + assert_equal(s.upcase, s2.upcase) + assert(s.object_id != s2.object_id) + + s2 << "kissekotte" + s << "kissekotte" + assert_equal(s, s2) + + s3 = @c["oj"] + assert_equal(s2, s3) + assert_equal(s2.object_id, s3.object_id) + end + + def test_multi_transaction + @c["oj"] = "hehu" + + t = @tm.begin + t2 = @tm.begin + + @c["oj", t] = "kiss" + @c["oj", t].upcase! + @c["oj", t2].upcase! + + assert_equal(:commited, t.commit!) + assert_equal(:aborted, t2.commit!) + assert_equal("KISS", @c["oj"]) + end + + def test_multi_chest + @c["oj"] = "hehu" + @c2["bla"] = "hirr" + + t = @tm.begin + + @c["oj", t].upcase! + @c2["bla", t].upcase! + + t.abort! + + assert_equal("hehu", @c["oj"]) + assert_equal("hirr", @c2["bla"]) + + t = @tm.begin + + @c["oj", t].upcase! + @c2["bla", t].upcase! + + assert_equal(:commited, t.commit!) + assert_equal("HEHU", @c["oj"]) + assert_equal("HIRR", @c2["bla"]) + + t = @tm.begin + + @c["oj", t] = "bark" + @c2["bla", t] = "boll" + @c2["bla"] = "burk" + + assert_equal(:aborted, t.commit!) + assert_equal("HEHU", @c["oj"]) + assert_equal("burk", @c2["bla"]) + + t = @tm.begin + t2 = @tm.begin + + s = @c["oj", t] + s2 = @c["oj", t2] + s3 = @c2["bla", t] + + s.swapcase! + s2.swapcase! + s3.swapcase! + + assert_equal(:commited, t2.commit!) + assert_equal(:aborted, t.commit!) + assert_equal("hehu", @c["oj"]) + assert_equal("burk", @c2["bla"]) + end + + def test_chained_linking + @c["oj"] = "hehu" + s = @c["oj"] + @c2["oj"] = s + s2 = @c2["oj"] + s2.swapcase! + assert_equal("HEHU", @c["oj"]) + assert_equal("HEHU", s2) + assert_equal("HEHU", @c2["oj"]) + end + + def test_transaction_store_load_update_commit + @c["oj"] = "hehu" + oj = @c["oj"] + + t = @tm.begin + + @c["oj", t] = "hirr" + oj2 = @c["oj", t] + + assert(oj != oj2) + assert_equal("hehu", oj) + assert_equal("hirr", oj2) + + assert_equal(:commited, t.commit!) + assert_raise(Archipelago::Treasure::UnknownTransactionException) do + oj2.size + end + assert_raise(Archipelago::Treasure::IllegalJoinException) do + @c["oj", t] + end + + assert_equal("hirr", oj) + assert_equal("hirr", @c["oj"]) + + t = @tm.begin + @c["oj", t] = "blar" + @c["oj"] = "kuk" + assert_equal(:aborted, t.commit!) + assert_equal("kuk", @c["oj"]) + + t = @tm.begin + @c["oj", t] = "bungk" + @c.delete("oj") + assert_equal(:aborted, t.commit!) + assert_equal(nil, @c["oj"]) + + t = @tm.begin + @c["oj"] = "baba" + @c.delete("oj", t) + assert_equal(:commited, t.commit!) + assert_equal(nil, @c["oj"]) + + t = @tm.begin + @c["oj"] = "bagi" + @c.delete("oj", t) + @c["oj"] = "ooqoq" + assert_equal(:aborted, t.commit!) + assert_equal("ooqoq", @c["oj"]) + + @c.delete("oj") + t = @tm.begin + @c["oj", t] = "ghgh" + assert_equal(nil, @c["oj"]) + assert_equal(:commited, t.commit!) + assert_equal("ghgh", @c["oj"]) + end + + def test_transaction_store_load_update_abort + @c["oj"] = "hehu" + oj1 = @c["oj"] + + assert(@c.active_transactions.empty?) + + t = @tm.begin + oj2 = @c["oj", t] + assert_equal(oj1, oj2) + + @c["oj", t] = "hehu2" + + assert_equal([t], @c.active_transactions) + + assert_equal("hehu2", @c["oj", t]) + assert_equal("hehu2", oj2) + assert("hehu2" != @c["oj"]) + assert("hehu2" != oj1) + assert(@c["oj"] != @c["oj", t]) + assert(oj1 != oj2) + assert_equal(@c["oj", t], @c["oj", t]) + assert_equal(oj2, @c["oj", t]) + + t2 = @tm.begin + assert_equal(@c["oj"], @c["oj", t2]) + oj3 = @c["oj", t2] + assert_equal(oj1, oj3) + assert(oj2 != oj3) + + @c["oj", t2] = "bar" + assert_equal(oj3, @c["oj", t2]) + assert(@c["oj", t] != @c["oj", t2]) + assert(oj2 != oj3) + + assert_equal("hehu", @c["oj"]) + assert_equal("hehu", oj1) + + assert_equal(tranny_sort([t, t2]), + tranny_sort(@c.active_transactions)) + + oj1.replace("brumma") + oj2.replace("bajs") + + assert_equal("brumma", @c["oj"]) + assert_equal("bar", @c["oj", t2]) + assert_equal("bajs", @c["oj", t]) + + @c.delete("oj") + @c.delete("oj", t) + + assert_equal("bar", @c["oj", t2]) + assert_equal(nil, @c["oj", t]) + + t.abort! + + assert_equal([t2], @c.active_transactions) + assert_raise(Archipelago::Treasure::IllegalJoinException) do + @c["oj", t] + end + assert_raise(Archipelago::Treasure::UnknownTransactionException) do + "hej" == oj2 + end + + t2.abort! + + assert_raise(Archipelago::Treasure::UnknownTransactionException) do + oj3.size + end + assert(@c.active_transactions.empty?) + end + + private + + def tranny_sort(a) + a.sort do |x,y| + x.transaction_id <=> y.transaction_id + end + end + +end Deleted: tags/release_0_2_3/hyperactive/README =================================================================== --- trunk/hyperactive/README 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/hyperactive/README 2006-11-29 05:45:13 UTC (rev 85) @@ -1,34 +0,0 @@ -= This is HyperactiveRecord, a not-at-all drop-in replacement for ActiveRecord::Base - -It uses archipelago for persistence, and is meaningful only in an environment where the server process doesnt restart at each request. This means that cgi environment is not really an option. - -== Dependencies: - -Hyperactive::Tree:: RBTree: http://raa.ruby-lang.org/project/ruby-rbtree - -== Sub packages: - -Hyperactive::Record:: The base class itself, providing you with cached selectors and rejectors, along with cached finders for any number of attributes. -Hyperactive::Tree:: A collection class that contains any number of Hyperactive::Records in a tree structure. Scales logarithmically, so use with care. - -== Examples: - -To define a Record subclass that has the properties @active and @city that are indexed in two ways: - - class MyClass < Hyperactive::Record - attr_accessor :active, :city - select(:active_records, Proc.new do |record| - record.active == true - end) - index_by(:city) - end - -This will allow you to find all active MyClass instances by: - - MyClass.active_records - -Or finding all MyClass instances connected to Stockholm by: - - MyClass.find_by_city("Stockholm") - - Copied: tags/release_0_2_3/hyperactive/README (from rev 83, trunk/hyperactive/README) =================================================================== --- tags/release_0_2_3/hyperactive/README (rev 0) +++ tags/release_0_2_3/hyperactive/README 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,41 @@ += This is HyperactiveRecord, a not-at-all drop-in replacement for ActiveRecord::Base + +It uses archipelago for persistence, and is meaningful only in an environment where the server process doesnt restart at each request. This means that cgi environment is not really an option. + +== Sub packages: + +Hyperactive::Record:: The base class package itself, providing you with cached selectors and rejectors, along with cached finders for any number of attributes. +Hyperactive::Hash:: A collection class that contains any number of Hyperactive::Records in a hash like structure. +Hyperactive::List:: A collection class that contains any number of Hyperactive::Records in a list like structure. + +== Usage: + +To use Hyperactive in the simplest way, just run the script/services.rb script from the Archipelago +distribution to get a few services running, then subclass Hyperactive::Record::Bass and create +objects with MyBassSubclass.get_instance (not new!). They will be automagically +stored in the network, and fetchable via their instance.record_id. + +By loading Hyperactive::Record you will automatically define Hyperactive::Record::CAPTAIN which will +be an Archipelago::Pirate::Captain that is your interface to the distributed database. + +== Examples: + +To define a Record subclass that has the properties @active and @city that are indexed in two ways: + + class MyClass < Hyperactive::Record + attr_accessor :active, :city + select(:active_records, Proc.new do |record| + record.active == true + end) + index_by(:city) + end + +This will allow you to find all active MyClass instances by: + + MyClass.active_records + +Or finding all MyClass instances connected to Stockholm by: + + MyClass.find_by_city("Stockholm") + + Deleted: tags/release_0_2_3/hyperactive/Rakefile =================================================================== --- trunk/hyperactive/Rakefile 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/hyperactive/Rakefile 2006-11-29 05:45:13 UTC (rev 85) @@ -1,60 +0,0 @@ - -require 'rake' -require 'rake/testtask' -require 'rubygems' -Gem::manage_gems -require 'rake/gempackagetask' - - -spec = Gem::Specification.new do |s| - s.platform = Gem::Platform::RUBY - s.name = "hyperactive" - s.version = "0.2.2" - s.author = "Martin Kihlgren" - s.email = "zond at troja dot ath dot cx" - s.summary = "A base class for persistent objects that uses archipelago for persistence. Useful for Ruby on Rails models for example." - s.files = FileList['lib/hyperactive.rb', 'lib/hyperactive/*.rb', 'tests/*', 'GPL-2'].to_a - s.require_path = "lib" - s.autorequire = "hyperactive" - s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') - s.has_rdoc = true - s.rdoc_options << '--line-numbers' - s.rdoc_options << '--inline-source' - s.extra_rdoc_files = ["README"] - s.add_dependency('archipelago', '>= 0.2.0') -end - - -SOURCE_FILES = FileList.new do |fl| - [ "lib", "tests" ].each do |dir| - fl.include "#{dir}/**/*" - end - fl.include "Rakefile" -end - -Rake::GemPackageTask.new(spec) do |pkg| - pkg.need_tar = true -end - -task :default => [:units] do -end - -desc "Run all tests" -Rake::TestTask.new(:units) do |t| - t.pattern = 'tests/*_test.rb' - t.verbose = true - t.warning = true -end - -desc "Run all benchmarks" -Rake::TestTask.new(:bench) do |t| - t.pattern = 'tests/*_benchmark.rb' - t.verbose = true - t.warning = true -end - -desc "Package a gem from the source" -task :gem => "pkg/#{spec.name}-#{spec.version}.gem" do - puts "generated latest version" -end - Copied: tags/release_0_2_3/hyperactive/Rakefile (from rev 84, trunk/hyperactive/Rakefile) =================================================================== --- tags/release_0_2_3/hyperactive/Rakefile (rev 0) +++ tags/release_0_2_3/hyperactive/Rakefile 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,60 @@ + +require 'rake' +require 'rake/testtask' +require 'rubygems' +Gem::manage_gems +require 'rake/gempackagetask' + + +spec = Gem::Specification.new do |s| + s.platform = Gem::Platform::RUBY + s.name = "hyperactive" + s.version = "0.2.3" + s.author = "Martin Kihlgren" + s.email = "zond at troja dot ath dot cx" + s.summary = "A base class for persistent objects that uses archipelago for persistence. Useful for Ruby on Rails models for example." + s.files = FileList['lib/hyperactive.rb', 'lib/hyperactive/*.rb', 'tests/*', 'GPL-2'].to_a + s.require_path = "lib" + s.autorequire = "hyperactive" + s.test_files = Dir.glob('tests/*_test.rb') + Dir.glob('tests/test_helper.rb') + s.has_rdoc = true + s.rdoc_options << '--line-numbers' + s.rdoc_options << '--inline-source' + s.extra_rdoc_files = ["README"] + s.add_dependency('archipelago', '>= 0.2.0') +end + + +SOURCE_FILES = FileList.new do |fl| + [ "lib", "tests" ].each do |dir| + fl.include "#{dir}/**/*" + end + fl.include "Rakefile" +end + +Rake::GemPackageTask.new(spec) do |pkg| + pkg.need_tar = true +end + +task :default => [:units] do +end + +desc "Run all tests" +Rake::TestTask.new(:units) do |t| + t.pattern = 'tests/*_test.rb' + t.verbose = true + t.warning = true +end + +desc "Run all benchmarks" +Rake::TestTask.new(:bench) do |t| + t.pattern = 'tests/*_benchmark.rb' + t.verbose = true + t.warning = true +end + +desc "Package a gem from the source" +task :gem => "pkg/#{spec.name}-#{spec.version}.gem" do + puts "generated latest version" +end + Copied: tags/release_0_2_3/hyperactive/lib/hyperactive/hash.rb (from rev 83, trunk/hyperactive/lib/hyperactive/hash.rb) =================================================================== --- tags/release_0_2_3/hyperactive/lib/hyperactive/hash.rb (rev 0) +++ tags/release_0_2_3/hyperactive/lib/hyperactive/hash.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,172 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + + +require 'rubygems' +require 'archipelago' +require 'digest/sha1' + +module Hyperactive + + # + # The package containing the Hash class that provides any + # kind of index for your Hyperactive classes. + # + # Is supposed to be constantly scaling, but preliminary benchmarks show some + # problems with that assumption? + # + module Hash + + # + # A wrapper class that knows what key, value and list_element belong together. + # + class Element < Hyperactive::Record::Bass + attr_accessor :key, :value, :list_element + # + # Initialize a new Hash::Element with given +key+, +value+ and +list_element+. + # + def initialize(key, value, list_element) + self.key = key + self.value = value + self.list_element = list_element + end + end + + # + # A class suitable for storing large and often-changing datasets in + # an Archipelago environment. + # + class Head < Hyperactive::Record::Bass + + attr_accessor :list + + include Archipelago::Current::ThreadedCollection + + # + # Initialize a Hash::Head. + # + # NB: Remember to call create on the new instance or use Head.get_instance to get that done automatically. + # + def initialize + self.list = nil + end + + # + # Return the size of this Hash. + # + def size + self.list.size + end + + # + # Return the value for +key+. + # + def [](key) + element = Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction] + if element + return element.value + else + return nil + end + end + + # + # Returns whether +key+ is included in this Hash. + # + def include?(key) + Hyperactive::Record::CAPTAIN.include?(my_key_for(key), @transaction) + end + + # + # Insert +value+ under +key+ in this Hash. + # + def []=(key, value) + self.list = Hyperactive::List::Head.get_instance_with_transaction(@transaction) unless self.list + + if (element = Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction]) + element.value = value + else + element = Element.get_instance_with_transaction(@transaction, key, value, nil) + self.list << element + element.list_element = self.list.last_element + Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction] = element + end + end + + # + # Delete +key+ from this Hash. + # + def delete(key) + self.list = Hyperactive::List::Head.get_instance_with_transaction(@transaction) unless self.list + + return_value = nil + + element = Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction] + if element + Hyperactive::Record::CAPTAIN.delete(my_key_for(key), @transaction) + self.list.unlink!(element.list_element) + return_value = element.value + element.destroy! + end + return return_value + end + + # + # Will yield to +block+ once for each key/value pair in this Hash. + # + def each(&block) + self.list = Hyperactive::List::Head.get_instance_with_transaction(@transaction) unless self.list + + self.list.each do |element| + yield([element.key, element.value]) + end + end + + # + # Remove all key/value pairs from this Hash. + # + def clear! + self.list = Hyperactive::List::Head.get_instance_with_transaction(@transaction) unless self.list + + self.list.t_each do |element| + element.destroy! + end + self.list.clear! + end + + # + # Clear everything from this Tree and destroy it. + # + def destroy! + self.clear! + super + end + + private + + # + # Get my private key for a given +key+. + # + def my_key_for(key) + Digest::SHA1.hexdigest("#{Marshal.dump(key)}#{self.record_id}") + end + + end + + end + +end Deleted: tags/release_0_2_3/hyperactive/lib/hyperactive/list.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/list.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/hyperactive/lib/hyperactive/list.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,154 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - - -require 'rubygems' -require 'archipelago' - -module Hyperactive - - # - # A package that simplifies putting dumb linked lists in the distributed database. - # - module List - - # - # A List element. - # - class Element < Hyperactive::Record::Bass - attr_accessor :previous, :next, :value - end - - # - # A List head. - # - class Head < Hyperactive::Record::Bass - attr_reader :first_element, :last_element, :size - - # - # Create a Head. - # - # NB: As usual, dont call this. Use Head.get_instance instead. - # - def initialize - @size = 0 - @first_element = @last_element = nil - end - - # - # Return the first value of the list. - # - def first - @first_element.value - end - - # - # Return the last value of the list. - # - def last - @last_element.value - end - - # - # Push +v+ onto the end of this list. - # - def <<(v) - if @first_element - new_element = Element.get_instance - new_element.value = v - new_element.previous = @last_element - @last_element.next = new_element - @last_element = new_element - else - start(v) - end - @size += 1 - return v - end - - # - # Push +v+ onto the beginning of this list. - # - def unshift(v) - if @first_element - new_element = Element.get_instance - new_element.value = v - new_element.next = @first_element - @first_element.previous = new_element - @first_element = new_element - else - start(v) - end - @size += 1 - return v - end - - # - # Remove the last value from this list and return it. - # - def pop - v = nil - if size > 1 - element = @last_element - @last_element = element.previous - @last_element.next = nil - v = element.value - element.destroy - else - v = @first_element.value - @first_element.destroy - @first_element = @last_element = nil - end - @size -= 1 - return v - end - - # - # Remove the first value from this list and return it. - # - def shift - v = nil - if size > 1 - element = @first_element - @first_element = element.next - @first_element.previous = nil - v = element.value - element.destroy - else - v = @first_element.value - @first_element.destroy - @first_element = @last_element = nil - end - @size -= 1 - return v - end - - private - - # - # Start this list with +v+ when it has no other values. - # - def start(v) - @first_element = @last_element = Element.get_instance - @first_element.value = v - end - - end - - end - -end Copied: tags/release_0_2_3/hyperactive/lib/hyperactive/list.rb (from rev 83, trunk/hyperactive/lib/hyperactive/list.rb) =================================================================== --- tags/release_0_2_3/hyperactive/lib/hyperactive/list.rb (rev 0) +++ tags/release_0_2_3/hyperactive/lib/hyperactive/list.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,239 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + + +require 'rubygems' +require 'archipelago' + +module Hyperactive + + # + # A package that simplifies putting dumb linked lists in the distributed database. + # + module List + + # + # A wrapper class that knows its previous and next List::Elements as well as its value and the id of its list. + # + class Element < Hyperactive::Record::Bass + attr_accessor :previous, :next, :value, :list_id + + include Archipelago::Current::ThreadedCollection + + # + # Initialize this List::Element with given +previous_element+, +next_element+, +value+ and +list_id+. + # + def initialize(previous_element, next_element, value, list_id) + self.previous = previous_element + self.next = next_element + self.value = value + self.list_id = list_id + end + + # + # Yield to +block+ once for this and each following element. + # + def each(&block) + element = self + while element + yield(element) + element = self.next + end + end + end + + # + # A List head. + # + class Head < Hyperactive::Record::Bass + attr_accessor :first_element, :last_element, :size + + include Archipelago::Current::ThreadedCollection + + # + # Create a List::Head. This is in essence the linked list. + # + # NB: Remember to call create on the new instance or use Head.get_instance to get that done automatically. + # + def initialize + self.size = 0 + self.first_element = self.last_element = nil + end + + # + # Unlinks the given +element+ and reconnects the List around it. + # + def unlink!(element) + raise "#{element} is not a part of #{self}" unless element.list_id == @record_id + + if size > 1 + if element == self.first_element + self.first_element = element.next + self.first_element.previous = nil + elsif element == self.last_element + self.last_element = element.previous + self.last_element.next = nil + else + element.previous.next = element.next + element.next.previous = element.previous + end + else + if element == self.first_element + self.first_element = self.last_element = nil + else + raise "#{element} is not a part of #{self} even though it claims to be" + end + end + self.size -= 1 + element.destroy! + end + + # + # Remove all elements from this List::Head. + # + def clear! + self.first_element.t_each do |element| + element.destroy! + end + self.first_element = self.last_element = nil + self.size = 0 + end + + # + # Destroys this list and all its elements. + # + def destroy! + self.clear! + super + end + + # + # Return the first value of the list. + # + def first + if self.first_element + self.first_element.value + else + nil + end + end + + # + # Return the last value of the list. + # + def last + if self.last_element + self.last_element.value + else + end + end + + # + # Push +v+ onto the end of this list. + # + def <<(v) + if self.first_element + new_element = Element.get_instance_with_transaction(@transaction, self.last_element, nil, v, @record_id) + self.last_element.next = new_element + self.last_element = new_element + else + start(v) + end + self.size += 1 + return v + end + + # + # Push +v+ onto the beginning of this list. + # + def unshift(v) + if self.first_element + new_element = Element.get_instance_with_transaction(@transaction, nil, self.first_element, v, @record_id) + self.first_element.previous = new_element + self.first_element = new_element + else + start(v) + end + self.size += 1 + return v + end + + # + # Remove the last value from this list and return it. + # + def pop + v = nil + if size > 1 + element = self.last_element + self.last_element = element.previous + self.last_element.next = nil + v = element.value + element.destroy! + else + v = self.first_element.value + self.first_element.destroy! + self.first_element = self.last_element = nil + end + self.size -= 1 + return v + end + + # + # Yield to +block+ once for each value in this list. + # + def each(&block) + element = self.first_element + while element + yield(element.value) + element = element.next + end + end + + # + # Remove the first value from this list and return it. + # + def shift + v = nil + if size > 1 + element = self.first_element + self.first_element = element.next + self.first_element.previous = nil + v = element.value + element.destroy! + else + v = self.first_element.value + self.first_element.destroy! + self.first_element = self.last_element = nil + end + self.size -= 1 + return v + end + + private + + # + # Start this list with +v+ when it has no other values. + # + def start(v) + self.first_element = self.last_element = Element.get_instance_with_transaction(@transaction, nil, nil, v, @record_id) + end + + end + + end + +end Deleted: tags/release_0_2_3/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/hyperactive/lib/hyperactive/record.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,400 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - - -require 'rubygems' -require 'archipelago' - -# -# A utility module to provide the functionality required for example -# if you want to use archipelago in a Ruby on Rails project *hint hint*. -# -module Hyperactive - - # - # The package containing the base class Bass that simplifies - # using archipelago for generic database stuff. - # - module Record - - # - # The default database connector. - # - CAPTAIN = Archipelago::Pirate::Captain.new - - # - # A tiny callable class that saves stuff in - # indexes depending on certain attributes. - # - class IndexBuilder - # - # Get the first part of the key, that depends on the +attributes+. - # - def self.get_attribute_key_part(attributes) - "Hyperactive::IndexBuilder::#{attributes.join(",")}" - end - # - # Get the last part of the key, that depends on the +values+. - # - def self.get_value_key_part(values) - "#{values.join(",")}" - end - # - # Initialize an IndexBuilder giving it an array of +attributes+ - # that will be indexed. - # - def initialize(attributes) - @attributes = attributes - end - # - # Get the Tree for the given +record+. - # - def get_key_for(record) - values = @attributes.collect do |att| - if record.respond_to?(att) - record.send(att) - else - nil - end - end - key = "#{self.class.get_attribute_key_part(@attributes)}::#{self.class.get_value_key_part(values)}" - end - # - # Call this IndexBuilder and pass it a +block+. - # - # If the +argument+ is an Array then we know we are a save hook, - # otherwise we are a destroy hook. - # - def call(argument, &block) - yield - - # - # If the argument is an Array (of old value, new value) - # then we are a save hook, otherwise a destroy hook. - # - if Array === argument - record = argument.last - old_record = argument.first - old_key = get_key_for(old_record) - new_key = get_key_for(record) - if old_key != new_key - (CAPTAIN[old_key] ||= Hyperactive::Tree::Root.get_instance).delete(record.record_id) - (CAPTAIN[new_key] ||= Hyperactive::Tree::Root.get_instance)[record.record_id] = record - end - else - record = argument - (CAPTAIN[get_key_for(record)] ||= Hyperactive::Tree::Root.get_instance).delete(record.record_id) - end - end - end - - # - # A tiny callable class that saves stuff inside containers - # if they match certain criteria. - # - class MatchSaver - # - # Initialize this MatchSaver with a +key+, a callable +matcher+ - # and a +mode+ (:select, :reject, :delete_if_match or :delete_unless_match). - # - def initialize(key, matcher, mode) - @key = key - @matcher = matcher - @mode = mode - end - # - # Depending on @mode and return value of @matcher.call - # may save record in the Hash-like container named @key in - # the main database after having yielded to +block+. - # - def call(argument, &block) - yield - - record = argument - record = argument.last if Array === argument - - case @mode - when :select - if @matcher.call(record) - CAPTAIN[@key][record.record_id] = record - else - CAPTAIN[@key].delete(record.record_id) - end - when :reject - if @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) - else - CAPTAIN[@key][record.record_id] = record - end - when :delete_if_match - if @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) - end - when :delete_unless_match - unless @matcher.call(record) - CAPTAIN[@key].delete(record.record_id) - end - end - end - end - - # - # A convenient base class to inherit when you want the basic utility methods - # provided by for example ActiveRecord::Base *hint hint*. - # - # NB: After an instance is created, it will actually return a copy within your local machine - # which is not what is usually the case. Every other time you fetch it using a select or other - # method you will instead receive a proxy object to the database. This means that nothing you - # do to it at that point will be persistent or even necessarily have a defined result. - # Therefore: do not use MyRecordSubclass.new to MyRecordSubclass#initialize objects, - # instead use MyRecordSubclass.get_instance, since it will return a fresh proxy object - # instead of the devious original: - # my_instance = MyRecordSubclass.get_instance(*the_same_arguments_as_to_initialize) - # - class Bass - - @@create_hooks_by_class = {} - @@destroy_hooks_by_class = {} - @@save_hooks_by_class = {} - - # - # The host we are running on. - # - HOST = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" - - # - # Call this if you want to change the default database connector - # to something else. - # - def self.setup(options = {}) - CAPTAIN.setup(options[:pirate_options]) - end - - # - # Return our create_hooks, which can then be treated as any old Array. - # - # These must be callable objects with an arity of 1 - # that will be sent the instance about to be created (initial - # insertion into the database system) that take a block argument. - # - # The block argument will be a Proc that actually injects the - # instance into the database system. - # - # Use this to preprocess, validate and/or postprocess your - # instances upon creation. - # - def self.create_hooks - self.get_hook_array_by_class(@@create_hooks_by_class) - end - - # - # Return our destroy_hooks, which can then be treated as any old Array. - # - # These must be callable objects with an arity of 1 - # that will be sent the instance about to be destroyed (removal - # from the database system) that take a block argument. - # - # The block argument will be a Proc that actually removes the - # instance from the database system. - # - # Use this to preprocess, validate and/or postprocess your - # instances upon destruction. - # - def self.destroy_hooks - self.get_hook_array_by_class(@@destroy_hooks_by_class) - end - - # - # Return our save_hooks, which can then be treated as any old Array. - # - # These must be callable objects with an arity of 1 - # that will be sent [the old version, the new version] of the - # instance about to be saved (storage into the database system) - # along with a block argument. - # - # The block argument will be a Proc that actually saves the - # instance into the database system. - # - # Use this to preprocess, validate and/or postprocess your - # instances upon saving. - # - def self.save_hooks - self.get_hook_array_by_class(@@save_hooks_by_class) - end - - # - # Create an index for this class. - # - # Will create a method find_by_#{attributes.join("_and_")} for this - # class that will return what you expect. - # - def self.index_by(*attributes) - attribute_key_part = IndexBuilder.get_attribute_key_part(attributes) - self.class_eval <self as - # key. If self doesnt exist in the +hash+ - # it will recurse by calling the same method in the - # superclass until it has been called in Hyperactive::Record::Base. - # - def self.get_hook_array_by_class(hash) - return hash[self] if hash.include?(self) - - if self == Bass - hash[self] = [] - return self.get_hook_array_by_class(hash) - else - hash[self] = self.superclass.get_hook_array_by_class(hash).clone - return self.get_hook_array_by_class(hash) - end - end - - end - end -end - Copied: tags/release_0_2_3/hyperactive/lib/hyperactive/record.rb (from rev 83, trunk/hyperactive/lib/hyperactive/record.rb) =================================================================== --- tags/release_0_2_3/hyperactive/lib/hyperactive/record.rb (rev 0) +++ tags/release_0_2_3/hyperactive/lib/hyperactive/record.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,518 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + + +require 'rubygems' +require 'archipelago' + +# +# A utility module to provide the functionality required for example +# if you want to use archipelago in a Ruby on Rails project *hint hint*. +# +module Hyperactive + + # + # The package containing the base class Bass that simplifies + # using archipelago for generic database stuff. + # + module Record + + # + # The default database connector. + # + CAPTAIN = Archipelago::Pirate::Captain.new + + # + # A tiny callable class that saves stuff in + # indexes depending on certain attributes. + # + class IndexBuilder + # + # Get the first part of the key, that depends on the +attributes+. + # + def self.get_attribute_key_part(attributes) + "Hyperactive::IndexBuilder::#{attributes.join(",")}" + end + # + # Get the last part of the key, that depends on the +values+. + # + def self.get_value_key_part(values) + "#{values.join(",")}" + end + # + # Initialize an IndexBuilder giving it an array of +attributes+ + # that will be indexed. + # + def initialize(attributes) + @attributes = attributes + end + # + # Get the key for the given +record+. + # + def get_key_for(record) + values = @attributes.collect do |att| + if record.respond_to?(att) + record.send(att) + else + nil + end + end + key = "#{self.class.get_attribute_key_part(@attributes)}::#{self.class.get_value_key_part(values)}" + end + # + # Call this IndexBuilder and pass it a +block+. + # + # If the +argument+ is an Array then we know we are a save hook, + # otherwise we are a destroy hook. + # + def call(argument, &block) + yield + + # + # If the argument is an Array (of old value, new value) + # then we are a save hook, otherwise a destroy hook. + # + if Array === argument + record = argument.last + old_record = argument.first + old_key = get_key_for(old_record) + new_key = get_key_for(record) + if old_key != new_key + (CAPTAIN[old_key] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction)).delete(record.record_id) + (CAPTAIN[new_key] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction))[record.record_id] = record + end + else + record = argument + (CAPTAIN[get_key_for(record)] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction)).delete(record.record_id) + end + end + end + + # + # A tiny callable class that saves stuff inside containers + # if they match certain criteria. + # + class MatchSaver + # + # Initialize this MatchSaver with a +key+, a callable +matcher+ + # and a +mode+ (:select, :reject, :delete_if_match or :delete_unless_match). + # + def initialize(key, matcher, mode) + @key = key + @matcher = matcher + @mode = mode + end + # + # Depending on @mode and return value of @matcher.call + # may save record in the Hash-like container named @key in + # the main database after having yielded to +block+. + # + def call(argument, &block) + yield + + record = argument + record = argument.last if Array === argument + + case @mode + when :select + if @matcher.call(record) + CAPTAIN[@key, record.transaction][record.record_id] = record + else + CAPTAIN[@key, record.transaction].delete(record.record_id) + end + when :reject + if @matcher.call(record) + CAPTAIN[@key, record.transaction].delete(record.record_id) + else + CAPTAIN[@key, record.transaction][record.record_id] = record + end + when :delete_if_match + if @matcher.call(record) + CAPTAIN[@key, record.transaction].delete(record.record_id) + end + when :delete_unless_match + unless @matcher.call(record) + CAPTAIN[@key, record.transaction].delete(record.record_id) + end + end + end + end + + # + # A convenient base class to inherit when you want the basic utility methods + # provided by for example ActiveRecord::Base *hint hint*. + # + # NB: When an instance is created you will actually have a copy within your local machine + # which is not what you usually want. Every other time you fetch it using a select or other + # method you will instead receive a proxy object to the database. This means that nothing you + # do to it at that point will be persistent or even necessarily have a defined result. + # Therefore: do not use the instantiated object, instead call my_instance.save + # to get a proxy to the object stored into the database. + # + class Bass + + @@create_hooks_by_class = {} + @@destroy_hooks_by_class = {} + @@save_hooks_by_class = {} + @@load_hooks_by_class = {} + + # + # The host we are running on. + # + HOST = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" + + # + # Call this if you want to change the default database connector + # to something else. + # + def self.setup(options = {}) + CAPTAIN.setup(options[:pirate_options]) + end + + # + # Works like normal attr_reader but with transactional awareness. + # + def self.attr_reader(*attributes) + attributes.each do |attribute| + define_method(attribute) do + value = instance_variable_get("@#{attribute}") + if Archipelago::Treasure::Dubloon === value + if @transaction ||= nil + return value.join(@transaction) + else + return value + end + else + return value + end + end + end + end + + # + # Works like normal attr_writer but with transactional awareness. + # + def self.attr_writer(*attributes) + attributes.each do |attribute| + define_method("#{attribute}=") do |new_value| + if Archipelago::Treasure::Dubloon === new_value + new_value.assert_transaction(@transaction) if @transaction ||= nil + instance_variable_set("@#{attribute}", new_value) + else + instance_variable_set("@#{attribute}", new_value) + end + end + end + end + + # + # Works like normal attr_accessor but with transactional awareness. + # + def self.attr_accessor(*attributes) + attr_reader(*attributes) + attr_writer(*attributes) + end + + # + # Return our create_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent the instance about to be created (initial + # insertion into the database system) that take a block argument. + # + # The block argument will be a Proc that actually injects the + # instance into the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon creation. + # + def self.create_hooks + self.get_hook_array_by_class(@@create_hooks_by_class) + end + + # + # Return our destroy_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent the instance about to be destroyed (removal + # from the database system) that take a block argument. + # + # The block argument will be a Proc that actually removes the + # instance from the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon destruction. + # + def self.destroy_hooks + self.get_hook_array_by_class(@@destroy_hooks_by_class) + end + + # + # Return our load_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent the instance about to be loaded (insertion + # into the live hash of the database in question) that take a block argument. + # + # The block argument will be a Proc that actually puts the + # instance into the live hash. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon loading. + # + def self.load_hooks + self.get_hook_array_by_class(@@load_hooks_by_class) + end + + # + # Return our save_hooks, which can then be treated as any old Array. + # + # These must be callable objects with an arity of 1 + # that will be sent [the old version, the new version] of the + # instance about to be saved (storage into the database system) + # along with a block argument. + # + # The block argument will be a Proc that actually saves the + # instance into the database system. + # + # Use this to preprocess, validate and/or postprocess your + # instances upon saving. + # + def self.save_hooks + self.get_hook_array_by_class(@@save_hooks_by_class) + end + + # + # Create an index for this class. + # + # Will create a method find_by_#{attributes.join("_and_")} for this + # class that will return what you expect. + # + def self.index_by(*attributes) + attribute_key_part = IndexBuilder.get_attribute_key_part(attributes) + self.class_eval <(o) + if Record === o + @record_id <=> o.record_id + else + 0 + end + end + + # + # Save this Record instance into the distributed database and return a proxy to the saved object. + # + # This will also wrap the actual insertion within the create_hooks you have defined for this class. + # + def create + @record_id ||= Digest::SHA1.hexdigest("#{HOST}:#{Time.new.to_f}:#{self.object_id}:#{rand(1 << 32)}").to_s + @transaction ||= nil + + Hyperactive::Hooker.call_with_hooks(self, *self.class.create_hooks) do + CAPTAIN[self.record_id, @transaction] = self + end + + proxy = CAPTAIN[@record_id, @transaction] + + return proxy + end + + # + # Will execute +block+ within a transaction. + # + # What it does is just set the @transaction instance variable + # before calling the block, and unsetting it after. + # + # This means that any classes that want to be transaction sensitive + # need to take heed regarding the @transaction instance variable. + # + # For example, when creating new Record instances you may want to use + # get_instance_with_transaction(@transaction, *args) to ensure that the + # new instance exists within the same transaction as yourself. + # + # See Hyperactive::List::Head and Hyperactive::Hash::Head for examples of this behaviour. + # + def with_transaction(transaction, &block) + @transaction = transaction + begin + yield + ensure + @transaction = nil + end + end + + # + # This will allow us to wrap any write of us to persistent storage + # in the @@save_hooks as long as the Archipelago::Hashish provider + # supports it. See Archipelago::Hashish::BerkeleyHashish for an example + # of Hashish providers that do this. + # + def save_hook(old_value, &block) + Hyperactive::Hooker.call_with_hooks([old_value, self], *self.class.save_hooks) do + yield + end + end + + # + # This will allow us to wrap any load of us from persistent storage + # in the @@load_hooks as long as the Archipelago::Hashish provider + # supports it. See Archipelago::Hashish::BerkeleyHashish for an example + # of Hashish providers that do this. + # + def load_hook(&block) + Hyperactive::Hooker.call_with_hooks(self, *self.class.load_hooks) do + yield + end + end + + # + # Remove this instance from the database calling all the right hooks. + # + # Freezes this instance after having deleted it. + # + # Returns false without destroying anything if any of the @@pre_destroy_hooks + # returns false. + # + # Returns true otherwise. + # + def destroy! + Hyperactive::Hooker.call_with_hooks(self, *self.class.destroy_hooks) do + CAPTAIN.delete(@record_id, @transaction) + self.freeze + end + end + + private + + # + # The key used to store the collection with the given +sym+ as name. + # + def self.collection_key(sym) + "Hyperactive::Record::Bass::collection_key::#{sym}" + end + + # + # Get an Array from +hash+ using self as + # key. If self doesnt exist in the +hash+ + # it will recurse by calling the same method in the + # superclass until it has been called in Hyperactive::Record::Base. + # + def self.get_hook_array_by_class(hash) + return hash[self] if hash.include?(self) + + if self == Bass + hash[self] = [] + return self.get_hook_array_by_class(hash) + else + hash[self] = self.superclass.get_hook_array_by_class(hash).clone + return self.get_hook_array_by_class(hash) + end + end + + end + end +end + Deleted: tags/release_0_2_3/hyperactive/lib/hyperactive/tree.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/tree.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/hyperactive/lib/hyperactive/tree.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,241 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - - -require 'rubygems' -require 'archipelago' -require 'rbtree' - -module Hyperactive - - # - # The package containing the Hash-like Tree class that provides any - # kind of index for your Hyperactive classes. - # - module Tree - - # - # A class suitable for storing large and often-changing datasets in - # an Archipelago environment. - # - # Is constructed like a set of nested Hashes that automatically create - # new children on demand, and will thusly only have to check the path from - # the root node to the leaf for changes when method calls return (see Archipelago::Treasure::Chest) - # and will only have to actually store into database the leaf itself if it - # has changed. - # - class Root < Hyperactive::Record::Bass - - attr_accessor :elements, :subtrees - - WIDTH = 1 << 3 - - # - # Dont call this! Call Root.get_instance(options) instead! - # - def initialize(options = {}) - @width = options[:width] || WIDTH - @elements = {} - @subtrees = nil - end - - # - # Deletes +key+ in this Root. - # - def delete(key) - if @elements - @elements.delete(key) - else - subtree_for(key).delete(key) - end - end - - # - # Returns the size of this Root. - # - def size - if @elements - @elements.size - else - @subtrees.t_collect do |tree_id, tree| - tree.size - end.inject(0) do |sum, size| - sum + size - end - end - end - - # - # Returns all keys and values returning true for +callable+.call(key, value) in this Root. - # - def select(callable) - if @elements - @elements.select do |k,v| - callable.call(k,v) - end - else - @subtrees.t_collect do |tree_id, tree| - tree.select(callable) - end.inject([]) do |sum, match| - sum + match - end - end - end - - # - # Returns all keys and values returning false for +callable+.call(key, value) in this Root. - # - def reject(callable) - if @elements - @elements.reject do |k,v| - callable.call(k,v) - end - else - @subtrees.t_collect do |tree_id, tree| - tree.reject(callable) - end.inject([]) do |sum, match| - sum + match - end - end - end - - # - # Puts +value+ under +key+ in this Root. - # - def []=(key, value) - if @elements - if @elements.size < @width - @elements[key] = value - else - split! - subtree_for(key)[key] = value - end - else - subtree_for(key)[key] = value - end - return value - end - - # - # Returns the value for +key+ in this Root. - # - def [](key) - if @elements - return @elements[key] - else - return subtree_for(key)[key] - end - end - - # - # Returns an Array containing +callable+.call(key, value) from all values in this Root. - # - def collect(callable) - rval = [] - self.each(Proc.new do |k,v| - rval << callable.call(k,v) - end) - return rval - end - - # - # Does +callable+.call(key, value) on all values in this Root. - # - def each(callable) - if @elements - @elements.each do |key, value| - callable.call(key, value) - end - else - @subtrees.t_each do |tree_id, tree| - tree.each(callable) - end - nil - end - end - - # - # Clear everything from this Tree and destroy it. - # - def destroy - if @elements - @elements.each do |key, value| - value.destroy if value.respond_to?(:destroy) - end - else - @subtrees.each do |tree_id, tree| - tree.destroy - end - end - freeze - super - end - - # - # Clear everything from this Root. - # - def clear - unless @elements - @subtrees.each do |tree_id, tree| - tree.clear - end - @subtrees = nil - end - @elements = {} - end - - private - - # - # Finds the subtree responsible for +key+. - # - # Does it in this ugly way cause the nice way of just doing modulo gave - # really odd results since all hashes seem to give some modulo values - # a lot more often than expected. - # - def subtree_for(key) - key_id = Digest::SHA1.new("#{key.hash}#{self.record_id}").to_s - @subtrees.each do |tree_id, tree| - return tree if tree_id > key_id - end - return @subtrees.values.first - end - - # - # Split this Tree by creating @subtrees, - # then putting all @elements in them and then emptying @elements. - # - def split! - raise "Cant split twice!" unless @elements - - @subtrees = RBTree.new - @subtrees.extend(Archipelago::Current::ThreadedCollection) - step = (1 << 160) / @width - 0.upto(@width - 1) do |n| - @subtrees["%x" % (n * step)] = Root.get_instance(:width => @width) - end - @elements.each do |key, value| - subtree_for(key)[key] = value - end - @elements = nil - end - - end - - end - -end Deleted: tags/release_0_2_3/hyperactive/lib/hyperactive.rb =================================================================== --- trunk/hyperactive/lib/hyperactive.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/hyperactive/lib/hyperactive.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,23 +0,0 @@ -# Archipelago - a distributed computing toolkit for ruby -# Copyright (C) 2006 Martin Kihlgren -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - -$: << File.dirname(__FILE__) - -require 'hyperactive/hooker' -require 'hyperactive/record' -require 'hyperactive/tree' -require 'hyperactive/list' Copied: tags/release_0_2_3/hyperactive/lib/hyperactive.rb (from rev 83, trunk/hyperactive/lib/hyperactive.rb) =================================================================== --- tags/release_0_2_3/hyperactive/lib/hyperactive.rb (rev 0) +++ tags/release_0_2_3/hyperactive/lib/hyperactive.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,23 @@ +# Archipelago - a distributed computing toolkit for ruby +# Copyright (C) 2006 Martin Kihlgren +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +$: << File.dirname(__FILE__) + +require 'hyperactive/hooker' +require 'hyperactive/record' +require 'hyperactive/hash' +require 'hyperactive/list' Copied: tags/release_0_2_3/hyperactive/tests/hash_benchmark.rb (from rev 83, trunk/hyperactive/tests/hash_benchmark.rb) =================================================================== --- tags/release_0_2_3/hyperactive/tests/hash_benchmark.rb (rev 0) +++ tags/release_0_2_3/hyperactive/tests/hash_benchmark.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,66 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class HashBenchmark < Test::Unit::TestCase + + def setup + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + assert_within(20) do + Set.new(Hyperactive::Record::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) + end + assert_within(20) do + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + end + end + + def teardown + @c.stop! + @c.persistence_provider.unlink! + @c2.stop! + @c2.persistence_provider.unlink! + @tm.stop! + @tm.persistence_provider.unlink! + end + + def test_set_get + h = Hyperactive::Hash::Head.get_instance + r = Hyperactive::Record::Bass.get_instance + hash_test("Hyperactive::Hash", h, r, 100) + end + + def test_regular_hash_set_get + Hyperactive::Record::CAPTAIN["h"] = {} + h = Hyperactive::Record::CAPTAIN["h"] + r = Hyperactive::Record::Bass.get_instance + hash_test("Hash", h, r, 100) + end + + private + + def hash_test(label, h, r, c) + bm("#{label}#set/get/delete", :n => c * 1) do + f = rand(1 << 32) + h[f] = r + x = h[f] + h.delete(f) + end + bm("#{label}#set", :n => c * 10) do + f = rand(1 << 32) + h[f] = r + end + bm("#{label}#set/get/delete (big)", :n => c * 1) do + f = rand(1 << 32) + h[f] = r + x = h[f] + h.delete(f) + end + end + +end Copied: tags/release_0_2_3/hyperactive/tests/hash_test.rb (from rev 83, trunk/hyperactive/tests/hash_test.rb) =================================================================== --- tags/release_0_2_3/hyperactive/tests/hash_test.rb (rev 0) +++ tags/release_0_2_3/hyperactive/tests/hash_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,76 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class RecordMatcher + def initialize(i) + @i = i + end + def call(k,v) + k == @i + end +end + +class HashTest < Test::Unit::TestCase + + def setup + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + Hyperactive::Record::CAPTAIN.update_services! + assert_within(10) do + Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort + end + assert_within(10) do + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + end + end + + def teardown + @c.stop! + @c.persistence_provider.unlink! + @c2.stop! + @c2.persistence_provider.unlink! + @tm.stop! + @tm.persistence_provider.unlink! + end + + def test_select_reject + h = Hyperactive::Hash::Head.get_instance + r1 = Hyperactive::Record::Bass.get_instance + r2 = Hyperactive::Record::Bass.get_instance + h[r1.record_id] = r1 + h[r2.record_id] = r2 + assert_equal(r1.record_id, h.t_select(RecordMatcher.new(r1.record_id)).first.first) + assert_equal(r1.record_id, h.t_reject(RecordMatcher.new(r2.record_id)).first.first) + end + + def test_delete + h = Hyperactive::Hash::Head.get_instance + r = Hyperactive::Record::Bass.get_instance + h[r.record_id] = r + assert(h.include?(r.record_id)) + h.delete(r.record_id) + assert(!h.include?(r.record_id)) + end + + def test_set_get + h = Hyperactive::Hash::Head.get_instance + h2 = {} + 10.times do + r = Hyperactive::Record::Bass.get_instance + h[r.record_id] = r + h2[r.record_id] = r + end + + h2.each do |k,v| + assert_equal(v, h[k]) + assert_equal(v, Hyperactive::Record::CAPTAIN[h.record_id][k]) + end + end + +end Copied: tags/release_0_2_3/hyperactive/tests/list_benchmark.rb (from rev 83, trunk/hyperactive/tests/list_benchmark.rb) =================================================================== --- tags/release_0_2_3/hyperactive/tests/list_benchmark.rb (rev 0) +++ tags/release_0_2_3/hyperactive/tests/list_benchmark.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,48 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class ListBenchmark < Test::Unit::TestCase + + def setup + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + assert_within(20) do + Set.new(Hyperactive::Record::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) + end + assert_within(20) do + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + end + end + + def teardown + @c.stop! + @c.persistence_provider.unlink! + @c2.stop! + @c2.persistence_provider.unlink! + @tm.stop! + @tm.persistence_provider.unlink! + end + + def test_set_get + l = Hyperactive::List::Head.get_instance + r = Hyperactive::Record::Bass.get_instance + bm("List::Head#push/pop", :n => 1000) do + l << r + l.pop + end + bm("List::Head#push", :n => 1000) do + l << r + end + bm("List::Head#push/pop (big)", :n => 1000) do + l << r + l.pop + end + end + +end Deleted: tags/release_0_2_3/hyperactive/tests/list_test.rb =================================================================== --- trunk/hyperactive/tests/list_test.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/hyperactive/tests/list_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,76 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'test_helper') - -class ListTest < Test::Unit::TestCase - - def setup - @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) - @c.publish! - @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) - @c2.publish! - @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) - @tm.publish! - Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, - :tranny_description => {:class => 'TestManager'}) - Hyperactive::Record::CAPTAIN.update_services! - assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort - end - assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] - end - end - - def teardown - @c.stop! - @c.persistence_provider.unlink - @c2.stop! - @c2.persistence_provider.unlink - @tm.stop! - @tm.persistence_provider.unlink - Archipelago::Disco::MC.clear! - Hyperactive::Record::CAPTAIN.update_services! - assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.empty? - end - assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.empty? - end - end - - def test_push_unshift_pop_shift - l = Hyperactive::List::Head.get_instance - assert_equal(0, l.size) - l << (r = Hyperactive::Record::Bass.get_instance) - assert_equal(1, l.size) - assert_equal(l.first, l.last) - assert_equal(r, l.first) - l << (r2 = Hyperactive::Record::Bass.get_instance) - assert_equal(2, l.size) - assert_equal(r2, l.last) - assert_equal(r, l.first) - assert_equal(r2, l.first_element.next.value) - l.unshift(r3 = Hyperactive::Record::Bass.get_instance) - assert_equal(3, l.size) - assert_equal(r3, l.first) - assert_equal(r2, l.last) - assert_equal(r, l.first_element.next.value) - assert_equal(r2, l.first_element.next.next.value) - assert_equal(r3, l.last_element.previous.previous.value) - r4 = l.shift - assert_equal(r3, r4) - assert_equal(2, l.size) - assert_equal(r, l.first) - assert_equal(r2, l.last) - r5 = l.pop - assert_equal(1, l.size) - assert_equal(r, l.first) - assert_equal(r, l.last) - assert_equal(r2, r5) - r6 = l.pop - assert_equal(0, l.size) - assert_equal(nil, l.first_element) - assert_equal(nil, l.last_element) - end - -end Copied: tags/release_0_2_3/hyperactive/tests/list_test.rb (from rev 83, trunk/hyperactive/tests/list_test.rb) =================================================================== --- tags/release_0_2_3/hyperactive/tests/list_test.rb (rev 0) +++ tags/release_0_2_3/hyperactive/tests/list_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,134 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class Collector + attr_accessor :es + def call(e) + @es ||= [] + @es << e + end +end + +class ListTest < Test::Unit::TestCase + + def setup + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + Hyperactive::Record::CAPTAIN.update_services! + assert_within(10) do + Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort + end + assert_within(10) do + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + end + end + + def teardown + @c.stop! + @c.persistence_provider.unlink! + @c2.stop! + @c2.persistence_provider.unlink! + @tm.stop! + @tm.persistence_provider.unlink! + end + + def test_clear + l = Hyperactive::List::Head.get_instance + l << (r = Hyperactive::Record::Bass.get_instance) + assert_equal(1, l.size) + assert_equal(r, l.first) + assert_equal(r, l.last) + l.clear! + assert_equal(0, l.size) + assert_equal(nil, l.first) + assert_equal(nil, l.last) + end + + def test_destroy + l = Hyperactive::List::Head.get_instance + l << (r = Hyperactive::Record::Bass.get_instance) + r1 = l.first_element.record_id + r2 = l.record_id + l.destroy! + assert(!Hyperactive::Record::CAPTAIN.include?(r1)) + assert(!Hyperactive::Record::CAPTAIN.include?(r2)) + end + + def test_each + l = Hyperactive::List::Head.get_instance + l << (r = Hyperactive::Record::Bass.get_instance) + l << (r2 = Hyperactive::Record::Bass.get_instance) + l << (r3 = Hyperactive::Record::Bass.get_instance) + c = Collector.new + l.t_each(c) + assert_equal(3, c.es.size) + assert(c.es.include?(r)) + assert(c.es.include?(r2)) + assert(c.es.include?(r3)) + end + + def test_unlink + l = Hyperactive::List::Head.get_instance + l << (r = Hyperactive::Record::Bass.get_instance) + l << (r2 = Hyperactive::Record::Bass.get_instance) + e = l.last_element + l << (r3 = Hyperactive::Record::Bass.get_instance) + assert_equal([r, r2, r3].sort, l.t_collect do |ec| ec end.sort) + l.unlink!(e) + assert_equal(2, l.size) + assert_equal(r, l.first) + assert_equal(r3, l.last) + assert_equal(r3, l.first_element.next.value) + l.unlink!(l.last_element) + assert_equal(1, l.size) + assert_equal(r, l.first) + assert_equal(r, l.last) + l.unlink!(l.first_element) + assert_equal(0, l.size) + assert_equal(nil, l.first) + assert_equal(nil, l.last) + assert_equal([], l.t_collect do |e| e end) + end + + def test_push_unshift_pop_shift + l = Hyperactive::List::Head.get_instance + assert_equal(0, l.size) + l << (r = Hyperactive::Record::Bass.get_instance) + assert_equal(1, l.size) + assert_equal(l.first, l.last) + assert_equal(r, l.first) + l << (r2 = Hyperactive::Record::Bass.get_instance) + assert_equal(2, l.size) + assert_equal(r2, l.last) + assert_equal(r, l.first) + assert_equal(r2, l.first_element.next.value) + l.unshift(r3 = Hyperactive::Record::Bass.get_instance) + assert_equal(3, l.size) + assert_equal(r3, l.first) + assert_equal(r2, l.last) + assert_equal(r, l.first_element.next.value) + assert_equal(r2, l.first_element.next.next.value) + assert_equal(r3, l.last_element.previous.previous.value) + r4 = l.shift + assert_equal(r3, r4) + assert_equal(2, l.size) + assert_equal(r, l.first) + assert_equal(r2, l.last) + r5 = l.pop + assert_equal(1, l.size) + assert_equal(r, l.first) + assert_equal(r, l.last) + assert_equal(r2, r5) + r6 = l.pop + assert_equal(0, l.size) + assert_equal(nil, l.first_element) + assert_equal(nil, l.last_element) + end + +end Deleted: tags/release_0_2_3/hyperactive/tests/record_test.rb =================================================================== --- trunk/hyperactive/tests/record_test.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/hyperactive/tests/record_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,158 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'test_helper') - -class MyRecord < Hyperactive::Record::Bass - attr_accessor :bajs - - def self.save_hook(instance, &block) - $BEFORE_SAVE += 1 - yield - $AFTER_SAVE += 1 - end - save_hooks << self.method(:save_hook) - - def self.create_hook(instance, &block) - $BEFORE_CREATE += 1 - yield - $AFTER_CREATE += 1 - end - create_hooks << self.method(:create_hook) - - def self.destroy_hook(instance, &block) - $BEFORE_DESTROY += 1 - yield - $AFTER_DESTROY += 1 - end - destroy_hooks << self.method(:destroy_hook) -end - -class RecordTest < Test::Unit::TestCase - - def setup - @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) - @c.publish! - @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) - @c2.publish! - @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) - @tm.publish! - Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, - :tranny_description => {:class => 'TestManager'}) - Hyperactive::Record::CAPTAIN.update_services! - assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort - end - assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] - end - $BEFORE_SAVE = 0 - $AFTER_SAVE = 0 - $BEFORE_DESTROY = 0 - $AFTER_DESTROY = 0 - $BEFORE_CREATE = 0 - $AFTER_CREATE = 0 - end - - def teardown - @c.stop! - @c.persistence_provider.unlink - @c2.stop! - @c2.persistence_provider.unlink - @tm.stop! - @tm.persistence_provider.unlink - Archipelago::Disco::MC.clear! - Hyperactive::Record::CAPTAIN.update_services! - assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.empty? - end - assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.empty? - end - end - - def test_index_find - MyRecord.class_eval do - index_by :bajs - end - r1 = MyRecord.get_instance - r1.bajs = "brunt" - r2 = MyRecord.get_instance - r2.bajs = "brunt" - r3 = MyRecord.get_instance - r3.bajs = "gult" - r4 = MyRecord.get_instance - r4.bajs = "beige" - - assert_equal(Set.new([r2.record_id,r1.record_id]), Set.new(MyRecord.find_by_bajs("brunt").collect(Proc.new do |k,v| - v.record_id - end))) - assert_equal([r3.record_id], MyRecord.find_by_bajs("gult").collect(Proc.new do |k,v| - v.record_id - end)) - - r1.destroy - - assert_equal([r2.record_id], MyRecord.find_by_bajs("brunt").collect(Proc.new do |k,v| - v.record_id - end)) - - end - - def test_select_reject - MyRecord.class_eval do - select :brunt do |record| - record.bajs == "brunt" - end - reject :not_brunt do |record| - record.bajs == "brunt" - end - end - r1 = MyRecord.get_instance - r1.bajs = "brunt" - r2 = MyRecord.get_instance - r2.bajs = "brunt" - r3 = MyRecord.get_instance - r3.bajs = "gult" - r4 = MyRecord.get_instance - r4.bajs = "beige" - - assert_equal(Set.new([r2.record_id,r1.record_id]), Set.new(MyRecord.brunt.collect(Proc.new do |k,v| - v.record_id - end))) - assert_equal(Set.new([r3.record_id,r4.record_id]), Set.new(MyRecord.not_brunt.collect(Proc.new do |k,v| - v.record_id - end))) - - r1.destroy - r2.destroy - r3.destroy - r4.destroy - - assert_equal(Set.new, Set.new(MyRecord.brunt.collect(Proc.new do |k,v| - v.record_id - end))) - assert_equal(Set.new, Set.new(MyRecord.not_brunt.collect(Proc.new do |k,v| - v.record_id - end))) - end - - def test_create_update_destroy - r = MyRecord.get_instance - assert(Archipelago::Treasure::Dubloon === r) - assert_equal(1, $BEFORE_SAVE) - assert_equal(1, $AFTER_SAVE) - assert_equal(1, $BEFORE_CREATE) - assert_equal(1, $AFTER_CREATE) - r.bajs = "brunt" - assert_equal(2, $BEFORE_SAVE) - assert_equal(2, $AFTER_SAVE) - assert_equal("brunt", r.bajs) - assert_equal("brunt", Hyperactive::Record::CAPTAIN[r.record_id].bajs) - i = r.record_id - assert_equal(r, Hyperactive::Record::CAPTAIN[i]) - r.destroy - assert_equal(1, $BEFORE_DESTROY) - assert_equal(1, $AFTER_DESTROY) - assert_equal(nil, Hyperactive::Record::CAPTAIN[i]) - end - -end Copied: tags/release_0_2_3/hyperactive/tests/record_test.rb (from rev 83, trunk/hyperactive/tests/record_test.rb) =================================================================== --- tags/release_0_2_3/hyperactive/tests/record_test.rb (rev 0) +++ tags/release_0_2_3/hyperactive/tests/record_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -0,0 +1,152 @@ + +require File.join(File.dirname(__FILE__), 'test_helper') + +class MyRecord < Hyperactive::Record::Bass + attr_accessor :bajs + def initialize + @bajs = nil + end + def self.save_hook(instance, &block) + $BEFORE_SAVE += 1 + yield + $AFTER_SAVE += 1 + end + save_hooks << self.method(:save_hook) + + def self.create_hook(instance, &block) + $BEFORE_CREATE += 1 + yield + $AFTER_CREATE += 1 + end + create_hooks << self.method(:create_hook) + + def self.destroy_hook(instance, &block) + $BEFORE_DESTROY += 1 + yield + $AFTER_DESTROY += 1 + end + destroy_hooks << self.method(:destroy_hook) +end + +class RecordTest < Test::Unit::TestCase + + def setup + @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) + @c.publish! + @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) + @c2.publish! + @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) + @tm.publish! + Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + :tranny_description => {:class => 'TestManager'}) + Hyperactive::Record::CAPTAIN.update_services! + assert_within(10) do + Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort + end + assert_within(10) do + Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + end + $BEFORE_SAVE = 0 + $AFTER_SAVE = 0 + $BEFORE_DESTROY = 0 + $AFTER_DESTROY = 0 + $BEFORE_CREATE = 0 + $AFTER_CREATE = 0 + end + + def teardown + @c.stop! + @c.persistence_provider.unlink! + @c2.stop! + @c2.persistence_provider.unlink! + @tm.stop! + @tm.persistence_provider.unlink! + end + + def test_index_find + MyRecord.class_eval do + index_by :bajs + end + r1 = MyRecord.get_instance + r1.bajs = "brunt" + r2 = MyRecord.get_instance + r2.bajs = "brunt" + r3 = MyRecord.get_instance + r3.bajs = "gult" + r4 = MyRecord.get_instance + r4.bajs = "beige" + + assert_equal(Set.new([r2.record_id,r1.record_id]), Set.new(MyRecord.find_by_bajs("brunt").t_collect(Proc.new do |k,v| + v.record_id + end))) + assert_equal([r3.record_id], MyRecord.find_by_bajs("gult").t_collect(Proc.new do |k,v| + v.record_id + end)) + + r1.destroy! + + assert_equal([r2.record_id], MyRecord.find_by_bajs("brunt").t_collect(Proc.new do |k,v| + v.record_id + end)) + + end + + def test_select_reject + MyRecord.class_eval do + select :brunt do |record| + record.bajs == "brunt" + end + reject :not_brunt do |record| + record.bajs == "brunt" + end + end + r1 = MyRecord.get_instance + r1.bajs = "brunt" + r2 = MyRecord.get_instance + r2.bajs = "brunt" + r3 = MyRecord.get_instance + r3.bajs = "gult" + r4 = MyRecord.get_instance + r4.bajs = "beige" + + assert_equal(Set.new([r2.record_id,r1.record_id]), Set.new(MyRecord.brunt.t_collect(Proc.new do |k,v| + v.record_id + end))) + assert_equal(Set.new([r3.record_id,r4.record_id]), Set.new(MyRecord.not_brunt.t_collect(Proc.new do |k,v| + v.record_id + end))) + + r1.destroy! + r2.destroy! + r3.destroy! + r4.destroy! + + assert_equal(Set.new, Set.new(MyRecord.brunt.t_collect(Proc.new do |k,v| + v.record_id + end))) + assert_equal(Set.new, Set.new(MyRecord.not_brunt.t_collect(Proc.new do |k,v| + v.record_id + end))) + end + + def test_create_update_destroy + r = MyRecord.get_instance + assert(Archipelago::Treasure::Dubloon === r) + assert_equal(1, $BEFORE_SAVE) + assert_equal(1, $AFTER_SAVE) + assert_equal(1, $BEFORE_CREATE) + assert_equal(1, $AFTER_CREATE) + r.bajs = "brunt" + assert_equal(2, $BEFORE_SAVE) + assert_equal(2, $AFTER_SAVE) + assert_equal("brunt", r.bajs) + assert_equal("brunt", Hyperactive::Record::CAPTAIN[r.record_id].bajs) + i = r.record_id + assert_equal(r, Hyperactive::Record::CAPTAIN[i]) + r.destroy! + assert_equal(1, $BEFORE_DESTROY) + assert_equal(1, $AFTER_DESTROY) + assert_equal(nil, Hyperactive::Record::CAPTAIN[i]) + end + +end Deleted: tags/release_0_2_3/hyperactive/tests/tree_benchmark.rb =================================================================== --- trunk/hyperactive/tests/tree_benchmark.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/hyperactive/tests/tree_benchmark.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,66 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'test_helper') - -class TreeBenchmark < Test::Unit::TestCase - - def setup - @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) - @c.publish! - @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) - @c2.publish! - @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) - @tm.publish! - Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, - :tranny_description => {:class => 'TestManager'}) - assert_within(20) do - Set.new(Hyperactive::Record::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) - end - assert_within(20) do - Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] - end - end - - def teardown - @c.stop! - @c.persistence_provider.unlink - @c2.stop! - @c2.persistence_provider.unlink - @tm.stop! - @tm.persistence_provider.unlink - end - - def test_set_get - h = Hyperactive::Tree::Root.get_instance - r = Hyperactive::Record::Bass.get_instance - hash_test("Tree", h, r, 100) - end - - def test_regular_hash_set_get - Hyperactive::Record::CAPTAIN["h"] = {} - h = Hyperactive::Record::CAPTAIN["h"] - r = Hyperactive::Record::Bass.get_instance - hash_test("Hash", h, r, 100) - end - - private - - def hash_test(label, h, r, c) - bm("#{label}#set/get/delete", :n => c * 1) do - f = rand(1 << 32) - h[f] = r - x = h[f] - h.delete(f) - end - bm("#{label}#set", :n => c * 10) do - f = rand(1 << 32) - h[f] = r - end - bm("#{label}#set/get/delete (big)", :n => c * 1) do - f = rand(1 << 32) - h[f] = r - x = h[f] - h.delete(f) - end - end - -end Deleted: tags/release_0_2_3/hyperactive/tests/tree_test.rb =================================================================== --- trunk/hyperactive/tests/tree_test.rb 2006-11-28 13:40:30 UTC (rev 73) +++ tags/release_0_2_3/hyperactive/tests/tree_test.rb 2006-11-29 05:45:13 UTC (rev 85) @@ -1,75 +0,0 @@ - -require File.join(File.dirname(__FILE__), 'test_helper') - -class RecordMatcher - def initialize(i) - @i = i - end - def call(k,v) - k == @i - end -end - -class TreeTest < Test::Unit::TestCase - - def setup - @c = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest.db"))) - @c.publish! - @c2 = TestChest.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("chest2.db"))) - @c2.publish! - @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) - @tm.publish! - Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, - :tranny_description => {:class => 'TestManager'}) - Hyperactive::Record::CAPTAIN.update_services! - assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort - end - assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] - end - end - - def teardown - @c.stop! - @c.persistence_provider.unlink - @c2.stop! - @c2.persistence_provider.unlink - @tm.stop! - @tm.persistence_provider.unlink - Archipelago::Disco::MC.clear! - Hyperactive::Record::CAPTAIN.update_services! - assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.empty? - end - assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.empty? - end - end - - def test_select_reject - h = Hyperactive::Tree::Root.get_instance - r1 = Hyperactive::Record::Bass.get_instance - r2 = Hyperactive::Record::Bass.get_instance - h[r1.record_id] = r1 - h[r2.record_id] = r2 - assert_equal(r1.record_id, h.select(RecordMatcher.new(r1.record_id)).first.first) - assert_equal(r1.record_id, h.reject(RecordMatcher.new(r2.record_id)).keys.first) - end - - def test_set_get - h = Hyperactive::Tree::Root.get_instance - h2 = {} - 10.times do - r = Hyperactive::Record::Bass.get_instance - h[r.record_id] = r - h2[r.record_id] = r - end - - h2.each do |k,v| - assert_equal(v, h[k]) - assert_equal(v, Hyperactive::Record::CAPTAIN[h.record_id][k]) - end - end - -end From nobody at rubyforge.org Wed Nov 29 00:56:20 2006 From: nobody at rubyforge.org (nobody at rubyforge.org) Date: Wed, 29 Nov 2006 00:56:20 -0500 (EST) Subject: [Archipelago-submits] [86] trunk/hyperactive/lib/hyperactive/record.rb: more documentation Message-ID: <20061129055620.2A848524097B@rubyforge.org> Revision: 86 Author: zond Date: 2006-11-29 00:56:19 -0500 (Wed, 29 Nov 2006) Log Message: ----------- more documentation Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/record.rb Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-29 05:45:13 UTC (rev 85) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-29 05:56:19 UTC (rev 86) @@ -317,11 +317,11 @@ # # Will define a method called +name+ that will include all - # existing instances of this class that when sent to +matcher+.call - # return true. Will only return instances saved after this selector + # existing instances of this class that when yielded to the +block+ + # returns true. Will only return instances saved after this selector # is defined. # - def self.select(name, &block) + def self.select(name, &block) #:yields: instance key = self.collection_key(name) CAPTAIN[key] ||= Hyperactive::Hash::Head.get_instance self.class_eval < Revision: 87 Author: zond Date: 2006-11-30 17:06:57 -0500 (Thu, 30 Nov 2006) Log Message: ----------- moving out indexing and transaction support to separate modules Modified Paths: -------------- trunk/hyperactive/lib/hyperactive/hash.rb trunk/hyperactive/lib/hyperactive/record.rb trunk/hyperactive/lib/hyperactive.rb trunk/hyperactive/tests/hash_benchmark.rb trunk/hyperactive/tests/hash_test.rb trunk/hyperactive/tests/list_benchmark.rb trunk/hyperactive/tests/list_test.rb trunk/hyperactive/tests/record_test.rb Added Paths: ----------- trunk/hyperactive/lib/hyperactive/index.rb trunk/hyperactive/lib/hyperactive/transactions.rb Modified: trunk/hyperactive/lib/hyperactive/hash.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/hash.rb 2006-11-29 05:56:19 UTC (rev 86) +++ trunk/hyperactive/lib/hyperactive/hash.rb 2006-11-30 22:06:57 UTC (rev 87) @@ -76,7 +76,7 @@ # Return the value for +key+. # def [](key) - element = Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction] + element = Hyperactive::CAPTAIN[my_key_for(key), @transaction] if element return element.value else @@ -88,7 +88,7 @@ # Returns whether +key+ is included in this Hash. # def include?(key) - Hyperactive::Record::CAPTAIN.include?(my_key_for(key), @transaction) + Hyperactive::CAPTAIN.include?(my_key_for(key), @transaction) end # @@ -97,13 +97,13 @@ def []=(key, value) self.list = Hyperactive::List::Head.get_instance_with_transaction(@transaction) unless self.list - if (element = Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction]) + if (element = Hyperactive::CAPTAIN[my_key_for(key), @transaction]) element.value = value else element = Element.get_instance_with_transaction(@transaction, key, value, nil) self.list << element element.list_element = self.list.last_element - Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction] = element + Hyperactive::CAPTAIN[my_key_for(key), @transaction] = element end end @@ -115,9 +115,9 @@ return_value = nil - element = Hyperactive::Record::CAPTAIN[my_key_for(key), @transaction] + element = Hyperactive::CAPTAIN[my_key_for(key), @transaction] if element - Hyperactive::Record::CAPTAIN.delete(my_key_for(key), @transaction) + Hyperactive::CAPTAIN.delete(my_key_for(key), @transaction) self.list.unlink!(element.list_element) return_value = element.value element.destroy! Added: trunk/hyperactive/lib/hyperactive/index.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/index.rb (rev 0) +++ trunk/hyperactive/lib/hyperactive/index.rb 2006-11-30 22:06:57 UTC (rev 87) @@ -0,0 +1,194 @@ + +module Hyperactive + + module Index + + # + # A tiny callable class that saves stuff in + # indexes depending on certain attributes. + # + class IndexBuilder + # + # Get the key for the given attributes and values. + # + def self.get_key(klass, attributes, values) + "Hyperactive::IndexBuilder::#{klass}::#{attributes.join(",")}::#{values.join(",")}" + end + # + # Initialize an IndexBuilder giving it an array of +attributes+ + # that will be indexed. + # + def initialize(klass, attributes) + @klass = klass + @attributes = attributes + end + # + # Get the key for the given +record+. + # + def get_key_for(record) + values = @attributes.collect do |att| + if record.respond_to?(att) + record.send(att) + else + nil + end + end + key = self.class.get_key(@klass, @attributes, values) + end + # + # Call this IndexBuilder and pass it a +block+. + # + # If the +argument+ is an Array then we know we are a save hook, + # otherwise we are a destroy hook. + # + def call(argument, &block) + yield + + # + # If the argument is an Array (of old value, new value) + # then we are a save hook, otherwise a destroy hook. + # + if Array === argument + record = argument.last + old_record = argument.first + old_key = get_key_for(old_record) + new_key = get_key_for(record) + if old_key != new_key + (CAPTAIN[old_key] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction)).delete(record.record_id) + (CAPTAIN[new_key] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction))[record.record_id] = record + end + else + record = argument + (CAPTAIN[get_key_for(record)] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction)).delete(record.record_id) + end + end + end + + # + # A tiny callable class that saves stuff inside containers + # if they match certain criteria. + # + class MatchSaver + # + # Initialize this MatchSaver with a +key+, a callable +matcher+ + # and a +mode+ (:select, :reject, :delete_if_match or :delete_unless_match). + # + def initialize(key, matcher, mode) + @key = key + @matcher = matcher + @mode = mode + end + # + # Depending on @mode and return value of @matcher.call + # may save record in the Hash-like container named @key in + # the main database after having yielded to +block+. + # + def call(argument, &block) + yield + + record = argument + record = argument.last if Array === argument + + case @mode + when :select + if @matcher.call(record) + CAPTAIN[@key, record.transaction][record.record_id] = record + else + CAPTAIN[@key, record.transaction].delete(record.record_id) + end + when :reject + if @matcher.call(record) + CAPTAIN[@key, record.transaction].delete(record.record_id) + else + CAPTAIN[@key, record.transaction][record.record_id] = record + end + when :delete_if_match + if @matcher.call(record) + CAPTAIN[@key, record.transaction].delete(record.record_id) + end + when :delete_unless_match + unless @matcher.call(record) + CAPTAIN[@key, record.transaction].delete(record.record_id) + end + end + end + end + + module Indexable + + def self.append_features(base) + super + base.extend(ClassMethods) + end + + module ClassMethods + # + # Create an index for this class. + # + # Will create a method find_by_#{attributes.join("_and_")} for this + # class that will return what you expect. + # + def index_by(*attributes) + klass = self + self.class.class_eval do + define_method("find_by_#{attributes.join("_and_")}") do |*args| + CAPTAIN[IndexBuilder.get_key(klass, attributes, args)] + end + end + index_builder = IndexBuilder.new(self, attributes) + self.save_hooks << index_builder + self.destroy_hooks << index_builder + end + + # + # Will define a method called +name+ that will include all + # existing instances of this class that when yielded to the +block+ + # returns true. Will only return instances saved after this selector + # is defined. + # + def select(name, &block) #:yields: instance + key = self.collection_key(name) + CAPTAIN[key] ||= Hyperactive::Hash::Head.get_instance + self.class.class_eval do + define_method(name) do + CAPTAIN[key] + end + end + self.save_hooks << MatchSaver.new(key, Proc.new do |arg| + yield(arg) + end, :select) + self.destroy_hooks << MatchSaver.new(key, Proc.new do |arg| + yield(arg) + end, :delete_if_match) + end + + # + # Will define a method called +name+ that will include all + # existing instances of this class that when yielded to +block+ + # does not return true. Will only return instances saved after this + # rejector is defined. + # + def reject(name, &block) #:yields: instance + key = self.collection_key(name) + CAPTAIN[key] ||= Hyperactive::Hash::Head.get_instance + self.class.class_eval do + define_method(name) do + CAPTAIN[key] + end + end + self.save_hooks << MatchSaver.new(key, Proc.new do |arg| + yield(arg) + end, :reject) + self.destroy_hooks << MatchSaver.new(key, Proc.new do |arg| + yield(arg) + end, :delete_unless_match) + end + + + end + + end + + end + +end Modified: trunk/hyperactive/lib/hyperactive/record.rb =================================================================== --- trunk/hyperactive/lib/hyperactive/record.rb 2006-11-29 05:56:19 UTC (rev 86) +++ trunk/hyperactive/lib/hyperactive/record.rb 2006-11-30 22:06:57 UTC (rev 87) @@ -26,133 +26,17 @@ module Hyperactive # + # The default database connector. + # + CAPTAIN = Archipelago::Pirate::Captain.new + + # # The package containing the base class Bass that simplifies # using archipelago for generic database stuff. # module Record # - # The default database connector. - # - CAPTAIN = Archipelago::Pirate::Captain.new - - # - # A tiny callable class that saves stuff in - # indexes depending on certain attributes. - # - class IndexBuilder - # - # Get the first part of the key, that depends on the +attributes+. - # - def self.get_attribute_key_part(attributes) - "Hyperactive::IndexBuilder::#{attributes.join(",")}" - end - # - # Get the last part of the key, that depends on the +values+. - # - def self.get_value_key_part(values) - "#{values.join(",")}" - end - # - # Initialize an IndexBuilder giving it an array of +attributes+ - # that will be indexed. - # - def initialize(attributes) - @attributes = attributes - end - # - # Get the key for the given +record+. - # - def get_key_for(record) - values = @attributes.collect do |att| - if record.respond_to?(att) - record.send(att) - else - nil - end - end - key = "#{self.class.get_attribute_key_part(@attributes)}::#{self.class.get_value_key_part(values)}" - end - # - # Call this IndexBuilder and pass it a +block+. - # - # If the +argument+ is an Array then we know we are a save hook, - # otherwise we are a destroy hook. - # - def call(argument, &block) - yield - - # - # If the argument is an Array (of old value, new value) - # then we are a save hook, otherwise a destroy hook. - # - if Array === argument - record = argument.last - old_record = argument.first - old_key = get_key_for(old_record) - new_key = get_key_for(record) - if old_key != new_key - (CAPTAIN[old_key] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction)).delete(record.record_id) - (CAPTAIN[new_key] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction))[record.record_id] = record - end - else - record = argument - (CAPTAIN[get_key_for(record)] ||= Hyperactive::Hash::Head.get_instance_with_transaction(record.transaction)).delete(record.record_id) - end - end - end - - # - # A tiny callable class that saves stuff inside containers - # if they match certain criteria. - # - class MatchSaver - # - # Initialize this MatchSaver with a +key+, a callable +matcher+ - # and a +mode+ (:select, :reject, :delete_if_match or :delete_unless_match). - # - def initialize(key, matcher, mode) - @key = key - @matcher = matcher - @mode = mode - end - # - # Depending on @mode and return value of @matcher.call - # may save record in the Hash-like container named @key in - # the main database after having yielded to +block+. - # - def call(argument, &block) - yield - - record = argument - record = argument.last if Array === argument - - case @mode - when :select - if @matcher.call(record) - CAPTAIN[@key, record.transaction][record.record_id] = record - else - CAPTAIN[@key, record.transaction].delete(record.record_id) - end - when :reject - if @matcher.call(record) - CAPTAIN[@key, record.transaction].delete(record.record_id) - else - CAPTAIN[@key, record.transaction][record.record_id] = record - end - when :delete_if_match - if @matcher.call(record) - CAPTAIN[@key, record.transaction].delete(record.record_id) - end - when :delete_unless_match - unless @matcher.call(record) - CAPTAIN[@key, record.transaction].delete(record.record_id) - end - end - end - end - - # # A convenient base class to inherit when you want the basic utility methods # provided by for example ActiveRecord::Base *hint hint*. # @@ -164,6 +48,9 @@ # to get a proxy to the object stored into the database. # class Bass + + include Hyperactive::Index::Indexable + include Hyperactive::Transactions::Accessors @@create_hooks_by_class = {} @@destroy_hooks_by_class = {} @@ -176,58 +63,6 @@ HOST = "#{Socket::gethostbyname(Socket::gethostname)[0]}" rescue "localhost" # - # Call this if you want to change the default database connector - # to something else. - # - def self.setup(options = {}) - CAPTAIN.setup(options[:pirate_options]) - end - - # - # Works like normal attr_reader but with transactional awareness. - # - def self.attr_reader(*attributes) - attributes.each do |attribute| - define_method(attribute) do - value = instance_variable_get("@#{attribute}") - if Archipelago::Treasure::Dubloon === value - if @transaction ||= nil - return value.join(@transaction) - else - return value - end - else - return value - end - end - end - end - - # - # Works like normal attr_writer but with transactional awareness. - # - def self.attr_writer(*attributes) - attributes.each do |attribute| - define_method("#{attribute}=") do |new_value| - if Archipelago::Treasure::Dubloon === new_value - new_value.assert_transaction(@transaction) if @transaction ||= nil - instance_variable_set("@#{attribute}", new_value) - else - instance_variable_set("@#{attribute}", new_value) - end - end - end - end - - # - # Works like normal attr_accessor but with transactional awareness. - # - def self.attr_accessor(*attributes) - attr_reader(*attributes) - attr_writer(*attributes) - end - - # # Return our create_hooks, which can then be treated as any old Array. # # These must be callable objects with an arity of 1 @@ -297,69 +132,6 @@ end # - # Create an index for this class. - # - # Will create a method find_by_#{attributes.join("_and_")} for this - # class that will return what you expect. - # - def self.index_by(*attributes) - attribute_key_part = IndexBuilder.get_attribute_key_part(attributes) - self.class_eval < Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) @tm.publish! - Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) assert_within(20) do - Set.new(Hyperactive::Record::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) + Set.new(Hyperactive::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) end assert_within(20) do - Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] end end @@ -36,8 +36,8 @@ end def test_regular_hash_set_get - Hyperactive::Record::CAPTAIN["h"] = {} - h = Hyperactive::Record::CAPTAIN["h"] + Hyperactive::CAPTAIN["h"] = {} + h = Hyperactive::CAPTAIN["h"] r = Hyperactive::Record::Bass.get_instance hash_test("Hash", h, r, 100) end Modified: trunk/hyperactive/tests/hash_test.rb =================================================================== --- trunk/hyperactive/tests/hash_test.rb 2006-11-29 05:56:19 UTC (rev 86) +++ trunk/hyperactive/tests/hash_test.rb 2006-11-30 22:06:57 UTC (rev 87) @@ -19,14 +19,14 @@ @c2.publish! @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) @tm.publish! - Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) - Hyperactive::Record::CAPTAIN.update_services! + Hyperactive::CAPTAIN.update_services! assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort + Hyperactive::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort end assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] end end @@ -69,7 +69,7 @@ h2.each do |k,v| assert_equal(v, h[k]) - assert_equal(v, Hyperactive::Record::CAPTAIN[h.record_id][k]) + assert_equal(v, Hyperactive::CAPTAIN[h.record_id][k]) end end Modified: trunk/hyperactive/tests/list_benchmark.rb =================================================================== --- trunk/hyperactive/tests/list_benchmark.rb 2006-11-29 05:56:19 UTC (rev 86) +++ trunk/hyperactive/tests/list_benchmark.rb 2006-11-30 22:06:57 UTC (rev 87) @@ -10,13 +10,13 @@ @c2.publish! @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) @tm.publish! - Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) assert_within(20) do - Set.new(Hyperactive::Record::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) + Set.new(Hyperactive::CAPTAIN.chests.keys) == Set.new([@c.service_id, @c2.service_id]) end assert_within(20) do - Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] end end Modified: trunk/hyperactive/tests/list_test.rb =================================================================== --- trunk/hyperactive/tests/list_test.rb 2006-11-29 05:56:19 UTC (rev 86) +++ trunk/hyperactive/tests/list_test.rb 2006-11-30 22:06:57 UTC (rev 87) @@ -18,14 +18,14 @@ @c2.publish! @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) @tm.publish! - Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) - Hyperactive::Record::CAPTAIN.update_services! + Hyperactive::CAPTAIN.update_services! assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort + Hyperactive::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort end assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] end end @@ -56,8 +56,8 @@ r1 = l.first_element.record_id r2 = l.record_id l.destroy! - assert(!Hyperactive::Record::CAPTAIN.include?(r1)) - assert(!Hyperactive::Record::CAPTAIN.include?(r2)) + assert(!Hyperactive::CAPTAIN.include?(r1)) + assert(!Hyperactive::CAPTAIN.include?(r2)) end def test_each Modified: trunk/hyperactive/tests/record_test.rb =================================================================== --- trunk/hyperactive/tests/record_test.rb 2006-11-29 05:56:19 UTC (rev 86) +++ trunk/hyperactive/tests/record_test.rb 2006-11-30 22:06:57 UTC (rev 87) @@ -37,14 +37,14 @@ @c2.publish! @tm = TestManager.new(:persistence_provider => Archipelago::Hashish::BerkeleyHashishProvider.new(Pathname.new(__FILE__).parent.join("tranny1.db"))) @tm.publish! - Hyperactive::Record::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, + Hyperactive::CAPTAIN.setup(:chest_description => {:class => 'TestChest'}, :tranny_description => {:class => 'TestManager'}) - Hyperactive::Record::CAPTAIN.update_services! + Hyperactive::CAPTAIN.update_services! assert_within(10) do - Hyperactive::Record::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort + Hyperactive::CAPTAIN.chests.keys.sort == [@c.service_id, @c2.service_id].sort end assert_within(10) do - Hyperactive::Record::CAPTAIN.trannies.keys == [@tm.service_id] + Hyperactive::CAPTAIN.trannies.keys == [@tm.service_id] end $BEFORE_SAVE = 0 $AFTER_SAVE = 0 @@ -140,13 +140,13 @@ assert_equal(2, $BEFORE_SAVE) assert_equal(2, $AFTER_SAVE) assert_equal("brunt", r.bajs) - assert_equal("brunt", Hyperactive::Record::CAPTAIN[r.record_id].bajs) + assert_equal("brunt", Hyperactive::CAPTAIN[r.record_id].bajs) i = r.record_id - assert_equal(r, Hyperactive::Record::CAPTAIN[i]) + assert_equal(r, Hyperactive::CAPTAIN[i]) r.destroy! assert_equal(1, $BEFORE_DESTROY) assert_equal(1, $AFTER_DESTROY) - assert_equal(nil, Hyperactive::Record::CAPTAIN[i]) + assert_equal(nil, Hyperactive::CAPTAIN[i]) end end