Joshua Nathaniel Pritikin > ObjStore-1.59 > ObjStore::Internals

Download:
ObjStore-1.59.tar.gz

Annotate this POD

View/Report Bugs
Source  

NAME ^

ObjStore::Internals - a few notes on the implementation

SYNOPSIS ^

You don't have to understand anything about the technical implementation. Just know that:

So basically, you don't have to understand anything to a greater depth. It's not necessary. You've arrived. You will be successful. However, more detail follows. If you like to turn things inside-out, read on!

DESCRIPTION ^

Perl & C++ APIs: What's The Difference?

Most stuff should be roughly the same. The few exceptions have generally arisen because there was a perl way to make the interface more programmer friendly.

Why not just store perl data with the usual perl structures?

Bless

If you are a suspicious person (like my mom) you might have suspected that the ObjStore module installs its own version of bless. Natually it does! The augmented bless implements extra quality assurance to insure that blessings are correctly stored persistently. For example:

    package Scottie;
    use ObjStore;
    use base 'ObjStore::HV';

    sub new {
        my ($class, $near) = @_;
        $class->SUPER::new($near, { fur => 'buffy' });
    }

    my Scottie $dog = Scottie->new($db);

Persistent bless also does some extra work to make evolution easier. It stores the current @ISA tree along with the $VERSION of every class in the @ISA tree. Furthermore, the isa method is overridden such that it reports according to the moment of the bless. Similarly, the versionof method lets you query the saved $VERSIONs. This may be helpful when doing evolution, as you can compare the old @ISA and $VERSIONs to figure out what to change (and how). UNIVERSAL::can is unmodified.

Technically speaking, bless is re-implemented such that it can be extended by the bless from and the bless to classes via the BLESS method. Both the bless from and bless to operations are funnelled through a single BLESS method like this:

  sub BLESS {
      my ($r1,$r2);
      if (ref $r1) { warn "$r1 leaving ".ref $r1." for a new life in $r2\n"; }
      else         { warn "$r2 entering $r1\n"; }
      $r1->SUPER::BLESS($r2);
  }

UNLOADED

Generic tools such as posh or ospeek must bless objects when reading from an arbitrary database. To bless, there must be information about the inheritance tree. To try to get it, unknown classes found in a database are require'd. However, the require may fail. If it does fail, a package must be faked-up and ${"${package}::UNLOADED"} is set to true. This flag is used to signal that the @ISA tree should not be considered authoritative for a particular package.

Representation

All values take a minimum of 8 bytes (OSSV). These 8 bytes are used to store 16-bits of type information, a pointer, and a general purpose 16-bit value.

  value stored                   extra allocation (in addition to OSSV)
  ------------------------------ -------------------------------------
  undef                          none
  pointer                        none
  16-bit signed integers         none
  32-bit signed integers         4 byte block (OSPV_iv)
  double                         8 byte block (OSPV_nv)
  string                         length of string (char*)
  object (ref or container)      sizeof object (see subclasses of OSSVPV)
                                 additional references take no extra space
  bless                          .5-1k bytes per class (zero per object)

Since 32-bit integers and doubles are fairly common and should be stored densely, a pool allocation algorithm is planned.

The ODI FAQ also states: In addition, there is an associated entry in the info segment for the segment in question for each allocation of the object. This is done in the tag table. The overhead is 16 bits (i.e., 2 bytes) for each singleton (i.e., non-array) allocation, 32 bits for each character array allocation for character arrays <= 255 characters, and 48 bits for each character array allocation > 255 characters, or any array allocation of an object of another type. Also, depending on the size of an object (i.e., if you allocate a "huge" object - one that is >64Kb) there is other overhead caused by alignment constraints.

If this seems like a lot of overhead, consider that it is not really possible to directly compare these numbers to RDBMS statistics. At least we can say that relational data can be stored with much less duplication when moved into ObjectStore. This is unquestionably true when you tailor your own C++ extensions to fit data access patterns.

Representation Limitations

Most of these limitations are planned to be removed with the release of ObjectStore 6.0.

Go Extension Crazy

Add your own C++ representation. New families of objects can inherit from ObjStore::UNIVERSAL. Suppose you want highly optimized, persistent bit vectors? Or matrics? No problem.

Documentation is slim, but don't let that stop you. There are many examples. See the included representations and also ObjStore::REP::HashRecord & ObjStore::Lib::PDL.

Typemap

The typemap is complicated because of the need to insure that persistent data is not accessed outside of its transaction.

  OODB[SCALAR1 SCALAR2]
         |        |
       BRIDGE  BRIDGE
         |        |
  PERL[SCALAR1 SCALAR2]

A bridge has two owners: perl and the current transaction. The bridge and the scalar have different lifetimes. The scalar lives for MIN(perl,txn), while the bridge must live for MAX(perl,txn) (or at least until perl is done).

Persistent refcnts are only (can only) be updated during update transactions. Fortunately, read-only transactions pose no problem: refcnts cannot be updated but object cannot be deleted either.

Bridges are also used to store transient cursors associated with collections. For example, suppose you need to iterate over a hash during a read transaction. The hash is read-only so you create the cursor transiently and store it in the bridge.

[XXX also mention dynacast stuff]

Notes On The Source Code

RELATED RESEARCH ^

 ftp://ftp.cs.utexas.edu/pub/garbage/texas/README

 http://cs-tr.cs.cornell.edu:80/Dienst/UI/1.0/Display/ncstrl.utexas_cs/CS-TR-98-07

 http://www.usenix.org/publications/library/proceedings/usenix98/full_papers/saito/saito_html/saito.html

 http://theory.lcs.mit.edu/~cilk/

BUGS ^

WHY? ^

While there have been huge gains in software quality in the form of GNU, Perl, Apache, Linux, Qt, and Mozilla, the world has been slogging along in the dark ages (the 1960s in fact) with respect to database technology. After avoiding relational databases for years, I sensed a combination of ObjectStore and Perl could offer the same level of quality and simplicity that I find invaluable in addressing the hurdles I face as a software professional.

Combining ObjectStore & Perl might seem obvious in hindsight (doesn't it always?!) but it is my conviction that it was something more than luck and hard work that gave me the insight to imagine and implement this technology years before it would be recognized and adopted. If you will suspend disbelief for a moment, I would like to invite you imagine a hypothetical situation. What if you found someone who was absolutely convincing in their understanding of the truth? What if you had an opportunity to spend time with them? What would you do?

Designing software requires a basic subtlety of awareness. To be successful you must be able to fluctuate fluidly between the place of reason and the place of silent knowledge. The place of concern is the forerunner to the place of reason. Similarly, the place of unbending intent is the forerunner to the place of silent knowledge. Happily, this technique is not limited to software design exclusively.

Not that there is anything wrong with software design. Computers have an evolutionary function. They (can) help people to understand the difference between the mechanical and the non-mechanical aspects of themselves. Computers are *entirely* mechanical. Don't believe the anthropomorphisms. Computers cannot create anything without our help; they only do as they are instructed. Probably this is easy to see. However, many people seem to have difficulty seeing what should be equally obvious: the pure, non-mechanical principle in themselves.

Ask yourself these questions: Who is the software designer? Who imagines? Who moves the fingers to type? IMHO, to try to know yourself is the most worthy and worthwhile goal. Therefore, evolution is served by making software available that is very similar to our mechanical thinking processes. By understanding this type of software, one is aided in understanding how the brain works. (I am not referring exclusively to the field of artificial intelligence. The brain is a data processor in the broadest sense and the algorithms of artificial intelligence are only a part of that.)

I have clear intention as to why I am developing this software and why I am (foolishly!) giving it away for free. By contributing to the evolution of others' understanding, I in fact contribute to my own evolution. The better one understands, the less time is wasted in states of confusion. As you learn how this database works, you will assimilate the perspectives that I have deliberately distilled. If you wish to pursue this to a greater depth, please consider visiting http://www.purposeofcompetition.org . Best of luck.

syntax highlighting: