Is there a spec document for SOP objects pack format

May 22, 2013 at 10:33 AM
Hi,
I am studying the source codes of SOP recently, it is really a great project.
One situation that prevents me from knowing its detail better is, SOP supports several types for storing, and the pack/unpack logic for them are scattered in different places.
Sometimes it is difficult for me to imagine what a tree looks like in disks.
Is there a pack/unpack spec document for SOP support objects?
Where can we find a .dta file layout document?

I guess if more people get to know the low level details, more people can join this project and enhance it.
Coordinator
Jul 22, 2013 at 8:33 AM
Edited Jul 23, 2013 at 3:30 AM
I agree, documentation is an area that needs work as well... 'pardon I have missed this posting for a long while and only saw it tonight. I hope below helps a bit... 'until a spec doc about this subject matter is created.

Object Serialization
SOP is open ended when it comes to Object serialization. SOP doesn't attempt to dictate or drive the serialization, but instead it treats the Objects (be it "internal" Algorithm business logic or User object) as a black box.

Each internal SOP Algorithm business logic, e.g. - B-Tree, implements its own Object serialization method and format. B-Tree Algorithm serializes each Node on a contiguous block on disk. The Node is a structure containing Entries and Pointers (long integer file offsets) to other Nodes comprising the B-Tree. Each entry contains the Key & Value (or file offset if Value is not persisted part of the Node) pair defined by the user.

A File can contain one or more ObjectStores and each ObjectStore can actually contain ObjectStores as members. There is a dta File layout submitted part of the Documentation, somehow I couldn't find it as well. Anyway, I'll make sure it is there if it got lost.

IInternalPersistent
Internal SOP Algorithm business logic implements the IInternalPersistent interface for defining how objects will be packed/unpacked to/from disk.

Here is the IInternalPersistent interface definition for illustration:
public interface IInternalPersistent : IWithHintSize
{
    void Pack(IInternalPersistent parent, System.IO.BinaryWriter writer);
    void Unpack(IInternalPersistent parent, System.IO.BinaryReader reader);
    DataBlock DiskBuffer { get; set; }
    bool IsDirty { get; set; }
}
IPersistent
Key &/Or Value data that are of standard types such as string, int, long, short, float, decimal, etc... are persisted using respective type serializer method of BinaryWriter and BinaryReader to achieve best performance and efficient use of storage.

For user defined structures (POCOs), User can choose to implement IPersistent interface to have total control on how the Key &/or Value Objects' data will be packed or unpacked to/from disk. Also, there are built-in support for other .net serialization types such as Xml Serialization, BinarySerialization. Other serialization types can be easily added in the user facing Helper classes, e.g. - StoreFactory. Pls. see the SOP sample programs for code illustration of how these serialization methods are utilized.

Here is the IPersistent interface definition for illustration:
public interface IPersistent : IWithHintSize
{
    void Pack(System.IO.BinaryWriter writer);
    void Unpack(System.IO.BinaryReader reader);
    bool IsDisposed { get; set; }
}
Coordinator
Jul 22, 2013 at 8:36 AM
Here is link to the Structure page which discusses the Server, File & ObjectStore layout: https://sop.codeplex.com/wikipage?title=Structure