Structure

Sample physical and logical data file layout to illustrate SOP's file structure and main controller/logic hubs -
ObjectServer, FileSet and object Stores:
ObjectServer
    SystemFile (Filename: c://SopBin/Server.dta)
             Store
                     Key1, Value1
                     Key2, Value2
                     Store2Key, Store2
                            Key1, Value1
                            Key2, Value2
                            Store3Key, Store3
                                   Key1, Value1
                                   Key2, Value2
                     //...
    FileSet
             File1 (Filename: d://ServerFiles/File1)
                     Store
                            Key1, Value1
                            Key2, Value2
                            Key3, Value3
                            Key4, Value4
             File2 (Filename: d://ServerFiles/File2)
                     Store
                            Key1, Value1
                            Key2, Value2
                            Store2Key, Store2
                                    Key1, Value1
                                    Key2, Value2
                                    Store3Key, Store3
                                            Key1, Value1
                                            Key2, Value2
                     //...
             File3 (Filename: d://ServerFiles/File3)
                    //...
  • Object Server - contains the System File of which, its object Store is mainly used by SOP to store for "internal use" information. It is also available for use as general purpose application object Store, but recommended use-case is for it to be used as a Container of user's application Stores (see Object Store description below for details). Object Server FileSet member is a collection of File objects. It has API allowing user to manage (create, delete, get) File members.
  • File - a container of object Stores. Built-in Store member can contain application data (key & value pairs) and/or as Container of other application Store(s), nesting level is limited only by host computer's resources. Each File can be given its own user defined preference, which is used as source of default settings that "controls" performance vs. resource allocation logic in data management and queries in Object Stores. Each File can be stored on a different folder or a different disk drive than the other File in the Object Server.
  • Object Store - a container of objects. It is high performance, cursor-based (see Movexxx, CurrentKey/CurrentValue API set), sorted, duplicates allowed dictionary of key and value pairs. An Object Store can contain "nested" Object Store(s) as member value(s), thus making it "hierarchical".

Object Stores are implemented internally using SOP's flavor of (modified) B-Tree algorithm for fast and scalable performance on item management and searching. This B-Tree algorithm implementation eliminates (or minimizes) performance impact due to B-Tree Node branch "load balancing", unlike more classical or traditional B-Tree algorithm implementations.
Since everything is stored using data Stores in SOP database, everything is indexed essentially. :)

Design highlights for the File construct:
  • Give user option to group together in a File similar in usage Object Stores. Each Store in the same File can have the same user defined preference which drives memory usage (caching) and data mapping on disk. Example, setup a File with big datablock size (datablock is smallest allocation unit on disk) for storing entities occupying large amount of data. All object Stores in this File will benefit from large blocks. Use another File for storing your typical entities (fairly sized data). Fine tune the preferences (set memory limits for caching, etc...) for each File and you have a good setup that balances the "tradeoffs" between memory, disk utilizations and performance.
  • Since File can be physically stored on a different disk drive, it allows end-user to take advantage of parallel I/O (multiple HDD heads) just by data layout across Files. Each object Store can be managed/used on a different thread of execution and those object Stores physically stored on different HDD will then fully take advantage of different HDD's drive heads doing parallel I/O. This is not to mention each File can also be stored in a RAID drive and thus, because of SOP's unbuffered, asynchronous overlapping I/O, object Store pretty much takes advantage of RAID structure "design optimizations" set up by IT, doing parallel I/O to occur. Each time object Store does "bulk data" disk mapping, it will utilize multiple HDD heads as available in the RAID drive configuration. Some RAID settings physically stores disk segments across different HDD drive in the RAID array. SOP was designed and written to take advantage of these disk configurations, one of the many scaling "design features" of SOP, all built-in and not needing for end-user to do anything programmatically, aside from simply organizing the Server, File and Store layout in a manner how we organize files intuitively. SOP API also was maintained simple to use without any requirement for learning these advanced disk mapping techniques.

Usage of SOP in an exceptional language such as c# for your data persistence needs allows very high performance, fine grained logic control in your data access and queries. You are sure to hit and use that B-Tree index 100% everytime!

Last edited Oct 2, 2012 at 6:25 AM by grecinto, version 1

Comments

No comments yet.