B-Tree .Net

a.k.a. Scalable Object Persistence (SOP)

version 4.8 released today, 10/17/2015!

v4.8 adds the following key enhancements:

* Native support for LINQ Queries is now supported! In v4.8, SOP's Linq to Objects extension (see Query function) expose a callable function that returns an efficient wrapper to the Store. Since SOP operates in the context of your class (POCO), all of Linq to Objects feature functionalities are automatically supported & inter-operates natively with the B-Tree Store. No translation occurring, data is served directly from the Store!

* Very efficient Multi Reader, Single Writer locking model for Store parallel access is also now available.
Pls. find unit tests & Sample demo programs (Simulator & LINQ Acceptance tests/demo) for these two new features.

* New demo: single user harness demo showing ability to do "initial load" insert of 100 million records in 35 minutes using my personal mac mini, or about 47,619 IO per second (IOPS) in this commodity PC. Typical SSDs have around 40 thousand IOPS hardware limit, thus, at this speed achieved, it will not matter whether the data engine is written in c/c++ or c#. It is like we've successfully hosted, embedded the fastest RDBMS' core data driver in our Apps. That is, cutting all of the connection points & layers, the hoops that the data has to travel from the Application to the RDBMS server, that low-level driver portion where data is cached as blocks & responsible for mapping to & from memory & disk.

* New automated testing based release process guarantees all of publicly accessible releases are of good quality and to have passed meticulous *scenario based, functionality* test suites.

v4.7 features threaded Data Reliability - support for multi-threaded Stores & Transactions, large data "smart" file streaming, more efficient memory utilization & addition of VirtualCache, a .Net ObjectCache implementation for ultra high speed, low footprint, scale-able caching!


SOP is a modern database engine within a code library. It is categorized as a NoSql engine, but which because of its scale-ability, is considered to be an enabler, coo-petition/player (wannabee) in the Big Data space.

Integration is one of SOP's primary goals, its ease of use, API, being part/closest! to the App & in-memory performance level were designed so it can get (optionally) utilized as a middle-ware for current RDBMS and other NoSql/Big Data engines/solution.

Code uses the Store API to store & manage key/value pairs of data. Internal Store implementation uses an enhanced, modernized B-Tree implementation that virtualizes RAM & Disk storage. Couple of key enhancements to this B-Tree as compared to traditional implementations are:
* node load optimization keeps it at around 75%-98% full average load of inner & leaf nodes. Traditional B-Trees only achieve about half-full (50%) average load. This translates to a more compressed or more dense data Stores saving IT shops from costly storage hardware.
* leaf nodes' height in a particular case is tolerated not to be perfectly balanced to favor speed of deletion at zero/minimal cost in exchange. Also, the height disparity due to deletion tends to get repaired during inserts due to the node load optimization feature discussed above.
* etc... a lot more enhancements waiting to be documented/cited as time permits.

The above listed enhancements together with the data structure and algorithms employed to take advantage of computer's memory (RAM) to keep more frequently used objects readily available to the application in their native form primarily distinguishes this implementation with the rest. Thus, the overall solution was branded virtualization of RAM and Disk due to the seamless-ness & effectivity offered.

SOP addresses data management scale-ability internally, at the data driver level, so when you use SOP code library, all you have to do is focus on authoring your application data solution. Nifty algorithms such as use of MRU data cache to keep frequently accessed data in memory, bulk I/O operations, B-Tree index usability optimizations, data bucketing for large data scenarios, etc... are already pre-baked, done at the driver level, so you don't have to.

Via usage of SOP API, your application will experience low latency, very high performance scalability.

Performance Level

Click on above link to navigate to SOP Performance Level details page.


FREE: Free to use, reuse, extend, no royalty fee on redistribution, no licensing fee, etc...

However, if you feel a sudden urge to motivate us further, you can give your token of appreciation & donate here: https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=TQ6ZW4SYBBNF2 

Release History

This is the 4th installment of the B-Tree Gold code library originally written in c++ & was released as Open Sourced on late '90s in programmer's haven and similar developer hub internet sites. In version 4.0 (released sometime 2009), the biggest functionalities comprising RAM & Disk virtualization were completed. Today in version 4.8, many key enhancements were added to further refine, optimize and streamline the code library & its performance.

Use-Case Scenarios

With open sourcing c# compiler & asp.net/.net, there will be lesser opposition in switching over and embracing the MS stack/.Net. In light of this, frameworks such as SOP which is one of the top of the line or top of the line modern data storage solutions, 'can offer big rewards for IT shops.

Real world use-case: create/host your website in godaddy.com, use Windows hosting w/ ASP.Net, use SOP to take advantage of 100GB of file system storage godaddy.com give away for free part of economy hosting. That means one can host a dynamic website w/ almost unlimited data storage capacity, with very affordable fees. :)

I'm actually prototyping it now, creating a sample website with this solution and soon to open source it as well, as a sample real world solution. (see Portals Gallery for details)

Anyway, 'just wanted to add some "beef" of what SOP brings over to the table. As another alternative, 'could be better in some use-case, e.g. - than typical MySQL, Sql Server data storage on popular hosting sites (such as godaddy).

Click on above "Use-Case Scenarios" link to navigate to other use cases' discussion page.


Click on above link to navigate to SOP logical and physical structure details page.

Store/Data Driver Algorithms

Click on above link to navigate to SOP's Store/Data Driver Algorithms details page.

Sample usage


using(var server = new ObjectServer("c:\\SopBin\\OServer.dta"))
    var store = server.StoreNavigator.GetStore<int, string>("SystemFile/MyStore");
    store.Add(1, "Hello World!");
    store.Add(2, "Foo Bar");

This is a very basic, sample usage of SOP framework. It will create (or open if existing) transaction protected SOP server data file containing a single data Store named "MyStore" (key is int, value is string type) in the SystemFile. This Store is (B-Tree) indexed as all Stores in SOP are, :). We are inserting following key/value pairs:

  • key: 1 value: "Hello World!"
  • key: 2 value: "Foo Bar"

At end of the block, we commit the Transaction. No need to define an index as the Store itself is the B-Tree index.

Another example for managing different kinds of objects in the same Store:

using(var server = new ObjectServer("c:\\SopBin\\OServer.dta"))
    var store = server.StoreNavigator.
GetStore<string, object>("SystemFile/MyStore");
new Person{ Firstname = "Joe", Lastname = "Moe" } );
new Address{ Street = "123 Love Lane",
City = "Fremont", State = "Ca" } );
    store.Add("JoeSS#", "603-45-1110");
server.Commit(); }

The code example shows inserting three kinds of objects with different string keys. A Person, an Address and a SS#. Person and Address structure definition was not shown for brevity.

Build Instructions

Click on above link to navigate to Build Instructions page.

Sample Programs

Click on above link to navigate to Sample programs page.

Last edited Nov 19 at 9:37 PM by grecinto, version 266