This project is read-only.

Big Data .Net

Mar 2, 2014 at 7:35 PM
Edited Mar 19, 2014 at 5:15 PM
Good morning to All! I never felt better in my life than today; I think management of my recently diagnosed sleeping disorder (sleep Apnea) attributes to some of key behavioral/focus improvement of yours truly. Anyway, ‘back to business… :)

This document is a work in progress, an initial stab discussing the designs of the upcoming SOP Server for Big Data.

The designs are a perfect match to SOP's key foundational assets allowing an almost seamless enhancement to the SOP framework, i.e. – addition of the Server component that will enable high throughput workload distribution/sharing across machines.


Enabling multi-machine (or Process) workload sharing is a fun challenge, :). Indeed, there are many ways to skin a cat. At the high level, all we are doing is adding a Server application/component to the SOP Framework, which allows user to group together multiple instances of this Server piece and each instance, knowing how to efficiently delegate object management/retrieval tasks (from storage) to one another based on load.

Efficiency is the word. How do we achieve very high level of efficiency for multiple concurrently or parallel running processes? Each process can take advantage of resources such as multiple connectivity channels to the Storage device(s), full utilization of high speed RAM, delegation and data management trafficking, etc…

At the core of a database engine such as SOP, approaches on data management trafficking (a.k.a. – cross Transactions data management) dictate the level of performance on the holistic level. Specifically, the method used to prevent data corruption on concurrently and parallel running processes affect the overall throughput.

Initially, a good amount of effort was invested on exploring the idea of implementing current proven approaches such as MVCC/ACID transactions and facilities to provide it, such as row/field level versioning/locking, lock escalation, etc…

However, modern Systems such as SOP should not be bounded to follow the current techniques. The goal is to achieve next level of performance throughput, not to follow and reinvent necessarily. :)

The following key enhancements of a system are identified:
• Reduced/minimal lock contentions
• Ability to offer some level of MVCC; ‘different transactions will have some ACID attributes but not fully. Couple of main enhancements to traditional techniques in order to resolve and enhance performance are:
  • System will not require Object (equivalent of a row) level versioning and tracking, which can add cost
  • Management action(s) on same Object(s) across ~simultaneously committed Transactions will be allowed & replaced by auto merging/conflict resolution facilities. This key feature is not done or rather can't be done by the big boys in RDBMS, e.g. - Oracle, for some technical reasons. As SOP is geared for working very closely with your Application, e.g. - as embedded DB engine. Interfaces can be authored for automatic merging and conflict resolution based on your Application business rules without impacting performance. This key feature also allows SOP Server to scale much better, simplify cross transactions' data management conflict resolution & drop any unnecessary costs such as object (row) versioning and tracking.

Dynamic Clustering, a.k.a. Grid?

Users will be able to setup SOP Server across machines and group them together to create a Cluster. I.e. - a group of SOP servers working together to distribute workload of object management and searching. Cluster configuration will be automatic. Server will automatically self register and be able to rejoin the cluster after a failed event such as App crash or sudden hardware reset. Due to the mentioned characteristics and these additional ones: absence of bottlenecks, virtualization in the DNA(Sop core) providing unlimited scalability & ability to distribute almost any run time tasks to the members, Sop on the cluster it seems will be a Grid like Server software.
Do we dare call it the Sop Grid? :)

Sample Clustering Scenario:

SAN resident SOP Files used to store BayWind Corp’s world-wide Customer Sales information:
• Object Store System File on SAN 1 disk
• File 1 on SAN 1 disk
• File 2 on SAN 2 disk
• File 3 on SAN 3 disk
• File 4 on SAN 4 disk

Cluster Member Machines and SOP Servers:
• Machine A with Web App 1 hosting SOP Server instance 1
• Machine B with Web App 2 hosting SOP Server instance 2
• Machine C with Web App 3 hosting SOP Server instance 3
• Machine D with Web App 4 hosting SOP Server instance 4

Initial release of SOP for Big Data.Net will be fully adapted to the latest Microsoft Web technologies such as ASP.Net, IIS and web load balancers.

Each instance of SOP will be cooperating members of the web farm cluster, each sharing workload of data retrieval and management on behalf of the host application.

Each SOP instance will have two potential roles, being the Cluster Controller and/or a Cluster Member. The Controller will provide Cluster services such as Member registration/tracking, Cluster resource utilization limits enforcement, client to ObjectStore instance pairing/reuse, etc...

A Member provides services to manage, instantiate, reuse ObjectStores of Files in the Cluster.

Transaction, Concurrency & Locking
Due to Sop's unique virtualization that ensures more frequently utilized objects reside in-memory, it requires an algorithm for multiple transaction of concurrent/parallel processes optimal usage of objects in-memory with the necessary resource locking. Following details describe this:
  • Reading Objects from a Store will not require locks. e.g. - multiple Apps in the farm reading data from the same Store abc will be able to read in parallel with no contention even if there are write operations occurring
  • Current SOP Transaction will be used to support Cluster wide transaction that are committed on demand &/or batch intervals. This means transaction is used mostly to provide hardware failure data protection, to the Apps, each management action is committed when done
  • Management operations on different ObjectStores in write mode can run concurrently or in parallel if in remote machine
  • Controller will distribute ObjectStore in write mode instantiation across Members in the cluster
SQL Server Sync
An interface for timed interval syncing of data with SQL Server Database(s) will be provided. Being able to make the data available in SQL Server delayed only by a bit enables all of the available tools for SQL Server to be usable as they are today. Your investment on these tools will not go to waste.

This solution, in fact, provides optimization in SQL I/O, 'serving as a natural, lightweight batching mechanism. e.g. - Data (Bulk) Sync that occur every five minutes will be so much more optimal as opposed to 100 or thousands of simultaneous client triggered data management actions to target SQL DBs.

Multi-Dimensional Scaling
In Sop Next, big ObjectStore can be configured. Big Stores differ than ordinary types in that, it is a logical construct comprised of physical Stores that actually manage data for load balancing.
Big Store if running in a Cluster can distribute sub Store hosting across the Cluster members based on load. Thus, addresses two types of scale within the Store level:
  • data partitioning across Sop Files (Database equivalent in Rdbms)
  • data management distribution across Cluster members
  • MetaStore1 Big Store
  • Comprised of 100 phys. Stores to manage 20 GB of data each
  • Each store data stored in a dedicated File
  • Each File is given a dedicated channel to separate SAN disk with guaranteed I/O & transfer speeds
  • Aggregated capacity is about 2 Terrabytes of data spread across 100 Stores instances of which are distributed in the Cluster
  • Estimated Aggregated Performance should surpass anything that is built today
Query the Cluster
If time permits, a Query engine that knows how to efficiently utilize the Cluster resources will be added. LINQ for (Sop) Entities will allow developers on querying the Stores without need for understanding the Sop Cluster and where their query gets executed.

Engine can guarantee optimal load distribution anytime.

Simple Store Navigation
As SOP Stores are hierarchical, a URI path based store retrieval API will be made. This should allow developers to focus on authoring Store data retrieval and data management operations and less on navigating the SOP File(s) and their Store/Sub-Store hierarchy.

Sample valid Store URI paths:
"SystemFile/Store1" - implicitly references SystemFile/ObjectStore/Store1
"SystemFile/Store2/Store2.1" - implicitly references SystemFile/ObjectStore/Store2/Store2.1

"File1/Store1/Store1.1" - implicitly references File1/ObjectStore/Store1/Store1.1.

Sample code using the new StoreNavigator API which understands the new Store URI paths:
// server instantiation not shown for brevity.
var storeNav = new Sop.StoreNavigator(server);
var store = storeNav.GetStore<string, string>("SystemFile/Store2.3/Store2.3.1");
// your code to retrieve or manage store data goes here...