Corrupted store after a couple of bulk inserts ?!

Nov 21, 2012 at 10:18 AM
Edited Nov 21, 2012 at 10:45 AM

I played a bit with SOP, and it's very interesting!

But i have a problem after a couple of bulk inserts. See the simple test program below.

It opens a store, insert 10.000 simple type values, commit and close.
Then it opens the store and reads all items. After some iterations the store seems to be corrupted. Any idea what's going wrong ?

 

 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Sop;

namespace SopTest
{
    class Program
    {
        const string filename = @"E:\SopBin\store.dta";

        static void Main(string[] args)
        {
            for (int i = 0; i < 100; i++)
            {
                Populate(10000);
                if (!ReadAll())
                    break;
            }
        }

        static void Populate(int count)
        {
            Console.Write("{0} | Write {1} items ... ", DateTime.Now, count);
            
            IObjectServer server;
            ISortedDictionary<string, int> store;

            server = ObjectServer.OpenWithTransaction(filename);
            IStoreFactory sf = new StoreFactory();
            store = sf.Get<string, int>(server.SystemFile.Store, "MyStore");

            Int32 maxID = store.Count;

            for (int i = 0; i < count; i++)
            {
                Int32 id = (Int32)store.GetNextSequence();
                string name = string.Format("1111-2222-3333-4444-{0}", maxID + i);

                store.Add(name, id);
            }

            server.Commit();
            server.Dispose();
            server = null;

            Console.WriteLine("done.");
        }


        static bool ReadAll()
        {
            Console.Write("{0} | Read items ... ", DateTime.Now);

            IObjectServer server;
            ISortedDictionary<string, int> store;

            server = ObjectServer.OpenWithTransaction(filename);
            IStoreFactory sf = new StoreFactory();
            store = sf.Get<string, int>(server.SystemFile.Store, "MyStore");

            if (store.MoveFirst())
            {
                store.HintBatchCount = 103;
                int Ctr = 0;
                do
                {
                    Ctr++;
                    Int32 p = store.CurrentValue;
                } while (store.MoveNext());
                if (Ctr != store.Count)
                {
                    Console.WriteLine("{0} found, expected {1} [FAILED]", Ctr, store.Count);
                    return false;
                }
                else
                    Console.WriteLine("{0} found [OK]", Ctr);
            }

            server.Commit();
            server.Dispose();
            server = null;

            return true;
        }
    }
}

Coordinator
Dec 11, 2012 at 10:01 PM

I will need to take a look on this tonight. I get it that only the "bulk" reads (store.HintBatchCount = 103) use-case seems problematic in your case above as you've mentioned. It is possible that the bulk read cache can get out of sync in this case above. So for now, maybe if you can not use the bulk read just as workaround.

I'll get back with more details & fix if there is a problem found and get corrected.

Thx much,

-G.

Coordinator
Dec 11, 2012 at 10:38 PM

O btw, Bulk Read in this case doesn't give you any performance advantage because both key & data are stored on the same segment. Bulk Reads is a feature created for optimizing bulk reading on cases where the Data is on a different segment than the Key. Just fyi. tx

Coordinator
Dec 13, 2012 at 5:34 AM

ABauer: no issue found. I ran your program above 2 times (insert/read 2 million entries in 10K batch increments) and all operations were successful. No corruption nor any issue detected. I don't have idea how you ran your program or if it is the same program as above to give you more insights on this topic.

Btw, your program above can get optimized, e.g. - Same ObjectServer instance can be kept open between Populate and Read methods. You don't need to dispose it and reopen on Read method. Also, you can cache the Store and keep it open, etc... This to take advantage of SOP's store caching.

Anyway, I thank you very much for this opportunity to exchange thoughts and explore SOP's stability. Until next...

-G.

Dec 13, 2012 at 7:15 AM

Thank you for the response and the detailed information. The program above was only a simple test, but i will try it again. Maybe i have to learn a bit more about SOP to better understand the architecture.

Alex

Coordinator
Jan 7, 2013 at 11:32 PM

Hi Alex,

Awesome, pls. feel free to let me know if I can be of help on anything SOP.

Thx & Happy New Year!

-G.