Mono compatibility question again

[mhshirley]

Is STSdb compatible with mono?

This question was asked in August 2011 with respect to mono 2.10.2.
I'm trying my test application today under mono 3.10.0 for the first time, and it fails with the error below.
(It runs successfully under .net 4.5.50938.)
The exception was TypeInitializationException, with an inner exception, I think, of System.ArgumentNullException: Argument cannot be null.

Right now, I don't know whether I can generate a smaller example, because it runs for a while and then fails deep inside STSdb. I have no idea what triggered it. The object types being stored are this:

publicclassPacketKey
{
publicintSession;
publicintAPID;
publiclongTimestamp;
}

as keys, and byte arrays as values.

Here's the full transcript.
[mshirley@sg-loaner22-snrf Debug]$ mono Test1.exe
32
0
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013266185604.HK.0
4.201761 Mbits/sec
363.0853 Mbits/sec
1.453 Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013266205609.HK.0
218.312 Mbits/sec
311.9429 Mbits/sec
25.512 Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013266225611.HK.0
181.2583 Mbits/sec
121.8404 Mbits/sec
Infinity Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267005615.HK.0
53.10927 Mbits/sec
136.3195 Mbits/sec
0.204129 Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267025620.HK.0
121.1493 Mbits/sec
92.67711 Mbits/sec
5.624 Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267045624.HK.0
85.49773 Mbits/sec
90.73283 Mbits/sec
Infinity Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267065628.HK.0
26.74925 Mbits/sec
59.66115 Mbits/sec
0.2596842 Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267085632.HK.0
82.15411 Mbits/sec
80.62637 Mbits/sec
17.616 Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267105635.HK.0
69.17587 Mbits/sec
55.83477 Mbits/sec
Infinity Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267125642.HK.0
18.00337 Mbits/sec
63.23154 Mbits/sec
0.1452043 Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267145643.HK.0
58.84238 Mbits/sec
58.90692 Mbits/sec
11.608 Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267165648.HK.0
54.4644 Mbits/sec
53.76662 Mbits/sec
Infinity Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267185651.HK.0
14.02292 Mbits/sec
51.19821 Mbits/sec
0.1217094 Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267205656.HK.0
49.51355 Mbits/sec
47.81969 Mbits/sec
46.72 Mbits/sec
/home/mshirley/code/STSdb/Test1/hk/SC_HK_2013267225659.HK.0
System.AggregateException: One or more errors occurred ---> System.TypeInitializationException: An exception was thrown by the type initializer for STSdb4.General.Extensions.SortedSetHelper`1 ---> System.ArgumentNullException: Argument cannot be null.
Parameter name: method
at System.Linq.Expressions.Expression.Call (System.Linq.Expressions.Expression instance, System.Reflection.MethodInfo method, IEnumerable`1 arguments) [0x00000] in <filename unknown>:0
at System.Linq.Expressions.Expression.Call (System.Linq.Expressions.Expression instance, System.Reflection.MethodInfo method, System.Linq.Expressions.Expression[] arguments) [0x00000] in <filename unknown>:0
at STSdb4.General.Extensions.SortedSetHelper`1[System.Collections.Generic.KeyValuePair`2[STSdb4.Data.IData,STSdb4.Data.IData]].CreateFindMethod () [0x00000] in <filename unknown>:0
at STSdb4.General.Extensions.SortedSetHelper`1[System.Collections.Generic.KeyValuePair`2[STSdb4.Data.IData,STSdb4.Data.IData]]..ctor () [0x00000] in <filename unknown>:0
at STSdb4.General.Extensions.SortedSetHelper`1[System.Collections.Generic.KeyValuePair`2[STSdb4.Data.IData,STSdb4.Data.IData]]..cctor () [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at STSdb4.General.Extensions.SortedSetExtensions.Spli t[KeyValuePair`2] (System.Collections.Generic.SortedSet`1 set, Int32 count) [0x00000] in <filename unknown>:0
at STSdb4.Database.RecordSet.InternalSplit (Int32 count) [0x00000] in <filename unknown>:0
at STSdb4.Database.RecordSet.Split (Double percentage) [0x00000] in <filename unknown>:0
at STSdb4.WaterfallTree.WTree+LeafNode.Split () [0x00000] in <filename unknown>:0
at STSdb4.WaterfallTree.WTree+InternalNode+Maintenanc eHelper.Split (Int32 index) [0x00000] in <filename unknown>:0
at STSdb4.WaterfallTree.WTree+InternalNode+Maintenanc eHelper.Do (System.Object state) [0x00000] in <filename unknown>:0
at System.Threading.Tasks.TaskActionInvoker+ActionObj ectInvoke.Invoke (System.Threading.Tasks.Task owner, System.Object state, System.Threading.Tasks.Task context) [0x00000] in <filename unknown>:0
at System.Threading.Tasks.Task.InnerInvoke () [0x00000] in <filename unknown>:0
at System.Threading.Tasks.Task.ThreadStart () [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.Wait (Int32 millisecondsTimeout, CancellationToken cancellationToken) [0x00000] in <filename unknown>:0
at System.Threading.Tasks.Task.Wait () [0x00000] in <filename unknown>:0
at STSdb4.WaterfallTree.WTree+InternalNode.Maintenanc e (Int32 level, STSdb4.WaterfallTree.Token token) [0x00000] in <filename unknown>:0
at STSdb4.WaterfallTree.WTree+Branch.MaintenanceRoot (STSdb4.WaterfallTree.Token token) [0x00000] in <filename unknown>:0
at STSdb4.WaterfallTree.WTree.Sink () [0x00000] in <filename unknown>:0
at STSdb4.WaterfallTree.WTree.Execute (IOperationCollection operations) [0x00000] in <filename unknown>:0
at STSdb4.Database.XTable.Flush () [0x00000] in <filename unknown>:0
at STSdb4.Database.XTable.Execute (IOperation operation) [0x00000] in <filename unknown>:0
at STSdb4.Database.XTable.Replace (IData key, IData record) [0x00000] in <filename unknown>:0
at STSdb4.Database.XTable.set_Item (IData key, IData value) [0x00000] in <filename unknown>:0
at STSdb4.Database.XTable`2[Test1.PacketKey,System.Byte[]].set_Item (Test1.PacketKey key, System.Byte[] value) [0x00000] in <filename unknown>:0
at Test1.Form1+<button9_Click>c__AnonStorey1.<>m__1 (System.IO.FileInfo f) [0x00000] in <filename unknown>:0
at TLMdb.CCSDS.FileTreeWalker.Walk (System.IO.DirectoryInfo di, TLMdb.CCSDS.WalkFunction f) [0x00000] in <filename unknown>:0
at TLMdb.CCSDS.FileTreeWalker.Walk (System.String dir, TLMdb.CCSDS.WalkFunction f) [0x00000] in <filename unknown>:0
at Test1.Form1.button9_Click (System.Object sender, System.EventArgs e) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Control.OnClick (System.EventArgs e) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Button.OnClick (System.EventArgs e) [0x00000] in <filename unknown>:0
at System.Windows.Forms.ButtonBase.OnMouseUp (System.Windows.Forms.MouseEventArgs mevent) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Button.OnMouseUp (System.Windows.Forms.MouseEventArgs mevent) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Control.WmLButtonUp (System.Windows.Forms.Message& m) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Control.WndProc (System.Windows.Forms.Message& m) [0x00000] in <filename unknown>:0
at System.Windows.Forms.ButtonBase.WndProc (System.Windows.Forms.Message& m) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Button.WndProc (System.Windows.Forms.Message& m) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Control+ControlWindowTarget.O nMessage (System.Windows.Forms.Message& m) [0x00000] in <filename unknown>:0
at System.Windows.Forms.Control+ControlNativeWindow.W ndProc (System.Windows.Forms.Message& m) [0x00000] in <filename unknown>:0
at System.Windows.Forms.NativeWindow.WndProc (IntPtr hWnd, Msg msg, IntPtr wParam, IntPtr lParam) [0x00000] in <filename unknown>:0
--> (Inner exception 0) System.TypeInitializationException: An exception was thrown by the type initializer for STSdb4.General.Extensions.SortedSetHelper`1 ---> System.ArgumentNullException: Argument cannot be null.
Parameter name: method
at System.Linq.Expressions.Expression.Call (System.Linq.Expressions.Expression instance, System.Reflection.MethodInfo method, IEnumerable`1 arguments) [0x00000] in <filename unknown>:0
at System.Linq.Expressions.Expression.Call (System.Linq.Expressions.Expression instance, System.Reflection.MethodInfo method, System.Linq.Expressions.Expression[] arguments) [0x00000] in <filename unknown>:0
at STSdb4.General.Extensions.SortedSetHelper`1[System.Collections.Generic.KeyValuePair`2[STSdb4.Data.IData,STSdb4.Data.IData]].CreateFindMethod () [0x00000] in <filename unknown>:0
at STSdb4.General.Extensions.SortedSetHelper`1[System.Collections.Generic.KeyValuePair`2[STSdb4.Data.IData,STSdb4.Data.IData]]..ctor () [0x00000] in <filename unknown>:0
at STSdb4.General.Extensions.SortedSetHelper`1[System.Collections.Generic.KeyValuePair`2[STSdb4.Data.IData,STSdb4.Data.IData]]..cctor () [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at STSdb4.General.Extensions.SortedSetExtensions.Spli t[KeyValuePair`2] (System.Collections.Generic.SortedSet`1 set, Int32 count) [0x00000] in <filename unknown>:0
at STSdb4.Database.RecordSet.InternalSplit (Int32 count) [0x00000] in <filename unknown>:0
at STSdb4.Database.RecordSet.Split (Double percentage) [0x00000] in <filename unknown>:0
at STSdb4.WaterfallTree.WTree+LeafNode.Split () [0x00000] in <filename unknown>:0
at STSdb4.WaterfallTree.WTree+InternalNode+Maintenanc eHelper.Split (Int32 index) [0x00000] in <filename unknown>:0
at STSdb4.WaterfallTree.WTree+InternalNode+Maintenanc eHelper.Do (System.Object state) [0x00000] in <filename unknown>:0
at System.Threading.Tasks.TaskActionInvoker+ActionObj ectInvoke.Invoke (System.Threading.Tasks.Task owner, System.Object state, System.Threading.Tasks.Task context) [0x00000] in <filename unknown>:0
at System.Threading.Tasks.Task.InnerInvoke () [0x00000] in <filename unknown>:0
at System.Threading.Tasks.Task.ThreadStart () [0x00000] in <filename unknown>:0


[mshirley@sg-loaner22-snrf Debug]$ mono -V
Mono JIT compiler version 3.10.0 (tarball Tue Nov 4 09:23:12 PST 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. Home | Mono
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
LLVM: supported, not enabled.
GC: sgen
[mshirley@sg-loaner22-snrf Debug]$

[mhshirley]

This is an attempt at replicating the error above by compiling from source. While it still doesn't work, the error is different. I'll confirm that the compiled version of STSdb I downloaded corresponds to the source I downloaded (although I just clicked on the download links and did nothing special). However, I think this still reflects a bug when running under mono.


Starting with STSdb_4.0.5-src.rar


I compiled it under monodevelop 5.4. One of the files had the wrong
case for the initial character and the solution didn't initially
compile.


$ mono -V
Mono JIT compiler version 3.10.0 (tarball Tue Nov 4 09:23:12 PST 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. Home | Mono
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
LLVM: supported, not enabled.
GC: sgen


My application inserts a series key/value pairs. The keys are instances:


public class PacketKey
{
public int Session;
public int APID;
public long Timestamp;
}


The value are byte arrays.


My code is simple. It creates the engine and inserts data on the gui
thread. It calls commit() every 5000 table writes.

When I run it, it hangs inside commit(), and the mono
debugger reports that there are no active threads. That is, the gui
thread has silently died.


When I have the debugger halt on System.Exception, STSdb gets an
exception with the error string "Internal error: rotation". This
error is on one of the thread pool threads.


If I continue, I get three instances System.AggregateException, and
then all threads are dead.


The debugger shows the call site of the function that threw the
exception but not the code that threw it, The function that throws the
exception is held in a variable, so it's not easily apparent what code
threw the exception. If I have monodevelop search the STSdb project
for the strings "Internal error" or "rotation", it finds nothing. If
I grep -r the STSdb project directory, that also finds nothing.
However, there are some exceptions whose message strings are
calculated at runtime, so I've stopped looking for the place where the
exception is thrown for now.


I've modified the code so that it writes only new byte[0] into the
table using the same keys as before. The "Internal error: rotation"
exception is still thrown, so the exception is sensitive to the structure of the
waterfall tree and not to the values. I have no idea how to get
the waterfall tree into the state where the exception is thrown
without the specific sequence of writes I'm using. Would it be
helpful if I wrote the keys to a file and changed my program to read
that and posted that?


Note that this runs fine under windows.

[mhshirley]

Hello. I view this question as a bug report (failure under mono of a scenario that works properly under microsoft's CLR), but I don't have a sense for whether this question was noticed by the developers or whether they would be interested in addressing it and want more detailed information. Should I have posted this in a different forum?
- Mark Shirley

[k.dimitrov]

Hello. I view this question as a bug report (failure under mono of a scenario that works properly under microsoft's CLR), but I don't have a sense for whether this question was noticed by the developers or whether they would be interested in addressing it and want more detailed information. Should I have posted this in a different forum?
- Mark Shirley

Greetings.

Sorry for the slight delay. We are currently working on the issues with Mono.

[mhshirley]

I'd like to follow up on this again, since it's been a while.
In the 4.0.7 release announcement, you point to this bug: https://bugzilla.xamarin.com/show_bug.cgi?id=25751
in mono's bugzilla. It looks to me as though that bug is specific to the sgen gc (cementing the spit nursery). I've tried my app under mono-boehm, and it also fails, although sometimes with System.IndexOutOfRangeException in ConcurrentDictionary. and sometimes with a vm crash. Perhaps the root cause is something else.

[k.dimitrov]

Greetings.

We have made very thorough tests and bypassed all of the code that might be "Mono specific" (runtime code generation) and yet, sometimes this crash occures under Mono. It seems that there is a problem with the Mono runtime that might occur when the runtime itself is under high loads.

[mhshirley]

Sorry. I wasn't clear. Perhaps the problem is a different mono bug than the one you've linked to above since there are crashes under the boehm collector as well as the generational collector.