Tuesday, November 20, 2018

The Things They Carried, by Tim O'Brien

book cover I started reading The Things They Carried as a recommendation for writing style and it is, indeed, a very deep personal work. Tim O'Brien writes about the Vietnam war in most if not all of his work, but this novella is a collection of short stories all brought together under the mantle of a sort of a confession. It's a mosaic, each piece beautiful, but together creating the artistic vision of the true war.

I liked the subtlety, most of all. The characters are not overly complex, but they are portrayed in a very personal manner, with details that are important for the overall meaning of the book. I loved how O'Brien described soldiers going to war (instead of running away to Canada, as he almost did) because they were too embarrassed not to. Died in the war because they were afraid to die of shame. Too cowardly to run.

At just 150 pages, the book shows not how the training went, or how the shooting was, it presents everything from the viewpoint of the people there. How it takes over every feeling you have, how it changes you into this creature that is completely different from the man (or woman) who left. It's not about maneuvers or tactical prowess or strategies of survival. They are all meaningless. The important part is to keep a semblance of sanity.

The titled refers to the trinkets people carry to remind them of who they are. And they carry much more: hopes, wounds, fears, diseases, the ever growing arsenal of pointless weapons and ammunition and so on. A bit depressing, but a damn good read.

Friday, November 16, 2018

Adding logging to your objects without changing them (much)! Using MarshalByRefObject and RealProxy.

I was attempting to add very detailed logging (tracing) to my application. In order to do that, I had to change hundreds of objects. That wouldn't do. Fortunately, .NET has a nice feature called RealProxy. Let's see a quick and dirty example:
    class Program
    {
        static void Main(string[] args)
        {
            JsonConvert.DefaultSettings = () => new JsonSerializerSettings
            {
                ReferenceLoopHandling = ReferenceLoopHandling.Ignore,
                NullValueHandling = NullValueHandling.Ignore,
                Formatting = Formatting.Indented
            };

            Test test = new Test();
            test.field = "test";
            test.Property = "Test";
            string outObject;
            string refObject = "ref";
            var result = test.Method("test1", "test2", ref refObject, out outObject);
            Console.WriteLine(JsonConvert.SerializeObject(new object[] {
                    test,
                    result,
                    refObject,
                    outObject
                })
            );
            Console.ReadKey();
        }
 
    }
 
    class Test
    {
        public string field;
        public string Property { get; set; }
        public string Method(string parameter1, string parameter2, 
                                ref string refObject, out string outObject)
        {
            if (parameter1 == null)
                throw new ArgumentNullException("Parameter one cannot be null");
            refObject += " reffed";
            outObject = "outed";
            return $"{parameter1} : {parameter2}";
        }
    }

So I defined an object with a field, a property and a method. That's what my application does. It then displays the object and the result of the method call. Here is the result:
[
  {
    "field": "test",
    "Property": "Test"
  },
  "test1 : test2",
  "ref reffed",
  "outed"
]

I would like to log everything that happens with my Test object. Well, as such I can't do anything, I need to change the code a bit, like this:
    class Program
    {
        static void Main(string[] args)
        {
            JsonConvert.DefaultSettings = ()=>new JsonSerializerSettings
            {
                ReferenceLoopHandling = ReferenceLoopHandling.Ignore,
                NullValueHandling = NullValueHandling.Ignore,
                Formatting = Formatting.Indented
            };

            Test test = (Test)new LoggingProxy(new Test()).GetTransparentProxy();
            test.field = "test";
            test.Property = "Test";
            string outObject;
            string refObject = "ref";
            var result = test.Method("test1", "test2", ref refObject, out outObject);
            Console.WriteLine(JsonConvert.SerializeObject(new object[] {
                    test,
                    result,
                    refObject,
                    outObject
                })
            );
            Console.ReadKey();
        }
 
    }
 
    class Test: MarshalByRefObject
    {
        public string field;
        public string Property { get; set; }
        public string Method(string parameter1, string parameter2, 
                                ref string refObject, out string outObject)
        {
            if (parameter1 == null)
                throw new ArgumentNullException("Parameter one cannot be null");
            refObject += " reffed";
            outObject = "outed";
            return $"{parameter1} : {parameter2}";
        }
    }
 
    class LoggingProxy : RealProxy
    {
        private readonly object _target;
 
        public LoggingProxy(object obj) : base(obj?.GetType())
        {
            _target = obj;
        }
 
        public override IMessage Invoke(IMessage msg)
        {
            if (msg is IMethodCallMessage methodCall)
            {
                var arguments = methodCall.Args.ToArray();
                var result = methodCall.MethodBase.Invoke(_target, arguments);
                return new ReturnMessage(
                                result, 
                                arguments, 
                                arguments.Length, 
                                methodCall.LogicalCallContext, 
                                methodCall);
            }
            return null;
        }
    }

So this is what I did above:
  • I've inherited my Test class from MarshalByRefObject
  • I've created a LoggingProxy class that inherits from RealProxy and implements the Invoke method
  • I've replaced new Test(); with (Test)new LoggingProxy(new Test()).GetTransparentProxy();

Running it we get the same result.

Time to add some logging. I will write stuff on the Console, too, for this demo. Here are the changes to the LoggingProxy class:
    class LoggingProxy : RealProxy
    {
        private readonly object _target;
 
        public LoggingProxy(object obj) : base(obj?.GetType())
        {
            _target = obj;
        }
 
        public override IMessage Invoke(IMessage msg)
        {
            if (msg is IMethodCallMessage methodCall)
            {
                var arguments = methodCall.Args.ToArray();
                string typeName;
                try
                {
                    typeName = Type.GetType(methodCall.TypeName).Name;
                }
                catch
                {
                    typeName = methodCall.TypeName;
                }
                try
                {
                    Console.WriteLine($"Called {typeName}.{methodCall.MethodName}" +
                                              $"({JsonConvert.SerializeObject(arguments)})");
                    var result = methodCall.MethodBase.Invoke(_target, arguments);
                    Console.WriteLine($"Success for {typeName}.{methodCall.MethodName}" +
                                              $"({JsonConvert.SerializeObject(arguments)}): " +
                                              $"{JsonConvert.SerializeObject(result)}");
                    return new ReturnMessage(
                                    result,
                                    arguments,
                                    arguments.Length,
                                    methodCall.LogicalCallContext,
                                    methodCall);
                }
                catch (Exception exception)
                {
                    Console.WriteLine($"Error for {typeName}.{methodCall.MethodName}" +
                                              $"({JsonConvert.SerializeObject(arguments)}): " +
                                              $"{exception}");
                    return new ReturnMessage(exception, methodCall);
                }
            }
            return null;
        }
    }

It's the same as before, but with a try/catch block and some extra Console.WriteLines. Here is the output:
Called Object.FieldSetter([
  "RealProxyTests2.Test",
  "field",
  "test"
])
Success for Object.FieldSetter([
  "RealProxyTests2.Test",
  "field",
  "test"
]): null
Called Test.set_Property([
  "Test"
])
Success for Test.set_Property([
  "Test"
]): null
Called Test.Method([
  "test1",
  "test2",
  "ref",
  null
])
Success for Test.Method([
  "test1",
  "test2",
  "ref reffed",
  "outed"
]): "test1 : test2"
Called Object.GetType([])
Success for Object.GetType([]): "RealProxyTests2.Test, RealProxyTests2, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"
Called Object.FieldGetter([
  "RealProxyTests2.Test",
  "field",
  null
])
Success for Object.FieldGetter([
  "RealProxyTests2.Test",
  "field",
  "test"
]): null
Called Test.get_Property([])
Success for Test.get_Property([]): "Test"
[
  {
    "field": "test",
    "Property": "Test"
  },
  "test1 : test2",
  "ref reffed",
  "outed"
]

A bounty of information. The first lines are the setting of the field, Property and the execution of Method. But then there are a GetType, a field getter and a Property getter. What's that about? That's JsonConvert, serializing the Test proxy.
Warning: I've copied methodCall.Args into a local property called arguments. At first glance it might appear superfluous, but methodCall.Args is not really an array of object, even if it appears that way in the debugger. It is read-only, changing any of its items has no effect.

Remember how we defined the proxy in Program.Main? We can use a method in our proxy object, let's call it Wrap:
    public static T Wrap<T>(T obj)
        {
            if (obj is MarshalByRefObject marshalByRefObject)
            {
                return (T)new LoggingProxy(marshalByRefObject).GetTransparentProxy();
            }
            return obj;
        }

Conclusion:

While this works, there are a series of issues related to it:
  1. RealProxy is only available for .NET Framework, not .NET Core (see DispatchProxy for that)
  2. Serialization is not so straightforward as in this demo (see my blog post about Newtonsoft serialization
  3. You need to inherit from MarshalByRefObject, so this solution doesn't work for classes that already have a base class
  4. Performance wise you should make sure you are not logging or executing anything unless the correct log level is set (something like logger.LogTrace(JsonConvert(...)) would not work because the JSON serialization occurs no matter before executing LogTrace)
  5. Also, this wrapping is not free. With a simple proxy that did nothing than execute the code it took 43 times more time to run. Of course, that's because the actual execution of setting properties or returning a string is basically zero. When I added a Thread.Sleep(1) in the method, it took almost the same amount of time. Just don't use it in performance sensitive applications

Final thoughts: in the same namespace with RealProxy there is a ProxyAttribute class that at first glance seems to be even better: you just decorate a class with the attribute and BANG! instant AOP. But it's not that simple. First of all it works only on object that inherit from ContextBoundObject which itself inherit from MarshalByRefObject. And while it seems like a good idea to just replace MarshalByRefObject with ContextBoundObject in the code above, know that no generic class can inherit from it. There might be other restrictions, too. If you make Test inherit ContextBoundObject, the debugger will already show new Test() as being a transparent proxy, without wrapping it with any code. It might still be usable in certain conditions, though.

Thursday, November 15, 2018

Newtonsoft Json OutOfMemoryException

I was trying to log some stuff (a lot of stuff) and I noticed that my memory went to the roof (16GB in a few seconds), then an OutOfMemoryException was thrown. I've finally narrowed it down to the JSON serializer from Newtonsoft.

First of all, some introduction on how to serialize any object into JSON: whether you use JsonConvert.SerializeObject(yourObject,new JsonSerializerSettings { <settings> }) or new JsonSerializer { <settings> }.Serialize(writer, object) (where <settings> are some properties set via the object initializer syntax) you will need to consider these properties:
We will use these classes to test the results:
class Parent
{
    public string Name { get; set; }
    public Child Child1 { get; set; }
    public Child Child2 { get; set; }
    public Child[] Children { get; set; }
    public Parent Self { get; internal set; }
}
 
class Child
{
    public string Name { get; set; }
    public Parent Parent { get; internal set; }
}

For this piece of code:
JsonConvert.SerializeObject(new Child { Name = "other child" }, settings)
you will get either
{"Name":"other child","Parent":null}
or
{
  "Name": "other child",
  "Parent": null
}
based on whether we use Formatting.None or Formatting.Indented. The other properties do not affect the serialization (yet).

Let's set up the following objects:
var child = new Child
{
    Name = "Child name"
};
var parent = new Parent
{
    Name = "Parent name",
    Child1 = child,
    Child2 = child,
    Children = new[]
    {
        child, new Child { Name = "other child"}
    }
};
parent.Self = parent;
child.Parent = parent;

As you can see, not only does parent have multiple references to child and one to himself, but the child also references the parent. If we try the code
JsonConvert.SerializeObject(parent, settings)
we will get an exception Newtonsoft.Json.JsonSerializationException: 'Self referencing loop detected for property 'Parent' with type 'Parent'. Path 'Child1'.'. In order to avoid this, we can use ReferenceLoopHandling.Ignore, which tells the serializer to ignore circular references. Here is the output when using
var settings = new JsonSerializerSettings
{
    Formatting = Formatting.Indented,
    ReferenceLoopHandling=ReferenceLoopHandling.Ignore
};

{
  "Name": "Parent name",
  "Child1": {
    "Name": "Child name"
  },
  "Child2": {
    "Name": "Child name"
  },
  "Children": [
    {
      "Name": "Child name"
    },
    {
      "Name": "other child",
      "Parent": null
    }
  ]
}

If we add NullValueHandling.Ignore we get
{
  "Name": "Parent name",
  "Child1": {
    "Name": "Child name"
  },
  "Child2": {
    "Name": "Child name"
  },
  "Children": [
    {
      "Name": "Child name"
    },
    {
      "Name": "other child"
    }
  ]
} 
(the "Parent": null bit is now gone)

The default for the ReferenceLoopHandling property is ReferenceLoopHandling.Error, which throws the serialization exception above, but we can also use ReferenceLoopHandling.Serialize besides Error and Ignore. In that case we get a System.StackOverflowException: 'Exception of type 'System.StackOverflowException' was thrown.' as it tries to serialize at infinitum.

PreserveReferencesHandling is rather interesting. It creates extra properties for objects like $id, $ref or $values and then uses those to define objects that are circularly referenced. Let's use this configuration:
var settings = new JsonSerializerSettings
{
    Formatting = Formatting.Indented,
    NullValueHandling = NullValueHandling.Ignore,
    PreserveReferencesHandling = PreserveReferencesHandling.Objects
};

Then the result will be
{
  "$id": "1",
  "Name": "Parent name",
  "Child1": {
    "$id": "2",
    "Name": "Child name",
    "Parent": {
      "$ref": "1"
    }
  },
  "Child2": {
    "$ref": "2"
  },
  "Children": [
    {
      "$ref": "2"
    },
    {
      "$id": "3",
      "Name": "other child"
    }
  ],
  "Self": {
    "$ref": "1"
  }
}

Let's try PreserveReferencesHandling.Arrays:
var settings = new JsonSerializerSettings
{
    Formatting = Formatting.Indented,
    NullValueHandling = NullValueHandling.Ignore,
    ReferenceLoopHandling=ReferenceLoopHandling.Ignore,
    PreserveReferencesHandling = PreserveReferencesHandling.Arrays
};

The result will then be
{
  "Name": "Parent name",
  "Child1": {
    "Name": "Child name"
  },
  "Child2": {
    "Name": "Child name"
  },
  "Children": {
    "$id": "1",
    "$values": [
      {
        "Name": "Child name"
      },
      {
        "Name": "other child"
      }
    ]
  }
}
which annoyingly adds an $id to the Children array. There is one more possible value, PreserveReferencesHandling.All, which causes this output:
{
  "$id": "1",
  "Name": "Parent name",
  "Child1": {
    "$id": "2",
    "Name": "Child name",
    "Parent": {
      "$ref": "1"
    }
  },
  "Child2": {
    "$ref": "2"
  },
  "Children": {
    "$id": "3",
    "$values": [
      {
        "$ref": "2"
      },
      {
        "$id": "4",
        "Name": "other child"
      }
    ]
  },
  "Self": {
    "$ref": "1"
  }
}

I personally recommend using PreserveReferencesHandling.Objects, which doesn't need setting the ReferenceLoopHandling property at all. Unfortunately, it adds an $id to every object, even if it is not circularly defined. However, it creates an object that can be safely deserialized back into the original, but if you just want a quick and dirty output of the data in an object, use ReferenceLoopHandling.Ignore with NullValueHandling.Ignore. Note that object references cannot be preserved when a value is set via a non-default constructor such as types that implement ISerializable.

Warning, though, this is still not enough! In my logging code I had used ReferenceLoopHandling.Ignore and the exception was quite different, an OutOfMemoryException. It seems that even with circular references checked, JsonSerializer will messes up some times.

The culprits? Task<T> (or async lambdas send as parameters) and an Entity Framework context object. The solution I employed was to check the type of the objects I send to the serializer and, if any of the offending types, replace them with the full names of their types.

Hope it helps!

Monday, November 12, 2018

Wix (Windows Installer) changing the color of a control

This is a simple gotcha related to changing the color of a control. Let's say you have a label that you want to present in a different color. Normally you would do something like this:
<!-- I put it somewhere in Product.wxs -->
<TextStyle Id="WixUI_Font_Normal_Red" FaceName="Tahoma" Size="8" Red="255" Green="55" Blue="55" />
 
<!-- somewhere in your UI -->
<Control Id="LabelRed" Type="Text" X="62" Y="200" Width="270" Height="17" Property="MYPROPERTY">
     <Text>{\WixUI_Font_Normal_Red}!(loc.MYPROPERTY)</Text>
</Control>

Yet for some reason, it doesn't work when the control is a checkbox, for example. The simple explanation is that this is by design: only text controls can change color. The solution is to split your control into the edit control without a text, then add a text control next to it with the color you need.

Here is an example of a checkbox that changes the label color based on the check value:
        <Control Id="DoNotRunScriptsCheckbox" Type="CheckBox" X="45" Y="197" Height="17" Width="17" Property="DONOTRUNSCRIPTS" CheckBoxValue="1"/>
 
        <Control Id="DoNotRunScriptsLabel" Type="Text" X="62" Y="200" Width="270" Height="17" CheckBoxPropertyRef="DONOTRUNSCRIPTS">
          <Text>!(loc.DoNotRunScriptsDescription)</Text>
          <Condition Action="hide"><![CDATA[DONOTRUNSCRIPTS]]></Condition>
          <Condition Action="show"><![CDATA[NOT DONOTRUNSCRIPTS]]></Condition>
        </Control>
 
        <Control Id="DoNotRunScriptsLabelRed" Type="Text" X="62" Y="200" Width="270" Height="17" CheckBoxPropertyRef="DONOTRUNSCRIPTS">
          <Text>{\WixUI_Font_Normal_Red}!(loc.DoNotRunScriptsDescription)</Text>
          <Condition Action="hide"><![CDATA[NOT DONOTRUNSCRIPTS]]></Condition>
          <Condition Action="show"><![CDATA[DONOTRUNSCRIPTS]]></Condition>
        </Control>

Wix Setup (Windows Installer) saving user input values

So you have one of those annoyingly XMLish setups from Windows Installer and you want to preserve the values you input so they are prefilled at future upgrades. There are a lot of articles on the Internet on how to do this, but all of them seem to be missing something. I am sure this one will too, but it worked for me.

Let's start with a basic setup.
<?xml version="1.0" encoding="UTF-8"?>
<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
  <Fragment>
    <UI Id="DatabaseAuthenticationDialogUI">
      <Property Id="DATABASEDOMAIN" Secure="yes"/>
      <Dialog
        Id="DatabaseAuthenticationDialog"
        Width="370"
        Height="270"
        Title="[ProductName] database authentication"
        NoMinimize="yes">
        <Control Id="DatabaseDomainLabel" Type="Text" X="45" Y="110" Width="100" Height="15" TabSkip="no" Text="!(loc.Domain):" />
        <Control Id="DatabaseDomainEdit" Type="Edit" X="45" Y="122" Width="220" Height="18" Property="DATABASEDOMAIN" Text="{80}"/>

So this is a database authentication dialog, with only the relevant lines in it. We have a property defined as DATABASEDOMAIN and then an edit control that edits this property. Ideally, we would want to make sure this property is being saved somewhere at the end of the install and it is retrieved before the install to be populated. To do this we will first define a DATABASEDOMAIN_SAVED property and load/save it in the registry, then link it with DATABASEDOMAIN.

First, there is the issue of where to put this code. Personally, I put them all under Product, as a separate mechanism for preserving and loading values. I am sure there are other places in your XML files where you can do it. Here is how my Product.wxs code looks like (just relevant lines):
<?xml version="1.0" encoding="UTF-8"?>
<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"
     xmlns:util="http://schemas.microsoft.com/wix/UtilExtension">
  <Product ... >
    <Property Id="SAVED_DATABASEDOMAIN" Secure="yes">
      <RegistrySearch Id="FindSavedDATABASEDOMAIN"
                      Root="HKLM"
                      Key="SOFTWARE\MyCompany\MyProduct"
                      Name="DatabaseDomain"
                      Type="raw" />
    </Property>
    <SetProperty Id="DATABASEDOMAIN" After="AppSearch" Value="[SAVED_DATABASEDOMAIN]" Sequence="ui">
      <![CDATA[SAVED_DATABASEDOMAIN]]>
    </SetProperty>
  </Product>
 
  <Fragment>
    <Directory Id="TARGETDIR" Name="SourceDir">
      <Directory Id="ProgramFiles64Folder">
        <Directory Id="MyFolder" Name="MyFolder">
          <Directory Id="INSTALLFOLDER" Name="MyProduct">
            <Component Id="InstallFolderComponent" Guid="c5ccddcc-8442-49e8-aa17-59f84feb4deb">
              <RegistryKey
                Root="HKLM"
                Key="SOFTWARE\MyCompany\MyProduct"
                >
                <RegistryValue Id="DatabaseDomain"
                               Action="write"
                               Type="string"
                               Name="DatabaseDomain"
                               Value="[DATABASEDOMAIN]" />
                </RegistryKey>
              <CreateFolder/>
            </Component>
          </Directory>
        </Directory>
      </Directory>
    </Directory>
  </Fragment>
</Wix>

This is what happens:
  1. We search in the registry and set the value for DATABASEDOMAIN_SAVED.
  2. We set the DATABASEDOMAIN value from the DATABASEDOMAIN_SAVED value, if that is set. Note that Sequence is set to "ui". This is very important, as the default value is "both". In my case I spent hours to figure out why the values were written in the registry, but then would never change again. It was because there are two sequences: "ui" and "execute". The code would read the value from the registry, the user would then change the values, then, right before installing anything, the value would be read from the registry AGAIN and would overwrite the user input.
  3. Finally, when we install the product we save in the registry the value of DATABASEDOMAIN, whatever it is.

This should be it, but there are a few gotchas. One of them is checkboxes. For Windows Installer the value of a checkbox either is or isn't. It's not set to 0 or 1, true or false or anything like that. So if you save the value attached to an unchecked checkbox control, when read, even if empty, it will be set. Your checkbox will always be set from then on. The solution I used was adding a prefix, then setting the value for the checkbox only if that value is what I expect it to be. Here it is, in a gist:

    <!-- Product.wxs -->
    <!-- this doesn't change -->
    <Property Id="SAVED_DONOTRUNSCRIPTS" Secure="yes">
      <RegistrySearch Id="FindSavedDONOTRUNSCRIPTS"
                      Root="HKLM"
                      Key="SOFTWARE\MyCompany\MyProduct"
                      Name="DoNotRunScripts"
                      Type="raw" />
    </Property>
    <!-- here, however, I check for Val1 to set the value of the property to 1 -->
    <SetProperty Id="DONOTRUNSCRIPTS" After="AppSearch" Value="1" Sequence="ui">
      <![CDATA[SAVED_DONOTRUNSCRIPTS = "Val1"]]>
    </SetProperty>
    <!-- Note the Val prefix when saving the value -->
    <RegistryValue Id="DoNotRunScripts"
                   Action="write"
                   Type="string"
                   Name="DoNotRunScripts"
                   Value="Val[DONOTRUNSCRIPTS]" />
 
    <!-- DatabaseSetup.wxs -->
    <!-- Note the checkbox value -->
    <Control Id="DoNotRunScriptsCheckbox" Type="CheckBox" X="45" Y="197" Height="17" Width="17" Property="DONOTRUNSCRIPTS" CheckBoxValue="1"/>

I Hope that helps people.

Sunday, November 11, 2018

Artemis, by Andy Weir

book cover The Martian was a total surprise as it took the world by storm and became beloved by all science, tech and fast talking pun lovers in the world. Artemis, on the other hand, will not be. Andy Weir writes another book about a supersmart tech savvy character with a great sense of humor, this time solving technical problems on the Moon. Is it as good as The Martian? I don't know, because now I have a template and knew what to expect. Without the element of surprise, it's difficult to compare the two books. Yet I finished reading the novel in a day. That's gotta count for something.

Stylistically it is first person, filled with dialogue and action. It's incredibly easy to read. This time, the main character is a young woman and the location is a city on the Moon. Andy Weir is nothing if not optimistic, so in his book we do get there. I thought the action was following the pattern of danger-action-solution too closely, so much in fact that at one time I saw the entire story as a big adventure game. I will bet you lots of slugs that Weir loved Sierra games as much as the next geek (that being me).

Bottom line is that I liked the book, I gobbled it up and enjoyed it to the end. I didn't like Jazz Bashara as much as Mark Watney and, while the technological descriptions kept me interested, I still think that space mechanics and Martian farming trump EVA shenanigans and vacuum welding any time. It was still damn entertaining, though.

Saturday, November 10, 2018

Oryx and Crake (MaddAddam #1), by Margaret Atwood

book cover All in all I liked Oryx and Crake. Margaret Atwood is writing in a very technical manner, at least it felt like that to me, with scenes and characters constructed in a certain way. That is both instructional and a bit off putting, because once you become aware of it, it distracts from the plot. The plot itself is simple enough: the world has ended and there is this half crazed old man who is reminiscing over what happened. At first it is all unnervingly vague, with things being kept from the reader as the feelings and mental issues of the main (and single) character of the book are being declared. Then, when vagueness cannot be sustained anymore, it is all told as a linear story of what happened, with clear explanations of everything. No mystery left.

To me, it felt like Atwood wanted to warn about the way we treat our world and she just exaggerated some things while letting others be whatever she wanted them to be. From a scientific and sociological standpoint some of the things described are spot on, others are complete bogus. I would go into details, but I don't want to spoil the story. One example is the marketing brands that she cooked up and that are all (and I mean all) phonetically spelled descriptions or sometimes puns. All companies, all product names, all new animal names are like that and it gets really tiresome after a while. Also people are able to do wonderful (or horrible) new things in some areas and not advance at all in others. The timeline is a little screwed up, too.

My favorite quote, the one that made me think the most, was: "All it takes," said Crake, "is the elimination of one generation. One generation of anything. Beetles, trees, microbes, scientists, speakers of French, whatever. Break the link in time between one generation and the next, and it's game over forever" . He is referring to technology after a post apocalyptic event. There are few people that understand any instructions left by makers of advanced technology and the ones that can are ill equipped for survival.

The book does feel like it was written as standalone, with the possibility of being continued, which is a good thing: no ending cliffhangers and too much setting up the next books. The title is weirdly misleading, though, as both Oryx and Crake appear very little in the memories of the main character. Why was the book named so? It makes little sense. I am not sure if I will be reading the next books in the MaddAddam trilogy.

Bottom line: good, although a tad bland writing. A single character reminiscing is all that happens in the book. Grand ideas that most of the time make sense, but sometimes fall flat. Scientifically things don't go as she described.

Sunday, November 04, 2018

Analyzing memory leaks in .NET applications running in production

Intro


I had this situation where an application that was running in production was getting bigger and bigger until it would devour all memory. I had to fix the problem, but how could I identify the cause? Running it on my computer wouldn't lead to the same increase in memory, what I wanted to see was why a simple web API would take 6Gb in memory and keep growing.

The process is simple enough:
  1. Get a memory dump from the running process
  2. Analyze the memory dump

While this post relates specifically to process memory dumps, one can get whole memory dumps and analyze those. There are two types of process memory dump formats: mini and full. I will be talking about the full format.

Fortunately, for our process, there are only two steps. Unfortunately, both are fraught with problems. Let's go through each.

Getting a memory dump from a running process


Apparently that is as simple as opening Task Manager, right clicking on the running process in the Details tab and selecting Create dump file. However, what this does is suspend the process and then copy all of its memory in a file in %AppData%/Local/Temp/[name of process].dmp. If the process is being watched or some other issue occur, the dumping will fail. Personally, I couldn't dump anything over 2Gb when I was trying to get the info from w3wp.exe (the IIS process) as the process would restart.

There are a lot of other ways of getting a process memory dump, including crash dumps, periodic dumps, event driven dumps. ProcDump, one of the utilities in the free SysInternals suite, can do all of those. The Windows Error Reporting (WER) system also generates dumps for faulty or unresponsive application or kernel. Other options for collecting a dump are userdum.exep, Windows Debugger (ntsd or windbg). Process Explorer, of course, can create two types of memory dumps for a process: mini and full.

Analyzing a process memory dump


This is where it gets tricky. What you would like is something that opens the file and shows a nice interactive interface explaining exactly what hogs your memory. Visual Studio does that, but only the Ultimate version gives you the memory analysis option. dotMemory from JetBrains is great, but it also costs a lot of money.

My personal experience was using dotMemory in the five days in which it is free and it was as seamless as I would have wanted: I opened the file, it showed me a pie graph with what objects were hogging my memory, I clicked on them, looked where and how they were stored and even what paths could be taken to create those instances. It was beautiful.

Another option is SciTech's .NET Memory Profiler, but it is also rather expensive. I am sure RedGate has something, too, but their trials are way to intrusive for me to download.

I have been looking for free alternatives and frankly all I could find is WinDBG, which is included in Windows Debugging Tools.

Windows Debugger is the name of the software. It is a very basic tool, but there are visualizers which use it to show information. One of them is MemoScope, which appears unmaintained at the moment and is kind of weird looking, but it works. Unfortunately, it doesn't even come close to what dotMemory was showing me. If analyzing a random .NET app dump showed that the biggest memory usage was coming from MainWindow (makes sense) which then had a lvItems that held most of memory (it was a ListView with a lot of data), WinDbg and therefore MemoScope show the biggest usage comes from arrays of bytes. And it also makes sense, physically, but there is no logical context. A step by step memory profiling guide using WinDbg can be found here.

Conclusion


I am biased towards JetBrains software, which is usually amazing, although I haven't used it a quite a while because their obnoxious licensing system (you basically rent the software). I loved how quickly I got to the problem I had with dotMemory. .NET Memory Profiler from SciTech is way cheaper and seems to point in the right direction, but doesn't really compare in terms of quality. Anything else I've tried was quite subpar.


Doing it yourself


As they say, if you want to do something right, you gotta do it yourself.

There is a Windows API called MiniDumpWriteDump that you can use programatically to create the dumps. However, as you've seen, that's not the problem, the dump analysis is.

And it's hard to find anything that is not very deep and complicated. Memory dumps are usually analysed for forensic reasons. There are specialists who have written scores of books on memory dump analysis. What's worse, most people focus on crash dumps, with memory leak analysis usually done while debugging an application.

I certainly don't have the time to work on a tool to help me with this particular issue. I barely had the time to look for existing tools. But if you, guys, can add information to this blog post, I am sure it will help a lot of people going through the same thing.

Pretty Pictures


I didn't find a nice way of putting the images in the post's text, I am adding them here.








Friday, November 02, 2018

MemoryCache.Default suddenly not working

MemoryCache has a really strange behavior where all of a sudden, without any warning or exception, it stops caching. You use Set or Add, it doesn't matter. It returns true for Add, it throws no error for Set, but the cache stubbornly remains empty. This behavior can be replicated easily by running this line:
MemoryCache.Default.Dispose();
From that moment on, MemoryCache.Default will not cache anything anymore.

Of course, you will say, why the hell would you dispose the default object? First of all, why give me the option? It's not like I can set the Default instance after I dispose it, so this breaks a lot of things. Second of all, why pretend to continue to be working when in fact it is not? Most disposable classes throw exceptions if they are used after being disposed. And third of all, it might not be you who does the disposing.

In my case, I was trying to be a good boy and declare all my dependencies upfront. I was using StructureMap as a IoC framework and for my class that was using a MemoryCache I declared a constructor parameter of the basest type I could: ObjectCache. Then I registered MemoryCache.Default as the instance used when needing an ObjectCache. The problem is that each unit test would initialize and tear down the StructureMap container and it would take all the instances used that implement IDisposable and dispose them, including our friend MemoryCache.Default. As I couldn't find a simple way to tell StructureMap to not dispose the object, I had to create a MemoryCacheWrapper that implemented ObjectCache and was NOT IDisposable and was using MemoryCache.Default for all methods and use that as the singleton instance for an ObjectCache in StructureMap.

If you are working with a .NET version prior to .NET 4.5, you may encounter the same issue with a bug in .NET: MemoryCache Empty : Returns null after being set