Monthly Archives: June 2011

EventScavenger 4.2

I recently had the ‘pleasure’ of having my EventScavenger tool being used to track the security event logs of a couple of domain controllers for a ‘full’ enterprise company. It may be temporary for us to help a specific issue but in the process I had to re-look at how the collector service (the heart of EventScavenger) performs under really high stress. In case you don’t fully appreciate it, with full auditing for successful and failed events these event logs full up within an hour or so – and they have been set to something like 100MB+! Events literally flood into the logs at (sometimes) 10000 plus events in a single minute.

Now, I’ve never had any performance problems with the collectors until now. I’ve had a single collector running on a plain old pc gathering around 50 logs or more running 24×7 for 4-5 years. In all those cases each individual event log never have more than 20MB of data in them and almost never full of data that gets overwritten within an hour or so. The problem with the domain controller event logs are the sheer volume of events written ‘all the time’ plus the collector is running on a separate pc with the sql database on a remote system that is shared with dozen of other systems. The end result is that events retrieved from the logs cannot (1) be read quickly enough and (2) written quickly enough to the database. The first problem is not something I can do anything about now – other than having a collector installed on each of the domain controllers. This is not feasible as it won’t be allowed and they are Windows Core machines anyway.

To address the second issue I did some research into how to submit batches of records to the database since doing it one by one just isn’t fast enough. I know about the new table variable in newer versions of sql server but unfortunately the server that was provided is only sql 2005 which does not support that. Fortunately it does support the xml data type even as a parameter. I found an article ‘Sending Multiple Rows to Database for Modification‘ on codeproject that specifically deals with sql 2005 that describes 2 ways to accomplish this: using a delimited string or the xml data type. The delimited option won’t work for me as it is limited to about 4000 characters and the amount of data I deal with is orders of magnitude more than that. The xml data type allows for up to 2GB in a single shot!

I did quite a few performance tests and there is a real increase in number of records that can be processed using batches – despite the possible overhead of first converting the rows to xml and then sql server converting it back to a table (in memory). Interestingly, increasing the batch size beyond a certain point (1000 as I tested) does not increase the overall throughput. It more or less stay linear. The problem with that is you increase the chance that a single failure caused by one row can make the whole batch fail. So the best is to have a batch size that is big enough to warrant the benefit but not too big so it can cause failures.

Example code

A basic example of the sql xml handling:

Declare @xml  XML
SET @xml = N’
<rs>
<r cn=”machineName” l=”logName” etc />

</rs>’
SELECT
T.Row.value(‘@cn’, ‘VARCHAR(255)’) MachineName,
T.Row.value(‘@l’, ‘varchar(255)’) LogName
FROM   @xml.nodes(‘/rs/r’) AS T(Row)

To generate the xml on client side (simplified code):

System.IO.MemoryStream outstream = new System.IO.MemoryStream();
using (System.Xml.XmlTextWriter wrt = new System.Xml.XmlTextWriter(outstream, Encoding.UTF8))
{

wrt.Formatting = System.Xml.Formatting.Indented;
wrt.Indentation = 1;
wrt.IndentChar = ‘ ‘;
wrt.WriteStartDocument();
wrt.WriteStartElement(“rs”);

foreach (EventLogEntry ele in eventEntries)
{

wrt.WriteStartElement(“r”);
wrt.WriteAttributeString(“cn”, machine);
wrt.WriteAttributeString(“l”, logName);


wrt.WriteEndElement(); //r

}

wrt.WriteEndElement(); //rs
wrt.WriteEndDocument();
wrt.Flush();

outstream.Position = 0;
InsertEventLogEntries(outstream);

}

public InsertEventLogEntries(System.IO.Stream xml)
{

string sql = “InsertEventLogEntries”;
System.Data.SqlTypes.SqlXml sxl = new System.Data.SqlTypes.SqlXml(xml);
SqlParameter[] parms = new SqlParameter[] {
new SqlParameter(“@xml”, SqlDbType.Xml) { Value = sxl}
};
using (SqlCommand cmnd = new SqlCommand(sql, insertConn))
{
cmnd.CommandType = CommandType.StoredProcedure;
cmnd.CommandTimeout = CommandTimeout;
cmnd.Parameters.AddRange(parms);
try
{

cmnd.ExecuteNoQuery();

}
catch { throw; }

}

So with this change it is possible to handle larger sets of inserts into the database. Now only if there was a way to make reading an event log faster…

The future of development on Windows

Thanks to a link on Slashdot (yes, I read that too) I came across an article on Arstechnica about the (possible) future of development on the Windows platform in the future.

After the initial announcement that Microsoft is changing the fundamental way Windows 8 and writing custom applications they went into a ‘information freeze’ and are not giving more info on the subject. The reaction of several bloggers and the speculation that follows isn’t making things better. The Html5 and JavaScript message may not be the all and everything.

However, from the leaked (or released) testing versions so far it seems the situation is a bit more complex but perhaps not as bad as some developers are fearing. The author of the article goes on explaining the history around how longhorn was planned but ultimately dropped to support only existing (at that time) technologies. It can only be that the internal politics inside Microsoft causes so much turmoil and the ‘old’ side won. A lot of newly planned technologies got dropped in the process and the result was Windows XP without all those features promised (like WinFX etc). Then that Windows XP and the hackers story started and further derailed any possibility of new technology getting into the core platform. Even now with Windows 7 the .Net framework and related technologies are really only an ‘add-on’ and not used by the core of the OS. This is (according to the article) where Windows 8 is going to change things again – big time!

If the article proof to be accurate then there may be some hope for Microsoft still. The problem is just that old enemy – themselves, as internal fighting against change might make things more difficult. However, because of the changes in the market recently (last few years) Microsoft might not have the luxury anymore to just play around and ignore the internal politics. They have been surpassed by several competitors (yes, not just Apple) and survival is not guaranteed anymore. Any bugger-ups now can actually cause the company to go the way of the dodo. They better get it right. Perhaps with this thread hanging over their heads they are taking things more seriously and that is why they are not talking openly anymore about internal changes. The only next planned release of a build of the new Windows is scheduled around September so we’ll have to wait and see if/how things have changed.

In the mean while it might be a good thing to continue learn how to use XAML and WPF… (even though WPF is giving me headaches 😉 )

Microsoft vs Apple – stock prices compared over time

I came across some interesting information comparing the history of Microsoft and Apple’s stock prices during the years. It contains a number of dates on a scale with major events or milestones for both companies plus what their stock prices were at the time. Apple might have struggled in the beginning but recently they have been soaring to extreme levels while Microsoft’s has been static for most of the last 10 years – despite some good products that were released in those times (Windows 7, Kinect etc.)

I’m not going to draw conclusions about the information – you can judge for yourself. If you are working from Microsoft then you’d probably be a bit worried though.

Read more »

Embedding a Form inside another control

This is something I was thinking about at the beginning of my TDI (Tabbed document interface) ‘adventure’ but I didn’t think it would be easy or possible back then. Then I came across a custom tab control (A highly configurable MDI tab control from scratch) that actually allowed you to use plain old WinForms to add to it. The control itself was written in VB and ultimately I abandoned using it but I was intrigued with how it could use a plain WinForm. It actually has some real advantages like.

  • A form you can design easier in the IDE (than a user control).
  • A form can be tested/used separately on it own without being embedded onto something else first.
  • And I had a blank thinking of more reasons…

If you try to just add a Form to another container – like a panel, tabpage etc. and try to run it you would get the error: ‘Top-level control cannot be added to a control.‘. This happens because Forms are marked as Top Level controls. Fortunately, there is a simple way around it. You need to mark the form as not being a ‘Top level control’ like this: (assuming you have a panel called panel1)

Form f = new Form();
f.TopLevel = false;
panel1.Controls.Add(f);
f.Show(); //this is needed to actually have the form display itself inside the container

That will make the form load inside the panel without run-time errors – but it would look ugly if you just leave it like that. To properly ’embed’ it you need to add some more code:

Form f = new Form();
f.TopLevel = false;
f.MdiParent = null;
f.FormBorderStyle = System.Windows.Forms.FormBorderStyle.None;
f.Dock = DockStyle.Fill;
panel1.Controls.Add(f);
f.Show();

So, there you have it!

Form2 f = new Form2();
            f.TopLevel = false;

Tab control

I’ve come across a tab control that I can customize for my needs in creating a MDI window replacement. It is based on the KRBTabControl article from Karub Yaz?l?m on CodeProject.

I’ve added some built-in context menu that adds a few options for closing a tab, closing all tabs, closing all other tabs and view a list of the available tabs from a modal dialog window. The rest is pretty much the original control except for some enums that I moved around.

My version can be found here.

* Update *

I’ve added an event to the control to fire when a tab page get removed. The code to achieve this looks like this:

public event TabPageRemovedDelegate TabPageRemoved;
private void RaiseTabPageRemoved(TabPageEx tab)
{

if (TabPageRemoved != null)

TabPageRemoved(tab);

}

The ‘RaiseTabPageRemoved’ method then gets called from the main OnMouseClick plus my new context menu events to close tabs.

User interfaces

The following is just some ideas/thoughts/comments on user interfaces.

There are lots of things that can and has been said about computer user interfaces. Ever since the first computers the way a person interact with a computer has been changing – but very slowly. If you go by popular science fiction then we should be talking and even mind controlling computers by now but I don’t think we’re quite there yet, if ever.

The last 2 decades or so most development into computer user interfaces has revolved around the Graphical User Interface (GUI) and the keyboard and the mouse. Now and then some different way to interact with the GUI has come along like touch screens and even the failed light pen. Voice input/output has been tried but this is generally the exception. I can’t help thinking about Star Trek when I think about interacting with a computer by voice… ‘Computer, tuna mayo sandwich please!” – of course you need a replicator as well 🙂 In those scenarios there (usually) were no real GUI to talk about as well. Ah, we can only dream…

The point it that most of these interfaces still use some computer screen with text and pictures (graphics) to display what the computer ‘is thinking about’. Most of that is also limited to 2D interfacing – with the exception of games where you ‘play’ a character moving around a virtual 3D world. I have seen attempts of creating a generic 3D GUI but it was ‘still’ displayed on a 2D monitor (with keyboard and mice as user input). Even today with more and more devices that uses ‘touch’ like in touchscreen we’re still stuck in a 2D  world. It turns out processing voice and sound as input is very ‘computing heavy’. Outputting sound/graphics is of course very easy but that is generally a ‘dumb’ or scripted way of dealing with a computer. Any tape recorder could do that – like what they did in the 60’s/70’s. However, if you want to interact with a computer the input part is also, if not the most, important.

To my knowledge, even the best speech recognition software today still take a lengthy time to be ‘trained’ to (fairly) accurately accept some voice commands. Even then it is usually only from one person. If another guy comes around the whole story has to be done from scratch (almost). Also, most ‘applications’ don’t support that way of user interfacing anyway. You really have to admire our human brains for doing these things without all that effort! We are much better at interfacing verbally and visually with each other. It’s a shame that our machines that we create cannot do anything similarly (by that I mean to do these things so easily).

The same I suppose can be said about ‘graphical’ input but at least it seems things in that department is changing a bit. With Microsoft’s Kinect technology it is becoming easier to interact with a computer (mostly graphically). It is a shame that they first only released it for a gaming console (despite that they claim it is more than a gaming platform). At least now with the new SDK we can start using it for a PC as well. Unfortunately I don’t have one – and the wife won’t allow it soon for financial reasons.

In the mean while I’m also still stuck in the 2D interfacing world, writing programs that still use the keyboard and mouse. But it isn’t all bad really. This way has served us for many years and probably will for many more. ‘If it isn’t broken don’t fix it’. Still, I can dream.

 

More comments about the Duke

Lets face it. There are people and there are people. Some moan just because they can, other because they have nothing better to do… Since the launch of Duke Nukem Forever lots of people have complaint about lots of stuff (about the game) but so what? If you really don’t like it then, do the world a favor and don’t play the freaking game! You’re just spoiling it for those of us that want to try and enjoy it. Sure, it is not a piece of art, nor the best graphics, best game, best whatevar, it is just plain old Duke having fun with some aliens (and so on).

Must say, the ‘ending’ is a bit ‘different’ and it makes you wonder whether there will be a sequel or not. Some of the themes and quotes in the game might be offensive to some people but that goes with the territory. Personally, I hope they make a follow up game that is just as ‘bad/good’ without adding to much modern crap or political correctness. It just wouldn’t be Duke otherwise!

Hail to the King baby!

Windows 8 and compatibility

Yesterday I saw an article on Ars technica about how Microsoft is planning to give Windows 8 a totally new API (Application Programming Interface) that will break compatibility with most existing and past development tools – like Visual Studio. If this proof to be true then I think Microsoft has just signed their own death warrant (from an individual developer point of view). So much for having a usable Operating System (from a developmental view). All the experience we ‘developers’ gained the last decade or so will be basically useless (except HTML and JavaScript).

In a way I can understand why Microsoft would be doing such a thing. They are desperately trying to play catch-up in the market and needs to do more and more radical things just to stay relevant and alive . They have never been strong in creating a development technology and sticking with it. There is too much internal politics and infighting going on. The result is that they are failing faster and faster each time. But don’t count them out yet – Apple was once in that position and jumped back into life. For Microsoft’s sake lets hope they can also do it. Perhaps they must use their billions while they still have some to buy Steve Jobs 😉 – or ask uncle Bill to come out of retirement.

I must admit I really like C# so it would be a sad day if I have to hang up my C# ‘gloves’, so to speak.
Well, if I have to abandon the development knowledge gained the last decade I would rather choose something that will not limit me to Windows and Microsoft only. Ironically it will probably turn out to be something in the line of HTML and JavaScript. Perhaps Microsoft has a good idea there but it won’t help them keeping the skills in ‘their court’ since these are open standard technologies. In the next couple of years things could get really interesting (or chaotic…)

Multilingual applications

Living in the good old S of A it is natural for us to be bilingual or even multilingual. So what do we do if we have to create software that can/must be used by people of multiple language backgrounds? Traditionally Microsoft had the concept of resource files that you can use to create multiple versions of the same application for different languages or locales. Problem with this is once the executable has been compiled it is also fixed. What happens if you want the application to be multilingual at ‘run-time’?

Well, I’ve had to do something like this for myself. I first looked at resource files but it just did not solve the whole issue on its own. So I started creating something for myself that works for my application. Part of the solution still use .Net resource files for storing xml data that gets use in the language look up procedure. The idea is to load the text of controls and windows at the time the window open since control and window text get loaded at run-time anyway. The following is an explanation of the classes I created to facilitate my solution. It is a fully working solution and already in use inside a proper application. There may be other and even better or smarter solutions ‘out there’ but this one works for me.

LanguageHandler

This is the central class that handles all the language setting/reading etc. functionality. It encapsulate the list of ‘phrases’ used in the application and provides methods to retrieve words or phrases depending on a look-up key. Simply translating words alone is not good enough so the focus is on whole ‘phrases’.

The following is a partial view of the class without methods:

public static class LanguageHandler
{

private static List<Phrase> phraseList = new List<Phrase>();

#region Properties
private static string languageID = “en-za”;
public static Language AppLanguage {

get {

switch (languageID) {
case “af”:
return Language.Afrikaans;
case “en-za”:
return Language.English;
default:
return Language.Afrikaans; }

}
set {

switch (value) {
case Language.Afrikaans:
languageID = “af”;
break;
case Language.English:
languageID = “en-za”;
break;
default:
languageID = “en-za”;
break; }

}

public static string PhraseSource { set; get; }
#endregion

}

PhraseList.xml

The definition of the phrases can probably be stored in any data form but I choose a plain simple xml file to host the raw phrase data. This make it easy to maintain and port. It is also included into the application as a file resource although theoretically it could be located anywhere as long as the application can reach it.

The structure is also very basic as it is simply a serialized version of List<Phrase>

<?xml version=”1.0″ encoding=”utf-16″?>
<ArrayOfPhrase xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xmlns:xsd=”http://www.w3.org/2001/XMLSchema”>
<phrase id=”someId” default=”Some default”>

<phraseEntry lang=”af”>Afrikaanse waarde</phraseEntry>
<phraseEntry lang=”en-za”>English value</phraseEntry>

</phrase>
<phrase id=”someId2″ default=”Some default 2″>

</ArrayOfPhrase>

LoadPhrases method

Loading the data is as simple as deserializing the data. I have a helper class (not shown) to simply take the xml file and deserialize it:

#region LoadPhrases
public static void LoadPhrases()
{

if ((PhraseSource != null) && (PhraseSource.Length > 0))

phraseList = SerializationUtils.DeserializeXML<List<Phrase>>(PhraseSource);

}
#endregion

GetLanguagePhrase method

The main (and only relevant) method for retrieving stuff from the class is GetLanguagePhrase. It simply takes one parameter – the id value of the phrase required.

#region GetLanguagePhrase
public static string GetLanguagePhrase(string languagePhraseID)
{

if (phraseList.Count == 0)

LoadPhrases();

Phrase phrase = (from p in phraseList where p.LanguagePhraseID == languagePhraseID select p).FirstOrDefault();
if (phrase != null)
{

PhraseEntry phraseEntry = (from pe in phrase.Phrases where pe.LangId == languageID select pe).FirstOrDefault();
if (phraseEntry != null)

return phraseEntry.Text;

else

return phrase.DefaultValue;

}
throw new Exception(“Undefined language phrase ID”);

}
#endregion

SetControlTextAndTip methods

To make it a bit easier to set the text (and tooltips if available) of controls (my implementation use WinForms) I created a few helper methods to automatically set the text on the control based on a phrase id specified.

#region SetControlText
public static void SetControlTextAndTip(string languagePhraseID, params ToolStripItem[] tsis)
{

string text = GetLanguagePhrase(languagePhraseID);
foreach (ToolStripItem tsi in tsis)
{

tsi.Text = text;
tsi.ToolTipText = text;

}

}
public static void SetControlTextAndTip(string languagePhraseID, params Control[] crtls)
{

string text = GetLanguagePhrase(languagePhraseID);
foreach (Control crtl in crtls)
{

crtl.Text = text;

}

}
public static void SetControlTextAndTip(string languagePhraseID, params ColumnHeader[] columnHeaders)
{

string text = GetLanguagePhrase(languagePhraseID);
foreach (ColumnHeader ch in columnHeaders)
{

ch.Text = text;

}

}
#endregion

The Phrase and PhraseEntry classes

These two classes are really simple. For the purpose of this article they could be a lot simpler but since I built a separate ‘editor’ they have a bunch of attributes to make editing easier. This editor plus a few method inside LanguageHandler will have to wait for another article.

[Serializable, XmlType(“phrase”)]
public class Phrase : IComparable
{

[XmlAttribute(“id”),
Browsable(true),
CategoryAttribute(“Identifier”),
DefaultValueAttribute(“”),
DescriptionAttribute(“Phrase ID”),
ReadOnly(true)]
public string LanguagePhraseID { get; set; }
[XmlElement(“phraseEntry”),
Browsable(true),
CategoryAttribute(“Value details”),
DefaultValueAttribute(“”),
DescriptionAttribute(“Phrase entries”)]
public List<PhraseEntry> Phrases { get; set; }
[XmlAttribute(“default”),
Browsable(true),
CategoryAttribute(“Value details”),
DefaultValueAttribute(“”),
Editor(typeof(MultilineStringEditor), typeof(UITypeEditor)),
DescriptionAttribute(“Default value. Use ‘\\r\\n’ to indicate crlf.”)]
public string DefaultValue { get; set; }

#region IComparable Members
public int CompareTo(object obj)
{

Phrase otherPhrase = (Phrase)obj;
return LanguagePhraseID.CompareTo(otherPhrase.LanguagePhraseID);

}
#endregion

}

[Serializable, XmlType(“entry”), TypeConverter(typeof(PhraseEntryConverter))]
public class PhraseEntry
{

public PhraseEntry() { }
public PhraseEntry(string phraseEntryDef)
{

if (phraseEntryDef.Contains(“:”))
{

LangId = phraseEntryDef.Substring(0, phraseEntryDef.IndexOf(“:”));
Text = phraseEntryDef.Substring(phraseEntryDef.IndexOf(“:”) + 1);

}

}

[XmlAttribute(“lang”),
Browsable(true),
CategoryAttribute(“Phrase entry”),
DefaultValueAttribute(“”),
DescriptionAttribute(“Language”)]
public string LangId { get; set; }
[XmlText(),
Browsable(true),
CategoryAttribute(“Phrase entry”),
DefaultValueAttribute(“”),
Editor(typeof(MultilineStringEditor), typeof(UITypeEditor)),
DescriptionAttribute(“Text”)]
public string Text { get; set; }

public override string ToString()
{

return LangId + “:” + Text;

}

}

Using it all

Using LanguageHandler is really easy. Typically in a form’s onload event I call a private local method that sets all the texts of the controls that need to be set. There is no real performance issue to worry about since all the phrase data is stored in memory. One disadvantage is that normally the form need to be closed and reopened if the ‘language’ has change. This is not really a problem since changing the language is something that should not happen often. Technically this private LoadLanguageResources method could be ran any number of times whenever you want. The only place where I usually call it multiple times is on the ‘Options’ dialog where you choose the language being used.

If you have to use the same phrase multiple times you can always store the value in a local string variable.

The following is a short example of how to use it:

static class Program
{

/// <summary>
/// The main entry point for the application.
/// </summary>
[STAThread]
static void Main()
{


if (Properties.Settings.Default.LanguageID == “English”)

LanguageHandler.AppLanguage = Language.English;

else

LanguageHandler.AppLanguage = Language.Afrikaans;

LanguageHandler.PhraseSource = Properties.Resources.PhraseList;
LanguageHandler.LoadPhrases();

Form mainForm = new MainForm();

Application.Run(mainForm);

}

public partial class MainForm : Form
{

….

private void LoadLanguageResources()
{

Text = LanguageHandler.GetLanguagePhrase(“AppName”);
LanguageHandler.SetControlTextAndTip(“File”, fileToolStripMenuItem);
LanguageHandler.SetControlTextAndTip(“NewPerson”, newPersoonToolStripMenuItem);

}

private void MainForm_Load(object sender, EventArgs e)
{

LoadLanguageResources();

}

}

Summary

As you can see the solution is not really complicated. I really like simple solutions that ‘just works’. This one has proven to be working for my needs so far. I have over 340 different phrases, some are very long themselves like message box prompts.

How to make you computer talk at start-up

If you want to try some different.

http://microsoftfeed.com/2011/how-to-make-your-computer-talk-at-windows-7-startup/

Now you can set it up to greet you – ‘Hello Dave. I’m HAL 9000..” hehe