If you are wondering why we bother struggling on with a 16-bit application when it would be much easier to support if ported to 32-bit then that makes two of us.
There are several reasons why this doesn't happen:
1. Cost/Benefit
Taking a moderately sized 16-bit VB and C code base and porting them to 32-bit would take a long time and be quite risky. The business cannot see a reason to sink a lot of developer-hours into an endeavour which - in their eyes - at best is going to result in an indistinguishable application and at worst will cost a huge amount of time and money. That's not mentioning the opportunity cost of having developers working on the port when they could be making beneficial modifications to the existing 16-bit code base. Management sees no value in the 32-bit port.
This sounds incredibly naive and dangerous considering if Microsoft removes 16-bit support from a future OS we will be in serious trouble and need to port to 32-bit or stop making money.
2. Third Party Components
The functionality of the application relies on some 3rd party components for UI controls, ZIP compression and other bits and pieces. The components are supplied as VBX files which are analogous to ActiveX controls in 32-bit land.
With some of the components we are supplied both 16-bit VBX's and 32-bit OCX's (ActiveX) which is a definite plus if we are going to do the port. For the rest of the controls we would be forced to either find 32-bit equivalents that are available and supported, or write our own from scratch. Either option immediately increases the risk involved. Writing our own ActiveX controls is outside the expertise of our staff and finding equivalent controls would require an amount of massaging of the existing code to get the new controls to fit properly.
3. Deprecated Product
Currently there is an effort to re-invent the product as an online solution in Flash/Flex rather than distributing the application on disc to our customers. The effort has been ongoing for a number of years now and is still yet to see a full scale commercial release.
We are hesitant to pour a lot of effort into porting the existing application to 32-bit when it will be retired some time in the near future. This has been the official company line for years although there is still active development on the product by a team of about six so my faith in the product being retired any time soon is low.
4. Tried It Before
At the request of the then lead developer then product was designated to be ported from 16-bit VB3 and VC++ 1.52 to the (then) ultra-modern VB6 and VC++ 6.0. Full liberty was taken with the structure of the VB6 code including liberal usage of classes and other advanced features.
Major headaches were encountered straight away when it was realised that there are subtle differences between how VB3 and VB6 utilise memory. For example, VB3's strings are ASCII with a 16-bit length prefix whereas VB6's strings are UTF-16 with a 32-bit length prefix. Considering we use a lot of Get and Put commands in VB to read and write data to and from disk, there is a lot of hassle involved in porting these modules as the formats are totally incompatible. From memory there is also something different about the Len statement between the two versions which made measuring structure sizes impossible.
When the port was more than three quarters done it was entered for initial QA and immediately raised red flags on the performance front. The 32-bit application was an order of magnitude slower than the 16-bit original largely due to VB6's terrible implementation of classes which brings the interpreter to a crawl.
After several rounds of performance enhancements the 32-bit version ended up several times slower than it's antique counterpart which was still completely unacceptable. Later on the project would be canned entirely in favour of the new web based version which has been written and rewritten six times now by various teams.
It turns out that it's not so elementary to replace a dinosaur. If it works, people will use it- whether it is written in VB3 or the latest and greatest web language du jour matters not to the end user. The web is definitely where we want to end up with the product but it is a hard sell when the user is used to a very responsive rich desktop application and is shown a slow, mostly mouse-based Flash/Flex application. Would you be happy with that upgrade?
Sunday, June 21, 2009
Visual C++ 1.52
A lot of the more complex functionality in our application is offloaded into 16-bit C/C++ DLLs which are authored with the last 16-bit Microsoft C++ IDE- Microsoft Visual C++ 1.52. The DLLs are mainly plain C for compatibility with VB3 and include VBAPI.H/LIB to allow manipulation of VB3 arrays, strings and variants. In some places we have used MFC but I try not to as it is just another dependency to cause problems.
In comparison with VB3, VC++ is lightning fast as it is compiled rather than interpreted. Similar to VB3, it has a fully fledged debugger with breakpoints, stack traversal and variable inspection.
Most of what we do is looking up information in huge (multi gigabyte is huge in the 16-bit world) data files. Some of these files have custom hand-written compression and/or "encrypted" with a simple ROT-style algorithm to avoid trivial data harvesting by curious users. Performing this decryption and decompression in VB3 code is out of the question as it is far too slow and in some cases completely impossible.
The one major trick I can share with you in VC++ 1.52 is that there are certain conditions under which the debugger will refuse to recognise a breakpoint. You have a line marked with a breakpoint and you absolutely know it has been executed but the IDE doesn't break into the code and acts as if the breakpoint doesn't exist. The solution is to force a software breakpoint in assembler (ASM) to trigger the debugger to break. In ASM, you issue software interrupt 3 and because C has inline assembler, it's a one-liner:
Because this is compiled into the DLL, this breakpoint will be hit every time the instruction is executed. Sometimes - such as in the guts of an inner loop - this is undesirable so you must make it conditional:
Apart from that, VC++ 1.52 is very solid and not too different from it's grandchild VC++ 6.0, which I have used extensively. The main difference is you are targeting the 16-bit Windows API, rather than the Win32 API. Most things that you take for granted when using the Win32 API are gone- all you can really achieve with the Windows API is synchronous file I/O, network I/O and registry modifications.
It is possible to use the Win32 API from 16-bit code using a process called thunking. This will be detailed in a post of it's own as it is rather complicated, at least in the way that we use it.
In comparison with VB3, VC++ is lightning fast as it is compiled rather than interpreted. Similar to VB3, it has a fully fledged debugger with breakpoints, stack traversal and variable inspection.
Most of what we do is looking up information in huge (multi gigabyte is huge in the 16-bit world) data files. Some of these files have custom hand-written compression and/or "encrypted" with a simple ROT-style algorithm to avoid trivial data harvesting by curious users. Performing this decryption and decompression in VB3 code is out of the question as it is far too slow and in some cases completely impossible.
The one major trick I can share with you in VC++ 1.52 is that there are certain conditions under which the debugger will refuse to recognise a breakpoint. You have a line marked with a breakpoint and you absolutely know it has been executed but the IDE doesn't break into the code and acts as if the breakpoint doesn't exist. The solution is to force a software breakpoint in assembler (ASM) to trigger the debugger to break. In ASM, you issue software interrupt 3 and because C has inline assembler, it's a one-liner:
asm int 3;Because this is compiled into the DLL, this breakpoint will be hit every time the instruction is executed. Sometimes - such as in the guts of an inner loop - this is undesirable so you must make it conditional:
if (strcmp(value, "expected value") == 0)I can't count the number of times that single instruction has saved my bacon. ASM FTW!
asm int 3;
Apart from that, VC++ 1.52 is very solid and not too different from it's grandchild VC++ 6.0, which I have used extensively. The main difference is you are targeting the 16-bit Windows API, rather than the Win32 API. Most things that you take for granted when using the Win32 API are gone- all you can really achieve with the Windows API is synchronous file I/O, network I/O and registry modifications.
It is possible to use the Win32 API from 16-bit code using a process called thunking. This will be detailed in a post of it's own as it is rather complicated, at least in the way that we use it.
Saturday, June 20, 2009
Visual Basic 3.0
There are many things to love and to hate about Visual Basic 3.0 (VB3). Released in 1993, this tool was revolutionary in its time, providing a WSIWYG form (dialog) editor when the next best thing was hand coding forms in C++. The fact that anyone with a mouse could now design a GUI application with basic logic made ranks of developers nervous that they would become obsolete.
Pros:
The VB3 IDE (for want of a better acronym) is replete with line-by-line debugging, breakpoints, variable inspection, stack traversal and a surprising number of modern features.
The form designer is very simple and really not that different from the UI designer in Visual Studio from recent years. Click and drag to create your controls, give them an ID then refer to them in code by that ID in event handler functions. Visual Basic 3 the language is quite feature-complete for a dinosaur. All of the mainstays are there- If/Else/ElseIf, Switch/Case, While loops and the infamous GoTo. It has garbage collection and dynamic typing so you don't need to worry too much about types or memory allocation if your application is moderately sized.
It is entirely imperative with no concept of user classes. As with C, classes can be approximated by creating a module and having all public functions take a user defined type as the first argument. If you really need classes then VB3's integration with 16-bit C DLLs is remarkable so if VB3 doesn't meet the performance requirements you can easily drop to C for a lightning fast binary search lookup or other required heavy lifting.
Cons:
There are two major limits we run into when maintaining a large VB3 code base. First is the limit on number of controls per form. On the main form of our application there are images, grids, a toolbar, menus and invisible controls such as timers.
Because (I assume) VB3 has an internal numeric identifier for a control and that ID must be an 8-bit number because there is a limit of 256 controls on any single form. This leads us to the ridiculous situation where if something is to be added to the main form, I must ask my manager which toolbar button or other control should be removed. He understands this is simply a limitation of the tool so usually prepares his change requests with what should be added and hence which controls should be removed.
The next major limitation is the number of global variables. Internally VB3 stores all global symbols (global variables and constants, public module functions, external function declarations) in a table of fixed size. I'm not sure how big this is but it must be around 1,000 entries. VB3 will let you know right away when you go over the limit as you will no longer be able to run the application in the IDE or compile it to EXE.
I can understand this limit but the thing that bugs me is that constants are included in the global symbol table. We have huge lists of constant integers that act as enumerations. For example, because we ship the product worldwide we have a list of natural languages that the application may be running in. That's about 30 global symbol table entries taken up right there. This forces us to eliminate global constants and replace them with literal values (magic numbers) in the code that consumes them. This is just asking for trouble but it's the only way we can squeeze in new event handlers and global functions. This is very common:
Also, because of this limit we are forced to combine functions where possible to keep the global symbol table entries down. This leads to some very confusing code and breaks down the idea of having modules as classes. Spaghetti code, here we come!
Event handling is done by creating a function in the form's module named controlName_eventName() and all of the plumbing is done behind the scenes for you. Because it is so easy to learn the language and the event handlers are such a natural place to put code, many VB3 application are spaghetti with no real separation of UI code from business logic or data access. With discipline you can create a two or three tiered application using VB3 but it really does tempt you to break the rules and put in filthy hacks.
Visually, in a VB3 application your GUI controls are limited to those available in Windows 3.11 and makes your application stick out like a sore thumb when running on a more recent version. For a legacy application I guess this doesn't really matter. Plenty of libraries and government departments are still running character mode DOS applications but these days it's via terminal software running on XP or Vista. Users that have to use an old application daily don't have the luxury of complaining about it- if it does the job then nobody cares what it looks like.
One of my pet peeves with VB3 is that the project file doesn't keep track of the breakpoints introduced into code. This means that you can be in the middle of an investigation into strange behaviour carefully setting breakpoints and observing variable values when for no reason the IDE will GPF (crash) and everything's gone. Because the IDE crashes reasonably frequently - several times a day if I'm using it full time - you end up with a Ctrl-S hair trigger. Every line of code I write is followed by a save to make sure it isn't lost. I have seen several hours work simply disappear due to an IDE crash and vowed to never let it happen again.
Pros:
The VB3 IDE (for want of a better acronym) is replete with line-by-line debugging, breakpoints, variable inspection, stack traversal and a surprising number of modern features.
The form designer is very simple and really not that different from the UI designer in Visual Studio from recent years. Click and drag to create your controls, give them an ID then refer to them in code by that ID in event handler functions. Visual Basic 3 the language is quite feature-complete for a dinosaur. All of the mainstays are there- If/Else/ElseIf, Switch/Case, While loops and the infamous GoTo. It has garbage collection and dynamic typing so you don't need to worry too much about types or memory allocation if your application is moderately sized.
It is entirely imperative with no concept of user classes. As with C, classes can be approximated by creating a module and having all public functions take a user defined type as the first argument. If you really need classes then VB3's integration with 16-bit C DLLs is remarkable so if VB3 doesn't meet the performance requirements you can easily drop to C for a lightning fast binary search lookup or other required heavy lifting.
Cons:
There are two major limits we run into when maintaining a large VB3 code base. First is the limit on number of controls per form. On the main form of our application there are images, grids, a toolbar, menus and invisible controls such as timers.
Because (I assume) VB3 has an internal numeric identifier for a control and that ID must be an 8-bit number because there is a limit of 256 controls on any single form. This leads us to the ridiculous situation where if something is to be added to the main form, I must ask my manager which toolbar button or other control should be removed. He understands this is simply a limitation of the tool so usually prepares his change requests with what should be added and hence which controls should be removed.
The next major limitation is the number of global variables. Internally VB3 stores all global symbols (global variables and constants, public module functions, external function declarations) in a table of fixed size. I'm not sure how big this is but it must be around 1,000 entries. VB3 will let you know right away when you go over the limit as you will no longer be able to run the application in the IDE or compile it to EXE.
I can understand this limit but the thing that bugs me is that constants are included in the global symbol table. We have huge lists of constant integers that act as enumerations. For example, because we ship the product worldwide we have a list of natural languages that the application may be running in. That's about 30 global symbol table entries taken up right there. This forces us to eliminate global constants and replace them with literal values (magic numbers) in the code that consumes them. This is just asking for trouble but it's the only way we can squeeze in new event handlers and global functions. This is very common:
' These global constants are commented out because
' we don't have room for them
' Global Const gEnglish% = 1
' Global Const gSpanish% = 2
...
Switch gCurrentLanguage
Case 1 ' gEnglish = 1
Case 2 ' gSpanish = 2
Also, because of this limit we are forced to combine functions where possible to keep the global symbol table entries down. This leads to some very confusing code and breaks down the idea of having modules as classes. Spaghetti code, here we come!
Event handling is done by creating a function in the form's module named controlName_eventName() and all of the plumbing is done behind the scenes for you. Because it is so easy to learn the language and the event handlers are such a natural place to put code, many VB3 application are spaghetti with no real separation of UI code from business logic or data access. With discipline you can create a two or three tiered application using VB3 but it really does tempt you to break the rules and put in filthy hacks.
Visually, in a VB3 application your GUI controls are limited to those available in Windows 3.11 and makes your application stick out like a sore thumb when running on a more recent version. For a legacy application I guess this doesn't really matter. Plenty of libraries and government departments are still running character mode DOS applications but these days it's via terminal software running on XP or Vista. Users that have to use an old application daily don't have the luxury of complaining about it- if it does the job then nobody cares what it looks like.
One of my pet peeves with VB3 is that the project file doesn't keep track of the breakpoints introduced into code. This means that you can be in the middle of an investigation into strange behaviour carefully setting breakpoints and observing variable values when for no reason the IDE will GPF (crash) and everything's gone. Because the IDE crashes reasonably frequently - several times a day if I'm using it full time - you end up with a Ctrl-S hair trigger. Every line of code I write is followed by a save to make sure it isn't lost. I have seen several hours work simply disappear due to an IDE crash and vowed to never let it happen again.
Backward Compatibility
Microsoft goes to great lengths to ensure that software written for a previous OS works on the next version. If businesses realise that upgrading to the latest version of Windows will cause their mission critical legacy apps to stop working then they will simply not upgrade (see: Vista).
This means that software written for Windows 3.11 can run on Vista and 7 with only minor modifications. The mechanism by which this works is called Windows on Windows (WoW), which entails 16-bit applications being hosted in a process known as the NT Virtual DOS machine (NTVDM.exe).
System calls are intercepted by NTVDM which mimics a native 16-bit OS. The running application is none the wiser that it is running in a virtual machine and goes about its business as if it's running on Windows 3.11. These system calls are presumably delegated to the host OS then re-packaged to be returned to the host application. This process - going from 16-bit to 32-bit and back - is called "thunking" and is a topic I will go into detail about in further posts.
There are several factors that make 16-bit programming a headache. First of all the tools are not meant for development at the scale that we are doing it. The 16-bit memory model limits you to around 1MB of total memory so memory allocation has to be managed very carefully. Stepping over that line will result in a crash. In future posts I will detail the extreme contortions we go through to stay under the memory limit and within the boundaries.
This means that software written for Windows 3.11 can run on Vista and 7 with only minor modifications. The mechanism by which this works is called Windows on Windows (WoW), which entails 16-bit applications being hosted in a process known as the NT Virtual DOS machine (NTVDM.exe).
System calls are intercepted by NTVDM which mimics a native 16-bit OS. The running application is none the wiser that it is running in a virtual machine and goes about its business as if it's running on Windows 3.11. These system calls are presumably delegated to the host OS then re-packaged to be returned to the host application. This process - going from 16-bit to 32-bit and back - is called "thunking" and is a topic I will go into detail about in further posts.
There are several factors that make 16-bit programming a headache. First of all the tools are not meant for development at the scale that we are doing it. The 16-bit memory model limits you to around 1MB of total memory so memory allocation has to be managed very carefully. Stepping over that line will result in a crash. In future posts I will detail the extreme contortions we go through to stay under the memory limit and within the boundaries.
Tools of the Trade
The nineties called- they want their programming tools back.
Developing and maintaining 16-bit apps requires only the latest and greatest toolset:
Microsoft Visual Basic 3.0
Microsoft Visual C++ 1.52
Microsoft Visual Source Safe
Shrinker
Various 3rd party VBX controls
You can acquire a copy of these tools at your local museum. Actually I think VC++ 1.52 is still available on MSDN if you really want it. Oh, did I not mention that this blog will contain a fair amount of sarcasm and loathing? Well it will.
Developing and maintaining 16-bit apps requires only the latest and greatest toolset:
Microsoft Visual Basic 3.0
Microsoft Visual C++ 1.52
Microsoft Visual Source Safe
Shrinker
Various 3rd party VBX controls
You can acquire a copy of these tools at your local museum. Actually I think VC++ 1.52 is still available on MSDN if you really want it. Oh, did I not mention that this blog will contain a fair amount of sarcasm and loathing? Well it will.
First Post!
This blog is going to be mainly programming orientated with a dash of oldschool. Major shout out to Raymond Chen who knows full well the pain of backcompat.
At the moment my work partially involves maintaining a legacy 16-bit Windows application that must run on every Microsoft OS from Windows 7 RC all the way back to Windows 95 (!). We only recently dropped support for Windows for Workgroups 3.11.
Because of this, I have become intimately acquainted with the 16-bit compatibility layer (NTVDM/WOW) in Windows NT4 and above which is the only thing keeping our app running in a 32-bit environment. Needless to say we are really glad Microsoft stands its ground on backward compatibility of its OSes.
I will be posting insightful retrospectives on investigations I have performed to keep a monolithic legacy code base able to run on modern machines and why on earth I would bother.
Stay tuned!
At the moment my work partially involves maintaining a legacy 16-bit Windows application that must run on every Microsoft OS from Windows 7 RC all the way back to Windows 95 (!). We only recently dropped support for Windows for Workgroups 3.11.
Because of this, I have become intimately acquainted with the 16-bit compatibility layer (NTVDM/WOW) in Windows NT4 and above which is the only thing keeping our app running in a 32-bit environment. Needless to say we are really glad Microsoft stands its ground on backward compatibility of its OSes.
I will be posting insightful retrospectives on investigations I have performed to keep a monolithic legacy code base able to run on modern machines and why on earth I would bother.
Stay tuned!
Subscribe to:
Posts (Atom)