Today’s post, the second in the series, describes the scene in 1990, when image display officially appeared in MATLAB for the first time.
My 1990
For most of 1990, I was finishing up my Ph.D. in electrical engineering at Georgia Tech, specializing in digital signal processing and digital image processing. My thesis was about image compression techniques, with one part about subband filter banks, and the other part about parametric edge modeling. I graduated in December and immediately moved to Chicago (cold) to join the faculty of the University of Illinois at Chicago in the Department of Electrical Engineering and Computer Science.
The EECS department set me up with my initial equipment and software requests:
- Sun Sparcstation (I really did love that workstation!)
- 600 dpi laser printer
- MATLAB
- Mathematica
I think it was MATLAB 4, but I don’t remember for sure. I found Mathematica interesting but not that useful for my work, so it faded pretty quickly.
PCs in 1990
In 1990, a typical high-end PC had an Intel 80486 CPU running at a clock speed of 25 or 33 MHz, with 2 or 4 MB of RAM.1 Can you imagine doing image processing research and algorithm development with only 4 MB of RAM? Compare that with today’s CPUs that have many computational cores, all running at multi gigahertz clock speeds, and accessing dozens of gigabytes of RAM. Or with GPUs containing far more cores and running even faster.
Typical PC monitors had a resolution of 640x480 or 800x600 and were “8-bit color,” meaning that they could only display 256 colors at a time. (There were high-end Silicon Graphics workstations available with 24-bit color capability.)
Microsoft Windows 3, the first commercially successful Windows version, was released in May.2
MATLAB 4
1990 saw the first MATLAB 4 releases. As best I can recall, the earliest platform releases were on the PC and Sun workstations, and it took roughly two years for it to be ported to the zoo of all MATLAB platforms, including a variety of Unix-based workstations, a couple of Mac CPU architectures, VAX, and the Cray supercomputer.
One of the most important MATLAB 4 features was in the math area: the introduction of sparse matrices. Sparse matrix linear algebra greatly expanded the scope of computational algorithm work that could be done in MATLAB.
Regarding the MATLAB language in version 4, the important thing to remember is that MATLAB still had only one data type—the matrix. The MATLAB 4 matrix was two-dimensional, and its elements were double-precision floating-point values. If you were trying to do image processing in MATLAB, then, every pixel took up 64 bits of memory. Image processing using a matrix-based language was appealing, but at that point in MATLAB history, and for a few years after, it was tough going.
For the image display story, though, the biggest deal about MATLAB 4 was Handle Graphics.
Handle Graphics
The new MATLAB 4 graphics system, called Handle Graphics, was a huge leap forward. Although true object-oriented programming features would not appear in MATLAB until much later, Handle Graphics enabled graphics creation and programmatic manipulation via pseudo objects such as line, patch, surface, text, axes, and figure. Graphics objects lived in a tree, with figures being children of the root, axes being children of figures, and lines, patches, etc., being children of axes. Handle Graphics also included uicontrol objects for implementing GUIs (or apps, in today’s terminology).
Handle Graphics objects had properties that could be queried and modified by name, using the set and get functions, like this:
x = get(line_handle, 'XData');
set(axes_handle, 'Position', [.1 .1 .8 .8], 'XLim', [-1 1], 'YLim', [-1 1])
Handle Graphics influence on modern MATLAB design
Objects
The Handle Graphics pseudo objects heavily influenced the design of later object-oriented programming features in MATLAB, first with MATLAB 5 in 1997, and then later with MATLAB R2008a. The emphasis on the direct querying and setting of object properties was perhaps the most important factor.
PV pairs
This function call:
set(axes_handle, 'Position', [.1 .1 .8 .8], 'XLim', [-1 1], 'YLim', [-1 1])
is an example of what MathWorks developers used to call PV pairs (or param-value pairs). PV pairs were successive pairs of input arguments. The first argument in a pair was the name of a Handle Graphics object property, and the second argument was its value.
Several key PV pair behaviors took root and are prevalent in today’s MATLAB designs:
- Parameters can be specified in any order.
- Parameters are not required inputs.
- Parameters have default values.
- The same parameter can appear multiple times in a call.
- Any number of PV pairs can be accepted in a call.
Once variable arguments lists entered the language (MATLAB 5, 1997), toolbox developers began incorporating the PV syntax style. They are now called named arguments and have improved syntax. Here is a documentation sample showing the modern form:
Low-level and high-level plotting functions
Handle Graphics introduced a distinction between low-level and high-level graphics functions. I’m not sure whether that terminology was documented, but MathWorks developers used it.
A high-level graphics function, such as plot, clears away any previous graphic in the current axes and resets the axes, unless you have previously called hold on. For example, suppose we start with a simple plot and give it a title:
plot([1 2], [1 10])
title("Up and to the right")
(The plot title above honors long-time MathWorker Michael M. IYKYK)
Then we call plot again:
plot([1 2], [10 1])
You can see that the second call to plot cleared the earlier plot, as well as the axes title. The function plot is a high-level function that can and usually does have side effects on existing graphics objects.
Compare that with the behavior of line:
figure
line(XData = [1 2], YData = [1 10])
title("Up and to the right")
Now call line again.
line(XData = [1 2], YData = [10 1])
The second call did not clear out the previous line, and it did not change the axes title. The function line is a low-level function. It is more of a pure object-creation function with few side affects.
Understanding low-level vs. high-level behavior, side effects, and interaction with the MATLAB hold on and hold off commands will be important for discussing some of the design issues for the various image display functions over the years.
Colormaps
MATLAB 4 Handle Graphics established a couple of important design precedents regarding colormaps. First, a colormap would be represented as a matrix such that each row is one color. For a color space with 3 components, such as RGB, a colormap would be a $P\times 3$ matrix, where $P$ is the number of colors.
Second, colormap component values would be in the range $0\le v\le 1$, establishing a representation that was mathematical in spirit, rather than a more programming-oriented representation such as using integers from 0 to 255.
The IMAGE object
One of the new graphics objects in MATLAB 4 was the image object. I believe that the clown image was used back then to demonstrate image display.
clear
load clown
whos
Name Size Bytes Class Attributes
X 200x320 512000 double
caption 2x1 4 char
map 81x3 1944 double
image(X)
colormap(map)
Indexed image display model
In MATLAB 4, the image object always used an indexed image display model. With an indexed image, the color of each displayed image pixel is determined by using that pixel value as an index into the colormap. For example, here are the color component values of the $(10,10)$ pixel:
k = X(10,10)
k = 56
c = map(k,:)
c = 1x3
0.8672 0.5781 0.1250
What does that color look like? One way to see it is to use image to display a single-pixel image:
image(56)
colormap(map)
Flexible spatial coordinate model
One unusual and powerful feature of the new image object was its spatial data coordinate model. By default, the image is displayed using something like ordinary pixel coordinates. Look again at the clown image.
image(X)
colormap(map)
size(X)
ans = 1x2
200 320
X has 200 rows and 320 columns, and you can see that the $x$ -axis and $y$ -axis are labeled accordingly. In the $(x,y)$ plane, each pixel is a $1\times 1$ square. But you can easily make image pixels have different $(x,y)$ sizes by setting the XData and YData properties of the image. Below, the clown image is displayed in a $2\times 2$ spatial region, centered around the $(x,y)$ origin.
image(X, 'XData', [-1 1], 'YData', [-1 1]) % MATLAB 4 syntax
colormap(map)
This was a very significant design decision, one that allows images to be displayed together with other spatial data. Here is the image with a superimposed $1\times 1$ box centered at the $(x,y)$ origin.
hold on
plot([-.5 .5 .5 -.5 -.5], [-.5 -.5 .5 .5 -.5], "y")
hold off
The $(x,y)$ domain flexibility of MATLAB image display had a deep, long-lasting impact. As image processing techniques developed and evolved in MATLAB, many of the applications areas involved “images” that were not photographs. Instead, they were other kinds data matrices that were often visualized as images, and processed using image processing operations such as filtering. For example, DEM (digital elevation model) data is often displayed as images. With that kind of data, the $(x,y)$ plane might be projected map coordinates with physical units of meters. One image pixel might occupy $3\times 3$ meters on that plane.
High-level and low-level syntaxes
The image function has an unusual syntax and behavior design, in that it sometimes acts as a high-level graphics function, and sometimes as a low-level graphics function. Look at the clown display again. (Are you weary of seeing this image yet?)
image(X)
colormap(map)
Notice how the $y$ -axis increases downward instead of the upward? That is not the default axes behavior. When you call the function using the syntax image(X), which MathWorks developers sometimes called a convenience syntax, the function acts as a high-level function. In that mode, it has a side effect on the axes—it changes the axes YDir property to 'reverse'.
Alternatively, if you call image using only PV pairs (or named arguments today), then it acts as a low-level function, and it doesn’t have that side effect.
clf
image('CData', X) % MATLAB 4 syntax
colormap(map)
IMAGE object issues and limitations
Rectangular pixels
From its very beginning in 1990, a common user complaint about the image function is that it produced an image display that appeared to be “stretched.” The phrase “nonsquare image pixels” featured prominently in many tech support cases.
To understand the effect, consider this square, $256\times 256$ image that was very widely used in the 1980s for image deblurring research:
Here is the display produced by the image function:
A = imread("cameraman.tif");
image(A)
colormap(gray(256))
grid off
The image looks stretched horizontally because the graphics system automatically adjusts most kinds of plots so that they fill the axes, and the default axes shape is wider than it is tall.
I mentioned earlier that the image(A) syntax has the side effect of making the $y$ -axis increase downward. With the benefit of several decades of hindsight, I claim it would have been better to also set the data aspect ratio to be 1:1. That’s what axis equal does:
axis equal
Spatial pixel-width ambiguity
This issue is, admittedly, one that many people did not care about very much. Only someone obsessed with proper edge-case handling would give it much thought. When it comes to edge cases involving the sizes of matrices and arrays in MATLAB, I am that person.
Consider the XData property of an image object. It is a two-element vector that specifies the $x$ -axis pixel centers corresponding to the first and last columns of the image matrix. Given the number of columns, N, and the XData property, we can compute the spatial coordinate system width of each image pixel:
pixel_width = (N - 1) / (xdata(2) - xdata(1))
And when there is only 1 column? Oops, the pixel width is $0/0$, or as MATLAB would say, NaN. This sort of thing results in extra conditional code branches that are necessary for handling the irregular behavior.
Hindsight (and edge-case obsession) says it would have been better to choose a representation that behaves nicely when N is 1, and even when N is 0.
Poor-quality interpolation
Given the very slow CPU and graphics card speeds in 1990, image display using nearest-neighbor interpolation with no antialiasing was a reasonable design decision for the time. It was, in fact, probably the only practical way to go. However, the resulting image visualization is often poor.
I will have much more to say about this issue in at at least a couple of future posts in this series. For now, you can check out my MATLAB Central blog post, “How to display images with bilinear interpolation and antialiasing.”
High memory use
As I mentioned near the beginning of this post (and wow, that was a long time ago, congratulations on making it this far!), the only kind of variable in MATLAB 4 was a 2-D matrix of double-precision, floating-point values. Image processing researchers using any other programming language were accustomed to image data taking 8 bits of memory per pixel. In MATLAB, it was always 64 bits per pixel. On a PC with 4 MB of RAM, that hurt.
Binary, grayscale, and truecolor image representations not supported
The indexed image model was limited and often not very useful for image processing algorithms and applications. Other more useful models, such as binary, grayscale, and truecolor were not supported. (The term truecolor was commonly used at the time to describe a 24-bit per pixel representation, with 8 bits each for the red, green, and blue color components of each pixel.)
That said, there actually was a way to display a truecolor image in MATLAB 4, if you happened to have a high-end Silicon Graphics (SGI) workstation with truecolor image capability. It was a very silly way, but it worked.
You could convert a truecolor image into an indexed image, with no loss of color fidelity, by creating a huge colormap that contained the color of every single image pixel. (Go ahead, read that sentence again.)
If R, G, and B were $256\times 256$ matrices containing the red, green, and blue color components of a $256\times 256$ truecolor image, then your image display code could like this:
map = [R(:) G(:) B(:)];
X = reshape(1:65536, 256, 256);
image(X)
colormap(map)
Good times.
The first MATLAB default image
I mentioned earlier that the various graphics object properties all have default values. That includes the data properties, which is why you get real line plot when you call line with no input arguments.
line
It looks that way because the default line XData is [0 1], and the default YData is also [0 1].
For the image object, the pixel values are stored in the CData property, and the image CData property has a default value. In MATLAB 4, the default CData was an Easter egg of sorts, because the default value was actually a picture of a person.
In MATLAB 4, if you called image with no input arguments, it might have looked something like this:
With a few axes tweaks and a grayscale colormap, it could look like this:
That guy is Damian, one of the developers of the MATLAB 4 Handle Graphics system.
As some of you know, that is not the end of the story about the MATLAB default image. Stay tuned.
Next up
For the next post in the series, I’ll jump ahead in time, all the way to 1993. That year saw the release of Image Processing Toolbox v1 and the IMSHOW function.