Notes for vil/vil2

Good points about existing vil1

• The functional notation
  • vil1_image im = vil1_load("filename")
• Support for processing very large images on disk.
• Support for lots of image types

Problems with existing vil1

• It doesn't support planes very well.
• Pointer indexing rather than arithmetic indexing
  • Pointers are slower than arithmetic on most modern platforms
  • It is hard to build a 2d image from a slice of 3d data
• The vil1_image smart pointer stuff is confusing.
  • Is it a view, is it a smart ptr, is it a smart smart ptr?

There are some fundamental divisions

data directly accessible	v	data downloadable
data type known at compile time	v	data type known at run time
data ordering known a priori	v	data ordering known after image loading

Do these not indicate that trying to have a single image type (or hierarchy) is incorrect? Does the single vil1_image hierarchy actually give you anything?

vil merges all the assumptions on the left in vil_image_view<T>, and those on the right in vil_image_resource

Background

Before ISBE, Manchester adopted VXL we used an image type which provided

1. Good plane support
2. Arithmetic indexing, exposed in the interface
3. A world to image transform
4. A polymorphic class hierarchy whose base class could represent images of any dimension.

We heavily used all these features, none of which were available in vil1. To get round this problem we wrote our own public VXL-compatible image library - mul/mil. It did all the above stuff, using vil1 to do the file loading and saving. We also designed the library in terms of views of images. We first informally raised the idea of rewriting vil1 to support our code at the Zurich VXL meeting in March 2001, with encouraging responses. After spending 18 months noticing that we were duplicating more and more code that was available in vil1, the short-notice arrangement of a meeting in Providence on the 27th Sep 2002, and the encouraging comments from other VXL members gave us the spur to design this proposal.

Philosophy

We want a single "normal" and "easy" image class that we can point new users at. This should also be the default class for doing actual pixel manipulation.

We want to keep the type proliferation down. In particular, we do not want to have to write or maintain vil_image_process_func() for more image types than absolutely necessary. So we enforce the following restriction in vil's API

• vil_image_resource_sptr is used when you don't know an image's pixel type, and you can't immediately access the pixels.
• vil_image_view<T> is used when you know the type, you know the structure and you can immediately access the pixels

By keeping this division clean we can avoid type proliferation. There are limited exceptional cases. One is when you absolutely must have a function that works on a view without knowing its pixel type - e.g. vil_load("filename") which returns a vil_image_view_base_sptr.

This division leads naturally to a division of labour.

• A vil_image_process_func() that actually does the pixel value calculations should take a concrete vil_image_view<T>.
• If someone wants a super-duper vil_image_process_func() (size-no-problem, runtime pixel-polymorphic, etc) then write a function that takes vil_image_resource_sptr. However sub-contract the actual pixel processing to vil_image_process_func(vil_image_view<T>).

A vil_image_view<T> is a view in the sense that it provides a view of some image data. You can make concrete copies of views cheaply, without needing to copy the underlying image data. As a service the view memory-manages the image data as well, so you don't need to worry about the underlying data at all.

vil_image_resource is the abstract base of polymorphic class tree. The design is such that most work can be done on the base class API. You can cast it to its concrete type if you need to set some obscure property. To get a functional style, most operations on a vil_image_resource, should be on a vil_image_resource_sptr.

We intend to uphold this API design rigorously. Whilst we have no intention of preventing users from doing whatever they like with their own code, we believe that the current design is locally optimal within a wide radius of convergence. We think that any VXL level-2 libraries should use these API guidelines as well.

Design decisions:

vil_image_view<T>::set_size(..) should be virtual and in the base class

It is in mil_image_2d. Resizing can be slow so virtual function is no problem. It can be useful to set the size of a image without knowing its type. Decision IMS and TFC.

vil_memory_chunk should not be templated.

You could have a memory chunk containing rgb, and choose to view it as a vil_image_view<vil_rgb<vxl_byte>> with nplanes=1, or vil_image_view<char> with nplanes =3; It is hard to get the type of the smart pointers correct without linking the types T of the vil_memory_chunk<T> and vil_image_view<T>. We can put a image type in a member variable if it is necessary (e.g. for vsl IO). Decision IMS and TFC.

Use (i, j, p) default names for index parameters, and ni(), nj(), nplanes(), for sizes.

Want to avoid use of (x,y) to avoid giving the impression that pixels are in a Cartesian co-ord system. It is good to link index name and size name to avoid for loop mistakes. Alternative was row, col, plane, and rows, cols, and planes. Inner loops should have short variable names. Decision at Providence meeting.

vil_image_view<T> should have a sparse interface.

Methods like set_to_window(..), flip_ud(), transpose(), can all be provided as standalone functions without much addition overhead. Class API's should be small rather than large. Use Doxygen's \relates option to help users find related functions. Decision IMS.

Using arithmetic indexing scheme

i.e.
vil_image_view<T>::operator(i,j,p) {return *(top_left_ptr + i * istep + j* jstep + p*planestep);}
Don't use pointer indexing.
vil_image_view<T>::operator(i,j) {return raster_ptrs[j][i];}
Multiplication is fast on modern processors. Pointers require extra memory lookup which is slower, and can reduce cache hit rate. Pointers do not generalise well to planes, or 3D. Decision confirmed at Providence meeting.

Best order for index parameters is i,j,plane.

There is no preferred indexing order with arithmetic indexing. You don't know whether the planestep is 1 or istep is 1. However, for ease of use it is worth having a consistent interface. Two alternatives are (plane, i, j) and (i, j, plane). The former is probably more natural, however, the latter allows us to use default argument plane=0, which is very useful for keeping interface clutter down. Decision at Providence meeting.

Use vil_image_view_base_sptr in preference to vil_image_view_base *.

Main uses of this are in vil_load(..) and vil_image_resource::get_view(..). These functions cannot return a real vil_image_view<T> because that would involve knowing T in advance. They cannot return references because the real object will get destroyed as the function ends before it is assigned to a variable in the calling level. We use the smart over the raw pointer to avoid a likely source of memory leaks. However, most code in vil should operate on vil_image_view<T>. Decision at Providence meeting.

vil_image_view<T> should not be derived from vil_image_resource

As explained above these are actually two different types. Efficient access to ni, nj for vil_image_view<T> whilst matching interface for vil_image_resource would require a lot of complexity. Decision confirmed at Providence meeting.

There should be no general vil_image_resource::set_properties(..)?

Its existence would imply the ability to set arbitrary properties. You can always have a specialist set_obscure_tiff_property(..) as a member function of vil_tiff_file_image. Decision by IMS and AWF. There will be a vil_image_resource::get_properties(..) with all properties documented in vil/vil_properties.h. This allows for partially shared properties such as bits per component or physical pixel height

vil_image_resource::get_view(..), etc. will not allow you to specify planes.

It is easy to split off individual planes later. Although plane specifications could give you a memory advantage - a factor of 3 is not big enough to be useful for v. large images. Not having planes specifications can make programming vil_file_image_resource::get_view(..), etc. a lot easier and potentially faster. Decision at Providence meeting.

Index and size types should be unsigned int.

Signed has the advantage that it is much easier to pick up -ne overflow errors. Unsigned has the advantage that it is more natural, makes the fixed 0,0 origin clear, and it reduces the number of assertion checks needed (admittedly by exposing you to the -ne overflow errors:) We can catch some -ne overflow errors by assert (i != (unsigned)(-1)). Decision at Providence meeting.

Use function overloading and normal filename prefix for functions on vil_image_view<T>, and functions on vil_image_resource objects.

Where function overloading doesn't discriminate (e.g. vil_load("filename") ) then use vil_load_image_resource("filename") for vil_image_resource objects. Decision IMS and TFC

vil_image_view<T>::set_size(ni, nj) will make no changes if size is same. It will detach from original underlying data and create new data if different.

This has some odd effects. If you have multiple views to the same data then set_size may or may not move view to different data. However, it is very useful when treating views as ordinary images, and allows very efficient use of workspace images. For example multiple calls to an image processing class often need identically sized image workspaces. Using this set_size means that it doesn't have to reallocate memory on a regular basis. mil_image has been using this design for 18 months with no problems. Decision at Providence meeting.

vil_image_view<T>::operator=() can be smart but should do fast view transforms only.

Allowing
vil_image_view<vil_rgb<vxl_byte> > im_rgb(ni, nj);
vil_image_view<vxl_byte> im_planes = im_rgb;
and
vil_image_view<vil_rgb<vxl_byte>> pnm = vil_load("filename.ppm");
makes for easy use. However , expensive operations should not have short names. So this should not include doing conversions such as byte to float conversion. These conversions should be done by separately named functions. Decision at Providence meeting.

vil_image_view<T>::operator=() should take base class and base class sptr.

Equals is defined as vil_image_view<T>::operator=(const vil_image_view_base &. This is an operator= from its own base class. This doesn't appear to cause any problems, and allows the above fast view transformations between pixel types. Additionally vil_image_view<T>::operator=(const vil_image_view_base_sptr &) allows for the simple conversion from the return value of many functions such as vil_load. These need to return base class sptrs to reduce memory leaks and because their interface cannot know the pixel type. Decision IMS

Should vil_load should return a base class sptr.

(Similarly vil_image_resource::get_view(..)) They could return vil_image_view_base * or vil_image_view_sptr Both appear to work. The smart pointer version has the advantage of not needing users to be careful about deletion. Decision by consensus on the reflector.

All pixel types should be explicit about type length (e.g. vxl_uint_16)

Alternative is to have vil_image_resource::get_view() automatically pick the best standard type for that platform (e.g. short) The problem with this becomes clear on 64 bit platforms. What types should 16bit or 32bit data get loaded into (The same platform can't define short as both.). Decision at Providence meeting. Question - do the float types need to be specific or can we assume IEEE standard lengths? 

All operations will default to scalar pixels and multiple planes. vil_image_resource hierarchy will mostly work only with scalar pixels.

Nothing prevents you from using vil_rgb pixels, and we will provide appropriate support. However planes are more general than components, and vil_image_resource hierarchy can be simpler if it only thinks about scalar pixels. In particular it may not be feasible to consider every possible compound pixel type in a switch() statement, but it is feasible to consider every possible scalar pixel type. Decision confirmed at Providence meeting.

Base class image type polymorphic in dimension and pixel type, and containing world to image transforms will be provided in a separate level2 library - vimt.

vimt benefits from being able to use both vil and vgl (and possibly vnl). Decision TFC.

Deal with const pointer data issues pragmatically inside vil.

Pointers to data are often passed in as const, but may have the const removed. This is because it is rather complicated to keep track of whether the data is going to be changed or not. Just try to be sensible, OK? Decision TFC.

All standard image processing algorithms will go into vil/algo

We will dump the vipl interface. All image processing code would be written in terms of vil_image_view<T> first. Then write another function which uses the first function, but does all the stitching stuff. Decision at Providence meeting.

All vil/algo image processing functions will have their input images as the first parameters.

e.g. vil_sobel<T>(const vil_image_view<T>&source, vil_image_view<T> &dest_i, vil_image_view<T> &dest_j);

This is different from the operator=(..) style, but is more common in other function libraries. Decision at Providence meeting.

Best names are vil_image_view, vil_image_resource

A beginner will expect to see the word image in the main image type. However it aids understanding to be reminded that this is a view so vil_image_view. vil_image_resource was originally vil_image_data. However that was thought to be confusing and might get confused with the vil_memory_chunk class. So vil_image_resource. Decision at Providence meeting and by vote on vxl-maintainers.

Use set_size(..) rather than resize(..).

The STL has introduced the notion of a data preserving resize. Since our code doesn't do that, we use set_size(..) instead. Decision Amitha and consensus on the reflector.

raster and plane step types should be vcl_ptrdiff_t.

This is strictly the correct type to add to a pointer to get another pointer. It isn't signed int on platforms with 64-bit address bus but 32-bit data. Decision IMS.

vcl_complex<float> is a scalar pixel type.

Whilst useful under certain circumstances, it is not sensible default behaviour to allow vil to treat a complex<float> as a two plane float image. Therefore vil_image_resource derivatives should treat complex<float> as ones of its default pixel types. Decision - FW and IMS.

vil_image_view<T> should have a constructor which allows interleaved or separated plane storage.

This is the single most useful specialist memory format for a view, and so deserves to be easily created. Decision PVr and IMS.

There should be versions of vil_convert_*() which take and return vil_image_view_base_ptrs.

There are already versions which take vil_image_view<T>s. However when loading an image it is often useful to force it into a particular pixel-type, arrangement, etc. We cannot assign it to a concrete vil_image_view<T> because that will fail if the types are incompatible. If we want to allow vil_load() to remain easy to use then we cannot solve this problem by providing vil_convert(vil_image_resource_sptr). This is the 3rd place where we allow the use of vil_image_view_base_ptrs (after vil_load() and vil_image_resource::get_view().) Decision IMS and TFC.

n.b. consistency suggests providing vil_convert(vil_image_resource_sptr) versions.

file resources should return pixel values in least significant bit position.

File types that have odd pixel widths (e.g. DICOM can use 12-bit images) should not scale those pixels to fill the full range of the C++ pixel-type that they return. So our 12-bit DICOM image may return vxl_uint_16 pixels with values from 0 to 4095. Decision IMS, TFC and Marc Laymon (GE).

 

Remaining questions

There is now no way to get a bit-compressed pbm file into memory efficiently. Is this a problem? We could solve it by deriving a vil_image_view_of_compressed_bits, or similar.

Possible improvements

Add a get_pixel(void *buf,i,j) and put_pixel(void *buf,i,j) to vil_image_resource.

These might be useful for efficient implementation of some algorithms. They could be implemented in terms of get_copy_view(..) and put_view(..) in the base class. However, in some cases you need to probe/set a pixel at a time and the overhead of allocating a vil_image_view<T> for each get_view(..) is quite expensive. Note that the cost of allocating the vil_memory_chunk is irrelevant. If the image is already in memory, no new vil_memory_chunk is allocated. If the image isn't already in memory, the cost of the new memory chunk is smaller than the disk access time. Note, one alternative may be to bring back the vil_image_resource::fill_view(...) which modifies the contents of an existing view and doesn't allocate any memory. IMS.

Add block processing implementation to the vil_image_resource derivatives for very large images.

The basic idea of dealing with large images was designed into vil1. It deferred loading the whole image into memory until actually needed. You could then load only one block of the image as required. This basic idea has been passed into vil2/vil. However, as with vil1, this is only a design outline and API. A lot of implementation to do block-by-block processing efficiently is missing.

The first step is to add preferred_block_size and preferred_block_origin to vil_property.h. Both of these unsigned[2] properties would be provided by file resources and used by the processing resources to do the work one block at a time. I guess many of the existing file resources will specify blocks that are one or more full rasters. Those file formats designed to deal with large images (including TIFF) can store their pixels in rectangular tiles. The processing resources should be able to use several of their supplier's blocks together up to some locally decided (or globally specified) limit. The processing resources would need to export these properties including any modification (e.g. vil_transpose().) 

Quite how vil_convolve_1d_resource should deal with a single column for a preferred block size (when vil_convolve_1d_resource operates on rows) - I don't know. It would take some experimentation to get an efficient solution. Until then we can always rely on the fact that the preferred_block_size, etc are just hints and can safely be ignored at the loss of efficiency. IMS.