9. vsl: Binary I/O

Chapter summary: This section describes how to save and restore objects using a binary stream. It details how to add the appropriate functions to each class to make use of this facility.

All objects in VXL should be able to save themselves to a stream (eg a file) and restore (load) themselves from a stream (file). The main functions provided for this purpose are vsl_b_write(os,object); and vsl_b_read(is,object&);.

The binary IO for the core libraries (vbl, vil, vgl and vnl) is implemented in `clip-on' libraries which live in the io subdirectories of each library (thus the declaration of the function vsl_b_write(vsl_b_ostream&,vnl_vector const&); lives in the file `vnl/io/vnl_io_vector.h'.

However, it is recommended that I/O for other libraries be provided by writing b_write(os); and b_read(is); functions in each class. See the `Design Notes' section below.

9.1 Supported Platforms

The binary I/O code is known to work across the following hardware/OS/compiler combinations, but probably also works on most other platform/compiler combinations:

1. Intel - Linux - gcc-2.95 and gcc-3.0
2. Intel - WindowsNT - vc++
3. SGI - MIPS - MipsPRO CC
4. Sun - Solaris - gcc-2.95
5. DEC alpha - OSF - gcc-2.95 and gcc 3.0 (64 bit!)

Thus binary files produced by any of the above should be readable by any other of the above. There is of course a minor exception: large numbers (like integers larger than 4294967295) saved on a 64-bit platform cannot be read on a 32-bit platform.

9.1.1 caveats

The code has been designed to work on as many platforms as possible. However if your platform uses any of the following, then it will probably not work (as presently coded.)

1. A middle endian word encoding scheme.
2. Chars of length other than 8 bits.
3. Non-IEEE format floats and doubles.

9.2 Using Binary I/O

To save an object to a file, simply do the following:

  vxl_myclass my_object;
  
  // Fill my_object
  
  vsl_b_ofstream bfs("my_object.bvl");
  if (!bfs)
  {
    vcl_cerr<<"Failed to open my_object.bvl for output."<<vcl_endl;
  }
  else
  {
    vsl_b_write(bfs,my_object);
    bfs.close();
  }

To load/restore the object from a file:

  vxl_myclass my_new_object;
  
  vsl_b_ifstream bfs("my_object.bvl");
  if (!bfs)
  {
    vcl_cerr<<"Failed to open my_object.bvl for input."<<vcl_endl;
  }
  else
  {
    vsl_b_read(bfs,my_object);
    bfs.close();
  }

It is recommended that the default extension name for your binary files is .bvl. This extension does not appear to be used by any other program. In many cases however, you will want to pick a new extension to indicate the contents of a file. For example, we store active shape model objects with ending .asm.

The classes vsl_b_ifstream and vsl_b_ofstream are simple wrappers around real vcl_ifstream and vcl_ofstream objects. These wrappers ensure that you open a file with CR/LF conversion turned off, and they should also allow lots of common misuses to be caught at compile time.

The functions vsl_b_write(os,X) and vsl_b_read(is,X) are defined for all reasonable cases, including all inbuilt types, most classes in vcl and the classes in the core vxl libraries.

When you write a new class, you should add the appropriate functions to allow easy use of binary I/O (see below).

Or for simplicity we provide the utility functions which would allow you to write:

  #include <vsl/vsl_quick_file.h>
  vxl_myclass my_object,my_new_object;
  
  vsl_quick_file_save("my_object.bvl",my_object);
  vsl_quick_file_load("my_object.bvl",my_new_object);

9.2.1 Saving multiple objects

One can use exactly the same approach to save a set of objects

  vxl_myclass my_object;
  vxl_my_other_class my_other_object;
  
  // Fill objects
  // ...
  
  vsl_b_ofstream bfs("my_object.bvl");
  if (!bfs)
  {
    vcl_cerr<<"Failed to open my_object.bvl for output."<<vcl_endl;
  }
  else
  {
    vsl_b_write(bfs,my_object);
    vsl_b_write(bfs,my_other_object);
    bfs.close();
  }

(and similarly for loading them).

A standard rule for ensuring trouble free I/O is

Always write the input and output code in tandem - the output should precisely mirror the input.

9.2.2 Binary I/O by baseclass pointer

When using polymorphism, there are frequently times when one needs to save and restore an object just using a base class pointer to it. vsl provides facilities to do this.

Assuming class my_derived is derived from class my_base, the following will work.

To save an object by baseclass:

  my_derived d;
  
  my_base *b = &d;
  
  vsl_b_ofstream bfs("data.bvl");
  vsl_b_write(bfs,b);
  ...

To restore an object:

  // Make application aware of possible classes that it might see in the file
  
  vsl_add_to_binary_loader(my_derived());
  vsl_add_to_binary_loader(my_derived2());
  ...
  
  my_base *b = 0;
  
  vsl_b_ifstream bfs("data.bvl");
  vsl_b_read(bfs,b);
  // b now points to the correct class which has been created
  // on the heap and filled with the data from bfs
  ...

Note that the read function will only work if the application has been made aware of each of the possible derived classes that it might come across in the file. This is done using calls to vsl_add_to_binary_loader(my_derived()) (see appendix for details).

To reduce the pain of doing this, many libraries have a function that adds all the relevant derived classes (eg xxxx_add_all_binary_loaders() where xxxx is the library name).

9.2.3 Which files do I need to include/link?

In general the vsl_b_read and vsl_b_write functions use Koenig Lookup - that is the location of their declaration depends on their parameters.

The vsl_b_stream objects and vsl_b_write and vsl_b_read functions for fundamental data types are declared in <vsl/vsl_binary_io.h>. If you want to load or save a vcl_vector, the appropriate vsl_b_write and vsl_b_read functions will be in <vsl/vsl_vector_io.h>. Likewise for most of the other vcl classes. The vsl library contains the implementation of all of this.

When reading/writing by baseclass pointer, you need to include vsl/vsl_binary_loader.h.

If you want to load or save a vgl_point_2d, you will need to include <vgl/io/vgl_io_point_2d.h> and similarly for all other Level-1 VXL libraries. You will need to include the vgl_io library. For Level-2 libraries, the situation varies. If binary io has been defined at all for a level-2 library, it might be included in the library itself, e.g. the io functions for vpdfl_gaussian are declared in the same file as the Gaussian, vpdfl/vpdfl_gaussian.h>. Alternatively, it might be in a clip-on library in the same form as the Level-1 libraries above.

9.2.4 How to save templated objects

The situation for templated objects is the same as above, except that you need to ensure that the appropriate (templated) vsl_b_read and vsl_b_write functions are explicitly instantiated. This instantiation is achieved by placing a file in the relevant "Templates" folder.

An example template file, is shown below. It enables saving of a 2d array of "hjk_model"s (a completely made up plain class).

  // This is my_module/hjk/Templates/vbl_array_2d_io+hjk_model~-.cxx
  #include <vbl/io/vbl_io_array_2d.txx>
  #include <hjk/hjk_model.h>
  VBL_IO_ARRAY_2D_INSTANTIATE(hjk_model);

The vbl_io_array_2d.txx file contains the VBL_IO_ARRAY_2D_INSTANTIATE macro and the hjk_model.h file contains the io header declarations for a plain class.

Another example template file, allowing the saving of a vector of vgl_point_2d objects, is shown below.

  // file = my_module/hjk/Templates/vsl_vector_io+vgl_point_2d~-.cxx
  #include <vsl/vsl_vector_io.txx>
  #include <vgl/io/vgl_io_point_2d.h>
  VSL_VECTOR_IO_INSTANTIATE(vgl_point_2d<double>);

The vsl_vector_io.txx file contains the VSL_VECTOR_IO_INSTANTIATE macro and the vgl_io_point_2d.h file contains the io header declarations for vgl_point_2d<double>.

You should now be able to load and save templated objects with lines such as:-

  vcl_vector<hjk_model> hjk_model_vec;
  vsl_b_ofstream bfs("hjk_model_vec.bvl");
  if (!bfs)
  {
    vcl_cerr<<"Failed to open hjk_model_vec.bvl for output."<<vcl_endl;
  }
  else
  {
    vsl_b_write(bfs,hjk_model_vec);
    bfs.close();
  }

NB, the template instantiation files should be placed in your own libraries (ie here "hjk") to avoid creating unnecessary and unused versions of a given templated function.

9.3 Tidy Printing with vsl_indent

The utility functions and class in vsl_indent give a way of putting indentation into output streams to give more legible printed output.

If each class implements it's printing (print(os) or print_summary(os)) in such a way that at the beginning of each new line one inserts an indentation using

  os<<vsl_indent()<<"Rest of stuff.."<<vcl_endl;

and increases and decreases the current indentation for the stream with vsl_indent_inc(os) and vsl_indent_dec(os), then one can easily generate readable output for complex nested sets of classes.

It's use is best described by example:

  vcl_cout<<vsl_indent()<<"No Indent"<<vcl_endl;
  vsl_indent_inc(vcl_cout);
  vcl_cout<<vsl_indent()<<"1 Indent"<<vcl_endl;
  vsl_indent_inc(vcl_cout);
  vcl_cout<<vsl_indent()<<"2 Indent"<<vcl_endl;
  vsl_indent_dec(vcl_cout);
  vcl_cout<<vsl_indent()<<"1 Indent"<<vcl_endl;
  vsl_indent_dec(vcl_cout);
  vcl_cout<<vsl_indent()<<"No Indent"<<vcl_endl;

This produces output of the form

No Indent
  1 Indent
    2 Indent
  1 Indent
No Indent

Example of use in class output:

  class Fred
  {
   public:
    void print(vcl_ostream& os) const { os<<vsl_indent(os)<<"Fred's data"; }
  };
  
  vcl_ostream& operator<<(vcl_ostream& os, Fred const& fred)
  {
    os<<"Fred: "<<vcl_endl;
    vsl_indent_inc(os);
    fred.print(os);
    vsl_indent_dec(os);
    return os;
  }
  
  class Jim
  {
   private:
    Fred fred_;
   public:
    void print(vcl_ostream& os) const
    {
       os<<vsl_indent()<<fred_<<vcl_endl
           vsl_indent()<<"Jim's other data";
    }
  };
  
  vcl_ostream& operator<<(vcl_ostream& os, Jim const& jim)
  {
    os<<"Jim: "<<vcl_endl;
    vsl_indent_inc(os);
    jim.print(os);
    vsl_indent_dec(os);
    return os;
  }
  
  main()
  {
    Jim jim;
    vcl_cout<<jim<<vcl_endl;
  }

This produces output:

 Jim:
   Fred's data
   Jim's other data

If Jim were then included as a member of another class, Harry, one could get output of the form

  Harry:
   Harry's basic data
   jim1:
     Fred's data
     Jim's other data
   jim2:
     Fred's data
     Jim's other data

and so forth. The author humbly suggests that this makes the summaries quite readable.

9.4 Error Detection

IO is often prone to errors beyond the control of the programmer. In particular, files can be come corrupted, given to programs that can't read a new format, read on platforms that do not support large enough numbers.

vsl attempts to detect as many error conditions as possible. It prints an error message to vcl_cerr and sets the fail bit on the input stream. Any objects that were being loaded when the error occurred should be consistent at least as far as being able to delete the object safely.

During the opening of a binary input stream, vsl also checks for a schema version number, and magic number that confirm that the stream was written by vsl.

It is easy to detect the error condition as the example shows

  vsl_b_ifstream bfs_in(path);
  if (!bfs_in)
  {
    vcl_cout << "Could not open " << path
             << " for reading as binary IO" << vcl_endl;
    return;
  }
  vsl_b_read(bfs_in, my_obj);
  if (!bfs_in)
  {
    vcl_cout << "Unable to read my_obj" << vcl_endl;
    return;
  }
  bfs_in.close();

This document was generated on May, 1 2013 using texi2html 1.76.