AUUG-Vic/CAUUG: Summer Chapter Technical Conference '98



Java Object Serialization in parsable ASCII format

Enno Davids, Metva P/L.
Enno.Davids@metva.com.au





ABSTRACT
Object serialization, the act of writing an object onto some medium in a serial, byte-at-a-time manner is essential for communications systems and persistence of data in Object Oriented systems like Java. With the advent of JDK 1.1 we are provided with means to allow this to happen in an automatic manner. We examine ways to achieve the same effect under all versions of Java and we examine the benefits of using a parsable ASCII format to allow objects to serialize themselves in human readable/manipulable form.




Introduction

Object serialization is the new jargon for an old sport in computers. It is the act of writing a data object in a serial, byte-at-a-time manner. Being able to do this allows us to write such data objects to files and communications channels and is thus fundamental to most useful sorts of programs at some time or another. For this process to be most useful of course, it must also be reversible. That is, we must be able to recover an analogue of the original object by de-serializing the data stream. This is where serialization differs from say debugging output, where the output format is styled to be human readable and often structural information about the data is lost.

Traditional serialization is usually done in some binary manner, principally so as to minimize the amount of time spent transmitting data or the amount of space spent storing the data. These pragmatic reasons have driven the decision on how the serialization format should be structured.

This paper revolves around a format which is structured to be human readable. Such a format allows us to use the serialization mechanism for purposes for which it may not initially have been created. Examples are as a configuration format for a program or as an aid to debugging, by allowing us to examine all parts of the data object.


Historical Approaches to Serialization

Serialization is as was stated in the beginning, not a new invention. Indeed, with the advent of computer communications systems, serialization became commonplace. One of the best known examples of serialization is the RPC/XDR suite of protocols which featured not only a standard format for serialized data to appear in, the eXternal Data Representation or XDR, but also means to automate the serialization process (called marshaling in the jargon of RPC). RPC in fact goes further by not merely allowing for data to pass to and from serial format, but by allowing this data to take place in remote processing, hence the name Remote Procedure Call for the package as a whole. More recently, other players have entered the arena such as CORBA, ASN.1 and a slew of API's from Microsoft. Sun has in fact also provided a new entry in the field with the Java JDK1.1 which offers easy to use and quite sophisticated serialization facilities and a Remote Method Invocation package (RMI) which offers functionality intended to solve a similar problem domain to that the older RPC dealt with.

To understand the simplicity of the new Java offerings, lets look at what we had to do before it arrived on the scene. Traditional Object Oriented systems used their facility of bundling procedures with instances of data objects to allow us to bundle a serialization interface and a deserialization interface with each object. These interfaces are then responsible for reading & writing the instance from and to a serial stream. In general each procedure simply deals with each simple field of the data structure in a straightforward manner and for each complex data type, simply calls down to similar routines provided in that object to perform the same function. This method works for any language in fact, it is merely the case that the OO languages allow the bundling of data with routines which know about dealing with that data in a more natural manner and that in fact leads to some simplification and in fact ease of maintenance. These are in fact some of those benefits the OO pundits have been telling us about for years and they are in fact real.


JDK1.0

So, having waved hands and made some motherhood statements, its time to in fact have a look at some code. JDK1.0 offered no facilities for serialization as standard, but in fact all the building blocks necessary to implement the above method are provided. One facility in particular makes the whole thing come together very neatly indeed.

So lets look at a class with some serialization code in it. For Java, we start by defining an interface for our serialization routines. This means we can declare that classes which take part in the serialization process implement this interface. This allows the single inheritance Java class model of the application to remain intact in the face of the serialization code which is often an important consideration. Our interface code looks like this:


As we can see the interface does little more than tell the Java compiler which methods a class implementing the interface is required to provide.

This in turn brings us to the class itself. By way of example we have a simple class with only a few fields. The read method simply reads each fields in turn and the write method similarly writes each field in turn.


Note that we have a nested object as one field and we assume it to also implement the Serial interface. This allows us to simply use the same routines to cause it to serialize itself as part of this process. The need for the null constructor isn't immediately obvious but will become clear.

Its also worth noting that what we've shown is all that is required to be done on a per class basis. As was suggested, we simply deal with each field once in the read method and once in the write method. There is of course a bit more support code required but this is provided only once and can be reused for as many objects and streams as we need. To kick off the process and to manage the stream to which we write, we provide a class which manages the serialization for us. In fact, much of the subtlety is hidden by simply subclassing the JDK's DataOutputStream class. Initializing then consists of passing in a more fundamental stream type which we pass to our superclass and providing the method which starts off the serialization process. Note that, we also inherit a number of utility methods for writing primitive types to the stream from our superclass.


The serial format then consists of the class name written as a string (at line 16) followed by the fields we chose to write, in the order we chose to write them (by calling out to the writeObj() method of the object at line 17). Note that when we encountered our nested object in our class we chose to call out to the writeObj() method of the stream rather than that of the object itself, meaning that the nested class appears in the stream in an exactly congruent manner.

The choice of using the class name as a marker in the stream for the object data is an interesting one. Traditionally, we would have chosen to emit a packet type or some other token which stood in stead of the class name. RPC in fact had a registry for recording such values (and remember that in their model a class usually implied an action) and standards such as ASN.1 have complex syntaxes for deriving the names themselves. Typically, the packet type approach simply means that the object supplies a constant in some manner to the serializer which it emits.

On input then, the packet type is examined and the appropriate type of class is instantiated and told to deserialize itself from the remaining data in the stream. This usually manifests as a switch statement with one case for each of the packet types/classes we intend to serialize.


While this arrangement works satisfactorily, we can in fact do slightly better in Java. This is because Java provides a facility where we can instantiate a class for which we have the name at run time. In this case, the choice of writing the class name as a string starts to make more sense. This lets us write our top level Input handler like this:


And that's it. We have all the infrastructure we need to read and write classes, provided we prepare them in a manner similar to the example class we showed at the beginning. You'll note that this is where the need for a constructor with no arguments occurs. The newInstance() method being only able to trigger such constructors. The Reflection API which arrived with JDK1.1 offers the ability to build argument lists and thus trigger other types of cionstructor in addition to this service which is all the earlier version of Java had available. Fortunately, this service is all we need here.

For completeness, lets have a look at what we need to do in the calling code:


As with most examples, the bulk of the code above is irrelevant to the example itself. The things of importance are, creating the streams at lines 15 and 16. You can see that for simplicity we simply write the data to a file here. Line 17 is where all the magic happens and that's it. The file is closed and we have various bits of housekeeping but all the important things have happened.

As simple as the example and indeed the methodology is, there are a few observations we can make and a few conclusions we can draw.


Its also worth noting that so far, we haven't done anything with parsable ASCII. So far, this is just another binary serial format. It is possible when we're writing the fields to write extra data such as field names and to write the data in ASCII rather than binary though. This is more of an implementation detail than a feature or limitation of the methodology. All this leads us fairly naturally to JDK1.1's version of serialization.


JDK1.1

JDK1.1 offers a new serialization API built right into the facilities offered by the base classes. This support is offered by virtually all of the base classes (the exceptions being security/sandbox related). Unlike the methodology above, the mechanism used doesn't require the classes to be serialized to actively participate in the process. All such classes must do is implement Serializable which is an interface which serves as a marker only and doesn't require any methods to be implemented (in a manner similar to Java's cloneable interface).

In a manner similar to our serialization for JDK1.0, JDK1.1 provides master ObjectInputStream and ObjectOutputStream classes which operate at the top level to control the process as a whole. In fact, the code in the target class consists of nothing more than adding the implements clause to the class header and the client code looks remarkably similar to that we wrote above. We have in fact contrived through careful choice of names to make it no more than a simple substitution of Object for Obj in the example.

So how does the JDK1.1 implementation compare in terms of the issues we mentioned above? Lets revisit them one at a time.


All in all, JDK1.1 has a great serialization format. Its not without shortcomings and chiefly amongst these is an insensitivity to errors in the serial data stream, but this may not be of importance to your application or may be worked around with a reliable transport.

Once again though, its a binary format. Unlike our JDK1.0 methodology above, its harder to bend this one to the needs of an ASCII format. Happily though, JDK1.1 also offers us new API called reflection which allows objects to examine the structure of underlying classes at run time. This is principally provided as a mechanism to implement the Java Beans component API but is available for other uses.

Parts of the JDK1.1 ObjectStreams are built upon the foundation which reflection offers. This means we can use reflection ourselves to perform similar functions.


ASCII serialization

This brings us to the ASCII serialization code. By making use of the reflection API, we will aim to make the serialization code as low impact as the native JDK1.1 system is. We specifically don't want to have to instrument each class with code to read and write itself. We will allow such methods to exist so that classes may customize their input/output.

Like the other formats we will start by defining an interface. In common with the JDK1.1 approach, this is only required by those classes which want to offer overriding methods in place of the default actions of these classes.


Unlike JDK1.1 we have not provided an opt-out mechanism, but the Serializable interface itself could be used if we wish and an if (obj instanceof Serializable) test added to code as appropriate.

We'll start by looking at the output code as its simpler and because the skeleton it uses is the same as that of the input code (as we take the same steps, perform the compatible inverse I/O's in the same order) in order to read the same object which was written.

We start by noting that the AsciiObjectOutputStream is subclassed from the JDK1.1 PrintWriter class. This gives it access to a print and a println method which operates on the stream it is passed. This is the stream we pass into the constructor. Additionally the constructor also sets a small amount of local state, notably the amount of whitespace we indent the output by. The whitespace is principally for humans who may examine the stream and is incremented before and decremented after nested objects.


This routine is the top level of the serialization proper. Principally it writes a class header and arranges to emit all of the fields of the class object in question. Its worth noting that the fields in question must largely be public or at least accessible at the top level to be serialized. We'll see why in the next section of text.


This brings us to the first of the utility routines. In order to emit all the fields of a class we can either use the getFields() method of the reflection API or the getDeclaredFields() method. The principle difference between these two methods is that the getDeclaredFields() method provides a list of all the fields declared in a single class, whereas getFields() returns a list of all the accessible fields of a class and all its superclasses. Essentially then, getFields() is aware of the Java scope and visibility rules and applies the access restrictions to winnow the list of fields. In contrast, getDeclaredFields() tells us about all the fields but only for the immediate class. Superclasses must be dealt with explicitly. That is the purpose of this utility routine. To traverse the superclass chain and iterate along the fields of each superclass.

The reason for choosing to deal with all the fields of a class rather than just the accessible ones is that this way we expose the fact that we cannot serialize all the fields of a class. The JDK1.1 serialization doesn't suffer from this limitation by using a native (that is machine code) method which bypasses such access restrictions. If we were to simply process the list of accessible fields, we would silently fail to process such fields. This is also the reason why we require all the fields of classes we wish to serialize this way to be accessible, i.e. usually public. For the purposes of debugging and configuration we are ostensibly creating this facility for, this is not unreasonable. There may be circumstances where it does not fit the model though.


This is the workhorse routine. Principally, it consists of two structures. The first is a loop which iterates along the fields of the class. The second is a series of sequential if statements which deal with the sorts of fields we may find.

Firstly we deal with the primitive data types supported by Java. These are the low level types and must be handled specially. The next special handling is for array types. This is forced upon us by the way the reflection API deals with such data. As each array is itself composed of one of the other underlying types we must then handle that underlying type for each element of the array. For simplicity we only handle single dimensional arrays for the moment. Lastly, we handle objects which are themselves classes. These are treated as Strings, for which we have a special encoding, and other classes for we recursively call ourselves.


Next we have two routines which deal with a single field and a single element of an array respectively. The array handler simply extracts the array element and then hands it off to the single field handler.


Finally, we come to two routines which complete the handling of fields. One is the routine which extracts the type name of the field. Once again arrays require special handling. Lastly, there is a routine which encapsulates the special handling required for Strings.


Thats all that we require to serialize our objects. Next we look at the flip side of the process. Most of this is simply running the plumbing backwards inside the same framework. What differences exist are completely due to the different processes. Parsing an input stream presents some different conceptual and practical problems to writing an output stream. Many of those differences are encapsulated in the parsing support code at the end of the class.


Not a lot to say here that wasn't said above and isn't said in the comment below. Over and above that, there isn't anything really complex here. These parsing routines are fairly plain vanilla and don't do anything special or tricky.


So having seen the bulk of the code, lets have a look at how we use it. This example is similar to the one we had at the start of the paper. It differs only in that it reads its output back in. It then re-writes the newly read classes a second time to a new file. This allows us to use the UNIX diff utility to compare the files. Any differences imply bugs in the process.


Finally lets have a look at the output. Remember there's nothing too special about the format. It's only meant to be human readable and unambiguous enough to read back in. This contrasts nicely with the toString() diagnostics in the comment lines at the head of each class.


So where does that leave us? What we have so far is far from perfect. It does allow us to achieve the aims we set ourselves and by this yardstick I am happy to declare ti a success. We can use this code to output an arbitrary class and usually a complex groups of classes for debugging purposes. With care, humans can edit together configuration information which can be parsed by this code. Indeed, the output is about as good as we can expect it to get. It is certainly sufficient for our purposes.

Clearly, the greatest shortcoming of the current code is that it is finnicky about its input. That is, the input code expects the right fields in the right order with compatible data. That's not really flexible enough to make most humans happy for a long time. The next version of this code can be expected to build keyed tables of the input and search that table/those tables for the input required. Missing input would then simply imply acceptance of default values from the default constructor. This would expand the role the null constructor from that of merely constructing an object to constructing one with known defaults. Humans coding inut manually (i.e. editing a configuration file for instance) would then be ina position to accept those defaults or override them.

The ability to elide uneccessary information from the stream, when authored by a humans at least, is the principle motivation for writing field names in such a format at all. While they may be used for checking (notably absent in this implementation) they are more useful as a lookup key for the assignment of data to the fields it belongs in.

The other notable shortcoming of this code is the inability to handle multi-dimensional arrays, a simple matter of recoding some of the handling and some exposure to the problems we earlier alluded to about serializing object graphs with cycles and loops in them. Once again, these facilities were not seen as central to the felxible configuration format we were principally seeking. Adding some support to correctly serialize and recover such objects is not onerous, merely another complicating factor we did need specify at this point in time.

It is also interesting to note that we have written enough data that with access to the classes which compose the Java compiler (a Java application itself in most environments) we could compile the class as we receive the data and only then hand the stream to the newly compiled class to deserialize. What processing can then sensibly be undertaken is another matter, but the objects can be recovered from ASCII streams as in this case at least, the stream is essentially self describing.

That's about it for now. More work will be happening on this as if nothing else, we have a use for this at work and hence have an interest in pursuing it. The changes we forsee are the relaxation of input requirements discussed above and perhaps a formalization of the grammar of the output stream, which till now, has developed in an ad hoc manner and bears the scars of this process. That's about all I have planned at this stage and that in turn reflects the basic fact that this is a good start.

The code shown here should be available and with a little luck later versions may also be available. If you can't find them, and believe they are or should be, drop me a note. If you've found this useful or feel there are facilities missing which woudl enhance the code, I'd be interested in hearing about it too. Other than that, Good Luck!


Acknowledgements

It would be remiss of me not to acknowledge the support of Australian Business Access (ABA) and the colleagues with whom I work on the SecurEcommerce project there. Their support has been instrumental during the development of this code and in fact the JDK1.0 serialization code shown here is a distillsation of the work of many individuals at ABA.


References

[1] Java in a Nutshell, 2ed.
David Flanagan,
O'Reilly & Associates, Inc. ISBN 1-56592-262-X
[2] Java Reflection API documentation.
http://java.sun.com/products/jdk/1.1/docs/guide/reflection/index.html

Sun Microsystems Inc.



Document last revised: Thu Feb 12 11:01:40 EST 1998



Author's papers