Monday, April 4, 2011

The Many Images Of LWUIT

When we initially came out with LWUIT we inherited some baggage from J2ME in the form of several image types: loaded, RGB (builtin), RGB (lwuit) and Mutable. We quickly added Indexed and StaticAnimation. We later on added then replaced SVG support (then replaced it again) but the image landscape remained relatively static until this past year.
In LWUIT 1.4 we finally introduced EncodedImage which to us was a complete epiphany in how images "should" work in mobile devices. We now also have an even newer approach to SVG, Multi-Image & Timeline. And I won't even go into LWUIT4IO which has the FileEncodedImage and FileEncodedImageAsync options...

I will now try to explain the pros/cons and logic behind every image type and how its created:

  • Loaded Image - this is the basic image you get when loading an image from the jar or network using Image.createImage(String)/Image.createImage(InputStream)/Image.createImage(byte[], int, int).
    In MIDP calling getGraphics() on an image like this will throw an exception (its immutable in MIDP terms), this is true for almost all other images as well.  This is strictly a MIDP restriction and might not apply for all platforms.
    The image is encoded based on device logic and should be reasonably efficient.
  • RGB Image (internal) - close cousin of the loaded image. This image is created using the method Image.createImage(int[], int, int) and receives ARGB data forming the image. It is usually (although not always) a high color image. Its more efficient than the LWUIT RGB image but can't be modified, at least not on the pixel level.
  • RGBImage (LWUIT) - constructed via the RGBImage class constructors this image is effectively an ARGB array that can be drawn by LWUIT. On many platforms this is quite inefficient but for some pixel level manipulations there is just no other way.
  • IndexedImage/StaticAnimation - Both are deprecated and replaced by the more efficient and effective EncodedImage/Timeline. IndexedImage's allow storing images using a palette array and a byte per pixel (indexed images must not contain more than 256 colors). Static animations add to that by animating frames using a line difference algorithm. 
  • EncodedImage - created via the encoded image static methods, the encoded image is effectively a loaded image that is "hidden". When creating an encoded image only the PNG (or jpeg etc.) is loaded to an array in RAM. Normally such images are very small relatively so they can be kept in memory without much effect. When image information is needed (e.g. pixels, dimension etc.) the image is decoded into RAM and kept in a weak/sort reference.
    This allows the image to be cached for performance and allows the garbage collector to reclaim it when the memory becomes scarce.
    Encoded image is not final and can be derived to produce complex image fetching strategies such as lazily loading an image from the filesystem.
  • SVG - SVG's can be loaded directly via Image.createSVG() if Image.isSVGSupported() returns true. When adding SVG's via the resource editor fallback images are produced for devices that do not support SVG.
  • Multi-Image - This is seamless to developers who receive a multi-image as an EncodedImage. In the resource editor one can add several images based on the DPI of the device (one of several predefined family ranges). When loading the resource file irrelevant images are skipped thus saving the additional memory.
    Multi-images are ideal for icons or small artifacts that are hard to scale properly. They are not meant to replace things such as 9-image borders etc. since adapting them to every resolution or to device rotation isn't practical.
  • Timeline - Timeline's allow rudimentary animation and enable GIF importing using the resource editor. Effectively a timeline is a set of images that can be moved rotated, scaled & blended to provide interesting animation effects. It can be created manually using the Timeline class.

All image types are mostly seamless to use and will just work with drawImage and various image related image API's for the most part with caveats on performance etc. For animation images the code must invoke images animate() method  (this is done automatically by LWUIT when placing the image as a background or as an icon! You only need to do it for drawImage code).
All images might also be animated in theory e.g. my gif implementation returned animated gifs from the standard Loaded Image methods and this worked pretty seamlessly (since Icons's and backgrounds just work). To find out if an image is animated you need to use the isAnimation() method, currently SVG images are animated in MIDP but most of our ports don't support GIF animations by default (although it should be easy to add to some of them).

Performance and memory wise you should read the above carefully and be aware of the image types you use. The resource editor tries to conserve memory and be "clever" by using only encoded images, while these are great for low memory they are not as efficient as loaded images in terms of speed. Also when scaled these images have much larger overhead since they need to be converted to RGB, scaled and then a new image is created. Keeping all these things in mind when optimizing a specific UI is very important.

    0 comments:

    Post a Comment