Perceiving Systems, Computer Vision

Modeling appearance change in image sequences

1997

Conference Paper

ps


As Gibson noted, the world is made up of surfaces that ``flow or undergo stretching, squeezing, bending, and breaking in ways of enormous mechanical complexity.'' These events result in a wide variety of changes in the ``appearance'' of objects in a scene. While motion and illumination changes are examples of common scene events that result in appearance change, numerous other events occur in nature that cause changes in appearance. For example, the color of objects can change due to chemical processes (eg., oxidation), objects can change state (eg., evaporation, dissolving), or objects can undergo radical changes in structure (eg., exploding, tearing, rupturing, boiling). In this paper we formulate a general framework for representing appearance changes such as these. In so doing we have three primary goals. First, we wish to ``explain'' appearance changes in an image sequence as resulting from a ``mixture'' of causes. Second, we wish to locate where particular types of appearance change are taking place in an image. And, third, we want to provide a framework that generalizes previous work on motion estimation. We propose four generative models to ``explain'' the classes of appearance change illustrated above. A change in ``form'' is modeled as the motion of pixels in one image to those in the next image. An image at time t+1 can be explained by warping the image at time t using this image motion. Illumination variations may be global, occurring throughout the entire image due to changes in the illuminant, or local as the result of shadowing. Here we model illumination change as a smooth function that amplifies/attenuates image contrast. By comparison, specular reflections are typically local and can be modeled, in the simplest case, as a near saturation of image intensity. The fourth class of events considered in this paper is iconic change. We use the word ``iconic'' to indicate changes that are ``pictorial.'' These are systematic changes in image appearance that are not readily explained by physical models of motion, illumination, or specularity. A simple example is the blinking of the eye shown above. Examples of physical phenomena that give rise to iconic change include occlusion, disocclusion, changes in surface materials, and motions of non-rigid objects. In this paper we consider iconic changes to be object specific and we ``learn'' models of the the iconic structure for particular objects. These different types of appearance change commonly occur together with natural objects; for example, with articulated human motion or the textural motion of plants, flags, water, etc. We employ a probabilistic mixture model formulation to recover the various types of appearance change and to perform a soft assignment, or classification, of pixels to causes. We use the EM-algorithm to iteratively compute maximum likelihood estimates for the deformation and iconic model parameters as well as the posterior probabilities that pixels at time t are explained by each of the causes. These probabilities are the ``weights'' illustrated below and they provide a soft assignment of pixels to causes.

Author(s): Black, M. J. and Yacoob, Y. and Fleet, D. J.
Book Title: Advances in Visual Form Analysis
Pages: 11-20
Year: 1997
Month: May
Publisher: Proceedings of the Third International Workshop on Visual Form

Department(s): Perceiving Systems
Bibtex Type: Conference Paper (inproceedings)
Paper Type: Conference

Address: Capri, Italy

Links: projetct

BibTex

@inproceedings{Black:AVFA:1997,
  title = {Modeling appearance change in image sequences},
  author = {Black, M. J. and Yacoob, Y. and Fleet, D. J.},
  booktitle = {Advances in Visual Form Analysis},
  pages = {11-20},
  publisher = {Proceedings of the Third International Workshop on Visual Form},
  address = {Capri, Italy},
  month = may,
  year = {1997},
  doi = {},
  month_numeric = {5}
}