This box searches only this space. The box at the upper right searches the entire iPlant wiki.

Skip to end of metadata
Go to start of metadata

In preparation of designing evaluation metrics for semantic reconstruction, below is a non-exhaustive list of ways fitting can fail.  The most interesting failure modes are those in which the raw structure matches well, appearing mostly correct using a purely geometric evaluation metric, but are still obviously incorrect to a human observer.

An ideal metric will both (a) identify these errors and (b) robustly evaluate good matches in the presence of these errors.   If identifying the error is too hard, the metric should at least be able to penalize it. 

Key:

Green: Ground truth
Blue: Model

Good match:

Type 0: Unary Errors

Part correspondence failure

Subtype (a): No match (to GT)


Subtype (b): Missing data

Type 1: Pairwise Errors

Correct part correspondence

Incorrect internal correspondence

Subtype (a): Early start/end

Subtype (b): Late start/end

Subtype (c): Internal inclusion



Model part includes extra unnecessary internal structure

Subtype (d): Internal omission

Model omits internal structure from the ground-truth.
Complement of “Internal inclusion” subtype

Subtype (e): Routing error



Ordering of internal correspondences is not preserved.

Comments:

  • Types (a) through (d) are identified by structure that is unaccounted-for  in either the model or the ground-truth.  It could be argued that they are simply geometric errors

Type 2: Correspondence errors

Partial incorrect part correspondence

Subtype (a): Many to one


(Many model to one GT)

Subtype (b): One to Many

Subtype (c): Mis-branch / Identity switch

Subtype (d): Node mis-location

 

Derivative subtypes:

  • Many-to-many – Combo of subtypes (a) and (c)

Comments:

  • Need many-to-many correspondences to identify most of these explicitly.
    • Geometry-level correspondence could do it
    • One-to-one part correspondences will penalize these implicitly. 
      • Subtypes (a) and (b) will result in skip/insertion type Topological errors, as well as early start/stop pairwise errors.
      • Subtype (c) becomes its own Topological error: generation reversal.
      • Node mislocation may be considered a type of pairwise error?
        • Or a separate category?

Type 3: Topology errors

Subtype: Node skip


Model omits one or more intermediate nodes between matches.

Subtype: Node insertion

Model inserts one or more intermediate nodes. 
(Complement of “Node Skip”)

Subtype: Late Birth


Model’s root node corresponds to non-root in GT

Subtype: Early Birth

Model’s root node corresponds to nothing in GT. 
(Complement of “Late Birth”)

Subtype: Wrong Subtree



Child occurs in wrong subtree

 

Comments:

  • These require unambiguous part-level correspondences.  Unrealistic?
  • No labels