Last semester (fall 2012), the SLIC team released a brand new version of the SLIC website (http://slic.arizona.edu/).

From Oct 25, 2012 to approximately Feb 8, 2013, Alexander Danehy, the main student developer in charge of the SLIC Portal, added a lot of tweaks to the main interface. One of the big changes was the inclusion of course lectures, and the subsequent ability to browse our video collection by different categories.

We've installed jsTree as the primary browsing/filtering system. The tree allows fine-granularity selection from our predetermined hierarchy. It's worth noting that we've discussed, tried and tested a lot of options along the way:

a.     drop-down checkboxes

b.     walking menu

c.     walking checkboxes

d.     walking, no-click menu

e.     single-selection drop-down lists (previously, user  could only select from categories)

A few changes were made to the video page as well:

  • added the ability to jump to a slide by its number
  • added information about the current number of slides shown
  • show segmentation data for slides - whether or not a slide is active, its time in the video (or multiple times, if it was shown repeatedly), or if isn't shown at all
  • added a visual indicator of a slide not being shown in a video (on the video, as well as in the SLIC view on the main page)
  • recruited a new team member to work on the multiple slide appearances
  • modified advanced search - no longer an arrow but a full-text button; added sorting

In an effort to make searching even more powerful, we looked into automatic speech extraction and transcript generation. One of the students, Kavinfranco Devadhas, who joined the project in the fall, began researching the CMU Sphinx project -- an open-source speech recognition software. Getting it to work and modifying it to suit our needs proved to be a challenging exercise. However, he was able to make some progress: he successfully added slide words to the dictionary to improve recognition by about 10-20%. He also worked on a publication idea, which required using slide words to automatically tune the Sphinx parameters. The preliminary results showed that this could be a promising direction. Unfortunately, due to a job offer, he was not able to continue this work in the spring semester. The project is in a state at which the next student can continue to build it up and test new ideas.

Another focus of the SLIC team was the development of a dedicated mobile app. Benjamin Dicken joined the SLIC project at the very end of the spring 2012 semester. He spent the summer trying to revive the Android app written by the former SLIC member, Steven Gregory. Unable to get it to work (due to the drivers' incompatibility and a wide variety of other problems), he continued to develop the mobile API written for the Android app that was originally further extended by another former student, Derek Leverenz. Instead of starting a new Android app from scratch, we decided to leverage Ben's Objective-C knowledge, and by the start of the fall semester, we had a proof-of-the-concept prototype interface for iOS.

At the end of November, we met with Wayne Peterson, who is the UA assistant director of Web / Mobile Services. We discussed the possibility of putting the SLIC app in the university enterprise development repository to allow for the internal testing and the future release of the app. We got a very enthusiastic response. Unfortunately, due to Ben's graduation in the spring of 2013, and an overwhelming number of other commitments, he was not able to continue working on the app in the spring. The project was put on hold until the team finds a new developer. In the mean time, all design decisions related to the main website are made to make sure that the website is usable by the mobile users (as soon as we switch to the HTML5-based video player).

At the end of the fall semester, the SLIC team got 3 new members: Haziel Zuniga, Mark Fischer and Matthew Burns. Stay tuned to learn about their projects and contributions.

Macports patch

This is a minor item but I'm proud of it.  Part of my research support is to advance vision algorithms like my work on locally-linear structures, for example the trail finding algorithm that Scott Morris, Kobus Barnard and I have developed.  This algorithm is, we believe, widely applicable to stringy-looking things (in the absence of tip growth), and one domain we've looked to cross into is that of neuron phenotyping.  So the IVI lab has a bit of collaboration with the Restifo Lab in the Department of Neuroscience.  We've developed image-inference software for them before, and we still provide a bit of support.

Recently they've decided they would like to run their custom software themselves.  As I've said before, we research people often (unintentionally) make that somewhat difficult:  we run our own code, so we know just how it has to be configured and run.  In this case, the configuration was the snag:  they want to run on a Macintosh (I hear that's a popular brand nowadays).  Our code for them required Gnuplot, and specifically an older output option for Gnuplot.  Well, you can't get that option on a Macintosh.

You can't -- until now!  Although I had little experience developing software on a Mac, I found out about Macports, which is a framework for installing things like Gnuplot on Macintosh, and I learned enough to modify the Gnuplot port, stuff in the missing option and get it working.  Furthermore, I submitted a patch and ticket to Macports.org.  Six days ago my patch was incorporated into the trunk Portfile.  So, this little support request has the wider consequence of advancing (in a small way) the evolution of Macintosh open-source software.

Pollen RCN wrap-up

So this is from a few weeks back, but still worth a mention.  For the month of February, I worked with a small group (Martha Narro, Ravi Palanivelu, Nirav Merchant, Salika Dunatunga, and my advisor Kobus Barnard) to develop a good demo of iPlant's Bisque platform in action.  Martha and I worked hard writing a clear, focused tutorial, reminiscent of the "lab protocol" format that Ravi recommended as a good way to speak to biologists in a way that builds on their existing knowledge.

During that same period, I made a large number of minor changes to tweak and streamline the Pollen Tube Tracker.  For example, the input parameters were rather cryptically named, at least from the point of view of a plant biologist.  I added more help text and altered the wording to communicate their meanings better.  One instance of that is the "spot size" parameter.  It was formerly specified in pixels, and instead we changed it to microns, and the module extracts the pixel-to-micron conversion factor from the image metadata.  Sometimes these changes have unanticipated consequences.  When I made the above change, soon the tracker started to take days and days to run.  Why was that?  It turns out if the user sets the minimum spot size smaller than the equivalent of 2 pixels, then spot detector goes bonkers -- it sees one-pixel spots everywhere.  That's not really a problem for the spot detector, but its excessive output consequently clogs up the tracker.

So that's why user testing is important.  Research people like me tend to write interfaces for a single user (oneself), and it's a bit humbling to have to face again the challenge of HCI, when the user cannot literally read (i.e., share) the mind of the developer.

The goal was the March 2 RCN meeting, and iPlant had, I think, a good showing.  I think all the iPlant representatives talked to a ton of people interested in various computing requirements.  Many of the questions I received had the common theme of "That looks similar to what I need, but . . ." and in all cases, the specifics of user data, or image models, were very significant.  One colleague of Ravi's had color images, and wanted to track both pollen tube tips and pollen tube nuclei.  Another wanted to track locally-linear structures (actin fibers) that are moving in all kinds of crazy ways:  squirming, twisting, growing and breaking.  I bet in 100 years these sort of adaptations will be easy, but we aren't there yet.

Plotting in Bisque

So I've been working on my Bisque module -- specifically, I'm trying to make the output of the pollen-tube-tracker module immediately useful, at least a bit.  (The module generates comprehensive output in CSV or XML form, but no one can read all that at a glance.)  Specifically, what our pollen expert requested was a plot of pollen tube velocities, versus time.  That sounds like a very reasonable request, no?  He sketched out for me a simple cartoon, with one trace per tube, with tube velocity as the ordinate against time as the abscissa:


It turns out, that's not so easy.

How Bisque Works

One reason it isn't easy is because a Bisque module is doesn't get to speak directly to the users.  I want to explain what I mean, so here's a little sketch of how the architecture looks from my perspective:


You, the biologist user, are running a browser and working through the interface Bisque provides. In this diagram, I'm "mod.py," a Python script that gets invoked every time the user picks my module to perform an analysis. My job is to mediate between Bisque and the binary executables (represented by engine.bin) that do the hard work. In the pollen-tube case, there are three binary executables, and they expect to run in a Unix environment, with their input in the form of named files (named just so) in a local filesystem. The engines need command-line arguments, and if anything goes wrong they send a message in English to standard output. Whereas Bisque, instead, gives me the URL of the input files. It gives me user parameters in a slightly different format (e.g., sizes could be in microns rather than pixels). It doesn't understand English error messages.

My job (as mod.py) is to be the bridge: download the images from the URLs and save them locally. Adapt the parameters. Run the engines and interpret their outputs. Detect errors. Translate engine output into a Bisque-acceptable response. Like any translator, this means mod.py has to understand both parties' "languages" and sometimes do a bit of extra explaining, but it's not that complicated. The Bisque developers have provided code to help out with the hardest parts, like that of downloading images and uploading XML output. Thus mod.py is roughly 700 lines of python -- not very long.

The high-level interface to the module is defined by a module-definition file, which is mdef.xml in the above diagram.  It defines a module's inputs and outputs.  This is how, for example, you the user can give me, the script, a bunch of numeric parameters to control the tracker.  Inside mdef.xml there is a list of input parameters; when you choose my module, Bisque reads mdef.xml and lets you tweak the parameters before clicking the "run" button.  When "run" fires, Bisque shunts those parameters over to mod.py.

So Where's My Chart?

It's not enough to say that mdef.xml spells out the input parameters for mod.py; more accurately, it acts as a contract between Bisque and mod.py.  The output from mod.py, which is exclusively XML, must conform to the structure delineated in mdef.xml.  So it defines the output interface (from mod.py and to Bisque), too.  Bisque by default supports a few kinds of output. The output XML can define an overlay of aggregate graphical primitives (called GObjects) on the input image. You can produce some text-based summary output. And Bisque will always let you have the raw XML, if you ask for it.

If you want a chart, it gets a bit tricky. You can make simple charts by specifying a single xpath, which will extract a single vector of variables from the output.  You can plot them, or draw a histogram, or a few other things, but mdef.xml is essentially an interface.  An interface should answer the question "what is the output?" rather than "how do we make the output?"

Furthermore, mod.py cannot really generate a plot easily.  Its job is to bridge between Bisque and the engine, not to offer reinterpretations of the engine output.  (In fact I violate my own advice here a bit, because my module does augment the engine output with a reinterpretation that lacks the time dimension.  I call the outputs "time-indexed" and "time-collapsed," because it's easier to visualize the time-collapsed output in Bisque, although the time-indexed one is more complete.  It's kind of a hack.  If possible I'd like to eliminate that behavior.)

One of the hallowed rules of software design is to separate interface from implementation.  In this case, that means separate the user-interface (the charts) from the XML output that is implemented by mod.py+engine.bin. This same principle is one part of the Model-View-Controller architecture: the view and the model should be decoupled.  Long story short: it's a dumb idea to shoehorn the plot into mdef.xml or mod.py. We need a new entity: please welcome plot.js, something called a "renderer" by the Bisque devs.  As the name suggests, it's a chunk of javascript that runs in your browser, and reinterprets module-output in the form of a plot.  It can answer the question, "How do we make this part of the output?" The only requirement is that mdef.xml must cite plot.js as part of its user interface.

Not quite there

After cribbing the javascript renderer from RootTipMulti I was eventually able to produce a plot. It still looks terrible, but I'm making progress:

Let's for a moment focus on the positives. I had never written any JavaScript before now. I was unfamiliar with the Bisque interface to javascript renderers until now. I've successfully adapted this module to scrape out tube velocity from the module output. Debugging was difficult here because the JS isn't stored on my server, but it's shipped by Bisque to the end user, so I have to re-run the module for every change. The xpaths generated by the module are interpreted by Bisque, from which the plot is produced, so I'm submitting callback functions and it's all kind of confusing. Yet we seem to have the rudiments of a plot here! I was very glad to see the above plot, after many hours of obscure refusals from Bisque.

However, my 4th grade teacher would weep at the sight of this chart.  With the missing labels, the missing units, the numeric formatting, the legends -- it's a trainwreck.  So, that's what I've got to improve next, and as usual with Bisque, I don't really know what to do. Apparently the plotting is performed by a JavaScript library called EXT-JS, and obviously its defaults are lousy, so I've got to learn how to make it behave better.

Seed edges (updated)

As I mentioned earlier, we are partway along building a generative model for flatbed-scanner seed images.  We have a silhouette component to the model, but we also want to use edge information, since edges are going to provide essential clues to where one kernel ends and another begins (even if they touch).  To that end, I've got a Canny edge detector working on seed images.  Here's a picture:

The input to the edge detector was a silhouette of seed kernels.  The picture here shows randomly colored edge pixels, i.e., the edge detector has identified which pixels are edges of shapes, and (though you cannot see it), it has stored the edge direction, as well as the location, at each colorful point.

One difficulty here is that there is no perfect edge detector; this one includes a number of parameters (thresholds and a scale factor) that help it distinguish true edges from image noise.  Those factors depend on image characteristics like seed size.  If one tries different seeds, or a seed sample with more dirt, or if one changes the scan resolution, the edge detector might not do as well.  We will have to pay attention how to make the system robust enough to be useful.

The next step is to integrate the edge information into the image model, i.e., into the likelihood function.  That's more a problem of modeling than programming, but fortunately it's an issue our lab has been working on for quite awhile.

Update (17:58):  Here's a visualization of the edge direction of those edge pixels, as a tangent line, in case the above was not clear.  Each edge point has not only a location, but also a direction.

Delete me
Bisque module works

The pollen-tracking module for Bisque is good enough to push into the world.  I believe it works well enough to be useful, and I hope the end users will concur.  I look forward to their feedback about how it can be improved.  I myself know a number of improvements I'd like to see.  But software can eternally be improved, and I know iPlant wants to launch Bisque on November 1, so let's call version 1 complete.  Kobus and I have talked, and we need to coordinate with Nirav and Ravi how to wrap up v1 nicely.

I have noticed that the browser used for Bisque matters.  The experience is nicer on Chrome than Firefox.  In particular, the graphical rendering and the user interface are a bit quirky in Firefox, but not in Chrome (which I believe is the developers' primary dev platform).

Bisque -- much better now

Ok, The Bisque situation is much improved since I last posted.  It seems obvious now the "right" approach all along was to install a separate Bisque engine on bovary itself.  Wish I'd perceived that earlier.  Anyway, now I can develop modules there without disrupting anyone else, and I don't have to wrestle with firewalls or configuration riddles-of-the-sphinx.

I've got the 4D module alive there, i.e., it starts; but I've encountered (or re-encountered?) a bug in the C++ inference code.  So we begin another round of getting the CVPR code fixed up, with the original developer.  Hope that doesn't take too long.

New SLIC blog post

The past couple of weeks the SLIC team has been busy increasing the security of the interface and ensuring its robustness against SQL injections and other malicious attacks. In the process, we have modularized some of the components, fixed minor bugs and added new features. As soon as we can reliably receive feedback from the site, we will be ready to release this new version to the world.

Currently I'm trying to get my local version of Bisque configured so that its "engine server" speaks to its main "server," both of which interact through different ports (27000 and 8080, respectively).  I think the two processes are not communicating.  I tried the simple test of copying the generic "MyData" and "ShExample" modules into the same directory, named "foobar" and "xyzzy."  With quite a bit of cajoling, I got the engine server to recognize foobar but not xyzzy, so that they turn up if one does a GET from localhost:27000/engine_service.  Unfortunately the main server does not even seem to be heeding what its putative engine is providing.  I've made as many intelligent guesses as I can, restarted the servers over and over and over, and even tried quite a few unintelligent guesses in desperation, but nothing is working.  Te documentation is no help, and I'm pretty frustrated.  I could succeed if I had a locally-running Bisque server, but it's a herculean task to get it configured correctly.  I could succeed if I had a locally-running Bisque engine-server that is registered with the one on bovary, but it's impossible to convince anyone with authority to open an external port in the firewall.  There has to be a better way.

Seed affinities

Good news and bad news.  First the good news.  The affinity sampler works pretty OK for different isolated seed silhouettes.  Here are a few examples with lima beans, barley, and wheat (respectively):


   

I've also made a few adjustments to from the earlier version, so that (1) magnification in X and Y directions is correlated, and (2) likelihood and prior seem to be a little more in balance, I guess.

  1. Correlated magnification:  this change represents additional prior knowledge that hadn't occurred to me yet.  The affinity model permits magnifying the X and Y axes by different amounts, which I think is appropriate, but I had let those magnifications move completely independent of each other.  That left the door open for the sampler to project one of the seeds into a thin filament.  However, when a seed is small in one direction, doesn't that suggest it is likely to be small in the perpendicular direction?  I added a correlation, therefore, to the two magnifications (which is kind of a pain because independent variables are so much easier to work with).
  2. Balancing prior and likelihood.  This is a subject that Kobus tells me often arises in a statistical model, but is somewhat difficult to specify precisely.  In general, if and when the likelihood and prior get into a tug-of-war, we want them to be roughly matched in strength.  We don't want one to be able to overwhelm the other.  Unfortunately it seems to be something of an empirical art how to achieve this balance. It would be nice if we had a principled way to find the best balance.

To show an example of what I'm talking about, below I show the lima bean example with two alternative models.  In the first one below, the prior does not correlate the two magnifications, so the bean projection gets super-skinny.  Also, the likelihood sort of overpowers the prior, so the sampler is "terrified"* of expanding the proposal outside the bounds of the red bean.  It gets kind of stuck, therefore, and wastes a lot of time on weak proposals, and doesn't get to a plausible magnification until it's too late.  In the second example, the likelihood is underpowered, and so the sampler "barely listens" to the data; the effect is aimless wandering.
 

* hope you don't mind me anthropomorphizing a bit.

So that's the good news.  The bad news is that the model is incomplete.  Kobus has convinced me that I need to add edge information into the likelihood.  The above is working nicely on isolated seeds, but touching seeds will need more clues about how to fit a template to the data -- a lesson that Kobus and Joe Schlect discovered working with furniture images.

Here's a picture of me stressing out a simple statistical inference engine.  I'm asking it to find an affinity to match up a triangle and square.  Of course, you and I know that a triangle and a square cannot ever be related by an affinity:  straight lines in become straight lines out (by definition), so triangles in become triangles out -- never a square.  Thus, the poor program is squirming this way and that to make the best of an impossible situation.  It's proposing all kinds of affinities, each of which projects the blue triangle (the original) into a new location (the green triangle).  The green triangle dances around as the program accepts new proposals.  Eventually it almost stops moving as the acceptance ratio drops.

To restate this:  I didn't animate the green triangle, but I wrote five components that did:

  • I made up a simple likelihood function that measures "quality of fit" by counting uncovered pixels, i.e., the red and green pixels above.  (Black and blue pixels do not count, nor do yellow pixels, which represent alignment.)
  • I chose affine transforms as the relationship between the given data (which are represented above by the red square and blue triangle).  Furthermore, I constructed a prior function that represents a-priori assumptions about the transform: it can magnify or shrink but not too much, it can rotate any amount, it can skew but only a little, etc.  In other words, some affinities are more probable than others.
  • I designed a proposer function that takes an affinity in, and tweaks it randomly.  It makes a small random change to all seven parameters.  It also can randomly flip the shape in the horizontal or vertical direction.
  • I implemented a sampling method called the Metropolis algorithm, which is one of the simplest MCMC sampling algorithms.  It can be thought of as an optimization procedure.
  • I typed in the coordinates of the square and triangle, gave them to the program, and said go.

For a little dramatic interest, I initialized the affinity so that the blue triangle maps onto itself (visible only in the first frame), but it's not really a sensible choice for compact shapes.  In practice, one would initialize so that the centroids are aligned.

Affinities

Yesterday and today I've been busy writing code to implement a model and method describing an affinity between two sets of seed pixels.  Here's the basic idea:

  • Two seeds out there, s1 and s2, have somehow produced a list of pixel locations:  s1 = ?{(x1,y1),(x2,y2), . . . , (xN,yN)} and s2 = {(u1,v1),(u2,v2), . . . (uM,vM)}
    .  We don't use color or shape, just pixel locations: s1 and s2 are silhouettes.
  • Our model states that these two data are explained by an affinity A.  An affinity is a restricted kind of transformation on pixels.  We postulate that A (which is unknown) will make s1 look like s2.  In other words,  A(s1) = s2 + noise.
  • Taking a generative, Bayesian approach, we want a statistical model p(A,s1,s2) = (k)L(s1,s2; A) p(A).  L is a function called the likelihood -- in this case, the likelihood of the seed data, presupposing we know A (which we don't).  p(A) represents the prior plausibility of some A.  Finally, k is constant of proportionality that we ignore.  We take this approach because if we try a sufficient bunch of candidate affinities A1, A2, A3, . . ., plugging each one into p(A,s1,s2), we often can find an affinity A' that yields a relatively high value for p(A',s1,s2).  That best answer A' can be said to "explain" the data; it's as if we've developed an "understanding" of the data.
  • The above story has a few blank sections still:  the likelihood L, the prior p(A), and the proposal process for A1, A2, etc.
  • For the likelhood, let's start from a nice big constant number and deduct 1 every time there's a pixel in s2 that is absent in A(s1), or vice versa.  So if all their pixels align, there will be a high score (no deductions).
  • For the prior p(A), I've decomposed A into seven more-or-less independent factors (discussed earlier) -- A might magnify or shrink in 2D, but not by a lot.  "Not by a lot" sounds vague, but we can be more quantitative using Gaussians -- say, along an axis, by a factor that on average is 1 and 68% of the time is no more than, say, 40% above or below unity.  Where did I get that?  I made it up, but I'm trying to draw from empirical observations of seeds, and I think I can be a bit sloppy in my guessing.  Likewise I can roughly model my prior knowledge of seed variation by casting it in terms of shear, translation, and rotation.  Rotation is easiest to model, because it's a "known unknown":  the seed could be rotated in absolutely any direction.
  • One annoying detail is that if A magnifies, and if the pixels of s1 are at least 1 unit apart, then some pixels of A(s1) will be more than one unit apart!  This could make s1 look like stripes or splinters!  So I wrote code to augment s1 with pseudopixels between the actual pixels, which will fill in the gaps and prevent splintering.
  • The last major piece is the proposal process.  We will use an approach in the large family of Markov-chain Monte Carlo methods, i.e., its proposals are a mixture of memory and randomness.  A Markov chain partially retains what else has been tried recently, and how well it worked, but then proposes a stochastic increment from that state -- it can be described as an "exploration" of a space.  In this case, a seven-dimensional space.
  • As is always the case with real code implementations, there are other chores:  developing a "language" for representing seeds in a text file, writing code to interpret and produce that language, writing test code to make sure the individual components actually do what I intended, and writing code to display what is happening in text or graphics.
Two things

I worked on two iplant-related things today, Bisque and Seed-counting.

  • The big Bisque goal is to get the pollen-tube tracking algorithm, tentative dubbed "ebtrack," running as a Bisque module.  To that end, I've got the back-end C++ code running on Vision, my local installation of Bisque (the one I got running yesterday).  This meant resolving a few known software dependencies (boost and lapack), and discovering a few previously-unacknowledged software dependencies (imagemagick and ncurses).
  • For seed counting, I ?worked on a distribution over random "affinities" (affine transforms) that could model the difference in shape between the shapes of two isolated, clean seeds.  I think we can factor this into seven independent dimensions (six Gaussian, one Uniform).  Specifically:  two shear parameters, two dilation parameters, two translate parameters (all Gaussian) and one rotation angle (uniform).  This implies a so-called "proposal distribution" that will be easy to draw from, and I can use a technique like Stochastic Dynamics to find a parameter set that will give a good alignment between two seeds.
Local Bisque is finally running

With the generous help of Kris Kvilekval, I've just managed to get a Bisque Server and Bisque Engine up and running on a local machine.  This is very good news, because it means I'll be able to develop user interfaces for our pollen-tube-tracking modules.

The user interface part of a Bisque module requires stopping and restarting the Bisque server with every major change, so that that change gets noticed.  Consequently, the development would be very disruptive to a live server.  That's why we wanted a local instance (my own private Bisque).  But we had a number of headaches, mostly due to the firewalls around this machine.  I've been working on this installation off and on since mid September, so it's a relief to get it going now.


Blank slate:  One user, no images.