Eideticker Debrief

About Eideticker

Our performance tests currently rely on metrics accessible by web content (Date.now, events, setTimeout, etc.). Many of these sources of information are loosely defined or just lies, implemented that way to improve performance of real web pages. Performance tests built on these metrics are useful in their own right, but it's very hard to use them to measure what users actually perceive: pixels appearing on screen. The problem is made even harder by process separation, GPU rendering, async scrolling and animation, and async rerendering (fennec).

Chris Jones, original specification

Fall 2011: Initial Prototype

  • Experiments with HDMI videocapture
  • Data format to store video capture data
  • Harness development

Takeaways

  • Start from a good requirements/functional specification
  • Don't be afraid to build new things from scratch
  • Modular libraries are good
  • Using the right data formats and structures can save a lot of time

Spring - Summer 2012: Checkerboarding measures

  • Recognizing key blockers for shipping
  • Dashboard development and deployment
  • Tools along the way: Orangutan

Takeaways

  • Know your customer
  • Presentation matters
  • Don't be afraid to build new things from scratch
  • Don't block on other people (e.g. I.T.)
  • Omit unnecessary details
  • Modular libraries and tools are good

Late summer 2012 - Fall 2012: Moar fennec

  • Startup measures
  • Backfilling code, automation robustness
  • Galaxy Nexus support
  • Gecko Profiler Integration

Takeaways

  • Making a system accessible and useful to developers is key
  • There's sometimes less to say about this type of work, but it's really important

Fall 2012: B2G work begins with panda

  • Trying to get eideticker working with the panda...
  • ... a.k.a. "trying to get the panda to do anything useful at all"

Takeaways

  • Don't bother building things on a bad foundation
  • It takes resources to bring up a new hardware platform, in many cases not worth the effort

January - Summer 2013: B2G with cameras

  • So much experimentation...
  • ... eventually got something that sorta worked ...
  • ... but who wanted it?

Takeaways

  • ???

Summer 2013 - Summer 2014: Productizing Eideticker for FirefoxOS

  • Dave Hunt saves the day: Jenkins! Lego!
  • Interest from QA and FirefoxOS perf
  • Still having a trouble finding a niche

Takeaways

  • Great team members can make a big difference
  • Try to see earlier when product isn't meeting business objectives or needs...
  • ... technical success, coolness, or cleverness are not good measures of this

Shutting eideticker down

  • Android data no longer actionable ...
  • ... fixing this too much effort for return on investment. Working on reporting tools (Perfherder) providing a much larger bang for buck (# of regressions filed and acted upon).
  • "I'll be back?"