Migrating this blog to wordpress

I will be moving these posts over to wordpress and incorporating into my lab webpage.  The new link will be:

https://tattersalllab.com

Thermal Camera Showdown - FLIR ONE vs. FLIR SC660

Last summer, I had a high school volunteer (Padraic Odesse) join my lab for a few hours a week to help out my graduate students with data analysis.  To assist him with learning about imaging, technology, and experimental design, I set up a simple experiment for him to test.  Since I am often asked by people whether the new consumer oriented thermal imaging products are 'up to snuff' or useful as research devices, I gave my lab assistant the task of capturing images with a high end thermal camera (FLIR model SC660, which has 640x480 image resolution and thermal resolution ~0.03C) and with the $300 dollar iPhone attachment (FLIR ONE, image resolution 80x60, enhanced up to 160x120, not sure its technical specs on temperature detection).

Here is what the FLIR ONE looks like, nestled next to my lab standard for temperature measurement (accuracy ~0.2C), a thermocouple meter from Sable Systems International :

The camera is a nifty attachment to my iPhone 5, although I look like I'm carrying a bulky phone around with me.  The attachment is actually two cameras in one.  The one lens is the lens that allows long wavelength infrared radiation to pass through to the detector and the other lens is probably very similar to an iPhone camera.  Because they are adjacent to each other, they very nearly capture a similar field of view.  The FLIR software lets you do neat things with overlaying the images, but to be honest, I find the whole outline overlay to be annoying.  Great fun for non-professionals, but annoying if you wanted to use this camera for scientific purposes.

The FLIR ONE has a little pull down switch that turns the device on, but also acts as a shutter control.  I'm not 100% certain what FLIR does, but given how their other cameras work, what is likely is that the when the shutter is pulled down, the thermal sensor is receiving a constant signal that it uses to "calibrate" itself.  FLIR calls this "tuning", not "calibrate" and I am putting things in quotes since I don't quite know.  What other thermal imagers tend to do is use a shutter like this as a reference signal, and also for Non-Uniformity Correction of the pixel array.  If the reference signal is at a known and constant temperature, you can, in essence, calibrate the thermal detector to a more stable source.  It's clearly more complicated than that, but suffice to say, the FLIR ONE demands to be tuned on a regular basis!

Here is the competition, my FLIR SC660 (it has a built in tuning function):

Mounted on a tripod ~0.4m from our temperature source:

Here is our constant temperature source, a circulating water bath with precision temperature control:

Which is easy to confirm temperature with the thermocouple meter:

Since distilled water has a high and known emissivity, it makes it an easy target to produce thermal images from that will be consistent over time (plus/minus error and resolution of the controller).

Here is a sample image taken with the FLIR SC 660:

Here is a sample image taken with the consumer product, FLIR ONE:

As I describe elsewhere (I will post a link to a review paper that will be in press in a month or so), the FLIR ONE product saves its default image as a composite of a thermal image with a haloed/outline image in the usual FLIR JPG format.  So, if you have other FLIR software, you can extract the actual thermal image and do proper thermography on the image.  If you don't have their software, then you are basically stuck with whatever the spot sensor tells you what the temperature is.  I would not trust the spot temperature measurements.  The software does not allow you to set the emissivity precisely or numerically, and distance and other parameters are not available.  Use the spot temperatures cautiously!

Error Analysis

So, my lab assistant diligently captured an image with the two cameras, taken at the same distance and at 8 different water temperatures (5 to 40C).  We now had two camera's data capturing the same source under identical conditions, and so we analyzed the water temperatures using FLIR Thermacam Rearcher Pro, setting object distance to 0.4 m, emissivity to 0.98 (http://www.infrared-thermography.com/material-1.htm), and reflected and atmospheric temperatures to 20C (which was the room temperature.

So, here are the results:

At a quick glance, they both perform quite well.  Look at those R2 values!  Rarely see that in biology. But these are engineered devices.  See how the FLIR ONE deviates in temperature at the low and high temperatures.  This tells me that there is some atmospheric contamination or attenuation of the signal the camera picks up.  The reason things look really accurate near room temperature is because the formula for calculating temperature predicts that if all temperatures are similar to each other, the contaminating radiation signal will be similar to the object signal.  

If you narrow in on how large the error is this will make a bit more sense.  Take the difference in Camera - Actual temperature for all 8 measurements.  The mean absolute error will tell you whether the camera is reading higher or lower than actual temperature (absolute value simple removes the negative sign).   The standard deviation of those measurements is a metric for the resolution/sensitivity of the measurements.

Mean absolute error for the FLIR SC 660: 0.128°C
SD of the error for the FLIR SC 660: 0.0873°C

Mean error for the FLIR ONE: 0.603°C

SD of the error for the FLIR ONE: 0.41°C

On the whole, both cameras perform rather well under these optimized conditions, although clearly the SC 660 performs better.  An error of ~0.128°C is quite low, whereas the FLIR One error is ~5 times higher at 0.603°C.  This kind of error may be related to the electronics, warm-up time, or the sensitivity of the sensor itself, to name a few.  

So, for measuring purposes, the FLIR ONE isn't too bad.  Certainly not as inaccurate as I had feared it might be (except see below).  But, remember, we were performing the analysis using thermography software that allows for some correction of environmental variables, and we were constantly tuning the FLIR ONE.  Depending on your measurement purposes, you might be ok, but the best error you will get under optimal conditions appears to be greater 0.6°C.

Confounding Issues with the FLIR ONE

Because we ran the error analysis above under ideal conditions, we wanted to be a little more realistic with the FLIR ONE since it is likely to be used by scientists as an affordable alternative to work in the field.

The possible confounding issue I am referring to is the warm-up time and the camera's physical temperature itself.  As is usual with cell phones, people are likely to keep the device in their pocket, or maybe leave it exposed to the elements.  If you then turn the thermal camera on and take a spot measurement, there is bound to be error associated with the lack of tuning, but also because the camera temperature is changing while you are making the measurements, which would confound the tuning process itself.

So we took as realistic an approach as possible ran the measurements 3 different ways. We stored the FLIR ONE in a cold room (5°C), a warm room (40°C) and at room temperature (20°C) for 30 minutes prior to use at room temperature.  Then, we turned the thermal camera on and rapidly started recording the spot temperature estimates every 10 seconds, tuning the camera whenever it demanded (there is a tilde ~ symbol that tells you when to tune).  Here are the resulting data (apparently we only ran the cold transfer for ~180 seconds, but the data are still obvious:

If you keep the camera in a cold room, it will severely underestimate temperature (by an average of 5.9°C!!).  It does not return to normal temperatures within 180 seconds.

If you keep the camera in a warm room, it will overestimate temperature (by an average of 1.0°C).  Strangely it does not seem to converge toward the real temperature within our 10 minute long experiment.  Note in the error analysis measurements before, the FLIR ONE did pretty well at measuring objects ~20°C, so this overestimate is due to persistent temperature effects on the device itself.

Finally, the rapid on data show you how the FLIR ONE really performs from rapid turn on (without a thermal equilibration problem), through various cycles of tuning.  Each time it jumps in temperature corresponds to when we tuned it.  It eventually converges toward the real temperature after ~8 minutes. 

Its overall error is ~0.89°C during this warm-up period.

So, can you use the FLIR ONE for real science?  Lot's of ifs and buts, and if you are prepared to calibrate it and check it, go for it (with the caveats above about a 5 times level of error).  I like mine for teaching purposes since it is portable and fast to use, but I will still use my SC 660 for most of my research purposes, due to the versatility of video capture functions, image resolution, and electronic stability.  If you choose to publish a scientific paper with a FLIR ONE, don't ask me to be the reviewer.  I would insist on a calibration curve and assurance that these parameters have all been assessed.

Fine Print

I am not affiliated with FLIR, nor do I receive revenue, salary, or funding from them, so this post should not be interpreted as an endorsement.  Over the years,  I have sent them a lot of business from researchers who ask my advice on thermal imaging products, but I can safely claim that I have received no freebies, hand-outs, or in-kind contributions from FLIR.  Of course, I would be most grateful if FLIR did give me a free thermal camera or a news lens for all the fine press I give them!  If anyone from FLIR is reading this, please take note: you're a big company and perhaps you might want to support basic research. 

Why Scientific Discovery is Important & Why Science is Supposed to be Fun

It has been a while since I blogged, and I could go into details about a desire to 'disconnect from the online world' or give excuses that 'there are too many things that occupy my time', both of which are true...but that is for another time.

I wanted to post some reflections following the publication of a paper that represented contributions from a number of brilliant collaborators, particularly because we need a good news story in basic sciences.  I also wanted to reflect a little to give younger scientist reason to be positive about science.

Last week, we (collaborators from Brasil and Canada) published a basic discovery paper entitled: Seasonal reproductive endothermy in the tegu lizard" in Science Advances, the open access sister journal to Science, published by the AAAS.

The discovery has been well covered in the media already (see links here), and I will not go into too many details here, since it is all openly available at the link above, except to say that we discovered an animal that is usually considered to be ectothermic ("cold-blooded" is the horribly inaccurate term used by non-scientists) can become endothermic ("warm-blooded" - another equally uninformative and antiquated term to describe a complex array of metabolic changes), and it does so during the reproductive season.  As careful scientists, we did not comment in the study on their reproductive activities or indeed we had no measurements of what they were doing, so the link to reproduction is indirect at the moment.  The study was simply an observation that they become warmer during this time of the year and that the degree to which they warm up cannot only be explained by the usual suspects (e.g. behavioural thermoregulation, thermal inertia).

Here are the results of the study, depicted simply as a thermal image:

Shown above is a small group of tegu lizards sharing the same burrow (they do so voluntarily, although probably with a little "aggro").  This image was captured in the early morning hours (~5-6am) before sunrise during the breeding season, so the tegus should have been as cold as they could be from the previous day of basking.  One is a little bit cooler than the other two in temperature, but all three are quite a bit warmer than their surroundings.  They are achieving this through a rise in metabolism and because of an escape from the outside air, are able to slow their rates of heat loss.

Indeed, occasionally, a tegu doesn't make it back into the burrow at night, so that the next morning when we find it, it is quite frigid.  This is because the endothermy isn't operating at the same capacity of that of a bird or mammal and that it is not necessarily thermoregulatory in nature.  Here is a rather cool tegu found trapped outside a burrow on a cool night (~6am), with a low body temperature:

Anyhow, what is more important for me to write about is something I only thought about after the media started calling us, and that is that not all science follows the painfully boring description kids are given in school about the laborious scientific method.  Allow me to explain by way of giving the "human side" to a fun research collaboration, while also writing this as a parable for the way that science truly proceeds.

The Best Laid Plans of Lizards and Men Often go Awry

This study began as a 2 year long MSc project for Colin Sanders, who was working with Dr. William Milsom at the University of British Columbia (where I did my post-doc).   Colin, Bill and myself went down to Brasil in 2003/2004 to initiate a project examining the year-long changes in heart rate, breathing rate and body temperature in tegu lizards in a semi-natural environment.  The initial basis for the study was to examine a lizard that would hibernate (i.e. go dormant) at relatively warm temperatures.  Our Brasilian colleagues (Abe and Andrade) had laboratory data suggesting that tegus would go dormant in the winter.  We had valid physiological reasons for why this was a interesting question (not initially about endothermy) and that was Colin's MSc project.  How did we do this?  We implanted little electronic devices that would measure an EKG and EMG and body temperature continuously and save this data to a computer.  The entire project was instrument heavy and indeed, our equipment was tied up in Brazilian customs for 6 weeks, nearly preventing the launching of his research!  I wrote a Matlab script to analyse the massive amount of raw data we were collecting from the radio-transmitting telemeters.  Colin had to endure months of staring at the computer screen to verify the script was working properly.  All totalled, we collected Gigabytes of data to computers that had no DVD drives and no network access.  Suffice it to say, Colin had a lot of sleepless nights and frustrations of running my rather unsophisticated computer code.  Skype and email were our friends, but it made discovery a distant and future goal.

Fast forwarding to when Colin returned to Canada, he now had a massive amount of data on the daily changes in heart rate, breathing rates and body temperatures in a cohort of tegu lizards and still had to make sense of that.  In many ways, this project was a highly sophisticated Natural History project.  We were initially interested in knowing whether tegus went into their burrows and reduced metabolism in the winter and needed to do this under natural conditions, which is where our Brasilian colleagues came in.

Collaboration and Camaraderie

We were able to pursue this research because of the friendly and collaborative nature of two key individuals, Dr. Augusto Abe and Dr. Denis Andrade.  Since 2002, they have hosted myself in their lab (and homes!) for up to 1 or 2 months, almost on a yearly basis.  They both have interesting research programs into South American evolutionary physiology and comparative biology and are incredibly knowledgeable about the natural history of reptiles and amphibians of Brasil.  They had both studied tegu hibernation in the lab as well.  Not only was their lab designed to allow us to study tegus in their (almost) natural habitat, but visiting their labs is always a fruitful journey of discovery.  My hats off to Augusto for having the foresight to build a world class facility to encourage and foster collaborations in our discipline, and my deep appreciation to Denis for being the on-site "supervisor" of the research in Brasil.  It is not easy to remotely manage collaborations.

Check, Refine, Eliminate other Explanations

Following our initial, but unplanned-for observations of lizards warming up in the reproductive season, we spent a few years (yes, this was a slow cooker project) discussing the possible explanations and trying to devise ways to convince ourselves that the heat was due to metabolic heat production.  Colin's heart rate data did provide the best evidence of this, but we already recognized that reviewers would be inherently skeptical, since we were ourselves.  At this stage, Dr. Cleo Leite and my student Viviana Cadena were invited to carry the torch and help provide the evidence that the tegus were not only capturing heat from the sun very efficiently and storing it.  Bill and myself had a lot of friendly arguments about this, but Bill convinced us all that the initial observations were real and important.  As a side project to his other research, Cleo collected three years worth of data on tegus during the reproductive period and transferring them back and forth from outside burrows to inside the lab where we had better control over ambient temperature, and along with Viviana's thermal images, we were able to show that tegus were still showing rather impressive amount of endothermy (2 to 3 degrees above ambient temperature) when placed in the lab for a week without access to insulating burrows, or nest material.

Science is an Ongoing Learning Exercise

Along with wise and thoughtful contributions from all my co-authors, I was tasked with the unenviable job (hey, this is my story, so I can tell it this way) of writing the data up for publication.  All of us suffer from procrastination when it comes to writing.  Scientific writing is not simply formulaic, although there are clear and simple ways to present science accurately, I also think the best scientific papers also tell a story.  So, I drafted the first version of the manuscript and knew we had a really cool observation but had to do my homework since we would need to place the work into the appropriate theoretical context.  The evolution of endothermy has a long history of big thinkers testing hypotheses for how the ancestral condition of ectothermy could have overcome the energetic costs inherent to endothermy.  How were we to put our years of observations into this context?  Anyhow, I think we were all pushed a little outside our comfort zone.

Don't Be Dismayed by Rejection

Once you think you have a potentially impactful paper, you do want it to be published somewhere where it will be read!  It also is true (almost a cliche) that many scientists will aim for the top journals, only to receive that dreaded auto-email reply from Nature claiming that "...your manuscript is of insufficient immediate interest to our broader readership to justify its publication in Nature."  That is fine.  Push on and don't let that manuscript stagnate on your computer.  So, that is what we did.  We endured our rejection, made headways with a subsequent submission and actually were pleased with the chance to have a widely publicised paper that was also available in an open access venue.

Take Home Message

Although the media attention surrounding discoveries are usually nice, they very often exaggerate the story and forget to comment on the importance of the team as well as the important role of accidental discovery.  Serendipity should be more valued in science.  It rarely is, and I worry that we train our students to become hypercritical hypothesis-driven automatons.  Although I still think that science should be hypothesis-driven (you still need to know what you hope to achieve and be able to communicate this effectively when writing a research grant or defending your idea to a graduate committee), you also need the freedom to "follow your nose" if you come across a novel observation.  Why can't we have both?  Why can't we train people to think scientifically, accurately and carefully, while recognizing the value of thinking outside dogmatic constraints.  Yet, granting agencies demand outcomes and outputs and governments like to link research to applications (read: commercial applications and jobs).  Where is the joy in this approach?  How does basic research fit into this?

My advice to junior scientists?  Don't give up, but be prepared to think clearly and seek your motivation to do science (i.e. discovery research) from your own inner enthusiastic and curious kid, rather than your colleagues or your mentors!

Funding Agencies Deserve their Credit

There are some funding agencies that do value basic research, and my hats off to them!

This research was funded by the Natural Sciences and Engineering Research Council of Canada</a> (to W.K.M. and G.J.T.),  FAPESP, and CNPq through the INCT in Comparative Physiology</a> (proceeding nos. 2008/57712-4 and 573921/2008-3). D.V.A. was supported by research grants from the FAPESP (proceeding nos. 2010/05473-6 and 2013/04190-9) and CNPq.