Migrating this blog to wordpress

I will be moving these posts over to wordpress and incorporating into my lab webpage.  The new link will be:

https://tattersalllab.com

Thermal Camera Showdown - FLIR ONE vs. FLIR SC660

Last summer, I had a high school volunteer (Padraic Odesse) join my lab for a few hours a week to help out my graduate students with data analysis.  To assist him with learning about imaging, technology, and experimental design, I set up a simple experiment for him to test.  Since I am often asked by people whether the new consumer oriented thermal imaging products are 'up to snuff' or useful as research devices, I gave my lab assistant the task of capturing images with a high end thermal camera (FLIR model SC660, which has 640x480 image resolution and thermal resolution ~0.03C) and with the $300 dollar iPhone attachment (FLIR ONE, image resolution 80x60, enhanced up to 160x120, not sure its technical specs on temperature detection).

Here is what the FLIR ONE looks like, nestled next to my lab standard for temperature measurement (accuracy ~0.2C), a thermocouple meter from Sable Systems International :

The camera is a nifty attachment to my iPhone 5, although I look like I'm carrying a bulky phone around with me.  The attachment is actually two cameras in one.  The one lens is the lens that allows long wavelength infrared radiation to pass through to the detector and the other lens is probably very similar to an iPhone camera.  Because they are adjacent to each other, they very nearly capture a similar field of view.  The FLIR software lets you do neat things with overlaying the images, but to be honest, I find the whole outline overlay to be annoying.  Great fun for non-professionals, but annoying if you wanted to use this camera for scientific purposes.

The FLIR ONE has a little pull down switch that turns the device on, but also acts as a shutter control.  I'm not 100% certain what FLIR does, but given how their other cameras work, what is likely is that the when the shutter is pulled down, the thermal sensor is receiving a constant signal that it uses to "calibrate" itself.  FLIR calls this "tuning", not "calibrate" and I am putting things in quotes since I don't quite know.  What other thermal imagers tend to do is use a shutter like this as a reference signal, and also for Non-Uniformity Correction of the pixel array.  If the reference signal is at a known and constant temperature, you can, in essence, calibrate the thermal detector to a more stable source.  It's clearly more complicated than that, but suffice to say, the FLIR ONE demands to be tuned on a regular basis!

Here is the competition, my FLIR SC660 (it has a built in tuning function):

Mounted on a tripod ~0.4m from our temperature source:

Here is our constant temperature source, a circulating water bath with precision temperature control:

Which is easy to confirm temperature with the thermocouple meter:

Since distilled water has a high and known emissivity, it makes it an easy target to produce thermal images from that will be consistent over time (plus/minus error and resolution of the controller).

Here is a sample image taken with the FLIR SC 660:

Here is a sample image taken with the consumer product, FLIR ONE:

As I describe elsewhere (I will post a link to a review paper that will be in press in a month or so), the FLIR ONE product saves its default image as a composite of a thermal image with a haloed/outline image in the usual FLIR JPG format.  So, if you have other FLIR software, you can extract the actual thermal image and do proper thermography on the image.  If you don't have their software, then you are basically stuck with whatever the spot sensor tells you what the temperature is.  I would not trust the spot temperature measurements.  The software does not allow you to set the emissivity precisely or numerically, and distance and other parameters are not available.  Use the spot temperatures cautiously!

Error Analysis

So, my lab assistant diligently captured an image with the two cameras, taken at the same distance and at 8 different water temperatures (5 to 40C).  We now had two camera's data capturing the same source under identical conditions, and so we analyzed the water temperatures using FLIR Thermacam Rearcher Pro, setting object distance to 0.4 m, emissivity to 0.98 (http://www.infrared-thermography.com/material-1.htm), and reflected and atmospheric temperatures to 20C (which was the room temperature.

So, here are the results:

At a quick glance, they both perform quite well.  Look at those R2 values!  Rarely see that in biology. But these are engineered devices.  See how the FLIR ONE deviates in temperature at the low and high temperatures.  This tells me that there is some atmospheric contamination or attenuation of the signal the camera picks up.  The reason things look really accurate near room temperature is because the formula for calculating temperature predicts that if all temperatures are similar to each other, the contaminating radiation signal will be similar to the object signal.  

If you narrow in on how large the error is this will make a bit more sense.  Take the difference in Camera - Actual temperature for all 8 measurements.  The mean absolute error will tell you whether the camera is reading higher or lower than actual temperature (absolute value simple removes the negative sign).   The standard deviation of those measurements is a metric for the resolution/sensitivity of the measurements.

Mean absolute error for the FLIR SC 660: 0.128°C
SD of the error for the FLIR SC 660: 0.0873°C

Mean error for the FLIR ONE: 0.603°C

SD of the error for the FLIR ONE: 0.41°C

On the whole, both cameras perform rather well under these optimized conditions, although clearly the SC 660 performs better.  An error of ~0.128°C is quite low, whereas the FLIR One error is ~5 times higher at 0.603°C.  This kind of error may be related to the electronics, warm-up time, or the sensitivity of the sensor itself, to name a few.  

So, for measuring purposes, the FLIR ONE isn't too bad.  Certainly not as inaccurate as I had feared it might be (except see below).  But, remember, we were performing the analysis using thermography software that allows for some correction of environmental variables, and we were constantly tuning the FLIR ONE.  Depending on your measurement purposes, you might be ok, but the best error you will get under optimal conditions appears to be greater 0.6°C.

Confounding Issues with the FLIR ONE

Because we ran the error analysis above under ideal conditions, we wanted to be a little more realistic with the FLIR ONE since it is likely to be used by scientists as an affordable alternative to work in the field.

The possible confounding issue I am referring to is the warm-up time and the camera's physical temperature itself.  As is usual with cell phones, people are likely to keep the device in their pocket, or maybe leave it exposed to the elements.  If you then turn the thermal camera on and take a spot measurement, there is bound to be error associated with the lack of tuning, but also because the camera temperature is changing while you are making the measurements, which would confound the tuning process itself.

So we took as realistic an approach as possible ran the measurements 3 different ways. We stored the FLIR ONE in a cold room (5°C), a warm room (40°C) and at room temperature (20°C) for 30 minutes prior to use at room temperature.  Then, we turned the thermal camera on and rapidly started recording the spot temperature estimates every 10 seconds, tuning the camera whenever it demanded (there is a tilde ~ symbol that tells you when to tune).  Here are the resulting data (apparently we only ran the cold transfer for ~180 seconds, but the data are still obvious:

If you keep the camera in a cold room, it will severely underestimate temperature (by an average of 5.9°C!!).  It does not return to normal temperatures within 180 seconds.

If you keep the camera in a warm room, it will overestimate temperature (by an average of 1.0°C).  Strangely it does not seem to converge toward the real temperature within our 10 minute long experiment.  Note in the error analysis measurements before, the FLIR ONE did pretty well at measuring objects ~20°C, so this overestimate is due to persistent temperature effects on the device itself.

Finally, the rapid on data show you how the FLIR ONE really performs from rapid turn on (without a thermal equilibration problem), through various cycles of tuning.  Each time it jumps in temperature corresponds to when we tuned it.  It eventually converges toward the real temperature after ~8 minutes. 

Its overall error is ~0.89°C during this warm-up period.

So, can you use the FLIR ONE for real science?  Lot's of ifs and buts, and if you are prepared to calibrate it and check it, go for it (with the caveats above about a 5 times level of error).  I like mine for teaching purposes since it is portable and fast to use, but I will still use my SC 660 for most of my research purposes, due to the versatility of video capture functions, image resolution, and electronic stability.  If you choose to publish a scientific paper with a FLIR ONE, don't ask me to be the reviewer.  I would insist on a calibration curve and assurance that these parameters have all been assessed.

Fine Print

I am not affiliated with FLIR, nor do I receive revenue, salary, or funding from them, so this post should not be interpreted as an endorsement.  Over the years,  I have sent them a lot of business from researchers who ask my advice on thermal imaging products, but I can safely claim that I have received no freebies, hand-outs, or in-kind contributions from FLIR.  Of course, I would be most grateful if FLIR did give me a free thermal camera or a news lens for all the fine press I give them!  If anyone from FLIR is reading this, please take note: you're a big company and perhaps you might want to support basic research.