Beetle Larva Cocoon, Backups and Pi Failures

Noticed this Beetle Larva (not sure, do you know?: ) forming a cocoon over the past few days, at this point it enters the Pupa stage where it will only be able to wiggle! I'm not sure how long this will last, and since it's in the compost bin I have to decide whether or not to disturb it (by turning the compost), or leave it be to observe!

I'm curious, over a ~6 hour period there is a significant color change in the Larva/Pupa. It's not clear to me what causes this. There's documentation about this with Butterfly Larva > Pupa transitioning, and I'm guessing it helps the Pupa / Cocoon to blend in better with its surroundings, but what process is causing this?

The Larva measures approximately 17mm x 2mm, and is buried about 20cm into the compost bin. Another reason why using a scanner is nice, we can make quick/easy/accurate measurements based on pixels.



Unrelated to the soil life, I've had two amusing technical issues over the past few days:
One: While I was toiling around with a script it looks like I deleted almost a day's worth of images. That's frustrating, but the timing of it is somewhat appropriate. I'd purchased a 1 TB drive awhile back to store images on but hadn't hooked it up in quite some time.

I think it would be appropriate to have the Pi save images to both the SD Card and the Hard Drive,  and once a day tar+gzip the files off to Amazon S3. This would afford me the option to play with one set of images knowing that if I did something horribly wrong I have a local backup.

Two: Around 1:30am yesterday the Raspberry Pi Model 2 B I was using croaked. I'm not sure what the status is beyond it not getting on the network (Hooked up via Ethernet), and an SD Card that was working in another Pi provides the same result. I'm still without an HDMI Monitor here, so everything is done headless, I'm currently unable to see what might be happening in the boot process. Thankfully a second Pi was ready to take its place.

Having a $35 computer in my case means it's easy to have a backup Pi. Thank you Raspberry Pi Foundation : )

Lessons here:
 - Have a process for swapping the Pi out when something goes wrong
 - - Backup SD Card
 - - Backup Pi
 - Have some notification that things are not as they should be
 - - Pi going off the network is not a huge deal, though is a symptom that the Pi may be down
 - - Not sure if there's a better way to alert me that there may be severe issues.
 - Have a process for saving images to two separate locations
 - - SD Card
 - - External USB Drive
 - Have a process for saving scripts in the same way
 - Automate Amazon S3 Sync of files.

Fungus, Frame Rates and Glaciers

Fungus!
Radiolab has a wonderful podcast on trees, roots, and fungus. It's a delicious introduction that really puts into perspective how interconnected everything is. As in Trees eat Salmon. Don't believe me? Listen to the podcast or read this article.

The same day I listened to this I noticed a few bright white growths in the compost, neat! There's been lots of fungal growth in the compost pile, but I haven't specifically seen this one yet, at least not in these concentrations.

Fungus doesn't move as fast as the bugs we're seeing. I could probably capture one scan per hour and feel pretty good about capturing movement in fungus. But I'm capturing a scan every 15 minutes, and the video playback at this rate causes bug movement to look like a hastily recorded stop motion animation.

Frame Rates!
So I changed the scan to run every 2 minutes, and I've captured a day's worth of footage.

Prior: 1 scan / 15 minutes = 96 scans a day, played back at 30 Frames Per Second = 3 second video.
Now: 1 scan / 2 minutes = 720 scans a day, played back at 30 Frames Per Second = 24 second video.

While I like the smoother movement, I don't necessarily like how long the video lasts. Ooh, but screens these days can all play back at 60 FPS.

60FPS: 1 scan / 2 minutes = 720 scans a day, played back at 60 FPS = 12 second video. Looks even smoother, and plays back faster!

30 FPS	60 FPS

I had an option to either re-encode the video from images at 60FPS, or convert and re-encode the video at 60 FPS. I had not done the latter before, so:

avconv -i sc_20160805.mp4 -f rawvideo -b 50000000 -pix_fmt yuvj420p -vcodec rawvideo -s 768x1080 -y temp.raw

avconv -f rawvideo -pix_fmt yuvj420p -s:v 768x1080 -r 60 -i temp.raw -c:v libx264 sc_20160805at60fps.mp

Not sure what I'll do in the future, for now I'll just continue capturing and encoding videos at 30 FPS.

Glaciers?
This is really all about storage.  I'm capturing a 600 DPI image once every 2 minutes. It's saved as a JPG which greatly reduces the storage space (with some image loss), but still costs 7MB / image.

7MB / image
720 images / day
5 GB / day of storage.

The Raspberry Pi has a 32GB SD Card on it, of which I can use ~28GB. That's only 5 days of storage. Not to mention the terabyte or so of storage I've already offloaded over the past couple years of doing this.

Amazon S3 is a pretty easily accessible storage method, it costs $0.03 / gigabyte. Right now I have around 300GB's on it which costs ~$9 / month. Given the increase in scanning frequency and resolution, I'll increase that by about 150 GB / month (5GB / day * 30 days).

That means I'll be paying an additional $4.50 (150 GB * .03 / GB) every month. Not so bad until a year goes by and now I'm paying $54 / month and increasing.

But I just realized there's this thing called Amazon Glacier, which is sloooooooooooow for retrieving data, but only costs $.007 / GB, and I can setup S3 to automatically dump any files that haven't been used in X days into Glacier storage. This sounds quite ideal. I'd love to here if there are better storage options. I do have a local drive I can store stuff on, but I trust Amazon a lot more than I trust my cheap drive.

Yay for Glaciers!

7 Day 4k Test, Broken Images, Jitter

A seven day 4k Test. I'm not sure how well this works as I've yet to actually view it at 4k resolution on a screen that supports it. If anyone out there does, let me know how it looks ;)

The workaround I came up with for removing broken images so far has been pretty reliable. Out of 74 days a total of 7104 images have been captured and 329 were removed. 7104-329 = 6775 images, but somewhere I'm missing 20.

Jitter, with the Canon LiDE 20 scanner it would not always start in exactly the same location. This wasn't very noticeable at lower resolutions, but when zooming in on a higher resolution scan and flipping through a few images the jitter would become distracting.

Though still noticeable it's not nearly as bad as it was.

July Compost Bin!

The full video of July's compost bin can be seen below. I don't think we've ever had any sort of schedule for turning the compost, but now that I can see it I've started turning when the bulk of it is a delicious brown color...  I should read up on composting...

Mushroom caps seem to pop up regularly a week or two after the turning, but I turn the compost too quickly to see what might be feeding upon the mushrooms/hyphae. Fungus being one of the most important decomposers in our world, I'm curious to see what feeds upon it. It took me a moment to remember that we (humans) are one of those creatures, though I won't be capturing that image with a buried scanner any time soon. Apparently some forms of fungus feast upon other fungus, another example of just how amazing fungus is.

I'm still scanning at 600 DPI which gets me super excited to see what new things can be seen that weren't before.  I'm a little less excited by the data storage requirements...  If all goes well in a few days I should have a week of scrolling footage at 4k : ) Anyhow, last month's video at 768 horizontal pixels:

More Compost and a brief 4K Video Test

19 Days of Compost featuring Tomatoes, Carrot Tops, Corn Husks and Mushrooms. It's interesting to see so many mushroom caps growing up within the compost. For some reason I had assumed these would only be featured above in the sun light. Mushrooms don't photosynthesize, so that assumption was a poor one....

The scanner has been running this entire time at 300 DPI, which gives me an image width of approximately 2550 pixels (8.5" * 300 pixels per inch). The 4K UHD standard is 3840 pixels wide, and I'd love to see a video like the one above done at 4k. So I've upped the scanning resolution to 600 DPI, and will capture that for hopefully a few weeks. A 3 second test of that can be seen here:

Unfortunately, I don't have a display capable of 4k. Hopefully this is setup properly!

30 Days of Compost + Panning Video + Removing 'bad' images

1. This is 2743 images taken over ~30 days.

2. A scan happens every 15 minutes, that means I should have ~2,880 images.

3. Sometimes the scanner jams halfway through a scan. This isn't horrible, but the resulting image has a streak of repeating pixels starting where the scanner jammed. This is particularly jarring when images are played back as a video.

4. I liked the effect of panning across the images (see earlier post), but wanted to clean things up + timestamp the images.
       
So two short scripts are run:
Clean Images:
#!/bin/bash
for file in *.jpg
do
        convert $file -crop 768x1+0+10 image_a.jpg
        convert $file -crop 768x1+0+30 image_b.jpg
        if [ $(compare -metric RMSE image_a.jpg image_b.jpg NULL: 2>&1 \ | sed 's/.*(//' | cut -c1-4 | cut -c 4-) -gt 0 ]

        then
                echo "$file keep"
        else
                echo "$file remove"
                mv $file brokenimages/
        fi
done
rm image*.jpg

Left image bad. Jammed about 3/5ths through the scan. Right image good!

For every jpg image in a directory we'll create two temporary images, a single row of 768 pixels located 10 and 30 pixels down from the top. We'll use imagemagick's compare tool to get a rough idea of how different they are, this feedback looks like:

350.361 (0.00534616) for images with little to no difference

2145.33 (0.0327357) for images with some difference

I'm using BASH, it doesn't like floating point numbers. We get lazy and grab the second number past the decimal point. If this number is > 0, we can assume we're dealing with two images with enough difference that it's good to keep. This could fail, but so far it's worked with ~3,000+ images. It's not particularly fast, taking approximately 3 seconds / image on a Pi 2 Model B. Running it retroactively isn't fun, but checking images as they are scanned (once every 15 minutes) is plenty fast.

Panning
In order to get the panning / scrolling affect, I crop each image down to 1920x1080 pixels starting at the top. With (almost) every image I shift the cropped area down 1 pixel. I like the idea that I can slowly pan through the entire length of the scanned area in the time it takes to run through each image, but I have more images than pixels height to move through:

2743 images = 2743 pixels to shift.
3484 = original image height.
-1080 to account for the video height
2404 = 3484-1080. Total number of pixels I need to shift down.

If I move down one pixel for each image, we'll be 339 pixels below the bottom of the scanner. I could try shifting partial pixels but that involves more work. Again laziness, so instead we just don't shift every 6th image. This isn't exact, but it gets us close to the bottom of the scanner area.

Is it noticeable in the video? : )

Cycling through all the images in a directory, we grab the date and time from the file name and set that aside as ndate & ntime. Using the imagemagick tool convert we crop a 1920x1080 chunk out of each 2480x3484 image starting at the top, shifted 280 pixels in from the left. We add a small 400x30px background bottom center and pop the $ndate and $ntime values as a text caption on top of this background.

Each image processed we increment the value $a by 1, if $a is < 6, we shift the cropped location down 1 px. If the value of $a is equal to 6, we don't shift the the image down and we reset the value of $a to 0.

This image is saved with the prefix "temp_" and a four digit number starting with 0000. Saving in this naming format makes it easy for us in the next step to wrap these images up in a nice mpeg video file.

#!/bin/bash

a=0

shift=0

count=0

for file in *.jpg;

do

        counter=$(printf %04d $count);

        ndate="${file:3:4}\/${file:7:2}\/${file:9:2}"

        ntime="${file:12:2}:${file:14:2}:${file:16:2}"

        convert $file -crop 1920x1080+280+$shift - | convert -background '#0008'   \

                -gravity center \

                -fill white     \

                -size 400x30    \

                -pointsize 24   \

                -kerning 2.5    \

                -font Courier   \

                caption:"${ndate} ${ntime}"     \

                -       \

                +swap   \

                -gravity south  \

                -composite      \

             "temp_$counter".jpg; 

if [ $a -eq 6 ];

then

        let shift=$shift+0

        a=0

else

        let shift=$shift+1

        let a=$a+1

fi

echo -e "$file\t$count\t$shift"

        count=$(($count+1))

done

And then we need to turn these images into a video file:

avconv -y -r 30 -i temp_%04d.jpg -r 30 -vcodec libx264 -crf 20 -g 15 CompostCam30Days.mp4

This probably took an hour to process on a ~6 year old Mac Laptop. It could be done on the raspberry pi, but the converting and timestamping of images would have taken longer.

Nothing too crazy here, all the hard work has already been done by ImageMagick and avconv.  But it's fun, and I should probably turn the compost : )

Guide Posted and More Compost Cam!

Posted a guide on how to make a SoilCam / Rhizotron! Lots of things I would love to add / edit / modify, but I think it's at a point where not posting it now would mean it never would : )

Apparently ffmpeg and avconv had a security hole in their concat command. This confused me for longer than I'd like to admit before I realized they had dropped the command in a newer release. Thankfully with mpeg.ts files you can simply cat them together and then convert.

The Compost Cam continues to run and I've captured about two weeks of data. The broken images make it a bit jarring, cleaning those out is pretty easy, but even with I still find it fascinating to watch what's happening with my food!

The jarring is a result of the scanner jamming, presumably due to too much pressure from the soil pressing in on it. Al has recommended a few modifications (enclosing it in a half cylinder), I wonder if it's pressure on the edge, or the back where a plastic linear gear rail resides that causes this? 

The scanner is mounted so the scan travels down, perhaps that orientation allows the gears to slip? 

Either way, figuring out how to make the scanner last longer and skip less would be the ideal next steps for this whole project.... 

Holy Compost Cam Batman!

Approximately 500 images taken over 5 days, roughly once every 15 minutes. A 6.4" x 4.2" view panning down 11 inches of compost located in my backyard.

It surprises me how insanely active this is. My wife started the compost a few years ago and I'm not sure when we last emptied/sifted things. Towards the bottom half the soil/compost is shifted away from the scanner which puts it out of focus, but there's still tons of activity.

I'm capturing a full 8.5 x 11.5" area at 300 DPI and this video is panning vertically across a small portion of it. We've recently put in a bunch of cilantro, strawberry, carrot, and bits and pieces of leaks along with pieces of brown paper bag and potato skins. To be able to see all of this turn to soil would be fascinating!

Compost, Hypha Growth and Instructables

Almost done with the guide/instructable. I need to run through the whole thing one more time to see if all of it adds up properly. In the mean time I've buried two more scanners. The first one a few weeks ago was buried with a flower (pictured in prior post), the Canon LiDE 200 seized up after ~7,000 scans.

The second scanner (another LiDE 200) was buried in our compost bin a couple days ago, and today I started seeing hypha growth along the head of a strawberry that arrived yesterday. In about 8 hours the hypha grew ~6mm. Impressive! I wonder how much further it will grow.

I know very little about fungus, but I remember reading that some forms of fungus form hypha strands capable of ensnaring nematodes. Totally wicked, and would be amazing to capture. Though I'm not sure scanning at 600 or 1200 dpi would suffice. Some details on this here: http://botany-lab.blogspot.com/2015/03/arthrobotrys-carnivorous-fungus.html

Mmmmmmmmm.

Flowers!

Polishing up an instructable on assembling a SoilCam means burying another scanner! In the process I realize I've had a 4 Port Plugable USB hub running for over a year outside, through a Michigan winter. Neat : )

This burial is in a tub that a year ago we used to grow potatoes. While still located outside, having it in a tub makes it easier to have inside which resolves having to worry about how you're going to get power outside. Though if portability is required, a five gallon bucket would be a better option (soil is heavy!) and would still fit the scanner.

Millipede Mating

I think this brings things full circle. Two weeks ago there was a barrage of baby pill bugs, last week we saw death occur as a hungry beetle larvae happened upon a slug, and this week we have millipede mating!

What we can see occurs over a period of 6 hours, and at some point it looks like three millipedes are involved. It's difficult to tell if they are the same millipedes participating throughout the entirety.

I had originally incorrectly titled this post "Centipede Mating". You can see between 16:02:01 and 16:57:02 that each body segment has two pairs of legs. If these were Centipedes you would only see one pair per segment. Assuming these are mating Millipedes we might get to see a slew of babies in a few weeks! Apparently they start off with only three pairs of legs and only after a few molting sessions do they appear as we're used to seeing them.

Amazing piece of info: Millipedes are part of the Myriapod family (many feet), and were the first oxygen breathing animal to walk on land! And last, these creatures are a significant contributor to decomposition, that crazy process that results in this wonderful pile of material we call soil that keeps you and I fed. Next time you're eating anything or simply going somewhere on a walk, thank the Millipedes.

 - https://en.wikipedia.org/wiki/Pneumodesmus
 - https://en.wikipedia.org/wiki/Myriapoda
 - http://www.biokids.umich.edu/critters/Myriapoda/

Image stabilization issues
Some of the videos I compile tend to shake left and right, a result of the scanner occasionally starting at a slightly different position. It always scans the same distance and looking at one image next to another it's difficult to see any difference. Play them back in a video and it becomes very apparent that the images don't line up.

I've used FFMPEG's deshake option to smooth out this jitter, but it's difficult to remove all of it. Going through each image one at a time is a bit too time consuming, so we need to either resolve the initial problem that causes it, or figure out a more accurate work around.

A few thoughts on resolving / working around this:

1. Figure out how the scanner returns home.
 - At 600 DPI even a millimeter off will result in a 20pixel shift. When viewing the whole 8.5x11" area it's not very obvious, when zooming into a smaller area it is. The intended use of these scanners from the manufacturer likely did not include building timelapse videos : )

2. Can we compare the first twenty rows of pixels from left to right of sequential images?
 - Once we find a certain percentage match, trim all rows to the left off.

3. Apply a ~5mm solid white border to the glass scanning plate
 - Use imagemagick to locate the border and crop within the boundary.

Death of a Slug

A week ago a dying worm slowly crawled into a small pocket of space. I was interested at the possibility of seeing what the decomposition process would look like. A few days in and this happens:

I didn't expect to see something so violent! The timelapse videos shows in 10 seconds what in reality took nearly 12 hours. Maybe the speed at which it plays back is what makes it feel so violent? 

It would be great to develop a system that allows for real time viewing, but that will have to wait.

Not sure what type of slug was eaten, the predator in this case appears to be some sort of Beetle Larvae? You can also make out the worm in the first few seconds of video, the view of which is quickly obscured as the Beetle Larvae makes its way in.

Young Pillbugs

A few days ago a couple small explosions of tiny white bugs appeared. Zooming in shows what appear to be hundreds of tiny Pillbugs (Armadillidium vulgare, roly-polies : )! Action begins approximately 9 seconds in.

According to this article these young crustaceans have already spent 10 to 14 weeks riding around in the pouch of a female Pillbug. I wonder if we could look back at previous video and identify any of them?

The area represented in the video is approximately ~8.1cm x 4.5cm (3.2" x 1.8"). This puts the young Pillbugs at ~1.5mm (1/16") in length.

Most of the Pillbugs quickly disperse beyond the viewable area within a few seconds. A few slow down and some can be seen feeding, or being fed on.

So much happening in such little space, so much more happening that's barely visible, and I can only begin to imagine how much is going on that isn't visible at all at this level.

Youtube Playlists, Current Image, Zoomed Views

Weekly videos are now being posted to this playlist on youtube.

Talking with George Albercook and Greg Austic got me wondering (in very rough) terms how the natively captured scanned images compare with a camera.

Scan DPI: 600
Physical Area: ~11.7x8.5"
Pixels:7015x5076

Equivalent to a 35 megapixel camera. Pew pew megapixels! Unfortunately, the videos being posted are processed down to ~1920x1080.

It would be interesting to see the timelapse at 4k on an appropriate monitor, but it's more interesting to start looking closer.  Since we're capturing the images at a higher res than being displayed, we can crop out a 1920x1080 section of the native res image and turn that into a video. Here's what that looks like over the course of a week:

The video above is of a ~3.2" x 1.8" splice of the earth. One week = 604800 seconds. This plays back in ~66 seconds. Life at ~9163x!

The scanner being used can capture images at 2400 DPI, and scanners that can capture at 9600 DPI aren't terribly expensive (though they do get bulkier). Capturing a full image at that res is probably too much for the Raspberry Pi to handle, and the storage space would be excrutiating. 

But capturing even a square inch of space at that DPI would be fascinating, and I think doable : ) 

For now, I wonder what square inch would be most interesting to capture at 2400DPI?

Also, latest images are being posted at the "Latest Image" tab above. Approximately ~10 minute delay.

Rain - Underground

Rain in Ann Arbor!

Ann Arbor saw between 1 and 3 inches of rain on Sunday, June 14th. It's interesting to see what the rain does to freshly (~two week) turned soil.

Playing around with imagemagick a bit more. I'm splicing a 100x100px section and getting the 'average brightness' of it. Scans are taken every 5 minutes, but to keep things simple I'm only sampling every third image, or roughly every 15 minutes.

Then I repeat that process 100px down, continuing until we get to the bottom of the image. Imagemagick spits out a pile of numbers like so:

Reading Date	Rainfall (in.)	0-100	100-200	200-300	300-400	400-500	500-600	600-700	700-800	800-900
0:00:00	0	9598.33	13316.9	10462.3	10667.8	10865.2	10513.7	10696.4	10353.2	10342.7
0:15:00	0	9460.71	13131.9	10450.9	10675.1	10881.2	10513.1	10743.1	10371.1	10354.6
0:30:00	0	9237.39	12896.3	10400.8	10698.7	10877.8	10542.1	10744.6	10392.1	10368.6
0:45:00	0	9155.23	12850.5	10315	10650.4	10851.8	10505.7	10729.3	10370.2	10382.1
1:00:00	0	9487.32	12982.9	10343.6	10668.1	10847	10512	10734.3	10374.6	10384
1:15:00	0	9519.43	12857.8	10288.2	10630.3	10810.6	10464.2	10685.2	10365.9	10339.7
1:30:00	0	9623.34	12937.3	10331.2	10609.4	10802.6	10485.5	10736.8	10393.2	10421.1
1:45:00	0	9516.39	12664.1	10087.5	10552.4	10765.2	10437.9	10715.7	10369.7	10345.9
2:00:00	0	9497.97	12638.1	10084.9	10480.2	10656.9	10417.9	10720.8	10366.6	10347.4
2:15:00	0	9586.58	12693.7	10088.4	10481.7	10597.6	10347.9	10712.9	10380.3	10333.1
2:30:00	0	9770.57	12844.1	9968.46	10395.3	10489.6	10198.7	10651.9	10324.4	10265.1
2:45:00	0	9660.3	12739.6	10046.8	10414.7	10511.4	10188	10619.8	10355.5	10314.7
3:00:00	0	9605.28	12698.8	10024.2	10384.2	10493.6	10182	10594	10318.9	10277.6
3:15:00	0	9569.22	12469.8	10023.2	10386.9	10474.1	10120.6	10551.8	10289.6	10249.6
3:30:00	0	9575.44	12534.9	10039.6	10409	10488.7	10132.1	10541.7	10279	10242.2
3:45:00	0	9509.7	12525.6	10005.7	10370.4	10443.8	10094.8	10502.8	10236.6	10190.7
4:00:00	0	9598.07	12539.1	9968.55	10371.2	10460.4	10099.1	10510.9	10232.1	10183.5

Reading Date and Rainfall (in.) are coming from elsewhere. The first 100px is mostly black, this part of the scanner is above ground and since it's night there is little for the scanner's light to reflect off. A graph of the 400-500px region from 00:00 to 4:00:00 looks like:

The numbers above unfortunately represent the day prior to the rainfall. Whoops. In the morning I'll have numbers for the proper day and be able to compare them with data from the City of Ann Arbor's Rain Gauges. Some time later this week I'll post the results.

I'm wondering if I can show to some degree of reliability how far and how quickly the rain is penetrating into the soil with this setup. I guess I should build/buy some soil sensors at this point to compare : )

Also, currently using Imagemagick's identify -format '%[mean]' command to infer "image brightness", but I'm really not sure what the command is doing / how it comes up with the numbers it spits out....

So much to learn!

Updated: 
Comparison of rainfall vs. image brightness for June 14th. This is a 100px snapshot ~4 inches below the surface. Both the rainfall and image brightness values were remapped from 0 - 100.

I'm doing a couple things here that I'm pretty sure are bad ideas.
1. I'm analyzing the jpeg, not the original tiff file.
2. I'm analyzing a section of a copy of the original jpeg, more loss : )
3. I have no real clue what I'm doing with the math. I think I used the same method I used to remap and constrain light sensor values on an arduino project from years ago for this (return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min;)...
4. Wheeeee!

Imagemagick

Imagemagick is a wonderful suite of tools that allow people to do lots of interesting things with images. Combining it with very minimal scripting knowledge lets one easily automate processing of large batches of images.

I used ImageMagick's compare tool to highlight in red the difference between 286 sequential images captured over 24 hours. I then used IM's convert tool to remove all other colors, create a transparent background, and finally stack each image onto the next.

Avconv was then used to turn these into a short 10 second video.

It's not terribly clear, but it does highlight the path(s) worms are taking. The large blob of red that shows up at the top is sunlight penetrating the first inch or two of topsoil/debris.

Very little attention to any sort of detail has been taken with this. This was a terribly fun distraction from documenting the actual setup - which I should really finish up in the next week or two : ) Regardless, I am constantly amazed at the amount of blood sweat and tears that are freely available in the software world.

Portability & Visitors

I took a portable version of the scanning setup to the beaches of Lake Michigan last week. It was interesting, not as successful as I would have liked it to be but I did learn a bit, took some notes, and have a better idea of where things can be improved.

The home setup continued to work fine while I was gone. The biggest issue I have to resolve is storage, switching to a portable USB drive should resolve that, but I'll miss the joy of uploading everything to a remote server, crunching all the images into a video and uploading to youtube / wherever at high speed. It doesn't make sense at this point to justify spending $50 / month when a 1TB drive can be purchased for $70.

I

Aside from technical fun, it looks like we've had a creature of some sort dig a hole in front of the scanner! No good images of the animal itself, the limited depth of field only gives us blurry images of anything more than 1 or 2 mm away from the scanner.

Documentation Progress + Battery Power

Brief updates

 - Incomplete process notes for Raspberry Pi setup

 - Rough draft of Scanner Prep

 - Rough draft of Burial Process

Lots of work still needed on the Pi setup process.

 - Still trying to decide how much information to include here

 - - Is it necessary to discuss initial image / OS installation, network setup, 

 - - This version assumes everything happens on the Pi - lots of pros & cons with this : )

 - Still need to compile a section on device setup (hub/scanner/wifi) along with pros & cons of different device choices.

I would love to spend more time with someone discussing and either building or linking to information that helps myself and others understand what can be seen in the images.

 - Root Growth

 - Insect life / processes that we're able to see

 - Soil type

 - Seasonal differences

Last, another objective has been to get battery powered version running. I spent a couple hours today setting up a battery (on loan from the wonderful AADL). At roughly 500mA idle and 900mA scanning (figuring 1 min/scan) should provide plenty excess energy to document for a solid day. (27,000mAh Battery).

Though my current understanding is that I can not charge the battery AND power the setup at the same time. The specs for the battery specify a max of 1 Amp output. With a WiFi dongle power hitting 1007mAmps for roughly 30 seconds at a time. It was recommended I avoid exceed the max amperage, even if just a little.

Documentation

Slowly documenting things! I'm aiming for minimal documentation that communicates as much as possible. This is proving to be difficult : ) This also meant burying another scanner : )

I currently plan on breaking documentation into three sections:
1. Setting up the Raspberry Pi to automate scans
 - Process documented, needs testing and significant clean up before starting rough draft

2. Preparing a scanner for burial
 - Photos taken and rough draft done, needs review

3. Burying the scanner
 - Photos taken and rough draft done, needs review

The last two sections are pretty straightforward. The first section needs a couple dedicated afternoons cleaning up the existing documentation, testing from scratch, and finding a balance of what+how much to communicate. I would love to see if the entire project could be done by a group of young students, and that brings up a number of questions.

What and how much to communicate really depends on the goal of whoever's perusing it. It's somewhat amusing that the easiest way to document is to document everything in excruciating detail. Which is probably a buzz kill for most...  Trying to take into account things like:
 - Would it help to include automated publishing of images + videos.
 - - Do we cover rsync, backing up, ssh, etc.? Or discuss GoogleCL?
 - Is local access to images and videos preferred, and then they can decide what to do with it?
 - - Do we discuss methods on Windows & Apple to process images and video?
 - What's a good price point to assume? Minimal: ~$100. Ideal: ~$150.
 - - This changes how much data can be recorded and methods of storage.
 - Do we keep it to the command line environment?
 - - Aside from viewing videos it's all been SSH. This can be a huge barrier for some.
 - Do we discuss open source software, hardware?
 - - This whole project would likely not have happened without it.

Mmmmm, food for thought! Sounds like I need to spend some time communicating with people who might want to use it, and what their goals and desires might be.

Scanner Unresponsive - What Next : ) ?

For the past 24 hours the scanner has not been functioning. Initially the top half of the image displayed the same repeating line of pixels, and a power cycle resulted in the device no longer appearing (lsusb) on the Pi

Both problems have separately occurred before. The former problem happened when the scanner was first buried. Dirt compacted too forcefully against the scanner meant the sensor was unable to complete a full pass, resulting in a similar image. The latter happened with the device reappearing after a day or two and several power cycles un/replugs, cause unknown.

So far the scanner has:
 - Spent ~6 months below ground
 - Captured ~17,000 images
 - Survived above-ground temps well below freezing

I'll wait a day or two more before going beyond power cycle and un/replugging the device : )

Meanwhile:
 - Automated uploading images to a remote server (Linode) and building daily videos.
 - - Next Step: Automate compilation of weekly and monthly videos, post online.

- Automated backup of the original images to Amazon S3 + file cleanup.
- - Next Step: Set it up to run regularly (right now I'm running the script instead of setting it up as a Cron job)

- Documentation / Build Guide.
- - Next step: Use the backup Canon Lide 20 to assemble the physical build guide
- - Next step: Review notes on existing process and create:
- - - Simple Pi guide (Just to view images as you like from the scanner)
- - - Timelapse Pi Guide (Include automation for creating timelapse video)

Beyond the above, there are two variations on this project that I would like to pursue:
1) Building a cylindrical scanner. Scrap the frame/body of an existing scanner, mount it to a glass cylinder and have it scan 360 degrees. I think it would be easier (physical constraints) to get it to scan from the outside in, but I think it might be more interesting to get it to scan from the inside out (buried in the ground). Similar to existing Mini-Rhizitrons, but with a 360 degree view.

2) Building a webcam version. Similar to existing Mini Rhizotrons, but significantly cheaper. This is how this project originally started. The image quality and viewing area will be reduced, but we'll have a significantly higher frame rate and nearly real time (ms delay) viewing. Taking an image once a day seems fine for tracking root growth, but tracking the path of certain bugs below ground seems like a 1 frame / minute frequency.

It would be wonderful to include additional data for both build options, ie:
 - Moisture
 - Humidity
 - Temperature