Friday, August 12, 2016

Beetle Larva Cocoon, Backups and Pi Failures

Noticed this Beetle Larva (not sure, do you know?: ) forming a cocoon over the past few days, at this point it enters the Pupa stage where it will only be able to wiggle! I'm not sure how long this will last, and since it's in the compost bin I have to decide whether or not to disturb it (by turning the compost), or leave it be to observe!

I'm curious, over a ~6 hour period there is a significant color change in the Larva/Pupa. It's not clear to me what causes this. There's documentation about this with Butterfly Larva > Pupa transitioning, and I'm guessing it helps the Pupa / Cocoon to blend in better with its surroundings, but what process is causing this?

The Larva measures approximately 17mm x 2mm, and is buried about 20cm into the compost bin. Another reason why using a scanner is nice, we can make quick/easy/accurate measurements based on pixels.

Unrelated to the soil life, I've had two amusing technical issues over the past few days:
One: While I was toiling around with a script it looks like I deleted almost a day's worth of images. That's frustrating, but the timing of it is somewhat appropriate. I'd purchased a 1 TB drive awhile back to store images on but hadn't hooked it up in quite some time.

I think it would be appropriate to have the Pi save images to both the SD Card and the Hard Drive,  and once a day tar+gzip the files off to Amazon S3. This would afford me the option to play with one set of images knowing that if I did something horribly wrong I have a local backup.

Two: Around 1:30am yesterday the Raspberry Pi Model 2 B I was using croaked. I'm not sure what the status is beyond it not getting on the network (Hooked up via Ethernet), and an SD Card that was working in another Pi provides the same result. I'm still without an HDMI Monitor here, so everything is done headless, I'm currently unable to see what might be happening in the boot process. Thankfully a second Pi was ready to take its place.

Having a $35 computer in my case means it's easy to have a backup Pi. Thank you Raspberry Pi Foundation : )

Lessons here:
 - Have a process for swapping the Pi out when something goes wrong
 - - Backup SD Card
 - - Backup Pi
 - Have some notification that things are not as they should be
 - - Pi going off the network is not a huge deal, though is a symptom that the Pi may be down
 - - Not sure if there's a better way to alert me that there may be severe issues.
 - Have a process for saving images to two separate locations
 - - SD Card
 - - External USB Drive
 - Have a process for saving scripts in the same way
 - Automate Amazon S3 Sync of files.

Saturday, August 6, 2016

Fungus, Frame Rates and Glaciers

Radiolab has a wonderful podcast on trees, roots, and fungus. It's a delicious introduction that really puts into perspective how interconnected everything is. As in Trees eat Salmon. Don't believe me? Listen to the podcast or read this article.

The same day I listened to this I noticed a few bright white growths in the compost, neat! There's been lots of fungal growth in the compost pile, but I haven't specifically seen this one yet, at least not in these concentrations.

Fungus doesn't move as fast as the bugs we're seeing. I could probably capture one scan per hour and feel pretty good about capturing movement in fungus. But I'm capturing a scan every 15 minutes, and the video playback at this rate causes bug movement to look like a hastily recorded stop motion animation.

Frame Rates!
So I changed the scan to run every 2 minutes, and I've captured a day's worth of footage.

Prior: 1 scan / 15 minutes = 96 scans a day, played back at 30 Frames Per Second = 3 second video.
Now: 1 scan / 2 minutes = 720 scans a day, played back at 30 Frames Per Second = 24 second video.

While I like the smoother movement, I don't necessarily like how long the video lasts. Ooh, but screens these days can all play back at 60 FPS.

60FPS: 1 scan / 2 minutes = 720 scans a day, played back at 60 FPS = 12 second video. Looks even smoother, and plays back faster!

30 FPS
60 FPS
I had an option to either re-encode the video from images at 60FPS, or convert and re-encode the video at 60 FPS. I had not done the latter before, so:

avconv -i sc_20160805.mp4 -f rawvideo -b 50000000 -pix_fmt yuvj420p -vcodec rawvideo -s 768x1080 -y temp.raw

avconv -f rawvideo -pix_fmt yuvj420p -s:v 768x1080 -r 60 -i temp.raw -c:v libx264

Not sure what I'll do in the future, for now I'll just continue capturing and encoding videos at 30 FPS.

This is really all about storage.  I'm capturing a 600 DPI image once every 2 minutes. It's saved as a JPG which greatly reduces the storage space (with some image loss), but still costs 7MB / image.

7MB / image
720 images / day
5 GB / day of storage.

The Raspberry Pi has a 32GB SD Card on it, of which I can use ~28GB. That's only 5 days of storage. Not to mention the terabyte or so of storage I've already offloaded over the past couple years of doing this.

Amazon S3 is a pretty easily accessible storage method, it costs $0.03 / gigabyte. Right now I have around 300GB's on it which costs ~$9 / month. Given the increase in scanning frequency and resolution, I'll increase that by about 150 GB / month (5GB / day * 30 days).

That means I'll be paying an additional $4.50 (150 GB * .03 / GB) every month. Not so bad until a year goes by and now I'm paying $54 / month and increasing.

But I just realized there's this thing called Amazon Glacier, which is sloooooooooooow for retrieving data, but only costs $.007 / GB, and I can setup S3 to automatically dump any files that haven't been used in X days into Glacier storage. This sounds quite ideal. I'd love to here if there are better storage options. I do have a local drive I can store stuff on, but I trust Amazon a lot more than I trust my cheap drive.

Yay for Glaciers!

Thursday, August 4, 2016

7 Day 4k Test, Broken Images, Jitter

A seven day 4k Test. I'm not sure how well this works as I've yet to actually view it at 4k resolution on a screen that supports it. If anyone out there does, let me know how it looks ;)

The workaround I came up with for removing broken images so far has been pretty reliable. Out of 74 days a total of 7104 images have been captured and 329 were removed. 7104-329 = 6775 images, but somewhere I'm missing 20.

Jitter, with the Canon LiDE 20 scanner it would not always start in exactly the same location. This wasn't very noticeable at lower resolutions, but when zooming in on a higher resolution scan and flipping through a few images the jitter would become distracting.

Though still noticeable it's not nearly as bad as it was.

Tuesday, August 2, 2016

July Compost Bin!

The full video of July's compost bin can be seen below. I don't think we've ever had any sort of schedule for turning the compost, but now that I can see it I've started turning when the bulk of it is a delicious brown color...  I should read up on composting...

Mushroom caps seem to pop up regularly a week or two after the turning, but I turn the compost too quickly to see what might be feeding upon the mushrooms/hyphae. Fungus being one of the most important decomposers in our world, I'm curious to see what feeds upon it. It took me a moment to remember that we (humans) are one of those creatures, though I won't be capturing that image with a buried scanner any time soon. Apparently some forms of fungus feast upon other fungus, another example of just how amazing fungus is.

I'm still scanning at 600 DPI which gets me super excited to see what new things can be seen that weren't before.  I'm a little less excited by the data storage requirements...  If all goes well in a few days I should have a week of scrolling footage at 4k : ) Anyhow, last month's video at 768 horizontal pixels: