Oh the magic of technology!

So yesterday, having just gotten my paycheck, I headed for Costco to stock up on toilet paper and storage bins.  As is my wont, I wandered through each of the aisles with my cart, weighing possibilities, and fantasizing about how each new wonder I found would alter my life for the better. 

It's a game I play, put something in the basket, think about how wonderful or helpful it would be and then an aisle or so later, after having considered all those options and then how better or more responsibly to use my money, I return to the scene of the crime and return the merchandise to the shelf from whence it came.

Yesterday was different.  Well, not entirely different but different all the same.

I toured the TV aisles near the front of the store ("oh that 46" Sony seems wonderful"), passed by the monitors and securities system ("maybe that will help us find out who is entering our apartment when we're gone"), then turned down the office supplies ("mechanical pencils for Ian?").  Each time I pondered and then continued on.

That is until I turned the corner as I left the office supplies aisle.  Turn left -- mattress, turn immediately left again and there it was...

My NeatReceipts scanner.  Woo Hoo!!

I've mentioned it in previous posts but, in brief, this is the portable bar scanner that is one of a suite of hardware and software from NeatCo which allows me to scan images of receipts and to capture data from them automatically.

It uses OCR technology and other logic to pick up method of payment, credit card last 4 digits (xxxx-1234), total sales, vendor names, dates of purchases etc. and it will both connect to Quicken for data exchange with that software but will also allow me to export data from the receipts into Excel or a .csv file that I can then pull into my project database.

Last night I scanned 88 receipts and captured dates, totals, tax and vendors for all of them.

88!!  That's just crazy talk.

Tonight I try to customize the database to capture my other data like time, employee, databaseIDNo, etc.

I wonder if I can code a system to automatically update an online database with purchase data as I make the purchases (the scanner is very portable and comes with its own case) so those data reflect immediately on a web interface. 

Class Presentation

I've been primarily doing research and groundwork on processes and technologies to do my work this semester.   (Terms shown in parenthesis are labels attached to related posts in this blog.)

1. Processes  (process, artpractice)
   1. art practice
   2. project execution
2. Scanning receipts
   1. manual scanning  (process)
   2. neat receipts  (technology)
3. Receipt Data Store (technology)
4. Presentation Technologies  (technology, mapping)
   1. Google Maps - Art610 Magee
   2. Google Earth - sample trip
   3. SketchUp
5. Mapping (technology, mapping)
   1. Theory 
   2. Technologies 

Some Mapping Resources

Mapping resources that have some kind of GIS or temporal data component:

1. TimeMap "TimeMap is a mature, Java temporal web mapping/animation application. Use as a standalone web site or CD-ROM app, or as a sophisticated client-server system using an XML or SQL metadata directory to source distributed, incrementally-loaded spatial data."
2. iDataMap "iDataMap software allows users to map economic and demographic data with different geographical boundaries on the same map. Our software contains an algorithm that allows users to create maps that represent one or more economic and/or demographic variables from a particular data source. iDataMap software also allows users to overlay two maps so that the user can visualize how variables from two different data sources interact. "
   
3. Starmapper:  might be useful for non-literal interpretation of map data.  "Starmapper is a maping utility that can read a map of "nodes", some files with "reports" and draw some pretty pictures. Originally for Stars! game, now it's Stars! and beyond! (you can make your own data model and plug it in)"  
4. Geodoggy:  "GeoDoggy (Perl CGI) is a web-based interface and national map dataset that enables users to render maps of specific portions of the US, DC, and US territories. The data is derived from the US Bureau of the Census Public Domain Tiger Line (TM) dataset."

Things about Google Earth

Things I've learned so far:

• I can import waypoint, tracks and route data from a GPS system.
  
• New placemarks are created using lat and long data.
  
• The data points imported into Google Earth from Google Maps have lat and log properties extracted and available in the waypoint's "Properties" dialog box.
  
• I can create folders which could allow me to organize data points by year of purchase.
• Google Earth animates zooming in to Street Level image display.
• GE Pro allows GPS data import into the map.
• GE Pro allows real-time GPS tracking
• GE Pro allows importing of spreadsheet data with up to 2500 points.
• GE Pro allows GIS data importing (ArcGIS?)
• Google has a product called "SketchUp" which allows us to create extruded 2D and 3D graphics.  This could be useful for representing multiple events (purchases) at a single location if it can be mashed up with Google Maps or Google Earth.

Things to find out:

• Can I automate "zooming" in to data points when Street Level images exist for them?
• Can I export placemarks I create in Google Earth to my My Maps map in Google Maps?
• Is it possible to programmatically alter the speed of movement from one waypoint to another based upon some piece of data?  This would allow me to simulate time differences among purchases.
• Find out how "real-time" GPS tracking works.
• What kind of GIS data are out there in the public domain that I may use in conjunction with my receipts?
• Get feature set and free vs. paid license feature comparison for SketchUp.

Resources:

• Google Earth Community - http://bbs.keyhole.com/ubb/ubbthreads.php/Cat/0
• Tutorial for creating custom kml's -- http://earth.google.com/intl/en/outreach/tutorials.html
• Instructions for putting a SketchUp graphic in Google Earth -- http://sketchup.google.com/support/bin/answer.py?hl=en&answer=36241
• Temporal data mapping: 
  1. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1839378
  2. http://www.esri.com/news/arcuser/1104/temporal.html

Google Maps data

I've been manually posting vendor locations into Google Maps using address info from receipts. On the screen, just above the map graphic, is a tantalizing link labelled "View in Google Earth." But instead of seeing some spiffy Google Earth magic, when I clicked it, all my browser did was download a file named Art610 Magee.kml.


So what happened?

Well, I have Google Earth installed on this computer but it didn't automagically launch. I'll have to fix that. In the meantime, I'm installing the Google Earth browser plug-in to see what happens when that's in place.

The cool part: Give me a file and I need to know what's in it. So, I opened the .kml file in Notepad and learned that it's just an XML data file that uses a data format that Google Earth recognizes.

Here's the DTD that I can discern from the file:

<kml xmlns="http://earth.google.com/kml/2.2">
<Document>
  <name></name>    /* the map name */
  <description><![CDATA[]]></description>
  <NetworkLink>
    <name></name>  /* the map name */
    <Link>                  
      <href></href>  /* URL to location on map */
    </Link>
  </NetworkLink>
</Document>
</kml>

That means a couple of things:

1. I can generate my own .kml files automatically from data so that they can be opened by Google Earth.


2. I can continue to map my vendors in Google Maps and upload all the URLs and other data shown below into my database using batch processing. This simplifies data collection dramatically.

Unfortunately, it appears only to export one data point at a time. I have to see if it's possible to download the entire set of map points all at once.

NeatReceipts

Info about the NeatReceipts scanner:

Sample Screen:

Software Specifications

• Images can be saved in JPEG, PDF and other popular formats
• Data can be sent to .XLS (Microsoft® Excel®), RTF (Word), Outlook, .OFX and .QIF (Quicken®), .TXF (TurboTax®), .IIF (QuickBooks®), .CSV, HTML, and PLAXO
• Reads U.S. and Canadian receipts and business cards
• Database can scale to approximately 1.5 million receipts 

Given the support for up to 1.5 million receipts and the ability to export .csv files, it seems like this would be an ideal long-term solution for doing the data entry.  

Also, the technical specs for the two available scanners mean I can get either

1. the mobile scanner (scans up to 600dpi, weighs under 11 ozs) which I can carry around in my backpack to do immediate or frequent scanning of receipts rather than waiting 'til I'm home, ($230) and
2. the adf tabletop scanner -- automatic document feeding up to 10 receipts at a time, up to 600dpi, one- or two-sided ($500).

My Google Map

Here is a link to my  Google Map, Art610 Magee, where I'm marking purchase locations.  It's listed as a private Map so you have to have the URL below to access it if the link doesn't work.

Art610 Magee

http://maps.google.com/maps/ms?ie=UTF8&hl=en&msa=0&msid=100628197335174745221.0004632bdd5570f26961b&z=9

Project progress Report

To date, I've scanned a hundred or so receipts.  

I research technologies to speed up the receipt data collection.  There is a product called "NeatReceipts" that purports to scan the receipts and pull the data off of them automatically.  It's out of my budget right now so I'm opting for a home-grown approach at the moment.

I've created a mySQL database for holding a bunch of information from each receipt.  Ideally I'd like to capture information about the actual items purchased but for the time being, I'm collecting:

• receipt info
• ID - a unique receipt identifier for the database (primary key)
• date
• time
• total
• vendor info 
• name
• store no
• address (street, city, state, zip)
• phone
• salesperson name or number
• Google Maps URL
• receipt image
• scanned?  (scanned receipt)
• quickened?  (in Quicken)

At the moment, I'm gathering the Google Maps URL info by manually mapping vendor locations in a map I've created. Once I find the location, I copy the link to the location and enter it into the database. By manually collecting these, I may be able to parse out lat/log or GPS data to use by some other means (once I find out the meaning of the geocode info in the URL) or I could use the URL for some other proof-of-concept "behavior" while I figure out the API.

I also want to find out the best way to manage the images so they can be referenced by any program I've created.  At the moment, I'm adopting the convention used frequently by document management systems -- the receipt image field holds the full pathname where the image file can be found rather than the actual image file itself.

I have about 30 receipts recorded in the database right now.  I plan on creating a better data entry mechanism than I'm currently using to speed up data entry.

Project process

So my first project is going to be to explore documenting my locations and movements using Google Maps or Google Earth and my receipts. It lays a groundwork for more complex explorations if I can setup the data store for the purchase data and the receipt images and learn how I can create a mashup that relates that data to locations in Google.

I have a bunch of questions and decisions about the process of creating this project. In particular, I don't know yet what I want to do with the physical artifacts (the receipts) but I have a really strong sense that they're going to be important to me in some aspect of this project so I want to handle them well from the outset.

Questions

Overarching question:  How do I capture all the data for the project so that it is project-agnostic. That is, I don't want to selectively collect data or alter receipts in a way that creates more work or limits me somehow for future projects. Where is the sweet spot between that and bogging myself down in data collection?

I really believe that a lot of the things that engage me in my art practice are going to deal with the relationship between virtual information (or even social) space and the physical world. As a result, I need to create a mechanism for managing the receipts once they're scanned and the data from them is collected.

I'll need to create serial numbers for the receipts for relating images with data records and, ultimately with the physical receipts.

I want to preserve the integrity of the receipts as much as possible without marking them with data that "reveals" the process of managing them. However, given degradation of paper and ink over time, I've decided that I'm going to have to mark them with serial numbers.

I've decided to scan the receipts before numbering them so that the digital artifact will show the original, unadulterated receipt.  Ironic, I think.  The real receipt will have "tool marks" from my work.  The digital one will be "purer" in that respect.

Do I add an image layer that has the serial number so that can be shown or not based upon how I plan to use the image? This is laborious so is it worth the effort?

What kind of numbering scheme should I use? Should they be serialized numbers or should I use a coding system? 

Look up Bates numbering and Dewey decimal system. Is there some numbering scheme that has some implied meaning just by its use (like legal or library in the two preceding examples) that might add an interesting character to this process? 

Do I buy a stamp that auto-increments and capture that data -- it's the least labor intensive but I have to add a field to the database to hold that information. Or do I write the auto-incremented primary keys from my data store on the images.

Are there any ideas about "mapping" that I want to explore with this project from my 511 Maps class or should I just do reading and research  for the second project?

I need to answer the following questions about the form of this project based upon my programming skills and the time allotted  Among them...

• Is there enough time in the term for me to learn the APIs to allow me to do a Google Maps (or Google Earth) mashup?


• I have to decide between continuing what has been a proof-of-concept manual mapping of receipt locations on Google Maps or figuring out how to map them dynamically.


• If I can't dynamically map receipts in the time allotted, what data do I include in the manual map points and is there any meaning in doing that.  Are there other Google Map features that I can use like coloring or tagging of map points, for example, to communicate something about the related data?


• Is there enough time in the term to show purchases temporally using Google Earth.  If not how do I represent multiple purchases at a single map point

Decisions so far...

• Scan each original image at 600dpi   Pros: I can enlarge images for printing or project them for an installation. Cons: Storage requirements are relatively huge for the entire corpus of receipts.

• Mark each image on back in pencil with "s" for scanned and "q" for entered into Quicken (if I record them there also).  Pros: Eases the tracking of which receipts have been recorded.  Cons: alters the original receipt with artifacts of the "process" of creating this work.

• Mark each image after scanning with a unique number as determined above. Pros/Cons, same as above.

• Continue scanning receipts and recording data while a numbering scheme is established.  Pros: continue progressing with receipt scanning.   Cons: have to go back to scanned receipts and number them correctly.

Practice Process

Okay, so I have one major project to tackle -- how to I construct an environment (virtual, physical, both?) that helps keep me on track doing my work for my Art610 Special Projects class. It's an important question beyond the immediate term because I think that this same effort can be applied to my art practice at large.

I've floundered in the 18 months since I've been in an art class. Work, math classes, personal priorities, etc. always have a way of asserting themselves when the imperative of creating some deliverable for a class assignment is gone. My electronics box sits full (of toys and promise) and gathers dust in my garage. I know where it is at all times. I know the lovely gadgets it contains. And it sits, waiting. My art practice is a larger priority for me but one, I tell myself, that has to wait until the more practical business of my life is addressed.

At that rate it will always slip down the list of priorities.

So this is what I've decided to do this term with an expectation that I will continue to apply these principles beyond the semester to my larger art practice at least until that creates a momentum that carries itself along.

I've set an alarm on my phone. Every Tuesday and Thursday at 5:45, 15 minutes before our scheduled class section is to start, an alarm will sound. For those days where we don't meet, that's when I set aside time to work on research or my projects.

My programming class deliverables are usually Tuesday or Thursday meaning that Monday and Wednesday are the crunch days on those so I think this will work well.

Tuesday is research day. Thursday is project day. That's the day when I plan project stuff, create infrastructure, organize, document, etc. Weekends I execute project work on whichever day we don't have a lacrosse game to go to. The rest of the time I commit as I can.

It's mundane, making decisions like that. But between my ADHD and my busy life, I really believe that if I don't, it won't happen.