Google Maps data

I've been manually posting vendor locations into Google Maps using address info from receipts. On the screen, just above the map graphic, is a tantalizing link labelled "View in Google Earth." But instead of seeing some spiffy Google Earth magic, when I clicked it, all my browser did was download a file named Art610 Magee.kml.


So what happened?

Well, I have Google Earth installed on this computer but it didn't automagically launch. I'll have to fix that. In the meantime, I'm installing the Google Earth browser plug-in to see what happens when that's in place.

The cool part: Give me a file and I need to know what's in it. So, I opened the .kml file in Notepad and learned that it's just an XML data file that uses a data format that Google Earth recognizes.

Here's the DTD that I can discern from the file:

<kml xmlns="http://earth.google.com/kml/2.2">
<Document>
  <name></name>    /* the map name */
  <description><![CDATA[]]></description>
  <NetworkLink>
    <name></name>  /* the map name */
    <Link>                  
      <href></href>  /* URL to location on map */
    </Link>
  </NetworkLink>
</Document>
</kml>

That means a couple of things:

1. I can generate my own .kml files automatically from data so that they can be opened by Google Earth.


2. I can continue to map my vendors in Google Maps and upload all the URLs and other data shown below into my database using batch processing. This simplifies data collection dramatically.

Unfortunately, it appears only to export one data point at a time. I have to see if it's possible to download the entire set of map points all at once.

NeatReceipts

Info about the NeatReceipts scanner:

Sample Screen:

Software Specifications

• Images can be saved in JPEG, PDF and other popular formats
• Data can be sent to .XLS (Microsoft® Excel®), RTF (Word), Outlook, .OFX and .QIF (Quicken®), .TXF (TurboTax®), .IIF (QuickBooks®), .CSV, HTML, and PLAXO
• Reads U.S. and Canadian receipts and business cards
• Database can scale to approximately 1.5 million receipts 

Given the support for up to 1.5 million receipts and the ability to export .csv files, it seems like this would be an ideal long-term solution for doing the data entry.  

Also, the technical specs for the two available scanners mean I can get either

1. the mobile scanner (scans up to 600dpi, weighs under 11 ozs) which I can carry around in my backpack to do immediate or frequent scanning of receipts rather than waiting 'til I'm home, ($230) and
2. the adf tabletop scanner -- automatic document feeding up to 10 receipts at a time, up to 600dpi, one- or two-sided ($500).

My Google Map

Here is a link to my  Google Map, Art610 Magee, where I'm marking purchase locations.  It's listed as a private Map so you have to have the URL below to access it if the link doesn't work.

Art610 Magee

http://maps.google.com/maps/ms?ie=UTF8&hl=en&msa=0&msid=100628197335174745221.0004632bdd5570f26961b&z=9

Project progress Report

To date, I've scanned a hundred or so receipts.  

I research technologies to speed up the receipt data collection.  There is a product called "NeatReceipts" that purports to scan the receipts and pull the data off of them automatically.  It's out of my budget right now so I'm opting for a home-grown approach at the moment.

I've created a mySQL database for holding a bunch of information from each receipt.  Ideally I'd like to capture information about the actual items purchased but for the time being, I'm collecting:

• receipt info
• ID - a unique receipt identifier for the database (primary key)
• date
• time
• total
• vendor info 
• name
• store no
• address (street, city, state, zip)
• phone
• salesperson name or number
• Google Maps URL
• receipt image
• scanned?  (scanned receipt)
• quickened?  (in Quicken)

At the moment, I'm gathering the Google Maps URL info by manually mapping vendor locations in a map I've created. Once I find the location, I copy the link to the location and enter it into the database. By manually collecting these, I may be able to parse out lat/log or GPS data to use by some other means (once I find out the meaning of the geocode info in the URL) or I could use the URL for some other proof-of-concept "behavior" while I figure out the API.

I also want to find out the best way to manage the images so they can be referenced by any program I've created.  At the moment, I'm adopting the convention used frequently by document management systems -- the receipt image field holds the full pathname where the image file can be found rather than the actual image file itself.

I have about 30 receipts recorded in the database right now.  I plan on creating a better data entry mechanism than I'm currently using to speed up data entry.

Project process

So my first project is going to be to explore documenting my locations and movements using Google Maps or Google Earth and my receipts. It lays a groundwork for more complex explorations if I can setup the data store for the purchase data and the receipt images and learn how I can create a mashup that relates that data to locations in Google.

I have a bunch of questions and decisions about the process of creating this project. In particular, I don't know yet what I want to do with the physical artifacts (the receipts) but I have a really strong sense that they're going to be important to me in some aspect of this project so I want to handle them well from the outset.

Questions

Overarching question:  How do I capture all the data for the project so that it is project-agnostic. That is, I don't want to selectively collect data or alter receipts in a way that creates more work or limits me somehow for future projects. Where is the sweet spot between that and bogging myself down in data collection?

I really believe that a lot of the things that engage me in my art practice are going to deal with the relationship between virtual information (or even social) space and the physical world. As a result, I need to create a mechanism for managing the receipts once they're scanned and the data from them is collected.

I'll need to create serial numbers for the receipts for relating images with data records and, ultimately with the physical receipts.

I want to preserve the integrity of the receipts as much as possible without marking them with data that "reveals" the process of managing them. However, given degradation of paper and ink over time, I've decided that I'm going to have to mark them with serial numbers.

I've decided to scan the receipts before numbering them so that the digital artifact will show the original, unadulterated receipt.  Ironic, I think.  The real receipt will have "tool marks" from my work.  The digital one will be "purer" in that respect.

Do I add an image layer that has the serial number so that can be shown or not based upon how I plan to use the image? This is laborious so is it worth the effort?

What kind of numbering scheme should I use? Should they be serialized numbers or should I use a coding system? 

Look up Bates numbering and Dewey decimal system. Is there some numbering scheme that has some implied meaning just by its use (like legal or library in the two preceding examples) that might add an interesting character to this process? 

Do I buy a stamp that auto-increments and capture that data -- it's the least labor intensive but I have to add a field to the database to hold that information. Or do I write the auto-incremented primary keys from my data store on the images.

Are there any ideas about "mapping" that I want to explore with this project from my 511 Maps class or should I just do reading and research  for the second project?

I need to answer the following questions about the form of this project based upon my programming skills and the time allotted  Among them...

• Is there enough time in the term for me to learn the APIs to allow me to do a Google Maps (or Google Earth) mashup?


• I have to decide between continuing what has been a proof-of-concept manual mapping of receipt locations on Google Maps or figuring out how to map them dynamically.


• If I can't dynamically map receipts in the time allotted, what data do I include in the manual map points and is there any meaning in doing that.  Are there other Google Map features that I can use like coloring or tagging of map points, for example, to communicate something about the related data?


• Is there enough time in the term to show purchases temporally using Google Earth.  If not how do I represent multiple purchases at a single map point

Decisions so far...

• Scan each original image at 600dpi   Pros: I can enlarge images for printing or project them for an installation. Cons: Storage requirements are relatively huge for the entire corpus of receipts.

• Mark each image on back in pencil with "s" for scanned and "q" for entered into Quicken (if I record them there also).  Pros: Eases the tracking of which receipts have been recorded.  Cons: alters the original receipt with artifacts of the "process" of creating this work.

• Mark each image after scanning with a unique number as determined above. Pros/Cons, same as above.

• Continue scanning receipts and recording data while a numbering scheme is established.  Pros: continue progressing with receipt scanning.   Cons: have to go back to scanned receipts and number them correctly.

Practice Process

Okay, so I have one major project to tackle -- how to I construct an environment (virtual, physical, both?) that helps keep me on track doing my work for my Art610 Special Projects class. It's an important question beyond the immediate term because I think that this same effort can be applied to my art practice at large.

I've floundered in the 18 months since I've been in an art class. Work, math classes, personal priorities, etc. always have a way of asserting themselves when the imperative of creating some deliverable for a class assignment is gone. My electronics box sits full (of toys and promise) and gathers dust in my garage. I know where it is at all times. I know the lovely gadgets it contains. And it sits, waiting. My art practice is a larger priority for me but one, I tell myself, that has to wait until the more practical business of my life is addressed.

At that rate it will always slip down the list of priorities.

So this is what I've decided to do this term with an expectation that I will continue to apply these principles beyond the semester to my larger art practice at least until that creates a momentum that carries itself along.

I've set an alarm on my phone. Every Tuesday and Thursday at 5:45, 15 minutes before our scheduled class section is to start, an alarm will sound. For those days where we don't meet, that's when I set aside time to work on research or my projects.

My programming class deliverables are usually Tuesday or Thursday meaning that Monday and Wednesday are the crunch days on those so I think this will work well.

Tuesday is research day. Thursday is project day. That's the day when I plan project stuff, create infrastructure, organize, document, etc. Weekends I execute project work on whichever day we don't have a lacrosse game to go to. The rest of the time I commit as I can.

It's mundane, making decisions like that. But between my ADHD and my busy life, I really believe that if I don't, it won't happen.

Process, process, process...

Right now at work we're re-examining everything about how we work, how we handle data and what the security and strategic implications of the decisions we make about the technologies and procedures to support those processes are. It's an exciting time. I love processes.

So my head is completely in that space as I struggle to start down a path of semi-independent study for my Advanced Projects class. What processes do I need to establish to succeed in the class -- to make meaningful progress on projects and to meet the requirements of the course itself.

So I suppose I have been engaged in meta-process assessments. I love meta things. I'm a meta kind of girl (we already know I'm not a blogging kind).

I've decided I have to focus on establishing two separate processes (actually three) to get through this term. The first -- how can I make certain that I carve enough time out of my schedule to do the research and project work that I have to do for my class. The second is an examination of the process required to execute my two projects. I don't know yet whether that same process will serve both projects or if I have to start a new one once the second effort is underway.

Oh the burden of blogging...

I'm not a blogging type of girl. I suspect, having grown up in a world where we learned about one another almost exclusively from live person-to-person contacts, going one step further and publishing my ideas and ruminations on a public site is really uncomfortable to me. I mean that to some extent we were able to control not only the content of the message about us but also how wide a net we cast in broadcasting that message. That is, tell one friend and a small group knows something; tell all your friends and that same news becomes common knowledge among your acquaintances. Put it on a website, however, and the guy that sits down next to you on BART may know more about how you think and what you value than many of your friends do. Worse, if you publish your picture, he knows who you are as he sits beside you and to you he's a complete stranger.

As I said -- uncomfortable.

To add to the irony, not only do I make my living creating mechanisms to share information via the Web, through web sites, internet portals, etc. but I'm also an early-adopter of emerging technology. I was the first of my friends to have a PC (I built it myself), a Palm Pilot, an iPod, my own domain name, an email account, etc. And yet I prefer the feel of paper, its toothiness and weight. I love my bright journals, red and black, with their satisfying heft and soft leather covers. I love how they give and flex when I feel inexplicably compelled to bend them.

I have also come to love the scratch of pencil on paper. I like the difference between filling a space with hash marks or taking my pinkie and smudging that space to create a satisfying, rich, dark fill. The marks on my virtual page, my blog, are each representations of 16 bit integer values. I can construct what I wish with them, within reason, but my tools are rigid. They are pre-defined, prescribed, legislated by "governing bodies." On the contrary, my random scratches with pencil, pen, erasers are my own. If I want to use ketchup, or masking fluid I may.

Blogging, web publishing in general, robs me of the tactile pleasures I get from writing.

Art610 Project Ideas

This semester I think that I will focus on exploring a single "idea" in two different ways. I have been collecting receipts for all purchases that I have made for the last three years and several months.

Here are some questions that I am interested that I may be able to address with my receipts

• Make tangible the amount and nature of data that can be collected just as a side-effect of living our lives in our commerce-centric culture.


• What "story" can people construct about a person based upon reviewing their purchasing data.


• Can purchase data be predictive of a person's behavior (where they'll be, when, what they'll do, etc.) If so, what questions does this raise about personal safety, the loss of anonymity in our culture, etc.


• What does "permanent" nature of record keeping mean for individual privacy?


• What does the public and long-term nature of some data mean in the context of mistake-making and redemption?


• Given impermanent and imperfect nature of paper record-keeping and the long-term nature of digital records, how does an individual defend themselves when challenged by entities like the IRS?

Possible projects

• Map the actual locations of purchases (differentiate between virtual purchase and "brick and mortar" ones. Use Google Earth or some other technology to move through space following the "path" of the purchases. How can we reconstruct the movement of a person through space and time based solely on purchase data.


• Feed purchase data into some kind of generative program to create graphics based upon it


• Create a locative media experience that "reveals" purchase behaviors as we pass by vendors. E.g., Name names, list purchases, etc.


• Contrast the receipts themselves as physical artifacts to their contemporary digital equivalents (impermanence of information, single-point of access, versus virtual access from anywhere)


• Qualify and quantify receipt data and find mappings between data perhaps using network mapping technologies. What patterns evolve in terms of nature of purchases, items purchased, locations, dates and/or times purchased. How can this data be represented to let someone discover who the "purchaser" is.

Preliminary Art610 Research proposal

I'm interested in learning more about how data is "mapped" in space and how data that doesn't have an inherent link to a physical place still may be mapped either in relationship to other data or in relationship to some objective external measure. In at least one of my projects I will be using a literal interpretation of mapping data by plotting purchase locations on a map. But I'm also interested in mapping relationships among the data themselves.

I want to explore three aspects of how researchers, scientists and artists represent data relationships visually. The first, is mapping of data so that data users can navigate among related information -- that is, they can find "neighborhoods" of related data and move within them, between them, etc. This is data mapping as a navigational aid. It's allows a stream-of-conscious movement through data following a path that is meaningful to the data user, perhaps revealing new ideas that have some relevance to them given where they are coming from (both literally and subjectively).

Second, I am really fascinated with how people derive "meaning" from data. In particular, I would like to understand better the ideas of Edward Tufte about visualizing data and learn both about other like-minded researchers as well as their detractors.

And finally, I am interested in what can be done artistically using data that has been mapped or quantified in some way that it can feed into some kind of generative system to create art that reveals the patterns or character of the data rather than the meaning of it.

For example, pieces by George LeGrady use commuter patterns to generate graphics that reveal nothing about the underlying data other than mirroring the frequencies of commuters through waveforms represented on pieces mounted in LA Subway entrances. Other pieces, like those commissioned by the new Seattle Public Library, tell a story about the movement of materials from the library out in to the public. What subjects, titles, topics are being checked out? How many people have borrowed books? How do those texts map to standard Dewey Labeling systems? All of these are represented graphically in such a way as to convey something about the borrowing public.