Checking a Directory for Changes with Python

This has been something I've been meaning to write a post about for a while, and I finally found myself with a free moment to document the process!

I recently needed to develop a python script that I could utilize for checking a specific file directory for changes.  By "changes", I mean any additions, deletions, or modifications to the files located within a specific directory.  For this exercise, I was not concerned with the type of change that was occurring, but only the time/date the change occurred.

Below is the script I came up with for this process, which reports on the 'Date Modified' column for a specific directory:

                                                                                                                               
#Quick and dirty method for checking last modified date
#for a folder or file path
#Tim Hohn 11/15/2017

#import modules
import os.path, time

#print date modified for specified file directory
print("Export_Test Folder - Last modified: %s" % time.ctime(os.path.getmtime("H:\Desktop\Test\Export_Test")))
                                                                                                                                

Running this script once every morning has been helpful towards keeping up with changes to a specific directory, especially in responding to new documents that have been uploaded that may require immediate attention.  Setting the script to run at a specific time through Task Scheduler, further assists the process and eliminates the need for me to remember to check the directory for changes every day.

Alternatively, in my research for the above process I came across a "watchdog" script that will specify the type of change (added or removed) that occurred.

                                                                                                                                 
#Watchdog designed to alert the user when files have been
#added, removed or updated within a directory
#Original Source Here:  http://timgolden.me.uk/python/
#win32_how_do_i/watch_directory_for_changes.html

#Import Modules
import os, time

#Set Path to Watch
path_to_watch = "H:\Desktop\Test\Export_Test"
before = dict ([(f, None) for f in os.listdir (path_to_watch)])

#Print Current Time
print time.asctime( time.localtime(time.time()) )
      
#Watch for changes to the directory specified in path_to_watch
while 1:
  time.sleep (0.5)
  after = dict ([(f, None) for f in os.listdir (path_to_watch)])
  added = [f for f in after if not f in before]
  removed = [f for f in before if not f in after]
  if added: print "Added: ", ", ".join (added)
  if removed: print "Removed: ", ", ".join (removed)
  before = after
                                                                                                                                  

So there you have it!  A couple of different methods for checking a directory for changes within python.

Ideas for next steps to make these scripts more useful:
          1.  Generate a text report of changes daily
          2.  E-mail report to personal e-mail daily
          3.  Run overnight and have report ready and sent to inbox in A.M.

How Did You Contribute To OpenStreetMap?

Since 2013, I have been contributing to various projects with the Humanitarian OpenStreetMap Team (http://tasks.hotosm.org/). While my contributions have been few and far between in recent years, taking part in these projects has been both enjoyable and rewarding for me (and I've met some really cool people along the way). Recently, I discovered a pretty neat website that allows you to view your individual OpenStreetMap (OSM) contribution statistics.

How Did You Contribute to OpenStreetMap?
(http://hdyc.neis-one.org/)

Looking at my contributions by year (see above), I took a wild dip from 2014 to 2015, after taking a new job in the GIS consulting world. By 2016 my OSM contributions became non-existent, as more and more of my time was gobbled up by work and life commitments (Who knew adopting a dog and getting engaged could take up so much of my time? ;P). Free time in the evenings was a rare treat, and sadly OSM fell by the wayside. In the Summer of 2017, I found myself with a new opportunity working with the State of Minnesota. With a ring on my finger and the puppy now a well-trained (sort of) fully grown 100-lb beast, I found myself with an itch to re-enter the OSM Community. As shown below, I have been contributing to several HOTOSM tasks related to Hurricane Maria in Puerto Rico. I couldn't be happier to be back, and would urge anyone who is interested to contribute as well! http://tasks.hotosm.org/


For the most part, my contributions have consisted of volunteer coordination and arm-chair digitizing. I have always felt an odd satisfaction from letting my mind go blank and digitizing geographic features, whether it was in ArcMap, QGIS, or Java OpenStreetMap (JOSM Editor). There isn't much to it, but I think there's something about getting into a rhythm and seeing points, lines, and polygons appear on a map, and seeing a project come together that is really satisfying to me. More than anything, it feels good to contribute to something so positive, and I hope to always find time for the OSM Community.

Using Microsoft Access to Track Tasks

Earlier this year I experienced an increase in workload, as I took on a more central roll in developing GIS solutions and project support within my company.

To deal with the increasing number of requests for GIS support I was receiving, I wanted to set up a GIS Task Tracker database/spreadsheet that I could use to easily input and track requests.  In addition, I would have a number of other GIS staff assisting with the project, so I felt task management would need to take center stage if the project was going to continue running smoothly.

I began with a template in Excel, but it quickly became clear that this approach wouldn't work....our spreadsheet became riddled with errors and many staff were forgetting to enter tasks.  I decided to scrap my original Excel document and use an Access database, in part because of a number of benefits I saw:

     1.  Access could be easily linked to our Outlook accounts.
     2.  Tasks could be directly e-mailed to GIS staff through this Outlook connection.
     3.  A task form could be utilized, for easily entering accurate task information.
     4.  GIS staff could quickly and easily communicate with project staff with regard to time frames and workload issues.
     5.  Attachments could be added to each individual request.
     6.  Finally, Excel is ultimately a tool for small scale data analysis.  It became clear that Access was the right tool for this complex data management/storage task.....perfect for this exercise.

In my research, I was fortunate to find an (almost) perfect template developed by Microsoft here:

https://support.office.com/en-us/article/Use-the-Task-Management-Access-Database-template-50376839-8cc6-4d29-ac1b-b9fce69605f8


The original template wouldn't need much editing on my end, just a few extra fields and changes to the primary form used for adding new tasks:

View of the task tracker in Access

Adding tasks in the table view was simplified with the creation of an entry form:

View of the Entry Form

With the Task Tracker in place, my workload became much more manageable and allowed me to generate queries and reports detailing the influx of recent requests.  I would highly recommend this database template for anyone looking to manage GIS requests within a medium or even large GIS work group.