Category Archives: projects

Pedestrian Safety in Manhattan

For the final project in my Realtime and Big Data Analytics class at NYU, I worked on an analysis of the effectiveness of pedestrian safety measures in Manhattan with fellow students Rui Shen and Fei Guan. The main idea behind this project was to look at the number of accidents occurring within a fixed distance of an intersection in Manhattan and determine if the accident rate correlated with any features of the intersection, such as the presence of traffic signals or high traffic volume. We used a number of big data tools and techniques (like Apache Hadoop and MapReduce) to analyze this data and found some rather interesting results.

The first step was to collect data about intersections, accidents, and various features of the intersections. To do this, we relied heavily on open source data sets. We extracted the locations of intersections, speed bumps, and traffic signals from OpenStreetMap. We used NYC Department of Transportation data for traffic volume information, traffic signal locations, and traffic camera locations. Finally, we used NYC Open Data for information on accident counts and traffic volume, as well as the locations of speed bumps, arterial slow zones, and neighborhood slow zones. Some of the data could be used mostly off of the shelf, but other datasets required further processing, such as normalizing traffic volume over time and geocoding the street addresses of traffic camera locations.

The next step was to merge the feature and accident data with the relevant intersections. To do this, we used big data tools to assign intersection identifiers to every corresponding feature and accident record. As Hadoop can’t natively handle spatial data, we needed some additional tools to help us determine which features existed within an intersection. There were three distinct types of spatial data that we needed to process: point data (such as accidents), line data (such as traffic volume) and polygon data (such as neighborhood slow zones). Fortunately, GIS Tools for Hadoop helped us solved this problem. The GIS Tools implement many spatial operations on top of Hadoop, such as finding spatial geometry intersections, overlaps, and inclusions. This toolkit also includes User Defined Functions (UDFs) which can be used with Hive. For this task, we used Hive and the UDFs to associate the feature and accident data with the appropriate intersections. We experimented with different sizes of spatial buffers around an intersection and decided that a twenty-meter radius captured most of the related data points without overlapping with other intersections.

Examples of the different types of spatial data we had to correlate with intersections: area data (blue), point data (red) and line data (green).
Examples of the different types of spatial data that could exist within an intersection: area data (blue), point data (purple) and line data (green).

Once all of the relevant data had an intersection identifier assigned to it, we wrote a MapReduce job to aggregate all of the distinct data sets into one dataset that had all of the intersection feature information in a single record. In the reduce stage, we examined all of the data for a given intersection and did some further reduction, such as normalizing the traffic volume value for the intersection or calculating the sum of all of the accidents occurring within the intersection buffer.

The last step was to calculate correlation metrics on the data. To do this, we used Apache Spark. We segmented the data set into thirds by traffic volume, giving us low, moderate, and high traffic volume data sets.  We then calculated Spearman and Pearson correlation coefficients between the accident rate and the individual features and then analyzed the results. Although most features showed very little correlation with the accident rate, there were a few features that produced a moderate level of correlation. First, we found that there is a moderate positive correlation between accidents and the presence of traffic lights. This seemed odd at first but on second consideration it made sense. I have seen many random acts of bravery occur at traffic signals where people would try to cross the street just as the light was changing. Second, we found that there was a moderate negative correlation between high traffic volume and accidents. Again, this was not immediately intuitive, but our speculation was that drivers and pedestrians would be more cautious at busy intersections.

As this project was only a few weeks long, we didn’t have time to do a more in-depth analysis. I think we would have found even more interesting results had we done a better multivariate analysis which would allow us to calculate correlation metrics across all variables instead of just examining single variant correlation. One observation that we made was that intersections in high-traffic business or tourist areas have different accident profiles than intersections in residential areas. Therefore, it would be wise to include more socio-economic information for each intersection, such as land-use information and population information.

Despite the time constraints, the small amount of analysis we did was very interesting and made me look at something as simple as crossing the street in a whole new light.

Live Streaming Video With Raspberry Pi


Much to my delight, I discovered that a pair of pigeons are nesting outside of my window. I decided to set up a live streaming webcam so I can watch the young pigeons hatch without disturbing the family. Instead of buying an off-the-shelf streaming solution, I used a Raspberry Pi and a USB webcam. Here is how I set up live streaming video using my Pi and Ustream.

For this project, I used a Raspberry Pi Model B+, a USB WiFi adapter, a microSD card, a USB webcam and a 5 volt power adapter. When selecting a USB webcam, try to get something on the list of USB webcams known to work with Raspberry Pi. It will save you a lot of headaches in the long run!


To start, download the latest Raspbian image and load it onto the SD card. My favorite tool for doing this on a Mac is Pi Filler. It’s no-frills, easy to use and free! It may help to connect the Pi to a monitor and keyboard when first setting it up. Once the Pi first comes up, you will be prompted to set it up using raspi-config. At this time, it’s a good idea to expand the image to use the full card space and set the internationalization options to your locale so that your keyboard works properly.

Once the Raspberry Pi boots up, there are a few things that need to be updated and installed. First, it’s a good idea to update the Raspbian image with the latest software. I also like to install webcam software, fswebcam, so I can test that the webcam works before setting up video streaming. Finally, you’ll need ffmpeg, which is software capable of streaming video. The following commands will set up the Raspberry Pi:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install fswebcam
sudo apt-get install ffmpeg

After installing the software, it’s a good idea to check whether the webcam works with the Raspberry Pi. To do this, simply take a picture with the webcam using fswebcam. This will attempt to take a single photo from the webcam. You can do this by running the following command:

fswebcam photo.jpg

If the photo looks good, then you are ready to set up streaming video. First, set up a Ustream account. I set up a free account which works well despite all of the ads.  Once you set up your video channel, you will need the RTMP URL and stream key for the channel. These can be found in Dashboard > Channel > Broadcast Settings > Encoder Settings.

Next, set up video streaming on the Raspberry Pi. To do this, I used avconv. The documentation for avconv is very dense and there are tons of options to read through. I found this blog post which helped me get started. I then made some adjustments, such as using full resolution video, adjusting the frame rate to 10 frames per second to help with buffering issues and setting the log level to quiet as to not fill the SD card with logs. I also disabled audio recording so I wouldn’t stream the laments of my cat for not being allowed to ogle the pigeons. I wrote this control script for my streaming service:


case "$1" in 
      echo "Starting ustream"
      avconv -f video4linux2 -r 10 -i /dev/video0 - pix_fmt yuv420p -r 10 -f flv -an -loglevel quiet <YOUR RTMP URL>/<YOUR STREAM KEY> &
      echo "Stopping ustream"
      killall avconv
      echo "Usage: ustream [start|stop]"
      exit 1

exit 0

Make sure the permissions of your control script are set to executable. You can then use the script to start and stop your streaming service. Before placing the webcam, it’s a good idea to see if you need to make any additional updates to the Raspberry Pi for your webcam to work. The webcam I chose, a Logitech C270, also required some modprobe commands to keep from freezing. Finally, it’s a good idea to add your control script to /etc/rc.local so that the streaming service automatically starts in case your Raspberry Pi accidentally gets rebooted.

And that’s it! There is a multiple second delay to the streaming service so within a minute you should see live streaming video on Ustream. One word of caution on working with the Raspberry Pi: be sure to shut down the Raspbian operating system before unplugging the Raspberry Pi. The SD card can become corrupted by just unplugging it. This will cause the operating system to go into kernel panic and refuse to boot.  Sadly, the only solution for this is to reinstall Raspbian and start all over again.

Once my webcam was up, I found that I had some issues positioning the camera effectively. To solve this, I bought a cheap mini camera tripod. I then dismantled the clip of my webcam and drilled a 1/4″ hole in the plastic so it would fit on the tripod. I put a 1/4″-20 nut on the top of the screw and I was good to go!


I will be live streaming the pigeon nest for the next month or so on this Ustream channel (Update: the baby pigeons have grown up and left the nest, so pigeon cam has been taken down).  I’ve learned a lot about pigeons by watching them every day. The squabs should hatch during the upcoming week and I am excited to watch them grow!


DIY LED Strip Controller

On a whim I decided to add LED lighting to my desk hutch. I already had a reel of LED strips but nothing to control them. As I wanted to build a controller that afternoon, I constructed it from parts which could be purchased locally. The controller I made has an on/off switch and two knobs: one to control the brightness and the other to control the color of the lights. Here is how I built it!

This project required the following parts: two 10K ohm potentiometers with knobs, an on/off switch, a project box, a 5 volt power supply, a power jack, some wire, and a small Arduino compatible microcontroller. RadioShack sells the Arduino Micro, but I used a Teensy 2.0 I had on hand as it is a much cheaper alternative. Of course, you also need some programmable LED strips. I used some 5 volt WS2812B LED strips (similar to Adafruit’s NeoPixel strips). It’s also useful but not required to have some connectors for the LED strips so that they can be detached from the controller. I used some JST connectors from my project stash.

The first step is to make holes in the project box. I did this with my trusty Dremel. Drill five holes: two for the potentiometers, one for the power switch, one for the power jack and a one for where the LED strip wires will enter the project box. Once the box is drilled out, place the power switch, potentiometers and power jack into the project box. Solder the power wires to the components.

IMG_9702Next, I assembled the LED strips. If you are adding connectors to the LED strips, solder those on to the strips. If you’re connecting multiple strips, be sure that you bundle the wires together properly. I’ve found that using colored wire or marking wires with different colors of tape makes it easier to keep everything straight.

IMG_9722Next, solder the potentiometers and LED strips. Mark the data lines for the LED strip and the potentiometers so you know which wire corresponds to a given component. It makes coding the microcontroller easier.

IMG_9761Next, solder the microcontroller.  Keep track of the pins and their corresponding data lines. When soldering the potentiomenter to the microcontroller, make sure to connect the potentiometer data wire to pins that can support the analogRead function. These pins generally begin with the letter ‘A’.

IMG_9767Now it’s time to program the microcontroller. The simple code can be found here. Update the code to reflect the length of your LED strip and the pins that correspond to your components. Be sure to test everything!

IMG_9771Once you’ve verified that everything works, tape up any solder joints so that there are no shorts. Close up the box and you’re done!




photo 2I recently taught myself how to knit. I was interested in the mechanics of knitting, especially how it’s possible to weave string into cloth with just a few simple tools and techniques. It was also appealing because it’s something that can be done in small sessions rather than requiring long spans of continuous attention. Furthermore, knitting is a skill that allows you to make really cool things.

Knitting during a flight delay
Knitting during a flight delay

To get started, I picked up an awesome introductory knitting book, Stitch N’ Bitch. The friendly folks at Knitty City also helped me pick up some needles (size 10) and yarn that were suitable for a beginner.

ScarfMy first project was this simple ribbed scarf. Admittedly, it took me a long time to figure out the purl stitch, but I finally had some success with the English method of knitting. At the beginning, I made some mistakes and had trouble getting the yarn tension right which resulted in some oddities in the knitted fabric. By the end of the scarf, however, the stitches were even and consistent, resulting in a cool stretchable ribbed pattern.

Ribbed fabricFor my second project, I wanted to try something a little harder. I made the Official Kittyville hat. I used a cheap wool yarn so it wouldn’t be an expensive mess if I screwed the whole thing up. This required needles that were smaller (size 7) than the ones I used for my first project. The hat also involved some new techniques, such as knitting in the round, decreasing size, using double-ended needles, and picking up stitches in the middle of the fabric. Overall, the stitches were still slightly too tight, but the end result came out well. It’s definitely something I will wear when it gets a little colder.

Kitty hat

I’ve added lots of knitting project ideas to my ever-growing project list. I want to try combining conductive yarn and Fair Isle knitting to make functional and attractive knit circuits. Of course, there are lots of great non-technical projects in there too, like these Dalek mitts.

Knitting on a train
Knitting on a train

And, by the way, did you know that knitting is good for your health? Sources say that it is an excellent stress reliever and could possibly have the same effect as meditation.  I’ve definitely found myself getting lost in the motions of moving the needles.

GPS Clock Upgrade

Ice Tube ClockToday I upgraded my Adafruit Ice Tube clock with a GPS module. The clock is beautiful but I found myself frustrated by having to continually update the time as the clock would slowly get out of sync. Fortunately, there is a handy guide on how to add a GPS module to the clock so that it can readjust the time based on GPS. It only took a few minutes to do the entire upgrade.

Clock and GPSFor the GPS module, you can use any 5 volt GPS module with 4800 baud TTL NMEA output. I chose the Parallax PMB-648 as it is relatively low cost and fits the specification perfectly. Once you have the GPS, it’s merely a question of disassembling the clock and soldering the GPS to the clock.

Solder TimeThe next step is to reprogram the clock with the new firmware. You will need a 6 pin AVR programmer for this. I used Adafruit’s USBtinyISP AVR programmer. Make sure you have AVRDUDE set up correctly. With the correct environment, it is simply an issue of pushing the new firmware that supports GPS timekeeping.

IMG_9030And that’s it. Just reassemble the clock and case and you are done. It was really cool to turn the clock back on and have it automatically adjust to the correct time. Now I don’t have to keep resetting the clock 🙂