Making the most detailed tweet map ever

By Eric Fischer on December 03 2014  www.mapbox.com (TOTH: www.sensorsandsystems.com)

I’ve been tracking geotagged tweets from Twitter’s public API for the last three and a half years. There are about 10 million public geotagged tweets every day, which is about 120 per second, up from about 3 million a day when I first started watching. The accumulated history adds up to nearly three terabytes of compressed JSON and is growing by four gigabytes a day. And here is what those 6,341,973,478 tweets look like on a map, at any scale you want.

© Mapbox © OpenStreetMap Improve this map. Data from the Twitter Streaming API
© Mapbox © OpenStreetMap Improve this map. Data from the Twitter Streaming API

I’ve open sourced the tools I used to manipulate the data and did all the design work in Mapbox Studio. Here’s how you can make one like it yourself.

You can follow Twitter’s stream of geotagged public tweets using the “statuses/filter” API to request tweets from a particular bounding box or the whole world. Before you can connect, you have toregister a Twitter API key and authenticate using it. I couldn’t find a simple library last year to generate the OAuth header for Twitter authentication, so I wrote this one. Once you have authenticated and connected to the filter API, you receive a steady stream of tweets in JSON format. They include a lot more metadata than you necessarily need to make a dot map, so I’ve been using this program to parse the JSON and pull out just each tweet’s username, date, time, location, client, and text.  Click to continue reading.

Team of young coders build location-based game, using OS OpenData

By , 1 December, 2014 8:00 am

Back in August, we shared news about our support at the Young Rewired State (YRS) Festival of Code and how we mentored young people at one of the centres at Totton College, which is just a stone’s throw from our Head Office.

Map Game (mapgameyrs.kissr.com) designed using OS OpenData
Map Game (mapgameyrs.kissr.com) designed using OS OpenData

Following that blog, we were contacted by another team of young coders – Jakob Metson, Tim Yeo, Gordon Lee and Solomon Foy – who told us how they’d also used OS OpenData during the week to build a game. The young coders told us they ‘were amazed by the sheer volume of maps supplied’ through our OS OpenData portal – so we thought we’d invite them to write a guest post, allowing them to share their awesome achievement with our fellow blog followers!

The team, who were aged between 8 and 14 years old and who were based in a centre located in London, discovered our OS OpenData portal via the YRS resources page. Here’s what they had to say about their project:

On Monday 28th July our group of four, took part in the YRS Festival of Code. The Festival takes place once a year to give young coders (18 or under) a platform to get together for a week, to create a project using various open data sets.

We worked on, and coded a project called ‘Map Game’ as we wanted to create a geographic game, because we felt that there should be a fun and simple way of learning more about England’s geography. We used the OS OpenData and were amazed about the volume and different types of maps made available. We decided to create a simple web game, where the user has to guess whether they are in the North, the Middle or the South of England.

As we were fairly novice in coding, one of the main challenges was managing to code the website, especially sorting through hundreds of maps and moving them to the website. We originally downloaded a large variety of different maps and decided to use OS VectorMap District raster images as these were clear, easy to read and provided the names of well-known places, rivers and roads (which are clues as to where in England you are).

The images came in very high quality which meant high detail, but they took too long to load in our website. However, it was very simple to convert them from TIFF format (high quality) to JPEG format (medium quality). With a little help we soon were able to create a fairly decent website and what we think is a good educational game – we certainly know more about England’s geography as a result of building it!

You can have a go, by checking it out here.

Top Ten of 2014 (from Sensors & Systems)

16 Dec 2014  Matt Ball in www.sensorsandsystems.com

In our regular end-of-the-year reflection, Sensors & Systems looks back over the past 12 months to come up with the top developments of 2014 that will have strong implications for geospatial industry growth and diversity in the coming years. Making the list are technology disruptions, acquisitions, modeling frameworks, mapping efforts and global change.


1. DigitalGlobe Launches WorldView-3 — The successful launch of the world’s most sophisticated commercial imaging satellite promises to bring a new range of potential uses, thanks to the high resolution and the infrared bands for greater spectral coverage. WorldView-3 will provide higher 0.31m resolution as well as greater global coverage. The new capabilities include the ability to see through smoke, to map crop types and health, to monitor oil and gas, and to explore for iminerals thanks to the ability to identify chemical composition.

2. Relaxed Imagery Restrictions — The relaxing of resolution restrictions by the U.S. Department of Commerce means that end users can acquire satellite imagery at up to 0.25m panchromatic and 1.0m multispectral starting in the first quarter of 2015. This move helps the commercial satellite imagery companies as well as the end user, given the superior coverage and frequency of satellite platforms for massive amounts of data collection as well as ongoing monitoring.

3. Drone-sized Laser Scanners — In the reality capture realm, both Velodyne and Riegl released LiDAR scanners small and light enough to be mounted on drone platforms. The portability and flexibility of this new capture mode should open up new opportunities for more rapid data collection and more complete 3D models for such hard-to-capture areas as mines or complex processing plants.

4. Google Purchases Skybox Imaging — The rumors were flying for months prior to the announced acquisition of the smallsat company for $500 million. The shared Silicon Valley location was certainly a factor, although likely more important is the focus on data and analytic opportunities with the planned high-resolution constellation with an eye on daily collection. The move impacted the stock of other commercial satellite companies thanks to its potential to satisfy most of Google’s insatiable demand for imagery. It’s interesting that the company was snapped up so quickly with only a few of the planned satellites in orbit.

5. Incubation Aimed at Proliferation — Esri has been expanding their support for startup companies through grants and licenses for their software. One of the more impressive aspects of the show floor at the Esri International User Conference this past year was the Startup Zone with passionate small companies that aim to expand the use and awareness of geospatial technologies and provide solutions upon the platform that Esri provides. The company wins when these companies start getting paid for their service and in turn pay Esri, but they also win by spreading the word through the evangelism of these passionate and thankful users.

6. Hexagon Geospatial Formed and Power Portfolio Released — The packaging of Hexagon Geospatial happened early in the year, splitting the geospatial tools from Intergraph and allowing for a fresh packaging of products. The new Hexagon packages are divided into Producer, Provider and Platform, giving users a bundling of capable software to support imagery processing and analysis, geospatial solutions, Web mapping, data portals and advanced modeling. The move to create the new company and the new packages provides greater clarity of offerings and reinforces the integration of the toolsets that were acquired and have been integrated over time.

7. Airbus in Action — One of the more interesting and ongoiing company stories is that of Airbus Defence and Space, which has undergone rebranding this past year with the change from Astrium. Just recently, the company sold the operations of their Spot 7 satellite to Azerbaijan as well as selling their Tokyo Spot Image company to PASCO. This divestment and partnership approach places more importance on regional partners and their ability to grow the business locally. It will be interesting to follow to see if there will be any further activity to place more of a focus on sensors and satellites and away from services.

8. FAA Opens UAS Use for Surveying and Monitoring The recent relaxation of restrictions for four companies to use unmanned aircraft systems (UAS) for surveying and monitoring follows a decision earlier in the year to grant exemptions to seven film companies working in both television and films. This exemption includes the drone maker Trimble and the surveying and mapping company Woolpert (who happen to be located in Dayton, Ohio where manned flight was born). This step may soon see limited exemptions for additional applications such as agricultural crop mapping and monitoring, utility and pipeline corridor inspections, and package delivery.

9. Coming Era of Real-time Geospatial Data — A number of very interesting data management tools from the likes of IBM and SAP are focused squarely on the rapid delivery of geospatial insight thanks to high-speed computing and advanced algorithms. A startup named SpaceCurve is also in this space. As geospatial data volumes continue to increase, these tools provide a distinct edge to not only make sense of all these feeds but to provide actionable information.

10. Increasing Cloud Commitments — Software companies across the full geospatial spectrum are making increased commitments to the delivery of hosted geospatial solutions as well as extending some of the data intensive heavy lifting of geospatial analysis to the infinite computing of the cloud. Esri’s repackaging of ArcGIS as Pro for desktop and Portal for Server provides increasing capabilities for Web GIS with data and infrastructure aimed at greater portability through the delivery of apps. Others are making similar moves to help organizations leverage their geospatial investments with greater access and integration into multiple simultaneous workflows.

– See more at: https://www.sensorsandsystems.com/dialog/perspectives/35376-top-ten-of-2014.html#sthash.ROZKH2qG.dpuf