\"Where Am I\", \"Take me to <<the park>>\", \"What's around me\"<\/code>. spatial audio<\/a> is used to provide more context for spatial direction of the information being called out to lighten their cognitive load and deliver an immersive user experience.<\/p>\n ![\"Image](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2016\/08\/citiesUnlockedApp.png\")
<\/p>\n
Features \/ Functionalities<\/h3>\n\n- Take Me To – turn-by-turn navigation routing instructions.<\/li>\n
- Location Context – automatic announcement of nearby points of interest(i.e. landmarks, sidewalks, crossings, etc) and location surroundings(i.e. roads, intersections, etc).<\/li>\n
- More Information – providing more contextual information about the closest points of interest.<\/li>\n<\/ul>\n
Technical Requirements \/ Challenges<\/h2>\n
Cities Unlocked announces location information throughout positional or head turn movements. The responsiveness and availability of our backend service had to be realtime. The planetary shapes we\u2019re working with are often comprised of complex geometry structures(i.e. parks and buildings). The expectation was our services can seamlessly onboard new cities to the platform, while maintaining acceptable performance for data volume growth.<\/p>\n
Our mapping data provider would also have to have a strong story around accessibility(i.e. crossings, curbs, sidewalks, stairs, etc). The data model also had to be flexible so we can easily add new places, features and tags.<\/p>\n
Elasticsearch Overview<\/h2>\n
Elasticsearch is an open source search engine based on Lucene<\/a>, and engineered to take data from any source to search and analyze it in realtime. Elasticsearch is considered an industry leader for developers seeking a distributed and highly scalable search solution.<\/p>\nElasticsearch is a document database ideal for high search volume scenarios such as full text-relevancy scoring or geo-spatial search. Data documents are serialized in JSON format and held in indexes internally managed by Lucene.<\/p>\n
Elasticsearch follows a clustered architecture where indices are evenly divided into shards, and horizontally scaled and distributed across a network of data nodes. Each data node on the cluster manages the delegated operations for its hosted shards. Elasticsearch offers a RESTful JSON based service API over HTTP allowing you to manage data indices and cluster nodes.<\/p>\n
Overview of the Solution<\/h2>\nArchitecture<\/h3>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2016\/08\/guidedogs-architecture.png\")
<\/p>\n
All our mapping data is sourced from OpenStreetmap<\/a> which allowed us to tap into a growing community of mapping enthusiasts contributing location information on a daily basis. OSM is a free, open and editable map of the world, which is often referred to as the Wikipedia for Maps. Our backend navigation services query OSM data using Elasticsearch running on Azure. Azure offers a managed search service<\/a> based on Lucene, but in this case study we\u2019ll discuss deploying our own Elasticsearch cluster to gain access to geo-spatial features not available in the Azure platform offering.<\/p>\nVector Tiles<\/h4>\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2016\/08\/vector-tile-example.png\")
<\/p>\n
\nImage Credit: Mapzen\r\n<\/code><\/pre>\n<\/div>\nWe implemented a vector tile based solution that collects mapping data as square shaped GeoTiles<\/a> into Elasticsearch. Each tile is a square area of land with a defined size and location, identified by a row, height and zoom level(X\/Y\/Z). The highest zoom level comprises of 4 tiles. Each zoom level divides the parent tile into four separate tiles. We used a zoom level of 16(so an average bounding box of ~460 meters x ~460 meters)<\/p>\nWe crowdsourced nearby geotile data based on the users position and a 3 kilometer radius distance. We\u2019re using Mapzens GeoJSON<\/a>-based Vector Tile Service<\/a>. Streets, buildings and parks are stored as GeoJSON LineString, while addresses and points of interest as a single GeoJSON point.<\/p>\nThe summary description for nearby points of interest is sourced from Wikipedia. We have created an Azure WebJob that keeps EleasticSearch up-to-date with any recent changes made in OpenStreetMap.<\/p>\n
Why Elasticsearch<\/h2>\n\n- We preferred an architecture that can scale horizontally and adapt to the expected data growth rate, support a multitenancy architecture, data-schema agnostic, support search\/filter\/analyze operations off complex geo-shape indices under 50 milliseconds.<\/li>\n
- The search engine needed to be distributed, fault-tolerant and load-balanced.<\/li>\n
- Full-Text Search<\/a> – With no visual interface available to validate the results of arbitrary user navigation request(s), finding an accurate full-text search engine was crucial. Lucene is a natural language search engine by design, supporting natural language relevancy ranking and fuzzy queries.<\/li>\n
- Our search ranking requirement was one of the reasons why we decided to not integrate with a more traditional geo-spatial offering like Postgres \/PostGIS.<\/li>\n
- Combining geoshape search with full-text search is powerful.<\/li>\n<\/ul>\n
Deploy Elasticsearch on Azure<\/h2>\n
Developers can now utilize preconfigured templates built by the Elasticsearch team to simplify the process of deploying a cluster to Azure.<\/p>\n
Prerequisites<\/h3>\n\n- Understand the fundamentals<\/a> of working with Elasticsearch indexes and queries.<\/li>\n
- An active azure subscription.<\/li>\n<\/ul>\n
Elasticsearch Cluster Setup<\/h3>\n
You can checkout this Gist<\/a> which will help you get started with setting up Elasticsearch on Azure.<\/p>\nVerify your Cluster Nodes are Online<\/h3>\n
Elasticsearch is packaged with Marvel: a visualization tool for monitoring your cluster status and health. At this point, your cluster should be showing a green status state in Marvel(http:\/\/my-kibana-url:5601\/app\/marvel).<\/p>\n
Loading planetary files into Elasticsearch<\/h3>\n
This section covers loading an OpenStreetmap planet file into the Elasticsearch cluster you created. I wrote a utility to automate the file parsing and data import here<\/a>. This utility works with binary planetary files<\/a>(.pbf) as a data source. You can follow the detailed instructions in the readme<\/a> to get started.<\/p>\nGeofabrik<\/a> provides pbf files at the continent, country and state \/ regional level. These files are updated on a daily basis.<\/p>\nMapzen recently released metro extracts<\/a>, which generates a custom planet file at the city or locality level. Custom bounding boxes are also supported. planet files from metro can take up to 20 minutes to generate. Metro extracts are updated on a weekly basis.<\/p>\nYou can start small with Rhode Island<\/a> for your first import.<\/p>\nYou can confirm that the OSM data was imported into your places index by accessing the document count API http:\/\/your-elastic-cluster:9200\/places\/place\/_count<\/p>\n
The utility uses a Go based pbf parser called pb2json<\/a>.<\/p>\n let config = {\r\n tags: featureTags.FEATURE_TAGS,\r\n leveldb: '.\/tmp',\r\n batch: 5000,\r\n file: process.env.PBF_DIRECTORY + '\/' + filename\r\n };\r\n\r\n pbf2json.createReadStream(config).pipe(denormalizer.stream(elasticService.elasticConnection()));\r\n<\/code><\/pre>\nI\u2019m using the elasticsearch.js<\/a> client to connect to my cluster.<\/p>\nfunction elasticClient(){\r\n if(VerifyEnvironmentVarsExist([\"ES_HOST\", \"ES_PORT\", \"ES_AUTH_USER\", \"PBF_DIRECTORY\", \"ES_AUTH_PWD\"])){\r\n return new elasticsearch.Client({\r\n host: [\r\n {\r\n host: process.env.ES_HOST,\r\n auth: process.env.ES_AUTH_USER + \":\" + process.env.ES_AUTH_PWD\r\n }\r\n ],\r\n log: {\r\n type: 'file',\r\n level: 'info',\r\n path: '.\/logs\/elasticsearch.log'\r\n },\r\n requestTimeout: 20000\r\n });\r\n }\r\n}\r\n<\/code><\/pre>\nThe bulk API is useful for writing a list of documents across multiple indexes. Elasticsearch provides the detailed response status code for all requested documents in the batch, as well as the processing time.<\/p>\n
let body = [];\r\n\r\n body = body.concat(transformToElasticBulk(tilesToIndex, tilesIndexHeader));\r\n body = body.concat(transformToElasticBulk(placesToIndex, placeIndexHeader));\r\n console.log('Requesting ' + body.length + ' for elastic indexing.');\r\n\r\n elasticClient().bulk({\r\n body: body\r\n },\r\n (err, response) => {\r\n if(err){\r\n<\/code><\/pre>\nRealtime On-Demand OpenStreetmap Elasticsearch<\/h3>\n