Author Archives: Grant Slater

Powering OpenStreetMap’s Future: A year of improvements from OpenStreetMap Foundation’s Site Reliability Engineer

Just over one year ago, I joined the OpenStreetMap Foundation (OSMF) with the goal of enhancing the reliability and security of the technology and infrastructure that underpins OpenStreetMap. Throughout the past year, I have worked closely with the Operations Working Group, a dedicated team of volunteers. Together, we have made significant progress in improving our processes and documentation, ultimately strengthening our collective effectiveness. I am immensely grateful for the support and collaboration within this group, and I am delighted to witness the remarkable strides we have taken in building a solid foundation for the future of OpenStreetMap.

I’ll go into a little detail below about what’s transpired. At a high level, I made it easier to manage deployment of the software running on our servers; hardened our network infrastructure through better redundancy, monitoring, access, and documentation; grew our use of cloud services for tile rendering, leveraging a generous AWS sponsorship; improved our security practices; refreshed our developer environments; and last but definitely not least, finalised migration of 16 years of content from our old forums to our new forums.

If you want to hear more from me over the course of the work last year, check out my talk at State of the Map 2022 and my interview on the GeoMob podcast. And I’d love to hear from you, email me at osmfuture@firefishy.com.

2022-2023 Site Reliability Details

Managing software on our servers

Containerised small infrastructure components (GitHub Actions for building)

I have containerised many of our small sites which were previously built using bespoke methods in our chef codebase as part of the “Configuration as code” setup. Moved the build steps to Github Actions. Setup a base for any future container (“docker”) based projects going forward. These are our first container / docker based projects hosted on OSMF infrastructure.

Our chef based code is now simpler, more secure and deploys faster.

Improved chef testing (ops onboarding documentation)

We use chef.io for infrastructure (configuration) management of all our servers and the software used on them. Over the last year the chef test kitchen tests have been extended and now also work on modern Apple Silicon machines. The tests now reliably run as part of our CI / PR processes. The tests add quality control and assurance to the changes we make. Adding ARM support was easier to add because we could use test kitchen before moving onto ARM server hardware.

Having reliable tests should help onboard new chef contributors.

Hardened our network infrastructure

Network Upgrades @ AMS (New Switches, Dual Redundant Links, Dublin soon)

Our network setup in Amsterdam was not as redundant as it should have been. The Cisco Small Business equipment we used we had out-grown. We had unexpected power outages due to hardware issues. The equipment was also limiting future upgrades. The ops group decided to replace the hardware with Juniper equipment which we had standardised on at the Dublin data centre. I replaced the equipment with minimal downtime in a live environment (<15mins).

Both Dublin and Amsterdam data centres now use a standardised and more security configuration. Each server now has fully bonded links for improved redundancy and performance. The switches have improved power and network redundancy. We are awaiting the install of the fully resilient uplinks (order submitted) in the next month.

Out of Band access to both data centres (4G based)

I built and installed an out-of-band access devices at each site. The devices are hard wired to networking and power management equipment using serial consoles. The out-of-band devices have resilient 4G link to a private 4G network (1NCE). The out-of-band access devices are custom built Raspberry PIs with redundant power supplies and 4x serial connectors.

Documentation of Infrastructure to easy maintenance (Racks / Power)

I documented each rack unit, power port (Power Distribution Unit), network connection and cable at the data centres. This makes it easier to manage the servers, reduces errors and allows us to properly instruct remote hands (external support provider) to makes any chances on our behalf.

Oxidized (Visibility of Network Equipment)

Our network and power distribution configuration is now stored in git and changes are reported. This improves visibility of any changes, which in turn improves security.

Config is continiously monitored and any config drift between our sites is now much easier to resolve.

Terraform Infrastructure as Code (improve management / repeatability)

Terraform is an infrastructure-as-code tool, we now use it for managing our remote monitoring service (statuscake) and I am in the process of implementing it to manage our AWS and Fastly infrastructure.

Previous these components were all managed manually using the respective web UIs. Infrastructure-as-code allows the Ops team to collaboratively work on changes, enhances visibility and the repeatability / rollback of any changes.

We manage all domains DNS using dnscontrol code. Incremental improvements have been made over the last year, including add CI tests to improve outside collaboration.

Grew our use of cloud services

AWS in use for rendering infrastructure. Optimised AWS costs. Improved security. Improved Backup. More in pipeline

Ops team have slowly been increasing our usage of AWS over a few years. I have built out multiple usage specific AWS accounts using an AWS organisation model to improve billing and security as per AWS best practise guidelines. We generously received AWS sponsorship for expanding our rendering infrastructure. We built the experimental new rendering infrastructure using ARM architecture using AWS Graviton2 EC2.
We haven’t previously used ARM based servers. As part of improvements to our chef (configuration as code) we had added local testing support for Apple Silicon (ARM), only small additions were required to add the required compatibility for ARM servers to chef.

We were impressed by the performance of AWS Graviton2 EC2 instances for running the OSM tile rendering stack. We also tested on-demand scaling and instance snapshotting for potential further rending stack improvements on AWS.
We have increased our usage of AWS for data backup.

Improved our security

Over the last year a number of general security improvements have been made. For example: Server access is now via ssh key (password access now disabled). We’ve also moved from a bespoke gpg based password manager for the ops team to using gopass (feature rich version of https://www.passwordstore.org/ ), gopass improves key management and sharing the password store.

Additionally we have also enhanced the lockdown of our wordpress instances by reducing installed components, disabling inline updates and disabling XMLRPC access. We are also working to reduce the users with access and removing unused access permissions.

Documented key areas of vulnerability requiring improvement (Redundancy, Security, etc)

Documentation on technical vulnerability: I am producing a report on key areas of vulnerability requiring improvement (Redundancy, Security, etc). The document can be used to focus investment in future to further reduce our expose to risks.

Refreshed our developer environments

New Dev Server

We migrated all dev users to a new dev server in the last year. The old server was end of life (~10 years old) and was reaching capacity limits (CPU and storage). The new server was delivered directly to the Amsterdam data centre, physically installed by remote hands and I communicated the migration, and then migrated all users and projects across.

Retired Subversion

I retired our old svn.openstreetmap.org code repository in the last year. The code repository was used since the inception of the project, containing a rich history of code development in the project. I converted svn code repository to git using a custom reposurgeon config, attention was made to maintain the full contribution history and correctly link previous contributors (350+) to respective github and related accounts. The old svn links were maintained and now link to the archive on github https://github.com/openstreetmap/svn-archive

Forum Migration

The old forum migration, we migrated 1 million posts and 16 years of posts to discourse. All posts were converted from fluxbb markdown to discourse’s flavour of markdown. All accounts were merged and auth converted to OpenStreetMap.org “single sign-on” based auth.

All the old forum links redirect (link to the imported) to correct content. Users, Categories (Countries etc), Thread Topics, and individual posts.

Meet Grant Slater, the OpenStreetMap Foundation’s new Senior Site Reliability Engineer

Thanks to the support of corporate donors, the OpenStreetMap Foundation has been able to hire its first employee, who is starting on 1 May 2022. Grant Slater and Guillaume Rischard, the Foundation’s chairman, sat down for a virtual chat.

Hi! Tell us about you?

Hi! I’m Grant Slater, and I’m the new Senior Site Reliability Engineer (SRE) working for the OpenStreetMap Foundation. I’m originally from South Africa, and now live in London (UK) with my wife Ingrida and our son Richard.

What do you do in OSM? Where do you like to map?

I’ve been mapping since 2006, mostly in the Southern Africa and in the United Kingdom. I have a strong interest in mapping the rail network of South Africa; holidays “back home” often involve booking railway trips across the country, with a GPS in hand.

My latest toy is an RTK GPS base station and rover. I’ll soon be mapping my neighbourhood with centimetre-level accuracy.

For the last 15 years, I’ve been part of the volunteer OpenStreetMap Operations Team who install and maintain the servers and infrastructure which runs the OpenStreetMap.org website and many other related services.

What are your plans for the new SRE job?

My main objective will be helping improve the reliability and security of the project’s technology and infrastructure.

One of my goals will be to improve the project’s long-term stability as we grow. OWG can’t work without volunteers, and I will be improving the Operation Team’s bus factor by also improving our processes, documentation, and by smoothing the path to onboarding new team members.

I will be helping to drive forward modernising the project’s infrastructure by reducing complexity, paying-down technical debt, and reducing our need to maintain undifferentiated heavy lifting, by tactically using Cloud and SaaS services, where suitable.

Is there anything else you’d like to say?

With time, I would like to see OpenStreetMap introduce new tools and services to improve our mappers’ access to opted-in passively collected data to improve the mappers’ ability to map and detect change.

Gamification! OpenStreetMap should always remain a fun and gratifying experience for all. We’re building an invaluable and unique dataset with far-reaching consequences for which we should be incredibly proud. Happy Mapping!

I would like to hear your feedback and suggestions, please email me osmfuture@firefishy.com

Grant gave a talk at State of the Map US (2013) – OSM Core Architecture and DevOps and is hoping to give an updated talk at State of the Map 2022 in Florence, Italy 19–21 August 2022.

https://www.openstreetmap.org/user/Firefishy
https://twitter.com/firefishy1
https://github.com/firefishy

Upcoming Maintenance

On Saturday 5th of July 2014 between 09:00 and 19:00 (GMT / UTC) we are moving our servers hosted by University College London to another data center.

The following services will be affected:

  • Search (nominatim.openstreetmap.org) will be unavailable. *
  • Slower map updates / Reduced tile rendering capacity. (Yevaud outage)
  • OSM Foundation websites and staging.blog.openstreetmap.org will be unavailable.
  • Taginfo (taginfo.openstreetmap.org) will be unavailable.
  • Development Server (errol) will be unavailable.
  • Some imagery services will be unavailable. (GPX Render, OS
    Streetview, OOC, AGRI, CD:NGI aerial)

Other OpenStreetMap provided services should not be affected – all of the following are expected to function normally:

  • www.openstreetmap.org web site will allow edits as per normal (iD or
    Potlatch).
  • API will allow map editing (using iD, JOSM, Merkaartor etc.)
  • Forum
  • trac (bug-tracker)
  • help.openstreetmap.org
  • tile serving (“View The Map” & “Export”)
  • Wiki
  • mailing lists
  • subversion and git (source code repositories)
  • donate.openstreetmap.org

Technical: We are moving all the servers listed here to a new UCL data center. The current building is being closed soon for refurbishment. The new data center has better server racks, power feeds, cooling and faster networking.

* Searches through the website will still work – we will redirect
them to another nominatim instance temporarily.

Sincerely
Grant Slater
On behalf of the OpenStreetMap sysadmin team.

OpenStreetMap Enhances User Privacy

Today, OpenStreetMap has enabled encryption (SSL) to all of the openstreetmap.org website, thereby enhancing the privacy of its users.

You can now browse the site at https://openstreetmap.org (note the ‘https’). This means your browsing activity is secure from snooping.

OpenStreetMap stands with the Open Rights Group and the Electronic Frontier Foundation in asserting greater Internet freedom, including the right to individual privacy. With this action providing the highest quality Free/Open Data Geographic resource to everyone.

We are proud to roll this out on the same day as the “Day We Fight Back” campaign.

Other aspects of privacy around OpenStreetMap are discussed on this wiki page.

OpenStreetMap infrastructure donation – Bluehost

Thanks to generous donations and active local community members, the OpenStreetMap distributed tile delivery infrastructure continues to grow.

Two tile servers, nadder-01 and nadder-02, have been added to the OpenStreetMap tile cache network.  Based in Provo, Utah, USA, these servers provide tiles to the Americas.

Map tiles are delivered to users based on their GeoDNS location. The OpenStreetMap Foundation seeks additional distributed tile servers. If you would like to donate a tile server and hosting, please see the Tile CDN requirements page on the wiki.

tile serving geodns map

We would like to thank the BOSS team (Bluehost Open Source Solutions) and especially Jared Smith at Bluehost.com for this generous donation to OpenStreetMap infrastructure.

The OpenStreetMap Foundation is a not-for-profit organization, formed in the UK to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data and to providing geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project. You can support OpenStreetMap by donating to the OpenStreetMap Foundation.

New tile rendering and CartoCSS stylesheet

The default OpenStreetMap.org “standard” map was switched across to a new rendering server setup over the last weekend.

In addition to new hardware, the rendering server also uses the new “openstreetmap-carto” stylesheet. This is a complete re-write of the old XML stylesheet to use CartoCSS, making it easier for our cartographers to work with. The style is designed to look as similar as possible to the old XML stylesheet.

Andy Allan presented a great talk at State of the Map US conference describing the reasons for re-writing the stylesheet: Putting the Carto into OpenStreetMap Cartography

Andy Allan Stylesheets

Andy will present a follow-up at State of the Map next month.

The “openstreetmap-carto” stylesheet is maintained on github

“openstreetmap-carto” is a good base for creating custom styles, and should be much easier to work with. If you want to help improve the style, or add new features, please fork it and contribute pull requests!

Please support OSM’s server hardware fundraising drive