Just over one year ago, I joined the OpenStreetMap Foundation (OSMF) with the goal of enhancing the reliability and security of the technology and infrastructure that underpins OpenStreetMap. Throughout the past year, I have worked closely with the Operations Working Group, a dedicated team of volunteers. Together, we have made significant progress in improving our processes and documentation, ultimately strengthening our collective effectiveness. I am immensely grateful for the support and collaboration within this group, and I am delighted to witness the remarkable strides we have taken in building a solid foundation for the future of OpenStreetMap.

I’ll go into a little detail below about what’s transpired. At a high level, I made it easier to manage deployment of the software running on our servers; hardened our network infrastructure through better redundancy, monitoring, access, and documentation; grew our use of cloud services for tile rendering, leveraging a generous AWS sponsorship; improved our security practices; refreshed our developer environments; and last but definitely not least, finalised migration of 16 years of content from our old forums to our new forums.

If you want to hear more from me over the course of the work last year, check out my talk at State of the Map 2022 and my interview on the GeoMob podcast. And I’d love to hear from you, email me at osmfuture@firefishy.com.

2022-2023 Site Reliability Details

Managing software on our servers

Containerised small infrastructure components (GitHub Actions for building)

I have containerised many of our small sites which were previously built using bespoke methods in our chef codebase as part of the “Configuration as code” setup. Moved the build steps to Github Actions. Setup a base for any future container (“docker”) based projects going forward. These are our first container / docker based projects hosted on OSMF infrastructure.

Our chef based code is now simpler, more secure and deploys faster.

Improved chef testing (ops onboarding documentation)

We use chef.io for infrastructure (configuration) management of all our servers and the software used on them. Over the last year the chef test kitchen tests have been extended and now also work on modern Apple Silicon machines. The tests now reliably run as part of our CI / PR processes. The tests add quality control and assurance to the changes we make. Adding ARM support was easier to add because we could use test kitchen before moving onto ARM server hardware.

Having reliable tests should help onboard new chef contributors.

Hardened our network infrastructure

Network Upgrades @ AMS (New Switches, Dual Redundant Links, Dublin soon)

Our network setup in Amsterdam was not as redundant as it should have been. The Cisco Small Business equipment we used we had out-grown. We had unexpected power outages due to hardware issues. The equipment was also limiting future upgrades. The ops group decided to replace the hardware with Juniper equipment which we had standardised on at the Dublin data centre. I replaced the equipment with minimal downtime in a live environment (<15mins).

Both Dublin and Amsterdam data centres now use a standardised and more security configuration. Each server now has fully bonded links for improved redundancy and performance. The switches have improved power and network redundancy. We are awaiting the install of the fully resilient uplinks (order submitted) in the next month.

Out of Band access to both data centres (4G based)

I built and installed an out-of-band access devices at each site. The devices are hard wired to networking and power management equipment using serial consoles. The out-of-band devices have resilient 4G link to a private 4G network (1NCE). The out-of-band access devices are custom built Raspberry PIs with redundant power supplies and 4x serial connectors.

Documentation of Infrastructure to easy maintenance (Racks / Power)

I documented each rack unit, power port (Power Distribution Unit), network connection and cable at the data centres. This makes it easier to manage the servers, reduces errors and allows us to properly instruct remote hands (external support provider) to makes any chances on our behalf.

Oxidized (Visibility of Network Equipment)

Our network and power distribution configuration is now stored in git and changes are reported. This improves visibility of any changes, which in turn improves security.

Config is continiously monitored and any config drift between our sites is now much easier to resolve.

Terraform Infrastructure as Code (improve management / repeatability)

Terraform is an infrastructure-as-code tool, we now use it for managing our remote monitoring service (statuscake) and I am in the process of implementing it to manage our AWS and Fastly infrastructure.

Previous these components were all managed manually using the respective web UIs. Infrastructure-as-code allows the Ops team to collaboratively work on changes, enhances visibility and the repeatability / rollback of any changes.

We manage all domains DNS using dnscontrol code. Incremental improvements have been made over the last year, including add CI tests to improve outside collaboration.

Grew our use of cloud services

AWS in use for rendering infrastructure. Optimised AWS costs. Improved security. Improved Backup. More in pipeline

Ops team have slowly been increasing our usage of AWS over a few years. I have built out multiple usage specific AWS accounts using an AWS organisation model to improve billing and security as per AWS best practise guidelines. We generously received AWS sponsorship for expanding our rendering infrastructure. We built the experimental new rendering infrastructure using ARM architecture using AWS Graviton2 EC2.
We haven’t previously used ARM based servers. As part of improvements to our chef (configuration as code) we had added local testing support for Apple Silicon (ARM), only small additions were required to add the required compatibility for ARM servers to chef.

We were impressed by the performance of AWS Graviton2 EC2 instances for running the OSM tile rendering stack. We also tested on-demand scaling and instance snapshotting for potential further rending stack improvements on AWS.
We have increased our usage of AWS for data backup.

Improved our security

Over the last year a number of general security improvements have been made. For example: Server access is now via ssh key (password access now disabled). We’ve also moved from a bespoke gpg based password manager for the ops team to using gopass (feature rich version of https://www.passwordstore.org/ ), gopass improves key management and sharing the password store.

Additionally we have also enhanced the lockdown of our wordpress instances by reducing installed components, disabling inline updates and disabling XMLRPC access. We are also working to reduce the users with access and removing unused access permissions.

Documented key areas of vulnerability requiring improvement (Redundancy, Security, etc)

Documentation on technical vulnerability: I am producing a report on key areas of vulnerability requiring improvement (Redundancy, Security, etc). The document can be used to focus investment in future to further reduce our expose to risks.

Refreshed our developer environments

New Dev Server

We migrated all dev users to a new dev server in the last year. The old server was end of life (~10 years old) and was reaching capacity limits (CPU and storage). The new server was delivered directly to the Amsterdam data centre, physically installed by remote hands and I communicated the migration, and then migrated all users and projects across.

Retired Subversion

I retired our old svn.openstreetmap.org code repository in the last year. The code repository was used since the inception of the project, containing a rich history of code development in the project. I converted svn code repository to git using a custom reposurgeon config, attention was made to maintain the full contribution history and correctly link previous contributors (350+) to respective github and related accounts. The old svn links were maintained and now link to the archive on github https://github.com/openstreetmap/svn-archive

Forum Migration

The old forum migration, we migrated 1 million posts and 16 years of posts to discourse. All posts were converted from fluxbb markdown to discourse’s flavour of markdown. All accounts were merged and auth converted to OpenStreetMap.org “single sign-on” based auth.

All the old forum links redirect (link to the imported) to correct content. Users, Categories (Countries etc), Thread Topics, and individual posts.

Statement of Purpose

The OpenStreetMap Foundation (OSMF) Operations Working Group (OWG) is looking for proposals to provision space in a data centre to continue to run the OpenStreetMap (OSM) project’s infrastructure.

Background Information

OSMF is a nonprofit organisation dedicated to supporting, but not controlling, the OpenStreetMap project. OSMF created the OWG in order to support OSM’s technical infrastructure, including the main website, API, data distribution, community sites, and manage them for the benefit of the project.

The map data created by OpenStreetMap and distributed through ​OSMF​ is the best free global map available. It powers services all over the world, including for companies such as Apple, Foursquare, Craigslist and Mapbox. Ple​ase see https://www.openstreetmap.org/about​ for more background information.

Scope of Work

The data centre provider must meet the requirements set out below.

Requirements

Primary

●  The data centre must be in the EU.
●  One rack (at least 40U) of space, at industry standard rack dimensions.
●  Power capacity at least 3kW w/ dual redundant supplies.
●  Cooling to keep the servers suitably cool, e.g: under 35 degrees celsius.
●  Secure cages, so that only access authorised by the data centre or OSMF is possible.
●  On-site “remote hands” to be able to receive and replace HDDs and press power buttons during weekday business hours and at least some service weekends and holidays.
●  Network connection capable of 1Gbit/s peak traffic and 500Mbit/s sustained.

Secondary

●  Control over the configuration of any upstream firewalls for the purposes of ensuring necessary ports are open.
●  Good peering connection to major European backbone network. Ideally within 20ms of our existing sites on JANET.

Additional questions

Please provide detailed information on:
●  The procedure for shipping parts to the data centre, and
●  The procedure for raising a ticket for “remote hands” work, and
●  Whether “remote hands” would be available outside of business hours, and
●  The site’s uptime and network reachability over the past year, and
●  The procedure for an OSMF representative to visit and access the data centre.

Term of Agreement

The agreement would start on or before 1st April 2018 and run for a minimum of 3 years (at
OSMF’s option), preferably renewable annually or on a longer basis after that. Any renewal or
cancellation on either side would need a minimum notice period of 3 months.

Terms and Conditions

If you have Terms and conditions or Acceptable Use Policies then you should submit them in editable form for legal review, where possible. T&C/AUP changes should be expected to ensure we meet the privacy and security commitments required for our users.

Schedule, Evaluation and Award Process

This RFP is expected to be open until 28th February 2018. Only applications received prior to this date can be considered for this RFP. All proposals will be received in confidence and will be kept private.

After the date above, all proposals will be evaluated by the OWG against the requirements set out above, after which OWG may contact candidate sites with follow-up questions or to arrange site visits. The final agreement will require legal review and approval by the OSMF board.

OSMF particularly welcomes responses from anyone willing to support the work of the foundation at minimal cost.

The successful candidate will be publicly thanked on the main OSM project website as well as OWG websites in accordance with OWG’s Hosting Provider Credit Policy.

Points of Contact

Many thanks for your interest. If you have any questions or proposals, please send them to
operations@osmfoundation.org​.