Category Archives: Operations

Hardware and system administration related posts. Anything related to the Operations Working Group

Server maintenance May 13-15

Some secondary services like the wiki may be affected by maintenance this weekend. Following this weekend, OpenStreetMap users may experience some slow uploads due to continued tuning after our recent database move.

On Monday May 9, our Operation Team smoothly moved the master OSM database server to Bytemark hosting, from Imperial College. Setting up multiple data center redundancy avoided further downtime due to planned power testing and maintenance at the Imperial data centre.

Over the weekend, we are switching some additional services to Bytemark, due to that power maintenance at Imperial. Primary OSM services will be operational, but the Wiki will be in read-only mode during this time. The slow uploads are a known issue, and are being addressed following the maintenance this weekend. More details on the talk mailing list.

Planned server outage – 9th May

The Operations Working Group are in the process of planning some hardware moves as a result of maintenance at our primary datacentre. The plans are subject to change, but these moves are expected to take place on Monday 9th May (a week from today), and this will mean a period of read only operation lasting for up to 24 hours (into Tuesday).

The read-only period will mean the OpenStreetMap website and map is still available, but no map editing will be possible during this time.

The ‘2016 May server maintenance’ wiki page has some more technical details, and will be updated with more precise timing information as the plans are finialised in the coming days.

We’re putting out this announcement to give some advanced notice, but we appreciate some people will be wanting to know more about the exact timings of the read-only period. Please have patience. The Operations Working Group are busy making plans, and will aim to firm things up in the coming days.

Target reached – Thank you!

Today we’ve reached our fund-raising target: £56,000 !

osm.org/go/ZUfMByg0Z

The Operations Working Group set out their detailed plans for the server upgrades they’d like to do: A big new database server, new tile servers and an imagery server. Thanks to your generosity, we now have the funds in place to make that a reality.

This has been our largest fund-raising target, but given the scale, reach and impact of OpenStreetMap, £56,000 is a small amount of money. For the thousands of data consumers, and millions of map end-users viewing OpenStreetMap, or the mega-corporations our maps are compared against. Don’t we deserve more? Lots of interesting discussions around this topic, but I will just say for now… Yes we do! And of course you’re very welcome to donate more: donate.openstreetmap.org

The OpenStreetMap Foundation remains open for donations. If you have ideas on how to get more recognition and big donations for OpenStreetMap, this is always welcome too!

For now though, we want to say a big Thank You again to Mapzen and Mapbox who made big donations to kick things off, to various other organisations who made donations to support us, and to all the individuals (more than 1200 of them!) who generously chose to put their personal money towards the OpenStreetMap cause.

OpenStreetMap tile CDN continues to grow

Since the last additions to our OpenStreetMap tile serving network in December, there has been a lot more server set-up going on.

The German tile cache server tabaluga is now retired and is no longer serving tiles. This may sound like bad news, but quite the opposite! Tabaluga has been replaced with a new server, katie, which has taken over its job.
The new tile cache server katie is still located in Falkenstein, Germany, and still hosted by Hetzner.

More good news: There are two tile cache servers in Germany now!
The second tile cache server, konqi, is located in Jena, Germany, hosted by EUserv.

The Russian tile cache server gorynych just had a memory and SSD upgrade, and with this it can deliver even more content.

There is another new server in Hungary. With this Hungary becomes one of 12 countries hosting OSM CDN servers.

Tile cache server sarkany is located in Budapest, Hungary, hosted by szerverem.hu.

With all of these, the CDN (Content Delivery Network) server count comes to 16 active servers.

Tabaluga was running, thanks to Freerk Ohling, at Hetzner since May 2013, and served its last tiles in January. Freerk approached us back in April 2013 to suggest we implement EDNS client subnet support (implemented in December 2014) and to offer us a sponsored tile cache server. Now he has also kindly sponsored new tile cache servers in Germany.

Tabaluga primarily served traffic to visitors from Germany. Approximately 56 million map tiles per day. (avg 652/sec, peaking at 1245/sec). Serving close to 1TB of data per day. It was the highest traffic OSM tile cache server.

OpenStreetMap tiles are free for everyone to use, but should be used with moderation. If you are a high traffic site you should look at switch2osm.org to find out how to use the data and keep the tiles available for everyone.

The OpenStreetMap Foundation seeks additional distributed tile servers. If your organisation would like to donate a tile server and hosting, please see the Tile CDN requirements page on the wiki. You can also support OpenStreetMap by donating to the OpenStreetMap Foundation.

The OpenStreetMap Foundation is a not-for-profit organisation, formed in the UK to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project.

Four New Tile Servers

Have you noticed faster tiles lately? Browsing the map on openstreetmap.org should now be even more responsive. Three new servers, started providing tiles over the last 2 weeks, joining a server which started earlier in the year.

Tile server saphira, located in London UK kindly hosted by Jump Networks.
Tile server viserion, located in Pula Croatia, kindly hosted by CARNet.
Tile server stormfly-02, Located in Corvallis USA, kindly hosted by OSUOSL.
Tile server longma, Located in Hsinchu Taiwan, kindly hosted by NCHC.

Map tiles are delivered to users based on their GeoDNS location. The OpenStreetMap tile content delivery network (CDN) now supports EDNS-client-subnet to improve locating the closest region tile cache.

OpenStreetMap tiles are free for everyone to use, but should be used with moderation. If you are a high traffic site you should look into switch2osm.org to find out how to use the data and keep the tiles available for everyone.

Thanks to generous donations and active local community members, the OpenStreetMap distributed tile delivery infrastructure continues to grow.

The OpenStreetMap Foundation seeks additional distributed tile servers. If you would like to donate a tile server and hosting, please see the Tile CDN requirements page on the wiki. You can also support OpenStreetMap by donating to the OpenStreetMap Foundation.

The OpenStreetMap Foundation is a not-for-profit organization, formed in the UK to support the OpenStreetMap Project. It is dedicated to encouraging the growth, development and distribution of free geospatial data and to providing geospatial data for anyone to use and share. The OpenStreetMap Foundation owns and maintains the infrastructure of the OpenStreetMap project.

OpenStreetMap Enhances User Privacy

Today, OpenStreetMap has enabled encryption (SSL) to all of the openstreetmap.org website, thereby enhancing the privacy of its users.

You can now browse the site at https://openstreetmap.org (note the ‘https’). This means your browsing activity is secure from snooping.

OpenStreetMap stands with the Open Rights Group and the Electronic Frontier Foundation in asserting greater Internet freedom, including the right to individual privacy. With this action providing the highest quality Free/Open Data Geographic resource to everyone.

We are proud to roll this out on the same day as the “Day We Fight Back” campaign.

Other aspects of privacy around OpenStreetMap are discussed on this wiki page.

OpenStreetMap infrastructure donation – Bluehost

Thanks to generous donations and active local community members, the OpenStreetMap distributed tile delivery infrastructure continues to grow.

Two tile servers, nadder-01 and nadder-02, have been added to the OpenStreetMap tile cache network. Based in Provo, Utah, USA, these servers provide tiles to the Americas.

Map tiles are delivered to users based on their GeoDNS location. The OpenStreetMap Foundation seeks additional distributed tile servers. If you would like to donate a tile server and hosting, please see the Tile CDN requirements page on the wiki.

We would like to thank the BOSS team (Bluehost Open Source Solutions) and especially Jared Smith at Bluehost.com for this generous donation to OpenStreetMap infrastructure.

New tile rendering and CartoCSS stylesheet

The default OpenStreetMap.org “standard” map was switched across to a new rendering server setup over the last weekend.

In addition to new hardware, the rendering server also uses the new “openstreetmap-carto” stylesheet. This is a complete re-write of the old XML stylesheet to use CartoCSS, making it easier for our cartographers to work with. The style is designed to look as similar as possible to the old XML stylesheet.

Andy Allan presented a great talk at State of the Map US conference describing the reasons for re-writing the stylesheet: Putting the Carto into OpenStreetMap Cartography

Andy will present a follow-up at State of the Map next month.

The “openstreetmap-carto” stylesheet is maintained on github

“openstreetmap-carto” is a good base for creating custom styles, and should be much easier to work with. If you want to help improve the style, or add new features, please fork it and contribute pull requests!

Please support OSM’s server hardware fundraising drive

Bulk GPX track data

Last year, at about this time, the OpenStreetMap Foundation made a big splash when it released the first Bulk GPS Point dataset. That may have been the largest publication of GPS point data ever. Many users were thrilled to have that raw data and to put it to good use. But they also wanted more. They wanted the track file details.

In fact, requests for bulk OpenStreetMap GPX track files go back to 2007 or earlier. Now, the combined talents of the OpenStreetMap community and the resources of the OpenStreetMap Foundation, make that dream a reality.

Announcing the GPX Planet

The scale of this dataset is fairly large.

848,000 GPX files
2.6 Trillion GPX points
260GB of GPS data (uncompressed)

Not everybody will be able to manage all of this data at once. It is likely, as with planet files and planet extracts, that somebody will offer GPX extracts.

What is this data good for?

Well, it’s interesting and there is a large quantity of it. Many OpenStreetMap contributors have been asking for it for years. Previous uses for this type of data have included

artistic organic trip data as maps
confirm turn restrictions
confirm one-way restrictions
confirm speed limits
and many more of your ideas in the comments

The future

There are hopes and plans to improve this service over time, including

quarterly updates
addition of vdop / hdop precision data, where available

You can help

You can help to improve this service by contributing your GPX track data. Upload your GPX files through your OpenStreetMap account. Collect new track files as you survey.

You can also help by using the data for something interesting and / or by publishing some code. OpenStreetMap user Zverik has started the ball rolling by creating gpx2pgsql to put the gpx data into your rendering tool chain.

Thanks

Thanks as always to the million-plus OpenStreetMap contributors over the eight years of the project so far. Especially, thanks to the thousands of contributors who collected and uploaded their GPX track files, and to Lars Franke, who wrote the first Java GPX dump code in 2010, and to Ian Dees who rewrote that into the current Python code, and to the many others who were involved in ways large and small.

So, where is it?

Find the GPX planet on the planet site and mirrors.

The image in this article is a visualization of some of the GPS point data in Europe. The image is licensed CC-By-SA and was created by Dave Stubbs.

Database maintenance

Updates included below the original post.

OpenStreetMap database unavailable on 31 March 2013

There is a current problem with the OpenStreetMap database and so the data API is currently unavailable due to hardware failures.

The current issues were first observed early on 31 March 2013 when the primary database server became non-responsive. The system administrators are aware of the problem and will resolve it as soon as possible. A complicating factor is the observation of a holiday at the hosting facility that prevents physical access to the server until Tuesday morning.

With the database unavailable, editing is not possible. Other database-related activities, such a new-user-signups, are also unavailable.

Other non-database-related services should operate normally. Tiles are being served with current data.

The system administrators will continue to take steps towards the return of normal operation. Given the aforementioned holiday at the hosting facility, normal operation may not return until some time Tuesday, 02 April 2013.