Category Archives: Operations

Hardware and system administration related posts. Anything related to the Operations Working Group

Licence redaction ready to begin

Hello all,

I’m pleased to announce that the licence change bot is ready to get underway.

Starting this week, we will be ‘redacting’ the contributions (less than 1%) from the live database that are not compatible with the new Contributor Terms and Open Database Licence (ODbL) – in other words, they will no longer be accessible. We are expecting to begin on Wednesday (11th July) assuming a couple of final setup details are completed by then.

The bot will run in the following order:

  1. Ireland
  2. UK
  3. Western Europe
  4. North America
  5. Australia
  6. rest of the world

Once it is complete, we will be ready to distribute data under the ODbL and we’ll advise of that with a separate announcement. The final pre-redaction dataset available under CC-BY-SA has now been generated at http://planet.openstreetmap.org/planet-120704.osm.bz2 . Where data has been redacted, any attempt to access it from the API or the site’s ‘browse’ pages will return a response to that effect.

Test runs have shown that the bot is functioning as we want it to, but we will of course be monitoring its progress. We are currently expecting it to take in the order of one month to complete; given the many variables I’m afraid we can’t give a more precise steer yet, but we’ll aim to keep everyone updated as it runs (via the announce@ and talk@ lists).

There will be no API outage and no other interruption to editing. When the bot is running in your area, please do save your edits frequently to minimise the likelihood of conflict.

(Separate messages are going to talk-ie@ and talk-gb@ as the first two areas to be affected. Please do forward and translate this for your local mailing lists.)

As you know we were expecting this to start just after 1st April and the complexity of the task incurred the delay. Thank you all very much for your patience in waiting for it to get underway. Thank you especially to those who have contributed to the code, whether by patches, suggestions or just helping to firm up the workings.

Posted to the mailing list by Richard for the OSMF board

License change still ongoing

Three weeks ago we mentioned we were still perfecting the ‘redaction bot’. The piece of code that goes through and redacts (removes/hides) any data that isn’t compatible with the new licence. Much to our dislike, it will take more time to get the bot working perfectly. Good news: the bot is now passing more tests than ever; bad news: still not all. Several people are working on this to make it work error-free.

Once these system tests are passing, live data testing will be conducted against a test server that is already configured and waiting. Subject to a successful test, a test of an isolated portion of the live database will be processed, most likely for the island of Ireland. If this goes successfully, the rest of the data will be processed.

On a positive note: in the last few weeks we’ve also managed to get agreement on several contributions to keep them in our database. We would like to thank all people who helped us make that happen.

We’ll give you another update next week.

Thank you all for your patience.

License change update: getting it right

With the new server successfully installed by our sysadmin team, we’re now onto the second part of our migration – the data ‘redaction’ work required to move to the Open Database License. We promised our first progress report next week, but lots of people have been asking, so here’s an update four days early.

The code changes to the OpenStreetMap API have been completed and successfully reviewed. openstreetmap.org is therefore ready to distribute the new data. (Thanks to Matt Amos for the code and Tom Hughes for the review work.)

The next part is the ‘redaction bot’. This is the piece of code that, for an area of OpenStreetMap data, goes through and redacts (removes/hides) any data that isn’t compatible with the new licence. This is the most crucial part of the whole process: we aim not to retain data whose creators haven’t given permission for it to be distributed under the Open Database License, and conversely, not to inadvertently delete anything from the vast majority which is compatible.

Since Wednesday we’ve been running tests against real-world data (thanks to Frederik Ramm for help with this). We’re not yet 100% happy with the results, so we are continuing to work on the code. As you would expect, we will not set the bot running until we are absolutely confident that it is producing accurate results. With the four-day Easter weekend just beginning, we currently expect that this will be next week. This puts us a few days behind schedule, but we owe it to our mappers to get this right.

If you’re a developer, you can help fix the currently failing tests: check out the code at https://github.com/zerebubuth/openstreetmap-license-change. If you’re a mapper, this gives you a few more days to get your area shipshape! And if you’re a data consumer, you can, of course, continue to use the data under our existing license, CC-BY-SA 2.0.

We’ll have a further update next week and, in any case, before the bot starts running.

API Read – Write returns

The sysadmin team completed the data base migration to the new DB server on schedule during the the morning of 04 April 2012. The API is now back to normal, Read – Write operation. Now the final steps of the license upgrade will proceed as outlined in the March – April service schedule announcement

Other items of possible interest as the license upgrade process proceeds:

osm.org map tile generation will recommence within the next few hours.

Replication diffs during the license upgrade period have started after community requests. These cc-by-sa data replication diffs are found in the redaction-period directory on planet. planet.openstreetmap.org/redaction-period. These diffs will only serve the period up until the switch to the new license. Mappers have requested these diffs for the redaction period. General consumers of OSM data may choose to consume these diffs or not at their discretion.

ODbL diffs will be located in another directory to be announced in future.

During the redaction period it is recommended that editors save their work early and often to reduce the chances of, and the complexity of conflicts with the back ground redaction process.

Bulk GPS point data


OpenStreetMap contributors have used track files from their GPSr devices for years while improving OSM data. They have shared those track files and the track points have been available to other mappers via editors and the web site. Now we are providing a way for you to get all of those points at once.

Announcing planet.osm.org/gps

This is the collected GPS point data from the first seven and a half years of OpenStreetMap. It is a very large collection of points and it is very raw data.

  • the compressed file is 7GBytes in size
  • uncompressed, the file is a 55GByte text file
  • the data consists of coordinate pairs only, with no track file or meta data
  • points were contributed by thousands of users
  • points were contributed as thousands of distinct track files
  • the data includes 2,770,233,904 points

Is this a big deal?

This might be the largest collection of Open Data GPS points published. Do you know of larger collections? Tell us in the comments.

Working with this file might not be your cup of tea. Over time, I expect that tools will emerge from the community to make this data easier to manage. For now, it is raw and it is extensive.

All of this data has been previously available to OpenStreetMap contributors in other forms, via editors and the web site. This file provides a new way to get the same data and to get all of it at once.

Example data

If you do decide to work with the file, this is the format that you can expect.

-900000000,1771882380
-900000000,1293757490
-891154290,1237501070
-877697750,1653442410
-871875000,1589069750
-871875000,1237507350
-843750000,1350007780
-843750000,1153132660
-843750000,1040632590
-843750000,1012507800
-843750000,1012507340
-824414060,1082922390
-815625000,1575007660
-805627440,1579614290
-805517580,1579284700
-805517580,1578845240
-804473020,1373773550
-787500000,1237507380
-787500000,1181257510
-787500000,1096882780
-787499970,1096882780
-778591613,1666901550
-778591613,1666898345
-778591384,1666911621

What format is that?

These are comma separated, raw lat / lon coordinates in a simple text format. To get the coordinates divide each number by 10**7. The points are sorted by location, starting in the far southeast of the globe (90 S,180 E) and moving northwest.

Thanks

Thanks as always to the hundreds of thousands of OpenStreetMap contributors over the seven-plus years of the project so far. Thanks to the syadmins for moving this data to a place where we can all access it.

This version of the GPS data file is CC-By-SA and published by OpenStreetMap and Contributors. The image in this article is a visualization of some of this point data in Europe. The image is licensed similarly and was created by Dave Stubbs.

Service schedule March – April 2012

The long awaited and eagerly anticipated license upgrade is coming soon, the conclusion of a multi-year process. To minimise disruption to OpenStreetMap mappers and users, we’re taking the opportunity to install our new database server (funded by your generous contributions) at the same time – reducing the total amount of downtime needed.

Please be aware of the following service schedule and the list of dates further on in this article. The license upgrade will start with the database server migration. All times and dates are subject to change: our volunteers are working flat out on this, so thanks in advance for your patience and support.

Mappers

There will be a several-day period of limited API availability. The API will be Read Only while the database is moved to the new database server, ramoth. This new database server was funded by your contributions during the December 2011 fund raising campaign. No map editing will be possible while the API is Read Only.

During the remainder of the upgrade, the API should operate normally. Please postpone bulk edits where possible, until after the license upgrade is complete. As always with system improvements, your patience while you find items to refine is appreciated. Consider monitoring the friendly OSM IRC chat channel, #osm on irc.oftc.net, if you have questions.

We would ask mappers who have not yet agreed (or otherwise) to the new terms to log into OSM before the downtime starts on 1st April (0800 UTC) and signal their intention. We are pleased that the vast majority of OSM data will be unaffected by the license change, and thank all the mappers who have thus far consented to their data being distributed under the new license.

Data consumers

The typical planet file scheduled for this week has been postponed. The final old license planet file will be created from 01 April 2012 data. It will be published once the planet file generation completes which may be delayed by a few days.

The old license replication diff service will stop when the database goes to Read Only mode for the server migration. A new license replication diff service will begin at the completion of the license upgrade from a new address. There will be another message with details for using the new diffs when you are ready to start consuming them.

Important Dates

These dates and times are subject to change without notice.

  • 1st April: Enter Read-only mode. 8am UTC
  • 4th April: End of downtime. Enter Read-write mode on new server. Our estimate is that this will be in the morning, but could be subject to change.
  • 5th-6th April: bbox-based live-data tests of rebuild logic.
  • 7th April: Start automated processing of all remaining non-clean objects.
  • 9th April: Progress report and estimation of remaining license upgrade time.
  • To be Determined: On completion of the processing, subject to satisfaction with the outcome we can re-declare the dataset to be ODbL. Immediately afterwards, a first new license planet file will be generated/published, and diff creation will resume.

Thanks

Thanks, as always, to the many people who make OpenStreetMap great. These people include: the countless mappers who have improved the data, the operations volunteers who make so many things “just work”, the programmers who make participating in OSM easier every day, the donors who provide the hardware and hosting we rely upon.

Many of you ‘overlap’ in more than one of these areas. Please be aware that the thanks are cumulative. 🙂

[As excerpted from Dermot McNally’s announcement. Context added from additional sources.]

Database downtime – 20 March 2012

On Tuesday 20th of March 2012 between 13:45 and 16:15 (GMT / UTC) the
primary database server will unavailable due to emergency maintenance.

The following services will be affected:

  • www.openstreetmap.org web site will not allow user login or edits (Potlatch). [1]
  • API and map database editing (using JOSM, Merkaartor etc.) will be unavailable.
  • planet.openstreetmap.org will be available but no new diffs will be generated during the outage.
  • Forum (no logins)
  • trac (bug-tracker, no logins)
  • help.openstreetmap.org (no logins)

Other services will not be affected – all of the following are
expected to function normally:

  • tile serving (“View The Map” & “Export”)
  • Wiki
  • Nominatim (search)
  • mailing lists
  • subversion and git (source code repositories)
  • donate.openstreetmap.org

Technical: Database Server Smaug: Replacing faulty motherboard.
Supplier Engineer Onsite. We have contingency hardware available.

1: Maps will still be viewable on the openstreetmap.org homepage and
on other people’s websites.

Sincerely
Grant Slater
On behalf of the OpenStreetMap sysadmin team

OSMF Hardware Update

Servers, ramoth and bowser, in position.

The sysadmin team have brought some more hardware on-line for our delight. OpenStreetMap servers are named after dragons, taken from “Here be Dragons” the inscription denoting incomplete / unexplored places on historical maps. Learn more about OpenStreetMap dragons.

  • azure – Java-XAPI. Experimental. Provides read-only OSM data from a refreshed XAPI code base. Azure has recently received a long anticipated disk upgrade.
  • bowser – joins soup and fiddlestick as another Web Front End server. This server will make browsing the osm.org web site snappier for browsing the map, etc.
  • eustace – Web stats. Experimental. Tracks user behaviour across OSM servers to understand and improve user experience.
  • gorwen and orm –geoDNS tile caching. gorwen is kindly supplied and hosted by Teleservice Skåne AB GeoDNS serves tiles from the closest tile server. The sysadmin team hope to have a North American server available shortly. We seek a host for other, geographically diverse servers. If you are interested and not worried by 100Mbits/s, please speak to a sysadmin on #osm-dev at http://irc.osm.org
  • poldi – Nominatim. Provides search and geocoding of OSM data. Return of a local nominatim instance after a hiatus.
  • ramoth – is the second database server. The successful fund raising campaign of December 2011 led to the installation of this server. This server increases the reliability and performance of OSM database operations. Current status: in rack, being configured.

We also welcome the two newest members of the server team, Ian Dees (iandees) and Sarah Hoffman (lonvia). They will be maintaining the Java-XAPI and nominatim servers.

Photo credit

Photo by Firefishy.

Funding drive – Improving OSM reliability and performance

Pound Coins
Pound Coins photo by William Warby CC-BY-2.0


OpenStreetMap is growing fast. We’ve recently welcomed our 500,000th signed up user, and we’ve logged our 10.000.000 th update to the map. Over the next few weeks we’re running a fund-raising drive while we invest in server infrastructure to improve reliability and performance of OpenStreetMap. If you’d like to support the project in this way, or you know anybody else who would like to give OSMers an early Christmas present, visit our fund raising site:

donate.openstreetmap.org/server2011

You have the option to include your name on the donors list. We’re aiming to raise £15,000 (~ 23,000 U.S. dollars).  Let’s see how quickly we hit the target!

We wanted to run another fund raising drive, because last time we had a big one was back in 2009 (old blog post) and we were blown away by how quickly we raised the target amount. It seemed as though people were looking for an outlet for their generosity and goodwill towards the project. Since we’re planning to buy a new server, now seems like a good time to do it again.

The Operations Working Group, which has the important role of keeping core OSM services running smoothly, plans to invest in a new server. This will provide us with a database replica. This improvement is at the very core of the OpenStreetMap infrastructure, giving services greater resilience. It means we’ll bounce back quicker and easier in the event of a hardware failure. In time the new server will also bring about some performance improvements.  We have a wiki page with more technical details and plans for the new hardware.

We hope you’ll agree that, although these improvements are very much behind-the-scenes, they are important. Please give generously to help make them happen!

Tile Usage Policy

Tile image by vidalia_11 is licensed CC-By-SA

We’ve had to block some uses of the OpenStreetMap Foundation tile servers. This article describes what is happening and why. This article also describes how you can adapt if you are affected.

We’re really proud of the increased popularity of OpenStreetMap. We’ve seen seven-plus years of project growth in every measurable area. As the project has grown we’ve learned and adapted in many ways. The use of our tile server has grown faster than every other aspect of the project. One way that we are adapting now, is by restricting how our map tiles may be used.

Slow sign by DaveCrosby is licensed CC-By-SA

You’ll still be able to use our tiles in creative and interesting ways but the volume of use will be limited. We need to limit access to our tile server to only those users who don’t overburden our resources.

Those users who make large demands on our tile server will be slowed down by our throttling mechanism. This throttling mechanism is rarely triggered by mappers.

Problematic applications may show this image instead of a map.

Those applications which make exceptional demands in aggregate from their users will be blocked. The tile usage policy is on the OSM wiki.

So what can you do about this? How can you get the wonderful OpenStreetMap tiles for your mobile device?

You’ll find more advice about potential tile source alternatives on the wiki.

Read on if you would like to know more about the history of the OpenStreetMap tile server.

OSM tile server background

We started creating rendered images of our map data as a way to encourage our data contributors. Mappers enjoy seeing the results of their surveys on the OpenStreetMap web site, and they can be inspired to map in their neighbourhood the things they see other mappers surveying in other places. Mappers loved the tile server when it first appeared. Potential users often looked at the map in 2006 and said, “Hey, why is there a huge blank spot where my town should be?” Some of those potential users became the long-time contributors that we all know and love.

As more contributors mapped more neighbourhoods, more blank spots started to fill in. More potential users became actual users, and OpenStreetMap tiles started to appear in more places. The tile server became even more popular when rapid updates were enabled. Rather than updating the map every week, parts of the map could update as contributors added data. If you remember the weekly updates you also remember that funny tingle you had the first time you mapped something and it appeared on the map immediately; it seemed like magic, didn’t it?

The OpenStreetMap Foundation has had a tile usage policy for some time. From September 2008 it has been explicitly stated that bulk downloading of tiles was discouraged. OpenStreetMap kept growing. More people came to understand the awesomeness of OSM tiles.

Also in 2008, the Ordnance Survey started serving map tiles to users through their OS Openspace program. In July of 2011, Ordnance Survey served their one billionth tile to a user.

OpenStreetMap serves a Billion tiles every eleven days.

So we know a thing or two about providing awesome maps to users. We do it all with the crowd-sourced data from our contributors around the world, the volunteered time of our sysadmins who keep our servers running, and the generous donations of servers and bandwidth and funds.

We’ve had to become more restrictive of the use of our tile server over time. We’ve limited how many tiles you can consume in a period of time. These restrictions only affected the most-demanding of tile consumers. The everyday mapper never ran into a problem getting tiles to add data to OSM. That allowed the growing number of users to continue to have access to OpenStreetMap tiles without our resources being monopolized by one or two bad actors.

More and more mobile applications started using OpenStreetMap tiles. Many of them included a bulk downloading method so that tiles could be saved on the device at home, rather than downloading tiles at a punitive data rate. That bulk-downloading has always been problematic for OpenStreetMap because a single user will consume hundreds of times the resources of an average user. There are so many applications using OSM tiles, with so many users making unreasonable demands on our resources that it is affecting the quality of service for the average user. And that’s not fair.

So we’ve started blocking the applications that are causing us the most trouble, in addition to blocking users with problematic specific behaviours. We regret it, in a way. After all, we map because we want people to be able to use our data. But our resources have to be used in a way that everybody can share. We can’t have a small number of people consuming all of our resources.

So that’s why some people have started to see the “prohibited” tiles on their maps. Overuse. Or mobile applications that are causing overuse by a group.