The Importance of Open Data and APIs

Jun032013

To call this a “growing” movement would be an understatement. Open data and APIs are and will be critical in the future of web and mobile technology. If you haven’t heard of open data or APIs yet, now’s the time to catch up.

What is open data?

“Open data” is an umbrella term that includes open APIs for a variety of services, as well as static, downloadable data sets. Static data makes sense for situations in which the data won’t change, such as building locations, whereas APIs allow for a stream of dynamic data, whether it’s a stream of tweets or bus arrival times.

Current landscape of open data and APIs

Right now we’re seeing some maturity in the space as API conventions are converging. This is a hugely important trend in open data, as the publishing company for the API has a lot of responsibility when making changes or updates to their stream. The more developers rely on a stream of data, the more impact (whether positive or negative) a small change can have on the community. Conventions in API structure and management is a good step towards avoiding negative consequences from API changes, but at the same time, conventions can’t guarantee that the major players in the space – Twitter, Facebook, etc. – won’t change their API.

Some other changes we can expect to see in this space include flat data sets being offered as a live data feed where applicable, and the ability to increase the frequency and granularity of the data on a particular feed.

To get started with building a successful open data API, there are a few key steps.

Planning for an open data API

Planning is an integral part of the open data set-up process, since good planning will help mitigate problems in the deployment. However, many problems can be hard to foresee when building a new API data stream.

Right out of the gate, expect that you will have a number of applications that are immediately reliant on the API – even though it has just been released.

Any changes that break members’ apps will alienate members of your community and cause significant problems. Changing JSON or XML format, method names, or attribute names can all have major implications to your members’ structure for growth, so they’re all important considerations.

To plan for future growth, make sure to take into consideration:

  • How you’ll add new data sets, new methods and new formats
  • How you’ll update to new versions
  • How to manage varying access levels and throttle thresholds amongst your member base
  • How you’ll communicate API downtime, changes and upgrades, and which channels you’ll use.

Another helpful step in planning is to model the daily usage of the API, either in advance or actively during usage. This will help you plan for actual traffic, as opposed to monthly estimated levels. For example, in terms of a transit API, requests tend to be much higher during peak travel hours in the morning and evening. The system has to be able to handle this volume.

Managing your API via third-party services

The API management system acts as a mediator between members’ applications and the API. This service provides API authorization, throttling, usage tracking and a member self-service portal. As a proxy, it also protects the API feed from being overloaded or attacked.

There are several services that do this, including:

  • 3Scale: a cost-effective solution that offers a comprehensive management console, easy integration and easy set up.
  • Mashery: a more robust, enterprise-level offering.
  • Apigee
  • Apify
  • Layer7Tech

As with any other service, it’s important to identify and deal with errors that can come up with the API or the API management service.  In other words, your API can’t go down if a service upstream isn’t working.  The API still has to tell the apps downstream that there is an error in the data feed and the message has to be in an expected format.

Another aspect of API management services is that they can be used to normalize and amalgamate multiple feeds of data. The service will act as a one-stop shop for data, and can help make the process easier for developers who work with the feed. This is important, since the easier it is for developers to work with the data, the faster and more widely the API will be adopted, used and tested.

Once set up, there are also six key must-haves that will help ensure the success of your API.

Open API Must-Haves

Reliability and Uptime

To be effective, the API cannot go down. During the set-up and maintenance process, you need to anticipate and deal with external errors to make sure the interface layer doesn’t go down.

Analytics

You’ll have to know overall usage to determine growth. Understanding members and their applications will also help you understand the value of the data to their app. Once you know who is using the data and which functions are most popular, you can adjust and plan for growth.

Documentation

Solid documentation helps to encourage adoption and makes the process of using your API easier for developers. Code samples can strengthen the documentation and help developers see how others have integrated the API previously.

Support

You’ll need to have a channel, or several channels, via which developers can ask questions and get support regarding the API. The questions that come up should always be answered and resolved by other developers within your organization.

Marketplace

A prominent gallery of the apps that make use of the API helps to entice other developers to create apps, and can serve as inspiration and motivation.

Outreach

There will need to be channels for input and requests from the development community around your API. Either these, or other channels, will also need to be set up to alert members to downtime or upcoming changes to the API. Ideally, there will also be channels to gather feedback from the community, to hear how developers are using the API, what works, what doesn’t and suggestions for future use. A great way to start your outreach plans is to reach out to the open data community and groups in your city or industry. They can be very helpful, and you don’t need to duplicate their existing community.

Wrap-up

The concept of the API is very much anticipated and now expected. The concept is on the cusp of transitioning from technology to utility – new and changing to ubiquitous and reliable. It’s required by a world of connected devices – go and read up on the Internet of Things.  It will be one of many moving parts, thus the mechanics of each individual piece must not be undervalued. At the end of the day, in steady state, the reliability will be the most important thing.

Skip to sharing