Index

Introduction

This document provides a guide for those implementing an API Service served under api.apps.cam.ac.uk.

Once you have read through this page, create a post on our Developers' Hub project to start the process of publishing an API.

It is intended for those directly involved in the development or deployment of services and contains technical language. A glossary of technical terms appears at the end. Some familiarity with web application development and deployment is assumed.

APIs which are hosted under api.apps.cam.ac.uk appear on the API developers' portal. Anyone in the University with Raven credentials can register a new client application and request that it be allowed to call one or more APIs. Some APIs require manual approval but it is intended that the majority of APIs may be used by any registered client application.

Each client application has one or more sets of client credentials registered by the developer. These client credentials can be used to identify an application to the API Gateway and authenticate requests from it.

Overview of the API Gateway

The API Gateway sits between users of your API and a backend which implements the behaviour of your API. The API Gateway acts as a proxy which performs many of the common tasks required to implement an API.

Requests to your service are authenticated by the API Gateway and passed on to your backend. HTTP headers are used to add contextual information to the request. The following diagram illustrates the process:

Flow of a request through the API Gateway

Here a request from an application to https://api.apps.cam.ac.uk/colleges/botolphs has been received by the API Gateway. The API Gateway has authenticated the client application as being allowed to call the API and passes the request on to a backend service.

The request passed to the backend is authenticated using credentials specific to the API Gateway passed in an Authorization header, information about the original request is added via X-Forwarded-... headers and information on who is calling the API is added via X-Api-... headers.

Any response from the backend is passed back to the client application. Along the way CORS Headers are added to the response so that the API may be used from within a web-browser.

Service maturity

The UIS are adopting the Government Digital Services Agile Delivery Framework. The API Gateway service is moving towards the end of its discovery phase and we are seeking to gain a small number of APIs and client applications as part of the alpha phase.

Responsibilities

APIs under api.apps.cam.ac.uk are hosted via the API Gateway which is responsible for handling the following aspects of an API Service:

  • discoverability on the API developer portal,
  • interactive API documentation,
  • registration of client applications,
  • authentication via OAuth2,
  • per-client application rate limiting,
  • per-client application quota enforcement, and
  • threat protection agains denial of service attacks, etc.

After validating and authenticating incoming requests, the API Gateway passes them to a backend via HTTP.

As a Service Developer, you are responsible for:

  • keeping your API in line with the current UIS API technical standards,
  • keeping your API backwards-compatible and versioned,
  • providing documentation for your API in OpenAPI version 3 format,
  • if your service is to be limited to certain client application developers, documenting how one determines if a given developer is allowed to access your API,
  • authenticating requests from the API Gateway as having actually come from the API Gateway,
  • hosting the backend at a public URL using HTTP over TLS for API Gateway authentication and confidentiality, and
  • maintaining monitoring and alerting configuration for your backend.

Paths and backend routing

API Services are hosted under api.apps.cam.ac.uk at api.apps.cam.ac.uk/[API NAME] where [API NAME] is the name of your API. For example, a hypothetical supervision management API may have URLs which look like the following:

  • https://api.apps.cam.ac.uk/teaching/v1/supervision/ABC123 is the location of a particular "supervision" resource in the production service.
  • https://api.apps.cam.ac.uk/teaching/v2/contact/ABC123 is the location of a particular "supervision" resource in a newer backwards-incompatible version of the API which tracks the more general concept of "contact time"
  • https://api.apps.cam.ac.uk/teaching-test/v1/supervision/ABC123 is the location of a particular "supervision" resource in the staging service.

In this example, the API Gateway will forward all requests to endpoints starting with https://api.apps.cam.ac.uk/teaching/ to your production backend and all requests to endpoints starting with https://api.apps.cam.ac.uk/teaching-test/ to your staging backend. It is up to your backend to interpret the requests and form an appropriate response. Any response is forwarded back to the client application.

Request flow

To make use of an API, a client application first needs to exchange its long lived client credentials for a short-lived access token. This is done using an API which sits under https://api.apps.cam.ac.uk/oauth.

Instructions for registering a client applications are available on the Developer Portal.

When a client application makes a request to the API Gateway, the first part of the request path is matched against a list of available APIs and the access token presented by the client is checked to ensure it is valid. If the access token is invalid or if the client application is not registered to use the API, an error response is returned to the client application.

The API Gateway then checks per-application rate-limits and quotas to guard against accidental or intentional misuse. If the request fails one of these checks, an error response is returned to the client application.

HTTP headers are added to the request so that the backend can determine which application has called it, which OAuth2 scopes its access token is valid for and what the original request URL was. The backend can use these headers to deny access to resources depending on its own permissions model.

Finally the request has authentication credentials applied so that the backend may verify that the request originated from the API Gateway and the request is passed to the backend.

The backend processes the request as it sees fit and sends a response. The API Gatway passes the response back to the client application unmodified save for adding CORS Headers.

Developing and deploying a backend

This section discusses options for developing and deploying a backend. It is not meant to be an exhaustive set of options. If you have mature processes for developing and deploying applications, by all means make use of them.

Dynamic backends

A dynamic backend is implemented by code running on a server. This will often be a REST-ful wrapper around a database.

A backend is hosted just like any traditional web application. The UIS Cloud First policy requires that Cloud hosting solutions be preferred for web application hosting. Web applications products from Amazon, Azure or Google can be used to host them.

A dynamic backend must follow the UIS API technical standards. In particular the following aspects are required:

  • The API must be documented according to the OpenAPI specification version 3.
  • The backend must be served using HTTP over TLS from a publicly routable IP address.
  • The backend must authenticate the API Gateway using one of the authentication methods below.
  • If the backend forms URLs pointing at its own resources it must use the X-Forwarded-... headers passed by the API Gateway.

The API Gateway operators can assist in evaluating whether a backend meets the technical requirements of the API Gateway but it is ultimately the responsibility of those managing the backend service to ensure that it continues to do so.

Static backends

Some backends can simply be static files. For example, https://api.apps.cam.ac.uk/ravenstats is a single static document which provides information on device and operating system statistics for Raven.

There is no need to spin up a dedicated server for static APIs; one can host them via a static hosting service. The ravenstats backend, for example, is hosted as a Google Cloud Storage object. This means there is no VM to patch and no web-server to configure. Often the best part is no part.

The UK Government Bank Holiday API is another example of a minimal static API which is driven by a static file. The backend service for that API simply serves a copy of static files maintained as part of the source repository.

Even if your API is hosted by a static backend, it must still follow the UIS API technical standards and the API must still be documented according to the OpenAPI specification version 3.

Authentication

This section discusses the mechanisms the API Gateway can use to authenticate itself to a backend.

Libraries which can validate authentication credentials for the API Gateway are available for Java, Python, Go, .NET CLR languages, Ruby, PHP and JavaScript.

Currently the supported authentication methods are:

  • A time-limited OIDC JWT passed in the "Authorization" header which can be validated either by one of the libraries above or using any JWT library which supports verification of tokens.
  • A time-limited OAuth 2 access token passed in the "Authorization" header which can be validated by one of the client libraries listed above.

JWT-based authorisation is preferred since many JWT libraries are available allowing backends to be implemented in a wide variety of implementation languages.

Two-sided or "mutual" TLS authentication may be supported in future if there is a demand.

Additional authentication methods may be considered if there is a technical need. Authentication methods should not involve the direct transfer of credentials from the Gateway to the backend and instead should make use of an appropriate time-limited and/or public-key cryptography-based authentication scheme.

JWT authentication

The JWT scheme is the preferred authentication scheme. Requests from the API gateway will have an Authorization header of the following form:

Authorization: Bearer [TOKEN]

where [TOKEN] is a JSON web token.

JWTs are composed of three parts: a header, a payload and a signature. The header and payload are both JSON documents. The signature allows the payload to be cryptographically verified. The header is generally used to identify the key and algorithm corresponding to the payload signature.

Backends should verify the token in the following manner:

  • Check that the token header has a type claim (typ) of JWT and that the key identifier (kid) and algorithm (alg) claims match one of the public keys available from the JSON Web Key set specifying known public keys. Backends may cache the public keys available at that endpoint if necessary but should refresh them periodically.
  • Verify the JWT's signature using the appropriate public key and algorithm from the JSON Web Key set.
  • Check that the authorised party (azp) claim corresponds to the service account identity for the API Gateway. This is currently apigee-gateway@api-prod-a3dc87f7.iam.gserviceaccount.com but your backend should be configurable to accept multiple audiences to allow rotation of this identity in future.
  • Check the expiration time (exp) and issued at time (iat) to ensure that the token is currently valid.
  • Check the audience (aud) of the token. It should correspond to the base URL of your backend.

JWT libraries which perform verification in accordance with this specification are available for a wide variety of languages.

Requests to your backend which fail verification must not be acted upon and should return a response with a 403 Forbidden status code. Do not echo the received token in the response.

The following is an example header from a valid token:

{
  "alg": "RS256",
  "kid": "fed80fec56db99233d4b4f60fbafdbaeb9186c73",
  "typ": "JWT"
}

The following is an example payload which might be sent to a backend service hosted at https://janes-api-server.srv.uis.cam.ac.uk/frobnication-api/. Backends must not attempt to parse the payload until the signature has been verified. Backends must also accept additional claims which should be ignored.

{
  "aud": "https://janes-api-server.srv.uis.cam.ac.uk/frobnication-api/",
  "azp": "apigee-gateway@api-prod-a3dc87f7.iam.gserviceaccount.com",
  "exp": 1614090860,
  "iat": 1614087260
}

Contextual headers

The API Gateway sets a number of headers on incoming requests before passing them to a backend server.

The following headers allow the backend to identify which application is using it:

  • X-Api-Org-Name A name corresponding to which instance of the API Gateway forwarded the request. For production services, this will always be "cam".
  • X-Api-Developer-App-Id The unique id of the client application which has been authorised. Backends can use this as a stable "username" to identify the client application even if client credentials are rotated.
  • X-Api-OAuth2-ClientId The client id for the credentials the application used to authorise. Backends can use this to further restrict access to a particular set of credentials but care should be taken as developers can freely rotate these credentials.
  • X-Api-OAuth2-Scope The scope or scopes the application requested when authenticating. This is currently unused but may be used in future to associate credentials with limited privileges rather than an "all or nothing" approach to API usage.

The following headers allow the backend to form URLs pointing back to itself as they appear to users of the API Gateway.

  • X-Forwarded-For A list of IP addresses providing the original client application IP address and the IP addresses of any subsequent proxy servers.
  • X-Forwarded-Host The host name of the original request.
  • X-Forwarded-Proto The URL scheme of the original request. This will usually be "https".
  • X-Forwarded-Port The port which the original request was sent to. This will usually be 443 corrsponding to the usual port for HTTP over TLS.
  • X-Forwarded-Prefix The "prefix" or "base" where the API was accessed. For example, if your API sits at https://api.apps.cam.ac.uk/my-api/ then this will be my-api.

As an example, a backend service may construct a URL pointing to its widgets/12345 resource by concatenating the headers above as:

${X-Forwarded-Proto}://${X-Forwarded-Host}:${X-Forwarded-Port}/${X-Forwarded-Prefix}/widgets/12345

If X-Forwarded-Port is "443" and X-Forwarded-Proto is "https" this can be shortened to:

https://${X-Forwarded-Host}/${X-Forwarded-Prefix}/widgets/12345

As an additional layer of verification, backends may choose to reject requests where X-Forwarded-Proto is not "https" or where X-Api-Org is not "cam" by returning a response with a 403 Forbidden status code.

Backends must not hard-code assumptions about hostnames or prefixes when forming URLs pointing back at themselves.

Cross-origin Resource Sharing (CORS) Headers

Ordinarily a web-browser will not allow JavaScript running in the context of one web-site to request data from another. This is to avoid a class of attacks where an attacker uses a third-party site under their control to perform actions on another site on the users behalf without the user's consent.

CORS headers provide a mechanism for web servers to indicate that they expect to be called from other web pages. APIs are a clear example of this and so the API Gateway adds CORS headers to all outgoing responses automatically.

A corollary which follows from this is that your backend must not accept any authentication methods vulnerable to cross-origin attacks. In particular this applies to "session cookies". You backend should only support the authentication mechanisms supported by the API Gateway.

Monitoring and alerting

The API Gateway service can provide a very basic level of monitoring for your service but the API Gateway management team strongly recommend that you implement your own monitoring and alerting for any backend services. In particular the API Gateway can only report when a request to a backend fails, it does not currently proactively monitor the health of a backend.

The monitoring and alerting provision is likely to grow over time as the API Gateway service matures.

Staging and development environments

It is likely that you will have multiple instances of your service running. For example you may have a "production" instance, a "staging" instance for testing upcoming changes to "production" and a "development" instance for developing the service.

The API Gateway can support this model by hosting "unlisted" proxies for your staging and development instances. These proxies should use a different backend from your production instance.

For example, suppose you are developing an API to retrieve information on colleges. Your public API is advertised in the API catalogue as being at https://api.apps.cam.ac.uk/colleges and allows any developer to enable that API for their application.

You may also have a testing API at https://api.apps.cam.ac.uk/colleges-test and a development API at https://api.apps.cam.ac.uk/colleges-dev. Neither of these APIs are listed in the catalogue and client applications must be manually approved to use them.

The ...-test and ...-dev APIs each point to a different backend.

The API Gateway itself has a staging instance. Occasionally we may ask you to allow your backend to be called by this instance so that we can test bug-fixes or enhancements to the service. In this case your API is available on a different API Gateway, for example https://next.api.apps.cam.ac.uk/colleges, but requests still go to your production backend.

This arrangement is illustrated in the following diagram:

Use of multiple environment with the API Gateway

The API Gateway management team strongly recommend that staging and development instances be created for API-driven services.

Resources and further reading

This section lists resources which may be of use when developing or designing backends.

Tooling

  • Terraform is the recommended tool to provision hosting infrastructure.
  • Apicurio studio can be used to design and document APIs according to the OpenAPI specification version 3.

Support

Technical support can be provided by opening an issue on the API Gateway project in GitLab.

Opening an issue is also the preferred channel to start a discussion about getting your API hosted by the API Gateway service.

The API Gateway team can provide advice and guidance on the implementation of backend services but due to the heterogeneous nature of service implementation around the University it is unlikely that they will be able to recommend a specific technical solution for your use case.

Internal

External

Glossary

This section lists some technical terms used in this document along with a brief description:

  • API Initialism for Application Programmers' Interface, a mechanism for services to exchange information with each other and request that actions be performed.
  • API Gateway The service which collects multiple APIs in one place and provides a unified authentication and discovery mechanism.
  • Backend The service which implements an API. The API Gateway passes authenticated and authorised requests to an API to the backend for processing.
  • Client Application A service written by a third party which makes use of an API.
  • Client Credentials A non-secret "client id" or "client key" which identifies the client application to the API Gateway along with a secret "client secret" which acts as a password for the client application.
  • Cross-origin Resource Sharing (CORS) A machanism for a web server to indicate it understands the implications of the historically "leaky" web security model and will not trust credentials which are automatically added by browsers.
  • Developer Portal A website where developers of client applications may register their applications, browse available APIs, create client credentials and request that their applications be granted use of sensitive APIs.
  • Header Value attached to a HTTP request or response outside of the body content. Header values are keyed by a case-insensitive header name. Used to add contextual information to HTTP requests or responses and, in particular, used by the API Gateway for authentication to backend services and to pass information about the client application which originated an API request.
  • Hypertext Transfer Protocol (HTTP) An emerging lingua franca for server-to-server communication. Used alongside TLS to allow communication between the API Gateway and backend services.
  • Internet Protocol (IP) address Unique numeric address assigned to an Internet server or client. A publicly routable IP is, informally, an IP address which is not reserved for private use.
  • JSON Web Token (JWT) A standard which allows data to be encoded in a compact token which may be validated by a backend using public-key cryptography without requiring the use of a separate validation service.
  • OAuth 2.0 A standard for service-to-service and natural person-to-service authentication and authorisation.
  • OpenAPI A specification for documenting REST-ful APIs in a machine-readable format. This allows the API documentation on the developer portal to be interactive and for client libraries to be generated automatically.
  • OpenID Connect (OIDC) A set of standards built upon OAuth 2.0 which standardises various common patterns around OAuth 2.0.
  • Platform as a Service (PasS) Services which offer technical hosting services at a higher-level than raw VMs. Examples include web application container hosting and managed database hosting.
  • REST Acronym for Representational State Transfer, a set of semantics for representing resources within APIs which maps well on to the standard HTTP request methods. APIs which implement REST semantics are termed REST-ful.
  • Scheme A URL scheme is the part before :// which indicates which protocol can be used to access the resource. For URLs for resources vailable using HTTP over TLS this is usually "https".
  • Sensitive API An API which by its nature should not be offered to all client application developers within the University. This is usually only the case for APIs which deal in confidential personal data or provide elevated or privileged access to systems.
  • Serverless Term of art referring to the exclusive use of Platform as a Service products to implement a service. Colloquially derived from the concept that one does not directly manage or configure servers in a Platform as a Service ecosystem.
  • Status code The status code of a HTTP response indicates if the response corresponds to a succesful request or some error. Common status codes include 200 (OK) and 403 (Forbidden).
  • Transport Level Security (TLS) Used along with HTTP to implement API backends ensuring confidentiality, integrity and authority of content.

Last update: October 28, 2021