Weak Coupling

NB: this is a draft of two new chapters for ‘The API Book’ I’m writing.

Strong coupling and related problems

In previous chapters we tried to outline theoretical rules and principles, and illustrate them with practical examples. However, understanding principles of change-proof API design requires practice like nothing before. An ability to anticipate future growth problems comes from a handful of grave mistakes once made. One cannot foresee everything, but can elaborate a certain technical intuition.

So in following chapters we will try to probe our study API from the previous Section, testing its robustness from every possible viewpoint, thus carrying out some ‘variational analysis’ of our interfaces. More specifically, we will apply a ‘What If?’ question to every entity, as if we are to provide a possibility to write an alternate implementation of every piece of logic.

One important remark we’re stressing here is that we’re talking about alternate realizations of business logic, not about entity implementation variants. APIs are being changed to make something usable in the first place — something missing in the original design. Just re-implementing some interfaces makes no sense to your customers.

This observation helps narrowing the scope, sparing time on varying interfaces blindly. (Actually, there is always an infinite number of such variations, and it would be a Sisyphean labor to examine all of them.) We need to understand why such changes might be desirable, and then we may learn how they are to be made.

A second important remark is that many decisions allowing for such a variability are already incorporated in our API design. Some of them, like determining readiness, we explained in previous chapters in detail; some of them are provided with no comments, so it’s now time to explain the logic behind these decisions.

NB. In our examples the interfaces will be constructed in a manner allowing for dynamic real-time linking of different entities. In practice such integrations usually imply writing an ad hoc code at server side with accordance to specific agreements made with specific partner. But for educational purposes we will pursue more abstract and complicated ways. Dynamic real-time linking is more typical to complex program constructions like operating system APIs or embeddable libraries; giving educational examples based on such sophisticated systems would be too inconvenient.

For the beginning, let us imagine that we decided to give to our partners an opportunity to serve their own unique coffee recipes. What would be the motivation to provide this functionality?

• maybe a partner’s coffee houses chain seeks to offer their branded beverages to clients;
• maybe a partner is building their own application with their branded assortment upon our platform.

Either way, we need to start with a recipe. What data do we need to allow adding new recipes to the system? Let us remember what contexts the ‘Recipe’ entity is linking: this entity is needed to couple a user’s choice with beverage preparation rules. At first glance it looks like we need to describe a ‘Recipe’ in this exact manner:

// Adds new recipe
POST /v1/recipes
{
  "id",
  "product_properties": {
    "name",
    "description",
    "default_value"
    // Other properties, describing
    // a beverage to end-user
    …
  },
  "execution_properties": {
    // Program's identifier
    "program_id",
    // Program's execution parameters
    "parameters"
  }
}

At first glance, again, it looks like a reasonably simple interface, explicitly decomposed into abstraction levels. But let us imagine the future — what would happen with this interface when our system evolves further?

The first problem is obvious to those who read chapter 11 thoroughly: product properties must be localized. That will lead us to the first change:

"product_properties": {
  // "l10n" is a standard abbreviation
  // for "localization"
  "l10n" : [{
    "language_code": "en", 
    "country_code": "US", 
    "name", 
    "description" 
  }, /* other languages and countries */ … ]
]

And here the big question arises: what should we do with the default_volume field? From one side, that's an objective quality measured in standardized units, and it's being passed to the program execution engine. From other side, in countries like the United States we had to specify beverage volume not like ‘300 ml’, but ‘10 fl oz’. We may propose two solutions:

• either partners provide the corresponding number only, and we will make readable descriptions on our own behalf,
• or partners provide both the number and all of its localized representations.

The flaw in the first option is that a partner might be willing to use the service in some new country or language — and will be unable to do so until the API supports them. The flaw in the second option is that it works with predefined volumes only, so you can’t order an arbitrary volume of beverage. So the very first step we’ve made effectively has us trapped.

The localization flaws are not the only problem of this API. We should ask ourselves a question — why do we really need these name and description? They are simply non-machine-readable strings with no specific semantics. At first glance we need them to return them back in /v1/search method response, but that's not a proper answer: why do we really return these strings from search?

The correct answer lies a way beyond this specific interface. We need them because some representation exists. There is a UI for choosing beverage type. Probably name and description are simply two designations of the beverage type, short one (to be displayed on the search results page) and long one (to be displayed in the extended product specification block). It actually means that we are setting the requirements to the API based on some very specific design. But what if a partner is making their own UI for their own app? Not only two descriptions might be of no use for them, but we are also deceiving them. name is not ‘just a name’ actually, it implies some restrictions: it has recommended length, optimal to some specific UI, and it must look consistently on the search results page. Indeed, ‘our best quality™ coffee’ or ‘Invigorating Morning Freshness®’ designation would look very weird in-between ‘Cappuccino’, ‘Lungo’, and ‘Latte’.

There is also another side to this story. As UIs (both ours and partner’s) tend to evolve, new visual elements will be eventually introduced. For example, a picture of a beverage, its energy value, allergen information, etc. product_properties will become a scrapyard for tons of optional fields, and learning how setting what field results in what effects in the UI will be an interesting quest, full of probes and mistakes.

Problems we’re facing are the problems of strong coupling. Each time we offer an interface like described above, we’re in fact prescript implementing one entity (recipe) basing on implementations of other entities (UI layout, localization rules). This approach disrespects the very basic principle of designing APIs ‘top to bottom’, because low-level entities must not define high-level ones. To make things worse, let us mention that revers principle is actually correct either: high-level entities must not define low-level ones, since that simply isn’t their responsibility.

The rule of contexts

The exit from this logical labyrinth is: high-level entities must define a context, which other objects are to interpret. To properly design adding new recipe interface we shouldn’t try find better data format; we need to understand what contexts, both explicit and implicit, exist in our subject area.

We have already found a localization context. There is some set of languages and regions we support in our API, and there are requirements — what exactly the partner must provide to make our API work in a new region. More specifically, there must be some formatting function to represent beverage volume somewhere in our API code:

l10n.volume.format(value, language_code, country_code)
// l10n.formatVolume('300ml', 'en', 'UK') → '300 ml'
// l10n.formatVolume('300ml', 'en', 'US') → '10 fl oz'

To make our API work correctly with a new language or region, the partner must either define this function or point which pre-existing implementation to use. Like this:

// Add a general formatting rule
// for Russian language
PUT /formatters/volume/ru
{
  "template": "{volume} мл"
}
// Add a specific formatting rule
// for Russian language in the ‘US’ region
PUT /formatters/volume/ru/US
{
  // in US we need to recalculate
  // the number, then add a postfix
  "value_preparation": {
    "action": "divide",
    "divisor": 30
  },
  "template": "{volume} ун."
}

NB: we are more than aware that such simple format isn’t enough to cover real-world localization use-cases, and one either rely on existing libraries, or design a sophisticated format for such templating, which takes into account such things as grammatical cases and rules of rounding numbers up, or allow defining formatting rules in a form of function code. The example above is simplified in purely educational purposes.

Let us deal with name and description problem then. To lower the coupling levels there we need to formalize (probably just to ourselves) a ‘layout’ concept. We are asking for providing name and description not because we just need them, but for representing them in some specific user interface. This specific UI might have an identifier or a semantic name.

GET /v1/layouts/{layout_id}
{
  "id",
  // We would probably have lots of layouts,
  // so it's better to enable extensibility
  // from the beginning
  "kind": "recipe_search",
  // Describe every property we require
  // to have this layout rendered properly
  "properties": [{
    // Since we learned that `name`
    // is actually a title for a search
    // result snippet, it's much more
    // convenient to have explicit
    // `search_title` instead
    "field": "search_title",
    "view": {
      // Machine-readable description
      // of how this field is rendered
      "min_length": "5em",
      "max_length": "20em",
      "overflow": "ellipsis"
    }
  }, …],
  // Which fields are mandatory
  "required": [
    "search_title", 
    "search_description"
  ]
}

So the partner may decide, which option better suits them. They can provide mandatory fields for the standard layout:

PUT /v1/recipes/{id}/properties/l10n/{lang}
{
  "search_title", "search_description"
}

or create a layout of their own and provide data fields it requires:

POST /v1/layouts
{
  "properties"
}
→
{ "id", "properties" }

or they may ultimately design their own UI and don’t use this functionality at all, defining neither layouts nor data fields.

The same technique, i.e. defining a specific entity responsible for matching a recipe and its traits for the underlying systems, might be used to detach execution_properties from the interface, thus allowing the partner to control how the recipe is being coupled with execution programs. Then our interface would ultimately look like:

POST /v1/recipes
{ "id" }
→
{ "id" }

This conclusion might look highly counter-intuitive, but lacking any fields in ‘Recipe’ simply tells as that this entity possesses no specific semantics of its own, and is simply an identifier of a context; a method to point out where to look for the data needed by other entities. In the real world we should implement a builder endpoint capable of creating all the related contexts with a single request:

POST /v1/recipe-builder
{
  "id",
  // Recipe's fixed properties
  "product_properties": {
    "default_volume",
    "l10n"
  },
  // Execution data
  "execution_properties"
  // Create all the desirable layouts
  "layouts": [{
    "id", "kind", "properties"
  }],
  // Add all the formatters needed
  "formatters": {
    "volume": [
      { "language_code", "template" },
      { "language_code", "country_code", "template" }
    ]
  },
  // Other actions needed to be done
  // to register new recipe in the system
  …
}

We should also note that providing a newly created entity identifier by client isn’t exactly the best pattern. However, since we decided from the very beginning to keep recipe identifiers semantically meaningful, we have to live with this convention on. Obviously, we’re risking getting lots of collisions on recipe naming used by different partners, so we actually need to modify this operation: either partners must always use a pair of identifiers (i.e. recipe’s one plus partner’s own id), or we need to introduce composite identifiers, as we recommended earlier in Chapter 11.

POST /v1/recipes/custom
{
  // First part of the composite
  // identifier, for example,
  // the partner's own id
  "namespace": "my-coffee-company",
  // Second part of the identifier
  "id_component": "lungo-customato"
}
→
{
  "id": "my-coffee-company:lungo-customato"
}

Also note that this format allows us to maintain an important extensibility point: different partners might have totally isolated namespaces, or conversely share them. Furthermore, we might introduce special namespaces like ‘common’ to allow publish new recipes for everyone (and that, by the way, would allow us to organize our own backoffice to edit recipes).

Weak coupling

In previous chapter we’ve demonstrated how breaking strong coupling of components leads to decomposing entities and collapsing their public interfaces down to a reasonable minimum. A mindful reader might have noted that this technique was already used in our API study much earlier in Chapter 9 with regards to ‘program’ and ‘program run’ entities. Indeed, we might do it without program-matcher endpoint and make it this way:

GET /v1/recipes/{id}/run-data/{api_type}
→
{ /* A description, how to
     execute a specific recipe
     using a specified API type */ }

Then developers would have to make this trick to get coffee prepared:

• learn the API type of the specific coffee-machine;
• get the execution description, as stated above;
• depending on the API type, run some specific commands.

Obviously, such interface is absolutely unacceptable, simply because in the majority of use cases developers don’t care at all, which API type the specific coffee machine runs. To avoid the necessity of introducing such bad interfaces we created new ‘program’ entity, which constitutes merely a context identifier, just like a ‘recipe’ entity does. A program_run_id entity is also organized in this manner, it also possesses no specific properties, being just a program run identifier.

But let us ask ourselves a more interesting question. Our API allows for running programs on coffee machines with a known API. Let us imagine then that we have a partner with their own coffee houses with a plethora of coffee machines running different APIs. How would we allow the partner to get an access to the programs API, so they could plug their coffee machines to our system?

Out of general considerations we may assume that every such API would be capable of executing three functions: run a program with specified parameters, return current execution status, and finish (cancel) the order. An obvious way to provide the common interface is to require partner have this three functions support a remote call, for example, like this:

// This is an endpoint for partners
// to register their coffee machines
// in the system
PUT /partners/{id}/coffee-machines
{
  "coffee-machines": [{
    "id",
    …
    "program_api": {
      "program_run_endpoint": {
        /* Some description of
            the remote function call */
        "type": "rpc",
        "endpoint": <URL>,
        "format"
      },
      "program_state_endpoint",
      "program_stop_endpoint"
    }
  }, …]
}

NB: doing so we’re transferring the complexity of developing the API onto a plane of developing appropriate data formats, e.g. how exactly would we send order parameters to the program_run_endpoint, and what format the program_state_endpoint shall return, etc., but in this chapter we're focusing on another questions.

Though this API looks like absolutely universal, it’s quite easy to demonstrate how once simple and clear API end up being confusing and convoluted. This design presents two main problems:

1. It describes nicely the integrations we’ve already implemented (it costs almost nothing to support the API types we already know), but brings no flexibility in the approach. In fact, we simply described what we’d already learned, not even trying to look at a larger picture.
2. This design is ultimately based on a single principle: every order preparation might be codified with these three imperative commands.

We may easily disprove №2 principle, and that will uncover the implications of №1. For the beginning, let us imagine that on a course of further service growth we decided to allow end users to change the order after the execution started. For example, ask for a cinnamon sprinkling or for a contactless takeout. That would lead us to adding a new endpoint, let’s say, program_modify_endpoint, and new difficulties in data format development (we need to understand in the real time, could we actually sprinkle cinnamon on this specific cup of coffee). What is important is that both (endpoint and new data fields) would be optional because of backwards compatibility requirement.

Now let’s try to imagine a real world example which doesn’t fit into our ‘three imperatives to rule them all’ picture. That’s quite easy either: what if we’re plugging via our API not a coffee house, but a vending machine? From one side, it means that modify endpoint and all related stuff are simply meaningless: vending machine couldn't sprinkle cinnamon over a coffee cup, and contactless takeout requirement means nothing to it. From the other side, the machine, unlike people-operated café, requires takeout approval: an end user places an order being somewhere in some other place, then walks to the machine and pushes ‘get the order’ button in the app. We might, of course, require the user to stand in front of the machine when placing an order, but that would contradict the entire product concept of users selecting and ordering beverages and then walking to the takeout point.

Programmable takeout approval requires one more endpoint, let’s say, program_takeout_endpoint. And so we've lost our way in a forest of three endpoints:

• to have vending machines integrated a partner must implement program_takeout_endpoint, but doesn't actually need program_modify_endpoint;
• to have regular coffee houses integrated a partner must implement program_modify_endpoint, but doesn't actually need program_takeout_endpoint.

Furthermore, we have to describe both endpoints in the docs. It’s quite natural that takeout endpoint is very specific; unlike cinnamon sprinkling, which we hid under pretty general modify endpoint, operations like takeout approval will require introducing a new unique method every time. After several iterations we would have a scrapyard, full of similarly looking methods, mostly optional — but you would need to study the docs nonetheless to understand, which methods are needed in your specific situation, and which are not.

We actually don’t know, whether in the real world of coffee machine APIs this problem will really occur or not. But we can say with all confidence regarding ‘bare metal’ integrations that the processes we described always happen. The underlying technology shifts; an API which seemed clear and straightforward, becomes a trash bin full of legacy methods, half of which borrows no practical sense under any specific set of conditions. If we add a technical progress to the situation, i.e. imagine that after a while all coffee houses become automated, we will finally end up with the situation when half of methods aren’t actually needed at all, like requesting contactless takeout method.

It is also worth mentioning that we unwittingly violated the abstraction levels isolation principle. At a vending machine API level there is no such term as ‘contactless takeout’, that’s actually a product concept.

So, how would we tackle this issue? Using one of two possible approaches: either thoroughly study all the subject area and its upcoming improvements for at least several years ahead, or abandon strong coupling in favor of weak one. How would the ideal solution look from both sides? Something like this:

• higher-level program API level doesn’t actually know how the execution of its commands works; it formulates the tasks at its own level of understanding: brew this recipe, sprinkle with cinnamon, allow this user to take it;
• underlying program execution API doesn’t care what other same-level implementations exist; it just interprets those parts of the task which make sense to it.

If we take a look at principles described in previous chapter, we would find that this principle was already formulated: we need to describe an informational context at every abstraction level, and design a mechanism to translate it between levels. Furthermore, in more general sense we formulated it as early as in ‘The Data Flow’ paragraph in Chapter 9.

In our case we need to implement the following mechanisms:

• running a program creates a corresponding context comprising all the essential parameters;
• there is method to exchange the information regarding data changes: the execution level may read the context, learn about all the changes and report back the changes of its own.

There are different techniques to organize this data flow, but basically we always have two context descriptions and two-way event stream in-between. In case of developing an SDK we might express this idea like this:

/* Partner's implementation of program
   run procedure for a custom API type */
registerProgramRunHandler(apiType, (program) => {
  // Initiating an execution
  // on partner's side
  let execution = initExecution(…);
  // Listen to parent context's changes
  program.context.on('takeout_requested', () => {
    // If takeout is requested, initiate
    // corresponding procedures
    execution.prepareTakeout(() => {
      // When the cup is ready for takeout,
      // emit corresponding event
      // for higher-level entity to catch it
      execution.context.emit('takeout_ready');
    });
  });
  return execution.context;
});

NB: In case of HTTP API corresponding example would look rather bulky as it involves implementing several additional endpoints for message queues like GET /program-run/events and GET /partner/{id}/execution/events. We would leave this exercise to the reader. Also worth mentioning that in real-world systems such event queues are usually organized using external event message systems like Apache Kafka or Amazon SQS.

At this point a mindful reader might begin protesting, because if we take a look at the nomenclature of the entities emerged, we will find that nothing changed in the problem statement, it actually became even more complicated:

• instead of calling the takeout method we're now generating a pair of takeout_requested/takeout_ready events;
• instead of long list of methods which shall be implemented to integrate partner’s API, we now have a long list of context objects fields and events they generate;
• and with regards to technological progress we changed nothing: now we have deprecated fields and events instead of deprecated methods.

And this remark is absolutely correct. Changing API formats doesn’t solve any problems related to the evolution of functionality and underlying technology. Changing API formats solves another problem: how to make the code written by developers stay clean and maintainable. Why would strong-coupled integration (i.e. coupling entities via methods) render the code unreadable? Because both side are obliged to implement the functionality which is meaningless in their corresponding subject areas. And these implementations would actually comprise a handful of methods to say that this functionality is either unsupported, or supported always and unconditionally.

The difference between strong coupling and weak coupling is that field-event mechanism isn’t obligatory to both sides. Let us remember what we sought to achieve:

• higher-level context doesn’t actually know how low-level API works — and it really doesn’t; it describes the changes which occurs within the context itself, and reacts only to those events which mean something to it;
• low-level context doesn’t know anything about alternative implementations — and it really doesn’t; it handles only those events which mean something at its level, and emits only those events which could actually happen under its specific conditions.

It’s ultimately possible that both sides would know nothing about each other and wouldn’t interact at all. This might actually happen at some point in the future with the evolution of underlying technologies.

Worth mentioning that a number of entities (fields, events), though effectively doubled compared to strong-coupled API design, raises qualitatively, not quantitatively. program context describes fields and events in its own terms (type of beverage, volume, cinnamon sprinkling), while execution context must reformulate those terms according to its own subject area (omitting redundant ones, by the way). It is also important that execution context might concretize these properties for underlying objects according to its own specifics, while program context must keep its properties general enough to be applicable to any possible underlying technology.

One more important feature of event-driven coupling is that it allows an entity to have several higher-level contexts. In typical subject areas such situation would look like an API design flaw, but in complex systems, with several system state-modifying agents present, such design patterns are not that rare. Specifically, you would likely face such situations while developing user-facing UI libraries. We will cover this issue in detail in the ‘SDK’ section of this book.

The Inversion of Responsibility

It becomes obvious from what said above that two-way weak coupling means significant increase of code complexity on both levels, which is often redundant. In many cases two-way event linking might be replaced with one-way linking without significant loss of design quality. That means allowing low-level entity to call higher-level methods directly instead of generating events. Let’s alter our example:

/* Partner's implementation of program
   run procedure for a custom API type */
registerProgramRunHandler(apiType, (program) => {
  // Initiating an execution
  // on partner's side
  let execution = initExecution(…);
  // Listen to parent context's changes
  program.context.on('takeout_requested', () => {
    // If takeout is requested, initiate
    // corresponding procedures
    execution.prepareTakeout(() => {
      /* When the order is ready for takeout,
         signalize about that, but not
         with event emitting */
      // execution.context.emit('takeout_ready')
      program.context.set('takeout_ready');
      // Or even more rigidly
      // program.setTakeoutReady();
    });
  });
  /* Since we're modifying parent context
     instead of emitting events, we don't 
     actually need to return anything */
  // return execution.context;
});
}

Again, this solution might look counter-intuitive, since we efficiently returned to strong coupling via strictly defined methods. But there is an important difference: we’re making all this stuff up because we expect alternative implementations of lower abstraction level. Situations when different realizations of higher abstraction levels emerge are, of course, possible, but quite rare. The tree of alternative implementations usually grows top to bottom.

Another reason to justify this solution is that major changes occurring at different abstraction levels have different weight:

• if the technical level is under change, that must not affect product qualities and the code written by partners;
• if the product is changing, i.e. we start selling flight tickets instead of preparing coffee, there is literally no sense to preserve backwards compatibility at technical abstraction levels. Ironically, we may actually make our program run API sell tickets instead of brewing coffee without breaking backwards compatibility, but the partners’ code will still become obsolete.

As a conclusion, because of abovementioned reasons higher-level APIs are evolving more slowly and much more consistently than low-level APIs, which means that reverse strong coupling might often be acceptable or even desirable, at least from the price-quality ratio point of view.

NB: many contemporary frameworks explore a shared state approach, Redux being probably the most notable example. In Redux paradigm the code above would look like:

execution.prepareTakeout(() => {
  // Instead of generating events
  // or calling higher-level methods,
  // an `execution` entity calls 
  // a global or quasi-global
  // callback to change a global state
  dispatch(takeoutReady());
});

Let us note that this approach in general doesn’t contradict to loose coupling principle, but violates another one — of abstraction levels isolation, and therefore isn’t suitable for writing branchy APIs with high hierarchy trees. In such system it’s still possible to use global or quasi-global state manager, but you need to implement event or method call propagation through the hierarchy, i.e. ensure that low-level entity always interact with its closest higher-level neighbors only, delegating the responsibility of calling high-level or global methods to them.

execution.prepareTakeout(() => {
  // Instead of initiating global actions
  // an `execution` entity invokes
  // its superior's dispatch functionality
  program.context.dispatch(takeoutReady());
});
// program.context.dispatch implementation
ProgramContext.dispatch = (action) => {
  // program.context calls its own
  // superior or global object
  // if there are no superiors
  globalContext.dispatch(
    // The action itself may and
    // must be reformulated
    // in appropriate terms
    this.generateAction(action)
  );
}

This is a draft for upcoming ‘Backwards Compatibility’ section of the book I’m writing; the work continues on Github. I’d appreciate if you share it on reddit, for I personally can’t do that.