Model Services Versioning
Versioned APIs are helpful when integrating with systems on different release cycles. With a mobile app that is solely using application data it is reasonable to say that when we update the application, the mobile app updates too. But if you are doing back office integration with an accounting system, for instance, you can't expect the organization to engage the accounting system team whenever there is a new application release - especially as we move to more continuous releases.
To address this requirement, Model Services support versioning with API elements which lists the classes, attributes, and events to be exposed over the REST API.
Elements and Attributes
APIs
API - The main element for defining an API. It has an extension of (*.api). APIs manage a map between a public versioned Model Services API and the class/members that are available over that API.
The name of the API metadata consists of the name and version (e.g. finance.1.api).
<API description="API for finance deprecation will be on 2/2/2022">
<Classes>
<Class name="Person">
<Attributes>
<Attribute name="loginName" type="string" required="true"/>
<Attribute name="firstName" type="string"/>
<Attribute name="DATE_ATTRS" type="any" static="true" deprecated="true"/>
</Attributes>
<Events>
<Event name="createPerson" static="true" deprecated="true">
<Arguments>
<Argument name="loginName" type="string" required="true"/>
<Argument name="collectionArg" type="any" collection="true"/>
</Arguments>
</Event>
<Event name="eventWithVarArg" vararg="true"/>
</Events>
</Class>
</Classes>
</API>
APIs cannot be customized in sub-projects. If a new API is needed, it should be given a new name. This is enforced by the validator. i.e. If the base API is not the same as the customized or local API, it will throw an error.
name and version
The name of the API. It is used to access the API as well as the version. It is discussed in the Endpoints section below.
examples
Core finance would be named "finance". A client may call thier API something like "mosaic" that would include all of their customizations and extra modules. At the client both the "finance" and "mosaic" endpoints would be available. Internal development may choose either to work against. Our development partners, such as Grapevine6 would use the core so as to work at all client locations.
Two API files with the same name and version cannot exist.
description
Optional description of this API. Will show up in the IDL if it's specified.
Classes
List of Classes exposed over this version of the API.
name
The name of a class to be included in the API `e.g. Person`
Attributes
The list of attributes to expose over this API
name - The name of the attribute to include in the API
deprecated
true if the of the member's compatibility property at the time of API generation was deprecated. This is needed to keep track of what the state of the model was at the time of generation. Without it, we could change or delete members the moment they became deprecated. With it, we can only change or delete members when the oldest supported API has the member marked as deprecated.
Events
The list of events to expose over this API
name - The signature of the event to include in the API. Specified as the event name "/" number of arguments
Class Model Support
API elements can be created by hand but if the class model is annotated, this can be fairly automatic by using the API tool to perform generation.
compatibility
Backward compatibility: none|backward|deprecated
"backward" includes the class in the current versioned API. "deprecated" includes the class, but indicates that an alternative is preferred and that it may be excluded from future versions. All compatible members will be marked as "deprecated" on the API. "none" excludes the class from the current versioned API.
Class.Members (attributes and events)
compatibility
* none (or not specified)
* backward, (same previous behaviour if compatible was true)
* deprecated (the reason for deprecation should be shown in the description)
This is not just for Classes. It is also for Aspects and Augments.
Maintenance
When we are ready to commit to an API, we run the API tool. We specify the name of our new version to the tool. First validation runs then the apiGenerator. The generated API metadata may be placed in a seperate mixin.
We publish "core api"s that represent our core model and all modules (mixins) considered to be core for finance and for cloud. This is what we list in our website documentation and what development partners write extensions against. e.g. the core API for finance will likely be called "finance".
Vertical specific APIs may be published as well. e.g. "wm" for wealth management.
Client environments may publish their own APIs to reflect local customizations, or module inclusions, that they want to expose to downstream systems in a controlled contract and versioned way. e.g. "mosaic".
This does require that clients clean out and evolve their old APIs roughly on the same schedule as our core APIs. Maybe we want some way to enforce this. i.e. Mark "core" APIs as core - or have a concept of an API having a dependence on another API. If a client (or dependent) API exists that is older than a "core" API and it contains a reference to a member as "backward" that is that we have "deprecated" in core APIs, a warning or an error should be emitted. The client should then issue their own new version.
Validation
Validation compares the current model against the API metadata.
- A member can only be changed from compatibility = backward/deprecated to compatibility=none only if the last version is deprecated or it is not mentioned in the .api.
- A member can only be changed or removed if it does not show up on any versioned API, even as deprecated.
Additions of new compatible attributes are not an issue, just changes to existing compatibility=backward attributes.
Validation runs as part of metadata loading.
The apiGenerator iterates through all of the classes and creates api entries based on the compatibility propery settings on attributes and events.
Retiring a Version
Simply delete the API file for e.g. version 2.
Endpoints
`<root>/json/...` works as it does today and provides access to all public attributes/events.
`<root>/api/<name>/json...` returns Swagger and attributes/events that are only valid for that version.
Example
`<root>/api/finance.3/json/Person` --> attributes/events from the version 3 finance API
`<root>/api/finance.5/json/Person` --> attributes/events from the version 5 finance API
`<root>/api/finance.5/json?filter=Person/0` --> Swagger from the version 5 finance API for person
`<root>/json/Person` --> all public attributes
OpenAPI/Swagger will only return valid attributes for the requested version. It should also include styling for deprecated elements.
!!! note Version as Parameter or Header
We will also allow you to specify version as a parameter or header.
`<root>/json/Person?apiname=finance,apiversion=3` --> attributes/events from the version 3 finance API
`<root>/json/Person` HEADERS: APIName="finance", APIVersion="3" --> attributes/events from the version 3 finance API
Use Cases
### Add a Member
Just add it and make it compatible or not
### Rename a Member
Attribute: Mark the attribute as deprecated with a deprecated reason and either rename it later after the deprecation period is over, or create the new one now and move all values from the old attribute to the new one after the deprecation period and delete the old one.
Event: Just create the new event and point the old one to the new one. Both are available for the deprecation period then delete the old event.
### Delete an Member
Mark it as deprecated and once the deprecation period is over, delete it.
### Move an Member to a New Class
Attribute: Mark the attribute as deprecated with a deprecation reason and either move it after the deprecation period or create the new attribute in the new class now and move all values from the old attribute to the new one after the deprecation period is over and delete the old one.
Event: Just create the new event in the new class and point the old one to the new class event. Both are available fo the deprecation period then delete the old event.
### Add a Class
Mark the Class as backward.
### Rename a Class
Mark the old class as deprecated with a deprecated reason and either remove the class later after the deprecation period and move the values to the new class, or create a new class now and move all values from the old class after the deprecation period and the delete the old class.
### Delete a Class
Set the class as deprecated with a deprecated reason then delete the Class instances after the deprecation period.
### Making an Attribute Required that wasn't before or change its type
Mark the attribute as deprecated with a deprecated reason. Once the deprecated period is over, change the attribute to required or change its type and set it back to compatible. Some type conversions will be fine regardless, such as integer to decimal.
!!! note
The solution/process we take here should be similar to the approach to non-destructive schema changes for datasources and continuous deployment that we are working on presented below.
Database Schema Change Patterns (Model Changes follow same patterns)
**Column Rename**
Release 1: Modify Schema:
- ALTER TABLE customers ADD COLUMN correct VARCHAR(20);
- UPDATE customers SET correct = wrong WHERE id BETWEEN <RANGE>;
- UPDATE customers SET correct = wrong WHERE id BETWEEN <RANGE>;
- Multiple updates with ranges stops table lock escalation. Sharding can be used to improve execution time and reduce changes of table locking escalations
⇢ Modify Code: Read data from old column and write to both. Announce depreciation for wrong column
Release 2: Modify Code: Read data from new column and write to both. Steps maintains ability to roll code back to previous release
Release 3: Modify Code: Read data from new column and write to both. Steps maintains ability to roll code back to previous release
Release 4: Modify Code: Read/write data to new column only. Announce drop of support for wrong column
Release 4+ Quarantine Interval: Modify Schema: ALTER TABLE customers DELETE COLUMN wrong;
**Column Format Change**
-Treat as column rename
**Column Delete**
Release 1: Modify Code: Stop reading data, but continue to write
Release 2: Modify Code: Stop writing the data – remove any not null constraints from table
Release 2+ Quarantine Interval: Modify Schema: ALTER TABLE customers DELETE COLUMN wrong;
General/Future Thoughts
1. For a stable read-only API, Reporting is also a great addition. Like BI models before it, it can provide views that we could expose as versioned APIs. That said, it still has the *extra work* that SOA needs (but maybe it's just the reporting views) and it is only the Read part of CRUD.
2. Could we change persistence to allow us to write to two locations to support some of the processes discussed in micro-service API refactoring texts?
3. Calculated attributes and events and implementation changes? What happens if the implementation changes? Is this something we will consider important. I don't think so. I think a *versioned API* specifies a contract around the Service Model and the Information Model payload. Maybe do this via a *strict* qualifier? I'm not recommending this but think it should be mentioned.
4. "Indirection". This design doesn't address indirection to the business model. It could be done through calculated fields and other business model mechanisms but it doesn't provide SOADef-like capabilities. This is intentional - we found that SOA had too much overhead in design and maintenance. So currently, no indirection. One thought to explore in a future version, is to create a pass-through module (either in a mixin or in a nodeJS AFL server) that could be optionally generated and optionally modified to accomodate business logic "in the middle".
Tracability
Some of the work we have done recently should help us meet this requirement integration channel monitoring and statistics know that an API is being used and maybe by whom. So we should be able to reach out and say "hey this is going away soon and you need to upgrade". Clients just want control around how and when they upgrade.
MOCK
Capabilities could be added to the business model to provide default mock values for attributes.