It's Cards all the Way Down
- Atomic Data
- Names and Namespaces
- One Kind of Content
- Formatters
- Linking and Inclusion
- Sets and Rules
- Patterns
- Rules
- Model Views and Events
- Web of Applications
- The Web Service Model
- The Big Picture
- Cards on the Move
- Cards as Code
- Card Extended
Contents:
Introduction
In object oriented languages, there is always one master class that all the other classes inherit from. There is a root in the inheritance tree. In ruby, this is the Object class, in content we propose the Card. Developed in the context of a rails web application that shares and extends wiki concepts, cards are designed to support web interfaces and wiki practices, but as we shall see along the way, each of these features can be considered and packaged as optional components in a much larger space of applications development.
This content model was developed inside a particular web application, specifically a ruby and rails application with its MVC pattern. We began to notice a new pattern developing where the controller is cannonical, serving up pages or data according to a ReSTful protocol, and application development can be concentrated on implementing models and views. Actions are customized in events at specific times in the object life cycle.
Although the view process might need to reference controller actions in providing controls linked to them, we abstract that away in Formatters that can be defined according to the application's needs. The initial web application focused on rich html views that are matched with some front-end code, but a formatter can be defined to output json or xml, or even the ruby objects that would serialize into that data. Cards are not dependent on a web applicaiton context, but they are great for implementing one. Implementing text formatters for email and print use, and data formatters for AJAX friendly formats like JSON or XML gives a wide range of choices to the application architect.
We begin an exploration of the entire architecture from the bottom up. Along the way we will see how each feature supports the entire design while remaining distinct, and each is an axis of flexibility for the architect to extend the platform. The system of naming data supports wiki-style linking of content, as well as the URI namespace used in web contexts. It also is part of how you can specify rules in data, and other extensions in code according to a system of sets. The names for data in application space (cardnames) map consistently to code namespaces Not only does this fascilitate the creation of modular extensions, new code, but it can support modular systems architecture where an application is a network of services.
We don't know what to call this pattern yet, is it MoVE for Model, View, Event, or is it MoFoS for Model, Formatter, Sets? Whatever you call it, it represents a significant departure from the MVC pattern; something that fits well with the MVC pattern, but also moves closer to a user oriented model of connected data from the tabular architecture of relational databases.
I am setting out to describe this architecture, not to document it. It is an exploration for all of us. If you are a systems architect, you will learn how to apply this technology effectively and possibly help define it, and if you are a user of card based systems (Decko) I hope you will be able to understand it better at the level you encounter it and help us to make it more comprehensible for all, users and builders alike. Module and core developers are not far below the architects, and our data programmers (see Sharking) will find much of interest, but when designers, editors and product managers can understand how it empowers them in applying technology I will have met my goal.
Each axis of flexibility provided by the architecture is an opportunity for designers as well as architects. The designer can directly extend and modify the details of the implementation in data and if not directly in code, close to it because the small amounts of code in many modules are easy to repurpose. The architect can concentrate on flexible and stable systems. Designers and module coders will know what they can ask of architects and what features that the architecture naturally supports. Designers can create working prototypes from existing module libraries while doing iterative development with UI and domain developers to refine the system. At the other end of the scale the architects and systems programmers and engineers who can deploy card based components in a flexible networked applications architecture.
Atomic Data
Atomicity has a number of technical definitions mostly relating to making the actions on groups of data items behave as single events. That is part of the meaning here, but it is actually a pretty recent addition to the code base that adds support for grouping events at the data level. More generally it means that all data units are equivalent, that they each have certain properties and that they behave consistently individually and in groups by the interaction of those properties. Although we emphasize flexibility, the architect is unlikely to need to change the basics of these properties and how they interact. The axis of flexibiilty needs to already be there, adding a new axis is possible but not often the right solution.
The current card model has three things, and a couple more optional things. We'll start our exploration with these and branch out. First is the name, a card has to have a name to secure an identity and persistence in a collection, and we will look at the structure of names in the next chapter. Cards connect to other cards, and every card connects to a type, which is also a card. Any card with a multipart name connects to two more cards based on the name structure, a left and right, or trunk and tag. The last of the three things is content, which can contain references to other cards, and we will cover content and references right after names.
As you might expect, references to data are references to data atoms, cards. Other logic might dig into the internal representation of a card's content and parse it or split it according to the abstractions internal to the type, but for the platform, it is a single unit of data. It can have multiple references to other cards in the content, but model specific logic will be needed to parse those out.
Names and Namespaces
If cards are data atoms, then namespaces are collections or containers of those data atoms. We will wait to explore how multiple namespaces might relate and interact at a later point as we look at how they can be used in systems architecture and to implement networks of interlinked data. To facilitate linking to other systems, the logic of card names is available as a gem. For now we will confine ourselves to a single collection and internal relationships.
A simple name/card has only one part, lets start there. That means the name doesn't have any joint characters in it. One part means the name only links to the one card it names. We will use plus (+) for the joint character throughout, although some designers may prefer to use slash (/) because of the good analogy to file system an web paths. Because the name is how a single item is identified in a collection, one of the most important issues is what makes one name equal to another. That could be simply comparing the name strings, but that's not particularly friendly for users. Programmers sometimes want to require identifiers to match exactly and put restrictions on what strings are valid names, but in this system we want it to be easy to refer to the same thing in different ways. To make this work, any name string can be translated to a canonical version of the name called a key, and if two names have the same key, they will refer to the same card.
In simple terms it breaks the name up into words, singularizes simple plurals, makes the words all lower case and joins them with underscore (_). Most punctuation and symbols are considered just like a space or underscore and are stripped off the ends and reduced to only one underscore between words. There are only a few exceptions to this the joint character (+) and slash (/) aren't allowed in simple names, and the name gem can be configured to add additional banned characters or use an alternate joint. The other exception is asterisk (*) which is treated like an ordinary letter and included in the words of a key. Later we will see how this special character is used to define special sets of system cards used to help implement advanced features.
Now we can address non-simple names, ones with more than one simple part. You can think of the name as just an array of parts, but there's a bit more that happens when you name a card. Consider the cardname A+B+C, it has three parts, but let's look at what cards are related. The card A+B+C is composed of two parts, A+B and C, thetn A+B is A and B, so A+B+C, A+B, A, B and C are each cards that are parts of A+B+C. If any of them don't exist when A+B+C is created, they will get created then. Note that B+C is not involved at all, and won't be created with A+B+C.
Finally, there is a concept of relative or contextual names. When a name has variable or missing parts it doesn't actually refer to a specific card but can be evaluated in a particular context. We'll dig into this further when discussing linking and inclusion, but the simplest case is when one card has a relative link to a to another. Consider the card A linking to the relative name +B+C. The link would go to A+B+C. The card B+C is still not involved, just the relative name +B+C, which doesn't point to anything without a context card.
Although not mentioned specifically, name handling is a little English centric. You may have noticed the mention of singular form, mainly an English language transformation, but nothing else is limiting. It uses generalized character classes so it can keep all the word characters and fold all the special characters away in key processing. Soon, cards will have a language, maybe the name and content would share a single language unless we can think of a use case to split them. That means a single deck can include cards in many languages making it truely multilingual. Adding support for giving names and content a language is almost trivial, and that is all that will be added to the card core code. It demonstrates the power of the architecture that the details of how different languages are presented and organized are implemented like much of the system, as modular components that extend the core architecture. It makes them easy to modify and extend from an evolving reference implementation. The initual support for multilingual sites might be a bit primative, but it won't take updates to the core representation of cards beyond the basic data to represent the language for the main card attributes. The projects with active communities in many languages will be able to push the UX in downstream module changes.
One Kind of Content
Actually, there is considerably more flexibility in data representation than this title implies, but underneath the hood there is one way to store the content of a card. Different databases will have different capabilities to deal with large objects, and the default mysql implementation limits the content size to 64K, but abstractly it is not size limited. It is text data, and might not respond well to storing nulls, and adopting international character encoding standards, it is multi-linqual text in UTF 8 encoding. That doesn't mean you can't store objects, and common formats such as XML, JSON or YAML are essentially text based and are supported well by card formatters.
None of the common formats really support references directly, which makes the usage and syntax of coding links into text or objects another important axis of flexibility. As with everything about the card model, this coding is specific to sets of cards, so type based encoding/decoding of data formats and links is used in implementing basic types and features. Any content may be processed for references, and there is a modular parser that matches chunks, that are internal (name based) or external (typically URIs) links. This parsing feature impliments comments and literals so I can show {{inclusion}} syntax without processing it. The developer might use this feature for other types of content processing or translation.
In other words, the architect can use a wide range of representations for content and parse references however desired according to sets the system specifies. There is also a standard parser to be used and customized. It is used to implement the standard linking and inclusion syntax described below, and it is flexible enough to be used to implement a lot of other features we never thought of.
Formatters
This is another core component of the card system. Initially there was only one formatter to render different views of cards to view and edit them as components of a web page and I wanted to access cards in XML representations through a ReSTful controller. This required a major rewrite of the current rendering code and brought in the concept of rendering to different formats. Although we had views, we didn't yet have the system of defining views for sets, and the formatter became where the views are defined and their running context. When support for set based views was implemented, the way they were defined and looked up was changed to support sets. That implementation has evolved to support sets on views in a way parallel to model modules are loaded. Global views (on *all) are still defined as methods on a formatter, but the set-based views are defined on a module just like the model methods and events. We'll come back to this when covering code based rules in Model Views and Events.
This isn't too different than the rails system of views where the requested format is part of selecting view. What is a bit different is the use of blocks as well as file/directory names to give more flexibility for grouping views and set code. The view code is just blocks that return strings. If you must have the typical view templates, that kind of view was recently implemented.
We'll have to come back to cover how views relate to each other and help application builders implement a consistent and intuitive user experience. Views can be specific to controller actions as in edit views that will post updates as form data, or might be very simple in the case of data formatters. The UI support the idea of open and closed views under the control of the user via standard controls.
Linking and Inclusion
So we know that referencing other cards from content isn't part of the common content abstraction, it's just an internationalized text string, and we have a way to process the string into chunks. Basic cards and any other type that wants this processing have two ways to reference other cards, by linking or including them. Web Formatters are going to have views or helpers to render links as web links, and other formatters can handle them as appropriate to the format. Inclusion, however, will recursively render the included card in a new formatter instance (same formatter class) with the view determined from defaults and inclusion options. Getting all of this to work in the first place was a bit arduous, but now that it does it is fairly easy to copy the pattern if a developer wanted to implement a different inclusion syntax or later when we want to support links between decks both local and non-local.
In the reference implementation we borrow from wiki practice and use [[link|link text]] with the link text being optional. The link might have URL syntax and be an external link, or it can name a card. Inclusion syntax is {{name|options|options1...}} where name is a contextual name that can be converted to absolute cardname using the card doing the including as the context card. When a link names a card, it can also be a contextual name interpreted just like an inclusion name. In inclusion processing, the first set of options are used to render the immediate subcard. When you provide more option sets after additional pipe (|) characters, those are used for each deeper level of inclusion, or the items of a collection when the immediate card is a Pointer or Search.
The implementation tracks all of the internal references in a database table, and WQL searches use those tables to construct efficient queries on included or linked cards. When a card is renamed, you usually want all the name based references in card content to be automatically updated. The references table is used to find them all and update the content references. The system supports this though methods on the referencing chunks that the content parser produces so it can replace the names with updated versions that are then saved to the content object. If you extend the system with a new or modified reference type, you will have to implement these support patterns so that renaming and inclusion are supported by other system components.
Sets and Rules
Up to this point we have talked about the idea of sets without saying much about how they work. That's because all the ideas above are needed to explain it. From the initial implementations of the Set feature, specific sets were represented by cards, and the way of representing a set with a card was determined. We wanted to attach one or more Settings to a set of cards by creating rule cards. A rule isn't actually a cardtype, and the standard patterns don't support it as a set, but Set and Setting are core cardtypes.
A simple example of the type pattern would be 'Set+*type' to refer to all cards whose type is the Set card. The *syle card is a Setting that determines the CSS, so a rule would be Set+*type+*style. The contents of rule cards configure and customize with data, but you can do even more with code.
These sets and patterns also have code bindings so they can be used in the definition of modules to customize and extend the common data abstractions provided by cards. That is something built into the name system that we left out above. Any card can have a codename, and the most important feature of a codename is that it never changes. Good code will never use a cardname literal, and therefore re-naming a card cannot break code references to that card. That will become important as we move to code features.
Patterns
Lets start with our simple set card 'Set+*type'. It is an instance of a particular pattern that is connected in code. Currently, it is defined in this file: mod/01_core/set_pattern/03_type.rb
def label name
%{All "#{name}" cards}
end
def prototype_args anchor
{ :type=>anchor }
end
def pattern_applies? card
!!card.type_id
end
def anchor_name card
card.type_name
end
def anchor_id card
card.type_id
end
The codename of *type is 'type', and is actually referenced by the filename itself. Note how the codename appears in the generated version of the above file, in the header as Card::SetType and in the footer as register 'type' ... :
# -*- encoding : utf-8 -*-
class Card::TypeSet < Card::SetPattern
cattr_accessor :options
class << self
# ~~~~~~~~~~~ above autogenerated; below pulled from .../mod/01_core/set_pattern/03_type.rb ~~~~~~~~~~~
def label name
%{All "#{name}" cards}
end
def prototype_args anchor
{ :type=>anchor }
end
def pattern_applies? card
!!card.type_id
end
def anchor_name card
card.type_name
end
def anchor_id card
card.type_id
end
# ~~~~~~~~~~~ below autogenerated; above pulled from .../mod/01_core/set_pattern/03_type.rb ~~~~~~~~~~~
end
register "type", (options || {})
end
The module defines class methods on Card::TypeSet which inherits from Card::SetPattern. This pattern has an anchor, which in the example is the card 'Set', which should be a cardtype card. The card being passed in to anchor_id and anchor_name is the card being tested for membership in the set, and they return values that match the id/name of the pattern anchor (the Set card in the example).
Some sets don't have an anchor, they are global sets like *all or *rstar where we don't have any card parameters to the set. In that case, you do not define anchor_name or anchor_id. Here is an example:
file: .../set_pattern/04_star.rb def label name 'All "*" cards' end def prototype_args anchor { :name=>'*dummy' } end def pattern_applies? card card.cardname.star? end
Here all the work is in pattern_applies? This will be Card::StarSet and register as the 'star' pattern bound to the '*star' Set card.
This is the list of currently standard patterns:
- *all All Cards
- *plus All plus cards (name is not simple?)
- *star All with names beginning with *
- *rstar All cards with rightmost part (tag) beginning with *
- Type+*type All cards with type of 'Type'
- tag+*right All cards with tag of 'tag'
- Type+tag+*type_plus_right All cards whose left part has type 'Type' and has tag of 'tag'
- Any+Card+*self Just this card (Any+Card)
There are many other possible patterns and most of them are very simple to add.
Rules
Rules are just a tool the module developer can use. They are used for core features like permissions, look and feel, help messages, and the developer can create new ones for any purpose. You look up the card like this:
rule = card.rule_card(:code)
And access the card's type and content to implement a feature. Rule lookup will match the most specific rule, the lowest one on the list above. Generally only one rule will apply to any card, but the developer could combine all matching rules by using a different lookup method. In code rules, we often use ruby class overloading to combine rules; by loading the most specific pattern's rule last, any redefinition will override one from a less specific set.
Model Views and Events
We can think of all of these like rules in code. Back to our example set, we have this file: mod/05_standard/set/type/set.rb Which has all of the code customizations related to this set (Set+*type), and you can see in the path as 'type/set.rb'. The key difference is we look up a data rule when we want to use it, but we load all the code rules with the application and expect it to be there to run when called.
Easy enough to load up all the code, but how does each card instance get all the rules it needs? A typical way to do this with active record support is with subclasses and single table inheritance. We started that way with cards, and found that it made initializing cards a pain because you had to have the type very early on in the object initialization. And it doesn't support other kinds of sets redily either.
Instead, we load up the singleton class. In ruby and similar object oriented languages, an object has two classes. The one we normally think of with all the common methods, and one that has methods that only apply to this instance. The '*all' set (path .../set/all/name.rb) modules are just included on the regular (shared) class one time when they are loaded, but the rest are handled per-instance. When loaded, our example module will have defined a module Card::Set::Type::Set, and when we are initializing a card there is a step that can be optionally skipped to load the set modules. We skip it if we only want to see if a card exists or similar, and you lose the modules on cards retrieved from the cache. The card remembers whether this step was done, and it will check/load as needed by the code context.
To load the modules, it goes through the sets from least specific to most and include any modules it finds (remember *all is already loaded, so we don't do those). Our example 'Set+*type' has the type Set, so it is in it's own type. When we load 'Set+*type' (or other Set cards like '*rstar' or 'tag+*right') it will include Card::Set::Type::Set on that instance. It would also load Card::Set::Rstar because *type is a star card and is on the right. It loads the more specific (the Set::Type here) last so that if the same methods are defined, the most specific version will win.
Events build on active record callbacks. That system is designed for all objects of a class having the same events triggered. If I want some events that are specific to updating *account+*right, it will still want to run that event on all cards, but only cards in the set will have the final callback method defined. That just means we have to test whether the card has included the module that defined the event before triggering.
define_callbacks :approve, :store, :extend before_validation :approve around_save :store after_save :extend
Thats how our events relate to active record events. All other events are defined relative to (:before, :after or :around) these three: :approve, :store and :extend. If you need more, you could add them in a Card::Set::All module. You can even reference the active record events directly, but that will make your module depend on active record. If it depends on it for other reasons already, then you don't give up much, but if you can avoid it, the module will be more broadly applicable in the world of cards.
Views are similarly defined in sets, but they will also be inside a format block. When defined, they are defined into a formatter class or module, so views on *all sets are defined straight into the class (Card::TextFormat < Card::Format as an example), or into nested modules for sets: Card::TextFormat::Type::Set for views on Set+*type. Then when the formatter is initialized, always with a card to format, it will load the formatter modules for the card's sets onto the sigleton (instance) class. Because recursive inclusion will instantiate a new formatter instance for each included card, the set based views will get loaded on each sub-formatter in turn.
Web of Applications
To this point we have only mentioned the web to talk about context for the development of the cards model, and to note the lack of dependencies on rails outside of the model classes. A few more dependencies come about in formatters and views that are a bit web-specific and need to use rails route helpers and the like. In other words, we can completely separate this universal content model (cards) from the applications that might use them. Any dependence on web support is because the modules are implementing a formatter for a web application. Maybe in stardard mods, but not core.
We are very close to achieving this separation in code by way of an independant gem for cards, and other gems that make it into a deployable application. Starting with the card gem and its dependencies, all it takes to make this a rails engine is some routes and a single controller with about 300 lines of code and a few hundred more of config and glue code. There is a bunch of other stuff to support the application in the form of assets like javascript libraries and images as well as some generator commands that extend rails generators. There is more work to be done to complete support for common use cases for card based applications. Now they support generating a particular card based web application called wagn, but we also want to support rails developers who want to add cards to an existing rails application.
In practice, applications, web based and otherwise will be using groups of components packaged in different ways. Ruby and rails use bundler and gems, and front-end technology is more often based on node and other packaging and dependency management native to javascript. Now we add card mods to the mix and we need to think and design carefully the best ways to configure card based application.
Whether you are primarily using web protocols, or doing something else, the paradigm of a networked web of applications is well supported by the card model.
The Web Service Model
Although we are working pretty hard to package the card model independant from any web application, the model is designed to integrate with web applications. By keeping it independant, the architect can choose how and where to integrate it, and the application will not suffer unnecessary bloat by referencing components that are not always needed. Now we can begin to explore ways we can build systems out of webs of services and how the web service model is a great way to do that.
There is a simple idea about a class of web service abstractions, ReST for Representational State Transfer. In this abstraction, a web address names an object and the HTTP actions, POST, GET, PUT and DELETE can be mapped directly onto lifecycle actions on the persistant object named by the URL with the data represented in the action payload. With formatters, cards can be represented as serialized objects in one or more formats. Then some other system can modify that representation and send an updated version back where it is parsed and applied to the stored data. Interacting with a browser application via ajax isn't really fundamentally different than two networked applications interacting over a shared API. Either way we are transfering state of data objects using a networked representation.
Any given deck of cards can be assigned a base URI, either just a host, or a host and a path prefix, and it becomes a networked resource using the card's namespace as directly mapped onto URI paths. There is already a large body of industry practice around abstracting a web request interface within and between systems. In ruby and other languages there are the Rack specifications to pass web requests and responses between middleware components. They might be different components all using the rack gem in ruby. Rack is part of the internal routing of requests in rails, so it comes for free with rails. The Rack protocol is very close to how you would arrange to spawn a process to service a request passing in environment variables and returning some headers and an output stream with the body of the response. So, you could have middleware that is in-process and passing ruby objects, or middleware that is a network of service processes all using the same web-based abstraction, but maybe not over a network protocol at all.
By using a shared abstraction based on web protocols throughout the system we don't expect to have trouble at the interfaces. Systems architects sometimes talk about impedance mismatch between the components of a system using different technologies that are not well matched. A classic example is the difference between database tables and a group of interconnected objects of arbitrary type. Abstractions like Object-relational Mapping (ORM) as implemented in active record implementations provide one way to deal with this mismatch. The card abstraction is all about reducing this impedance mismatch between the storage in active record tables and web-based APIs. Names are well matched to web name spaces and content is well matched to chunks of text and markup that make up web pages. Formatters provide a range of external representations of the data that can be easily matched either to an external application or output channel, or to the input parsing for ReST actions to create and update cards.
SOA or Micro Services
Micro services, latest buzzword or important new methodology. Read a little below the surface and we see that this move is connected to a deeper trend of decentralized architectures. The earlier and clearly related idea of Service Oriented Architecture (SOA) led to a lot of designs where the service APIs were overdesigned, fragil and managed by a central authority. The move towards decentralization acknowledges that knowledge and innovation are best when they can be created and maintained anywhere in a networked heterogenious system. The more flexible your APIs are, the more able to evolve and to maintain connectivity with a changing network of services.
If you want to call this idea micro services, that's fine with me, but let us also declare it as a set of properties that all services should strive towards in order to be a well designed service. We can see these principles in operation in the design of decko as a platform. The first card based application, Wagn, is networked by virtue of being a web-app, but that's about it. The following chapters begin to explore the vision of card based platforms generally. As those chapters unfold, you will see how good design in the platform makes it easy, almost trivial after a little practice, to deploy services and network them according to good design practice in our increasingly distributed networks of innovation.
The Big Picture
This project is situated within the history of software development. The full sweep of this history has a ways to go to get to a full century unless you want to go back to Ada Lovelace devising algorythms for Babage's never build Analytical Engine. Longer than a career but beyond the lifetime of current practitioners. The birth of ruby and rails is very recent and still in very active development. The Internet, the Web and web applications are only twenty years old, is it surprising that we are discovering and evolving important core patterns? Component technologies come onto the scene as open source projects that introduced new technologies and were then adopted widely by software developers. This adoption is driven by desires to build well engineered solutions that will not be easily obsoleted by the next wave of change, and through a marketplace of competing solutions. In that marketplace, open source solutions have some superior properties when choosing for technological staying power. When a commercial vendor owns components that are necessary to build solutions, your code can be made obsolete by the whim or fortunes of that vendor. This kind of lock-in may be good for a vendor's bottom line, but it is never good for the projects that depend on it.
By strategically connecting these core technologies, the card model can become the bridge between the technological platform and a general content/business model. We can do any kind model logic and presentation in easy to write modules that are controlled by data rules, and we have a platform to easily implement web-based service oriented components as well as the means to combine a network of these components into a system. Emerging enterprise architectures are now built with mostly or only this type of component. If you can also build more of the components with cards, then you have both a deeper platform for sharing components, and components that closer to the space where users and designers interact. They can better describe and deliver features to use, and systems architects can pay attention to systems and operations issues. The more systems share this infrastructure, the more effort gets invested at both ends of this scale.
We will begin to see how this will work in the next chapters. We will be able to build out networked support services that handle the publishing and distribution of card modules that uses cards as its underlying technology. You'll be able build modules and content on different systems and only move the components onto production systems when they are released or published.