NoFlo

Quite a bit of time has passed since my two years of NoFlo post, and it is time to take another look at the state of the NoFlo ecosystem. To start with the basics, NoFlo is a JavaScript implementation of Flow-Based Programming: In computer ... [More] programming, flow-based programming (FBP) is a programming paradigm that defines applications as networks of “black box” processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally. FBP is thus naturally component-oriented. With NoFlo software is built by creating graphs that contain reusable components and define the program logic by determining how these components talk to each other. I started the NoFlo open source project six years ago in Mountain View, California. My aim was to improve the JavaScript programming experience by bringing the FBP paradigm to the ecosystem. At the time the focus was largely on web API servers and extract, transform, load (ETL) programs, but the scope has since expanded quite a bit: NoFlo is not a web framework or a UI toolkit. It is a way to coordinate and reorganize data flow in any JavaScript application. As such, it can be used for whatever purpose JavaScript can be used for. We know of NoFlo being used for anything from building web servers and build tools, to coordinating events inside GUI applications, driving robots, or building Internet-connected art installations. Flowhub Four years ago I wrote how UI was the missing part of NoFlo. Later the same year we launched a Kickstarter campaign to fix this. Our promise was to design a new way to build software & manage complexity - a visual development environment for all. This was wildly successful, being at the time the 5th highest funded software crowdfunding campaign. The result — Flowhub — was released to the public in 2014. Big thanks to all of our backers! Here is how Fast Company wrote about NoFlo: If NoFlo succeeds, it could herald a new paradigm of web programming. Imagine a world where anyone can understand and build web applications, and developers can focus on programming efficient components to be put to work by this new class of application architects. In a way, this is the same promise as the “learn to code” movement, which wants to teach everyone to be a programmer. Just without the programming. With Flowhub you can manage full NoFlo projects in your browser. This includes writing components in JavaScript or CoffeeScript, editing graphs and subgraphs, running and introspecting the software, and creating unit tests. You can keep your project in sync with the GitHub integration. Celebrating six years of NoFlo Earlier this year we incorporated Flowhub in Germany. Now, to celebrate six years of NoFlo we’re offering a perpetual 30% discount on Flowhub plans. To lock in the discount, subscribe to a Flowhub plan before June 12th 2017 using the code noflo6. Ecosystem While NoFlo itself has by no means taken over the world yet, the overall ecosystem it is part of is looking very healthy. Sure, JavaScript fatigue is real, but at the same time it has gone through a pretty dramatic expansion. JavaScript As I wrote around the time I started NoFlo, JavaScript has indeed become a universal runtime. It is used on web browsers, server-side, as well as for building mobile and desktop applications. And with NoFlo you can target all those platforms with a single programming model and toolchain. The de-facto standard for sharing JavaScript libraries — NPM has become the most popular software repository for open source modules. Apart from the hundreds of thousands of other packages, you can also get prebuild NoFlo components from NPM to cover almost any use case. Dataflow After a long period of semi-obscurity, our Kickstarter campaign greatly increased the awareness in FBP and dataflow programming. Several open source projects expanded the reach of FBP to other platforms, like MicroFlo to microcontroller programming, or PhpFlo to data conversion pipelines in PHP. To support more of these with common tooling, we standardized the FBP protocol that allows IDEs like Flowhub manage flow-based programs across different runtimes. Dataflow also saw uptake in the bigger industry. Facebook’s Flux architecture brought flow-based programming to reactive web applications. Google’s TensorFlow made dataflow the way to build machine learning applications. And Google’s Cloud Dataflow uses these techniques for stream processing. Tooling for flow-based programming One big area of focus for us has been improving the tooling around NoFlo, as well as the other FBP systems. The FBP protocol has been a big enabler for both building better tools, and for collaboration between different FBP and dataflow systems. Here are some of the tools currently available for NoFlo developers: Flowhub — browser-based visual programming IDE for NoFlo and other flow-based systems noflo-nodejs — command-line interface for running NoFlo programs on Node.js noflo-browser-app — template for building browser applications in NoFLo MsgFlo — for running NoFlo and other FBP runtimes as a distributed system fbp-spec — data-driven tests for NoFlo and other FBP environments flowtrace — tool for retroactive debugging of NoFlo programs. Supports visual replay with Flowhub NoFlo 0.8 NoFlo 0.8, released in March this year is probably our most important release so far. It introduced a new component API and greatly clarified the component and network lifecycle. With this release, it is easier than ever to build well-behaved NoFlo components and to deal with the mixture of asynchronous and synchronous data processing. It also brings NoFlo a lot closer to the classical FBP concepts. As part of the release process, we also fully overhauled the NoFlo documentation and wrote a new data transformations tutorial project. To find out more about NoFlo 0.8, watch my recent NoFlo talk from Berlin Node.js meetup: Road to 1.0 In addition to providing lots of new APIs and functionality, NoFlo 0.8 also acts as the transitional release as we head towards the big 1.0 release. In this version we marked many old APIs as deprecated. NoFlo 1.0 will essentially be 0.8, but with all the deprecated APIs removed. If you haven’t done so already, now is a good time to upgrade your NoFlo projects to 0.8 and make sure everything runs without deprecation warnings. We intend to release NoFlo 1.0 later this summer once more of our open source component libraries have been updated to utilize the new features. [Less]

Quite a bit of time has passed since my two years of NoFlo post, and it is time to take another look at the state of the NoFlo ecosystem. To start with the basics, NoFlo is a JavaScript implementation of Flow-Based Programming: In computer ... [More] programming, flow-based programming (FBP) is a programming paradigm that defines applications as networks of “black box” processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally. FBP is thus naturally component-oriented. With NoFlo software is built by creating graphs that contain reusable components and define the program logic by determining how these components talk to each other. I started the NoFlo open source project six years ago in Mountain View, California. My aim was to improve the JavaScript programming experience by bringing the FBP paradigm to the ecosystem. At the time the focus was largely on web API servers and extract, transform, load (ETL) programs, but the scope has since expanded quite a bit: NoFlo is not a web framework or a UI toolkit. It is a way to coordinate and reorganize data flow in any JavaScript application. As such, it can be used for whatever purpose JavaScript can be used for. We know of NoFlo being used for anything from building web servers and build tools, to coordinating events inside GUI applications, driving robots, or building Internet-connected art installations. Flowhub Four years ago I wrote how UI was the missing part of NoFlo. Later the same year we launched a Kickstarter campaign to fix this. Our promise was to design a new way to build software & manage complexity - a visual development environment for all. This was wildly successful, being at the time the 5th highest funded software crowdfunding campaign. The result — Flowhub — was released to the public in 2014. Big thanks to all of our backers! Here is how Fast Company wrote about NoFlo: If NoFlo succeeds, it could herald a new paradigm of web programming. Imagine a world where anyone can understand and build web applications, and developers can focus on programming efficient components to be put to work by this new class of application architects. In a way, this is the same promise as the “learn to code” movement, which wants to teach everyone to be a programmer. Just without the programming. With Flowhub you can manage full NoFlo projects in your browser. This includes writing components in JavaScript or CoffeeScript, editing graphs and subgraphs, running and introspecting the software, and creating unit tests. You can keep your project in sync with the GitHub integration. Celebrating six years of NoFlo Earlier this year we incorporated Flowhub in Germany. Now, to celebrate six years of NoFlo we’re offering a perpetual 30% discount on Flowhub plans. To lock in the discount, subscribe to a Flowhub plan before June 12th 2017 using the code noflo6. Ecosystem While NoFlo itself has by no means taken over the world yet, the overall ecosystem it is part of is looking very healthy. Sure, JavaScript fatigue is real, but at the same time it has gone through a pretty dramatic expansion. JavaScript As I wrote around the time I started NoFlo, JavaScript has indeed become a universal runtime. It is used on web browsers, server-side, as well as for building mobile and desktop applications. And with NoFlo you can target all those platforms with a single programming model and toolchain. The de-facto standard for sharing JavaScript libraries — NPM has become the most popular software repository for open source modules. Apart from the hundreds of thousands of other packages, you can also get prebuild NoFlo components from NPM to cover almost any use case. Dataflow After a long period of semi-obscurity, our Kickstarter campaign greatly increased the awareness in FBP and dataflow programming. Several open source projects expanded the reach of FBP to other platforms, like MicroFlo to microcontroller programming, or PhpFlo to data conversion pipelines in PHP. To support more of these with common tooling, we standardized the FBP protocol that allows IDEs like Flowhub manage flow-based programs across different runtimes. Dataflow also saw uptake in the bigger industry. Facebook’s Flux architecture brought flow-based programming to reactive web applications. Google’s TensorFlow made dataflow the way to build machine learning applications. And Google’s Cloud Dataflow uses these techniques for stream processing. Tooling for flow-based programming One big area of focus for us has been improving the tooling around NoFlo, as well as the other FBP systems. The FBP protocol has been a big enabler for both building better tools, and for collaboration between different FBP and dataflow systems. Here are some of the tools currently available for NoFlo developers: Flowhub — browser-based visual programming IDE for NoFlo and other flow-based systems noflo-nodejs — command-line interface for running NoFlo programs on Node.js noflo-browser-app — template for building browser applications in NoFLo MsgFlo — for running NoFlo and other FBP runtimes as a distributed system fbp-spec — data-driven tests for NoFlo and other FBP environments flowtrace — tool for retroactive debugging of NoFlo programs. Supports visual replay with Flowhub NoFlo 0.8 NoFlo 0.8, released in March this year is probably our most important release so far. It introduced a new component API and greatly clarified the component and network lifecycle. With this release, it is easier than ever to build well-behaved NoFlo components and to deal with the mixture of asynchronous and synchronous data processing. It also brings NoFlo a lot closer to the classical FBP concepts. As part of the release process, we also fully overhauled the NoFlo documentation and wrote a new data transformations tutorial project. To find out more about NoFlo 0.8, watch my recent NoFlo talk from Berlin Node.js meetup: Road to 1.0 In addition to providing lots of new APIs and functionality, NoFlo 0.8 also acts as the transitional release as we head towards the big 1.0 release. In this version we marked many old APIs as deprecated. NoFlo 1.0 will essentially be 0.8, but with all the deprecated APIs removed. If you haven’t done so already, now is a good time to upgrade your NoFlo projects to 0.8 and make sure everything runs without deprecation warnings. We intend to release NoFlo 1.0 later this summer once more of our open source component libraries have been updated to utilize the new features. [Less]

Quite a bit of time has passed since my two years of NoFlo post, and it is time to take another look at the state of the NoFlo ecosystem. To start with the basics, NoFlo is a JavaScript implementation of Flow-Based Programming: In computer ... [More] programming, flow-based programming (FBP) is a programming paradigm that defines applications as networks of “black box” processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally. FBP is thus naturally component-oriented. With NoFlo software is built by creating graphs that contain reusable components and define the program logic by determining how these components talk to each other. I started the NoFlo open source project six years ago in Mountain View, California. My aim was to improve the JavaScript programming experience by bringing the FBP paradigm to the ecosystem. At the time the focus was largely on web API servers and extract, transform, load (ETL) programs, but the scope has since expanded quite a bit: NoFlo is not a web framework or a UI toolkit. It is a way to coordinate and reorganize data flow in any JavaScript application. As such, it can be used for whatever purpose JavaScript can be used for. We know of NoFlo being used for anything from building web servers and build tools, to coordinating events inside GUI applications, driving robots, or building Internet-connected art installations. Flowhub Four years ago I wrote how UI was the missing part of NoFlo. Later the same year we launched a Kickstarter campaign to fix this. Our promise was to design a new way to build software & manage complexity - a visual development environment for all. This was wildly successful, being at the time the 5th highest funded software crowdfunding campaign. The result — Flowhub — was released to the public in 2014. Big thanks to all of our backers! Here is how Fast Company wrote about NoFlo: If NoFlo succeeds, it could herald a new paradigm of web programming. Imagine a world where anyone can understand and build web applications, and developers can focus on programming efficient components to be put to work by this new class of application architects. In a way, this is the same promise as the “learn to code” movement, which wants to teach everyone to be a programmer. Just without the programming. With Flowhub you can manage full NoFlo projects in your browser. This includes writing components in JavaScript or CoffeeScript, editing graphs and subgraphs, running and introspecting the software, and creating unit tests. You can keep your project in sync with the GitHub integration. Celebrating six years of NoFlo Earlier this year we incorporated Flowhub in Germany. Now, to celebrate six years of NoFlo we’re offering a perpetual 30% discount on Flowhub plans. To lock in the discount, subscribe to a Flowhub plan before June 12th 2017 using the code noflo6. Ecosystem While NoFlo itself has by no means taken over the world yet, the overall ecosystem it is part of is looking very healthy. Sure, JavaScript fatigue is real, but at the same time it has gone through a pretty dramatic expansion. JavaScript As I wrote around the time I started NoFlo, JavaScript has indeed become a universal runtime. It is used on web browsers, server-side, as well as for building mobile and desktop applications. And with NoFlo you can target all those platforms with a single programming model and toolchain. The de-facto standard for sharing JavaScript libraries — NPM has become the most popular software repository for open source modules. Apart from the hundreds of thousands of other packages, you can also get prebuild NoFlo components from NPM to cover almost any use case. Dataflow After a long period of semi-obscurity, our Kickstarter campaign greatly increased the awareness in FBP and dataflow programming. Several open source projects expanded the reach of FBP to other platforms, like MicroFlo to microcontroller programming, or PhpFlo to data conversion pipelines in PHP. To support more of these with common tooling, we standardized the FBP protocol that allows IDEs like Flowhub manage flow-based programs across different runtimes. Dataflow also saw uptake in the bigger industry. Facebook’s Flux architecture brought flow-based programming to reactive web applications. Google’s TensorFlow made dataflow the way to build machine learning applications. And Google’s Cloud Dataflow uses these techniques for stream processing. Tooling for flow-based programming One big area of focus for us has been improving the tooling around NoFlo, as well as the other FBP systems. The FBP protocol has been a big enabler for both building better tools, and for collaboration between different FBP and dataflow systems. Here are some of the tools currently available for NoFlo developers: Flowhub — browser-based visual programming IDE for NoFlo and other flow-based systems noflo-nodejs — command-line interface for running NoFlo programs on Node.js noflo-browser-app — template for building browser applications in NoFLo MsgFlo — for running NoFlo and other FBP runtimes as a distributed system fbp-spec — data-driven tests for NoFlo and other FBP environments flowtrace — tool for retroactive debugging of NoFlo programs. Supports visual replay with Flowhub NoFlo 0.8 NoFlo 0.8, released in March this year is probably our most important release so far. It introduced a new component API and greatly clarified the component and network lifecycle. With this release, it is easier than ever to build well-behaved NoFlo components and to deal with the mixture of asynchronous and synchronous data processing. It also brings NoFlo a lot closer to the classical FBP concepts. As part of the release process, we also fully overhauled the NoFlo documentation and wrote a new data transformations tutorial project. To find out more about NoFlo 0.8, watch my recent NoFlo talk from Berlin Node.js meetup: Road to 1.0 In addition to providing lots of new APIs and functionality, NoFlo 0.8 also acts as the transitional release as we head towards the big 1.0 release. In this version we marked many old APIs as deprecated. NoFlo 1.0 will essentially be 0.8, but with all the deprecated APIs removed. If you haven’t done so already, now is a good time to upgrade your NoFlo projects to 0.8 and make sure everything runs without deprecation warnings. We intend to release NoFlo 1.0 later this summer once more of our open source component libraries have been updated to utilize the new features. [Less]

Quite a bit of time has passed since my two years of NoFlo post, and it is time to take another look at the state of the NoFlo ecosystem. To start with the basics, NoFlo is a JavaScript implementation of Flow-Based Programming: In computer ... [More] programming, flow-based programming (FBP) is a programming paradigm that defines applications as networks of “black box” processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes can be reconnected endlessly to form different applications without having to be changed internally. FBP is thus naturally component-oriented. With NoFlo software is built by creating graphs that contain reusable components and define the program logic by determining how these components talk to each other. I started the NoFlo open source project six years ago in Mountain View, California. My aim was to improve the JavaScript programming experience by bringing the FBP paradigm to the ecosystem. At the time the focus was largely on web API servers and extract, transform, load (ETL) programs, but the scope has since expanded quite a bit: NoFlo is not a web framework or a UI toolkit. It is a way to coordinate and reorganize data flow in any JavaScript application. As such, it can be used for whatever purpose JavaScript can be used for. We know of NoFlo being used for anything from building web servers and build tools, to coordinating events inside GUI applications, driving robots, or building Internet-connected art installations. Flowhub Four years ago I wrote how UI was the missing part of NoFlo. Later the same year we launched a Kickstarter campaign to fix this. Our promise was to design a new way to build software & manage complexity - a visual development environment for all. This was wildly successful, being at the time the 5th highest funded software crowdfunding campaign. The result — Flowhub — was released to the public in 2014. Big thanks to all of our backers! Here is how Fast Company wrote about NoFlo: If NoFlo succeeds, it could herald a new paradigm of web programming. Imagine a world where anyone can understand and build web applications, and developers can focus on programming efficient components to be put to work by this new class of application architects. In a way, this is the same promise as the “learn to code” movement, which wants to teach everyone to be a programmer. Just without the programming. With Flowhub you can manage full NoFlo projects in your browser. This includes writing components in JavaScript or CoffeeScript, editing graphs and subgraphs, running and introspecting the software, and creating unit tests. You can keep your project in sync with the GitHub integration. Celebrating six years of NoFlo Earlier this year we incorporated Flowhub in Germany. Now, to celebrate six years of NoFlo we’re offering a perpetual 30% discount on Flowhub plans. To lock in the discount, subscribe to a Flowhub plan before June 12th 2017 using the code noflo6. Ecosystem While NoFlo itself has by no means taken over the world yet, the overall ecosystem it is part of is looking very healthy. Sure, JavaScript fatigue is real, but at the same time it has gone through a pretty dramatic expansion. JavaScript As I wrote around the time I started NoFlo, JavaScript has indeed become a universal runtime. It is used on web browsers, server-side, as well as for building mobile and desktop applications. And with NoFlo you can target all those platforms with a single programming model and toolchain. The de-facto standard for sharing JavaScript libraries — NPM has become the most popular software repository for open source modules. Apart from the hundreds of thousands of other packages, you can also get prebuild NoFlo components from NPM to cover almost any use case. Dataflow After a long period of semi-obscurity, our Kickstarter campaign greatly increased the awareness in FBP and dataflow programming. Several open source projects expanded the reach of FBP to other platforms, like MicroFlo to microcontroller programming, or PhpFlo to data conversion pipelines in PHP. To support more of these with common tooling, we standardized the FBP protocol that allows IDEs like Flowhub manage flow-based programs across different runtimes. Dataflow also saw uptake in the bigger industry. Facebook’s Flux architecture brought flow-based programming to reactive web applications. Google’s TensorFlow made dataflow the way to build machine learning applications. And Google’s Cloud Dataflow uses these techniques for stream processing. Tooling for flow-based programming One big area of focus for us has been improving the tooling around NoFlo, as well as the other FBP systems. The FBP protocol has been a big enabler for both building better tools, and for collaboration between different FBP and dataflow systems. Here are some of the tools currently available for NoFlo developers: Flowhub — browser-based visual programming IDE for NoFlo and other flow-based systems noflo-nodejs — command-line interface for running NoFlo programs on Node.js noflo-browser-app — template for building browser applications in NoFLo MsgFlo — for running NoFlo and other FBP runtimes as a distributed system fbp-spec — data-driven tests for NoFlo and other FBP environments flowtrace — tool for retroactive debugging of NoFlo programs. Supports visual replay with Flowhub NoFlo 0.8 NoFlo 0.8, released in March this year is probably our most important release so far. It introduced a new component API and greatly clarified the component and network lifecycle. With this release, it is easier than ever to build well-behaved NoFlo components and to deal with the mixture of asynchronous and synchronous data processing. It also brings NoFlo a lot closer to the classical FBP concepts. As part of the release process, we also fully overhauled the NoFlo documentation and wrote a new data transformations tutorial project. To find out more about NoFlo 0.8, watch my recent NoFlo talk from Berlin Node.js meetup: Road to 1.0 In addition to providing lots of new APIs and functionality, NoFlo 0.8 also acts as the transitional release as we head towards the big 1.0 release. In this version we marked many old APIs as deprecated. NoFlo 1.0 will essentially be 0.8, but with all the deprecated APIs removed. If you haven’t done so already, now is a good time to upgrade your NoFlo projects to 0.8 and make sure everything runs without deprecation warnings. We intend to release NoFlo 1.0 later this summer once more of our open source component libraries have been updated to utilize the new features. [Less]

In Flowhub you can create and edit full flow-based programming projects. The live mode enables introspecting running FBP systems. This weekend we rolled out Flowhub 0.19 which makes it easy to move between these modes. Live mode Flowhub’s live mode ... [More] is designed for making software introspectable — flow-based systems provide access to the graph and network state information, and tools like Flowhub then visualize it. Opening a system in live mode can be done with a URL that contains the connection details to the runtime. This means runtimes can make their live mode discoverable in many ways, from printing it at system start-up to a NFC tag or QR Code on a deployed device: In live mode, users can see the graph structure of the running software and the packets flowing through it. And with the right permissions, you can also navigate the possible sub-graphs and component source code. Edit as project To make changes to the running software, hit the Edit as project button. Flowhub will download the currently running software from the runtime and turn it into a project. In nutshell: Live mode provides a read-only view to a running system Project mode allows live programming a full project including its graphs and components Edit as project can be used to turn a live mode session into an editable project Projects can be synchronized with GitHub Edit as project also works with the example programs that ship with Flowhub. [Less]

In Flowhub you can create and edit full flow-based programming projects. The live mode enables introspecting running FBP systems. This weekend we rolled out Flowhub 0.19 which makes it easy to move between these modes. Live mode Flowhub’s live mode ... [More] is designed for making software introspectable — flow-based systems provide access to the graph and network state information, and tools like Flowhub then visualize it. Opening a system in live mode can be done with a URL that contains the connection details to the runtime. This means runtimes can make their live mode discoverable in many ways, from printing it at system start-up to a NFC tag or QR Code on a deployed device: In live mode, users can see the graph structure of the running software and the packets flowing through it. And with the right permissions, you can also navigate the possible sub-graphs and component source code. Edit as project To make changes to the running software, hit the Edit as project button. Flowhub will download the currently running software from the runtime and turn it into a project. In nutshell: Live mode provides a read-only view to a running system Project mode allows live programming a full project including its graphs and components Edit as project can be used to turn a live mode session into an editable project Projects can be synchronized with GitHub Edit as project also works with the example programs that ship with Flowhub. [Less]

In Flowhub you can create and edit full flow-based programming projects. The live mode enables introspecting running FBP systems. This weekend we rolled out Flowhub 0.19 which makes it easy to move between these modes. Live mode Flowhub’s live mode ... [More] is designed for making software introspectable — flow-based systems provide access to the graph and network state information, and tools like Flowhub then visualize it. Opening a system in live mode can be done with a URL that contains the connection details to the runtime. This means runtimes can make their live mode discoverable in many ways, from printing it at system start-up to a NFC tag or QR Code on a deployed device: In live mode, users can see the graph structure of the running software and the packets flowing through it. And with the right permissions, you can also navigate the possible sub-graphs and component source code. Edit as project To make changes to the running software, hit the Edit as project button. Flowhub will download the currently running software from the runtime and turn it into a project. In nutshell: Live mode provides a read-only view to a running system Project mode allows live programming a full project including its graphs and components Edit as project can be used to turn a live mode session into an editable project Projects can be synchronized with GitHub Edit as project also works with the example programs that ship with Flowhub. [Less]

Microservices — an architectural pattern we recommended in our 2012 International PHP Conference keynote — is pretty popular these days. There are many benefits to consider: Independent development and release lifecycle for each microservice ... [More] Ensuring clear API boundaries between systems Ability to use technologies most applicable for each area of a system In an ideal world, microservices are a realization of the Unix philosophy as applied to building internet services: writing programs that do one thing, and do it well; writing programs that work together. Just use a message queue However, when most people think about microservices, they think systems that communicate with each other using HTTP APIs. I think this is quite limited, and something that makes microservices a lot more fragile than they could be. Message queues provide a much better solution. This is mentioned in Martin Fowler’s Microservices article: The second approach in common use is messaging over a lightweight message bus. The infrastructure chosen is typically dumb (dumb as in acts as a message router only) - simple implementations such as RabbitMQ or ZeroMQ don’t do much more than provide a reliable asynchronous fabric - the smarts still live in the end points that are producing and consuming messages; in the services. When we were building The Grid, we went with an architecture heavily reliant on microservices communicating using message queues. This gave us several useful capabilities: Asynchronous processing If you have heavy back-end operations or peak load, a microservice might be swamped with work to be done. If your web server only needs to send work to a queue and not wait for result immediately, you have a lot more freedom on how to organize the work. Furthermore, you can split your service along different scalability characteristics — some systems may be network-bound, others CPU-bound etc Autoscaling Since the work to be performed by your microservices is kept in message queue, you can use the combination of the queue length and typical processing times to automatically determine how many instances of each service you need. Ideally you can use this to ensure that your processing times stay consistent regardless of how many users are on your system at a given time Dead lettering It is possible to configure RabbitMQ to place any failed operations into a dead letter queue. This gives you a full record of any failing operations, making it possible to inspect them manually, replay later, or produce new tests based on real-world failures Rate limiting Sometimes you’re dealing with external HTTP APIs that are rate limited. Once a microservice starts hitting rate limits, you can keep new requests in a queue until the limit lifts again In-flight updates A message queue can be configured so that non-acknowledged messages go back into the queue. This means that if one of your services crashes, or you deploy a new version of it, no work gets lost. When the service is back up again, it can pick up right where the previous instance left off Better tooling HTTP is something all backend developers are familiar with. With message queues you have to deal with some new concepts, and hence new tools are also needed. MsgFlo Client libraries for talking with common message queues exist for pretty much every language. However, this still means that you’ll have to handle things like message pre-fetch limits, acknowledging handled messages, and setting up queues yourself. And of course keeping track of what service talks to what can become a burden. We developed the MsgFlo tool to solve these problems. MsgFlo provides open source client libraries for bunch of different programming languages, providing a simple model to handle message-related workflow. To give you an overview of the dataflow between services, MsgFlo also provides a way to define the whole system as a flow-based programming graph. This means you’ll see the whole system visually, and can change connections between services directly from graphical tools like Flowhub. If you’re using Heroku, MsgFlo can also generate the Procfile for you, meaning that the services you’ve defined in your graph get automatically registered with the platform. GuvScale As mentioned above, queue-based microservices make autoscaling quite easy. If you’re on Heroku, then our GuvScale can be used to automate scaling operations for all of your background dynos. I wrote more about GuvScale in a recent blog post. Further reading If you’d like to explore using message queues for your services a bit more, here are couple of good articles: Event-driven Microservices Using RabbitMQ Microservice Communication with Queues There are also some MsgFlo example applications available: msgflo-example-imageresize is a web service for scaling images using autoscaled background workers imgflo-server is a web service that uses MsgFlo to send image processing jobs to background GEGL or NoFlo microservices [Less]

Microservices — an architectural pattern we recommended in our 2012 International PHP Conference keynote — is pretty popular these days. There are many benefits to consider: Independent development and release lifecycle for each microservice ... [More] Ensuring clear API boundaries between systems Ability to use technologies most applicable for each area of a system In an ideal world, microservices are a realization of the Unix philosophy as applied to building internet services: writing programs that do one thing, and do it well; writing programs that work together. Just use a message queue However, when most people think about microservices, they think systems that communicate with each other using HTTP APIs. I think this is quite limited, and something that makes microservices a lot more fragile than they could be. Message queues provide a much better solution. This is mentioned in Martin Fowler’s Microservices article: The second approach in common use is messaging over a lightweight message bus. The infrastructure chosen is typically dumb (dumb as in acts as a message router only) - simple implementations such as RabbitMQ or ZeroMQ don’t do much more than provide a reliable asynchronous fabric - the smarts still live in the end points that are producing and consuming messages; in the services. When we were building The Grid, we went with an architecture heavily reliant on microservices communicating using message queues. This gave us several useful capabilities: Asynchronous processing If you have heavy back-end operations or peak load, a microservice might be swamped with work to be done. If your web server only needs to send work to a queue and not wait for result immediately, you have a lot more freedom on how to organize the work. Furthermore, you can split your service along different scalability characteristics — some systems may be network-bound, others CPU-bound etc Autoscaling Since the work to be performed by your microservices is kept in message queue, you can use the combination of the queue length and typical processing times to automatically determine how many instances of each service you need. Ideally you can use this to ensure that your processing times stay consistent regardless of how many users are on your system at a given time Dead lettering It is possible to configure RabbitMQ to place any failed operations into a dead letter queue. This gives you a full record of any failing operations, making it possible to inspect them manually, replay later, or produce new tests based on real-world failures Rate limiting Sometimes you’re dealing with external HTTP APIs that are rate limited. Once a microservice starts hitting rate limits, you can keep new requests in a queue until the limit lifts again In-flight updates A message queue can be configured so that non-acknowledged messages go back into the queue. This means that if one of your services crashes, or you deploy a new version of it, no work gets lost. When the service is back up again, it can pick up right where the previous instance left off Better tooling HTTP is something all backend developers are familiar with. With message queues you have to deal with some new concepts, and hence new tools are also needed. MsgFlo Client libraries for talking with common message queues exist for pretty much every language. However, this still means that you’ll have to handle things like message pre-fetch limits, acknowledging handled messages, and setting up queues yourself. And of course keeping track of what service talks to what can become a burden. We developed the MsgFlo tool to solve these problems. MsgFlo provides open source client libraries for bunch of different programming languages, providing a simple model to handle message-related workflow. To give you an overview of the dataflow between services, MsgFlo also provides a way to define the whole system as a flow-based programming graph. This means you’ll see the whole system visually, and can change connections between services directly from graphical tools like Flowhub. If you’re using Heroku, MsgFlo can also generate the Procfile for you, meaning that the services you’ve defined in your graph get automatically registered with the platform. GuvScale As mentioned above, queue-based microservices make autoscaling quite easy. If you’re on Heroku, then our GuvScale can be used to automate scaling operations for all of your background dynos. I wrote more about GuvScale in a recent blog post. Further reading If you’d like to explore using message queues for your services a bit more, here are couple of good articles: Event-driven Microservices Using RabbitMQ Microservice Communication with Queues There are also some MsgFlo example applications available: msgflo-example-imageresize is a web service for scaling images using autoscaled background workers imgflo-server is a web service that uses MsgFlo to send image processing jobs to background GEGL or NoFlo microservices [Less]

Microservices — an architectural pattern we recommended in our 2012 International PHP Conference keynote — is pretty popular these days. There are many benefits to consider: Independent development and release lifecycle for each microservice ... [More] Ensuring clear API boundaries between systems Ability to use technologies most applicable for each area of a system In an ideal world, microservices are a realization of the Unix philosophy as applied to building internet services: writing programs that do one thing, and do it well; writing programs that work together. Just use a message queue However, when most people think about microservices, they think systems that communicate with each other using HTTP APIs. I think this is quite limited, and something that makes microservices a lot more fragile than they could be. Message queues provide a much better solution. This is mentioned in Martin Fowler’s Microservices article: The second approach in common use is messaging over a lightweight message bus. The infrastructure chosen is typically dumb (dumb as in acts as a message router only) - simple implementations such as RabbitMQ or ZeroMQ don’t do much more than provide a reliable asynchronous fabric - the smarts still live in the end points that are producing and consuming messages; in the services. When we were building The Grid, we went with an architecture heavily reliant on microservices communicating using message queues. This gave us several useful capabilities: Asynchronous processing If you have heavy back-end operations or peak load, a microservice might be swamped with work to be done. If your web server only needs to send work to a queue and not wait for result immediately, you have a lot more freedom on how to organize the work. Furthermore, you can split your service along different scalability characteristics — some systems may be network-bound, others CPU-bound etc Autoscaling Since the work to be performed by your microservices is kept in message queue, you can use the combination of the queue length and typical processing times to automatically determine how many instances of each service you need. Ideally you can use this to ensure that your processing times stay consistent regardless of how many users are on your system at a given time Dead lettering It is possible to configure RabbitMQ to place any failed operations into a dead letter queue. This gives you a full record of any failing operations, making it possible to inspect them manually, replay later, or produce new tests based on real-world failures Rate limiting Sometimes you’re dealing with external HTTP APIs that are rate limited. Once a microservice starts hitting rate limits, you can keep new requests in a queue until the limit lifts again In-flight updates A message queue can be configured so that non-acknowledged messages go back into the queue. This means that if one of your services crashes, or you deploy a new version of it, no work gets lost. When the service is back up again, it can pick up right where the previous instance left off Better tooling HTTP is something all backend developers are familiar with. With message queues you have to deal with some new concepts, and hence new tools are also needed. MsgFlo Client libraries for talking with common message queues exist for pretty much every language. However, this still means that you’ll have to handle things like message pre-fetch limits, acknowledging handled messages, and setting up queues yourself. And of course keeping track of what service talks to what can become a burden. We developed the MsgFlo tool to solve these problems. MsgFlo provides open source client libraries for bunch of different programming languages, providing a simple model to handle message-related workflow. To give you an overview of the dataflow between services, MsgFlo also provides a way to define the whole system as a flow-based programming graph. This means you’ll see the whole system visually, and can change connections between services directly from graphical tools like Flowhub. If you’re using Heroku, MsgFlo can also generate the Procfile for you, meaning that the services you’ve defined in your graph get automatically registered with the platform. Guv As mentioned above, queue-based microservices make autoscaling quite easy. If you’re on Heroku, then our Guv tool can be used to automate scaling operations for all of your background dynos. I wrote more about GuvScale in a recent blog post. Further reading If you’d like to explore using message queues for your services a bit more, here are couple of good articles: Event-driven Microservices Using RabbitMQ Microservice Communication with Queues There are also some MsgFlo example applications available: msgflo-example-imageresize is a web service for scaling images using autoscaled background workers imgflo-server is a web service that uses MsgFlo to send image processing jobs to background GEGL or NoFlo microservices [Less]