The Quiet Evolution of SOA

• April 8th, 2013
• shan

We’ve all found ourselves looking at an organization’s web services and commenting on how “It’s not really SOA”.  Maybe because the program still maintains point-to-point interfaces, or maybe the organization hasn’t put in place any form of governance, but for whatever reason, we declare that it simply isn’t comprehensive enough to be considered SOA.  That begs the question then: who is actually doing TRUE enterprise wide SOA?  Well… very few organizations.  Anne Manes famously declared that “SOA is dead” back in 2009.  So why is it that we still find ourselves evangelizing and building towards this vision?

The answer is that our understanding of what makes an SOA program successful has quietly evolved over the last few years.  Enterprise-wide re-platforming and re-architecture initiatives gave way to tactical adoption of SOA.  The success of SaaS and BPM adoption meant that organizations are implementing the principles of service orientation without explicitly calling it an SOA program.  And instead of trying to figure out just how to effectively measure SOA ROI at the enterprise level, much greater success has been found measuring the value created within a given portfolio and/or capability.

So while we Architects have not given up on the hope of achieving SOA utopia, we have become more realistic in our approach:

1. Identify a very specific problem to solve with an SOA approach, be it to reduce the time-to-market of a frequently changing business process, or to reduce the application footprint of a given line-of-business.
2. Demonstrate the value of SOA by successfully solving that problem.
3. Rinse and repeat.

At the end of the day, any plans for enterprise level SOA can only be built a critical mass of successful self-sustaining SOA capabilities/portfolios.

Fractal Governance

• March 28th, 2013
• shan

SOA landscapes today look very much like fractals.  An organization may have several internal capabilities presented as reusable services that connect to each other.  It may even connect to 3rd party and/or cloud based services.  But if you drill down into each of these services, you’ll likely see a composite application that is made up of several finer grained services interconnected together.  And as a math geek, I am naturally curious about all things related to fractals.

In fact the fractal pattern appears in almost anything that’s responsible for connecting things together: highway and road systems, power grids, the internet… the list goes on.  In all of these systems, there exists a hierarchical system of management and governance to regulate its functionality.  Each country, for example, have national standards and regulatory bodies that define how power is to be exchanged, managed, and consumed.  At the regional levels, there are additional standards and regulatory bodies that deal with region-specific decisions such as how much power to generate, pricing, and what equipment is to be installed where.  Similar structures are true for transportation and telecommunications.  So why is it that most organizations see SOA governance as an all-encompassing enterprise wide responsibility?

The interaction requirements and lifecycle characteristics of enterprise level composite services or business processes are very different from those of a utility service.  To paint the entire enterprise service landscape with an uniform set of standards and processes will either result in a high number of exceptions or a lowest common denominator scenario.  To be effective, an organization’s SOA governance model must match its SOA deployment model.  The governance model must exist not just at the top, but at a granularity that matches how the services are being deployed and managed.  Service Portfolio Managers, then are not just another role within the governance model, but micro versions of governance domains themselves.  Service Portfolio Managers must be allowed to define their own standards and processes that are appropriate for the specific services that they’re responsible for.  The SOA governance model for the enterprise must consider what standards and processes are appropriate for all services, which are appropriate for only the ones being consumed across the enterprise, and which ones should be left up to the Portfolios to govern themselves.

Brief Intro to SOA development

• March 12th, 2013
• Andre Racicot

As a hacker, I get a huge rush out of solving problems. It’s like a game for me, with the goal being always to find bigger and better dragons to slay…

Unfortunately, we all know it’s not always the case in our line of work! Usually, it’s some interesting problem hidden among piles of scut work. You know… That repetitive stuff that, while simple and pays the bills, often makes you wonder if all the good problems have been solved. That goes double if your client happens to be a big enterprise.

If you work for a company or have a client that has (or needs) a large IT Infrastructure, you’ve probably heard of Service-Oriented Architecture (SOA) before. You might have attended a meeting where a consultant had recommended such a system, or even had read up on it yourself. You might have noticed that it involves a bunch of mundane tasks like writing xml transforms, and WSDLs. I’ve been lucky enough to have been given a brief introduction to Oracle SOA Suite, and the accompanying JDeveloper tool that takes care of the tedious stuff, and lets you focus on what matters: Writing awesome code to solve interesting problems and look like a hero in your clients’ eyes. Of course, Oracle/JDeveloper is only one of the many packages out there for quickly deploying SOA apps. Other examples include TIBCO , IBM Websphere, and OpenESB.

This post isn’t some elaborate HowTo or Tutorial, but a brief intro to SOA from a programmer’s perspective. To demonstrate how using the right tools, you can have rich, reusable services up and running in a very short amount of time! The potential for this stuff is huge, and like it or not, it’s here to stay. Might as well go whole hog!

Zero to Hero in 30 minutes or less.

The following is just a quick intro at what can be done using jDeveloper. Like I said before, this is just one way of skinning the SOA cat. There are tons of other tools out there, some of them are even open-source. I’ll be posting some more elaborate tutorials, demos and HowTos later on.

The Dreaded XML Transform

XML transforms is possibly some of most boring work out there. Most of the time, you’re mapping an input from a web service to an object you’re then going to pass on to a database layer or some other service. Using jDeveloper, you can skip writing XSDs, and XSLTs, and just draw a few quick diagrams.

Business Process Execution Language

BPEL is how more complex SOA processes can be implemented. Usually by means of long boring XML definitions. Again, jDeveloper abstracts these in simple to read diagrams, resembling flow charts

Putting it all together

The transforms and BPEL modules you create using jDeveloper are then used by a composite. This is a high-level definition of what comes in from the outside, and where it gets routed. Every module is configurable and can route data in and/or out based on defined conditions (more on this in future posts)

In Closing

This is just a (very) brief intro on what using the right tools can do to make your mundane development tasks way easier and faster to complete. Your mileage may vary depending on which package/platform you use, but they all can shave hours or even days off your service development cycle. And after all, that’s what matters when you want to get back to hacking that brilliant solution to that interesting problem you haven’t had time to focus on!

eIDverifier by Equifax: The Implementation

• February 15th, 2013
• Marco Flores

At Tindr, we often get asked to fix broken software projects.  Typically a lot of bad things have happened along the way – underskilled development teams, a lack of discipline around code and testing, poor project management and the like.

In this particular project, we inherited the code with some functionality in what we were told was “ready for production” condition.   Part of that “complete” software was the code for eIDverifier, an Equifax product for authentication of personal information.  As far as we knew, that part of the project was working and so we focussed our efforts on the implementation of a new registration process plus some other minor and not so minor changes.

Houston, we have a problem!!!

It was during testing that we notices that all of our results from eIDverifier by Equifax were coming negative. The test cases should have been working but they simply weren’t.

Ok, so let’s try again, mmm… not the expected results. The documentation was not very helpful on this particular scenario — in fact is missing.  All this testing was done on a Sunday afternoon, so we had to wait until the next morning to contact Equifax to get some answers. The tip we received, although vague, gave us an idea of where to start looking.

After expending some time looking at the logs, the code we inherited and the actual API, we found where the problem was.  It turned out that the functionality never had a chance of working at all.   A few more minutes modifying the code and we were ready to test again. Et voila, we finally had it working correctly.

The fact was the code we inherited never actually worked, the connection and responses back from the API all were looking good but it was missing key pieces of the process, most of them in the Equifax documentation.

Unfortunately because of confidentiality rules, we cannot give any details of the problem or the solution, in fact you are strongly advised to contact Equifax for support.  But if you need a team to work on your app with some experience working with the API, you can contact us and we can get your service running in no time.

Play 2.1, Slick and JSON

• February 4th, 2013
• dave

As an exercise in familiarizing myself with Play Framework 2.x Scala, I am implementing a lightweight CMS for a  dynamic corporate website.

I was curious about what the new 2.x release offers with regards to JSON reads and writes as well as Slick for data access.

In this post, I'll present some code detailing how I used  Slick and the new macro enabled JSON utilities in Play 2.1 in order to implement a simple CMS.

 

 

The Model:

 

 

I prefer to use a case class as a parameter to Slick's table object, as opposed to tuples, as the case class will lead to trivial JSON read/writes later on.

I use case classes to provide different representations of the base model for the CMS: Article

The above case class represents the complete model (all colums in the database table). The class Article is the parameter to the table:

~ is a projection operator. It is a member of the class Column.

Here are some examples of basic CRUD with Slick. In these examples, I am making use of the various other case classes even though Article is still the class mapped to the Table class

Create:

 

Update:

In Slick, you describe the "Projection" (a set of table rows and columns) you wish to query or update by passing a Column projection using ~ to map(). This is equivalent the SELECT in SQL

Describe the particular rows you are interested in with filter(). This is equivalent to WHERE in SQL

As you can see, we are making use of scala's Try[T] introduced in Scala 2.10 and pattern matching from the calling code:

 

Reading from a JSON request is pretty simple, done the ArticleCreate implicit, the Play macro takes care of the rest. json.validate() takes the implicit and returns the monadicJsResult[T]

 

 

I'm also making use of a sealed trait as an enum:

 

In this case I need to include the implicit conversion:

 

I'm still figuring a lot of this stuff out, and I'm sure I've missed much, but getting a JSON request into the DB seemed pretty straightforward!

 

 

I prefer to use a case class as a parameter to Slick's table object, as opposed to tuples, as the case class will lead to trivial JSON read/writes later on.

I use case classes to provide different representations of the base model for the CMS: Article

The above case class represents the complete model (all colums in the database table). The class Article is the parameter to the table:

~ is a projection operator. It is a member of the class Column.

Here are some examples of basic CRUD with Slick. In these examples, I am making use of the various other case classes even though Article is still the class mapped to the Table class

Create:

 

Update:

In Slick, you describe the "Projection" (a set of table rows and columns) you wish to query or update by passing a Column projection using ~ to map(). This is equivalent the SELECT in SQL

Describe the particular rows you are interested in with filter(). This is equivalent to WHERE in SQL

As you can see, we are making use of scala's Try[T] introduced in Scala 2.10 and pattern matching from the calling code:

 

Reading from a JSON request is pretty simple, done the ArticleCreate implicit, the Play macro takes care of the rest. json.validate() takes the implicit and returns the monadicJsResult[T]

 

 

I'm also making use of a sealed trait as an enum:

 

In this case I need to include the implicit conversion:

 

I'm still figuring a lot of this stuff out, and I'm sure I've missed much, but getting a JSON request into the DB seemed pretty straightforward!

 

 

 

References:

 

 

• Learning Slick
• JSON Inception (Based on Scala 2.10 Macros)

Tindr Team Survives First Winter Camping Trip

• January 21st, 2013
• Mike Kelland

Tindr is proud to announce that members of the team have survived its first winter camping trip.  When the trip met with disaster in the form of unexpected rain that soaked sleeping bags and created what one member of the trip called a “pool” in the shelter, the team came together to quickly pack up and march out of the campsite. The trip started out with a whiteout and a slow drive behind snow plows to the put in point west of Ottawa.  After a long and arduous hike cross country, including some steep climbs and bushwacking when the trail was lost, we arrived at the target lake and put down fishing lines.  Lines down, we concentrated on building the shelter.  Based on past experience, the plan was to build a shelter rather than carrying in tents.  The first problem was that the tarps that we brought to build the shelter with were too small for comfort – it was going to be a “friendly” night.  As the day went on, we sawed down standing deadwood for the shelter and the fire and waited, fruitlessly, for the fish to bite. As the sun started to go down – still with no fish to show for our efforts – a light but persistent cold rain started to fall.  The rain chilled us all to the bone, despite our waterproof gear.  After some great dinner including Nick’s excellent carne asada, the first member of our group, Rachel, decided to head to bed.  Shortly after entering the shelter, she called out that everything in the shelter appeared to be soaked and that there was a “pool” in her sleeping bag.  Nick investigated and made the call that there was no way that it would be safe for us to stay the night in these conditions.  We packed up all of the gear and, despite the exhaustion across the group, we were quickly on our way back to the cars.  Fortunately we managed to find a trail that took us around the mountains on the way back rather than over them and kept us from having to bushwack our way out. We finally arrived back at the cars at 2am.  Unfortunately our adventure did not end there.  After the long day of heavy physical activity and the forced march out, no one was in any particular shape to safely drive.  We ended up limping slowly back to civilization by changing drivers and taking naps as we went with half the team taking a stop for the rest of the “night” at the first hotel along the route. All in all, it was an adventure which makes us all appreciate summer camping so much more.  And makes us look forward to next year where we can be vindicated with a more successful trip, and maybe some trout!

Is SOA Still a Thing?

• January 13th, 2013
• shan

The Oldest Buzzword Around

Service Oriented Architecture (SOA) isn’t a new concept by any means. It’s practically a decade old and, in IT years, that’s beyond the useful lifespan of just about all buzzwords. And that’s the problem; as a buzzword, SOA never attained the same level of popularity as Cloud or Big Data.  The concept of SOA was nebulous and how an organization could achieve SOA was even more unclear. Vendors were pitching anything ranging from just an asynchronous messaging infrastructure to a full blown process automation and orchestration suite as the “Conerstone of Enterprise SOA” solutions.  Further confusion was caused by product vendors trying to differentiate their products by pushing the importance of interoperability standards (ie. WS-*) claiming that other competing products weren’t truly “SOA” for one reason or another. This confusion created just as much negative stigma around the term SOA as positive sentiment.  While product marketing folks were focusing on the discussion of just what is and isn’t SOA, the Architects were quietly picking and choosing the concepts of SOA that they liked and evolving their enterprises’ IT landscapes.  

SOA – Still Alive & Kickin’

Fast forward to today. RESTful has fully taken over as the web service integration style of choice for the Internet, relegating SOAP for internal enterprise interactions and transactions that are considered “low throughput”.  JSON has gained traction in the same way over XML thanks to movement towards mobile computing and a renewed focus on making interfaces as lean byte-wise as possible.  No one thinks twice about decoupling the UI from the business logic and integrating using a set of web service calls.  And asynchronous messaging is practically the status quo method of propagating large amounts of data across the enterprise. So yes, the key SOA concepts of: 1 – developing applications that promote reuse 2 – decoupling functional application components to improve flexibility and agility 3 – standardizing the way interfaces are described and interacted with to promote predictable and consistent integrations are more prominent than ever.  Exposing Big Data stores as RESTful services is one of the most popular ways of integrating with these technologies.  And the SOA concepts of abstraction, service contracts, and reuse are at the foundation of SaaS solutions.  

It Just Makes Sense

SOA is at a level of maturity where it no longer benefits from having its own buzzword. After all, you don’t see organizations advertising that they’re a client-server shop or that they are prolific adopters of web architecture to differentiate themselves in 2013.  We’re at a point where sound architecture principles put forward by the proponents of SOA nearly a decade ago, have become just good architecture practice. The conversations today with IT executives should no longer be “Should you adopt SOA?” but “What should you do to better address reuse, flexibility, and consistency within your enterprise?”  

Ring In 2013 with Functional Programming

• January 7th, 2013
• Patrick Prémont

Learning a new programming language every year may be excessive, but this year is special. If you haven’t had the chance to do so before, now would be a great time to learn functional programming (FP) techniques, and to start applying them, possibly within the language you are already using. Here I’d like to briefly go over why FP is a great programming paradigm, especially for today’s software problems, and how conveniently it can be applied within a wide variety of languages.

The case for functional programming

With FP you will get software that is - more concise, - more reliable, - easier to maintain, - and more parallelizable (this is especially relevant now that multicore systems are everywhere). Let’s see how FP can perform this feat! Functional programming relies in part on higher order functions, meaning functions that have other functions as parameters or return values. This allows functions to represent more abstract and reusable concepts, such as traversals. This increased reuse contributes both to concision and reliability. Since higher order functions require developers to frequently pass functions as arguments, FP languages generally provide facilities to create anonymous functions very conveniently, in any scope. Functional programming also deals with immutable data, allowing variables and fields to be initialized, but never reassigned. This restriction makes it easier to verify properties of programs, both for programmers and for compilers: when writing or reviewing a piece of code, you never have to worry about the potential for some other part of the code to concurrently modify the data you are working with. This greatly facilitates parallel computation and the greater isolation between parts of the code makes it more modular and favors reuse. Without immutability, higher order functions would not be as useful because the caller would depend on the order in which the higher order function called its arguments functions (those could perform mutation in a way which is order sensitive). Finally, FP languages typically have more capable type systems. These allows developers to create more precise types. Many invariants which were described in comments and supported by test cases, can now be encoded within types and proven by the compiler. By simply checking the correspondence of types, the compiler is in fact carrying out a proof that all executions of the program will respect important program properties. Typically FP languages also relieve you from having to write most type annotations as the compiler is able to perform type inference. Such type based verification of programs can go very far. In fact it can prove full program correctness in languages which are capable of representing dependent types. However, those more precise types cannot generally be inferred automatically. Writing such proven programs represents more work and should therefore be reserved for highly critical code. Type systems also have great benefits in non-functional languages, but the presence of mutable data limits their ability to prove program correctness. With immutable data there is no such limitation and skillful use of the type systems can produce extremely reliable software. This type of compiler verified reliability greatly simplifies software maintenance, and even development itself. Changes can be made to source code with great confidence since the compiler is catching most errors. This enables even simpler and concise code since refactoring can be applied more liberally as program requirements evolve. The reliability benefits of FP are in large part due to the compiler’s ability to detect very large classes of errors. This automatic verification also makes the software easy to maintain since most incorrect changes are identified during compilation. And of course having more concise (yet readable) code also helps with maintainability.

Functional languages and libraries

If the above seems convincing, you may still be unconvinced about the practical applicability of these techniques, either in functional or mainly object-oriented languages.

Are functional languages sufficiently mature ?

Functional languages such as Scala, F#, OCaml and Haskell can now be used very effectively in demanding commercial applications thanks to more mature tools and libraries. This was clearly observable at the 2012 CUFP conference where a diverse selection of commercial users presented very positive experience reports. I invite you to judge for yourself by viewing some of the archived presentations.

Can these techniques be used in object-oriented languages ?

Many of them can. The result may not be as syntactically elegant as in functional languages, so some of the conciseness gains may be reduced, but you can still gain a lot in terms of reliability and support for concurrency and parallelism. In fact many non-functional languages have recently been revised to support anonymous functions and somewhat improved type inference: - C# 3.0 (2007) - C++11 (2011) - Java 8 (due in 2013) These developments are making functional techniques a lot more practical, and are supporting the development of libraries based on functional principles. Functional Java

Tindr Takes Scala on the Road!

• December 12th, 2012
• nicole

Tindr has gained a reputation as a leader in the Scala community. We thought it was time to take our knowledge and experience on the road. Between October and December 2012 we met with over 100 Scala enthusiasts and professionals across Ontario and Quebec. Ottawa We started close to home on October 23rd with our Ottawa Scala Meetup. We packed Mercury Grove for a night of talks and community networking. We kicked things off with talks by our very own Tindr team on their recent work with Scala and Play. We were also very fortunate to have Jaime Allen, a consultant and instructor with Typesafe. It was great to see the community out for a night together, and hear about some of the exciting projects going on. Montreal

ESB: To Adapt(er) or Not to Adapt(er)

• November 5th, 2012
• shan

One of the most commonly touted features of commercial Enterprise Service Bus products (ESB’s) is the out-of-box adapters for other COTS products (eg. SAP, PeopleSoft, Siebel).  But organizations who become dependent on these adapters and take a “use ‘em if you’ve got ‘em” approach inevitably find that the implementations are a lot more complex than advertised.  This is because the decision on whether or not to use adapters should be driven by the alignment of the organization’s skillsets and support structure, not the perceived simplicity in using out-of-box components. I’ve created a simple decision tree to help clients determine whether they’re better off using the adapters provided by their ESB vendor or whether they should consider some alternative method of integrating with the ESB: 1)      Is the integrating system a data component with no business logic layer and with no built-in ability to be exposed as a web service (eg. Database, LDAP, MQueue)?

1. Yes – Use the provided adapters.  There’s no point adding an additional layer of complexity when simple data integration is very mature among all the leading ESB products.  Also ask yourself if you’re doing data mappings and translations within the adapters?  Whether there will be a DBA on the ESB support team or not.  ESB’s are excellent at handling simple data translations (ie. Field-to-field).  Any complex data translations such as multi-table selects and data merges should be handled in the data tier where you have DBA teams who are equipped to manage them.
2. No – Go to question 2.

2)      Does the integrating system have the built-in ability to generate a web service or ESB-compatible API?

1. Yes – Go to question 3.
2. No – Use the provided adapters.  The whole point of SOA is to leverage existing assets as much as possible and to avoid building new custom code for integration.  Putting in a custom coded service layer between the ESB and the target defeats the objective.

3)      Will the support team for the integrating system be cross-trained on configuring the ESB adapters?

1. Yes – Use the provided adapters.  The support team of the integrating system will ensure that the adapters and reconfigured as necessary based on any changes in the application in question.
2. No – Expose web services and/or API’s on the integrating system.  The support team for that system are the experts on that application.  They know the data model and the implications of seemingly inconsequential details on the business logic in the backend.  I’ve seen many an adapter implementation go off the rails because the team implementing the ESB simply are not experts on every system they’re connecting to.  Let the team with the expertise manage the abstraction and register their endpoint on the ESB so you can still take advantage of features like version management and BAM.

Next time you’re opening the box to your shiny new ESB, resist the urge to take all the adapters out and deploy them everywhere possible.  Ask yourself in the long run, who’s going to own the integration point? And whether their lives will be made easier with the adapter or without?  An IT landscape that is well supported is much more important than an IT landscape that matches exactly to the vendor’s product sheet.

Contributing a Pusher module to the Play framework

• October 11th, 2012
• Alejandro Lujan

At Tindr, building great software with the Typesafe stack has been a fabulous experience. Tindr’s latest collaboration with the folks at Typesafe was for building Scala1, a mobile application to visualize event and speaker information related to Scala, Play! and Akka for the 2012 JavaOne conference. The App – available for iOS and Android – includes a Discussion feature, which can be done at three levels: global, event-specific or private. For communication between server and clients we chose to use Pusher, a hosted service that uses Websockets to provide a fast and easy-to-use API for web and mobile apps. It features client and server libraries for a number of platforms. Realizing there was no integration module between Pusher and Play 2, we decided it was a perfect opportunity to give back to a project we support. It will hopefully help others who want to try Pusher by providing a super easy way to integrate it with your Scala application.

The Pusher Module

(For the impatient: you can find the module hosted here) Our module provides a very simple way to integrate server capabilities into a Play 2 project, so that you can do stuff like the following: To illustrate, let’s look at a simplified use case of our own Scala1: The above diagram shows the Event Discussion feature. Once the user clicks on the Join the Discussion button on the event detail view, the app subscribes to a channel that represents that event (event-42 in the example). Sending a message makes an API call to our Play! server, which in turn forwards the message to Pusher. Pusher then relays the message to the subscribed clients. Pusher takes care of the Websocket details, and facilitates channel creation (just post to any channel name) and event processing (just provide a function to be executed when an event is received).

The work

Writing a module for Play 2 is relatively easy, once you figure out how to do it. There is currently no official documentation, although there seems to be an initiative to make that an official part of the framework’s documentation. However, these two links should take you far enough. Essentially, a module is a Play project that cannot be accessed directly – there is no routes file, so controllers are not directly available. Other than that, it has the same structure and access to libraries as any regular Play project.

Publishing the module

To make a Play 2 module available, it must be published as a Maven repository. We chose to do so using Github, following this guide. It essentially boils down to:

1. Create a GitHub page (pretty handy feature, allows you to host files publicly from a repository)
2. Package your module on a Maven friendly format
3. Publish the generated files on the GitHub page repository

The files are then hosted by GitHub and served as-is. No need to host your own dedicated Maven repository.

Give it a try

If you have a Play 2 project, give the module a try and let us know how it goes. It would be awesome if you want to contribute as well – the module is open source, of course!

Agile Governance in SOA

• October 11th, 2012
• shan

I know, at first glance it looks like the cramming together of 3 loosely defined and oft-debated buzzwords into the title, but got your attention didn’t it?  So if you will bear with me for a few minutes, I will explain how applying Agile to a much more abstract of an activity such as building an SOA Governance Framework works. SOA Governance is a fluid concept, with many differing, complementary, and overlapping point of views.  Is it a framework focused on lifecycle management of service assets?  How does it integrate with existing governance frameworks around other IT components such as infrastructure, database, packaged applications?  Or should it be focused around the monitoring and management of business capabilities.  The discussion could go on forever. Because of this fluidity, I’ve seen two major problems come up again and again with clients when it comes to creating an SOA Governance Framework: 1)      The client has invested in a shiny new SOA Governance Framework complete with templates, registries and repositories, architectural review boards, the works.  It takes a year to roll out.  And the moment it’s done, people start finding scenarios that don’t fit within the framework; do not apply it consistently; or refuse to use it altogether. 2)      Various architects within the client organization can’t agree on the scope, focus, or business case around such an ambiguous concept as SOA Governance, so it keeps on getting pushed off. The reason why the first problem occurs is that organizations are treating Governance like a traditional application build project.  You need to define it all up front and build it once.  And that can only lead to an SOA Governance Framework, or any Governance Framework for that matter, which is out-of-date before it’s even delivered.  Governance needs to constantly evolve to adapt to the changes in an organization.  Is the organization moving from being IT focused to being business focused?  Is there an increased uptake in COTS implementations over custom development?  Is the organization moving to an offshore support model?  All of these changes in the organization will impact how the services should be governed.  In short, the way the SOA Governance Framework is developed and managed over time needs to be flexible and adaptable to change.  Sounds like Agile, doesn’t it? The same general mentality also leads to the second problem: the idea that the SOA Governance Framework has to be perfect and all-encompassing the first time around.  The fact of the matter is that instead of being stuck in analysis paralysis, we’re always better off with having something rather than nothing.  Even an imperfectly managed SOA landscape is better than a completely unmanaged landscape.  One of the key tenets of SOA is to enable continuous improvement of business capabilities and efficiency by building it out in loosely coupled modules.  So why can’t we implement a SOA Governance Framework by starting small and building up additional components as we learn more about the landscape and become more experienced in working with it.  Isn’t it better to solve a real problem that we encounter than to create a solution in anticipation of a problem?  Once again, sounds a bit like Agile, doesn’t it? That was a long preamble.  So here’s our approach: 1)      Get the organization to agree on a core set of SOA Governance deliverables, creating a baseline of the framework.  In Agile terms, this list is our backlog and includes: - a Service Definition template - a Service Contract template - a Taxonomy - a Service Registry and Repository - and some kind of org chart from at least a support perspective 2)      For each deliverable defined in the backlog, analyze whether something similar already exists in the organization today or needs to be created.  Group similar/relevant deliverables together (ie. Service Definition template with Service Contract template).  Implement it and deliver it to the organization.  These are our sprints. 3)      While operating our SOA landscape, constantly be on the lookout for areas to improve.  Did we have any services that didn’t fit the definition template?  Did we have a project that identified inefficiencies within the architectural review process?  Is the service inventory becoming too cumbersome to manage?  Are people adhering to the change management process and if not, why?  Was there a re-org that requires us to rethink the key roles and responsibilities?  Frame these into deliverables and put them into the backlog. The key to success here is continuously looking at the applicability and appropriateness of the framework and keep it aligned to the rest of the organization.  If the users of the framework start to feel like they have to shoehorn something into a structure that no longer works for them, they will cease to use it.  So don’t fall into that trap and let’s apply some agility to our SOA Governance initiatives.

Building an API for your system

• September 18th, 2012
• Alejandro Lujan

Building an API might be one of the best things you can do for your system or product. Why, you ask?

1. You can expose your data and logic to other systems, effectively multiplying your potential user base.
2. It forces you to think hard about your data model and business logic.
3. It ups the ante when it comes to testing your system boundaries.

I won’t expand on (1), but I’ll share my experience with (2) and (3).

Think harder

If you’re used to building web applications, you’ve grown accustomed to interacting with your users through a (hopefully) neat interface. However with API users you are only showing them your raw data objects – no pretty images and css to hide behind. No makeup. As a developer, building an API makes you think harder about your data model and business logic. It makes you look at your system from outside. This introspection, if done right, has a tremendously positive effect on the quality of your system. You may be tempted to refactor some of your code to respond more nicely to a RESTful approach, simplify some of your design choices in favor of usability, and so on. Follow the temptation, binge on your neatifying impulses. Your future self will thank you.

Test. Now test again.

Since an API has no UI by definition, verifying its correctness naturally leads to building automated tests. I shouldn’t need to tell you this is also a good thing. All systems evolve in time. Data models expand and change, business logic mutates. When changes are introduced, and inevitably existing features break, running your test suite helps you identify where the problems are. It also helps you verify that you’ve actually fixed them. This has incredible value, and it pays out – specially in the case of APIs.

Tips

You can find valuable tips on how to build a good API here, here and here (that last one by a Google engineer). I strongly recommend doing some reading before designing your next API. To close, let me mention some fundamentals that I found particularly valuable:

1. Write tests before the implementation. A basic TDD technique that certainly aligns with the introspection aspect I mentioned before. Testing before building makes your tests somewhat less dependent on your implementation.
2. Document your API just enough. Too little, and it becomes unusable. Too much and the documentation itself becomes unmaintainable. I know, finding balance is hard.
3. Version your API, so that in the future you can evolve your interface without breaking existing API users.

On Re-Inventing The Wheel

• August 31st, 2012
• Mike Kelland

Every couple of weeks, I have the same conversation. The conversation usually starts with something like “we could really speed up development if we reused [random piece of vaguely related code]“. And it generally ends with a fresh understanding of what it is that software developers do all day.

The fundamental proposition is that what software developers really do all day when they’re trying to create a product, is apply a specific and specialized set of behaviours and actions against a giant pyramid of generic software. It’s a customization exercise.

Think about it this way – every software product has components that are generic – the functionality is very standard and common. These are kind of “solved problems”. So you think about an operating system – every piece of software needs some kind of operating system under it – we don’t need to worry about stuff like directly accessing the hardware and choosing the blocks of memory our program writes to. And that doesn’t really matter if I’m building a web app or a desktop app like Photoshop. It’s common and a solved problem.

Then take a step up and look at an SQL database. It’s still really common, there’s a large subset of problems that it solves that are really common. It doesn’t help me when I want to build Photoshop, but for pretty much all web apps, I’m in good shape. So it’s a bit more specific – it can’t be used for all software, but for a big subset of applications. Then you take another step up, etc, etc…

Essentially, the point at which the “specific-ness” *generally* overwhelms the number of problems that something can solve when we’re talking about database-backed web applications is at the level of frameworks. There are a bunch of different frameworks out there, but some of the major ones are Play! Framework (the one we use), Rails, .NET, Django, Cake, backbone.js, etc… there are also a bunch of higher level “toolsets” like require.js or LESS that also help out. These frameworks are specific enough to be helpful but generic enough to span the vast majority of the “solved problems” in the web application space. They define things like how the code will talk to the database in as few steps as possible and how the application will deal with the lifecycle of a web request and really common stuff like logging users in and out. Where they stop short is in defining and forcing you into a database layout and workflow for the application.

The problem with using someone else’s code is that they have already developed software that is more specific to their needs than the “solved problems”. Their code specifically does what it was intended to do in their application. What this means is that, if you want to use their code, you have to do a lot of *extra* work to get it torn down to a generic point where you can build it back up to your specific needs. So, you start with working code, then you need to have your developer spend 4-8 weeks understanding what the code currently does and all of its nuances, then you need to pull out large portions of their work that doesn’t apply to your app (hoping that you don’t break anything in the process) and then you need to build it back up to your requirements.

So in almost all cases, unless the products are nearly identical, you want to start fresh with all the generic frameworks that you need in order to accelerate your development. Using someone else’s stuff is generally a net negative rather than a net positive.

To follow this along the garden path, the place I see a lot of people get hung up is with products like Joomla, Drupal and WordPress. These are great content management systems and strong applications with great communities. Where people run into trouble is when they consider them to be as general as a framework and use them as equivalents. The content management problem is a specific one, and one that these systems solve admirably, but if your application is not fundamentally a content management application, shoehorning it into one of these systems is going to be a huge exercise in frustration, re-architecture and pain. And it’s almost never worth it.

Bottom line? It is exponentially more difficult to reuse code the more difference there is between the problems you’re trying to solve and the problems the existing code solves.

So go to town, feel good about re-inventing the wheel!

Who wants Push-ups?

• August 16th, 2012
• masashi

Among all the exercises you could do in the office, which one do you like most? “Push-ups” That’s the first one that came to mind. It’s one of the most basic and popular exercises in the world. At Tindr, we’ve been looking for ways to stay active at the office since most of us are sitting in front of monitors all day. Since we’re a start-up, we don’t have much of a budget for an exercise coach. So we had to look for an alternative solution. Being a shop full of developers, we naturally looked for an automated solution to this problem. So, I wrote a bash script that reminds us every hour to get off our chairs and do some pushups. Want to give it a try? We keep the script in Github, and you’ll need this to run it:

1. Mac OSX > 10.6
2. Shell: BASH
3. “afplay”, “killall”, and “osascript” enabled.

And to execute the script every hour, you can use a simple cron job with the following settings: Have fun! Masashi Kobayashi Software Developer @ Tindr.co

Prefetching and debugging in Google Chome and Firefox

• July 5th, 2012
• long

Like many of us, I’ve been using Chrome exclusively for my web browsing. It is faster, lighter, and more intuitive. The smart address bar has changed the way I browse and I can’t imagine using a browser that does not have that feature. Another feature built into Chome and Firefox is prefetching – the browser predicts what pages you want to see and loads them in the background, creating the feeling of a very fast browsing experience. This feature produced an unintended side-effect when I was debugging a web application. I had set up two views in Visual Studio using MVC – “shoppingCart”, “shoppingCartEntry”. The controller has two breakpoints; one for each view. After a few hits on both views, the address bar had them cached so I no longer had to type the pull page. Instead I could hit the down button to scroll to “shoppingCartEntry”. This is where the prefetching caused an issue. As I scrolled passed “shoppingCart”, chrome would fetch the page and trigger my breakpoints even though I hadn’t pressed enter/selected that address. This can cause problems if your second view alters data that your first view requires. Also, it can be confusing if you don’t know about prefetching and are left wondering why certain breakpoints are being triggered when you haven’t actually visited the page. To see how to disable prefeching in Chrome and Firefox, look to this link.

How to Fix a Broken Enterprise Software Development Project

• June 19th, 2012
• Mike Kelland

We’ve all heard of enterprise software development projects that went on forever, cost millions of dollars and, often, never resulted in a final working product. A quick Google trawl reveals that somewhere in the range of 30-40% of enterprise software development projects are outright failures and estimates of projects that “don’t meet their goals” range from 60 to 80%. How and why do we, as an industry, continue to tolerate such a terrible failure rate? And, more importantly, what can be done about it?

Software development projects are finicky beasts. They are research and development activities, fraught with uncertainty and risk. They are also infinitely flexible – susceptible to scope changes and requirement shifts. They require skilled resources who need to maintain deep focus on their tasks in order to remain productive. And they are conducted in ever changing business environments complete with external and internal stimuli.

The most commonly stated reasons that enterprise software projects fail are:

• Lack of User Input
• Incomplete & Changing Requirements
• Poor Execution

The most powerful factor to combat these issues is strong, technical project management that has a deep focus on shipping. Our project successes have proven, time and again, that software development projects that have strong and technical project management and that incrementally and consistently ship code come out successful.

It’s not as simple as it sounds. All of the factors involved in effectively fixing a broken software project are difficult to achieve and take discipline, experience and expertise. They require that the entire enterprise remains focused and interested in the success of the project. And they require some very special individuals in the management team who can deeply understand the technology and speak the language of the business.

Strong project management needs to be fully supported by their management up to the executive levels. It needs to be disciplined in its approach and focused on removing roadblocks. It needs to be able to isolate the development team when they need to focus. Developing software is an incredibly context-sensitive task. It has been described as building a house of cards in your mind. It takes between 30 minutes and an hour, depending on the developer, to get into context enough to be truly productive with the code. And it takes only a moment of changing focus to blow the entire house of cards away. Strong project management needs to be fully aware of this and needs to be able to accept and work within the demands of the business while maintaining the developers’ focus. They need to be able to create time-boxed “platforms” of certainty and stability for the development team against the backdrop of the chaotic reality that the business operates in.

The only way that a development team can consistently work productively towards a goal is when their requirements don’t change. But change is the only constant in business and requirements never remain the same. Strong project management needs to be permitted to isolate developers so that they can work on a subset of requirements – no changes allowed – for a period of time. This is one of the underpinnings of the Agile methodology – nothing changes within these time-limited islands of stability (called Sprints in Agile), but everything can change after one ends and before another begins.

Strong project management also needs to internally maintain discipline around the requirements that have been selected for the sprint in question and not allow anyone to waver or go off course within the actual sprint. They need to set and adjust reasonable expectations and use the appropriate tools to understand progress so that they can see far enough ahead to proactively adjust resource loads. Adding resources too near a deadline is one of the most assured ways of missing it.

Project management must also be technical. In the same manner that a general contractor is an experienced tradesperson, software development project management must intimately understand the technology that they are working with and the process of software development itself. Technical project managers are respected by their development team, they understand the impact of their decisions and they have felt the pain of scope changes in their own work.

Technical project managers intuitively understand how changes to scope will affect a project – they can guide and counsel stakeholders towards solutions and changes that minimize the impact on the project and work with the business to refine ideas and concepts into implementation details.

Above all, technical project management works tirelessly to reframe expectations and communicate impacts of changes in the pursuit of long term maintainable code. As requirements inevitably change, the underlying software can rapidly become unmaintainable and unworkable. Changes and updates that should have been simple, start taking longer and longer, to the point where even the smallest change takes months. Technical project management gives development teams the room to execute an effective test-build-refactor cycle to consistently ensure that the code is high quality and long-term maintainable.

Good software is never done. If a software project is successful both in the technical and the business sense, it will be under development forever as new features are added and the system is adjusted. Software projects that fail are often the victim of a lack of focus on maintainability where project management that does not understand the impact of their actions requests that features are “just added quickly”, resulting in a huge base of unmaintainable and unmanageable code. A very good barometer of whether a software project is going well or not is to determine how long it would take to implement a feature this month versus the same feature last month. If it takes significantly longer, the project is likely in trouble.

Project management must also have a focus on shipping. There are two things that happen when a software development project does not “ship” or deliver working, tested products with a subset of the overall feature requirements.

First, it tends to “spin” – because the product remains in a constantly ephemeral state, with no firm milestones for a demo of the product, all the “little” things that make code ready for use never get done and the system remains in a constant state of flux. As the end of the project approaches, small issues constantly crop up – features that were supposedly finished need “just a little more work” and the final polish of the system drags on and on. Shipping working code at predefined intervals eliminates spin by exposing the code to the harsh light of day and enabling visibility on what still actually needs to be done. It also trains the development team to ensure they fully complete a feature before moving on to the next and to ensure that it’s properly tested while they’re in the context of that feature.

Second, it diverges. The best way to communicate between the stakeholders and the development team and ensure a shared understanding of what the requirements actually mean is to sit around working code and try it out together. If a picture is worth a thousand words, working code is worth a million requirements. And the longer that a development team works without showing the stakeholders working software, the more that the software will diverge from the requirements.

Failed software development projects are the norm in the industry, and while it’s not easy, it is possible not to fall victim to this trend. The proper application of strong, technical project management with an intense focus on shipping can eliminate the major causes of failure by executing well against changing requirements and facilitating effective communication.

Manageable Single Page Javascript App with Play! framework, Backbone.js and Require.js

• April 29th, 2012
• dave

I’m not going to tell you why Play! (1.2.4) is so amazing, or why backbone.js is dyno-mite.

For my most recent project, I knew I needed some javascript (ie jQuery) for a richer user experience, and didn’t want to end up storing data model in the DOM, so backbone was the obvious non-frameworky answer.

Read the rest of this entry »

Deserializing and Parsing Geocoding data with FlexJson

• March 27th, 2012
• Alejandro Lujan

One of the latest features of our current project required that we query the Google Geocoding service to obtain the coordinates of a user-entered address. It was a very quick exercise thanks to FlexJson’s easy to use API. Google Geocoding exposes a Restful API that allows you to send an address and get the coordinates of that address in JSON or XML.

Read the rest of this entry »

Launching a Web “Property”

• February 16th, 2012
• Mike Kelland

We launched our first web “property” yesterday. Yes, that’s a very bad pun – the application we released into the world is a real estate app called HomeProof. HomeProof provides realtors, buyers and sellers with a Home History Report that reveals the history of a home including all of the insurance claims ever made against the home, whether or not the home has ever been the site of a Grow Op or a Clandestine Lab and a host of other items. And that’s just the launch data, we have big plans to add a lot more to the report.

Read the rest of this entry »