For Whom the Batphone CallsPosted: 05/04/2014
A Runbook for Enterprise Adoption of Open Source Software
Enterprise leaders wanting to do a “Dev and Ops” pivot need to own up to the fact that you have to earn you keep today when choosing this path.
If shit is broke, YOU should have the power to fix it. This idea of outsourcing all problems to a paid closed source vendor support call can be naive and lazy in many cases. You should have the power to be your own bat phone. When you call Batman, you are in essence calling yourself.
Retirement Home Enterprise
For the longest time, infrastructure and application support teams have operated in a super old workflow that was uninspiring. They got paid well to be effectively “glorified maintenance/janitorial staff.” An operations spreadsheet/app had a name on it that says they support X and Y. When X or Y breaks, they put a ticket in a queue and call people. These people look at some super basic stuff like 1) Is the server up and pingable?, 2) Is the app process running?, or 3) Is there a vague, nefarious network issue? If none of these things get hits, they shrug their shoulders and and open a ticket with the Vendor that they pay support to and then wait. They then report to some person that reports to some other ops person that they are “working on the issue” with the vendor.
The vendor then has a support engineer (L1/L2) work the call and ask for sane things like logs and read the case notes. The enterprise support person just acts like a drone in this case typing whatever the vendor support engineer says and accepts whatever the vendor provides as an answer. Unless an answer cannot be found, the enterprise support person just accepts the answer/fix and closes the case. If no fix can be found, it effectively is like reporting a bug (If it is not already well known). The fixing for the bug can be soon* or indefinite. Unless the enterprise customer is huge (spends loads on support & products) the bug fix doesn’t get addressed fast. It gets addressed on the schedule most convenient to the vendor. These support contracts are not cheap. This is such a lazy, lame approach. I call this the “Batphone Mentality” where the vendor software support superhero is always a support phone call away. This is especially true for closed source COTS apps whose heavy handed approach actually creates a self-reinforcing cycle of enterprise retirement home “Batphone” mentalities. The problem is that Batman worked for free for Gotham, while enterprises have to pay salaries for multiple external vendor Batmans.
“closed source COTS apps … heavy handed approach actually creates a self-reinforcing cycle of enterprise retirement home “Batphone” mentalities.”
The Emerging Open-Source Enterprise
When you decide that you want to control your destiny, get things fixed faster, or add features and functionality to applications and tools you need to run your business, you may be ready to make the open-source switch. If paying loads of money for support contracts is leaving you beholden to a vendor that is not transparent and open about product bug fixes or feature roadmaps, you are effectively trapped. Major commercial software vendors with closed source products maintain a kind of hegemony over enterprise customers in the following ways:
- Forcing the customer to follow often stringent tech-stack requirements to install and use the product.
- Forcing the customer to forgo support officially if they try to deviate any part of the tech-stack.
- Making money off parts of this “required” tech-stack with onerous licensing.
- Releasing new versions that require forklift upgrades that are uber costly and are often not backward compatible.
- Purposely not being open with support documentation and requiring pay/account gateways to get it.
- Forcing a god forsaken license key or licensing process that makes installs painful and limited.
- Dropping a product completely leaving the customer with no option.
“Major commercial software vendors with closed source products maintain a kind of hegemony over enterprise customers.”
For years enterprises just accepted this in large part due to the following reasons:
- They used closed source OS platforms (Windows/AIX/Mainframes) since viable open-source alternatives were not available.
- They bought software that was closed source because developing the software was hard and costly.
- Tools needed to write software for these closed-source platforms were not easily accessible or free ($1000+ for Visual Studio is pathetic).
- Technology perhaps was not a key part their business (non-tech companies).
- Staff that had requisite skills to overcome (1-4) were not widely available.
Enter the modern tech era Circa 2007 – 2014 where open-source software is almost an American birthright and as normal and accepted as apple pie. Open source operating systems, applications, and software development tools are widely available that can solve many (if not most) of the things an enterprise might need (Linux, OpenStack, Nagios, Hadoop, Tomcat, Apache, Rails, NodeJS, and on and on). Public code sharing repositories like Github, Gitlab, Gitorious, Cloudforge, Sourceforge, and GoogleCode have been a revolution in open-source workflow and project accessibility. Github in particular has been a big part of this inflection point in social coding. There has likely been more open-source momentum in the past 7 years than in that past 30. It is truly an amazing time. Decades from now, folks may look back and ask all us old-timers stary-eyed questions about “what was it like to live during the second open-source revolution?” like we we lived through the equivalent of the roaring twenties or some other historically significant time.
“There has likely been more open-source momentum in the past 7 years than in that past 30.”
While the open-source movement may have not started precisely in the San Francisco bay area, the area has been home to many leaders in the movement. Partly perhaps due to the roots (even today) of silicon valley as a haven for free thinkers and futurists. A place also were it was ok to be distrustful of governments and “the man.” The bay area hippie culture of the 60’s may have also contributed in part to a tradition of folks wanting things to be free or more accessible that continues. Today, there is still a vein of this benevolent revolutionary mindset behind every open-source project. The interesting thing that has happened is that the hippies have been replaced by benevolent libertarian capitalists. This is very cool. Many startups in the valley today often release their product day one as an open-source project. They simultaneously offer paid support for customers. In many cases, their products are based on open-source products but sold also as SaaS or cloud offerings. This is an amazing amount of new freedom for the enterprise.
“…the hippies have been replaced by benevolent libertarian capitalists.”
Be your own Batphone
If you want to control your destiny, get things fixed faster, or add features and functionality you need on your own schedule, you need to be using open-source software and hiring staff that are not drones or retirees. Here is proposal for how the modern enterprise can call their own Batphone.
1) Use open-source software
– get access to the source code now (there is a boat load out there github.com)
– check lists like this (http://www.osalt.com/)
2) Hire good developers for Apps and good developers for Infra
– Pay them well
– Empower them
3) Be very strategic about what commercial, closed source software you buy and use
– It needs to provide a better ROI for functionality, speed, reliability than an OSS alternative.
– Try to pick closed source software that at least is largely standards based (it can ease a transition off the product later if needed).
– Put pressure on the vendor to allow flexibility of the tech stack to use open-source projects (tomcat, node, etc).
4) Train your staff and engineers well
5) Accept the fact that you can now truly control your destiny.
On this last point, it is a function of having skilled developers and engineers that use open source projects and have a specific mentality. The mentality needed is really an “Anti-Batphone” mentality. Developers and Engineers have to own up to the fact that they are the last person standing. They are the last solider in the bunker and have to be McGiver or Will Wheaton to get it working. They have to believe there is no fucking Batphone. They are the Batphone.
A Sample Enterprise anti-Batphone Support Call Flow
1) Some important app is broke
2) Open-source monitoring software alerts the app owner/developer team
– Operations sees that same alert and opens a ticket
3) Internal support engineer starts working on it
– logs, processes, system level debugging
4) Something is broken with the code (a bug)
– Look at the damn source code, you have it!
5) Commit code changes with the fix to your CI/CD pipeline.
– Push to prod for the fix
– close the internal support case.
Paying for Support with an Open Source Vendor
Paying for support with a vendor that has an open-source version of it’s product is truly a win-win.
You get the luxury of official support.
The open-source vendor gets to make money and you still get the source code.
What if you cannot fix the bug with your internal staff? What if you need to pay for support?
No problem, do the following:
1) Locate a vendor that provides the product platform you need as open-source + support
2) Pay these good people for support.
3) Use their product and do the following for support issues
– Open issues on Github for problems and bugs
– Fix the bugs yourself and submit pull requests directly to the vendor github
4) Influence product roadmaps by submitting pull requests and collaborating with the vendor:
– this can be from your own internal staff if they are skilled enough.
– this can be through skilled contractors that you pay to develop modifications to submit as pull requests to the vendor.
– this can be through outright paying the vendor for adding the features directly.
5) If for whatever reason the open-source vendor decides to stop working on the project, closes shop, or will not accept your pull requests for product modifications, you still own the source code.
– you can take over the project
– you can fork the project and do your own thing
This is an awesome new world with loads of new freedom. Enterprises need to embrace this and control their destiny by using open-source software and using open-source vendors. You should have the power to be your own bat phone. When you call Batman, you are in essence calling yourself.