Alfresco Premier Services – New Blog

Posted on March 28, 2017 by ntmcminn

Screen Shot 2017-03-27 at 9.14.38 PM

I’m not shy about saying the best thing about my job is my team. Never in my career have I worked with such a dedicated, skilled and fun group of people. Whether you are talking about our management team or our individual contributors, the breadth and depth of experience across the Alfresco Premier Services team is impressive. Members of our team have developed deep expertise across our product line, from auditing to RM, from workflow to search. Some folks on the team have even branched out and started their own open source projects around Alfresco. We have decided to take the next step in sharing our knowledge and launch a team blog.

The inaugural post details some recent changes to the Alfresco Premier Services offerings that coincide with the digital business platform launch. We will follow that up in short order with new articles covering both content and process services including guidance on FTP load balancing and pulling custom property files into a process services workflow.

Lots of exciting stuff to come from the Premier Services Team, stay tuned!

A Few Thoughts on TEDxBirmingham

Posted on March 27, 2017 by ntmcminn

Screen Shot 2017-03-27 at 2.31.04 PM

Birmingham has come a long way since the days when the city was known as a dying steel town where a heavy handed local government turned fire hoses and police dogs on peaceful civil rights protestors. Today the city is better known for its leading edge medical research, its commitment to civil rights education and preserving its history, and its burgeoning food scene. It’s a city that has its eyes firmly planted on the future, made stronger by the mistakes of its past. Nowhere is this shift more evident than at TEDxBirmingham.

For those that don’t know, TEDx conferences are independently organized and licensed conferences in the spirit of the larger TED conferences. These things have popped up all over the world with more than 15000 events to date generating over a billion talk views. That’s some serious reach for events largely put together by volunteers. Four years ago, a group of people in Birmingham, Alabama decided to get in on the action and start a local TEDx conference. This was my first year to attend, and I was absolutely floored at the professionalism, passion and energy on display. It’s a reflection of the direction Birmingham as a whole is headed. I’d like to put down a few thoughts before the details fade into memory. The conference helpfully provided a program notebook in the goody bag, complete with a section to jot down notes on each talk.

The Talks

All of the speakers were outstanding in their own way. It’s clear that the coaching and practice for speakers paid off. There’s a certain cadence and consistency to the way TED speakers present. They don’t typically do a big introduction about themselves, instead launching right into the content. They take you on a journey, instead of just walking through some slideware. They are well rehearsed, sharing their story without notes. Most importantly, they are concise. Short, even. The idea is distilled into a concentrated essence, delivered as a personal story that brings the audience along in a way that seems almost hypnotic, effortless. A few of my favorite moments:

Brian Reaves was the first speaker of the day, and provided a strong open for the conference. Brian is an illusionist, which I thought an odd choice for a TED style event, especially as the opener. That thought disappeared along with several of his props on stage as he talked about how magic forces you to see the impossible as possible, and how reframing a problem or letting go of your current perspective can completely change the way you approach things. Start from “It can be done, I just need to figure out how”. Having Brian kick off the conference helped everybody open their minds a little bit for the rest of what followed.

Later on in session one, we heard from Dunya Habash. Dunya is a musician, filmmaker and refugee advocate. She spoke about visiting the Zaatari Syrian Refugee Camp in Jordan and how it challenged her notions of what it meant to be a refugee. Her talk was probably the most moving moment of the day, doing more to show the resilience, ingenuity and humanity of displaced persons than any media report I have ever seen. So much of what we hear and see about refugees is at best incomplete, at worst deliberately misleading. She also shared her thoughts on the way refugees are portrayed, driving home the point that consumer friendly media is failing us in ways we don’t even understand. The important stories are so much deeper than a Facebook post, so much more complex than a soundbite, and we owe them our full attention.

Unsurprisingly for an event connected in so many ways to UAB, many of the talks had a medical or public health angle. Dr. Michael Saag challenged our assumptions about eradicating hepatitis C. He ended with a concrete call to action to help us make eradication a reality by reducing the price of lifesaving drugs. I was a bit perplexed that he called out drug advertising as a major cost contributor, but never mentioned lobbying against it in his call. Dr. Julian Maha made a compelling case for expanding the way we address disability to include those invisible, sensory disabilities. Most interesting to me is how he positioned his story as a set of inclusive, reasonably easy to implement changes that can open the world to those with sensory challenges. Dr. Jayme Locke educated the crowd on the human cost of kidney transplant waiting lists and an innovative matchmaking network to improve the chances of donors and those needing a living donor getting connected. Will Wright unpacked the true cost of loneliness on health and challenged all of us to reach out to somebody that needs a friendly ear.

We also heard from some people working hard to change the social fabric of Birmingham. Honestly, social issues aren’t usually my focus area, but these sessions were some of the most moving and eye opening. Maacah Davis coined what may be my new favorite phrase for creating in a space full of constrictive (and often harmful) assumptions: “Artistic Claustrophobia”. Lara Avsar deconstructed how harmful both princess fantasies and Wonder Woman stereotypes can be to a young woman (especially poignant as I figure out how to raise my own daughter), and pointed to a better way in which the struggle, failure and resiliency are core values. Diedre Clark introduced us all to concept of Kuumba (the idea of leaving our community more beautiful and beneficial than we found it) and her innovative community arts program by the same name. Anne Wright took us on a heartbreaking journey through running a program for homeless men, and encouraged us to use our own hopelessness as common ground, a gateway from sympathy to empathy.

It wouldn’t be a TED conference without some representation from education, which Randall Woodfin filled in nicely. Mr. Woodfin has been a fixture in education in the city for some time, a steady, moderate and thoughtful voice for the children of Birmingham. Oddly absent from his introduction was the fact that he has declared his candidacy for mayor, but perhaps the TEDxBirmingham organizers felt it was best to keep politics on the sideline. Either way it was great to hear from our mayoral candidate about his views on education, community and how the two have to work hand in hand for the best outcomes. Elizabeth Bevan rounded out the day, sharing her passion for sea turtles, her job and her research. She gave us a glimpse into the present and future of researching animals in their natural environments with drones. Drones are a relatively new addition to the field biologist’s toolkit, but are already allowing researchers to observe behaviors never seen before.

The Interactive Sessions

I’ve been to a LOT of conferences, usually attending 3-4 a year on a variety of topics. I also present regularly on system architecture, content management, document security and many other topics. I love the TED format and wish more conferences would structure themselves in a similar way, maybe we’d have less “death by powerpoint”. Alternating between sit down sessions and walking around for some more interactive session time keeps the blood flowing, and gives you an opportunity to discuss what you’ve just heard with the other conference attendees. It also gives you a chance to ask follow up questions of the speakers, who usually had a nice crowd around them during every interactive session. Probably my favorite bit from the interactive sessions were the artists that were making art on site that reflected what was shared in the talks. My only complaint was that I didn’t get a chance to try out the VR/AR gear, it always had quite a line.

The Event in General

In a word, it was professional. Start to finish. The crowds were well managed. There were a ton of ambassadors around if you had questions and every one of them had a smile. The check in was fast. Lunch was easy. There was always coffee (this is huge). The presentations were well run. It’s hard to believe that event was entirely put on by volunteers. Unbelievably well done. About the only thing I might change is the addition of a speaker or two that has a bit of a technology focus, but I’m in the industry so that’s my bias showing. With AI and machine learning reshaping the world, and digital transformation completely rewiring companies from the ground up or disrupting them entirely there is a lot of ground to cover there.

I can’t wait for next year. If you live in Birmingham and want to see how inspired and inspiring your city can be, you need to be there too.

Image credit: tedxbirmingham.org. Hope y’all don’t mind that I borrowed it.

Content Services is in Alfresco’s DNA

Posted on March 15, 2017 by ntmcminn

I’m spending this week at Alfresco’s Sales Kickoff in Chicago, and having a blast. There’s a lot of energy across the company about our new Digital Business Platform, and it’s great to see how many people instantly and intuitively get how Alfresco’s platform fits into a customer’s digital transformation strategy. When content and process converge, and when we provide a world class platform for managing and exposing both as a service it’s a pretty easy case to make. We have some great customer stories to drive the point home too. It’s one thing to talk about a digital strategy and how we can play there, but it’s another thing entirely to see it happen.

Content management is undergoing a shift in thinking. Analysts have declared that ECM is dead, and content services is a better way to describe the market. For my part, I think they are right. Companies ahead of the curve have been using Alfresco as a content services platform for a long time. I decided to do a little digging and see when Alfresco first added a web API to our content platform. A quick look through some of our internal systems shows that Alfresco had working web services for content all the way back in 2006. It was probably there earlier than that, but that’s one of the earliest references I could easily find in our systems. That’s over a decade of delivering open source content services. Here’s a quick view of the history of content services delivery channels in the product.

API History

I don’t think any other company in the market space formerly known as ECM can say that they have been as consistently service enabled for as long as Alfresco. It’s great to see the market going to where we have been all along.

Open Source in an AI World. Open Matters More Now Than Ever.

Posted on March 6, 2017 by ntmcminn

121212_2_OpenSwissKnife

Technological unemployment is about to become a really big problem. I don’t think the impact of automation on jobs is in any doubt at this point, the remaining questions are mostly around magnitude and timeline. How many jobs will be affected, and how fast will it happen? One of the things that worries me the most is the inevitable consolidation of wealth that will come from automation. When you have workers building a product or providing a service, a portion of the wealth generated by those activities always flows to the people that do the work. You have to pay your people, provide them benefits, time off, etc. Automation changes the game, and the people that control the automation are able to keep a much higher percentage of the wealth generated by their business.

When people talk about technological unemployment, they often talk about robots assuming roles that humans used to do. Robots to build cars, to build houses, to drive trucks, to plant and harvest crops, etc. This part of the automation equation is huge, but it isn’t the only way that technology is going to make some jobs obsolete. Just as large (if not larger) are the more ethereal ways that AI will take on larger and more complex jobs that don’t need a physical embodiment. Both of these things will affect employment, but they differ in one fundamental way: Barrier to entry.

High barriers

Building robots requires large capital investments for machining, parts, raw materials and other physical things. Buying robots from a vendor frees you from the barriers of building, but you still need the capital to purchase them as well as an expensive physical facility in which you can deploy them. They need ongoing physical maintenance, which means staff where the robots are (at least until robots can do maintenance on each other). You need logistics and supply chain for getting raw materials into your plant and finished goods out. This means that the financial barrier to entry for starting a business using robots is still quite high. In many ways this isn’t so different from starting a physical business today. If you want to start a restaurant you need a building with a kitchen, registers, raw materials, etc. The difference is that you can make a one time up-front investment in automation in exchange for a lower ongoing cost in staff. Physical robots are also not terribly elastic. If you plan to build an automated physical business, you need to provision enough automation to handle your peak loads. This means idle capacity when you aren’t doing enough business to keep your machines busy. You can’t just cut a machine’s hours and reduce operating costs in the same way you can with people. There are strategies for dealing with this like there are in human-run facilities, but that’s beyond the scope of this article.

Low barriers

At the other end of the automation spectrum is AI without a physical embodiment. I’ve been unable to find an agreed upon term for this concept of a “bodiless” AI. Discorporate AI? Nonmaterial AI? The important point is that this category includes automation that isn’t a physical robot. Whatever you want to call it, a significant amount of technological unemployment will come from this category of automation. AI that is an expert in a given domain will be able to provide meaningful work delivered through existing channels like the web, mobile devices, voice assistants like Alexa or Google Home, IoT devices, etc. While you still need somewhere for the AI to run, it can be run on commodity computing resources from any number of cloud providers or on your own hardware. Because it is simply applied compute capacity, it is easier to scale up or down based on demand, helping to control costs during times of low usage. Most AI relies on large data sets, which means storage, but storage costs continue to plummet to varying degrees depending on your performance, retrieval time, durability and other requirements. In short, the barrier to entry for this type of automation is much lower. It takes a factory and a huge team to build a complete market-ready self driving car. You can build an AI to analyze data and provide insights in a small domain with a handful of skilled people working remotely. Generally speaking, the capital investment will be smaller, and thus the barrier to entry is lower.

Open source democratizes AI

I don’t want to leave you with the impression that AI is easy. It isn’t. The biggest players in technology have struggled with it for decades. Many of the hardest problems are yet to be solved. On the individual level, anybody that has tried Siri, or Google Assistant or Alexa can attest to the fact that while these devices are a huge step forward, they get a LOT wrong. Siri, for example, was never able to respond correctly when I asked it to play a specific genre of music. This is a task that a 10 year old human can do with ease. It still requires a lot of human smarts to build out fairly basic machine intelligence.

Why does open source matter more now than ever? That was the title of this post, after all, and it’s taking an awfully long time to get to the point. The short version is that open source AI technologies further lower the barriers to entry for the second category of automation described above. This is a Good Thing because it means that the wealth created by automation can be spread across more people, not just those that have the capital to build physical robots. It opens the door for more participation in the AI economy, instead of restricting it to a few companies with deep pockets.

Whoever controls automation controls the future of the economy, and open source puts that control in the hands of more people.

Thankfully, most areas of AI are already heavily colonized by open source technologies. I’m not going to put together a list here, Google can find you more comprehensive answers. Machine learning / deep learning, natural language processing, and speech recognition and synthesis all have robust open source tools supporting them. Most of the foundational technologies underpinning these advancements are also open source. The mots popular languages for doing AI research are open. The big data and analytics technologies used for AI are open (mostly). Even robotics and IoT have open platforms available. What this means is that the tools for using AI for automation are available to anybody with the right skills to use them and a good idea for how to apply them. I’m hopeful that this will lead to broad participation in the AI boom, and will help mitigate to a small degree the trend toward wealth consolidation that will come from automation. It is less a silver bullet, more of a silver lining.

Image Credit: By Johannes Spielhagen, Bamberg, Germany [CC BY-SA 3.0], via Wikimedia Commons

A Brief History of Screwing Up Software

Posted on March 3, 2017 by ntmcminn

bsod

Unless you live in a cave you have heard by now about the recent massive AWS outage and how it kind of broke the Internet for a lot of people. Amazon posted an account of what went wrong, and the root cause is the sort of thing that makes you cringe. One typo in one command was all it took to take a huge number of customers and sites offline. If you have been a software developer or administrator or in some other way have had your hands on important production systems you can’t help but feel some sympathy for the person responsible for the AWS outage. Leaving aside the wisdom of giving one person the power and responsibility for such a thing, I think we have all lived in fear of that moment. We’ve all done our fair share of dumb things during our tech careers. In the interest of commiserating with that poor AWS engineer, here are some of the dumbest things I’ve done during my life in tech:

Added four more layers of duct tape to the “infrastructure” that holds the internet together with several bad routing table choices.
Had my personal site hacked and turned into a spam spewing menace. Twice. Pay attention to those Joomla and Drupal security advisories folks, those that would do you harm sure do!
Relied on turning it off and then back on again to fix a deadlock I couldn’t find a root cause for. Embedded systems watchdog FTW.
Wrote my own implementation of an HTTP server. I recommend everybody do this at least once just so you can see how good you have it. Mine ended up being vulnerable to a directory traversal attack. Thankfully a friend caught it before somebody evil did.
Used VB6 for a real project that ended up serving 100x as many users as it was intended to. Actually, let’s just expand that to “used VB6”.
Done many “clever” things in my projects that came back to bite me later. Nothing like writing code and then finding out a year later that you can’t understand what you did. Protip: Don’t try to be clever, be clear instead.
Ran a query with a bad join that returned a cartesian product. On a production database that was already underpowered. With several million rows in each table.
Ran another query that inadvertently updated every row in a huge table when I only actually needed to update a handful. Where’s that WHERE clause again? Backups to the rescue!

Anybody that spends decades monkeying around with servers and code will have their own list just like this one. If they tell you they don’t they are either too arrogant to realize it or are lying to you. I’m happy to say that I learned something valuable from each and every example above and it’s made me better at my job.

What’s your most memorable mistake?

Technological Unemployment in Alabama

Posted on March 2, 2017 by ntmcminn

hands

Much has been written about the effect that technology has on jobs. While some (myself included) have seen our careers benefit immensely from the march of technological progress, many people have not been so fortunate. I grew up in the Detroit area from the late 70s to the early 90s and even back then I remember people talking about how many workers were going to lose their jobs to machines. I was young at the time, but the fear and uncertainty was palpable and was a regular topic of discussion in households connected to the automotive industry. Companies rejoiced at the prospect of workers that never took a day off and did their jobs consistently, correctly and without complaint. Human workers had a different view. The robots were coming, an unstoppable mechanical menace that would decimate employment.

Robots vs. “Robots”

Fast forward to the present day. In the words of the immortal Yogi Berra, it feels like “deja vu all over again”. Thought leaders around the world are talking about the effect of automation on jobs, and the scope is orders of magnitude larger than it was before. The rise of AI, machine learning, natural language processing, big data, cheap sensors and compute power and other technologies has expanded the scope of technological unemployment well past the assembly line. Even jobs that were considered safe ten years ago are now witnessing at the very least a sea change in how they are done, if not outright extinction at the hands of technology. Retail jobs face threats from cashier-free stores like Amazon Go. Truck drivers are watching helplessly as Otto completed their first driverless delivery. Personal drivers such as chauffeurs, taxi drivers and ride share drivers are nervous about self driving car technologies from Uber, Waymo, Ford, GM and others. Projects like the open-source Farmbot offer a glimpse into the future of farming where seeding, watering and weeding tasks are carried out completely by machine. Even previously labor intensive tasks related to harvesting are being automated. Timber and logging are on the block as well, with robots being deployed to take down trees and handle post-harvest processing. Construction is facing automation too. We already have robots that can lay bricks faster than a human worker, and companies around the world are experimenting with 3D printing entire structures.

While the physical manifestations of technology may be the most visible, they are not the only way technology is changing the face of work. When people hear the word “robot” they usually think of a physical machine. A Terminator. An industrial robot that welds car frames. A Roomba. A self driving car. That’s only part of the story. AI doesn’t need a physical body, it can live on the internet, basically invisible until you interact with it, and still do jobs a human does today. We are already seeing this in many spaces from healthcare supplemented by IBM Watson, to fully automated AI driven translation services that in the future will handle language tasks that people do today. In short, this isn’t just a “blue collar” issue, it’s much bigger than that. Consider what happens when you can just ask Alexa, and don’t need to call a human expert for help with a problem.

Sweet Home Unemployment

What does this have to with Alabama, specifically? I had not really given much thought to the specific impact on the southeast until I first read about Futureproof Bama. These folks have a mission “to help the state of Alabama prepare for the transition period for when robotics, artificial intelligence, and other autonomous innovations will make most traditional work obsolete.“. The point was really driven home for me when I had the chance to sit down and chat with Taylor Phillips at a recent Awesome Foundation Awesome Hour event and talk automation, jobs and the future of work. Why is this of particular concern to Alabama? Will we be impacted more than other areas of the country? Well, let’s take a look.

According to usawage.com (which calculates their numbers from BLS data) there are about 1.85 million Alabamians employed as of 2015. 3.7% of them are retail cashiers. 2% are freight, stock or material movers. 1.66% are heavy truck drivers. Light truck, delivery, industrial truck and tractor operators combine to add about another 1.1%. Tellers and counter clerks are another 1% or so. Various construction trades are another couple of points, as is agriculture. I could go on, but you can read the list for yourself. It isn’t hard to tally up the affected professions on this list and get to 20-30% of the population that is squarely in the crosshairs of automation in the short term. To put this in perspective, unemployment during The Great Depression peaked at around 25%. Will Alabama fare worse than other areas? I don’t know, an extensive analysis of affected employment categories across states would be necessary to answer that question with any kind of certainty. Regardless of how Alabama stacks up to other states, it’s clear that we face a period of great change in how we view work.

Sounds grim, but you might ask “Can’t these folks just find another job?”. Maybe, maybe not. If the entire job category disappears, then no, at least not in their chosen field. Even if a job doesn’t disappear entirely, a reduction in the number of available positions in a particular industry combined with a steady or increasing number of people that want that job will result in a serious downward pressure on salary. Good for employers, bad for employees. At a minimum, we’ll have a huge number of people that will need to retrain into a different job. Even then, it’s unlikely that we’ll have enough labor demand across industries to soak up the excess supply. All of this adds up to a very big problem that will require an equally big solution. We cannot legislate automation out of existence, nor can we simply ignore the people that are left out of work.

If this concerns you, and you want to join a group of people that are working hard to get Alabama ahead of the curve, pop over to Futureproof Bama and get involved.

My Favorite New Things in the Alfresco Digital Business Platform

Posted on February 28, 2017 by ntmcminn

alfresco-dbp

Everybody inside Alfresco has been busy getting ready for today’s launch of our new version, new branding, new web site, updated services and everything that comes along with it. Today was a huge day for the company, with the release of Alfresco Content Services 5.2, a shiny new version of Alfresco Governance Services, our desktop sync client, the Alfresco Content Connector for Salesforce, a limited availability release of the Alfresco App Dev Framework and refreshes of other products such as our analytics solution, media management and AWS AMIs / Quickstarts. Here are a few of my favorite bits from today’s releases (in no particular order).

The new REST API

Alfresco has always had a great web API, both the core REST API that was useful for interacting with Alfresco objects, and the open standards CMIS API for interacting with content. Alfresco Content Services 5.2 takes this to the next level with a brand new set of APIs for working directly with nodes, versions, renditions and running search queries. Not only is there a new API, but it is easier than ever to explore what the API has to offer via the API Explorer. We also host a version of the API explorer so you can take a look without having to set up an Alfresco instance. The new REST API is versioned, so you can build applications against it without worry that something will change in the future and break your code. This new REST API was first released in the 5.2 Community version and is now available to Alfresco Enterprise customers. The API is also a key component of the Alfresco App Development Framework, or ADF. Like all previous releases, you can still extend the API to suit your needs via web scripts.

Alfresco Search Services

Alfresco Content Services 5.2 comes with a whole new search implementation called Alfresco Search Services. This service is based on Solr 6, and brings a huge number of search improvements to the Alfresco platform. Search term highlighting, indexing of multiple versions of a document, category faceting and multi-select facets and document fingerprinting are all now part of the Alfresco platform. Sharding also gets some improvements and you can now shard your index by DBID, ACL, date, or any string property. This is a big one for customers supporting multiple large, distinct user communities that may each have different search requirements. Unlike previous releases of Alfresco, search is no longer bundled as a WAR file. It is now its own standalone service.

The Alfresco App Dev Framework

Over the years there have been a number of ways to build what your users need on top of the Alfresco platform. In the early days this was the Alfresco Explorer (now deprecated), built with JSF. The Share UI was added to the mix later, allowing a more configurable UI with extension points based on Surf and YUI. Both of these approaches required you to start with a UI that Alfresco created and modify it to suit your needs. This works well for use cases that are somewhat close to what the OOTB UI was built for, or for problems that require minimal change to solve. For example, both Explorer and Share made it pretty easy to add custom actions, forms, or to change what metadata was displayed. However, the further you get from what Share was designed to do, the more difficult the customizations become.

What about those cases where you need something completely different? What if you want to build your own user experience on top of Alfresco content and processes? Many customers have done this by building our their own UI in any number of different technologies. These customers asked us to make it easier, and we listened. Enter the Alfresco App Dev Framework, or ADF. The ADF is a set of Angular2 components that make it easier to build your own application on top of Alfresco services. There’s much more to it than that, including dev tooling, test tooling and other things that accelerate your projects. The ADF is big enough to really need its own series of articles, so may I suggest you hop over to the Alfresco Community site and take a look! Note that the ADF is still in a limited availability release, but we have many customers that are already building incredible things with it.

Admin Improvements

A ton of people put in a tremendous amount of work to get Alfresco Content Services 5.2 out the door. Two new features that I’ve been waiting for are included, courtesy of the Alfresco Community and Alfresco Support. The first is the trashcan cleaner, which can automate the task of cleaning out the Alfresco deleted items collection. This is based on the community extension that many of our customers have relied on for years. The second is the Alfresco Support Tools component. Support Tools gives you a whole new set of tools to help manage and troubleshoot your Alfresco deployment, including thread dumps, profiling and sampling, scheduled job and active session monitoring, and access to both viewing logs and changing log settings, all from the browser. This is especially handy for those cases where admins might not have shell access to the box on which Alfresco is running or have JMX ports blocked. There’s more as well, check out the 5.2 release notes for the full story.

The Name Change

Ok, so we changed the name of the product. Big deal? Maybe not to some people, but it is to me. Alfresco One is now Alfresco Content Services. Why does this matter? For one, it more accurately reflects what we are, and what we want to be. Alfresco has a great UI in Share, but it’s pretty narrowly focused on collaboration and records management use cases. This represents a pretty big slice of the content management world, but it’s not what everybody needs. Many of our largest and most successful customers use Alfresco primarily as a content services platform. They already have their own front end applications that are tailor made for their business, either built in-house or bought from a vendor. These customers need a powerful engine for creating, finding, transforming and managing content, and they have found it in Alfresco. The name change also signals a shift in mindset at Alfresco. We’re thinking bigger by thinking smaller. This new release breaks down the platform into smaller, more manageable pieces. Search Services, the Share UI, Content Services and Governance Services are all separate components that can be installed or not based on what you need. This lets you build the platform you want, and lets our engineering teams iterate more quickly on each individual component. Watch for this trend to continue.

I’m excited to be a part of such a vibrant community and company, and can’t wait to see what our customers, partners and others create with the new tools they have at their disposal. The technology itself is cool, but what you all do with it is what really matters.

Importing Legacy CSV Data into Elasticsearch

Posted on February 17, 2017 by ntmcminn

CSV to ES

I use Salesforce at work quite a bit, and one of the things I find endlessly frustrating about it is the lack of good reporting functions. Often I end up just dumping all the data I need into a CSV file and opening it up in Excel to build the reports I need. Recently I was trying to run some analysis on cases and case events over time. As usual, Salesforce “reports” (which are really little more than filtered lists with some limited predefined object joins) were falling well short of what I needed. I’ve been playing around with Elasticsearch for some other purposes, such as graphing and analyzing air quality measurements taken over time. I’ve also seen people on my team at Alfresco use Elasticsearch for some content analytics work with Logstash. Elasticsearch and Kibana lends itself well to analyzing the kind of time-series data that I was working with in Salesforce.

The Data

Salesforce reporting makes it simple to export your data as a CSV file. It’s a bit “lowest common denominator”, but it will work for my purposes. What I’m dumping into that CSV is a series of support case related events, such as state transitions, comments, etc. What I want out of it is the ability to slice and dice that data in a number of ways to analyze how customers and support reps are interacting with each other. How to get that CSV data into Elasticsearch? Logstash. Logstash has a filter plugin that simplifies sucking CSV data into the index (because of course it does).

The Configuration

Importing CSV data to Elasticsearch using Logstash is pretty straightforward. To do this, we need a configuration file for Logstash that defines where the input data comes from, how to filter it, and where to send it. The example below is a version of the config file that I used. It assumes that the output will be an Elasticsearch instance running locally on port 9200, and will stash the data in an index named “supportdata”. It will also output the data to stdout for debugging purposes. Not recommended for production if you have a huge volume, but for my case it’s handy to see. The filter section contains the list of columns that will be imported. Using filter options you can get some fine grained control over this behavior.

input {
file {
path => [“/path/to/my/file.csv”]
start_position => “beginning”
}
}

filter {
csv {
columns => [
“EventDateTime”,
“User”,
“ElapsedTime”,
“CustomerName”,
“…”,
“…”,
]
}
}

output {
elasticsearch {
hosts => “http://localhost:9200”
index => “supportdata”
}
stdout{}
}

Debugging

No project goes perfectly right the first time, and this was no exception. I use a Mac for work, and when I first tried to get Logstash to import the data it would run, but nothing would show up in the index. I turned on debugging to see what was happening, and saw the following output:

[DEBUG][logstash.inputs.file ] each: file grew:/path/to/my/file.csv: old size 0, new size 4674844
[DEBUG][logstash.inputs.file ] each: file grew:/path/to/my/file.csv: old size 0, new size 4674844
[DEBUG][logstash.inputs.file ] each: file grew:/path/to/my/file.csv: old size 0, new size 4674844
[DEBUG][logstash.inputs.file ] each: file grew:/path/to/my/file.csv: old size 0, new size 4674844
[DEBUG][logstash.pipeline ] Pushing flush onto pipeline

This block just repeated over and over again. So what’s going wrong? Obviously Logstash can see the file and can read it. It is properly picking up the fact that the file has changed, but it isn’t picking up the CSV entries and moving them into Elasticsearch. Turns out that Logstash is sensitive to line ending characters. Simply opening the CSV in TextWrangler and saving it with Unix line endings fixed the problem.

Now that I can easily get my CSV formatted event data into Elasticsearch, the next step is to automate all of this so that I can just run my analysis without having to deal with manually exporting the report. It looks like this is possible via the Salesforce Reports and Dashboards REST API. I’m just getting my head around this particular Salesforce API, and at first glance it looks like there is a better way to do this than with CSV data. I’m also looking into TreasureData as an option, since it appears to support pulling data from Salesforce and pushing it into Elasticsearch. As that work progresses I’ll be sure to share whatever shakes out of it!

Open Source is the Surest and Shortest Path to Digital Transformation

Posted on February 16, 2017 by ntmcminn

Back in 2013, Mike Olson, a co-founder of Cloudera, famously stated that “No dominant platform-level software infrastructure has emerged in the last 10 years in closed-source, proprietary form.”. He’s absolutely right about that. John Newton underscored this theme at a recent Alfresco Day event in NYC. He shared this slide as a part of his presentation, which I think does a great job showing how much of our modern platforms are dependent on the open source ecosystem:

open-components-slide

Platforms are more open today than they have ever been, with a few exceptions (I’m glaring annoyed at my iPhone as I write this). Quite a few companies seem to have figured out the secret sauce of blending open platforms with proprietary value-adds to create robust, open ecosystems AND be able to pay the bills in the process. This is very good news for you if you are pursuing a digital transformation strategy.

Why open source and open standards?

The advantages of open source are pretty well established at this point. Open projects are more developer friendly. They innovate faster. They can fork and merge and rapidly change direction if the community needs that to happen (although there are good and bad forks). Open has become the de-facto way that the digital business works today. I’d challenge you to find any team within your organization that isn’t using at least one open source project or library. Open has won. That’s the first big advantage of open source in digital transformation today: It’s ubiquitous. You can find a platform or component to fill just about any need you have.

Open is also faster to try, and removes a lot of friction when testing out a new idea. Effective digital transformation relies on speed and agility. It’s a huge advantage to simply pull down a build of an open source technology you want to try out, stand it up and get to work. That allows bad ideas to fail fast, and good ideas to flourish immediately. Since testing ideas out is effectively free in terms of dollar cost, and cheap in terms of time and cognitive investment, I think we also tend to be more willing to throw it out and start over if we don’t get the results we want. That’s a good thing as it ultimately leads to less time spent trying to find a bigger hammer to slam that square peg into a round hole. If you decide to go forward with a particular technology, You’ll find commercial organizations standing behind them with support and value added components that can take an open source core to the next level.

If digital transformation relies on speed of innovation, then open technologies look even more appealing. Why do open source projects tend to out-innovate their purely proprietary competitors? There are probably a lot of reasons. An open project isn’t limited to contributors from only one company. Great ideas can come from anywhere and often do. At their best, large open source projects function as meritocracies. This is especially true of foundational platform technologies that may have originated at or get contributions from tech leaders. These are the same technologies that can power your own digital transformation.

Open source projects also make the pace of innovation easier to manage since you get full transparency of what has changed version to version, and the visibility into the future direction of the project. Looking at pending pull requests or commits between releases gives you a view into what is evolving in the project so that you can plan accordingly. In a very real sense, pursuing a digital transformation strategy using open technologies forces you to adopt a modular, swappable, services driven approach. Replacing a monolithic application stack every cycle is not possible, but replacing or upgrading individual service components in a larger architecture is, and open source makes that easier.

Software eats the world, and is a voracious cannibal

There is a downside to this pace of change, however. Because open source projects innovate so quickly, and because the bar to creating one is so low, we often see exciting projects disrupted before they can really deliver on their promise. Just when the people responsible for architecture and strategy start to fully understand how to exploit a new technology, the hype cycle begins on something that will supersede it. Nowhere is this as bad as it is in the world of JavaScript frameworks where every week something new and shiny and loud is vying for developers’ limited time and attention. Big data suffers from the same problem. A few years ago I would have lumped NoSQL (I hate that term, but it has stuck) databases into that category as well, but the sector seems to have settled down a little bit.

There is also a risk that an open source technology will lose its way, lose its user and developer base and become abandonware. Look for those projects that have staying power. Broad user adoption, frequent commits and active discussions are all good signs. Projects that are part of a well established organization like the Apache Software Foundation are also usually a good bet. Apache has a rigorous process that a project must follow to become a full blown project, and this drives a level of discipline that independent projects may lack. A healthy company standing behind the project is another good sign, as that means there are people out there with financial interest in the project’s success.

Simply using open source projects to build your platform for transformation is no guarantee of success, but I would argue that carefully selecting the right open components does tilt the odds in your favor.

You Cannot Succeed at Digital Transformation Without Planning for Scale

Posted on February 13, 2017 by ntmcminn

TL;DR: Digital transformation == scale, just by its nature.

Digital transformation affects all areas of a business, from the way leadership thinks of opportunities to the way developers build applications, and it carries challenges throughout that chain. One of the biggest challenges for IT will come from achieving scale, often in unexpected places.

Why does digital transformation automatically mean scale?

Looking back at my last post on the journey to digital transformation, there are a few points where it should be obvious that you should be prepared to scale up. In the digitization phase, for example, it makes sense to plan for managing a large amount of content and metadata. Whether you are migrating from a legacy system, consolidating multiple repositories or ingesting a bunch of paper, your target repository will need to be ready to handle not only what you are bringing in today but what you plan to create and manage for the next several years. Deploying in the cloud eases this burden significantly, freeing you from having to provision a ton of storage or DB capacity that will sit idle until it is used or deal with adding capacity to an on-premises solution.

Digitalization also drives the need to scale up. Processes that were once done completely manually now get done via software. Along the way a ton of useful information is captured. Not only that information that is required to complete the process, such as attached documents, form data, assigned user, etc, but also metadata about the process itself. How long did it take for a specific step to be completed? Was it reassigned? To whom and how often? What is the current active task? How many instances of each process are in flight? All of this data is collected in a process management system. The more processes move from a manual process to automated or managed processes, the bigger this pool becomes. The net effect is an explosion in the amount of data that needs to be handled.

If digitization of content and digitalization of process lead to the need to scale, achieving digital transformation takes the problem and dials it up to 11. Digital transformation will flip things around and turn more people that were previously consumers of information into producers, whether they realize it or not. An employee working with a digital process may see some similarities in the types of information they are working with as they did before transformation, but behind the scenes there is a lot more data being created.

Alfresco’s platform is built for this kind of scale in both content and process. It is built on proven, scalable and performant open source technologies, and has been deployed by thousands of customers around the globe in support of large, business critical applications. Alfresco provides guidance in several areas to help you size your deployment, build in the cloud, and make smart decisions about how and when to scale.

What about those unexpected areas of scale?

Scaling your content and process platform as a part of a digital transformation strategy is expected from day one, and should be part of the roll out and maintenance plan built before the first application goes live. It may start with scaling content and process technology, but it does not end there. Let’s look at some common drivers of digital transformation. A few days spent reading a lot of articles, literature and opinion on digital transformation yields a wealth of reasons why companies might pursue it:

Improve the customer experience and become more customer centric
Get leaner, meaner and more efficient
Make better business decisions
Responding to an increased pace of technological change
New competitive threats or market opportunities
Demand for real time information and insights

Take a look at that list. Achieving any of these things will require more information to be captured and analyzed. Getting more customer centric means understanding what your customers need, where they are dissatisfied with the current experiences you offer them and what you can do to improve them. Becoming leaner and more efficient requires detailed metrics about processes so waste or delays can be identified and trimmed. Better business decisions and real time information mean drinking from a firehose of data from across the business.

Achieving digital transformation requires you to plan for scale and not where you might expect. It doesn’t just require you to plan for more content and more processes, but also for how to handle the data about those things that you will need to capture and analyze.

The non-technical side of scale

Ultimately digital transformation is not a technology problem, it’s a business problem. It is unsurprising then, that we’ll be faced with challenges that we didn’t expect as we scale up that have nothing to do with technology. Take, for example, support. If your digital transformation rests on open APIs provided by a stack of homegrown, cloud and vendor provided services, how do you route support tickets? If a user reports a problem, how can you narrow down the source and make sure that it gets handled by the right team? A support team can be quickly overwhelmed if they need to sift through a dozen irrelevant error reports to find the ones that they can actually address. The more services you rely on, the harder this problem becomes. This is where detailed monitoring of the service layer becomes important. Guess what that creates? More data. If you are using Alfresco technologies for content, process and governance, you have several options for keeping tabs on your services.

There are other non-technical areas that will be affected by scale as well. Documentation and discovery, for example. As the number of services rolled out in support of transformation increases, developers and business users alike need easy ways to find these services, and to understand how they work. This in itself becomes another service. Change management is another area that a business needs to be prepared to scale up. Digital transformation increases the pace of change in an organization by enabling more rapid response to changing business conditions or new opportunities. Without a solid framework in place to evaluate, decide on, execute and communicate change, digital transformation will have a hard time getting the traction it needs.

If you take away one thing from this and other articles about digital transformation, it’s this: Achieving transformation requires scale from your systems, processes and people and not just those that deal directly with technology. Don’t underestimate it and find yourself with a plan that cracks under the weight of change.