Enterprise Content Management

Armedia Blog

Archive for the ‘Process’ Category

The CRASH Report

January 10th, 2012 by Scott Roth

Cast software, the maker of software quality tools, released their second annual CRASH (Cast Report on Application Software Health) report in December. The report examined the “health” of world-wide software applications by examining the source code of 745 applications (~365 million lines of code), from 160 different companies, spanning 10 industry sectors, and 8 programming languages. The code examination flagged 1800 different types of development and architecture violations that compromise application “health” in 5 major categories. The categories were:

  • Robustness – The stability of an application and the likelihood of introducing defects when modifying it.
  • Performance – The efficiency of the software’s application layers.
  • Security – An application’s ability to prevent unauthorized intrusions.
  • Transferability – The ease with which an application can be transferred to a new maintenance team.
  • Changeability – An application’s ability to be easily and quickly modified.

Cast has drawn some interesting conclusions in their report. Here are a few I found notable.

  • The most secure applications seem to be large COBOL applications in the financial and insurance industry (that should be reassuring to everyone). The least secure applications were written in .Net.  Yikes!
  • J2EE applications scored worst in performance, primarily because of misunderstood technologies and frameworks.  Another contributing reason could be the high degree of modularity inherent in J2EE applications.
  • Transferability scores for applications in the government sector scored lower than in any other industry sector. Being a government contractor, this one strikes close to home. What conclusions or insights can be gleaned from this finding? One insight the report draws is that government agencies are spending ~73% (on average) of their IT budgets to maintain existing applications — more than any other industry sector. I ask you, where is the money in government IT contracting?
  • Transferability and Changeability scores were highest for applications developed using a classic waterfall style methodology, as opposed to an Agile methodology. Whoa! I didn’t see that one coming. (Cast found that the other three categories, Robustness, Performance and Security, were about equal between waterfall and Agile methodologies.) Perhaps because Agile projects are in a continual state of refactoring that they are never in an ideal state to be transferred.

All of the deficiencies identified in this report are termed “technical debt” and have an average cost (according to Cast) of $3.61 per line of code to repair — except for Java, which rings in at $5.42 per line of code. That’s a lot of money and consumes an enormous amount of IT budgets.  For the roughly 365 million lines of code used in the study, that’s $1.3 billion of technical debt.

In conclusion, let me quote from Cast’s own conclusion, who I think said it quite well: “The observations from these data suggest that development organizations are focused most heavily on Performance and Security in certain critical applications. Less attention appears to be focused on removing the Transferability and Changeability problems that increase the cost of ownership and reduce responsiveness to business needs. These results suggest that application developers are still mostly in reaction mode to the business rather than being proactive in addressing the long term causes of IT costs and geriatric applications.”

The 22 page executive summary can be downloaded from Cast here.
http://www.castsoftware.com/resources/resource/email-campaigns/cast-report-on-application-software-health?gad=HPH

The Fast/Good/Cheap Rule of Software Development

September 30th, 2011 by Scott Roth

Triangles have been a staple of mathematics, architecture and engineering for centuries. They have also become important in software development by way of a project management concept. You may have heard of the “Fast/Good/Cheap” rule. This rule uses a triangle to depict and constrain the attributes of a project that are usually in conflict during development: schedule, quality and budget. Ideally, a project should maximize speed (Fast), quality (Good), and efficiency (Cheap).

The rule of “Fast/Good/Cheap” states that a customer can choose only two of these attributes to maximize. The third, unchosen attribute, will naturally suffer. This is a physical constraint of a triangle, and a realistic constraint of a project.

If you consider the area of a triangle to be constant and represent the scope, or requirements of your software project, you can create numerous triangles to model the “Fast/Good/Cheap” rule. For example:

Remember, the area of the triangle (scope of the project) must remain constant. Therefore, a change to any one side of the triangle necessarily affects the other two. The implications are these:

  • Good & Fast = Expensive. High quality software will be produced very quickly to meet a customer’s time constraint. However, this approach will require additional staff, extended work hours, additional testing, etc. — all things that will drive up the cost of the project.
  • Good & Cheap = Slow. High quality software will be produced; however the project tempo will be slow. Other projects will take priority over this project and interrupt its schedule frequently.
  • Fast & Cheap = Poor Quality. The project will be done quickly and on a shoe string budget, and you will get what you pay for. Don’t expect all of the requirements to be met and expect some bugs and unpredictable behavior because the development team didn’t have the time or resources to thoroughly design or test the software.

In reality, everyone wants Good & Fast, but can’t afford it so they settle for Fast & Cheap. Then one of two things happen:

  1.  Inferior, non-compliant, buggy software is delivered that the customer is unhappy with and your company gets a bad reputation for poor quality.
  2. You and your staff absorb all the cost and overtime required to produce quality software on an unrealistic schedule with an insufficient budget (because you are perfectionists). You are lauded as heroes, but the project actually cost you money instead of making you money.

Ideally, you strive to strike a balance among these competing constraints, although some projects will justifiably favor one attribute over another.

All complex systems are difficult to model, and the software development process is no exception. This very simple model can be used to easily and quickly demonstrate to customers how changes to any of a project’s constraints (Fast/Good/Cheap) will effect the other constraints of the project. I have used this model numerous times to demonstrate how these constraints relate and the consequences of changing or over emphasizing one of them. A picture is worth a thousand words and customers usually get the “Fast/Good/Cheap” rule immediately once you draw the triangle. And best of all, beyond its obviousness, there is a lot of solid project management research and data to validate it. I will leave that reading to you.

SharePoint 2010 – REST Atom Service and JAVA, part 2

May 26th, 2011 by Tim Lisko

This is the second of two blogs where I demonstrate how to use the openCMIS library from a JAVA client to connect to a CMIS enabled SharePoint site through the REST Atom service of SharePoint.

My previous blog “SharePoint 2010 – REST Atom Service.” provided information on how to connect to a SharePoint site and examined some of the SharePoint site’s capabilities. In this blog I will show how to use the openCMIS library and REST Atom Service to: 1) Access and Create Folders; 2) Iterate over SharePoint folders; and 3) Add documents to a SharePoint folder. This blog builds on the previous one, so reference it if you aren’t able to follow the code in this one.

Accessing and Creating SharePoint Folders

The following exmple will show you how to get and create folder objects. An additional class library is imported for accessing folders and another for getting a CMIS object by path.

 
import org.apache.chemistry.opencmis.client.api.Folder;
import org.apache.chemistry.opencmis.client.api.CmisObject;

Folder folder = session.getRootFolder();
String folderprops=twolines + “ROOT FOLDER:” + newline +
tab + “ID: ” + folder.getId().toString() + newline +
tab + “name: ” + folder.getName() + newline +
tab + “path: ” + folder.getPath();//create new folder in the root folder
Map properties = new HashMap();
properties.put(PropertyIds.OBJECT_TYPE_ID, “cmis:folder”);
properties.put(PropertyIds.NAME, “mynewfolder”);
Folder newFolder = folder.createFolder(properties);//Check properties
String folderprops=twolines + “NEW FOLDER:” + newline +
tab + “ID: ” + newFolder.getId().toString() + newline +
tab + “name: ” + newFolder.getName() + newline +
tab + “path: ” + newFolder.getPath();

//Get another folder by path that already exists; CmisObject does not have a “getPath” method,
//so convert it to a CMIS folder
CmisObject cmisObj = session.getObjectByPath(“/2011″);
Folder fo = (Folder) cmisObj;

//Check properties
folderprops=twolines + “2011 FOLDER:” + newline +
tab + “ID: ” + fo.getId().toString() + newline +
tab + “name: ” + fo.getName() + newline +
tab + “path: ” + fo.getPath();

 

 

Results:

  ROOT FOLDER:
ID: -1
name: Shared Documents
path: /NEW FOLDER:
ID: 87
name: mynewfolder
path: /mynewfolder2011 FOLDER:
ID: 83
name: 2011
path: /2011

Things to note:

1) The path is relative to the Repository, so the “Shared Documents” library could be accessed at its top level with the path “/”.

Iterating over SharePoint Folders

The following exmple will show you how to iterate over the folder objects. An additional class library is imported iterating over CMIS objects.
 

 
import org.apache.chemistry.opencmis.client.api.ItemIterable;

FileList.setText(“FOLDER (Name; ID)” + newline + tab + “CHILDREN (Name; ID)”);folder = session.getRootFolder();
FileList.append(twolines+folder.getName()+”; ” + folder.getId());
tabs = tab;
this.getKids(folder);
}

private void getKids(Folder folder) {
//recursive method to go through library

ItemIterable children = folder.getChildren();
for ( CmisObject obj : children )
{
if(obj.getBaseTypeId().toString()!=”CMIS_FOLDER”)
{
FileList.append(newline+tabs+obj.getName()+”; ” + obj.getId());
}
}
children = folder.getChildren();
for ( CmisObject obj : children )
{
if(obj.getBaseTypeId().toString()==”CMIS_FOLDER”)
{
folder = (Folder) obj;
FileList.append(newline+tabs+”/” + folder.getName()+”; ” + folder.getId());
tabs = tabs + tab;
this.getKids(folder);
}
}
}

Results:

  FOLDER (Name; ID)
CHILDREN (Name; ID)Shared Documents; -1
/2011; 83
/05; 84
/03; 85
 

Things to note:

1) The root ID of SharePoint libraries is always “-1”
2) You can detect and display properties of files within the folders by using the base type “CMIS_DOCUMENT”.
3) The document ID produced from CMIS_DOCUMENT is not the one you see exposed in the SharePoint GUI, but it is the one you use to access a document in CMIS.

Adding a Document to a Folder

Putting documents into a SharePoint folder is typical operation you would need to do. I’ve shown how you get and create a folder, so now you have a place to put your file. A number of additional libraries are needed for this operation.
 

 
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.File;
import java.io.IOException;
import java.math.BigInteger;
import java.io.ByteArrayInputStream;
import org.apache.chemistry.opencmis.commons.data.ContentStream;
import org.apache.chemistry.opencmis.commons.impl.dataobjects.ContentStreamImpl;
import org.apache.chemistry.opencmis.client.api.Document;

//details of the fileChooser not shown here. The goal is to get an InputStream
File file = fileChooser.getSelectedFile();
InputStream is = new FileInputStream(file);
long length = file.length();
BigInteger bi = BigInteger.valueOf(length);
ContentStream contentStream = new ContentStreamImpl(file.getName(), bi, “text/plain”, is);
Map properties = new HashMap();
properties.put(PropertyIds.OBJECT_TYPE_ID, “cmis:document”);
properties.put(PropertyIds.NAME, file.getName());folder = session.getRootFolder();
Document doc = folder.createDocument(properties, contentStream, VersioningState.MINOR);

Things to note:

1) This code shows the essentials, but actual implementation should provide for the various exceptions that could arise, CmisConstraintException and IOException for example.
2) New ContentStreamImpl requires a BigInteger, hence the conversion of the file length from long.
3) VersioningState can be MAJOR, MINOR, NONE, or CHECKEDOUT. Whichever you use must match the versioning enabled in the SharePoint library.

I have shown how you can use the Chemistry openCMIS library and REST Atom Service to:
1) Access and Create Folders; 2) Iterate over SharePoint folders; and 3) Add documents to a SharePoint folder.

I hope this will “de-mystify” how to perform common DM operations in SharePoint using the CMIS standard!


Software Development and the Perils of Barrel-Dropping Niagara Falls

January 10th, 2011 by dbock

How do we avoid what philosopher George Santayana warned of when he said, “Those who cannot remember the past are condemned to repeat it”. Today’s platforms, frameworks and methodologies all help us learn from the past and can go a long way to help assure our optimum success, but what would your approach be if your life depended on it? Should it, does it, differ from your current approach, and what about the rest of your team. Is everyone on board? Is there a shared vision?

Consider this…do the following kinds of statements mean the same thing to everyone involved in your projects?

“We’re a RUP shop” ”They are CMMI level four”, ”This is an ITIL project”, “Our team is Agile”, “This client RFP specifies a Waterfall approach”.

RUP is a process framework whose creators intended it to be extended and individualized. CMMI levels are attained with great effort, but there is no assurance that they are adhered to or even appropriate in each project. ITIL is a vast collection of codified practices across multiple disciplines. “Agile” can just mean “we’re flexible” or it can suggest a strict adherence to SCRUM.  A Waterfall approach, one iteration and you’re done. Really? You and your team are all in the same boat, but are you sure you are all on the same page?

Danger sharpens the senses. What would our approach be if our lives depended on it? How would we heed Mr. Santayana’s warning? Would we all be in it together?

The Challenge:

Imagine this Associate Press headline:  First Team to ‘Barrel Drop’ Niagara Falls in Homemade Vessel Wins $1m. Maybe not even that far-fetched given the proclivities of today’s uber wealthy eccentric!  We may also find stories of some of the early Niagara barrel drop pioneers. Both tragic and fascinating. But what might we learn from them. Do they apply to us? And do we really need to go all the way to Niagara Falls to get it?

In October 1829, Sam Patch, who called himself “the Yankee Leapster”, jumped from a high tower into the Niagara gorge below the falls and survived; thus began a long tradition of daredevils trying to go over the Falls. On October 24, 1901, 63-year-old Michigan school teacher Annie Edson Taylor was the first person to go over the Falls in a barrel as a publicity stunt; she survived, bleeding, but virtually unharmed. Soon after exiting the barrel, she said, “No one ought ever do that again.”

Does Annie Taylor’s warning sound familiar? By the way, several later daredevils did not fare so well, and it wasn’t until 1989 that a team of two successfully made the plunge, and by successful I mean they lived to tell about it, once they paid their fines and were sprung from jail.

How dissimilar is this fictitious challenge to the last internal project or RFP your team agreed to take on, knowing full well of the perils that await you, and of the rule bending you might engage, in order to assure your team’s success? In both cases success depends largely on user and sponsor acceptance of the project deliverable.  The aforementioned eccentric uber wealthy provocateur  will only pay up if your team goes down together and is the first to successfully live to tell the story! What approach would you take? Would you take lessons, learned and not learned, from Sam Patch, Annie Edson Taylor, and those who followed them over the waterfall?

As I studied the fate of many a Niagara daredevil (I blame the History Channel!), I marveled at how dissimilar each of their designs were, and wonder how closely each of their approaches mimicked current software development methodologies. From pictures I studied of actual “barrels” used in both successful and unsuccessful attempts, it appears none of them borrowed or learned much from those before them. In one tragic example, a daredevil attempted to repeat his successful Niagara plunge elsewhere, but he failed to adjust his requirements properly to the new environment.

On July 2, 1984, Canadian Karel Soucek from Hamilton, Ontario successfully plunged over the Horseshoe Falls in a barrel with only minor injuries. Soucek was fined $500 for performing the stunt without a license. In 1985, he was fatally injured while attempting to re-create the Niagara drop at the Houston Astrodome. His aim was to climb into a barrel hoisted to the rafters of the Astrodome and to drop 180 feet (55 m) into a water tank on the floor. After his barrel released prematurely, it hit the side of the tank and he died the next day from his injuries.

I learned many of these daredevils had used the non-iterative one shot “waterfall approach” that Winston Royce first described and warned us about in a software development article he wrote in 1970. He neither endorsed nor named this approach, but only observed that it had become prevalent as software started being used by people other than those who had written it. Winston thought it was a bad idea, instead favoring an iterative approach, but waterfall non-the-less became the adopted standard of the United States Military, and later, the standard of Governments, consultancies, and companies throughout the world.

An “If your life depends upon it” approach:

If it is your team chasing the $1m, or even $3m for doing the plunge three times in a row, would you use a pure waterfall approach to go over the waterfall, or would you gather all of the collective knowledge of your team and do something else?

My money, and life, is on an Open, Agile, ITIL, RUP Waterfall hybrid approach. So…this does sound like an attempt to string together as many buzz words as possible in an effort to sound clever. But…you maybe surprised to recognize that even if your most successful projects fall squarely in one camp, they do borrow from the rest, even waterfall.

Open: No romance for “not invented here, or by Microsoft, etc”. An open source approach frees a team to borrow well documented and rigorously tested code from the community and use it without the burden of procurement or finance. There is little or no hard cost to trying multiple options, and creating solutions using open source projects as tools and building blocks.

There are currently over 250,000 projects available on Sourceforge, complete with user generated reviews and adoption metrics giving the user helpful information about each project, and world class commercial support available for most enterprise offerings.

Similarly, there are numerous options for canisters [barrels] that could be dug out of salvage yards and re-purposed for the barrel drop “application”. How about old brewers casks or refinery tanks that could be fortified and made habitable? They are already built and readily available, I bet we could even get a bunch of specifications regarding strength and durability that would give us clues as to which is most suitable. Think “Junk Yard Wars”.

This might not be a direct correlation to the open source movement, but it’s close. How many web applications are out there that are “pure Microsoft”, yet leverage the open source Hibernate project for persistence? Uh, pretty much all of them.

Agile: We could study the successes and failures of previous Niagara drops, treating those as initial iterations, and continuing with an iterative approach, tossing a few of our own iteratively developed designs over the Niagara after having equipped them with crash test dummies and sensors borrowed from nearby abandoned automotive plants. Unless it is a complete failure, our first prototype eventually becomes our production model, presumably with minor changes and additions in subsequent test iterations all the way up until “production”, and then perhaps even between  drop [iteration] two and three.

This notion was first explained to me by an old friend and colleague, Jon Kern, who in 2001 co-authored the Agile Manifesto. He and that cadre have come a long way since, and we owe them a great deal. Words to live and work by, and in this case, literally. Reprinted below verbatim.

We are uncovering better ways of developing software [barrels]by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software [barrels] over comprehensive documentation

Customer [inhabitant] collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

And regarding testing, and the Agile/”Extreme” notion of build the test before the code, well, Niagara Falls itself is the test, and it certainly is already built.

ITIL: The Information Technology Infrastructure Library delves into some pretty specific managed service delivery disciplines with a collection of well documented practices, but at its core it is a simple framework for producing repeatable results. ITIL prescribes an approach where one enters and addresses lowest hanging fruit in a continuous cycle of Strategy, Design, Transition, Operation, and Continuous Improvement.

Just makes too much darn sense not to use the ITIL foundational framework to approach this challenge. Keep in mind we are planning for a second and third drop. ITIL owes much to W. Edwards Deming.

RUP: Rational Unified Process is an iterative process framework which supports a one size and shape does not fit all approach to process, but does provide a standardized language around which groups can define and communicate their approach. We owe the term “Use Cases” to, Ivar Jacobson, one of the “Three Amigos” of RUP. He once told me that he blamed the term on his poor English skills, stating that what he really meant to say was “usage” cases. Creating a shared understanding the desired usage of a deliverable is central to success in delivery, and is a central element of RUP, and if RUP is not “Agile”, it is certainly iterative, and with RUP, everyone agrees on what the definition of “is” is.

Waterfall: One shot at requirements analysis, design, implementation, testing (validation), integration, and maintenance? No thank you. But on the other hand, how much iteration is required? Seems in this case the requirements are fairly binary. A reasonably safe and reusable Niagara barrel suited for five occupants and built on a low budget. Hence the statement which I have heard often repeated has some validity here, “a waterfall approach is best in situations where the requirements are well defined and unlikely to change”.

We think Annie Taylor took a pure waterfall approach (pun unavoidable) as no evidence to the contrary exists.  But would you?

Software design is in crisis

July 12th, 2010 by A.J. McClary

In the 90’s, when ECM solutions were rare, we could get away with designing solely toward requirements, but there is too much at stake nowadays. A recent study sampling various IT projects reported that:

  • 62% percent of projects fail to meet their schedules
  • 49% are over budget
  • 47% have higher than expected maintenance costs
  • And get this—25% are canceled before they are ever deployed!

If you’ve been in the software industry long enough, you’ve probably seen all of these things happen. The funny thing is, it doesn’t really come down schedule, cost, or requirements—it comes from bad design. When software companies think design, they’re thinking about contractual obligations and meeting commitments with their stakeholders. Here is a typical scenario:

Company wins contract. Client provides requirements. Company builds a solution, meeting the requirements. Testers validate requirements and application is deployed to a set of users. The users hate it and the client makes new requirements, company builds to those requirements, and the users hate the next iteration…and so on.

This can be resolved by utilizing user-centered design principles in your system development life-cycle. Software is not about code, it’s about people. Instead of our clients telling us what they want, we should be telling them what they need. As engineers, we can do a much better job building solutions then they can—that’s what they hire us for. To be successful, we need to incorporate user research, information architecture, interaction design, and usability testing into our process.

“Iteration 0” – 10 Tasks to Guarantee a Slammin’ User Experience

Let’s start with “Iteration 0”. This sprint is completely devoted to user experience. The point is to get out of habit of thinking of Java Beans and database schemas and start thinking about people. Here are tasks that need to be accomplished:

  1. Gather assumptions and requirements. Take some time to get acquainted with the requirements and begin to make assumptions based on your experience with the technology.
  2. Analyze competition. Familiarize yourself with the way your competition handles things. This is not necessarily the solution you should be striving for, but is excellent to have in your back pocket to show how your solution is better or to compare solutions to initiate change.
  3. Understand goals & tasks. Comprehensive user research, interviews, card sorting exercises, and contextual inquiries help identify user needs
  4. Develop personas and scenarios. Evangelize these with everyone on your team. They can be in the form of user stories, posters, work-flow diagrams, and profiles.
  5. Build a content strategy. Find out what content you have, what needs to be developed, what needs to be expanded, and what can be cut. Also create a schedule for delivering those missing gaps.
  6. Information architecture. It’s more than figuring out what content goes where. It’s also what information is most important to your users. Identify those needs and incorporate them into your design.
  7. Prioritize features. One of the greatest advantages of user centered design is that you often find some requirements barely impact your users. Those requirements can be re-prioritized so you can focus on what’s really important.
  8. Build wireframes and interaction designs. Setting expectations on functionality, how it should look, and how it should behave reduces future UI defects.  
  9. Design a prototype. Build something you can take with you to road shows that incorporates both visual and functional design. Update the prototype during future iterations so you always have an accurate portrayal of what the product will look like at launch.
  10. Validate usability. There are hundreds of techniques, including a usability inspection, paper prototyping, and eye tracking, but put something in front of your users frequently.

Points 8, 9, and 10 should be repeated throughout all future iterations until the project’s completion. When user experience drives design, you build products people love to use.

Doneness!

February 9th, 2010 by Jim Nasr

‘Tis the season for the big sales kickoffs, goal communication, quota setting and the inevitable fudding and jockeying that goes with that. We at Armedia are, of course, not much different and have been dutifully working our list for a while. All roads therefore lead to “Are we done yet?!” Well, some things may never be completely done. And, sometimes, you just don’t know if you’re done or how done is done. That’s kind of like dreaming about having a dream about having a dream…will it ever end! As intriguing as such enigma could be chances are you don’t want to deal with it when it comes to software development. In fact, unless you plan on becoming further fodder for Dilbert, that’s about the last thing you want to do.

Dilbert_software_development

So, how can you achieve “doneness” in software development? There are, presumably, lots of ways…and likely at least one or two not based on divine intervention! Based on our experience, doneness in development (particularly in large projects) is not as much to do with having a cadre of stud developers but having a thorough, systematic and review-laden process that is consistently adhered to. Who knew?!

Here is the gist of our approach to doneness (formal review steps are highlighted in brackets—cannot proceed to subsequent step unless review passes successfully):

Code Design

  1. Document requirements (or change to requirements) to scope level
  2. Get client signoff (FORMAL REVIEW)
  3. Document code design, changes to object model, peer review
  4. Review design (FORMAL REVIEW)
  5. Update system design document

Implementation

  1. Implement test cases
  2. Implement compiled stubs
  3. Review Stubs (FORMAL REVIEW)
  4. Implement services
  5. Integration code
  6. Code review (FORMAL REVIEW)
  7. Update documentation
  8. Documentation review (FORMAL REVIEW)
  9. Implement review changes
  10. Final commit and addition to main build
Copyright © 2002–2011, Armedia. All Rights Reserved.