But how so? Read on…

Part 1 – The challenge

NIEM is an important cog in a government case management solution. At some point in a case’s lifecycle, you may need to share the case-related information with other agencies or individuals, perhaps to get assistance in completing the case or maybe to give more information to another agency for a similar case that they’re working on. Whatever the reason, this information must be easily passed from one system to the next without being ‘lost in translation’.

NIEM (National Information Exchange Model) is a framework for sharing enterprise-wide information across all levels of government. It has a core set of elements that are standard across all the agencies and has more specific ones for the various entities and stakeholders. Agencies can now speak a common language, allowing one agency to publish data from its case management system to the XML standard, which can then be picked up and parsed by another agency that requires this information.

Now all you need is a case management system that lets you export data into the NIEM standard and then a parser to put it back in.

Pretty neat and simple, n’est-ce pas?

Almost. (Let’s face it, if it really were that simple, I’d have nothing to write about and I’d just put up some comics to keep you entertained.)

There are a few problems to watch out for:

• Before importing or exporting, the system must check the data’s security classification. A document requiring a classification level of Top Secret cannot be sent to an agency (or individual) with only Confidential level access. Perhaps only certain parts of the document are deemed to be Top  Secret, in which case, the publishing tool should be able to extract only those parts that are at the required access level.

• Large amounts of data may need to be continuously exported to, or imported from, an external system. Every time an update is made to case data, it may be necessary to push this information out to other agencies. So it may be better to export/import only the changed data and update the existing data. This reduces the time and cost of transfer and would help to reduce duplicated data.

• Data may be imported that already exists in the case management system. Checks need to be in place to handle matching data during import. When duplicate data is found, is it better to replace the data or create a new version of the existing data?

Armedia Case Management comes with a tool specifically to import from and export to NIEM that tackles these problems. Stay tuned for Part Two… The Solution, which discusses how ACM implements the NIEM publishing process.

Until then, here’s a comic strip to keep you entertained (don’t we all just love Dilbert)…

When you work for a while in the software you get numbed to “technologies have come and gone…” occasionally though some become commodities and others trend setters. We have seen that with many products like Apache, Tomcat, Lucene, Drupal …etc that have stabilized and matured over the past years with the help of increased development from the open source realm. Wait! Did I mention the word “Open Source” and going to talk about the enterprise content management?

So without any more ado, we have Alfresco- catering to a rapidly increasing demand of the enterprise content management solutions which is built over the open source technologies such as Spring, Hibernate, and Lucene platforms. Having done years of Documentum development and several Alfresco projects of late, I think there are some interesting overlaps and differences of approach that I feel would make the developers get adapted quicker.

With the wiki site overwhelmed with Introduction, API’s, Development, Deployment and the Forums to answer all the questions regarding the issues faced during the project phase, I am here to talk purely from the developers perspective on what’s the key areas that I happen to witness the difference from the Documentum space.

The road map for my next series of blog is going to cover each of the areas mentioned below in more detailed, code abundant and developer centric approach which will answer the questions:

• Does this feature exist in Documentum or Alfresco or both?
• If yes, how different is the approach?

So with that preamble, and in no particular order, I give you my list of the key areas I got hands on and learnt how different Alfresco is:

• Aspects

Custom data model is the core for any enterprise content management solution. The use of “Aspects” as its core is the fundamental concept for content modeling in Alfresco. Although in the form D6, aspects was introduced, how different is the use and approach in Alfresco is something I will take a deep dive in my next blog.

• Web Scripts

Alfresco Web Scripts brings together the world of content repository and the web. Being a Documentum developer earlier ways of interaction with the repository have been either using DFC API’s or DQL. In Alfresco, Web Scripts provide RESTful access to content within the repository and we can build our own interface using java script. A custom move operation is implemented using the Web Scripts and the comparison of the implementation with the Documentum would be a something to be noted.

• Integration

On my last project, we had requirements for the customers to be able to permanently redact Personally Identifiable Information (PII) from existing documents stored in the repository and version the original document upon save. For various reasons, we decided to integrate the 3^rd party tool Daeja ViewOne module to provide this capability. I will discuss the topic as part of this blog series.

I started this series based on my experience implementing Alfresco projects and I invite you to share any of your experiences with any part of the road map wherein you run into interesting twists and turns? Did you drive off the road to get some help? I welcome your feedback as the blog takes its shape. See you all soon in Take 2.

• 62% percent of projects fail to meet their schedules
• 49% are over budget
• 47% have higher than expected maintenance costs
• And get this—25% are canceled before they are ever deployed!

If you’ve been in the software industry long enough, you’ve probably seen all of these things happen. The funny thing is, it doesn’t really come down schedule, cost, or requirements—it comes from bad design. When software companies think design, they’re thinking about contractual obligations and meeting commitments with their stakeholders. Here is a typical scenario:

Company wins contract. Client provides requirements. Company builds a solution, meeting the requirements. Testers validate requirements and application is deployed to a set of users. The users hate it and the client makes new requirements, company builds to those requirements, and the users hate the next iteration…and so on.

This can be resolved by utilizing user-centered design principles in your system development life-cycle. Software is not about code, it’s about people. Instead of our clients telling us what they want, we should be telling them what they need. As engineers, we can do a much better job building solutions then they can—that’s what they hire us for. To be successful, we need to incorporate user research, information architecture, interaction design, and usability testing into our process.

“Iteration 0” – 10 Tasks to Guarantee a Slammin’ User Experience

Let’s start with “Iteration 0”. This sprint is completely devoted to user experience. The point is to get out of habit of thinking of Java Beans and database schemas and start thinking about people. Here are tasks that need to be accomplished:

1. Gather assumptions and requirements. Take some time to get acquainted with the requirements and begin to make assumptions based on your experience with the technology.
2. Analyze competition. Familiarize yourself with the way your competition handles things. This is not necessarily the solution you should be striving for, but is excellent to have in your back pocket to show how your solution is better or to compare solutions to initiate change.
3. Understand goals & tasks. Comprehensive user research, interviews, card sorting exercises, and contextual inquiries help identify user needs
4. Develop personas and scenarios. Evangelize these with everyone on your team. They can be in the form of user stories, posters, work-flow diagrams, and profiles.
5. Build a content strategy. Find out what content you have, what needs to be developed, what needs to be expanded, and what can be cut. Also create a schedule for delivering those missing gaps.
6. Information architecture. It’s more than figuring out what content goes where. It’s also what information is most important to your users. Identify those needs and incorporate them into your design.
7. Prioritize features. One of the greatest advantages of user centered design is that you often find some requirements barely impact your users. Those requirements can be re-prioritized so you can focus on what’s really important.
8. Build wireframes and interaction designs. Setting expectations on functionality, how it should look, and how it should behave reduces future UI defects.  
9. Design a prototype. Build something you can take with you to road shows that incorporates both visual and functional design. Update the prototype during future iterations so you always have an accurate portrayal of what the product will look like at launch.
10. Validate usability. There are hundreds of techniques, including a usability inspection, paper prototyping, and eye tracking, but put something in front of your users frequently.

Points 8, 9, and 10 should be repeated throughout all future iterations until the project’s completion. When user experience drives design, you build products people love to use.

So for those of you who may not have heard: The Andy Fastow Story is a far-fetched, weird and wonderful fantasy about a man who single-handedly (well almost…with a little help from a few close golfing buddies: Jeff, Ken, Michael, Rick, Ben and David, amongst others) brings down a giant company, nay, a whole industry, tailspins an entire country into recession, causes the loss of tens of thousands of job and billions of dollars of people’s savings, illegally bags tens of millions of dollars for himself and his best buddy Michael, etc… In this little tale, our hero Andy is magically transformed from an Ordinary Joe to Financial Genius with the help of his friend, and self-anointed god, Jeff. Andy then gets to make up “Andy’s World”, where everything is always wonderful for him and all bad things in the real world are shoved into a box called LJM and then a bigger box called LJM2 (and a couple of broken down pales called Chewco and Raptor). In Andy’s World, Andy and Michael get to make up all the rules. Everybody is invited to play in Andy’s World—at a fee. Everybody will win a participation prize. Everybody will lose a whole helluva lot. Errr, except for Andy and Michael of course—they always win. Until one day, when a nosy stranger starts asking questions like “Andy, how do you do it? …like, can you teach me how to stick really bad things into an ordinary box just like yours and have them just disappear and make me loads of money just like you?” Well, one thing leads to another and this nasty thing called the SEC starts poking and prodding Andy’s World, bugging Andy and asking him the same kinds of stuff; except this SEC thing is scary and has some gnarly militia working for it. The nerve of the SEC questioning whether Andy is on the up and up! Of course, just like all good fantasies, after a bout of the uglies (a little public humiliation and a few years outside the comfort of Andy’s World—chilling in an institution near Colorado), our hero rises at the end and gets to fight another day (presumably with some of those millions he bagged earlier). Unfortunately his friends, well except Michael again, didn’t do so well: poor Ken died, Ben and Rick went to jail, David became known as the world’s most infamous shredder, and Jeff was vilified around the world and got blamed for everything bad in Andy’s World. Oh, and, like millions of people’s lives’ got permanently messed up.

Pretty good story, eh?! I’ve even heard some rumors that it’s loosely based on real life…

Of course the moral of the Andy Fastow Story is that we don’t want another Andy Fastow story. Actually, no more stories at all of that genre. Sadly, it seems to be a trite plot prone to repeat. Who can forget the Bernie Ebbers or the Dennis Kozlowski Story? Or the Lehman and AIG Show? Don’t even start with Maddof! So many copycats, it’s hard to keep track.

The answer to stop these stories, one presumes, is to put in place some controls. Tools that support the controls. And, a scary, ugly and expensive editing process affectionately called the threat of eDiscovery. Oh, and a whole bunch of government type people, lawyers, accountants, bankers and consultants. Now, surely there will never be another Andy Fastow Story. There will never be another Arthur Andersen willing to edit another Andy Fastow Story. In fact, we won’t even need to have any more authors or heroes for these kinds of stories. Our tools will auto-document happy stories where everyone wins and everyone is treated fairly. So, that’s where ECM comes in!

Unfortunately, it seems, again, that reality is just not complying with our carefully thought out plot. Dang! In spite of noise to the contrary, and HUGE vendor propaganda promoting the discovery of gold reincarnated in the all signing, all dancing eDiscovery products tied to ECM solutions, it seems there is a “fiction gap” between is and would-like to be. In fact, records management en-masse (which one would presume to be a precursor to eDiscovery products) is seemingly not much more than email archival these days. All this seems to point to the obvious. What’s the subtext here?

Perhaps the subtext is in the incentives and disincentives. eDiscovery, empirically, seems to be a dance. The goal really is not full discovery or going to court—that would be insanely expensive for most organizations—but a happy settlement, which would be less insanely expensive. Of course to get to the settlement, you still need some level of eDiscovery…to know what is insane and what is less insane! Now, what if eDiscovery were easier? ECM, records management, search and eDiscovery tools all working in unison. A nirvana to be sure. But, would that not potentially lead to greater exposure of being discovered? Would the disincentive of the lengthy, costly and archaic discovery process now be replaced with the incentive of protecting yourself through the same lengthy, costly and archaic discovery process?! Certainly the US government (through the FRCP; Federal Rules of Civil Procedure) doesn’t hope so. In fact, since the December 1, 2006 amendments to the FRCP, effectively dealing with eDiscovery is very much supposed to be on the legal and corporate roadmap of most organizations; moreover they have the burden of responsibility for adequately managing discoverable information at their own cost.

So, where does that leave us? In the spirit of Mr. Fastow and company, I suspect somewhere between pay now or pay later…which is to say, little evidence of much competence at this time. In my opinion, effective and somewhat efficient eDiscovery is directly tied to successful deployment of ECM—properly indexing, managing, storing, retrieving and archiving information, and a streamlined implementation of Records Management—identifying, classifying, retaining and disposing of formal records based on corporate and regulatory compliance policies. I’m sure the eDiscovery vendors would not entirely, or possibly at all, agree with some of these assertions. But then again they may have incentives not to…

“What is a record?”     It is the fundamental question in records management. Without a clear answer to that question, one cannot determine record titles, develop and apply retention schedules, structure file plans or manage an “item” with software. The answer also makes the difference between establishing an “item” as a record, as opposed to being just a Word document, an email, a piece of paper in a file, or a map of a city.

The answer to the question begins with a definition. The primary definition for a “record” is stated in the ISO 15489: 2001 standard. It defines a record as

“…information created, received, and maintained as evidence and information by an organization or person, in pursuance of legal obligations or in the transaction of business.”

For the US federal government, the U.S. National Archives and Records Administration (NARA) is required by the Federal Records Act to use a slightly different and more explicit definition:

““records” include[s] all books, papers, maps, photographs, machine readable materials, or other documentary materials, regardless of physical form or characteristics, made or received by an agency of the United States Government under Federal law or in connection with the transaction of public business and preserved or appropriate for preservation by that agency or its legitimate successor as evidence of the organization, functions, policies, decisions, procedures, operations, or other activities of the Government or because of the informational value of data in them.”

Although this definition is slightly different than that in the ISO standard, NARA has incorporated the ISO definition into its regulatory guidance for federal agencies.

The result of these definitions is the segregation of items into “records” and “non-records” If an item does not meet the definition, it is not considered a record and, therefore, does not require any protection or preservation efforts – it may be disposed of when it usefulness for its owner ends.

If an item does meet the definition, it must also meet certain criteria to be considered a record. The criteria are designed to establish the trustworthiness of the potential record and are inherent to the definition. They are: authenticity, reliability, integrity, and usability. These criteria are defined as:

1. Authenticity: the record is a “true” item – an accurate representation of a transaction or activity, as purported by its creator at the time of creation

2. Reliability: the record can be relied upon to be a trusted source of the information it contains

3. Integrity: the record has not been and cannot be altered. It is complete in form and data

4. Usability: the record can be repeatedly retrieved throughout its lifecycle and used as an authoritative source of the information contained with in it.

If an item meets the definition, but fails to satisfy any one or more of the criterion of trustworthiness, the value of the item is suspect and cannot be considered a record. Thereafter, it is considered a non-record and can be disposed of at any time. Conversely, an item that meets all of the criteria of trustworthiness but does not meet the definition of a record, that item, too, is not a record. An example of this is a lunch menu for a nearby restaurant that is found in an employee’s desk. Although the menu may be a trustworthy representation of the restaurant’s bill of fare, if that restaurant is not connected to the employee’s organization, the menu is a non-record. (However, it could be a record for the restaurant.)

Often confusion arises in the application of the definition and criteria to an item when determining its records/non-record status. One major causes of this confusion surrounds the item’s media type. Another source of confusion is a failure to appreciate and evaluate the content and context of an item. Media type confusion arises from outdated perceptions of what a record have been. Until recently, many considered paper to be the only items that were records. The perception may have extended even to items preserved on microfilm or microfiche, but not much beyond that. The entire realm of electronic items was ignored. However, as ISO 15489 and the Federal Records Act make clear, any item, physical or electronic, can become a record if it is evidence of an entity’s business’ conduct and transactions. Therefore, potential record items include such things as paper documents, native electronic documents, scanned images of paper documents, structured data in a database, a chart, a photograph, a recording of voicemail, an instant message, a Twitter post, an email, or even, in the case of law enforcement, a gun, a knife, or DNA material. So long as the item meets the definition and satisfies the trustworthiness criteria, it becomes a record.

The confusion involving content and context is similar to that surrounding media type, as it concerns misinterpretations of how the definition and criteria are applied. One common misunderstanding is the assumption that all emails are records and should be preserved and protected as such. However, email itself is a media type, no different from a note, a letter, or a baseball bat. The factors that elevate an e-mail to record status are the content and the context of the email. At one end of the spectrum, an email between co-workers about where to have lunch, in most instances, would not be a record. At the other end, an email from a supervisor directing one of his/his employees to undertake certain job-related tasks clearly could be a record (assuming it also trustworthy). However, when context is applied with the content, it can change the status of the item. In the example of the email regarding lunch among co-workers, if business were conducted at the lunch, the email could become a record. If the is used by the organization to prove the time and date of the lunch for the Internal Revenue Service or other governmental entity, the email could become a record. Therefore, it is not the item itself that must be compared to the definition and criteria, but rather the content and context of the item in activity or transaction.

Any sound records management program is founded on the correct definition and trustworthiness criteria of a record. If the definition is absent, or is faulty, an organization’s entire records program can cause incorrect items to be treated as records. Every organization must, therefore, ensure that its record definition and the use of that definition are clear and unambiguous.

For more detail on this subject, please visit Armedia’s Resources page
and request the whitepaper entitled “Issues in Record Definition and Declaration.”

Solution-wise, this is actually good news for clients…in the longterm: richer functionality, better ease of use, tight focus on WCM, proven winner. Ultimately, WCM to EMC Documentum was the Super G to Tomba la Bomba: good, but not quite Aamodt-esque (apologies in advance for the indulgence in Winter Olympics rhetoric; seemed appropriate though given the wall to wall TV coverage and the three feet of snow we’re in here in DC). However, this direction does pose some intriguing questions:

• How tightly would Fatwire’s product work with EMC Documentum? Whose repository rules when it comes to WCM? Can the repositories truly co-exist merrily (possibly through CMIS)? Emmm…what happened to the whole one repository thing?
• What happens to clients’ existing Web Publisher (WP) based solution? Stick with it? Live without it? Migrate to Fatwire? Throw another 3^rd party tool in the mix? Start from scratch? Chuck the whole thing and Salesforce.com it?
• Operationally, who takes the lead, EMC Support or Fatwire? What about overlaps? How about licensing for existing solutions? Functionality vs. Risk?
• Technically, what’s the impact? What would the security model look like? How do you migrate?
• If the sunsetting of WDK and WP are both inevitable, what’s the recipe to upgrade? What tools and support will EMC provide to ease the process?

Of course, this is not an exhaustive list of questions and sharper minds than mine will surely have scripted-answers for them and more. History, though, tells me to tread with caution. Inevitably, enterprise class solutions are burdened with enterprise class complexity (and likely not just because of the technology). Time, budget, people, skills, political battlegrounds, management knowhow and many other factors may well play a part and tint the lens with which you view the questions.

So, where else might destiny’s child poke its nose? Of course, I don’t know for a fact, nor would want to speculate. However, looking at the gaps in the kitbag of some of the larger ECM vendors, I can’t help but wonder whether the next case management, records management, ediscovery, collaboration (read: Sharepoint) or content migration “partnership” announcement is just around the corner.

The case management system may not and probably should not be the evidence management system; however it should integrate with the evidence management system.  At a minimum, the case agent(s) need to understand the chain of custody of evidence associated with their case.  It would be better if it provided richer features like allowing the case agent to request evidence for charge out, reporting on state of evidence within case, electronic representation of evidence (i.e. picture) or etc.  The evidence administrator ensures that the evidence can be found when it is needed for whatever reason. 

The case management system should provide records management services. If deigned correctly, this will be transparent to the case agent.  As you clearly stated in “The People of Case Management”, the system has to make it easy for officers to document their investigation.  The records management system will ensure that the case adheres to the organization’s retention and disposition policies.  The records administrator maintains the file plan with its associated policies and enforces them during the lifecycle of the case. 

The evidence and records administrator supports the case agent by ensuring that physical and electronic material is tracked and available during the lifecycle of the case.

First and most importantly we have the case agent.  Case management is directed by a case agent or a team of case workers.  The case agent is supported by the organization and works within a framework of regulation and legal boundaries; still, the case agent is personally responsible for closing the case.

Every other benefit the organization gains from case management is derived from the case agent’s work.  Any case management system has to provide first-class support for the case agent’s needs.  Any new case management system has to be substantially better than the old system… No matter how bad the old system may seem, the case agents will have made it a part of their lives, and they will understand how to use the current system.  The introduction of the new system will make their lives harder in the short term.  The long term advantages of the new system must be very significant to make the effort worthwhile.  This is true even for electronic systems replacing a paper system.  Software developers think any electronic system is so superior to paper that they often neglect the hard work, thorough planning, and business analysis that goes into a deep understanding of the case agents and their day-to-day needs.

Law enforcement systems have to make it easy for the officer to document their investigation.  Software defect tracking systems have to make it easy for testers to enter defects and software developers to record the fixes.  Mortgage systems have to make it easy for underwriters to decide what to do.  Insurance systems have to make it easy for the adjusters to do the right thing.

Case agents have supervisors.  Supervisors are responsible for their case agents.  They have to know what cases each case agent is working.  Is each agent paying attention to the right cases?  Are the right cases assigned to the right agents?  Are the agents complying to the web of regulation and law and policy that affect their work?  Are the agents maintaining their case files?  Can I make a good case for hiring more case agents?  Any case management system has to make it easy for a supervisor to answer all these questions.

A case may involve a number of tasks that have to be fulfilled.  The tasks have to be assigned to people, usually by the case agent; and the task assignees have to report on the task status.  The assignee may or may not have access to all the details and background of the case file.  The case management system should make it easy for the case agent to create and assign tasks; for the assignees to receive the task; for the assignees to report on the task status; and for the agent to accept these reports.

The organization may have a separate group of people charged with mining value from the organization’s collective wisdom, as encoded in the repository of case files.  This is where electronic case management systems start to shine over their paper counterparts – if they are done correctly.  Analysts need to run reports, ad-hoc queries, and searches over the entire set of case files.  They may suggest changes to organization policy, point out non-compliance with existing policy, and propose changes in priority.  A case management system has to support analysts, without obstructing the case agents.

Any electronic system has administrators… it goes without saying that a case management system has to be ”administratable”.  I say this because, all too often, it does go without saying, and applications are released with no operational procedures, no way to edit lookup tables or drop-down lists, no backup windows, no upgrade path – in short, no joy for the poor underappreciated sysadmin.

In an interesting way, the public is sometimes a stakeholder in case management systems.  Most organizations have to prepare structured reports on their performance from a year-to-year or quarter-to-quarter basis.  How many crimes solved; how many defects closed; how many mortgages issued; how many claims resolved?  The public affairs officer needs accurate data from the case management system, without compromising ongoing cases.

Any organization considering a change to their case management system first has to understand who uses the system, how they use it, and whether the proposed change will make each user’s life easier.

Who uses your case management system?  Did I miss any prominent roles?

Case management obviously helps customers manage cases. So let’s back up and define a case.

A case is an incident of interest to some organization.  Law enforcement organizations have crimes.  Software development projects have defect reports.  Mortgage companies have loan applications.  Insurance companies have claims.  Crimes, defect reports, loan applications, and claims are numbered, tracked, and managed by these organizations.  They are all cases.

So we know what a case is.  A case also has stuff associated with it.  A case has people assigned to it; a history of actions taken by these people; tasks to be performed; relevant documents; a status; and a resolution.

The case agent focuses on the case and all its associated stuff.  That’s how crimes get solved, mortgages loaned, claims processed, and defects fixed.  The organization also pays attention to the collection of cases.  Data mining helps the organization improve.  How many cases did we process?  How much time to close a case?  Do we need to pay more attention to this kind of case, and less attention to that other kind?

Case management is how the organization manages a case and its stuff.  Case management is how the case agent defines, tracks, documents, and closes cases.  Case management helps the organization reflect on past performance and optimize future performance.

Nothing I’ve said so far implies anything about electronic support for case management.  Banks, law enforcement, and hospitals have managed cases for hundreds (even thousands) of years with no computers!

An electronic case management system should make life easier for the case agent, and give the organization more tools and better information.  It sounds obvious!  But you’d be surprised at how optimal paper processes can get, after decades or centuries of tuning.  I’ve seen applications try to replace one pencil scrawl with a dozen clicks, wizards, dropdowns, confirmations, and error messages.  I can say from certain knowledge that if a system is clunky, awkward, and hard for case agents to use – they won’t use it!  All the reporting, searching, information sharing, data mining, and metrics that organizations typically want – all of it is predicated on the case agent being willing to use the system to its full extent.

As my colleague has clearly stated in his blog, My name is NIEM, NIEM is not a silver bullet.  There are many issues to resolve:

1. Security of information once export from system.
2. Sharing of information between civilian and intelligence agencies – What if it contains data about US citizen?  How do we share with our international partners?
3. What if one system does not support all data elements and during the import data is lost?  Agency could be making decisions based on limited information.
4. When does the imported data become stale?  What if the case is active and new information contradicts the export data that has been shared?
5. Given the limited budgets of local and state law enforcements, how do they participate in collaborating to combat crime?
6. How do we combat data overload once data is being shared throughout the community?

These issues/concerns must and will be addressed because the alternative is not acceptable.  Our enemy wins when we allow distractions to take our eyes off accomplishing this goal of information sharing amongst the community.  As I review current and future procurements, I am glad to see the FBI, DHS and others make this a core element of their system.