The Electronic Journal of Information Systems Evaluation provides critical perspectives on topics relevant to Information Systems Evaluation, with an emphasis on the organisational and management implications
For general enquiries email administrator@ejise.com
Click here to see other Scholarly Electronic Journals published by API
For a range of research text books on this and complimentary topics visit the Academic Bookshop

Information about the European Conference on Information Management and Evaluation is available here

linkedin-120 

twitter2-125 

fb_logo-125 

 

Journal Article

IS Evaluation in Practice  pp169-178

Ann Brown

© Jan 2006 Volume 8 Issue 3, ECITE 2005 Special, Editor: Dan Remenyi, pp143 - 230

Look inside Download PDF (free)

Abstract

IS evaluation exercises continued to engender scepticism. The evaluation of IS investment is considered a 'wicked problem' and there are good reasons for this judgement. The topic has attracted many researchers. There is a substantial body of literature on the problems of measurement and the inadequacies of traditional investment appraisal methods. A wide range of alternative tools has been proposed to replace these approaches. But many surveys of actual practice have established little evidence of their use. Reported IS evaluation practice appears to be relatively unsophisti‑ cated or absent in many organisations. This paper draws on existing literature and case material to analyse the problem facing organisations when planning an IS evaluation exercise. It argues that the factors that can undermine the effectiveness of IS evaluation projects pose ma‑ jor problems. Management apathy may be a rational response to a complex and difficult exercise that often yields little benefit to the organisation.

 

Keywords: IS evaluation, Failure-prone decision process, IS Business value, IS evaluation project

 

Share |

Journal Article

A Holistic Framework on Information Systems Evaluation with a Case Analysis  pp57-64

Petri Hallikainen, Lena Chen

© Nov 2006 Volume 9 Issue 2, Editor: Dan Remenyi, pp45 - 104

Look inside Download PDF (free)

Abstract

This paper presents a framework for understanding IS evaluation in its broader context. The role of IS evaluation is emphasised on integrating the IS development process into business development process. The framework is applied to analyze a single IS project in details. The results show that sometimes formal IS evaluation might not be important or necessary, but rather it may be more important, with an informal and flexible evaluation process, to quickly gain experience of a new kind of business and system to maintain a leading position in the competitive market.

 

Keywords: information systems projects, IS evaluation, organisational context, holistic framework on IS evaluation

 

Share |

Journal Article

Economic Desirability and Traceability of Complex Products  pp201-212

Mordechai Ben-Menachem, Ilanit Gavious

© Nov 2008 Volume 11 Issue 3, Editor: Dan Remenyi, pp109 - 212

Look inside Download PDF (free)

Abstract

The real values and benefits of Information Technologies are difficult to quantify and frequently even to identify accurately. Existing financial models such as Net Present Value have proven insufficient for complex products, for long‑ term corporate goals. IS projects and software‑rich products are decided upon while ignoring critical financial aspects, as the distance between the corporate product vision and the reality that engineers see may be very large. This paper maps between economics vis‑à‑vis IS‑based product management via an inter‑disciplinary approach, looking at the needs and exigencies of corporate management, IS project, products and software engineering. The basis for the article is a discussion of the difficulties in evaluation of the economic desirability of complex, software‑ rich products. It presents a dynamic corporate‑level model for economic profit evaluation designed to deal with the unique characteristics of such products, over many variants and versions, and the entire lifecycle. Given the extreme uncertainty of costs, benefits, risks and timeframes projections, the model facilitates real time reporting via an information system designed for management of Products, Portfolios and Projects. Whereas existing project management techniques such as Earned Value Management provide a general basis for managing project level activity, our model provides a longer‑term view to assess economic affects of corporate strategies over time. This is provided by a dynamic, Management Information System based aggregation of all product information, over an entire product lifecycle, with the objective to provide a knowledge base for corporate dynamic decision‑making. Concomitantly, the model fulfils Sarbanes‑Oxley Act of 2002 requirements for management assertion traceability of valid and accurate measures. These aspects co‑joined, from Sarbanes‑Oxley, back through multiple products, over myriad versions, and through automated requirements, design and testing tools, all combine to form an auditable management feedback loop that can be leveraged at multiple corporate management levels. The paper represents a significant step towards quality product decision‑making via a model that is meaningful, while also useful as it is leveraged through an automated tool set.

 

Keywords: economic profit, information systems, IS management, IS evaluation, product management, project management, traceability

 

Share |

Journal Article

Evaluating Information Systems according to Stakeholders: a pragmatic perspective and method  pp73-88

Jenny Lagsten

© Jan 2011 Volume 14 Issue 1, ECIME 2010 Special Issue, Editor: Miguel de Castro Neto, pp1 - 166

Look inside Download PDF (free)

Abstract

In the last decade several researchers have addressed the problem that there does not seem to be much evidence of extensive use of interpretive evaluation approaches in practice. Researchers have though recognized the interpretive evaluation approach as well founded academically and theoretically offering potential advantages such as stakeholder commitment and learning opportunities. One reason for this non‑use could be that there are few, if any, interpretive evaluation methods ready at hand for evaluators in practice. An interpretive IS evaluation method means a method in support for doing evaluation as interpretation. This research presents a practical method for doing evaluation of information systems as a joint act of interpretation performed by the stakeholders of the information system in use. In our research we have expanded the interpretive philosophical base to embrace a pragmatic knowledge interest in order to underpin the overall strive for evaluation that is to contribute to change and betterment. The method presented is named VISU (Swedish acronym for IS evaluation for workpractice development). The process of evaluating accordingly to the VISU method has been extensively tested in practice and in theoretical grounding processes and is now considered ready for wider use. The research process for developing VISU has been conducted with canonical action research through parallel work with evaluation and method development in six episodes within two cases. VISU consists of prescribed actions that are anchored in a set of underlying principles stemming from the philosophy of American pragmatism. Evaluation according to VISU is performed in three phases; arrange, evaluate and develop. In the paper VISU is described according to phases, actions, main concepts and principles. The use of VISU is demonstrated through examples from a performed evaluation of an information system in support for social welfare services.

 

Keywords: IS evaluation, stakeholder model, interpretive IS evaluation method, pragmatism, action research

 

Share |

Journal Article

Research on IT/IS Evaluation: A 25 Year Review  pp276-287

Xingchen Song, Nick Letch

© Nov 2012 Volume 15 Issue 3, ICIME, Editor: Dan Remenyi, pp230 - 287

Look inside Download PDF (free)

Abstract

The conduct of IT/IS evaluation and its associated approaches, techniques and methods have been the subject of IS research for many years, particularly in the last two decades. This paper reflects on the body of knowledge which has emerged over the past twenty‑five years in order to identify where research efforts are focussed, what are the important issues in IT/IS evaluation research, and where future research efforts should be placed. This study presents a descriptive analysis of research on IT/IS evaluation over the last 25 five years, from 1986 to 2010, in five leading IS journals In total, 176 papers related to IT/IS evaluation are identified and reviewed in this study. Based on the Context, Content and Process model, IT/IS evaluation can be broken down to five interrelated elements: why evaluation is carried out, what is evaluated,whenwhen evaluation takes place, how evaluation is performed and who is involved in evaluation. Each of these elements are identified and classified in the sample research articles and based on this analysis, we propose a new perspective for classifying IT/IS evaluation approaches.

 

Keywords: IT/IS evaluation, literature review; Content, Context and Process (CCP) model, evaluation streams

 

Share |

Journal Article

Information System Evaluation through an Emergence Lens  pp38-47

Olgerta Tona, Sven A. Carlsson

© Jun 2013 Volume 16 Issue 1, ECIME 2012, Editor: Dr. David Sammon and Dr. Tadhg Nagle, pp1 - 84

Look inside Download PDF (free)

Abstract

Abstract: The development and expansion of evaluation theory and practice is at the core of several different disciplines. There exist different traditional Information System (IS) evaluation approaches, like experimental, pragmatic, constructivist, plu ralist and realist IS evaluation. IS evaluation approaches are influenced by the way they address to technology. Recently actor network theory (ANT) and sociomateriality are two influential information systems (IS) entanglement perspectives. Additiona lly, El Sawy identified three faces of IS views: connection, immersion, and fusion. In terms of IS evaluation approaches, connection and immersion view are the dominant views in which these approaches are positioned. We believe the IS fusion view calls fo r IS evaluation approaches to be revised. This paper uses the relational emergence theory, based on the philosophy of critical realism to theorize and operationalize the fusion view, as it lacks a theoretical grounding and as well to push forward the trad itional IS evaluation research approaches. At the core of relational emergence theory is the emergence concept, in which parts are structured by the relations among each other to create an entity as a whole. Based on this, we present and discuss the im plications for IS evaluation in terms of how to evaluate a process as well as the output of the process. The discussion on IS evaluation is illustrated through an empirical example, drawn on a longitudinal research study within a police organization. This paper concludes that in the fusion view, the evaluation process shall embrace a holistic perspective. The focus of the evaluation process shall be the emergent entity consisting of IS, users, task and processes structured by means of relationships among each other. The properties exhibited by this emergent entity shall be evaluated.

 

Keywords: Keywords: Information System evaluation, IS evaluation approaches, fusion view, IS views, relational emergence theory

 

Share |

Journal Article

Evaluating Success in Post‑Merger IS Integration: A Case Study  pp143-150

Maria Alaranta

© Jan 2006 Volume 8 Issue 3, ECITE 2005 Special, Editor: Dan Remenyi, pp143 - 230

Look inside Download PDF (free)

Abstract

Despite the importance of post‑merger IS integration to the success of the whole merger, post‑merger IS inte‑ gration literature remains scarce. This paper attempts to synthesise the often implicit or vague definitions of post‑merger IS integration success with those provided in the vast body of literature on IS evaluation. As a result, four categories of success issues for post‑merger IS integration are proposed: User satisfaction with the integrated software's system and information quality as well as its use; Efficient and effective IS integration management; Efficient IS staff integration; and IS ability to support the underlying motives of the merger.

 

Keywords: IS Integration, Mergers, Acquisitions, M&A, Success, IS Evaluation

 

Share |

Journal Article

Interpretative IS Evaluation: Results and Uses  pp97-108

Jenny Lagsten, Göran Goldkuhl

© Jun 2008 Volume 11 Issue 2, Editor: Dan Remenyi, pp51 - 108

Look inside Download PDF (free)

Abstract

One major reason for doing evaluations of information systems is to take actions based on the results of the evaluation. In order to make better use of interpretive evaluation processes in practice we need to understand what kinds of results such evaluations produce and the way that the results are used to be transformed into change and betterment in the organisation. We have developed, applied and studied a methodology in support for doing interpretive evaluation. In the paper we report the case of a performed action research study that has comprised an IS evaluation. Through this action research we have transformed the theoretical principles of the interpretive approach into a useful evaluation methodology in practice. The main emphasis in this study is on the results and the uses of the evaluation process. We make a brief theoretical overview of interpretive principles for IS evaluation and of the research on evaluation use, from the field of evaluation theory, and represent a framework for analysing influences from evaluation efforts. We use this framework to analyse and identify the results and uses of the performed evaluation in order to shed light on what kinds of results that interpretive evaluation may offer. We experienced the influence framework useful for locating and understanding the variety of results from interpretive evaluation processes. We conclude with a model depicting results and uses from interpretive IS evaluation processes. The main point we elaborate on in this paper is how evaluations influence the actions taken in the organisation in order to establish betterment. How people in the organisation use evaluation in order to establish betterment and change. Further we bounce back the insights on evaluation results and uses into the discussion on how to design interpretive evaluation processes and how to design evaluation methodology in support for those processes.

 

Keywords: IS evaluation, evaluation process, evaluation results, evaluation use, interpretative evaluation methodology

 

Share |