Thursday, October 1, 2009

Published Scientific Paper Evaluation 2

I am going to evaluate the paper entitled “Usability and Open Source Software” reviewed by Ate Chen-Chen. This is already a familiar topic to me because of our Elective 1 subject, which is about Free and Open Source Software. We have also touched the issue of usability of Open source software so I think this paper will be able to give us more information about that topic.

The paper studies the usability of open source software and suggests possible improvements for in usability. The paper first reviewed the existing evidence of usability of open source software. Then, it outlined how the characteristics of open source development influence the software itself.

It started with actually asking if there IS an open source usability problem – which, I think, is just appropriate because the study would be insignificant if there never was a problem with open source usability in the first place. With the help of the Internet, open source development was facilitated easily among developers and open source software is quickly gaining popularity. However, most of the users of open source applications are relatively more technically sophisticated that the average desktop user that uses standard proprietary software. According to them, there are many factors that may have contributed to this scenario, but the focus is on usability.

Usability is typically described in terms of five characteristics:
1. Ease of learning
2. Efficiency of use
3. Memorability
4. Error frequency and severity
5. Subjective satisfaction
It was stated in the paper that all software has usability problems, but the issue is: how does software produced by an open source development process compare with other approaches? The study was unable to arrange a controlled experiment to compare the alternative engineering approaches but it was able to compare similar tasks on existing software programs produced in different development environments.

Comparing open source and proprietary, since open source is public, the past versions are visible and you can see the progress but in proprietary, these are restricted and when they come out, they are already complete.

The study cited the development process of open source software that may have some influence on the poor usability of open source software. They used two main sources for this. The first is that the developers are not users. This problem in the development process refers to open source software being made by experts for experts. But then, they are also now being used by non-experts who could not see the usability of the software because it may be too technical already for the majority of them to comprehend. This flaw in open source development has been mitigated in commercial software where it recognized the non-technical user’s point of view and employed methods to balance their software in favor of users. The next problem the paper cited is that usability experts do not get involved in open source development. Another is that open source development tends to aim more at functionality than usability. This is only good if all users are technical experts. The next is that usability problems are harder to specify and distribute than functionality problems. I think this is the same as the saying “What is moral to me might not be moral to you”. In other words, it is difficult to define the exact standard of usability because of the varying technical capabilities of users. Another factor is the lack of budget for open source development. Since it is open source, it is mostly voluntary work and so open source developers work on small budgets. The prominence of commercial software also contributes to the poor usability of open source.

There are more issues in the open source development that were cited in the paper, however, I will skip those parts and focus now on what the study suggested as potential approaches to improving open source usability. From all the approaches the paper cited, I think that involving the end users is the best idea. After all, they are the ones who are going to use the software so they should be involved in some way in the development process.

That ends the summary of the paper – at least, if I were to summarize it. Now for the evaluation, the first thing I noticed is the title. It is very short – only four words, excluding “end” – but it clearly suggests what the paper is all about.

In terms of content, the paper is very informative. It provided the readers with the existing usability issues and suggested possible improvements. In terms of format, I do not think it followed the APA format but it is still acceptable because the flow of the paper is good and understandable. The abstract is also very short – only one paragraph – and I think, based on what I learned, is somewhat lacking. The abstract should give the readers an idea of what problem the study is trying to solve, what methodology it is going to use, and a summary of the findings. The paper stopped in defining the problem.

I also noticed that there is no formal methodology section in the paper and I honestly was confused what kind of methods they used. They did not clearly state how they were able to get the existing usability issues – did they conduct a survey, an experiment, or something? There were also no methods describing how they came up with the possible approaches. From what was given in the paper alone, I guess they just conducted an exhaustive literature review and did not actually do any technical experiment or something along that line. However, I would also like to commend them for the many sources they had for the entire study. I was only disappointed in the “usability and open source development” section because they heavily relied on only TWO references, which I think is not enough, considering that open source is very diverse and there may be a LOT of factors influencing the open source development process. Even in the possible approaches, they also only did something like a literature review. In fact, the whole paper felt to me like a long literature review because there were no formal methods described.

However, it was good that they discussed future works and I was glad to read that they suggested some deeper study about the usability of open source software. The study is only like an introduction to a series of more studies, hopefully more technical ones. They also presented some issues that also need further research.

Overall, I think the paper is average and this decision may have been influenced by the fact that I was expecting something technical from it and it failed to meet that expectation. Very Happy

No comments:

Post a Comment