Objectives

The objective of the case study is to get detailed information about the functionality of the Bibster and SWAP architecture. For this purpose the actions from each client and the network traffic for each query will be logged.

One the one hand, logging the actions of each client is required to get a particular result of the functionality of the system. On the other hand network traffic will be recorded to prove the efficiency of a semantically based peer-to-peer network.

The collected data will make it possible to evaluate several technologies and algorithms developed for Bibster and SWAP. For example the semantic based peer selection and duplicate detection can be benchmarked.

In detail we are searching for answers of the following research questions:

We would like to know if our system improves knowledge sharing and if it supports community awareness and formation.

SWAP aims to increase knowledge sharing and community activity in decentralized settings. The bibliographic scenario in specific is a case of a Distributed Communities of Practice (DCOPs): the system will connect groups of scientists active in the same area, but working at different geographic locations and affiliated to independent organizations (universities and research institutions).
While the Bibster tool may provide benefits by local use as well (i.e. installed on a single computer), we’re specifically interested in sharing between users. Less value is attributed to sharing between users and passive peers storing libraries of bibliographic data (such as DBLP) and to sharing activity within organizations, since such scenarios are already possible using alternative methods. With respect to community awareness, we’re interested if the system is able to create weak ties, i.e. "connect" scientists who have not known each other previously and were unaware of each other’s work.
Since the question relates to improvement, it is also important to establish a baseline for measurement. Alternatives usually refer to the way things were done before the introduction of the system (as identified during requirements analysis). Ideally, one would like to show an improvement in both the time required to perform a certain task and the quality of the work produced. In most cases, however, time saving is much easier to measure and can be more compelling for stakeholders, because it can be directly used to calculate ROI (savings in money terms).

We would like to find out how our system and certain features are valued by the user.

This question relates to the notion of user satisfaction. Since there is no organization behind this case study that would determine overall business objectives for the case study, the satisfaction of individual users is an important goal on its own.2 The answer to this question is also important for the next cycle of system development, when the features criticized by the users can be improved. Note that satisfaction may depend on the circumstances and characteristics of the user himself and to give context to these answers it is important to profile the users and their technical environment. This can also help to track down the reason of dissatisfaction (by clustering the population) and target the problem in a later version of the system.

We would like to know how ontology use benefited the system.

This question relates to the particular innovation of the SWAP project, namely adding ontology-based features to P2P systems. It is a specialization of the previous question, proposed for evaluation by the Technical Annex of the SWAP project. Ontologies are employed at several places in the system, of which we focus on peer selection and duplicate detection. With respect to peer selection, we would also like to validate the results of earlier experiments done with a simulation environment.

We would like to explore unknown properties of the system, such as the topologies

This question is motivated by scientific interest. Since SWAP pioneers the combination of semantic technologies and peer-to-peer, little is known about the physical, semantic or social topologies emerging through natural system use. This knowledge can be valuable input for designing better algorithms with respect to peer selection (semantic routing). Moreover, this question cannot be answered through the existing simulation platform.