Bazaar for Version Control

One of the MATSIQEL RDM requirements is that software must support multiple versions of the research data. Due to data protection and ethical constraints, only some project partners may see all the research data, others may see and update the data, whilst others are denied access to raw (partially anonymised data) but may see the processed data.

These requirements, especially version control caused us to consider version control software as used for software development. Such software (as used in Microsoft's SharePoint) also gives differential user access rights and permissions. However, SharePoint is commercial product, so can not be recommended for unfunded work (i.e. when there is no funding for IT software/hardware).

Since most of the project team are not Computer Scientists software is needed that has the lowest possible barrier to entry, or it will not be used. That is, in addition to satisfying the technical requirements, data management software needs to be:

·         Easy to Use

·         Free

·         Multiplatform (Windows, Mac, Linux)

We examined several open source software source control management products to identify one to complement our case study.

Git is the most popular distributed version control system. Written by Linus Torvalds, it is used to manage development of the Linux kernel. It is (reputedly) fast, and allows free hosting on GitHub for projects that use it. Git however is designed for efficient software development and so saves versions of files as collections of incremental changes on a base file. That is, any particular version is assembled from pieces. This is counter to MATSIQEL requirements where versions of research data arrive externally and are not necessarily increments.

The other alternative evaluated is called Bazaar. Bazaar is version control software 'for everyone'. Sponsored by Canonical and used to develop Ubuntu Linux, Bazaar claims

·         "Version control for everyone

·         Work offline

·         Any workflow

·         Cross platform support

·         Rename tracking and smart merging

·         High storage efficiency and speed

·         Any workspace model

·         Plays well with others"

Bazaar is also a distributed version control system. This avoids central reliance on a single bottleneck and does allow multiple workflow styles. In particular, it is straightforward to set up a web based repository that end users can access freely and appropriately. Bazaar has several graphic clients that integrate well with Windows. Most ordinary users will be able to use bazaar version control and access to centrally stored data with minimum impact on their usual workflow.

Data under Bazaar version control is simply stored (invisibly) in subdirectories. Graphic version histories are readily available. Also, since Bazaar does not exploit proprietary storage mechanism, a bazaar repository may be zipped, archived (e.g. in Sharepoint), and revived intact as needed.

Bazaar may be simply configured on cloud based web servers, which may be set up with the kind of access controls needed, granting differential access rights as needed by the MATSIQEL project. In summary Bazaar is a  multiplatform product that fulfils requirements for research data management in our case study project MATSIQEL since it supports repositories.

Posted on behalf of Jeremy Ellman

Comparison of IT infrastructures

The DATUM in Action project piloted two IT infrastructures: (i) use of existing standard office software and a secure shared network drive; (ii) an experimental prototype collaborative infrastructure environment – setting up a team site in a paid-for, cloud-based SharePoint service.

The requirements for the EU Team were:

• Access / data sharing by researchers in different institutions / countries (all within the EU or with safe harbour agreements)
• A filespace
• Access rights to be set up at the folder/sub-folder
• Automatic version control of files
• Automatic application of retention periods
• Email system
• Project wiki, for researchers to collaboratively develop documents/presentations
• A public-facing blog for dissemination
• Public-facing website

We have compared the way the two infrastructures met these requirements. Note: we are still testing the SharePoint prototype.

(1) Access / data sharing by researchers in different institutions / countries (all within the EU or with safe harbour agreements)

Standard office IT facilities

• The shared drive is accessible only to Northumbria University staff. The University is reluctant to give access to external people (understandably from a security viewpoint). Data sharing is by anonymisation of data and use of encrypted files through services such as Dropbox, and the use of encrypted laptops and data sticks.

SharePoint prototype

• Being a cloud service, access to all project researchers is easily arranged by giving them ids & passwords. The number of people able to use the site is governed by the price paid. However there are issues with data protection: the country of origin of the cloud service provider needs to be in the EU or have a safe harbour agreement. And how secure is the service? You would expect the cloud service provider to offer the same level of security as a University, but what access do the provider’s IT staff have to the data? And how are back ups handled if the provider goes bust? It is recommended that when using a cloud service provider a service level agreement should be drawn up. If SharePoint was made available by the University than the same external access issues would occur as with the standard office IT facilities.

(2) A filespace

• Standard office IT facilities & SharePoint prototype: exactly the same

(2a) Access rights to be set up at the folder/sub-folder

Standard office IT facilities

• This can be done, but scope is limited

SharePoint prototype

• Far more scope available

(2b) Automatic version control of files

Standard office IT facilities

• This has to be done manually, by adding version numbers to file names

SharePoint prototype

• Very flexible, detailed automatic versioning, with the ability for the site administrator to customise this

(2c) Automatic application of retention periods

Standard office IT facilities

• This has to be done manually, via the use of sub folders containing files of a given category: the whole sub folder can be deleted when required.

SharePoint prototype

• This is still under test

(3) Email system

Standard office IT facilities

• Outlook

SharePoint prototype

• Same email software, however in the SharePoint prototype this is integrated, so for example when updating files other people can be alerted by email that a file has been altered. This enables workflow processes. An email facility is a higher price option.

(4) Project wiki, for researchers to collaboratively develop documents/presentations

Standard office IT facilities

• A free service by a cloud provider would have to be used

SharePoint prototype

• The wiki is integrated with the team site

(5) A public-facing blog for dissemination

Standard office IT facilities

• A free service by a cloud provider would have to be used

SharePoint prototype

• The blog is integrated with the team site. A public blog facility is a higher price option.

(6) Public-facing website

Standard office IT facilities

• Pages set up on the University’s website

SharePoint prototype

• A website is integrated with the team site. A web facility is a higher price option.

SharePoint offers much more functionality (including automatic versioning for example), and integration of different facilities. However, this comes at a cost. Use of a cloud-based service would require funding, e.g. as an item within a proposal budget. Universities, who have not already done so, could set up SharePoint implementation, either across the institution or for a specific activity such as research. Basic SharePoint comes with the academic site license, however staff resources would be needed to set up and run the implementation. There is also the barrier to entry, i.e. all researchers would need to learn how to use a new system, and how to set up a team site. But this barrier is not high. It could be likened to the adoption of VLEs within universities. Initially there was opposition from some academic staff, now all staff use them as a standard system. The VLE is set up with standard module templates, and help guides and training is available. Similarly in SharePoint, a research project template for team sites could be made available which could then be customised by researchers for specific projects. However, uncontrolled use of team sites could lead to SharePoint sprawl.

Standard office IT facilities may represent fit-for-purpose supporting infrastructure for managing the data of much research that is conducted HEIs. HEIs / researchers should assess whether or not investing in a sophisticated system such as SharePoint is necessary. Are the benefits of adopting it for a research project great enough to outweigh the costs (financial, training, development) if the system is not already implemented?

DMP Template

The DATUM in Action project developed a customised data management plan (DMP). Initially we used the DATUM for Health DMP template which was developed from the DCC’s template (DCC Checklist for a Data Management Plan Post-Consultation (v2.2: 6th January 2010) https://dmponline.dcc.ac.uk/documents). However, feedback from the EU researchers, after they tried to complete this, showed that we had to radically alter the design. In summary this new approach was based on 3 main themes:

(1) to focus on the researcher and what is needed to help them conduct their research project on a day by day basis. The focus has moved away from data curation of shared data after project completion.

(2) to reduce bureaucracy, i.e. DMP lite. The front pages of the DMP enable the researcher to select the sections that are currently applicable to them and ignore the rest. Additionally, to reduce duplicating information held in other systems/documents the researcher is just asked to give the location of the relevant document (either a folder on a computer drive or a physical location for paper-based items).

(3) to embed decisions and actions so the DMP becomes a living document for the duration of the project.

The DMP template has been posted on the project website:

http://www.northumbria.ac.uk/sd/academic/ceis/re/isrc/themes/rmarea/datum/action/outputs/

Guidance for completing the DMP is currently being written and will be posted on the project website. 

Information Security Guidance

This has been posted on the project website:

http://www.northumbria.ac.uk/sd/academic/ceis/re/isrc/themes/rmarea/datum/action/outputs/