openSUSE Development and Integration Process (Details)

From Roadmap to Beta1

Version 1.0.3 - 29/08/2013

...

The Factory Development Model is explained here. For an overview of the development process see this page.

1. Roadmap and Feature Planning

Purpose

• To create the initial time-line for the release: the roadmap
• To identify the main technical goals for the new release

Entry Criteria

• E1.1 Previous release version of the distribution was released

Tasks

• T1.1 Create the roadmap description. This task is done taking the last roadmap as a template and working around the holidays in China, Germany, Czech Republic and USA. A detailed description about the rules used to build the roadmap is in [15]. The final roadmap need to be written in XML using the schema from [4]. This roadmap contains all the information from T1.1 and T1.2. This XML is published using the SIMILE JavaScript program [5] to create a time line web page as the official reference. In parallel with this, the roadmap is published in the wiki from the project [1], and maintained during the release process, linking the new milestones when they are released. The important dates (roadmap and freezes) are made explicit and easy to consult for users and developers.
• T1.2 Feature freeze calendar. Together with the roadmap calendar is the associated feature freeze calendar [1]. There are three stages of freezing: toolchain freeze after M3, base system freeze after M4 and full feature freeze after B1. After that the maintenance process starts. Freezes are scheduled at the end of milestone development periods to give time for development after a milestone release. They should not be placed too close to the next release as the frozen components should get a little testing before the milestone is out.
• T1.3 Determine the new features. On the mailing list (as the primary channel for discussing ideas and features) and through other community communication channels the new features expected for this release are discussed and defined. Some of those features can be documented on openFATE [6] (not widely used) or the wiki [3]. The features can be decided inside the community (in this moment this is done only by a really small number of contributors) or internally in SUSE when there is a relation between SLE and openSUSE: a developer can create a feature in FATE for SLE and then add openSUSE afterward (this kind of feature is not visible outside) At this moment we do not have a strict policy on using openFATE. Some features are also decided later without planning and documentation. If the feature is implemented after the freeze calendar, an explicit approval from the Factory Maintainer is needed for the integration.

Verification

• V1.1 Review the roadmap. The roadmap needs to be approved by the Product Manager and the Release Manager.
• V1.2 Roadmap maintenance. Small changes in the roadmap are not communicated, just executed. But if a delay threatens the release day, an announcement is expected, and if the delay is serious we need to communicate it widely [17] [18]. In every case T5 is involved, update the wiki.

Exit Criteria

• X1.1 The roadmap and the freeze calendar are published in the mailing list

Deliverables

• D1.1 XML Document where the roadmap is described [4]
• D1.2 The features documented in openFATE [6] and in the wiki [3]

2. Package Development

Purpose

• Create, update or fix packages in the devel projects or in Factory
• Be the starting point of the development process
• Develop and experiment without interfering with the devel projects and Factory packages

Entry Criteria

• E2.1 The developer is creating a new package that is not in Factory or in devel projects, but has the intention to integrate the package in the distribution
• E2.2 The developer wants to update a package that is currently in the distribution
• E2.3 The developer wants to fix a bug in a package that is currently in the distribution, in response to a bug report in Bugzilla or another channel [19]
• E2.4 There is a message from the messaging system (Hermes) [20] [21] describing an action that needs to be done in a previous submit request

Tasks

• T2.1 Create local package. If the package is new (does not yet exist in a devel project or in Factory), the developer creates a new package using the osc command line tool or the web interface. The creation of a package is a complex process well documented [22] and with policies which the developer must follow [23]. If the new package is stored in CPAN, Pypi or is a Ruby gem, there are scripts that help during the package creation, but in either case it is the packager who needs to submit the package.
• T2.2 Branch Package. If the package currently exist in one of the devel projects and /or in Factory, the package developer branches the package from the original project to the local home. This process is similar to copying the files, but a relation between the new copy and the original package is retained (stored in some metadata files). The developer still needs to follow the general policies referenced in T2.1
• T2.3 Fix / Update Package. The real work of the developer is expressed with this task. The package developer now can fix the bug, create a new feature or update the package. After the work is done, the developer submits the changes to OBS to launche the process described in "V2.2 Automatic checks complete successfully".
• T2.4 Submit to Devel Project. Once the package is created or fixed/updated, and V2.2 is completed successfully, the package developer creates a submit request for the Devel Project (see Glossary to understand the relation with Devel Project and Factory) to get the package included there. If he / she doesn't know which Devel Project is related with the packages (maybe because is a new packages), the Package Developer need to ask in opensuse-factory[47] mailing list. The Factory Maintainer can create a new Devel Project in case that there is not a good match for the package, or more frequently, point to the already existing Devel Project.
• T2.5 Automatic Submit. There are external scripts which update certain packages automatically. Those scripts are related to some very specific Devel Projects and are not generally used. When run they act like a Package Developer, updating the package and submitting directly to OBS (V2.2)

Verification

• V2.1 Is in Factory. The work flow of creating a new package for future integration into Factory is different from the work flows for fixing bugs or updating packages that are currently in Factory. With this verification point the developer makes an explicit decision to change the path of the work flow.
• V2.2 Automatic checks complete successfully. Every package needs to be committed in OBS which will build the package and run some standard tests. If compilation is successful and the package passes the tests, the developer can create a Submit Request.

Exit Criteria

• X2.1 The developer creates a submit request in OBS. This submit request will be received by the Package Maintainer or the Project Maintainer of the package.
• X2.2 The developer drops the change, aborting the development process.

Deliverables

• D2.1 Submit Request. This document is a digital document created by the Package Developer when the package is successfully created in OBS and deemed ready by the Package Developer. This document is used by the Package Maintainer and the Project Maintainer to integrate the changes of the new version of the package in the devel project.

3. Devel Project Integration

Before a package can be integrated into Factory, it needs to belong to a devel project. A devel project is a set of related packages that share certain features. For example, in devel:language:python are stored all the packages related to the programming language Python. LibreOffice:Stable is the devel project where are stored the packages of LibreOffice updated to the last stable version.

Purpose

• Provide a space where the main development (integrating feature releases) takes place.
• To guarantee a certain level of quality and integration before the package is finally integrated into Factory. This integration process follows the cycle checkout – fix – commit – submit request, very similar to the process described in 2. Package Development.

Entry Criteria

• E3.1 A submit request from a Package Developer to a devel project.
• E3.2 A decision by the Project Maintainer to make changes to a package directly in the Devel Project

Tasks

• T3.1 Integrate the submit request. The Project Maintainer can integrate the code coming from the submit request after the review is done in V1. Usually this integration is done using the OBS web interface or the osc command line tool. The Package Maintainer can only accept and integrate the submit request for his / her own packages.
• T3.2 Package checkout. The Project Maintainer has two options for working with the packages part of the devel project: one is use the exit criteria and branch the package in his/her own home project (changing roles into Package Developer). The other option is checkout the package directly from the devel project and submit a change there. The former option is more secure and clear, as it avoids collisions with new submit requests. The latter one is a faster approach more adequate for projects where the number of maintainer and developers is low. To help in this process, some devel projects use extensions to the OSC command line tool. For example, the GNOME project uses the plug-in [24] "collab" [25] to allow advanced workflows.
• T3.3 Fix / Update Package. The real work of the developer is expressed with this task. The Project Maintainer or the Package Maintainer can work, using as input the bugs from bugzilla [19], or the features planned for their own project.
• T3.4 Submit to Factory. Once the Package or Project Maintainer considers the package(s) in his project good enough he/she creates a submit request for Factory.

Verification

• V3.1 Review of the submit request. Before the integration, the Project Maintainer (or the Package Maintainer if the request is related with his / her own package) checks the request manually, to detect problems in the spec file, in the patches attached or in the compilation process. If no problems are found, the request is accepted and integration takes place (T3.1), otherwise the request is declined and Hermes sends a message to the creator of the submit request. Some devel projects have an internal review process, but there is not a shared policy by default.
• V3.2 Compile in OBS. Every package is committed to OBS. OBS will build the binaries according to the spec file [26]. If the compilation is successful, the Project Maintainer can create a Submit Request and send it to Factory or continue the integration process with the rest of the packages.
• V3.3 Automatic Security Review. For certain packages that use SETUID binaries or offer D-BUS services or PAM modules, an automatic security review takes place. The check is executed in the rpmlint phase of the package creation and if one of the security policies applies [27] the package can't be compiled. To lift the restriction on building the package the package maintainer must contact (open a ticket) the Security Team for review. As long as the package is not in the white list, it can't be submitted to Factory.

Exit Criteria

• X3.1 The Package or Project Maintainer creates a submit request for Factory.
• X3.2 The Package Maintainer decides to change role to Package Developer and branch the package in his/her home project.

Deliverables

• D3.1 Submit request to Factory. A similar submit request can be created and sent to Factory. The Project Maintainer can group a set of submit requests into one single request which can be accepted or declined as a whole.

4. Review Process

The review process is only done for Factory.

Purpose

• To improve the quality of the submit requests.
• Potentially simplify the integration of the request into Factory.
• To detect the most usual errors in: legal, technical (intro-package and inter-package) and security aspects.
• Avoid the problem of accepting interdependent packages. With the review you can assure that before you commit into Factory, all the related packages meet a certain level of quality and follow certain policies, preventing Factory breaking due to missing packages if some of them are not up to the quality standards.

Entry Criteria

• E4.1 A submit request is created by a Project Maintainer for Factory [28] [29] [30]

Tasks

Because this review process is a full verification step, this process is composed only by verification steps, not tasks

Verification

• V4.1 Legal Review. The legal review is split in two parts: (1) the script Factory-auto [10] checks if the license(s) in the package have changed or not. If there is no change and the license is in the license database this script will accept the request. (2) If the package contains a change of the license, or is a new package with a license not found in the license database, the legal team has to manually check the request and accept or decline.
• V4.2 Factory Auto Review. Factory-auto is a script [10] which does a shallow technical review, trying to find the most common problems usually found in requests. For example, this script checks if the associate package is currently successfully compiled in OBS. This check is fast and is used to decline the most evidently bad quality requests.
• V4.3 Technical Review. The review team checks the more complicated technical aspects of the request. The Technical Reviewer will check if the specification file follows the set of predefined policies [23] [26], look over quality of the code in patches and verify the content of the changelog file.
• V4.4 Check Repo Review. This review is also automatic. Is executed by the check-repo script [10] and checks for typical integration problems. For example, this bot checks if certain dependencies are present in the Factory project, that the dependencies from Base:build are all self-contained in this same special project, or that the new group of request doesn't create new problematic dependency cycles in Factory.

Exit Criteria

• X4.1 All the reviews are positive.
• X4.1 If there is a negative review from one of the verification points, the request is declined and Hermes sends a notification to the sender of the request. Some automatic checks flag the request if it fails but do not completely reject the package. If this is the case, an action is needed from the submitter so it can be considered as an exit criteria. OBS also allows comments on the request, some of which need an action to be taken by the submitter.

Deliverables

• D4.1 The review process changes the status of the request, showing at every moment what reviews are pending, accepted or declined.

5. Factory Integration

Purpose

• Bring all packages together in one tree, creating a consistent whole
• To be used as the base for the new products ISO image
• Ensure packages flowing into Factory follow feature freeze and timing rules
• Ensure consistency of other branches of Factory (milestones, Beta's, RC's, Release)

Entry Criteria

• E5.1 A submit request has been submitted from a devel project and has gone through the review process
• E5.2 The Factory Maintainer decides to fix an issue inside Factory

Tasks

• T5.1 Monitor Factory. On a periodic basis, the Factory Maintainer checks how Factory is doing, and if there are any problems (which solution is outside the scope of the Factory Maintainer role) he / she sends reminders to the correct roles, or reverts the packages that causes the problems. The Factory Maintainer uses three types of monitors: the Build Monitor showing the current stage of all process (failing packages) [31] [32], the Submit Request queues [33] and the Users.
• T5.2 Maintain XML files used by OBS to generate ISO images. The Release Maintainer updates and fixes the XML needed by KIWI [34] to generate the ISO images for the products. This process is not currently documented.
• T5.3 Integrate the submit request. The Factory Maintainer accepts and integrates the submit request from the devel project to Factory. In order to avoid an overload of the OBS servers, the Factory Maintainer accepts the request following some basic rules based on experience (as in, not documented) [9]:
  1. Do not accept a SR when a new build is in process.
  2. If a leaf package is accepted, OBS will trigger an small subset of packages.
  3. Base or core packages can lead to a lot of re-compilation in Factory. These are usually delayed until the end of the week.
• T5.4 Revert or remove packages. If the maintainer of the package does not respond in time or if the package causes great pains, the change can be reverted or the package removed.
• T5.5 Issue notification. The Factory Maintainer detects and notifies the Packages or Project Maintainers when there is an issue related with the package in Factory.
• T5.6 Create staging projects. To simplify the integration process and issue detection the Factory Maintainer can create staging projects. A staging project usually contains a new important package (like GCC or Perl) and all the dependencies and otherwise related packages. In this way the maintainer can detect new issues related only to the new package, avoiding issues caused by other integration actions or updates taking place.
• T5.7 Create Patterns. The Factory Maintainer creates and update the patterns needed for the installation [36]. The patterns format and purpose are described here [35] [37]

Verification

• V5.1 Is Factory open for submissions? Check for freezes or other reasons not to accept SR's
• V5.2 Brief review. The review process filters low quality submit requests, so the Factory Maintainer does not verify quality. Instead he/she judges if this is the proper time to accept the request and start the build process. The Factory Maintainer usually does a simple review of the request, and can decline or put it in hold if:
  1. The submit request is (related to) a package which is currently frozen AND the package is not fixing an issue
  2. The package will break so many other packages that a staging project will be needed

Exit Criteria

• X5.1 The package is integrated in Factory
• X5.2 The package is declined and a event in reported in Hermes
• X5.3 A bug report is created in bugzilla

Deliverables

• D5.1 Package integrated into Factory
• D5.2 A working Factory repository

6. Quality Assurance Process

The QA process can be done automatically using the openQA [11] [12] tool, but also can be done manually inside a virtual machine or a real machine.

Purpose

• Determine the quality of the product generated after T4. Factory Integration.
• Test ISO images which are candidates to become Milestones or Beta 1
• Try to identify real bugs (integration bug or package bugs) in the ISO build (not FIX problems)
• Quantify the final quality of build, detect errors and update Bugzilla with the new errors to be fixed.

Entry Criteria

• E6.1 There is a new automatic build generated by OBS
• E6.2 The Release Manager or the Factory Maintainer has decided to test a concrete build image, candidate as a final product (Milestone or Beta)
• E6.3 The QA team has decided to test a new feature or reproduce a bug reported in BNC [19]

Tasks

• T6.1 Create a new test. If the QA Team decides that the current coverage of the test set is not enough, they need to write a test that covers the new feature / use case.
• T6.2 Test maintenance. The way to make tests for Factory is using openQA. This tool requires the maintenance of certain components named needles, used in the tests to assert outcome conditions [46]. These needles need to be updated and adjusted when an important change in the artwork or other areas effecting the look and behavior of the distribution occurs.
• T6.3 Create bug reports. openQA detects problems in the distribution, but it does not specify the exact issue which affects the ISO image. The QA Reviewer will dig into the failed test, deduce what is wrong and document the findings in a bug report published in bugzilla. This bug is assigned to the correct developer to fix the bug in the T2. Package Development task or in T3. Devel Project Integration.

Verification

• V6.1 Test review. The tests are grouped according to small set of criteria: the phase of the test (installation process, text applications or graphical applications) and the relative importance (critical, important, normal). The tool can mark one ISO as failed if critical tests are not successful passed, or if certain tests fail in the installation process. If the final result in openQA is not green the ISO has to be debugged and a bug report be written (T6.3)

Exit Criteria

• X6.1 The important and critical test are green and the ISO is considered candidate for a Milestones or Beta release
• X6.2 An important bug is detected by the tool in some configuration and the ISO is rejected as a viable product candidate.

Deliverables

• D6.1 Test report. The application generates a test report with screen-shots and status of the test. This report is reviewed by the QA Reviewer.
• D6.2 Bug report. If a bug is found (critical or not), the QA Reviewer creates a bug report with the description of the bug and assigns it to the correct maintainer of the package.

7. Release Process

Purpose

• Make Milestone/Beta images available for testing [38] [39]
• Finish the process

Entry Criteria

• E7.1 It is 2 days before the chosen release date

Tasks

• T7.1 Update roadmap. The Release Manager adjusts and communicates new roadmap.
• T7.2 Build Images. Using KIWI, OBS creates ISO images that are specified in a XML document. Different products (ISO images) have different XML documents.
• T7.3 Generates signatures. The autobuild team generates signatures in the name of SUSE and generates a delta ISO against the last milestone
• T7.4 Calculate hashes. Release manager calculates the ISO MD5/SHA1 hashes of the images
• T7.5 Create list of changes for marketing. Release manager generates differences between current and previous milestone with dvddiff script and sends it to marketing together with other important notes. The resulting list is shortened to a more manageable number (between 10 and 30) based on the importance of the updates.
• T7.6 Branch sources for Beta. In case of Beta, release manager branches all the sources in Factory off. When the branch is made, Factory is open for development again.
• T7.7 Create announcement. Marketing manager creates announcement based on diff and the template on progress.o.o [40]
• T7.8 Sync image to mirrors. Release manager syncs the image and binary repository to the mirrors [41]. Release manager notifies mirror admins about the new iso's and the proposed release date by mailing mirror@opensuse.org. Release manager publishes the image on the mirrors (making files readable)
• T7.9 Update Staging. Release manager updates staging area, metalinks for bittorrent
• T7.10 Enable product in NPP. Release manager enables the new milestone as a product in the Novell Product Page in bugzilla
• T7.11 Publish announcements. Marketing manager publishes announcements

Verification

• V7.1 Pick image. Release manager decides, based on openQA output, that a potential release image is good enough. If the release is an early milestone, the Release Manager can decides to release using a different criteria than openQA output. The closer to the release, the more the Release Manager raises the threshold for releasing.
• V7.2 Discuss roadmap for Beta. If this is Beta, a meeting with the product manager and Management is set to discuss the state of Factory and the roadmap. There is a go / no-go meeting for the final release with the Release Manager, Controller, QA Reviewer, the Product Manager and management where it is decided to delay or not.
• V7.3 Check communication. The Release Manager checks if the delay (if any) is big enough that it has to be communicated
• V7.4 Check Sync state. The Release Manager checks if the image is synced and published to the mirrors

Exit Criteria

• X7.1 Release is available and announced

Deliverables

• D7.1 ISO images for all supported platforms and architectures
• D7.2 Release announcement text for news.o.o and mailing lists
• D7.3 In case of delay: announcement and new roadmap

8. Public Test

After the release process, a second QA is done by the community (developers and users). This test is not automatic and is usually done on real hardware

Purpose

• Determine the quality of the product generated after the release
• Identify real bugs in real hardware and in real use case
• Detect errors and update Bugzilla with the new errors to be fixed.

Entry Criteria

• E8.1 A new release (Milestone or Beta) is published on sofware.o.o [14]

Tasks

We can consider this action as a task or as a verification point. If this is a verification point, there are no real task assigned to this action.

Verification

• V8.1 Test in real machine. The user (or developer) can go to software.o.o [14] to download the release and install it in the real hardware. If he / she detect a bug a new bug report is created in bugzilla [19] or / and a mail is send to the Factory mailing list. Also, many users decide to test Factory directly updating the repositories [50], without installing the released media.

Exit Criteria

• X8.1 A new release is published

Deliverables

• D8.1 Bug report. If a bug is found, the tester creates a bug report with the description of the bug. The QA Reviewer can reproduce the bug and assign the it to the correct developer.

Project Organization

1. Roles and Responsibilities

There is a large number of people involved in Factory development. We have identified the following roles:

...

Some of these have one or more roles assigned in the Open Build Service. See an overview [42] [43] [44] [45].

Glossary

Branch

When a contributor wants to work on a package, the first step is to create a copy of the original package. OBS will establish a relation between the original package and the copied (branched packaged). Both packages can evolve separately until a merge process occurs (the integration process).

...

Digital document established by a developer and managed by OBS that describe the set of changes done for a source package.

References

[1] openSUSE Roadmap: http://en.opensuse.org/openSUSE:Roadmap

...