General workflow overview
For the sake of practicality here is a short summary of how we organize our work, using Gitlab issues, merge requests(MRs), and operations on those states.
The main entity that represents a certain amount of work to be done for the product, is the issue. Issues can be:
- Bug – incorrect or unwanted functionality that must be fixed
- Enhancement – a new feature or enhancement request that must be implemented
- Security – a security weakness that must be removed
- Performance – functionality with unsatisfying performance, which should be improved
- Research – a research issue, one that should present one or more options for general improvement in the product
- Infrastructure – an improvement, related with internal infrastructure, code refactoring and so forth
In general, there is one main project that aggregates the effort for the target, and where we set goals, milestones, and track issues.
In the context of Gitlab system, we use, issues are currently open in the context of target_project Gitlab project, which is our umbrella project, as they reflect some change in the product functionality, which will improve user experience or fix a problem in the product.
Issues can be opened in some of the other projects like frontend or core, in case there is work to be done that has no whatsoever visibility to the user of the product. But that should be avoided.
Every issue can be linked to one or more MRs, which represent the change in the source code of the product that will solve the problem described in the issue. The MRs, associated with an issue, can be implemented in one or more other source code project repositories like core, frontend, utils etc. The relation between the MR and the issue is discussed later in the document, where Issue Board and issue states are discussed.
Release lifecycle and Code Freeze
Our main development activities are submitted via merge requests to a main development tree, called next for each repository used in the product development. In the general case Gitlab names such branches master, but we use next, as this is the base for next release.
For each release version of the product we schedule a milestone with an end date. At the middle of the release QA engineers, having in mind the content of the release, the current progress etc, should determine what should be the Code Freeze period of the release.
Code Freeze period is a period of time, defined in a date before the release date, after which no new features will be added and only P0 bugfix issues will be handled in the release. At that date a version branch/tag will be created, and only testing will be done on it and critical fixes added.
The rest of the development will continue on the next branches of the relevant projects. But this is a rare situation, since the whole team is concentrated on stabilizing the release.
The stabilization fixes, done in Code Freeze phase, will be merged both to the next branch and to the release one.
Issue board
The board is the main reference for our workflow. It illustrates:
- Where we stand roughly in terms of amount of work to be done or already finished for given release
- How many issues are currently in development, review or in testing phase.
- Who is handling what, in terms of personal assignment of issues.
The columns in the Issues Board table reflect the state through which an issue goes through.
So looking at the table one might understand the basic life-cycle of an issue:
- Where the life-cycle of the issue starts
- What kind of transitions it goes through, in result to team’s operations on it
- When the work on an issue is considered finished.
States
Backlog
This is the first state of the issue, right after it gets open and according to its severity and functional effect gets assigned, by the product manager, to a certain milestone release. Then it gets to the backlog of that milestone release. In this state, the product manager with the help of the whole product team, analyses the issue in order to determine:
- Issue priority, in terms of which functionality is affected and how solving it, will facilitate achieving the final goal, a release with desired functionality with no bugs.
- Issue severity, which isn’t a mandatory field, but is useful as additional input from issue creator to product manager on priority decision. The severity levels are high, medium and low with current description:
- High: Affecting main functionality, that system isn’t accessible, main functionality doesn’t work or performance significantly drops
- Medium: Affecting supportive functionality, for example in case of a web collaboration system, that would be access list, user management, passwords etc., tasks that an administrator can handle using workarounds.
- Low: Affecting cosmetics functionality, for example look and feel, which doesn’t harm user experience.
- The type of functionality it affects, which can be preliminary analysis, but a nice starting point, done by labeling the issue with relevant projects, like:
- frontend
- core
- utils
- etc.
Development
In this state the issue is being handled by one or more developers.
The general policy for choosing an issue, is that a developer will pick the next issue from the Backlog according to, criteria roughly ordered by importance:
- issue priority
- issue severity
- issue project relevance (frontend/core/utils) best matching her/his expertise.
- nothing assigned for review, or already had reviewed one MR
- personal affinity
- ask product manager
NOTE: The idea here is to have balance between issues in development and in review, so that after a developer has finished development work on issue, should check if she/he can review an issue, according her/his competence and only after that pull new issue from backlog to develop.
When a developer starts working on an issue, she/he must assign himself to the issue and drag it in the board from the backlog to the development column.
The general policy should be one developer works on one issue, and creates one or more MRs that solve the issue.
There can be cases, in which one issue can’t be handled by one developer, or its handling will take significant amount of time, say a month or more. In those cases we should split the big issue, to smaller issues.
Each small issue should be as little as possible dependant from the others, and should be capable to be tested separately by QA. The BIG issue will move forward in states review, qa, etc only in case all related issues have passed to the relevant state, and when reviewed/tested the whole solution should be reviewed/tested. Once the first small issue is moved to develop, the BIG(Gitlab calls that Epic) issue will be also moved to “develop” stage and stay there, until the last small issue is closed. After the last small issue is closed, BIG issue goes to review, where we will make sure that all the target functionality is covered, and then move it to qa for full testing.
The way to do that in Gitlab, is to keep the BIG/Epic issue in the backlog and open separate issues, which to be added as related issues to the BIG/Epic one.
In cases where multiple developers are working on an issue, the rule of thumb should be that the big issue will be assigned to:
- the one who in advance is known to integrate the effort to achieve the final functionality targeted by the issue, OR
- the one who splits the job, OR
- anyone of the developers working on the issue
An issue may stay in a development stage for more than a week, without any visible progress, for example MRs created or issues raised in commentaries. It’s recommended, in that case, the developer, who is working on it, to log their problems and progress in a free form in the context of the issue, on a weekly basis.
The issue goes from development to review state, once the work on it’s done and the MR is ready to be merged, meaning also re-based to the latest version of the main development branch (in our case next).
The issue can go also back to the backlog, if the work on it has been suspended/stopped due to more important effort. This must be avoided as much as possible. When an issue is pushed back in the backlog its assignment can be changed.
Review
When an issue transitions to review state, it should be assigned to a product manager.
The product manager should assign it to a different person to review the changes made in the source code, in order the reviewer to help the implementer in its solution. During the stage through performing code reading/testing, online and/or offline discussion between the implementer and the reviewer, both are making sure best solution is provided for the issue.
The general policy should be, that the number of issues in review and development is equal to the number of developers, working on the product.
It can be the case that an issue goes back to development, if during review it was discovered that a significant amount of rework should be done to the solution, but in general at the end an issue should transit to qa state.
In order for the issue to transit to qa state the reviewer should approve the MR.
- When reviewer is done with the review and approves the MR, should assign the MR to the product manager.
- Then the product manager to merge it and make sure there is new version produced, including all the changes from the MR provided to the QA engineers.
On-hold
In case the changes in the issue can’t be tested separately and depend on other issues, after review stage they will be put on-hold by the product manager and she/he will move them to qa, only when the issue can be tested by QA engineer.
QA
After the issue has been reviewed and all relevant MRs merged to a release candidate version, the issue is moved to qa state. It’s the product manager’s responsibility to order the issues in the qa column according to their priority and the engineer that’s going to test the issue will pull issues to be tested according to that priority. There can be exceptions, in cases of issue being blocked by other issues or additional circumstances prevent the issue from being tested. If the issue is part of BIG-gerr one it might be the case that single issue can’t be tested in QA environment and only the BIG issue will be tested and all the others closed. At that stage the issue is tested that:
- the main problem/functionality is solved/achieved
- there is no degradation in functionality, that’s nothing working has been broken
- there is no degradation in performance, that’s main functionality has kept or improved its performance
- there is no degradation in usability, this can be quite subjective, but is and has to be checked
If the change can’t be tested by a QA engineer, the change will be tested in the review/development phase, a unit test provided and QA engineer will make sure we’ve no degradation introduced in the change handling the issue.
If all of those checks/tests have passed, the issue is closed. If there is problem discovered during testing either:
- a new issue is opened
- the current issue is returned to development stage and will be handled accordingly
with a strong preference towards the first approach, that’s opening a new issue.
Closed
Success!!!
The contents of an issue
Here are some notes on sections and labels in an issue, which can help:
- reproducing the issue
- solving and analysing the issue
- handling or opening another issue, that might be related to the current one
- provide feedback on our development/testing practices and thus improve the work process
The issue sections can be different, depending on the type of the issue, we’re using issue templates, which help the person, who opens the issue to give as much as possible structured and useful information to everyone, who later might be working on this issue.
Labels also depend on the issue type, and its state, for example when creating a new bug, the following labels are relevant:
- Severity: low / medium / high (optional)
- Discovery: field / R&D / initial testing / release candidate / automation (mandatory)
In case one thinks that the issue should be closed without any change resolution label is used:
- Resolution: duplicate / can’t reproduce / rejected (mandatory)
further discussed in the ping-pong section.
Possible workflow ping-pongs
QA <-> Develop ping-pong
With regards to the workflow, once QA opens a bug, after it’s assigned to Developer, the following flows are possible using the “resolution” label:
- Developer labels it as duplicate (and moves to QA phase using the board)
- QA agrees -> closes the bug
- QA disagrees -> if it’s not a duplicate, removes that label and moves to development phase -> Developer continues working on it
- Developer labels it as “can’t reproduce”:
- QA also can’t reproduce (investing the time needed for this, depending to the given priority) -> closes the bug
- QA reproduces -> removes this label and writes instructions in the issue about how to reproduce, and moves to development phase -> Developer continues working on it
- Developer labels it as “rejected” and moves it to the QA column:
- QA agrees -> closes the bug
- QA disagrees -> the product manager should decide There can, and will also be cases where a bug is labeled as ‘rejected’ by the product manager, while it’s in the backlog and it will be directly closed by the product manager.
In case one thinks that the issue should be closed without any change or there is misunderstanding in how-to-reproduce, expected behaviour etc, currently we discuss things offline, and either both developer and QA engineer agree or product manager decides how it should be handled.
Develop <-> Review ping-pong
As much as possible with time we will agree on and introduce objective rules that our source code will have to follow. As much as possible those rules will be monitored automatically. Using code climate and unit/integration tests Always during review of source code there will be subjective point of view and that’s perfectly OK :). So we think, it’s important to state and agree on common policy to follow, when there are different opinions when code review is done. If we see with time that this policy doesn’t prove to be helping the process we will re-negotiate. In general the goals of the code review are:
- To increase shared knowledge of the code base and common understanding of the problem and the solution in question.
- To make sure the code is a correct and does the job for fixing the problem or adding the new functionality in the best way.
- To make sure that the code is maintainable.
Having in mind the above still there can be cases in which different opinions can exist on the solution approach or implementation, between the code author and reviewer.
In those cases, a third party, that can also review the code, should be involved by the code author and she/he should resolve the dilemma.
Final thoughts
This is fairly detailed description, but still doesn’t cover all the situations of life that a team can come across. Still the main idea of the described practices and policies is:
- To keep the team focused on the main goal of the product
- To keep the process as smooth as possible, so both at any point in time to be able to deliver a working product
- To provide enough visibility for everyone to see how the small everyday tasks map to the main goal
- To provide feedback for the whole team, in case there is a thread for the goal to be achieved or process to stuck
- To facilitate teamwork, so in case of a problem one can rely on one’s teammates for help, since nobody’s goal is achieved if the common goal isn’t achieved.
