When a software organization reaches a certain size, with many different projects all running at once, inevitably someone realizes multiple project teams are writing code that seem to serve very similar pressure. This is usually followed by an attempt to bring together all this code into a library, managed by a single team.
This is an easy initiative to pitch. Software best practices already recommend avoiding code duplication, so you get all the technical benefits like fixing bugs in only one place, and spreading the cost of development across multiple clients. And of course, there's the financial aspect of it, where developers can instead work on other things instead of rewriting the same functionality in a different context. Sometimes code merging can be seen as so intrinsically good that it's impossible to even suggest that maybe it shouldn't be done.
The other factor that makes it difficult to argue against code merging is that the drawbacks, while real, are mostly intangible, and very difficult to put an exact price on:
- True maintenance effort - Measuring this is not as simple as it seems. You obviously can't just take the team that's currently maintaining one of the versions, and expect that team to be able to scale to supporting all the clients. But how much additional staffing you need can vary a lot depending on the library and the clients. For example, a team that supports a game engine that plans to support a variety of platforms, as well as Windows, MacOS, and Linux on the PC may find itself requiring a substantial expansion to develop, test, and support all features on all platforms.
- Schedule risk - The more components out of your control, the more difficult it becomes to maintain the schedule. Something that might have taken a couple of days might now take a week or more since requirements need to be discussed among devs from different teams, and both teams need to slot the work into their schedules. There can also be back and forth and delays if the code needs bug fixes, since the original developer may have moved onto another feature for another team.
- Brand damage - Bugs and other issues get resolved much more slowly as they have to get routed and scheduled through multiple teams. In some cases, an unhealthy relationship can develop where both sides become more interested in showing that the bug is not in their code, and less interest in finding the bug and fixing it. Remember - the more people who are "responsible" for something, the less responsibility there is overall.
- Lower morale - Losing control on all meaningful axes can lead to the client developers losing pride in their work and no longer caring about the work. After all, how many times can you apologize for yet another late or buggy feature because the library developers didn't deliver? On the flip side, how easy is it to stay motivated when you've done a lot of heavy lifting in the library and the client teams are getting all the credit for being so quick with development?
This is not to say whether it is or isn't worthwhile to merge software projects. It's simply a call to have a more informed dicussion about the true costs and benefits. Also, it's a call to better evaluate the outcome of any merging projects. If you only look for data showing your project was a success, then that's the only thing you'll find. Talk to all the people involved, including both management and developers, including both library team members and client team members, and try to do a qualitative assessment, in addition to the typical financial only assessment.
If you do decide to go ahead with a merge project, here are some questions to consider. There aren't any right or wrong answers. Instead it's meant to make sure all the groups are in general agreement about the responsibilities of everyone involved, and to provide a framework for hammering out workflow issues.
For concreteness, the following questions use a game engine as the code to be shared, and assumes separate "engine teams" and "game teams," the latter of who do the implementation for a specific game. It's easily generalizable to other software uses (web developers can substitute "web framework" and "web app" if they like).
Version 1.0 (April 2019)
Requirements
- Who will decide the changes for each release?
- Will it be the engine team, or the game team?
- Will it be someone in the management chain who owns both the engine and game teams? (At some places this can be the VP level)
- How will the changes for each release be chosen?
- Will it be based on the financial revenue of each game? Will that impact smaller or more experimental projects?
- Will it be based on who the project manager likes the best?
- Will it be based on who has the ear of the CEO or VP?
- What fallback plans are there?
- If a game can't get a feature into the engine, are they still expected to implement it on top of the engine? (Can have high impact on design and maintainability of the software)
- Is the game allowed to use a different engine? Is the game allowed to fork the engine?
- When different technical tradeoffs are needed, who makes the decision?
- Is it ok for the engine to add a new feature for a game if it reduces another game's fps by 20%?
- If the engine uses more resources (for example, RAM) in a new iteration, is the engine or the game required to make optimizations if necessary?
- How is the technical budget (CPU, GPU, RAM, storage) decided between the engine team and the game team?
Schedule
- When two games both need features from the engine, who decides the order?
- Will the game be penalized for slipping schedule due to engine changes?
- When and how will new versions of the engine be deployed?
- If the game engine needs to be shared (for example, a game and level editor both share the same engine), who makes the call on if and when upgrades happen?
- Is it ok for the game to force the upgrade, even if it causes the level editor to crash on launch?
- At the end of the release cycle, if the engine team needs to make changes that cause bugs in the game, can the game team be required to crunch?
- If the game engine needs to be shared (for example, a game and level editor both share the same engine), who makes the call on if and when upgrades happen?
Development
- What team has the responsibility of adding in new features?
- Does this change based on number of games using the feature?
- Can a flagship game commit several members of the engine team to develop features that will be used by that one game only?
- Can the engine team be required to accept patches from the game team?
- What is the technical quality required before acceptance?
- Are the technical expectations different between the game code and engine code?
- Can code that's buggy or code that has no documentation or tests be forcibly committed into the engine? (Beware, it's common to say "of course not" here, but then decide to make an exception when schedule crunch hits)
- Who decides whether the code has met the technical burden?
- What if the code works and is solid but does not fit the design of the engine? (Example: functional heavy code in an object oriented architecture)
- What is the technical quality required before acceptance?
- Who maintains the code?
- If submitted by a member of the game team, are they expected to continue maintainance, or does the engine team take over?
- If a member of the game team, will they be expected to maintain their code even after moving on to the next game?
- If the engine team, how much justification is required before they can reject a patch as unmaintainable?
- If submitted by a member of the game team, are they expected to continue maintainance, or does the engine team take over?
- Who is responsible for testing on what platforms?
- If the engine developers all run Windows, and one of the game teams wants to ship on Linux, who's responsible for the port?
- Is the engine team expected to know how to install any SDK's and build and debug on all platforms?
- What about platforms such as consoles that may require NDA's?
- If a game team wants to run on a specific platform, are they responsible for any platform specific fixes?
- If the engine team doesn't maintain all the platforms, which teams need to sign off before a new version is deployed?
- If the engine developers all run Windows, and one of the game teams wants to ship on Linux, who's responsible for the port?
Debugging
- When a bug is reported in the game, who is responsible for the debugging, the game team or the engine team?
- What is the burden of proof for a bug to moved between teams?
- Is a cluster of issues among different games using the same engine sufficient to assign next steps to the engine team?
- Is a fully reproducible test case necessary? (Can be very difficult for certain classes of bugs)
- Does the game team need to provide a patch?
- Does the engine team have any obligations before the burden of proof is met?
- Does the engine team need to help with debugging even when it's not clear it's an engine bug?
Compensation
- Will the engine team share in any bonuses if the game does well financially?
- Will this be everyone on the engine team?
- Will this happen even if the engine team needed to do no additional work? If not, will this provide a disincentive for any self initiated improvements?
- Will the engine team share in any non financial rewards? Promotions, launch parties, swag?
- How will the game team be evaluated independently of the engine team?