Problem Inquiry & Tool Proposal: Source Controlled PK3 Build System
- A-Bomb
-
Topic Author
- Dark Imp
-
Less
More
- Posts: 11
22 hours 20 minutes ago #1
by A-Bomb
Problem Inquiry & Tool Proposal: Source Controlled PK3 Build System was created by A-Bomb
Hi folks!
I'm just wondering if there is a need/want in the community to have something that I'm trying to enable for myself in my personal projects. Perhaps there's already a better existing solution or even a general best practice which avoids the issues I'm trying to solve entirely. In any case, essentially, I'd like to treat my mod projects more like software projects but this comes with some hassles which I've outlined below (TL;DR at the bottom).
WADs, PK3s and Source Control...
Reviewing changes to WADs in GitHub, for example, is not very transparent. WADs are comprised of both human-readable segments (text lumps) as well as binary segments (sounds, graphics, compiled ACS, etc.) but GitHub simply considers the whole WAD a binary file (you might be able to get around this by first committing a text-only WAD first but I've not tested this and, if true, it's still quite a hacky solution). It will not display line changes in the UI so in order to gain an understanding of a particular change, the reviewer/reader must pull in the change and view the diff themselves locally (and if there are binary changes mixed in, it won't be pretty or convenient to navigate). I suspect that other Git hosting services will be similar in this way. Additionally, all the same is true for reviewing changes to PK3s. The difference is that since PK3s are literally just zip archives they represent a more natural analogue to a conventional software project. This means you can make changes to your PK3 and then extract the contents to a corresponding "source archive root" directory in your local repo so that you can benefit from having the simple, transparent catalog of narrowly-scoped changes over time that modern source control enables. Still, this feels very... off. It feels like I'm maintaining a "ghost" repo alongside my PK3 in parallel... it feels like extra work... like after all the changes I make to my PK3 I have to remember to do this manual step where I navigate to my PK3, right click, select my archiver application, select "extract to...", choose my sources root and-- OOPS! I forgot that I had deleted some unnecessary lumps and folders in the PK3 as part of this release so now I have to go into the sources I just extracted to delete those lumps or delete the whole directory and rerun the extraction. I have to perform all these manual actions, make all these button clicks and error-prone judgements just to make sure that my source repo reflects the current state of my latest release... and even then I've still managed to introduce some degree of risk. The burden of potentially desynchronizing my sources from my release with every change starts weighing on me... maybe I put it off once or twice with the promise to myself that I'll re-sync my sources in a few commits. I hold myself to this promise at first but I inevitably start slipping up again regularly and then eventually I simply stop consistently updating my source repo altogether... but it doesn't have to be this way!
Building Consistency into Your Workflow...
What I propose is a build system which has 2 main jobs:
generates artifacts (compiles source trees from a local Git repo into PK3s for release)
synchronizes sources (extracts the contents of artifacts with the latest changes into corresponding source trees in a local Git repo to maintain a 1:1 relationship between release contents and sources)
The build system could even be extended to handle WADs. It could expose actions to take when making changes to the repo in the form of Git hooks (for example: if it detects, upon commit, that the latest local release differs from what's in the sources it can automatically synchronize them before proceeding with the commit or it can cancel the commit and then prompt the user to take appropriate action). It opens up the possibility to expand the potential tool suite to include people's favorite IDEs. The possibilities are vast I feel. To get a rough idea of what a workflow enabled by such a system could potentially look like you can check out the demo I've posted to YouTube: " DOOM Modding Tools: Source Controlled PK3 Build System? ".
In Conclusion...
I'd like to get some folks' opinions on this idea (whether it is worth the effort or necessary or even solves a problem in the first place, examples of existing efforts to achieve a similar goal, general questions and remarks, etc.)... and if you made it this far, I appreciate your time as well as any insights and/or advice that folks from the community may have to offer!
TL; DR...
I want to track my projects using source control but there are challenges associated with viewing/reviewing changes in GitHub as well as keeping changes between sources and releases in 1:1 sync; I want to know if there are existing solutions or general best practices to help with addressing/avoiding these issues (or whether I am simply hallucinating a problem); I've included a video demo which demonstrates what a solution could potentially look like; thanks!
I'm just wondering if there is a need/want in the community to have something that I'm trying to enable for myself in my personal projects. Perhaps there's already a better existing solution or even a general best practice which avoids the issues I'm trying to solve entirely. In any case, essentially, I'd like to treat my mod projects more like software projects but this comes with some hassles which I've outlined below (TL;DR at the bottom).
WADs, PK3s and Source Control...
Reviewing changes to WADs in GitHub, for example, is not very transparent. WADs are comprised of both human-readable segments (text lumps) as well as binary segments (sounds, graphics, compiled ACS, etc.) but GitHub simply considers the whole WAD a binary file (you might be able to get around this by first committing a text-only WAD first but I've not tested this and, if true, it's still quite a hacky solution). It will not display line changes in the UI so in order to gain an understanding of a particular change, the reviewer/reader must pull in the change and view the diff themselves locally (and if there are binary changes mixed in, it won't be pretty or convenient to navigate). I suspect that other Git hosting services will be similar in this way. Additionally, all the same is true for reviewing changes to PK3s. The difference is that since PK3s are literally just zip archives they represent a more natural analogue to a conventional software project. This means you can make changes to your PK3 and then extract the contents to a corresponding "source archive root" directory in your local repo so that you can benefit from having the simple, transparent catalog of narrowly-scoped changes over time that modern source control enables. Still, this feels very... off. It feels like I'm maintaining a "ghost" repo alongside my PK3 in parallel... it feels like extra work... like after all the changes I make to my PK3 I have to remember to do this manual step where I navigate to my PK3, right click, select my archiver application, select "extract to...", choose my sources root and-- OOPS! I forgot that I had deleted some unnecessary lumps and folders in the PK3 as part of this release so now I have to go into the sources I just extracted to delete those lumps or delete the whole directory and rerun the extraction. I have to perform all these manual actions, make all these button clicks and error-prone judgements just to make sure that my source repo reflects the current state of my latest release... and even then I've still managed to introduce some degree of risk. The burden of potentially desynchronizing my sources from my release with every change starts weighing on me... maybe I put it off once or twice with the promise to myself that I'll re-sync my sources in a few commits. I hold myself to this promise at first but I inevitably start slipping up again regularly and then eventually I simply stop consistently updating my source repo altogether... but it doesn't have to be this way!
Building Consistency into Your Workflow...
What I propose is a build system which has 2 main jobs:
generates artifacts (compiles source trees from a local Git repo into PK3s for release)
synchronizes sources (extracts the contents of artifacts with the latest changes into corresponding source trees in a local Git repo to maintain a 1:1 relationship between release contents and sources)
The build system could even be extended to handle WADs. It could expose actions to take when making changes to the repo in the form of Git hooks (for example: if it detects, upon commit, that the latest local release differs from what's in the sources it can automatically synchronize them before proceeding with the commit or it can cancel the commit and then prompt the user to take appropriate action). It opens up the possibility to expand the potential tool suite to include people's favorite IDEs. The possibilities are vast I feel. To get a rough idea of what a workflow enabled by such a system could potentially look like you can check out the demo I've posted to YouTube: " DOOM Modding Tools: Source Controlled PK3 Build System? ".
In Conclusion...
I'd like to get some folks' opinions on this idea (whether it is worth the effort or necessary or even solves a problem in the first place, examples of existing efforts to achieve a similar goal, general questions and remarks, etc.)... and if you made it this far, I appreciate your time as well as any insights and/or advice that folks from the community may have to offer!
TL; DR...
I want to track my projects using source control but there are challenges associated with viewing/reviewing changes in GitHub as well as keeping changes between sources and releases in 1:1 sync; I want to know if there are existing solutions or general best practices to help with addressing/avoiding these issues (or whether I am simply hallucinating a problem); I've included a video demo which demonstrates what a solution could potentially look like; thanks!
Please Log in or Create an account to join the conversation.