Problem Inquiry & Tool Proposal: Source Controlled PK3 Build System

Hi folks!

I'm just wondering if there is a need/want in the community to have something that I'm trying to enable for myself in my personal projects. Perhaps there's already a better existing solution or even a general best practice which avoids the issues I'm trying to solve entirely. In any case, essentially, I'd like to treat my mod projects more like software projects but this comes with some hassles which I've outlined below (TL;DR at the bottom).

WADs, PK3s and Source Control...
Reviewing changes to WADs in GitHub, for example, is not very transparent. WADs are comprised of both human-readable segments (text lumps) as well as binary segments (sounds, graphics, compiled ACS, etc.) but GitHub simply considers the whole WAD a binary file (you might be able to get around this by first committing a text-only WAD first but I've not tested this and, if true, it's still quite a hacky solution). It will not display line changes in the UI so in order to gain an understanding of a particular change, the reviewer/reader must pull in the change and view the diff themselves locally (and if there are binary changes mixed in, it won't be pretty or convenient to navigate). I suspect that other Git hosting services will be similar in this way. Additionally, all the same is true for reviewing changes to PK3s. The difference is that since PK3s are literally just zip archives they represent a more natural analogue to a conventional software project. This means you can make changes to your PK3 and then extract the contents to a corresponding "source archive root" directory in your local repo so that you can benefit from having the simple, transparent catalog of narrowly-scoped changes over time that modern source control enables. Still, this feels very... off. It feels like I'm maintaining a "ghost" repo alongside my PK3 in parallel... it feels like extra work... like after all the changes I make to my PK3 I have to remember to do this manual step where I navigate to my PK3, right click, select my archiver application, select "extract to...", choose my sources root and-- OOPS! I forgot that I had deleted some unnecessary lumps and folders in the PK3 as part of this release so now I have to go into the sources I just extracted to delete those lumps or delete the whole directory and rerun the extraction. I have to perform all these manual actions, make all these button clicks and error-prone judgements just to make sure that my source repo reflects the current state of my latest release... and even then I've still managed to introduce some degree of risk. The burden of potentially desynchronizing my sources from my release with every change starts weighing on me... maybe I put it off once or twice with the promise to myself that I'll re-sync my sources in a few commits. I hold myself to this promise at first but I inevitably start slipping up again regularly and then eventually I simply stop consistently updating my source repo altogether... but it doesn't have to be this way!

Building Consistency into Your Workflow...
What I propose is a build system which has 2 main jobs:
generates artifacts (compiles source trees from a local Git repo into PK3s for release)
synchronizes sources (extracts the contents of artifacts with the latest changes into corresponding source trees in a local Git repo to maintain a 1:1 relationship between release contents and sources)

The build system could even be extended to handle WADs. It could expose actions to take when making changes to the repo in the form of Git hooks (for example: if it detects, upon commit, that the latest local release differs from what's in the sources it can automatically synchronize them before proceeding with the commit or it can cancel the commit and then prompt the user to take appropriate action). It opens up the possibility to expand the potential tool suite to include people's favorite IDEs. The possibilities are vast I feel. To get a rough idea of what a workflow enabled by such a system could potentially look like you can check out the demo I've posted to YouTube: " DOOM Modding Tools: Source Controlled PK3 Build System? ".

In Conclusion...
I'd like to get some folks' opinions on this idea (whether it is worth the effort or necessary or even solves a problem in the first place, examples of existing efforts to achieve a similar goal, general questions and remarks, etc.)... and if you made it this far, I appreciate your time as well as any insights and/or advice that folks from the community may have to offer!

TL; DR...
I want to track my projects using source control but there are challenges associated with viewing/reviewing changes in GitHub as well as keeping changes between sources and releases in 1:1 sync; I want to know if there are existing solutions or general best practices to help with addressing/avoiding these issues (or whether I am simply hallucinating a problem); I've included a video demo which demonstrates what a solution could potentially look like; thanks!

I'm not sure if this is the right place to ask about this, since this site is meant for small scale things like single weapons and monsters that usually don't have the problems you are describing, at least not for me personally.

Most people here aren't programmers, just some guys who occasionally write some more or less simple decorate or zscript. Me personally, I once took some programming classes in trade school over 2 decades ago, not much has stuck  To be honest, I only really understood what you were proposing after watching the video

As far as I'm aware, the only member of Realm667 who uses GitHub is Inkoalawetrust, and he isn't very active here at the moment (probably busy with working on his Smart Marines mod).
I myself can't wrap my head around GitHub at all, I just stick to SLADE and do manual backups every now and then, never gave me any issues. For small things like my monsters, weapons and items I do everything in a pk3 file directly, for a little bit larger things like my gameplay mods I have a working folder that I only pack into a pk3 once it's ready for release. Anything beyond that is way too complicated for me and not worth the effort.
But I understand that for large scale projects this approach might not be sufficient anymore.

I'm curious what other members of this community say about this, maybe this turns out to be really useful to them.

Also if you haven't already done so, you might wanna post this over at the Zdoom forums too, maybe in the  Creation, Conversion, and Editing  section. The official Zdoom discord might also be a place to promote your ideas, as many modders with big projects regularly frequent that server.

Oooooooo I see... I had a feeling that maybe I was pulling a problem out of thin air   I appreciate that you took the extra time to sit through the video to understand what I was at least getting at! Also thank you for the insight into your process- it's entirely possible that the workflow I'm trying to create for myself isn't quite appropriate for most projects (possibly even my own!). In any case, I suppose I'll follow your suggestion and post to the zdoom forum section you mentioned JUST in case!