So here we are. The following will most likely be quite long, because it's not easy to give a full picture of where we are now without looking at how we arrived in the current situation.
Disclaimer: this is an attempt to describe the past year more or less accurately, some part of it only involves my own point of view, I am not speaking on behalf of the whole ETL dev team. In any case, don't take it personally, because infuriating people is not the objective here. Giving an explanation on the current situation in a more factual manner is, so TM knows what's the way ahead. Just drop me a PM if you feel to insult someone, I promise I won't even read it.
Let's quickly recap from the beginning.
// How ET:Legacy started
ET was released sometimes in 2003, and various mods flourished thanks to the available SDK. The entire source code was released in 2010, and quite a few projects started to improve the engine. Of all these efforts, only ET:L remains. In these early days, the work was primarily focusing on the engine side, but work started to improve the mod side too.
Because of license restriction and unlike most existing mods, the ET:L mod is built from etmain (silent, nq, nitmod and many others were based of the open etpub code). In other words, the mod was several years behind other popular mods, barebone and buggy and only slowly integrated features over time. It was also not clear what the objective should become in these early days, but it became gradually clear it should stay close to the original game design: no fun feature, nor any special weapons, just plain ET experience - but still aiming at fixing many of its shortcoming.
Development was done by a small number of people. A few others joined the project over time (including myself). Some aren't active anymore. It took quite a few releases over the year to reach a point were the mod was actually enjoyable. Since the mod wasn't popular, we couldn't get lot of feedback. Testing before a new release consisted of 2 or 3 night sessions of testing with a couple of guys we could find to help (7 or 8 people, over a few games). With these limited testing sessions, it's no surprise that the quality of each and every release was subpart. Every release was better than the last though, but still crucially lacking in stability and had very major issues we couldn't detect in time.
Also, the project had long delay between release. There is so many things to fix, so many lacking features, that getting a release out has never been easy: you need to reduce the planned features, cut what's not absolutely needed, and try to fix important bugs or at least workaround them as best as possible. The whole process requires tons of work, time and energy, that involved people feels usually exhausted - at least it always felt like it for me.
And such was the situation for 2.76: about 2 years in development, not seemingly near a state of being "good enough" for release. And basically not any feedback for 2 whole years. We knew the code was bad, we we had no choice other than to release it anyway to get the much needed feedback. We planned it for being a "just acceptable" release and planned a second release 2 or 3 months later to fix the important shortcoming the community of players would give us. It took about 3 months of crazy work to stabilize the code as much as we could, but by January of last year the release was out.
And indeed, 2.76 release went badly. But not for any of the reasons we could have predicted: a small, non optimal fix trying to limit the way admins can pollute players UI indiscriminately was badly received. I would lie if I described the way it went other than "being shit upon after giving some stuff for free" by some of the ET community. I personally managed it okayish, in part because I wasn't actively exposed in community forums, and because I expected 2.76 to be far from perfect. I also had similar experience in the past (in a complete other context), so this wasn't entirely new to me. I was however surprised by the intensity of the backlash. I know some members of the ETL team had understandably a hard time to deal with it.
// Continuous feedback
The planned short term release never happened. After some discussion with some input of the community, the planned scope changed, more bigger fixes were included, the protection feature required a complete redesign, etc. In a nutshell, we were back to the usual business of delaying any new release "until it's good enough".
Until then, there were a few servers running Legacy mod, but one in particular started to become regularly populated. We noticed these guys had some issues with a few 2.76 bugs that we had already fixed, so we arranged to provide a fixed version by compiling the latest code we were working on. It kinda happened organically: soon after we found out we had feedback on what worked and what other bugs were found, which helped to provide a new version with fixes and some new features we planned for 2.77. Collaboration with TM started here.
Of course, we followed some internal process to ensure not to send broken builds (fixing a bug sometimes can create 3 new bugs): each dev test the feature or bug he's working on locally, then we upload a build on our own test server. We check everything is seemingly working smoothly before sending the build to TM a few days later, or we work on a new build if it doesn't.
// What worked.
(and quite well)
Continuous feedback. It really helps. We catch bugs as soon as possible, we have early feedback, we can play test our own code live with many humans players. I have no doubt the overall quality of the code is much better than anything we released previously.
Log access. We had log access simply by asking to get them, and more recently, direct access to logs, allowing us to check them in case we need more info to solve an issue (I'm still tracking an issue in TM logs, but it's happening so rarely that I'll need more time. The good news is that we now know its occurrence is much rarer than initially thought).
Security. Remember Antum and how TM server suddenly crashed? A close collaboration helped to pinpoint the exact issue so we could fix it once and for all. That was a quite thrilling episode.
// What didn't work.
(this is longer than the above, but don't be fooled as these are much more specific than what actually worked)
No real testing done on our own test server. While devs check changes before approving new builds for TM, nobody else does. Our local independent testing by devs on their machine and testing on our test server isn't much different. Testing is done with 1 or 2 players on our test server, and 40 players on a full TM servers, but nothing is done in-between. The process doesn't scale up, it's all or nothing.
Sometimes shit happens and bug slip through. Some issues are very specific and can pass all of our checks because of a single variable. One recent example is a bug that happens only with a specific compiler, and it took time to understand why devs couldn't actually reproduce it.
Some annoying bugs cannot be reproduced locally, or on a server with only a small number of people. Some bugs are map dependent, some other depends on the number of players, or happen only in a very specific situation that will naturally arise more frequently on a full server. That was especially the case last summer/autumn, when each and every build we shipped to TM were seemingly affected by a crash we couldn't reproduce on our end.
Delay before build deployment. Continuous feedback works well when it is... continuous, but the more the delay between the time we provide the build and the time it is installed on the server, the more it becomes useless. TM admins have their reasons to not put a build right away (often due to admin availability), but sometimes the delay was simply too long, introducing a "bug loop": if it takes several weeks to get feedback on a build we provide, then we can only fix detected issues after these few weeks, and the next build will undoubtedly have new issues introduced during these few weeks without any feedback. For those that don't understand what we mean here: quick update = faster feedback = faster fixes = less bugs overall. At some point the delay was so bad (~6 or 8 weeks at some point last year) that we even internally joked about keeping severe bugs intentionally, just to entice admins to update faster to the next build that would solve it (we never did that obviously, the codebase has already enough of these). It wasn't always as bad of course, but quite a few times we had time to prepare a new build before the old one was even deployed on TM.
Very specific config setup. Because TM used to level up all players to maximum levels automatically, there was a whole range of features and bugfixes we never had feedback on. This is one reason the Prestige feature was introduced earlier than planned - beside the fact that vanilla behavior has an overall much better gameplay, because the original game balance has been built around it. I won't enter the detail of the Prestige feature feedback here, but to keep it short it's a good step in the right direction, but still underwhelming.
There are others:
Errors in TM configs which might have skewed feedback results. Over the past few months we've heard a lot of complaints about how the airstrike/artillery strikes were too numerous, with some demands to give more options to limit them - despite the feature having been reworked exactly to provide better airstrike limitation to server admins. All efforts to identify and reproduce the issue locally failed. It is only when plain access to TM logs/config were granted that some missing parenthesis were identified in the server config, preventing the existing cvars to be correctly taken into account. To this day I have no idea how the rest of TM config worked at all during all this time... Another similar situation was introduced with the prestige feature, with initial settings not set as requested which led to corrupted data we couldn't use to determine how well the feature worked. This lead to an extra database reset which didn't make anyone happy, but which was ultimately necessary.
Very superficial reports. I certainly don't want to dismiss the effort taken by the players that actually reported issues (thanks to all of you), but I feel the vast majority of bugs have been caught by devs playing on TM themselves. As an example, corpses aren't gibbed by explosions on the current TM version. I've seen that issue by randomly connecting to the server, and there is no way nobody didn't notice it at all for several weeks. Players simply don't bother to report issues.
Tensions between TM admins and ETL. When interacting in TM forums, I kinda noticed some patterns by "reading between the lines" and felt some people at TM feel we push update to your server just because "we do as we please" and are "playing with code without any consideration for the players". I expressed these concerns in the private forum here in a more detailed way. While I expected a more open discussion about that particular issue, only one single person at TM answered (Bystry, who followed ETL dev quite closely the past year or so). His post is actually very interesting, so I'd suggest whoever can read that forum to have a look. Bystry also made a suggestion to get out of the described "stalemate" by proposing a test server. I actually didn't follow up right away, waiting for more input from TM side, but unfortunately, nobody else ever looked at that post again and these concerns were lost to history. (To answer Bystry suggestion now: test servers never worked in the past, and certainly not on the long term. We tried many times. We have one already, right now. Nobody makes the effort to go to an extra server to do testing). Overall, I am pretty sure more communication between TM admins and ETL would have prevented many unnecessary tensions, but the stalemate remains.
Since then, a few things happened. On discord, it was made clear that ETL1/ETL2 aren't servers to get any testing on new builds, and that the builds should be tested before being sent to TM (paraphrasing a bit, but you get the idea). Yes, the build actually are tested, but it isn't a dichotomy "tested/not tested". It is "tested within the best of our abilities and limited time on a small scale on our test server that not many people play on" vs "full TM server battle-tested".
We can't do more by ourselves, certainly not considering our small size. If TM expects perfectly tested builds and no occasional issues, then there is no point in shipping new builds that will be lacking in one way or another. We try our best, but it's not sufficient. Bugs will happen, and sometimes big ones - for all of the reason mentioned above.
For all of these reasons, I cannot reasonably recommend any further build upgrade. Can't have the latest fixes, because they most probably come with their own set of new issues. Can't have the cake and eat it too.
// It's even worst
The current situation certainly produces lot of stress and frustration on dev side. It's never easy to ship a build knowing we might have missed something, and we usually connect on TM as soon as a new build is set to do debugging as fast as possible - in case something bad happens. But adding to this the need to manage expectation of potentially frustrated people, without a way to talk about concerns openly is something else entirely.
It's even worst when we implement critical features that have been public for months, talked more than once by the team, that we deem necessary to modernize the game, as the reason it wasn't done before doesn't exist on modern computers anymore. That inevitably create outrage because 15-year old habit are modified a bit. Worst, it creates high tensions in the dev team: nobody wants to get shit upon (we have had enough of it already), and standing ground to get stuff done instead of going for a quick revert right away is not easy (recent example: explosion rework).
There is a whole other chapter about the competitive side of the mod, but expending on that alone would certainly require twice as many word as already written above. To keep it short, let's say that everyone agree to make Legacy more appropriate for modern technology, until the point we actually change their habits built on 15-year old habits based on crust code that doesn't make any sense today. Be damned if you dare change anything, you fool! Also, be damned if you don't!
// The future
I probably ranted way too much already, so let's go to what you want to know: what happens now?
ETL Legacy servers have been flourishing lately, and many people around are asking for newer builds.
While ETL is open source and anybody can build by himself, the process isn't exactly easy. We didn't make it easy on purpose, as we tried to avoid spreading builds that might be buggy. So far, only TM got the build directly provided by us.
These latest builds have new features but also security fixes, and so far only TM can easily enjoy them.
Thus, it is time to make ETL test builds available to a wider range of servers and players, and also, time to end the exclusive relationship that started a year ago between TM and ETL dev team.
You can read about our plans here.
To quickly recap:
ETL will provide public snapshot directly on the website, for anyone to be tested
Players can test the latest ETL client (which was never distributed, we only shared the mod code with TM).
MacOS players can now play on latest macOS version, as 2.76 client doesn't work currently
Linux players can play on their latest distro version, as 2.76 client doesn't work anymore
Server owners can install the latest server and Legacy mod version
Regular update is highly encouraged (server side), or forced (client side). The stable version will still be available.
We won't provide the info directly to you (in this forum), nor we will post in each interested community forums, but make it available publicly on our website/discord. Everyone interested can follow and test.
These build are marked *unstable* by default even if they are more stable than anything before. Don't expect any stellar quality, bug will happen.
Bug reporting to *us*, on discord or github, is encouraged. We'll not go fish for bug reports in the various community forums.
Work for setting up these public builds has been done already. There are some non technical circumstances that slowed progress, but we hope to fix these shortly. It will certainly take time to create a servers base giving feedback, but we hope this will serve well for 2.77 release.
Of course, TM is welcome to join in testing these builds.
I'm tired after hours writing the above, so excuse me if some bad wording slip through. Many other aspect weren't touched on, but feel free to ask any question.
Edit: Holy cow, that went even longer than I imagined.
// Join us
Also, I almost forgot:
ETL team is small. Very small. We always need help, the more the better.
In case you still have too much sanity left, don't hesitate any further, join us! We provide no pay, only extra work after your 7-5 main job, no special status nor any recognition, you'll probably get shit on by the community at some point, and you'll be delighted to deal with the occasional moron demanding a quick release after throwing an insult or two.
Requirement: probably some coding skills, but artists are welcome too! Being a psycho to handle the pressure might be required, but maybe you are one already without knowing it! (NB: compensation for follow-up therapy sessions not included).