How we shot ourselves in the hand, leg, but found the strength not to make a control shot in the head.

[Apache Ant 2629]

Authors and editors: Xeons, Apache Ant, Papanda.

https://l2e-global.com/forum/attachments/fixes1-eu-png.119888/[/img]

 

 

After we came to our senses and even slept a little, now we would like to share with you our vision of this server opening, and show some inner workings, the process of starting the server.

 

The current gaming product is the result of innovations of the last year and a half, launched with the development of Gracia Final x3 in 2017 (togather with Shock team, under support of AdvExt team). Then we, for the first time, really got the opportunity to do complex and interesting things, to create gameplay, and not to open for the 10th year in a row the another one copy of the 2007-2010 years game with just changed rates.

 

During that cooperation, crazy courage and the desire to give something new - the Interlude Final project was born, which we started to implement a little later.

 

At the heart of the Interlude Final was a simple idea.

Interlude are good chronicles, which are loved by half of the entire audience of Lineage 2 (from all chronicles), but it has endlessly aged and has a number of unsolvable problems and limitations. We decided to create the same simple and understandable product, but to rebuild it a bit, to renew it, to use the good experience of following older chronicles without taking up much, so as not to spoil it. As a result, the formula chosen for the product database was: GF AdvExt engine, support for the maximum client at that time - High Five, and add the following innovations to the game process:

1. Transforms for support and tanks - to increase their playability in solo;
   
2. Pet evolution system - because they are cool and comfortable;
   
3. Revise character sets of skills to increase the diversity of professions and the composition of groups in pve, pvp, mass pvp, olympiad, etc.
   
4. Use the functionality and features of the HF client itself, such as ingame mail, alt+click buffs, shift in chat and other built-in facilities that the interlude lacks;
   
5. Forts, as additional gaming content;
   
6. System of skills enchantment close to the GF, as in the interlude it is completely outdated and morally impoverished.
   

In 2018, we had two successful launches of this product, received a lot of positive feedback, including the general mood that after this nobody wants to play again the old "blue-black" Interlude with permanent criticals. It was time to move on, and after the regular opening of the GF we began to plan the opening of the new Interlude Final.

 

As you can see, we rarely took such huge breaks, almost half a year passed (GF opening took place on October 26) between two openings on the PTS platform, but this time we worked up an appetite for creating the perfect product.

So. What key tasks we set for ourselves on this server:

1. Provide you with new Game Client.
   
   • Our experience with recent GF make a clear vision of the situation: it's NOT very good to gather 500 players with 5 FPS on Baium fight. The old client was developed for relevant at a time processors (and hardware in general), which had fewer cores. As many of you know, today's processors increase the number of cores (streams), but often reduce their clock speed compared to older processors that were relevant at the time of release of most versions of the clients/games.
     Moreover, new client - better look, modern interface, more available features for game design and just modern-look game of 2019.
     

[*]Create a modern Launcher\Updater for all our servers.

• We realize, that it's a 2019 year and less players are ready to use more then 2 steps to run the game. That's why we decide to create a All-In-One app to solve this issue.
  In additions to that, new game client is highly needed fast update delivering, so we make the only 1 way to run the client via Launcher, not via l2.exe.
  

[*]Next step in 3rd party software war.

• Eternal problem :)
  

[*]Upgrade site, forum and marge 2 big users database of E-Global and Averia.

• The final goal (the process of development of this infrastructure, still going) is creation of a society that would be interested in helping us develop products, being an energy source for the team.
  

[*]Make a few additional features to the game design\experience.

• Add some instance, but not pushing the limits. Instance zones - is a good way to help players of different categories to cooperate and communicate.
  
• Finally introduces new High-End content location.
  
• New system of visual changes, such as costumes, cloaks etc.
  
• Ingame tutorial and knowledge database.
  

Basicaly, we've decided to make VERY ambitious plans for next opening.

Unfortulitaly we didn't made to finish all of the assigned tasks, however we keep working to provide you with them, but more information later.

 

So, it's OBT date.

Surprisingly all is well, the attention of the audience is attracted, there are no really critical problems found.

The risks we could face were understood and we could cope with them. But as they say, everyone has a plan until he gets punched in the face.

 

Day of start! (1)

We are launching the server. The first thousand of players joined and the launcher, which is the only entry point to the game, simply could not withstand load and droped. A short diagnosis shows that the problem cannot be solved in 10 minutes. The server stops, and we expand the reserve capacity, then we make sure that the system is stable, reset the characters level and start the server again in two and a half hours.

 

Got into an unpleasant situation, spoiled your race for the first hours and it seems you can breathe out with relief, but...

Some Bulgarian dude appears, mom's DDoS-er, and he finds gaps in our system which appered because of a lot of new modules.

As a result: sweat + 2 server drops + half an hour of time = closed visible holes.

Of course, DDoS is not over, but it began to significantly less influence the gameplay.

 

In parallel to this process, our personal channels of communication, a forum, perhaps even the phones - are broken from the negative.

You are many, we are few, chances are 0.

From the huge players online on the forum and creating hundreds of messages per second - the forum dropped and public communication channels were paralyzed.

If you were going mad at that moment, you can imagine how we felt.

 

But it was definitely not our day and, of course, this was not the end.

The server began to drop every 20 minutes, and the level of our concentration and awareness by this time - well, you can imagine, after a few hours of severe stress... We take a half hour time-out and the workaround is successfully appied on the server - we let the players login the game.

The clock shows 3AM. Half the team already just fell asleep on the keyboards.

 

Hopefully, that is all - but hell NO:D.

The protection system that we managed to increase began to block the launch of client protection, because of this the network crashed and almost no one else could get into the game. At this very time - a couple of hundred players managed to login the game somehow and are exping. The aggression is growing.

We've "done".

We need to stop this, so we make a decision to shut down the server. We set a temporary announcement and everyone goes to bed so that in the morning, with a fresh mind, analyze this local apocalypse (at that time it seemed that this was the end of the universe) and decide on further actions.

 

Morning of a new day.

Gathering a meeting and brainstorm everything that happened.

What needs to be done to cover all problem areas and, most importantly, how much time we need for this, and when to start the server: today, tomorrow, in a week.

At the same time, the option “in a week” was dominant, since we no longer had a second right to make a mistake.

How do we understand this? - A record figure of 1,700 people on the forum, and none of them isn't really happy to see you.

We identified problem points, gave a quick assessment of what to do and how much time it takes, namely:

1. Rework the game launch system: launching the game through l2.exe - launch both the updater itself and the game so that there is a backup opportunity to get a working client if the launcher falls to the death of the brave;
   
2. Modify the network environment in terms of DDoS protection, in order not to add problems to ourself, but not to be substituted for a second blow;
   
3. Analyze and reliably close the causes of night drops;
   
4. Put an icon and light a candle for the health of the server.
   

The first three points were most likely superfluous and it was the last one that helped.

 

https://l2e-global.com/forum/attachments/1554843171286-png.119885/[/img]

As a result, the server was opened (2) for players at 18:00 CEST, without any particular difficulties, critical errors, forced reboots, thanks god.

 

And thanks - this is certainly not for us, we just do our job.

Thanks to you.

We are very grateful for your trust in us, for the chance to recover from what happened (we, of course, quietly wiped off the spattered face and back), and now we concentrate on delivering the planned, but still unfinished content in the nearest time:

1. New location;
   
2. The Battle Pass system for getting an extras (Battle Piece and Battle Coin that drop already, are the part of this system);
   
3. Zaken Instance (preserving the original one) and some interesting things around him.
   

I also want to share the statistics for the first day after the server start:

[spoiler=Beautiful Figures]Total - 12266 characters

1-19 lvl. - 6068 characters (49.5%)

20-39 lvl. - 4154 characters (33.8%)

40-51 lvl. - 1276 characters (10.4%)

52-60 lvl. - 601 characters (4.8%)

61-67 lvl. - 114 characters (1%)

68+lvl. - 53 characters (< 1%)

Satturday - 4363 unique players

Sunday (17:30) - 5160 and goes up!

During this day, 1674 characters (excluding merchants) spent in the game> 12 hours!

6200 characters - more than 3 hours!

320,322 posts in chat!

 

 

To summarize, I would like to apologize again for the caused trouble and say thank you for your support.

Weightedness and tenfold thinking over our decisions is what we have learned through hard pain.

On the other hand, risk and success are related concepts.

I’m sure we can go through everything together.

 

 

With :1f497:, Your E-Global team.

[reetveter 2]

You and your team did good work. Thanks for this fun server, we all hope we can enjoy this server for at least 1 year :). Kind regards, MonkeyBusiness crew.