EVE Evolved: Tech Chat with CCP Chimichanga and CCP Blaraka

EVE Online: Technische Modernisierung durch Quasar-Technologien im Fokus

CCP
- 20.02.25 - 01:05:43 - 6.366 - EVE Online

• Zusammenfassung
• Original

Der 'EVE Evolved' Stream beleuchtet die Modernisierung von EVE Online durch Quasar, ein Ökosystem aus Docker, Kubernetes und Golang. Ziel ist die Ablösung des Monolithen, um Ressourcen effizienter zu verwalten, die Stabilität zu erhöhen und die Downtime zu reduzieren. Verbesserungen im Chat-System und neue Features sind geplant.

EVE Online

00:00:00

00:00:00 Vielen Dank.

00:05:56 And here we are. Hello, everybody. Thank you for joining us for today's stream. Let me see. This is Eve Evolved. Eve Evolved is a brand that we use at CCP to sort of capture all of the tech changes, graphical updates, audio changes, anything that kind of doesn't fall within the purview of, you know, a game design change. And periodically we'll compile a bunch of the really cool things that we've been working on and present them to you through a series of dev blogs.

00:06:23 Earlier this month, there was a news item letting you know that there were several dev blogs coming out throughout February. The first one was about ship identity, which talked about the new ship cards, the new info screen, and the boarding cinematic, which is really, really cool. And just yesterday, a blog was published called Server Deep Dive, which was written by the gentleman to my left on your screen. On my left, your right.

00:06:49 Das war über die Monolith, Quasar und die Veröffentlichung wir haben, um die Performance oder Potenzial zu veröffentlichen, die wir von Skinner gemacht haben. Das ist wirklich cool. So, um, ich introduce sie jetzt. Wir haben die Technical Director, CCP Chimichanga, die ich glaube, ihr habt von verschiedenen Streamen gehört. Und wir haben auch Yves' Lead Distributed Systems Engineer, CCP Blaraka, die ich glaube, ihr war einfach nur für die erste Mal.

00:07:19 It's been almost six years, going to be six years in May this year. But yeah, definitely my first time on stream. Yeah, so it's good for the community to finally get to meet you. And during the course of this stream, get to know you a little bit better. So like I said at the top of the stream...

00:07:38 We're delving into Eve Evolved again for the month of February. We have one more dev blog yet to come, which I'm told is a bit of a monster. It's a huge blog on ESI. I think players will be very happy to hear what the progress we've been making on that. But for now, I would like to talk to these two fine gentlemen about the dev blog that was published the other day.

00:07:59 So, let me just open up with a sort of a generic, could each of you, starting with CCP Chimichanga, just tell us a little about what it is you do at CCP? Everything. So, a lot of what I do at CCP is more around kind of trying to...

00:08:21 Direkt how we're going to leverage certain technologies and how we use those inside of EVE Online. And the biggest part where I started about 11 years ago now is modernizing EVE Online and using things that have solved certain problems that we don't need to solve anymore. And then how we roll that in and connect all those together and achieve a different outcome than just continuously building the same thing over and over again.

00:08:45 The Definition of Insanity. Ceci Barako, I mean, what does a lead distributed systems engineer do?

00:08:55 Well, yeah, I'm working in a team that Nick, sorry, CCP Chimichanga. Oh, no, he's powered. He's a gutfather of. And in the team, me and my coworkers, we're working on building solutions to modernize EVE without having to do everything at once. So we're trying to take piece by piece.

00:09:23 Functionality-by-functionality, including new functionalities, and implement in a way not to have to deal with limitations of the old infrastructure and old code base. And the distributed part in my job description refers to the fact that we're building systems that are distributed, which is kind of obvious given that we're dealing with multiplayer online games.

00:09:50 Now, I'm sort of the tech layman here, obviously. Basically, Chat, my job on this stream is to be like that little Timmy from those old, you know, the old science shows where they have the professor and the little kid together, and the kid Jobs goes, oh, gee, Willica's professor, what does that mean? And, you know, it enables these other guys to sort of explain these things a little bit more. Then you get the Fallout version, we're all going to die. Yeah, that's right. Sorry, Guy.

00:10:16 Das ist die Aufgabe, was mein Job ist zu meiner Mutter. Ja, das ist das, was du tun. Meine Version ist, dass ich smarte Menschen nicht machen. Das ist die einfache Version. Oh, excellent, excellent. Das ist genau das, was wir brauchen. Ich bin die Leute.

00:10:40 Going to the term distributed. Now, is this where Quasar comes into things, CCP Chimichanga? I know we've been talking about Quasar a lot over the last couple of years. It sort of crops up now and then in relation to, you know, we make a reference to it when there's some sort of technical update or improvement. But is Quasar, is that what we mean by distributed? Because Quasar is taking these services out of the big chunker that is the monolith, I think we call it.

00:11:08 Yeah. See, I don't know what I'm talking about. So I think Quasar is just a label that we've kind of, an umbrella term that we've used for a lot of different technologies that we're using. This involves things like Docker, Kubernetes, Golang, gRPC, Protobuf, all sorts of fun stuff. And then using all those things together to kind of give us a platform where we can easily pull things out of EVE Online's ecosystem.

00:11:35 Und dann können wir das Horizontal, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr, wenn wir es nicht mehr.

00:11:56 But also horizontally, because the way that EVE is set up as a mesh network, we can only go so big before all of those things have to connect to one another. And if everything is to connect to everything, you end up with a quadratic resource problem, which has a ceiling on it as well. So all of that is basically taking all of those pieces and making sure that we can model them in a way where we can scale them in ways that make more sense.

00:12:22 A concrete example is if an engineer is worried about deploying something for an expansion or something that might go wrong at the beginning of that expansion, I can basically say, no, I'm good. I can throw money at it. We can go sideways, add so many more pieces of hardware to the equation that we don't have to worry about that. And then we can solve that after the fact if that's really a problem. So it allows us to really focus on specific pieces.

00:12:45 I had a question that came up. Sorry, carry on Bloraka, please. By all means. Yeah, I could just add that I think that it may be like Quasar as a name can be different things for different people. In general, from the outside, it would be exactly what ZP Chimichanga said. Basically, us using those more modern technologies to build things for EVE.

00:13:09 But internally, for us, I guess it will be the ecosystem that we build around those technologies that give us like building pieces to move faster building stuff. And, you know, like five years ago when I joined, like the foundation of Quaze already existed, but we didn't have a lot of those pieces to build from. We had to figure them out.

00:13:34 Aber jetzt, mit jedem Jahr, mit jedem Expansion, mit jedem neuen Ding, haben wir mehr Sachen, und wir können schneller werden. Das ist was Phaser ist geworden.

00:13:54 Ja, aber auch ein Trigger-Word.

00:14:21 Mikroservices are the devil. Mostly because nobody knows what that means. A big thing we concentrate on is domain services. So we have something that can, we have a service that's just for sovereignty. So everything that deals with sovereignty, we're trying to put into that service to try to deal with those things. And that does contain those features around Skyhooks, around the new soft hubs, around the upgrades for the soft hubs.

00:14:43 the resource network around soft hubs all those different pieces so that's all in one domain service for sovereignty so that we're we're basically taking those pieces figuring out the bigger domain that they they're involved with and how those are connected all right awesome and i i'm still not i'm sorry if i'm asking you to over clarify this but again with with the the quasar and the quote unquote getting stuff out of the monolith what is the what's the what's the benefit of doing so

00:15:11 The biggest benefit is that we have an ecosystem where we have more engineers that can hop into various different places and we have more technologies that can help us go faster. So there's a lot more momentum behind it. For example, because it's an ecosystem that's kind of started in the...

00:15:31 The confidence or the security of where EVE Online is right now and not like at the beginning of EVE's creation where they just had to get something out the door. We have a lot more things in place around like linting, around automated testing. So there's a lot more, there's a lot higher confidence levels in changing huge pieces of that tech and getting it out the door faster as opposed to pulling that one domino out of the monolith and hoping nothing else falls over. Right, okay. So it's...

00:15:57 was it like uh separating and like segregating components of the game so that uh you know yeah like you say it's not like a big gen wobbly pile of jenga uh tiles which could yeah then that part of the game might be offline for a little while but the whole cluster is not going to the bed like in some of our in some of our older talks uh like i think we did we talked about this and when we were in vegas in like 2019 or something like

00:16:23 In those talks, we were talking about the fact that you had to be careful about modifying EVE Mail or EVE Chat, because then everything else would explode. But now you can see that we're trying to modify things like chat again, because we've isolated them in a different way. I was about to say, I think you just went on my trigger. What's going on with EVE Chat today? No, we'll come back to that, but carry on, please. Yeah, that's part of what we're trying to do as well, is to kind of isolate those pieces. So if something needs to have more hardware than other pieces, then we can do that, and we can scale that.

00:16:51 horizontally and not have to affect anything else. We don't have to scale everything horizontally. One of the tangible things that can be visible in bad moments for players is that if, let's say, we have a bug in ifsoft system and it's a new domain service for that, we don't have to risk restarting whole cluster to redeploy the fix.

00:17:18 Really, really quickly actually deploy the fix to that system. Right. And the same with scaling what CCP Chamenchanga said already. If that feature that domain service governs became widely popular, which always makes us happy, we can just scale it easily without thinking about everything else.

00:17:44 And another thing also mentioned, but to maybe rephrase it a bit, you could do a lot of these things within a monolithic codebase, like having contracts between different pieces of monolithic codebase to not to be afraid to change a thing, its internal implementation of the thing, because you know that you have a basically API contract between the pieces of the code, but having separate services.

00:18:12 Das macht es viel einfacher zu optimieren, zu modifizieren, zu ändern, einfach nur die gleichen Kontrakte zwischen den Services, zwischen den Pieces der Backend-Code. Genau. Ich möchte zurückkommen zu etwas, was Chimichenga sagte, in der letzten Zeit, wie er gesprochen hat, wie unsere Konferenz verbessert von den ersten Zeiten.

00:18:39 One of the things that you mentioned in your dev blog, and I want to come start talking about a lot of stuff you said in there about like the server tick rate and so forth, that, you know, when Evil Online launched, you know, people were still using 56k modems. I was one of those. It took me about two days to download the client.

00:18:55 Und das war wirklich schlecht aus der Ausstrahlung, nicht nur für 56k. Das ist warum der 1hz tick oder 1-second tickrate für EVE Online war, weil wir hatten, um die höchstenstensten, für die Hardware zu machen, um es möglich zu machen, um es möglich zu machen, um es möglich zu machen, um es möglich zu machen, um es möglich zu machen, um es möglich zu machen, um es möglich zu machen, um es möglich zu machen.

00:19:18 Und jetzt haben wir entire megabits. Es ist crazy. Ja, ich weiß. Es ist wild, right? Wir haben Fiberoptics und all sorts of cool stuff. Aber ja, EVEs Cluster ist immer noch auf 1 Hz. Und du hast in der Blog gesagt, dass wir bei der Masse Testung für Skinner, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir, wenn wir

00:19:47 dass das auf die Serverververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververververver

00:19:56 Ja, so in this case, we have an older way of sending the client updates for any kind of cosmetic changes or like the turrets on your ship or any kind of like shield effects or those types of things. Those come through our simulation frames. And when we started the project for the third-party skins, which, oh, sorry, is the ship skins, we call them third-party skins because they have to delineate between our own first-party skins and the ones that the players make.

00:20:26 Und so we made the decision to not use that same system, because ultimately we wanted to kind of push in a bit and see what we could do inside of Quasar and what those pieces were capable of, because it's one of those systems where, yes, it's a scan, everybody should see it, but if it breaks, it's not the end of the world. It doesn't force any reimbursements or any kind of like...

00:20:49 auf unserer PlayerSupport oder PX oder sowas, so es ist die perfekte place zu versuchen und experimentieren mit etwas. Und dann, wenn wir das gemacht haben, ich glaube, eine der Dinge, die ich eigentlich in der Blog gemacht habe ist, dass uns das über Quasar übernommen ist, dass es nicht die EVE-Server gibt. None of das ist passiert. None der Skin-Change-Change-Itself tatsächlich touches die current EVE-Server existiert.

00:21:14 So, while we were doing these tests, we were able to do all of this without killing the test server that we were running everybody on because we were pushing it 100% through Quasar. So, nothing was going back to, like right now on TQ, for anything to do with ship skins, nothing is going back to London Data Center for code or for Datastore or anything like that. It's all in Quasar. It's all coming directly from everything that we have in AWS.

00:21:42 Right on. But how did that lead you to the revelation that we can push a lot more data around a lot faster versus before and where that might lead?

00:21:55 So, go ahead. Ja, but basically, for me, it's a good example of the situation you might get yourself in working in a role, like I'm working and everybody in the team is working. Basically, we could theoretically try to implement it the old way, that it would increase the load on the server.

00:22:21 Das bedeutet, dass wir wahrscheinlich etwas zu einem breiten Punkt kommen können, wo wir nicht die Server kalkuliert werden können und mehr Dinge während der Tick machen können. Oder wir können eine Art und Weise auswählen, die Dinge anders zu den Klienten. Und wir wählen diese Route. So, die Problem-Space in der Beginn war, dass wir nicht nur eine Systeme für 3rd-Party Skins oder Player-Dotter-Skins haben, sondern wir haben auch die Technik-Problem der Delivering-

00:22:49 um data about those schemes in a new way and how did that lead us to revelation i will let ccp chumichanga to answer well i think there's a lot of traffic i guess yeah yeah so like we we've always had a mechanism to send traffic to the client but this required us to build up a whole nother subsystem to achieve that um and this was basically required us to teach quasar

00:23:16 Where every ship was, not just the solar system, but actually the grid they were on, the bubble that they were in on the server side. So we had to actually emit events to process all that information so that when we said, hey, the ship's getting changed, go tell everybody that's inside of this bubble that the ship's getting changed. So all of that was needed to facilitate that. But by virtue of having that, it also opens the door for all sorts of other things, like being able to target and transmit simulation frames.

00:23:43 Or, if we invert that, we can also monitor that data in a way that we could also start writing NPC behaviors on top of that project and understand who's in grid, who's doing what where, and operate and build up NPC behaviors that don't have to run on the Eve server, that can run in their own ecosystem, that we could build up kind of different, more interesting behaviors of kind of like, I'm an old Eve player, so like.

00:24:10 Wenn die Sleeper-NPCs first kamen, dann haben wir die Behaviors ein bisschen mehr gemacht. Und ich weiß, dass wir das auch für die Incursions und die Tricholabianen sind. Es gibt viele NPC-Berbsen, aber sie können auch sehr expensive werden, weil das was passiert. Aber das ist der Unterschiede. Und dann, ultimately, wir wollten sehen, was passiert wenn wir...

00:24:36 If we give 500 people a button just to spam something so that they can see the effect of it. But also we can kind of measure how hot everything gets and what the actual limits we start running into. That might be sort of a big brain way of sort of trying to clear up local chat during massive fleet brawls as well if we give people basically the equivalent of a fidget spinner. So they're not just putting copy past that over and over again in local chat.

00:25:01 So one of the things I've always wanted to do, I haven't figured out a good way to do it, is I want a ship spinning leaderboard, but it's client specific data and it's not authoritative. So it would be very statistically significant and we'd have to have a relative leaderboard of some kind.

00:25:19 Get that kind of data flying across all the clients as well. I see an opportunity there for an enterprising third-party dev to sort of like come up with something. Just give them an endpoint for that and let them go nuts. They'll find all really cool ways of representing it. I was going to ask you just a couple of questions which come up in Twitch chat. Now, I don't know if these are anywhere within your wheelhouse, but since they're asking, I will ask shrugging and saying, I don't know, it's perfectly fine. One of the questions that came up was, is Quasar in any way related to Carbon?

00:25:50 It's related in the sense if you think about like the Unreal Engine and the Unreal Engine has like online subsystem modules. So not directly related in the sense of like it is part of the carbon framework that we're building for the engine itself. It's more of just how the engine will communicate with the rest of the ecosystem. So carbon is...

00:26:11 Ultimately, a client of this ecosystem. It's a very rich client for the ecosystem to render all these different pieces to show the user what's going on. We're getting to a place now where we're thinking more about our ecosystem as clients to the universe as opposed to just the desktop client. So that's kind of why you've seen more evolution in the launcher, for example, that we're trying to add more things like, great, now you can see your characters. Now you can see where they are, what they might be flying.

00:26:38 Evalve more of those different pieces, because we're treating everything like a client and not just the desktop client. So what Loraca was saying earlier about putting those APIs in place, it allows us to bring in more pieces of the ecosystem to use those APIs and do some cooler stuff. Because that honestly was the original, like the third-party API was the original kind of infancy of Quasar. Like that technology is still around and used heavily in Quasar right now.

00:27:06 Aber Quasar hat es ziemlich viel gebrochen. Und so wird das große ESI-Blog sein, dass wir das Ganze wiederholfen, und aufgleichen und aufgleichen Dinge, die noch nicht in eine Zeit haben. Ja, eigentlich, speaking von meinem Publikum, einige Leute kennen mich aus Slack für ESI, weil das war mein erstes. Das war mein erstes Training für ESI, eine lange Zeit.

00:27:33 Es ist all connected with VSI as well. I guess at some point we might review some information about that as well. Was that the third-party developers channel on the Tweet Fleet Slack? I think that's where a lot of third-party devs used to hang out. Yes. We now also have a spot for third-party devs. I think Discord has supplanted Slack, though Slack's in a lot of ways. We have third-party devs on our Discord. We also have a separate Discord for our EVE partners. And we have a lot of third-party developers.

00:27:58 We're Slack community old, that's how old we are. Yeah, yeah, that's right. Well, you know, like he goes back through phases, you know, it was like Slack IRC and I don't know, whatever it became before that. That's what was big on the internet when I was a kid. The other question was, okay, this is sort of a wishlisty thing, but I wonder if you know anything about why this hasn't been implemented before, but a user was, sorry, a viewer was saying that...

00:28:22 um, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es, wenn es,

00:28:39 Ja, das absoluten hat. Und ich glaube, das größte Problem ist, dass wir uns durch die Settings-Menü haben, und es verändert hat in den letzten Jahren oder so. Ja, ja, eigentlich. Wir haben versucht, das zu standardizieren, und das in der ganzen Welt ist, wie die ganze Welt der Game-Industrie ist, mit settings, die consistenten mit was manchen würde, in den anderen anderen Spielern, wie die Graphical-Settings, zum Beispiel. Aber dann, ich glaube, es ist auch ein Teil von classifying, was das blob von Informationen war in den Settings-File. So, separating die Dinge aus, das vielleicht...

00:29:07 Host-levels specific about where your window is versus more like gameplay specifically you're talking about with like overviews and those types of things. Or what channels you join, you know, when you fire up a fresh client. If you could just, once you've logged into the launcher, if it could just like fetch your profile from, I don't know, the computer under your desk maybe.

00:29:27 Das ist ein perfekte Beispiel von den Optionen, die sich öffnen, um diese Systeme zu entwickeln, weil dann andere Teilen der Ecosysteme können. Ja, das ist ein bisschen ein Jump-Forward, aber ich denke, dass dieses Problem-Space eigentlich mit einem anderen, was wir versuchen, zu machen. Das ist eine neue Weise, um die Daten zu entwickeln.

00:29:55 Two clients that we don't have to ship the data about the universe. For example, we have the client code as we do mostly right now, but just deliver it during runtime. And this is a bit different because this is a player authored thing, the settings, but this can be part of it. Like when we fully do that transition, which we already kind of prototyped within new sovereignty system.

00:30:25 Then those kinds of possibilities also open up for this kind of syncing of settings.

00:30:33 Yeah, that's a big part of what we're tearing apart and the paradigms that we have to break. To add on to that, a lot of how everything was built in EVE Online is built so that everything is in a perfect state when the client starts up, which means that if we want to change one thing in how ships are defined, we have to then make a server patch and a client patch and everybody has to download that client patch. So a big part of where we're going with Quasar is breaking that paradigm in the sense that things are dynamically available to the client.

00:31:02 So you've kind of seen this in the UI a bit in the sense of you might see things like lazy loading in the UI where you might get more pinwheels for certain pieces of the UI and not have to wait like a full second for the whole UI to pop up because it's waiting for all the information to show at once. So this is also kind of a cultural and paradigm shift was how we develop things in the client as well.

00:31:22 Right on. And the other question that sort of caught my eye was someone, so if you read the patch notes today, we announced that we've, you know, we think we fixed a long-standing bug when capital ships in particular would undock en masse from player-owned structures and just get yeeted in the middle of nowhere in the solar system, which is obviously not ideal, it's not what you want.

00:31:45 when you're undocking your huge expensive spaceship to go fight somebody uh do you know uh do you know anything the background about how that was resolved or what came of it i've heard some things but it's just basically one of those ones where it's like if this eat and we're like okay just don't do that oh yeah get rid of that and then that was kind of what we're testing like it's not been it's one of those things where it's really hard to reproduce because we think something is happening when there's a lot of things going on and there's something very very specific going on yeah

00:32:13 Even when we were spamming Capitals undocking with our own automated tools, it was difficult to regrow. And I know when we were doing some of the mesh tests, previous mesh tests, such as Skinner and things like that, low-key we were also hoping to reproduce this bug.

00:32:31 ...during those mass tests, because we'd get like 300-400 people to log in to CC, and then we'd tell you all to undock at once. And then carry out whatever the purpose of the mass test was. But in the background, we were like, did anyone end up at like 40 AU away from the station this time? Just trying to make it reoccur. But we do know that the ability to reproduce it...

00:32:52 was discovered thanks to like thanks for isd volunteers actually so the interstellar service department who volunteer and do a whole you know who are volunteers who do a whole bunch of different stuff everything from like twitch forum uh and twitch forum and and uh like the discord moderation uh to to the isd bug hunters to the people who just hang out in rookie help channel in game answer questions for new players and help them um

00:33:17 Often we get bug reports, they'll come in. Our QAs will triage them, see which ones they can fix. Maybe it's the 50th time this bug has been reported and that gets appended to the ongoing issue for that. And then they will attempt to try and repro as many of these bugs as possible so that they can try and figure out what the cause is. But there's just so many of them that we have a team within the ISD who specifically just go...

00:33:43 Ich finde, unsere ISD-Bug-Hunters sind wie speedrunners. Ja. Ja, wenn du diese Rock an 45-degrees bist, dann geht es um 4,000-miles-an-hours. Ja, ja. Ja, das ist einfach so, wie die Geometrie der Game ist, um den Ruf of den Map, oder so. Aber sie machen einen tollen Job. Und ja, es ist einfach so, wie ein Team-Method zwischen ihnen, dass der Reprodukt-Repro-Case...

00:34:10 The ability to repro came up and then, yeah, we deployed a fix today, which hopefully, hopefully has resolved it. But if it recurs, we'll just have to go back and take a look at it. But it has been troubling NullSec in particular for a while, but hopefully they can put their minds at ease now.

00:34:26 Oh, sorry, on that note as well, that's another good example of some of the tech that we've brought in with Quasar that we've used inside the EVE server as well, because that fix is actually part of a new feature flagging system that we're using with the vendor. So we're actually doing testing and production in the sense of it's turned on, we have that different code path in there, if we basically leave it on and don't see any actual negative effects of that, then...

00:34:55 We're going to assume it's fixed for now and then rip that out later. But if this fix caused massive other side effects, because again, pulling out that one domino, then we were able to turn it off without redeployment and those kind of things. So this topic actually connects us back to the blog that initiated this discussion, because yeah, we were trying to reproduce it during the same master that we were testing delivery of skins.

00:35:24 Das war die Techniker-Journey-Story. Wir hatten den ersten Mass-Test und wir hatten keine Spieler-Authard-Skins. Wir hatten nur die Liverie der alten Skins, mit dem neuen System.

00:35:42 So there was no new functionality for players. We were just testing technology in the background, because when we were developing Skinner, first we developed a foundation for it. And the foundation, just to summarize it, was first we need to track who's in the grid, which we didn't have before. We just had tracking who's in the solar system. Now we're also tracking who's in each grid. And then on top of that, ability to deliver information to all the clients.

00:36:12 that belong to characters that are in a given grid. So the service sends, I want this to be delivered to that grid, and every client that has a ship in that grid should receive that notice. So we were testing that, and that's a bit embarrassing story for me, but there were a lot of math tests. Some people might remember, there was a lot of math tests trying to repeat that.

00:36:33 Weil wir uns mit sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr, sehr.

00:36:58 We were talking about solar flares at that point in time. We were out of ideas. Yeah, it actually turned out that when we are, and this actually gives a bit of context of what we're doing with this, how this system of delivering information works. So we just sent one message and then there is a component like gateway component that funds out this message to all the clients that should receive it. And at that point...

00:37:27 Literally 10 minutes before a mask test, I personally added a tracing code because I wanted to have a better visibility on what's happening during that mask test. And that introduced the bug. And that's my defense, that in the end, when I was tasked with investigating what was the reason of the bug, I quite quickly found it out. But this was something that we had to deal with when we were developing Skinner.

00:37:57 Das zeigt, dass wenn man komplett neue neue Wände zu tun hat, man ist ein Risiko. Aber wir manage. Das ist das, was der Skinner story für mich persönlich ist, ist es ein success story. Ich weiß nicht, wie viele Skinner in der Ende sind, aber wir haben es geschafft, und in der Ende, in der Ende, in dieser Gespräche, in der Ende, in dieser Gespräche.

00:38:26 Underneath the Skinner, there is a lot of stuff that will allow us to build cool stuff in the future. Some above the block, really. Yeah, this is where Blaraka found out the old adage that debugging is like reading a mystery novel where you're the murderer. And you don't find out until the last chapter or something. Is that the twist at the end? Yes.

00:38:51 Ja, sehr Schutter Island. Ich erinnere mich. Ich erinnere mich. Oh, ich weiß, was es ist. Das ist ein sehr Focal Aha Moment. Das ist das, das ist die Sound of Progress, so. Embrace es. Embrace es. Ja. Let me see. So, ein paar mehr Fragen. Ich habe von Twitch Chat. So, ich werde diese Fragen zu you.

00:39:19 The way I want to do it is by coming to the hypothetical questions that you had at the end of the dev blog. You were like, if Quasar can provide 90 hertz traffic for 500 players on a small-scale test server running just three nodes, what could it do if scaled up to the full power of 230 tranquillian nodes with 30,000 active players? You know, what happens if the frames no longer have to wait a full second to be sent? What are the implications does this have for time dilation? So this is kind of the question that Nator and Twitch asked, what technical innovations are needed to remove time dilation?

00:39:48 Is that even... Is that even...

00:39:53 We can speak generally about a couple things here. We've had other projects that have been working with Python 3, and we know that Python 3, with a fork of the Eve engine, has given us, just at a minimum of switching over, a

00:40:22 23% performance increase. Was it 2.7 to 3? Actually, before you continue this train of thought, can you just talk about that movement a little bit? Because I think there might have been a blog about it at some point, but can you just sort of articulate? Yeah, so we do have a blog about that because that's part of the Carbon Initiative to get those pieces into Python 3. And I mean, there's not much to it. It's just, you know, you increment a number in your file. We're just running on a really old version of Python. Also, there's nuances everywhere, right? That's the tricky part.

00:40:50 We have four million lines of Python and there's some tricky bits in there because there's nuances between two and three, et cetera. So there's a lot of work that needed to be done to kind of comb through all those. You don't just press the update button and wait 10 minutes and oh, I'm running on Python three now. Great. Yeah, exactly. Yeah. Anyone running a Linux desktop knows that you never update everything at the same time.

00:41:11 But yeah, so one component of that is Python 3. Another component of that is, if you've seen some of my previous talks about Quasar, there's serialization, transmission, and basically fan out or multiplexing of messages. So the Python 3 part of it, if we take it and say, okay, well, we get a conservatively 20% performance increase, and then we know that serialization, transmission, and multiplexing is another 30% of the...

00:41:40 um required processing inside of a fleet fight we're staring down the barrel of 60 performance increase just conservative napkin math of of what we could possibly improve there in that sense um and so one of the one of the things that we're kind of throwing out there is that okay if we can go far enough to take the simulation frames off the eave server and get rid of that serialization transmission and multiplexing because the serialization for us is now done off-thread by

00:42:07 Proto-Buff, the transmission is now done off-thread by gRPC, and the multiplexing fanout is now done on a completely separate service where all the clients are connected to the public gateway for Quasar. So we're basically removing all of those pieces and offloading 30% of that work. Then if we add another 20% from just, in air quotes, switching to 5.3, we get another 20%. So actually my math is wrong, right?

00:42:35 We get 50%, right? So 50% performance increase. Because in my head, Python was originally supposed to be 30% and I'm still kind of bitter about that. But anyway, it's just 20%. So that's kind of what we know hypothetically right now. So it's not that it'll go away, it's that we'll be able to have more capacity. But it will also open up the door to like, what's the next hurdle that we have? And we know that the next hurdle that we're going to have is going to be around...

00:43:01 The rules engine for EVE Online and how it processes all that information and getting that out of the system that is currently doing it inside the EVE server. So that would be the next step of that. And then just to really dissect the bulk of that 50% that's left over, where is the time really being spent? But it allows us to clear the table to kind of think more, see the future a bit more clearly and get past the bigger pieces that are the big wins at least.

00:43:26 Is there maybe also a future then where apart from tie-dye, just the responsiveness of the client is enhanced because you're not waiting a full, you're not basically waiting a whole tick for something you do to be acknowledged or something happens to you to kick in. You just sort of, all right, I want to stop shooting my entropic disintegrator now. Instead of like going like 25% into the next cycle, it just sort of like cuts off when it's supposed to. Would that be some of the ways this improvement could manifest as well?

00:43:53 Ja, ja, things work like they're supposed to, it's crazy. Ja, that's right. It would just be such a weird sensation. It would probably feel wrong for a long time, you know what I mean? It's like something's wrong with my EVE client, I just double clicked in space and my ship immediately started flying in that direction. Aging younger, yeah, that would be strange. And if you're someone who plays with high latency, you might actually be able to escape the occasional bubble camp now.

00:44:19 Yeah, that's kind of my end goal. My end goal is I want to get the server and services performance to a degree where it's a struggle for our engine to keep up. That's what I want to get to. I want to get to the point where our servers are lighting your machine on fire, not the other way around. All right on. Another question that came up in the chat, Katrina Beckers. I think pretty sure the answer to this is yes, but they asked, can you hot deploy code in services?

00:44:45 Or in the sole dogma, or anywhere really, versus needing downtime. I know we do push out hot fixes from time to time without requiring a full server restart. Is this because we've been able to pull certain things out of that monolith so you can update them on the fly? You kind of said you could before when you were talking about how Sovereignty Services were now running remotely, and if something goes wrong with them, it's just going to be a few windows in the game that are broken rather than the whole client being screwed.

00:45:11 So we have a lot of different permutations of that. We even have an internal Wiki page that was made with pepper spiciness on the deployment. So the easiest one that we do is when we deploy a patch for the client. It's just the client that needs to update. The server doesn't need to go down.

00:45:30 Um, there's another one that we do is, is more of, there's a cowboy hat in the office that you have to wear, um, where you're live editing the code on TQ. Um, and that needs to be used rarely if we can't get a patch out or it's in the Goldilocks zone of not important enough to bring down TQ, but important enough to do something about. So you're like, you're like trying to fix the engine in the car while you're speeding down the highway. So that's very Min Matar sounding, but you shouldn't be.

00:45:55 And then we have the newer ecosystem with Quasar, where we're constantly rolling out updates to Quasar throughout the day, throughout the week. And by design, we've built it so that we can roll out updates to that code and try different things out and see what's wrong and kind of deal with certain things. And I think the other thing that Laraq was talking about earlier is the stuff that we tried in Sovereignty was to the point where we could get designers to make changes real time to like...

00:46:24 The planetary resources for Skyhooks. They can change that and deploy that now and the clients will get the new information and the clients don't need to run an update. We don't need to take down the server, none of those things. So at the end of the day, like we're trying to get to the point where the same way that I want to set the engine team on fire, I also want to set the design team on fire in the sense that they get to the point where they can actually just press a button and suddenly you guys have it in your hands.

00:46:50 Ja, und mit dem Updates zu Quasar-Domain-Services, ich denke, das ist eine wichtige Sache, dass es immer noch etwas passiert ist, aber in general, wenn es eine Neuheit zu synchronisieren Quasar-Service-Update mit 11.00 Uhr ist, dass wir etwas falsch gemacht haben. In general, die Dinge, die wir tun sind, sind alle Zeit. Wenn es eine Neuheit zu synchronisieren,

00:47:16 Some functionality shouldn't be yet available. That should be governed by something that is just a feature flag that can be turned on at 11, because this is where we want to make it available to people. But it's not tied to actual code deployment. And the same goes for fixing bugs. We can do it at any time. In fact, that's the main...

00:47:39 I think velocity improvement when it comes to fixing things that don't work like they should. We can be much, much faster. We can get out expansion at 11 a.m. At 2 p.m. we have the deployment of the fix and it doesn't require all the process that is required when it comes to the EVE server. The process might be either getting somebody approving using the cowboy hat.

00:48:08 Or bring Cluster down and bring it up again with the new code. But it's also the interesting thing because there's also residual culture there. Even though we've built these tools this way, we also see teams sometimes like, hey, we're going to make this change at 11. We're like, but you don't have to. It doesn't have to happen then. It's a brave new world. Embrace it, nerds.

00:48:30 Ja, das ist ein großer Teil der Geschichte von Eid Online. Die 11 Uhr ist ein wichtiges Milestone für jeden Tag. Ja, ich erinnere mich an Hilmar zu sprechen, lange Zeit über das Thema. Und es war so, die Dauntime. Er sagte, oh, wir müssen die Spiel verabschieden. Und dann werden wir das später lösen.

00:48:49 So there would only be a weekly downtime, then no downtime. And this was back in 2003, right? And then they got by with that. And I think the game was so unstable in the early months.

00:49:02 Probably even years that they needed basically that daily downtime to be able to deploy and update. And then what happened was people started to design features to account for that. Like you remember the old outposts. So before there were Citadels that you could put down. Switching corporations used to be this way too. Oh yeah, yeah, that's right. So yeah, the feature, you know, how is this going to work? Well, there's downtime at 11 a.m. Let's just, you know, connect it to that. No problem. And now you've got...

00:49:31 in der Game gradually, mehr of them, too, you know, being reliant on this thing, which ultimately you would one day like to get rid of. So sort of winding all that back is also a bit of a hassle. On that, though, I was going to ask you a little bit about the... Let me see. I have to start sort of bringing this for landing in a minute, but now I want to indulge myself by asking you some of the things that I care about. And one of them was like the no downtime.

00:50:00 um initiative that we've conducted two experiments on in the last couple of years just leaving the server up letting it run past 11 a.m and seeing what happens as time goes on the first time we ran we ran it we found a bunch of weird gremlins that were happening so we

00:50:15 Fixed a bunch of those and then like a year later we did it again. And we found those things were kind of pretty much resolved. So we stayed up longer and then found newer problems emerging. Is there no downtime, you know, is that like, or maybe like just having a weekly downtime or something like that. Is that achievable? Are we getting closer to that? So the end goal of Quasar will create no downtime.

00:50:39 Das ist die journey we're on. Ja, ich war mit einem sehr gewissen Mann, ein paar Wochen später, und er sagte mir, dass Ende der Down-Time nicht die Ziel ist, es sollte die Symptome der Greater Achievement sein. Das ist eine der meisten Inspirationalen Dinge, die ich noch gehört habe. Ich war wahrscheinlich drunk. Ja, ich weiß, aber es war ein großartiges Mann. Ich glaube, es war so gut. Aber bitte, sorry, continue.

00:51:05 Ja, ich meine, da gibt es viele Punkte. Da gibt es die Kulturelle. Wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun, wenn wir das tun,

00:51:31 Und das ist ein größer Teil des Problems. So, das ist sehr wichtig, wie lange wir können, die Cluster-Side-Side-Side-Side. Und dann, ich bin sicher, wir finden andere Dinge in der EVE-Client, als die Zeit geht, wo wir nicht ein Reset haben, wo wir nicht ein Reset haben, über 24 Stunden. So, da sind viel mehr Dinge zu haben, nicht nur all die Jobs, die wir hatten, oder die Kultur, die wir haben, oder die Muscle Reflex, die wir haben, um downtime zu haben. So, für mich, es ist mehr, dass wir, wenn wir keine downtime haben, dann...

00:52:00 mit Quasar. Das würde alles ändern. Ich glaube, was wir wollen, ist, dass wir nicht auf den Punkt haben, wo es kein Dump time gibt, sondern wir bauen neue Dinge und machen alle, die die Leute, die die Designers sind, wissen Sie.

00:52:26 Mit diesen Dingen sind wir nicht auf Dumpfangen. Dumpfangen kann einiges passieren, wegen der anderen Veränderungen, die ein bisschen schwierig sind. Es ist unmöglich, zu werden, aber gleichzeitig der flow des Spiels, sagen wir es nicht, ohne Dumpfangen zu sein, dass es die einzige Moment, die Dinge verändern können. So, ja, ein neues Content kann auf jeden Fall sein.

00:52:52 We want to have an event that starts at midnight, right? And it doesn't mean that we will have downtime at midnight, whatever. And then at some point, we will achieve that great achievement that the wise man was talking about. And there will be no downtime. I mean, I used to, when I was still a player living in the Australian time zone, I used to relentlessly...

00:53:17 Bugs CCP about downtime when I was at Fanfest or Eve Vegas or when they were visiting for Eve Down Under because in the Australian time I mean normally when downtime occurs it's it's it's in the afternoon the UK most of North America is asleep but when downtime happens at 11 a.m. here in Australia and New Zealand in that part of the world it's like right in the middle

00:53:36 der Primetime ist, wenn du mit deinen Freunden bist, oder du sitzt auf einer Gate, oder etwas ist passiert, und dann hast du die Pop-up gesagt, Dauwntime ist passiert in einer Stunde, und du hast zu sagen, okay, ich muss die Fleet safe sein, du hast, doch auf.

00:53:51 Write it out and then see and hope everyone logs back in afterwards, right? And doesn't go... I think that's also a huge part of it as well. Like when we're talking about not being able to transmit more things with the server, like how does the, you know, the meta change for when you can actually have real-time battles? Like what actually matters in that case when people don't have time to think? Like how does that change how designers build those components? Same way with downtime as well. Like there's a lot of content that has certain spawn mechanics or respawn mechanics and then having something that doesn't rely.

00:54:19 Das ist ein bisschen mehr interessant, wie es die MTOs von Mercenary Dens, zum Beispiel. Wie, wenn sie auf die Klocke arbeiten, wenn sie auf die Klocke arbeiten? Wenn sie auf die Klocke arbeiten? Wenn sie auf die Klocke arbeiten? Wenn sie auf die Klocke arbeiten? Wenn sie auf die Klocke arbeiten? Oder wenn sie auf die Klocke gehen? Das andere was ich möchte, von meinem persönlichen Interesse. Das ist ein schwieriges Thema für uns alle. Ich möchte über Chat sprechen.

00:54:46 Weil es nicht sterben. Es wird nicht sterben. Es wird nicht sterben, lange genug zu sagen. So, es war das Friday, ich würde sagen. Vielleicht war es das Friday. Es war eine gute Zeit für ein Deployment, aber es war eine gute Zeit. Every day ist es eine gute Zeit. Es war eine gute Zeit für die In-Game-Chat-System.

00:55:11 Und dann, ein paar Stunden später, hat er eine DevBlog über den Brave New World gemacht. Und dann die Chat war nicht optimistisch. Es war slow zu populateen Menschen in den Systemen. Und dann ist alles, dass die Texte, wenn man die Systeme verändert hat, was jetzt White instead of Red.

00:55:30 So we kind of just like just like quickly just take that put it over here grab the old chat plug it back in everything calm down and now we got to go back to the drawing board. Can you talk a little bit of okay so two things if you can you tell us a little bit what you know went wrong but if you're not in a position to actually discuss that can you just sort of give us some idea like like why is this one thing being so elusive so really hard for us to just sort of settle you know get to grips with.

00:56:00 Oh, that's a lot. So first off, chat was one of the first things that was attempted to be pulled out of the Eve server. And because it was the first attempt, there was a lot of, there's just a lot of inexperience in the organization when that happened. And so we did a lot of things and we made a lot of the classic mistakes of like, oh yeah, cool. Like we can pull this out and all the chat happens at XMPP. Oh wait, all of the auth is still happening on the proxies, which means a monolithic tentacle comes out and starts messing with the chat server.

00:56:30 So we've kind of improved. We spent a lot of time improving that and getting to a stable place. But then it's to the point now where we're now trying to untangle a bigger problem. We talk about how does tech unlock design capabilities. And in a sense, kind of how do we divorce the concept of communication and intelligence gathering? Because that's what we really want to unlock for designers in the sense of, okay, now we get into a system where we want.

00:56:55 Something to be able to switch a system into delayed mode or whatever the case might be. Or it projects more people into local than are actually there. Something along those effects. I know everyone just clutched their pearls, but like these are the kind of things that we want to unlock for designers to kind of toy with and play with in those ideas. And it's really hard to do that when it's in such an alien ecosystem that's not really connected to ours.

00:57:16 So this is why we're trying to bring chat into Quasar. And part of a lot of those pieces are actually even connected to Skinner and the Shipskins, because that was the first time when we were toying with this idea behind the scenes, it was the first time that we actually tracked the location of players, like knowing where they were, because that was important for local chat.

00:57:38 So that was something that we stood on top of and actually expanded for what Blark I was talking about with tracking things in certain grids. So even though that feature didn't necessarily get off the ground, it helped us get to where we are and kind of the details that are in that blog post as well. But there are a lot of pieces there. There's a lot of like, you mentioned the text color, like there's also the name color. There's like...

00:58:01 We found out that there were so many things in that little window telling people so much information that it kind of revealed itself to be like one of the biggest knots that we have in the ecosystem in the sense of it's where everything converges for character portraits, character names, standings, corporation membership, alliance membership, contact standings, war declaration standing, like all of these things intertwined to just make that one little color change.

00:58:28 So basically trying to unravel all of those, make sure all those pieces there and trying to make sure it's close to what it was before we started trying to touch any of it. To get to the point where we can start iterating on it and start playing with those ideas around the fact of what if you could have...

00:58:47 I imagine playing Supreme Commander back in the day, and I think it's one of the factions that had a radar scrambler where it made it look like there's 300 units coming your way, and you had no idea how many of them were real and what was the real ones. It reminds me of back in the day when we used to jettison Rifters from Orcas just to go like, yeah, yeah, we got like five people, let's fight. It'd just really be just two of us.

00:59:11 Für mich ist es auch ein touchy subject, weil ich den ersten Versuch auf den ersten Versuch zu werden, um den neuen System zu werden, die wir jetzt nicht verhindern können, weil wir andere Dinge zu tun haben. Aber vielleicht bin ich weitergekommen, aber ich denke, wir haben viel gelernt, wir sind sehr nahe, und irgendwann werden wir endlich zu machen, weil jetzt ist es mehr oder weniger, was Chimichanga gesagt hat, es ist eine Sache,

00:59:40 Making sure that the small levers that control how different distributed systems interact with each other work correctly. But all the building pieces are actually there and we kind of know that it will work. That's my view of things. And I hope we'll get it out sooner than later. Or at least not very late. It is inevitable. It's inevitable. Yeah. All right.

01:00:07 Cool. Okay, so it's almost four and I have to let you go. So I'll wrap these up. I'll start wrapping this up now. There was just a couple of things I wanted to call out. Oh man, there's one more question I want to get in from Eve Perfect Wing. Answer this really quickly. With something like this, you know, sometimes when there are large fleet battles and they go on for a long time and people are, you know, chewing on a Keepstar or something for so long when it eventually blows up, a kill mail doesn't get generated.

01:00:35 Und dann haben wir, so ein Poor Bastard, hat es zu manually go und re-construct es, wenn es für eine wirklich, wirklich kritische Sache, die die Leute wollen wissen. Ist das ein Symptom von Prolonged Fighting in Tie-Dye, also? So, kann das vielleicht bemitigated, wenn es etwas mit der höheren Tick-Rate ist?

01:00:57 Let's not go there then. We don't have time for it anyway. So that's fine. Let me see. So I just wanted to, yeah, we'll wrap things up. But before I do, just a shout out. So I think tomorrow or soon TM will be the third in the series of the Eve Evolved Dev Blogs that we're publishing this month. This next one is going to be all about ESI. And we've got some really good news for you on that front. I know that CCPE Stroopwafel has been in Twitch chat.

01:01:22 Und er hat einen sehr, sehr langen Dev-Blog über das. So, wenn du so, long, dry, technical documents möchtest, du kannst dich aus. Du kannst nicht sagen, dry. Er hat einen guten Weg zu machen, wenn du diese Dinge einstürztest. Wenn du ihn und erinnert hast, ist das Third Buddy, Dev-Blog, dass er, ich glaube.

01:01:37 Last month is really good. I also wanted to let you know that on the 28th, because we've talked about mass tests today and how valuable they've been, you know, like it was a Skinner mass test, but it helped us, you know, helped you sort of do all this research into tick rates. And I talked about how we sort of secretly were using mass tests to try and help us find that undocked bug.

01:01:58 The mass tests are really, really valuable to us. We've got another one planned for the 28th, so put that in your calendars. There'll be a forum post and a news item about that later, so you know the specifics and exactly what it's going to be about. And I also want to just encourage everyone to tune in tomorrow at 1400 UTC when we'll be doing Lift to Feed, which is our weekly dev room, where the community team just jumps in spaceships, flies around New Eden and gets shot in the face, and we have a blast. It's really, really good.

01:02:22 So, thank you, CCP Chimichanga and CCP Blaraka for coming and talking to us today about the server and Eve Evolves and taking these questions from Twitch Chat. Thank you, Twitch Chat, for coming out and watching us. And I guess, unless you guys have any PSAs of your own, we can stick a pin in it right there. You good? Yeah, I'm all good, man. All right, no worries. Thanks, Dan. No worries. All right, thanks a lot. Thank you, Twitch Chat. Goodbye for now. And look out for the next blogs. Bye.