As we enter 2025, it is a prime opportunity to look back on what we have achieved in the closing months of the network's 26th year of existence and direct our attention to the next steps for the organisation we are all so passionate about.
We, firstly, want to take a moment to thank members and all staff involved in the great success of Crowded Skies 2024. Across the day, over six thousand of you connected to the network enabling us to see a 16% increase in connections from last year and a thoroughly enjoyable day for all, even our Development Operations Team, who oversaw a smooth and stable network across our web systems, flight and voice servers across those 24 hours.
Looking to what the next 12 months has for our network, we seek to continue advancing the current web and software systems alongside new ones which our teams have excitedly begun developing. In the spirit of expanding our IVAO Family, our teams also enter 2025 advancing our continual wish to be inclusive and transparent with our community. A New Year's resolution is us placing a huge emphasis on working with external developers to contribute to the success of IVAO through open-source and third-party projects that ensures we propel our network with the best tech and tools for our community.
✈️ Altitude: Pilot Client
Altitude is currently responsible for 98.1% of pilot connections to our network in the last six months and continues to provide a stable platform for members across an array of flight simulators and operating systems.
👏 This is also a perfect occasion to acknowledge our hard-working developers who ensured that the network was ready for the release of Microsoft's Flight Simulator 2024 and that pilots could connect from the very first day.
With the implementation of Network Rules 2.3.10 and 2.3.11 back in December, we now urge pilots to ensure their version of Altitude is at least 1.13. Any versions lower than this will be unsupported to connect to the network beyond 31st January 2025. This will enable all users to experience and enjoy Voice Unicom implemented in November 2022.
Altitude 1.14 now has a release candidate version which will be proved by members of our Software Testing Group (more information below 👇) before public release. This version has some great new features, the most popular being Joystick push-to-talk, and many others such as visual aids to assist with debugging potential connection issues, the possibility to adjust the gain on your microphone with a visual volume meter to ensure you are not too loud nor too quiet and the added ability to ignore specific callsigns MTL models for those who may be using a third-party client such as JoinFS for close formation flights.
📻 Aurora: ATC Client
Aurora is accountable for 97.9% of ATC connections to our network for the last six months. The version for Windows continues to be maintained and updated with feature requests from our community. (Find out more below how you can recommend a feature!)
The platform continues, on a regular basis, to have multiple features added to provide ease and enjoyment for members utilising the software, such as new timers to assist with separation, the introduction of customisable traffic symbols and the further customisation of static labels allowing divisions and users to mirror the traffic labels used in real life at control centres.
As we seek to advance the Unix version of Aurora to match the level of features and usage as our Windows version, we are pleased to confirm that we will be converting this platform to be open-source. Members of our community will be able to contribute to the ever-successful ATC client. An announcement in due course will follow about this exciting opportunity on how to contribute.
📋 Features/ Radar Website
Back in July we soft-launched our new Features and Radar platforms. Within our organisation it remains pivotal that the community has an active voice in shaping the very software and web systems they use on a day-to-day basis.
To enable us to test the system on a live environment and finalise the processes our User Support Team would follow, we deliberately did not provide links to the systems but some inquisitive members managed to work out the location and began to help us test the platforms out.
Since July, our members can visit 🔗 https://features.ivao.aero and publish the changes or new implementations they would like to see across our current catalogue of software and web systems. Using this website, the community can support the ideas that they would love to see implemented and our teams will prioritise and implement the features with the most backing from the community. The website requires members to have an active IVAO account to make, and comment on, suggestions.
Accompanying the new Features website is the second platform, called Radar. The focus of Radar is to improve communication and provide updates to users related to bugs and other issues amongst our software and web systems. Members of our User Support team will add confirmed issues to this platform and inform affected members on how to keep up-to-date with the progression of a fix. You can see the current known issues by visiting: 🔗 https://radar.ivao.aero.
📜 Third-Party Forum
As we continue to develop on the community spirit of our network, we are pleased to announce a new sub-section of the IVAO Forum dedicated to enabling those who develop third-party applications, both software and web platforms, that utilises one of the network's many APIs or the new Aurora SDK that's on offer, to showcase their work and answer questions or queries regarding their platform.
🗺️ Webeye
Webeye continues to be one of the network's most popular tools with our fantastic web development team continuing to enhance it, week in, week out. In the last 30 days the Webeye has been visited over 2.2 million, yes 2.2 MILLION, times!
Our web development team continues to move forward the features of Webeye for the benefits of our users around the globe. In recent months we have seen the addition of the unique user count which displays the number of unique members connected in the last 24 hours, a new mode for streamer (requested by 379926) which hides the personal information of members and stops personal data being displayed whilst streaming Webeye
🧑💻 Tech Talk
On a daily basis our Web Development Team are continuously working to improve the stability and performance of the tools which members use. Many of the changes they make are not visible to the naked eye but have a significant improvement to the experience of community. To highlight a few of the changes and to intrigue those with a technical interest, our team has compiled a list of the changes they have made.
🗺️ Tracker
For many, the tracker website is simply the place to check where you have flown in the past but most will not know that this system is powered by a tremendous amount of data, primarily about a member's flight.
The team are now pleased to advise that we will now be storing all data for one year, both controllers and pilots, with the exception of the tracks which will be stored for three months. They are also looking to increase the frequency of position updates from once every 15 seconds, to once every five seconds offering a smoother track and enhanced flight data.
To increase the performance of the Tracker API a new cache layer was added directly at service level. To increase stability, a new HTTP library has been used. Performance and stability improvements have also been made to our tracking software (tracker-loader).
A great new addition to the codebase is a feature related to pilots who have been disconnected from the network and then re-connect shortly after, say due to an internet issue. Historically this would create two tracker sessions but thanks to the work of the team so long as a member re-connects in the air using the same callsign and within five minutes of disconnection, the tracker will automatically continue the previous session to keep the flight going. This was requested on our Features Website by 733193 and, with support of the community, was implemented by the team!
🧠 Caching Tracker Data
🪦 Our previous Tracker caching
The issues we were faced with was that each visitor to Webeye, inclusive of all bots and third-party scripts, were fetching data from us every 15 seconds. Getting all this information from the database and displaying it in the right format takes us around 150ms per endpoint, which is usually fine, but once you multiply it by the number of requests, it becomes a complete mess. Our first layer of caching is Cloudflare, all web traffic goes through them, and they will try to cache, as much as possible, the requests to our APIs as the same data is used for everyone in the same 15-second window.
By doing this we already prevent around 96.3% of requests routing to our servers which relieves a large amount of strain on our systems. As these requests are no longer hitting our servers, there is no reason to measure the latency as communication now is only between the user and Cloudflare. With these changes we now, therefore, only need to provide the data to Cloudflare servers, and whilst there are hundreds of their servers along, it is significantly lower than before.
You may also have noticed on Webeye that you can only see a member's personal data (such as First and Last Name) when you are logged in, but this disappears once you are logged out. This complicates the process as a method of authentication is needed to be completed on the API to verify if we can send this personal data or not to individual members. Cloudflare sadly cannot do much for us here because they have no real method to verify against our membership database. The portion of requests to verify if Cloudflare can show a member the personal data or not still has to go through a validation process with our servers.
As you can see from the image below, we still receive up to 50 requests per second reaching our servers for validation on an ordinary Monday Evening (image below from 6th January 2025) where 865 people were connected to the network. Note, this is just one endpoint of the many we offer and, in some instances, we are receiving ten times the number of requests on most other endpoints especially those which cannot be cached. That day alone (6 Jan 2025) we had 48 million requests and an average of 500 requests per second on our servers.
🆕 What we have implemented for Tracker caching
Once the requests arrive at our API, we can easily judge the difference between a valid user and a non-valid one. We then gather the information from the database, format it as needed and, before sending it back to the user, make a copy in a shared space called "Redis" where the data will be available to all copies of that API as we run between two and eight mirrored version to help support the load. Using this method, in a 15-second window, only the first authenticated and the first non-authenticated request will make a database request and cache it. After this, all the requests which follow will just query which version of the cached data to send back.
When doing the math, this represents a 99.75% reduction in requests made to the databases when receiving 50 requests per second. The results are seen in just 5ms instead of 150ms and provides some relief for our systems for the remaining time until the next 15-second update is due on Webeye. If we continue to use 6th January as an example and calculate both layers of caching at the busiest minute that day, 19:45z, we received around 97k requests to the API but only eight requests, two each at :00, :15, :30 and :45 seconds, were made to the database (99.9917% caching) instead of 3.36k (420 times less).
We could optimise this a little more by only making one request every 15 second and remove the personal data at the formatting stage but it would require more work which does match the benefits to implement it. I guess we can agree that eight requests, per minute, per endpoint is very well optimised and does not have the need to be over-engineered.
TLDR: We went from 96.3% caching to 99.9917% caching with ten lines of code and divided the number of Webeye map requests hitting our API by 420!
🛜 HTTP Library
🏚️ What we used to have
To be able to support the load that bares its weight on our systems which we have mentioned above, we had to ensure that we had an API which could serve the huge of data being requested in an extremely short amount of time. Our APIs are written in TypeScript but we were faced with performance issues around a year ago with an increase of visitors to Webeye and other surrounding factors. We decided we needed to move to another language made for performance and we selected, 🦀 Rust.
Our first implementation, which we relied upon for eight months, utilised a well-known Rust Web Framework, 🚀 Rocket. We faced an issue where for some reason, approximately once-a-week, one of the API copies would become unavailable, a little like a frozen flight simulator, and partially blocked the database because of this. It was enough of an issue that the API required restarting to get it functioning correctly again, but this restart was not something we could keep doing. We needed a permanent fix.
This had to be fixed ready for Crowded Skies in December and testing began around two months prior, seeing how far we could push this Rust API to ensure it could support the 6.2k members, alongside the hundreds of other third-party tools, trying to access Webeye that day. We observed that when a high number of requests arrived at the same time, and the server was already busy trying to support other APIs, this Rust API was very likely to freeze.
🔧 What we implemented
We looked into known issues within the Rocket framework that would explain such behaviour and realised that the mechanism it uses behind the scenes to handle multiple requests at the same time was not the best one.
We searched for an alternative and found another library well-known within the Rust community, ⚙️ Actix. It relies on *️⃣ Tokio, which is one of the best libraries for handling concurrent tasks. It took us around a week to adapt the code to the new library, test it, and make sure everything was optimised and functioning correctly before pushing it to production. Since this implementation we have not observed, nor has been reported, any freezing and the database has not reported any partial outages to blocked requests coming from that API as it had previously done so.
📈 Progression of new platforms
🎓 Training System
The highly-anticipated system by many of our community, which will enable us to completely overhaul the product we deliver, has some great inertia and is making good progress. We wish to thank our colleagues in the Training Department for their collaboration and partnership in deliver this exciting change for the network. We, regrettably, do not have a timescale for release yet but as soon as we do it will be announced through official channels.
👮 Supervisor System
Whilst this platform is an internal system, it is worthwhile that our wider community is aware of its development. Our team have been working on overhauling the outdated system currently in place to replace it with a modern, streamlined system that significantly reduces the workload of handling member's request for help and the suspension of troublesome individuals on a daily basis.
This new system enables Supervisors to no longer need to be connected to the network using our software, such as Altitude or Aurora, and can instead assist members straight from the web browsers including the ability to chat with those asking for help on the network.
🕰️ Migration from Legacy System(s)
Work continues to be underway to remove legacy systems slowly, but surely, and the progression of these two systems above are crucial for that progression. Once these two platforms are introduced an administrative platform, where members can sign up and see their profile, would be all that is left to say goodbye to the web systems which has served us since our inception.
🎨 IVAO Web Design System
To ensure that we offer a universal look, feel and design of all web systems we offer now and into the future of our organisation we have been working closely with the Public Relations Department to create a universal design system which we are preparing to convert from an artist's impression to physical lines of code.
This design system will ensure that web systems have an "IVAO-feel" about them and ensure all buttons, colours, fonts, layouts, forms, modals, everything web, looks the same (with an added independence, naturally). To foster that spirit of transparency mentioned above, this library/package will be made public.
⏳ Other changes
Additional changes have been made to the MTL catalog to acknowledge and display the author of the model used. Improved performance was made to the web proxy to enable efficient caching in preparation for Crowded Skies, databases were swapped from legacy platform to new ones in preparation for the completion of legacy web system migration alongside various bug fixes and performance optimisation when cleaning up the tracker and core databases.
🌍 World Server
With the planned de-commissioning of IvAc and IvAp in January 2025, we are preparing to make the next steps to enhancing the already fantastic World Server.
With the network now fully in ownership of the source code for the software which connects to our network we can implement much needed changes to enhance the visual experience for our members. New versions of world server implementing new features will be tested by our Software Testing Group in due course.
🙏 How can I help as a member?
💻 Software Testing Group
Our new Software Testing Group is live, and you can become a member! The involvement of the very community that uses our software on a daily basis is pinnacle to ensure we deliver quality and stability. Do you fancy trying out, testing and feeding back on Alpha versions of our software, then get involved! You can read more about shaping the future by visiting: 🔗https://beinvolved.ivao.aero/.
👫 Join our team
We have a variety of exciting vacancies within the Development Operations Department and the wider-network. You can see all current vacancies advertised on the IVAO forum and view each vacancy for more information on how you can make a difference.
🚧 Build your own
IVAO has a wealth of data accessible to help our community use their creative mindset and wealth of technological knowledge to build tools and software which can enable a user to enjoy the network further. You can view a list of our available APIs here and gain access to them through our Developer's Website.
