A New Way to Obtain GP Data (aka TLEs)

by Dr. T.S. Kelso
2020 May 27
Updated 2024 Sep 14

Background

The US government has provided GP or general perturbations orbital data to the rest of the world since the 1970s. These data are produced by fitting observations from the US Space Surveillance Network (SSN) to produce Brouwer mean elements using the SGP4 or Simplified General Perturbations 4 orbit propagator.

Many of you are familiar with this data in the form of TLEs or Two-Line Element Sets. TLEs were designed to provide the minimum data necessary to propagate the orbit of a resident space object (RSO) at a time when both bandwidth for transmission or digital storage were extremely limited. In fact, at the time, transmission might be via fax, hard copy (postal delivery), or even read over the phone and storage was handled using punch cards or magnetic tape.

While this format has served us well for many decades, it has not been without its share of problems. For example, the choice of a two-digit year caused many problems approaching Y2K—problems that were side-stepped by redefining what those two digits represented—but that Y2K problem persists fully 20 years into the 21st century. And now we are approaching another milestone where we will no longer be able to catalog all the objects we track within the 5-digit catalog number limitation of the TLE format.

One of the key drivers forcing us to consider tracking more than 100,000 objects is the activation of the Space Fence on Kwajalein Atoll. The Space Fence reached initial operational capability (IOC) on 2020 Mar 27 and is expected to track far more than the ~26,000 objects currently tracked by the SSN—perhaps by as much as an order of magnitude.

And we are expecting to see public availability of data from the Space Fence starting some time this summer (2020). The 18th Space Control Squadron (18 SPCS) has already transitioned internally to using 9-digit catalog numbers in support of these changes and we expect 18 SPCS to release data from the Space Fence using 9-digit catalog numbers.

The Solution

CelesTrak—working closely with Space Track—has begun making the GP data available via standard queries using the Orbit Mean-Elements Message (OMM) that is part of the Orbit Data Messages (ODM) Recommended Standard CCSDS 502.0-B-3 developed by The Consultative Committee for Space Data Systems (CCSDS) in November 2009. We are recommending the XML format of Version 2.0 of the OMM, as defined in XML Specification for Navigation Data Messages (CCSDS 505.0-B-3) to ensure future compatibility and interoperability.

There are XML and KVN (key-value notation) versions of the OMM standard and CelesTrak will provide all mandatory elements of those formats. Some elements may be blank (KVN) or null (XML), if not available via the current TLE format. An example might be that an object in the current analyst sat range (80000-series) typically will not have a name (OBJECT_NAME) or International Designator (OBJECT_ID).

Use of the new data queries is NOT required for most users at this time, since CelesTrak will continue to provide data for RSOs with 5-digit catalog numbers in the TLE/3LE or 2LE formats. The current focus is to provide software developers a way to test modifications to their code to support using the new OMM format and 9-digit catalog numbers. Legacy links to fixed .txt files will continue indefinitely, although links on web pages will eventually be transitioned to use the new GP query and allow users to define their default format. Of course, TLE formats will not support objects with catalog numbers above 99999.

Additionally, data will be provided in both JSON and CSV formats, using the same keywords and definitions as provided in the OMM standard (CCSDS 502.0-B-3, Table 4-1), although null/blank or redundant (e.g., CENTER_NAME = EARTH, REF_FRAME = TEME, TIME_SYSTEM = UTC, MEAN_ELEMENT_THEORY = SGP4) mandatory fields will not be included.

CelesTrak will work to ensure that all GP data received via 18 SPCS and Space Track will be ingested in a way that supports users requesting GP data in any of the TLE or OMM formats.

The Implementation

All GP queries on CelesTrak will take the form:

where {QUERY} is:

{QUERY} must be uppercase.

Allowed formats are:

The FORMAT specification is optional, but defaults to TLE (to support legacy queries).

Examples:

There are also queries to show the first or last GP data available. For example, if you wanted to see when the first 18 SDS GP data became available for the Transporter-11 mission (2024-149), which was launched 2024-08-16, you might use:

This type of query is also useful for getting the first data for objects in a debris event, which might take days or even months to get cataloged:

And we use the last GP query in things like our table of Lost Objects (objects which should have GP data but for which none was found in the last 30 days):

as well as in our table of Recently Decayed Objects:

These are custom versions of our general table queries, which layout basic information for both the GP and SupGP data, in an interactive table. These table queries use the same structure as the GP queries. Here the FORMAT specification is used to define the format of any data linked to the table. The content (columns) may vary, depending on the focus of the data.

Examples:

These table queries can also include a variety of flags to further customize the table for specific uses.

Flags:

Examples:

And for information on how to query SupGP data, see How to Perform SupGP Queries.

For Software Developers

Software developers looking for code to input or output these formats in a variety of languages are invited to check out the Space Data Standards web site developed by our partners at Digital Arsenal.

There is code to support C++, Kotlin, Java, C#, Go, Python, JavaScript, TypeScript, PHP, Dart, Lua, Lobster, Swift, and JSON Schema. There are also examples converting TLEs to OMMs in XML, KVN, JSON, CSV, and FlatBuffers. And there is a GitHub repository for the code that allows users to submit suggested changes.

Summary

Providing GP data in the OMM format provides a way forward for all software developers to continue to support using SGP4 in their applications and add support for 9-digit catalog numbers. In addition, it elimates the Y2K problem still coming NLT 2057 by using the ISO 8601-1 (WD) date and time standard and also supports the use of Unicode characters for satellite names from non-English languages.

FAQs

Q: Why don't we just modify the current 5-digit catalog numbers to include letters to increase the range of objects that can be represented?

A: There have been numbering schemes suggested that would extend the range of catalog IDs that would fit in a 5-character field of a TLE. The easiest would be to extend the current 'numbering' by allowing the leading character to go from 0-9 and then A-Z. At best, this would allow for tracking 360,000 objects (assuming you don't discard I and O, as has been suggested by some). That could work for the Space Fence, but ignores other potential large increases in objects tracked due to the development of large constellations (which currently propose as many as 100,000 new satellites) or additional large debris events that might occur due to the collision of large uncontrolled rocket bodies.

Even allowing all 5 characters to go from 0-9 and then A-Z would only allow 60,466,176 catalog IDs, but 18 SPCS is already assigning catalog numbers in their new analyst sat range of 7995xxxxx (or over 799,500,000). So, there would be no way to map these 9-digit catalog numbers to 5-character IDs.

And the ability to fit new characters into a field does not make the problem of interpreting the change in software go away, any more than the change in interpretation of the 2-digit year did for Y2K. All the code using these catalog numbers will have to be updated to change their interpretation, with a cascading set of implications. Including letters means the field will no longer be able to be simply validated by verifying that it is an integer and integer comparisons will no longer be possible. For example, some TLEs use leading zeros in the 5-digit field while others do not. But a value of 00964 parses as an integer the same way as 964.

Every software developer will need to update their code to adapt, so this is an opportunity to do that in a way that provides future flexibility and no longer relies on the limitations of fixed formats.

Q: Why do we have to use formats that are so verbose?

A: While it is possible to use formats that are less verbose, they do not provide the framework to ensure future interoperability that an international standard like the OMM provides. Software developers—particularly those developing to support systems for use in satellite operations, ensuring safety of flight, or national security—are strongly encouraged to use the recommended OMM XML standard. Others that do not support critical functions like these may choose the JSON or CSV formats based on the OMM standard (but not currently part of that standard), although there is always the possibility that these could change. CelesTrak has worked hard to ensure that doesn't happen, going back almost 35 years now, and will endeavor to make changes in a way that maintain backward compatibility whenever possible.

But the reality is that using a full XML version of the OMM for a single object takes about 1,200 characters (including all the overhead XML formatting for a set of OMMs) compared to the 168 characters of a three-line element set. That is a factor of 7 larger. When I first started CelesTrak in 1985, I had a 1200 baud (120 Bps) modem on the system for a single user at a time. Today, CelesTrak has a 1 Gbps connection that supports over 400,000 unique users a day and my home Internet service allows up to 350 Mbps—that's a factor of almost 300,000 times faster. And my hard disk at the time was a whopping 5 MB—I have several 10-TB drives on my home system (including 5-TB external drives that cost a tiny fraction of that 5-MB one) and CelesTrak has 400 GB of SSD storage. That's a factor of 80 to 1,000 times as much storage. And we will continue to see similar advancements in bandwidth and storage that make these differences irrelevant.

And the long-established (February 1998) XML format has extensive software support to allow easily ingesting the OMM XML data.

FAQ Addendum

Added 2024 Aug 30

Q: Why am I getting blocked trying to download GP data?

A: CelesTrak only checks for new GP data once every 2 hours, so there is no need for you to check more often. In fact, when you do, that uses limited resources needed to support ~300,000 unique users on CelesTrak each day. Because some users, if left unchecked, will download the same file every minute of the day (that's 1,440 times or 120x the update rate), every day, we have had to implement limits, which are enforced with temporary blocks.

If the IP address for you (or your process) is being blocked, CelesTrak sends a custom HTTP 403 error message explaining why you are being blocked:

Of course, that means you (and your processes) need to be checking for error messages. If you check the query being blocked in your browser (from the same IP address that your process is using), you should immediately see what's going on. If you still don't understand why you're being blocked, you can send me a screenshot, like the one above, which includes the IP address, and I will look into it. I often help users who think they are only making a small number of requests realize their process hit an unexpected response and just started hammering the system.

Your process can avoid this situation by checking for a successful response (an HTTP 200) and being prepared to handle unexpected responses. If some other response is received, your process should only try 2-3 more times before stopping and reporting the problem to a human. On CelesTrak, we follow this approach when downloading data from any other sites. When a serious problem is encountered, our processes actually send an SMS (text) message to ensure quick attention. And each of our processes maintains an easily accessible log file on our Dashboard to clearly state what happened.

Failing to have automated processes check for errors can not only waste CelesTrak's limited resources, it can cause users to blindly continue to use data that hasn't been updated for days, months, or even years.

For example, CelesTrak changed the primary domain to https://celestrak.org years ago when we became a non-profit on 2021 Apr 26. That means if you use the .com domain, CelesTrak tries to redirect your query to the .org domain and sends an HTTP 301 (Moved Permanently) response. Your browser knows how to handle that, but your process may not, causing it to hammer away until we block it. Once the system sends more than 10,000 HTTP 403 errors to an IP address in a day (yes, that happens almost every day), that IP address is put into the firewall and requires manual review to find and remove it.

To put this in perspective, let's look at a snapshot from 2024 Aug 29 (yesterday). Of the 864,374 successful accesses on the site (HTTP 200), 571,795 of them started out on the .com domain and received an HTTP 301—almost 3.5 years after the change. Not only does that mean CelesTrak has to execute (and log) twice as many queries, it could mean those users aren't getting any data.

On 2023 Dec 28 (8 months ago), we removed a number of legacy static .txt files, which only use the TLE format, in an effort to get users to prepare for running out of 5-digit catalog catalog numbers in the main part of the SATCAT (see notice on X). If you thought that happens at 99999, you may be surprised to discover that is not true (see notice on X). When we run out of 5-digit catalog numbers at 69999, new data will not be able to be created using the TLE format. In the meantime (just yesterday), we saw 30 of these deleted legacy files requested between 194 and 3,301 times each (42,082 times total). These users have likely received no data for as much as 8 months.

In fact, CelesTrak already provides GP data for USSF Space Fence analyst objects that use 6-digit catalog numbers. You can see that toward the end of this table but if you click on the link for the TLE-formatted data in the header, you will notice it does not include any of the 27xxxx catalog numbers (it only shows 8xxxx catalog numbers). Using a format like JSON will include all of the data. And if you look closely at the SupGP data for a recent Starlink launch, you will notice we are already using the 18 SDS 9-digit launch nominals catalog numbers (in the 799xxxxxx range). The same will be true for future Transporter and Bandwagon launches.

The bottom line here is that these blocks are in place to get users' attention that their processes are likely not working as expected. It is unlikely that anyone is manually requesting hundreds or thousands of downloads a day. But since we don't have user accounts, we can't just send you a message. So, we use these progressive steps to (hopefully) get your attention, so that you aren't left blissfully unaware that you may not be getting new data or of impending changes to data formats.

Q: How can I avoid getting blocked?

A: Now that you know why you are getting blocked, it's actually pretty easy to solve the problem.

First, turn off the process causing the issue. Once you do this, the temporary blocks will be automatically removed within 2 hours.

Next, modify your process to use the latest data you downloaded by default. If nothing else, this step will allow your process to continue working in the event of temporary Internet issues.

Finally, add a step before using the latest data to check the data file's timestamp to see if it is more than 2 hours old. If it is, re-download the data to that file and proceed to use it. Otherwise, just use the latest data. Pretty simple. Note that when set up this way, if you are a software developer testing your code, the process is still only going to download the data once every 2 hours (at most).

Q: How else can I help CelesTrak make the most out of its limited resources?

A: First and foremost, only download the data you need, when you need it (are ready to use it). Back in the day (circa 20 years ago), many processes were written to harvest all of those legacy static .txt files, many times a day. Often that data just took up disk space and never got used. Plus, downloading data every 12 hours, just so you have it, likely only meant it would be six hours old (on average) when you went to use it. Now that Internet speeds are faster and connections are more reliable, it's better to just grab the data when you need it, which can also randomize when that occurs (which spreads out the load on CelesTrak). And don't be that person that still needs to download all of the data—including that for the old Iridium satellites that are all long dead, uncontrolled, and no longer generate flares—oh yeah, and because that file is now gone.

Of course, if you aren't a software developer or are using an application written by someone else and are seeing problems with getting blocked, please be sure to pass this information along to them so that they can take the necessary steps to avoid it. That not only helps you, but others using the same software.