Dealing with re-flown radiosondes (e.g. r2home)

[helioshk]

It would be good to have an option to limit the length/time of a balloon path. Otherwise it could become a bit a mess with the massive reuse of sondes, and it takes very long to load.

[darksidelemm]

The sondehub interface and API was never really designed with radiosonde re-use in mind. We expected each launch to have different serial numbers.

Unfortunately yes, we are now seeing sondes re-used with the same serial number used each flight. I would have preferred to see different serial numbers used on subsequent flights, but that's difficult unless you start to use debug interfaces that Vaisala probably don't want their customers to use... and then you have a whole different problem of managing these modified serial numbers.
I did put some thoughts on this here: https://github.com/projecthorus/sondehub-infra/wiki/Re%E2%80%90use-of-Recovered-Radiosondes#serial-numbers
Vaisala probably don't want you to do this though, and I suspect their ground-station software (which I'm assuming is still in use on these re-flights) probably wont like the changed serials either.

More thoughts on this:

• Users can either select an age to display (up to 12 hours, mainly due to the huge volumes of data from the many many radiosonde fllights), OR they can search for a specific serial number.
• When searching for a specific serial number, the data might be quite old. We have to ignore the age selection otherwise many, if not most sondes that are searched for will never show up.
• For most historical sondes, the data comes from our S3 archive, not the 'hot' ElasticSearch (ES) database. We archive out to S3 because storing it all in ES is incredibly cost-prohibitive. This storage is a single JSON file per serial number. There is no way to select data by time when retrieving data for a serial, you get all or nothing. For 99.99% of cases, this is not a problem. For these re-flown sondes, it now is.
• Filtering or limiting time ranges could only realistically be done on the front-end. This won't help load times, it still has to process with all the data.
• Any kind of finer-grained filtering on the front-end would some kind of additional input fields. I don't know where these would go on the tracker page.

So in short, dealing with this is not easy, and I don't see how we're realistically going to handle it in the near-term.

Some more information on widespread this R2Home approach is going to be, and in what timeframes would help in assigning a priority on working out a solution. Right now we don't really have the developer time to solve a problem that's only really affecting one launch site.

[helioshk]

Any kind of finer-grained filtering on the front-end would some kind of additional input fields. I don't know where these would go on the tracker page.

A suggestion would be to use the Telemetry Graph. It allows a time selection to be made. This could be applied to the map, such that only this section of the path would be shown on the map.
Or the pulldown menu which is already used to only show sondes of a certain age (since last reception).

[TheSkorm]

Does r2home transmit anything unique per flight over xdata?

[helioshk]

I don't think it does.
Maybe @YohanHadji can say more about this.

[yphadji]

Hello there, sorry for the DB issues created by re-flying the sondes. For now, no specific data is transmitted over XDATA, not all sondes have a nice XDATA interface. For R&D flights from St.Gallen we can easily change the sonde serial between each flight, but for operational use the sonde serial will probably stay the same.

Some more information on widespread this R2Home approach is going to be

Single use radiosonde should disappear. Our ambition is to make all radiosondes reusable. It's going to take a while, maybe 10 years, but will eventually happen.

Isn't there a way to automatically filter each launch in a different database entry? Radiosondy is doing something where you have the sonde serial and then a letter added at the end if it's a reflown sonde, with the letter changing each time it's flown one more time. I would replace the letter approach with a number to not be limited to 26 reuses.

I have no idea how S3 or ES works but this is what I would have done if I had to do it for our database:

If new data point received, if this serial was already received in the past, if the last data point is more than 6h away, increment the reuse counter for this radiosonde serial and store the following data with the new tag "radiosondeSerial-reuseCounter"

Then when a user searches for the radiosonde serial, they would see a list of:

U425172-1
U425172-2
U425172-...

And be able to only load on the website the flight they're interested in.

Let me know if there is anything I can do to make things easier for the radiosonde tracking community.

[TheSkorm]

The pipeline for our ingestion and websockets doesn't have access to a DB, and is really something we'd prefer to not to add due to the extra cost, reduced reliability and increased latency. We'll have to think about this more