Liberation through Computer Literacy

[ejolson]

Presently I'm using spaghetti to measure the stochastic parameterisation of turbulence models

What is the accuracy of a piece of spaghetti as a measuring tool?

Not very good, though with many pieces the accuracy improves as does ones knowledge about the inaccuracy. I much prefer raspberry pie because spaghetti frequently gets caught in my throat. More information is here.

https://en.m.wikipedia.org/wiki/Spaghetti_plot

[Heater]

Strangely enough the algorithm I have in mind to tackle this problem involves spaghetti.

[bensimmo]

For sure people don't want to waste time on useless things on their shiny new and expensive super computers.

But they do brag about how fast they are at running industry standard bench marks. Which is very important, national pride and all that.

So why not beating the world record for the digits of Pi as a benchmark? Sounds like a natural to me.

Is it because they would be embarrassed by how slow that would make their big expensive machines look compared to amateur lash ups of a few PC boxes?

Once upon a time they would break how fast they could crunch through seti@honme units, ClimatePrediction, Rosetta@Home or Find-A-Drug or LHC and then BOINC messed it all up and benchmarking didn't work so well, graphics card became better than CPU's for a lot of them and we'll ...
Times move on. Doing useful stuff while you benchmarked didn't look as pretty.

[bensimmo]

@John_Spikowski,

Will objects have properties?

For the purposes of the challenge the objects only have position (x, y). With the range specified.

Of course that does not prevent one from building those coordinates into whatever object model you have of the otherway around extending that base property.

@ejolson

Great, you already have a solution in mind. Let's stick to 2D for us simple minded folk.

Actually there are a couple of changes I'd like to make to the problem spec.

1) Previously we have had issues actually timing things. Like the "warm up time" of interpreter engines and cache filling. Like the time taken to print the results. To that end we should specify it so that the actual algorithm time is measured. After all in a real, long running, applications it's the algorithms run time that matters not all that other junk. We should arrange the run time is longer, like requiring to find the distances of some number of different targets, such that run time is many seconds or a minute say.

2) I'd like to specify that always producing a complete list of all possible points within the range is not required.

To that second point, imagine you are somewhere in a big city and you want a list of restaurants within 2km of your location. That might be a big list. You might be happy with just being told of 5 of them that are nearby. Not even the 5 nearest ones necessarily, for example 5 that are all off in the same direction down a particular street would do for you to check out. But not outside the range.

I'm not sure how to express that idea properly yet or how we would evaluate and compare the results of different approaches, which would likely be different.

I think you are saying something like we do for out air quality project (PM2.5 sensors).

We have result in an SQL (or similar) database, they all have their GNSS location.
I have my Pi with GNSS (or phone, or Alexa) and ask it to give me the current reading.
I can grab the nearest by actual distance from my current point.
I can also grab the nearest 5 and take an average, or remove any that haven't sent data back for a while.

I first did this in Python3, cheap and easy as we don't have that many points to loop through at the moment but moved it to the server in some language (php/SQL?).

Scale it up to 1000s of point and the Python3 would be slow probably.
To be honest I would bother doing it in my own code and I would look at SciPi (and maybe KDtree ?) and leave it to people who are good at this
https://docs.scipy.org/doc/scipy/reference/

[davidcoton]

More information is here.

https://en.m.wikipedia.org/wiki/Spaghetti_plot

Thanks for the link. Enjoy your raspberry Pi(e).

Strangely enough the algorithm I have in mind to tackle this problem involves spaghetti.

The Italian Job?

[Heater]

davidcoton,

The Italian Job?

This recipe was inspired by an Italian but this particular spaghetti strand was cooked by a German.

[Heater]

bensimmo,

I think you are saying something like we do for out air quality project (PM2.5 sensors).

We have result in an SQL (or similar) database, they all have their GNSS location.
I have my Pi with GNSS (or phone, or Alexa) and ask it to give me the current reading.
I can grab the nearest by actual distance from my current point.
I can also grab the nearest 5 and take an average, or remove any that haven't sent data back for a while.

I first did this in Python3, cheap and easy as we don't have that many points to loop through at the moment but moved it to the server in some language (php/SQL?).

Scale it up to 1000s of point and the Python3 would be slow probably.

Yes, the challenge is kind of inspired by that geographical problem.

Rather than loop through thousands of points stored in a database why not let the database do the work:

Code: Select all

SELECT * FROM Places WHERE
    (Lat => 1.2393 AND Lat <= 1.5532) AND (Lon >= -1.8184 AND Lon <= 0.4221)
AND
    acos(sin(1.3963) * sin(Lat) + cos(1.3963) * cos(Lat) * cos(Lon - (-0.6981))) <= 0.1570;

See nice article about this here: "Finding Points Within a Distance of a Latitude/Longitude Using Bounding Coordinates" - http://janmatuschek.de/LatitudeLongitud ... SQLQueries

How fast it works will depend on your database of course. But I suspect it would be hugely faster than any home made looping through thousands of points in Python.

That sounds likes an interesting project you have there.

[bensimmo]

bensimmo,

I think you are saying something like we do for out air quality project (PM2.5 sensors).

We have result in an SQL (or similar) database, they all have their GNSS location.
I have my Pi with GNSS (or phone, or Alexa) and ask it to give me the current reading.
I can grab the nearest by actual distance from my current point.
I can also grab the nearest 5 and take an average, or remove any that haven't sent data back for a while.

I first did this in Python3, cheap and easy as we don't have that many points to loop through at the moment but moved it to the server in some language (php/SQL?).

Scale it up to 1000s of point and the Python3 would be slow probably.

Yes, the challenge is kind of inspired by that geographical problem.

Rather than loop through thousands of points stored in a database why not let the database do the work:

Code: Select all

SELECT * FROM Places WHERE
    (Lat => 1.2393 AND Lat <= 1.5532) AND (Lon >= -1.8184 AND Lon <= 0.4221)
AND
    acos(sin(1.3963) * sin(Lat) + cos(1.3963) * cos(Lat) * cos(Lon - (-0.6981))) <= 0.1570;

See nice article about this here: "Finding Points Within a Distance of a Latitude/Longitude Using Bounding Coordinates" - http://janmatuschek.de/LatitudeLongitud ... SQLQueries

How fast it works will depend on your database of course. But I suspect it would be hugely faster than any home made looping through thousands of points in Python.

That sounds likes an interesting project you have there.

It wasn't slow as such, but having to transfer every point from the database to my device was not the best plan.
It is all done on the SQL database now with a call. Not sure which we are suing, that not my side of things, I think it is Maria but may be switched to something else.

I believe we now use the Spacial funtion provided ('Point' rings a bell and spherical distance, it was great working it out from my suggestion.

Interestingly, or not, it was so I could take a PiZero/Pimoroni ScrollBot with a GPS attached and show peoppe their local PM2.5 count from our sensors (or DEFRA or the Councils etc) to enthuse anyone that would listen School being the initial target.

At least that was the plan, just need to get time and a bit of money to persue partII. Getting LoRaWAN/TTN setup around the city has taken over for a bit. Not to bad for what is basically a community interest group with no money
The main part isn't to bleat on about particle counts and climate health though as that far more complicated that putting a load of sensors up and telling them a number.
But to make them think, what could they do with a free to use slow but Long Distance 'WiFi' network or just fast but short range WiFi and a few relatively cheap ESP/Ardunio/Pi devices and sensors spread around.
The only stipulation so far is all data will be stored as open-data for anyone to use, analyse and make use of.

Hopefully all the UK's money pumped into Primary and Secondard school computing will give them the skills and encouragment to try thing for themselves.

[John_Spikowski]

THIS might be of interest to commercial app developers.

[Heater]

In my continuing search for heaven, https://www.raspberrypi.org/forums/view ... 0#p1556604, I have been assembling my collection of interesting coordinates in space.

I downloaded the US Navy Observatory CCD Astrograph Catalog (UCAC): https://www.usno.navy.mil/USNO/astromet ... -prod/ucac from the mirror given here: https://www.ap-i.net/skychart/pt/news/ucac4_catalog

There is some software in that archive to read, decode, filter that data set in Fortran, C and Python. After installing the GNU Fortran compiler I new have 113 million star records in CSV format. Nearly 20GB of it!

But damn it, now I have the irresistible urge to start plotting them some how....

[ejolson]

now I have the irresistible urge to start plotting them some how....

What? I thought you were going to search for 5 of the nearest 10 stars to any point in outer space.

[Heater]

Err, yeah, that.

I originally envisioned this little challenge as just an abstract problem. A flat space of some number of grid points, a bunch of random points occupied, find the nearest occupied points to some other randomly chosen point.

The star positions only came in by way of spicing things up with some real data that looks kind of random. Then I got side tracked looking at what these astronomers have been up to in recent years...

I'm slowly getting around to pulling out a subset of that huge star position data set as our test data. It's amazing how crowded the galaxy is!

I'm going to be brutal and just map those declinations and ascensions to points on a flat grid.

[jcyr]

Ooh, 113 million coordinates to project onto a flat screen. I've got just the thing to do that with!

[Heater]

Curious.

[Gavinmc42]

Ooh, 113 million coordinates to project onto a flat screen. I've got just the thing to do that with!

Space junk monitoring or asteroid mining?

I was impressed when Grass GIS and QGIS just worked on a Pi4.
Trying to learn OpenStreet map stuff so PM2.5 could be plotted.
I have some PM2.5 sensors and Lora feathers I want to connect to the Council LoRa network
Pi's at schools would be a good place to put them.

But Open Data and Open Gov and Open Education?
Might be easier to make my own LoRa network

[Heater]

Here in Helsinki they are really into the "smart city" idea and encourage citizen participation. https://helsinkismart.fi/citizen-city/

One neat project was exactly getting air quality sensors spread around the place a few years back. They tackled it in an interesting way. Basically Helsinki resident's were offered an air sensor IoT kit for a few euros, but they had to build it and get it working themselves. To that end a kind of maker space was provided where you could go and build it and learn from the guys there about the basic electronics, soldering and assembly required. From talking to the guys who ran it it seems it was very popular.

They just opened a huge and expensive new library building in the middle of the city. To my mind it's a bit short of actual books, but it has a huge area where you can go and use 3d printers, laser cutters, all kind of things. Including a proper electronics lab where you can get on with soldering and make use of scopes and such. Amazing.
https://helsinkismart.fi/portfolio-item ... ioCats=248

[John_Spikowski]

Ooh, 113 million coordinates to project onto a flat screen. I've got just the thing to do that with!

It looks like we are back on track with absurd.

[Heater]

John_Spikowski,

Could you elaborate on your definition of "absurd"?

As far as I can tell it includes anything that is:

• Just for fun.
• Done out of simple curiosity.
• Educational
• Of no immediate practical use, despite it's potential for such.

In this case my simple challenge idea has inadvertently led me to researching star catalogs and finding out what astronomers get up to. All very interesting. You may think it's absurd but you would have to argue that case with the US Navy Observatory who spent ten years on the observations that went into the data set I'm looking at.

Perhaps we should just fire all those absurd, time and money wasting astronomers, cosmologists, physicists, mathematicians and other useless scientific so called "researchers".

The common theme I notice is that "absurd" seems to be anything ScriptBasic can't do very well

[John_Spikowski]

The challenge started off finding objects in a grid and evolved into mapping the universe. My initial comment was lets focus on challenges that result in code that can be reused.

The ScriptBasic project stands on its own and doesn't require my contributions to be viable.

[Heater]

Ah, right. Do not worry John.

Through the awesome powers of abstraction I imagine the basic implementation of finding objects on a grid will be there. Ready to be used for whatever other purpose.

The only little twist here is where I'm proposing to get those object positions from. Might as use those stars hanging around in the galaxy and not doing much except looking pretty.

All that is needed is to map a subset of all those ascensions and declinations, in micro arc seconds, into integer grid points. On whatever size grid we end up with.

Perhaps the most expensive source of random numbers I have ever used!

[jcyr]

The challenge started off finding objects in a grid and evolved into mapping the universe. My initial comment was lets focus on challenges that result in code that can be reused.

The challenge hasn't changed. Given a set of 10^8 star's right ascension (RA) and south pole distance (SPD), easily mapped onto a 2D space, find the N closest points to an arbitrary point (RA,SPD).

[Heater]

The story so far...

The fourth U.S. Naval Observatory CCD Astrograph Catalog (UCAC4) contains 900 files of star data in binary. Totaling 8.3GB.

There is a python script to fix up a few errors in there. Done that.

There is a dump program included, in FORTRAN, that reads the binary files and spits out ASCII files, one star record per line. I managed to build that with GNU Fortran so now I have 30GB of CSV files!

These files are named z001.asc to z900.asc. That's 900 declination zones of 0.2 degree width, with zone number 1 starting at the South Pole and increasing toward north.

I'm tempted to stuff them all into my Cockroach SQL database to see how well it copes. From there I could select a nice busy patch of sky for our challenge data. Sort of equatorialish and looking to the middle of the galaxy.

[John_Spikowski]

looking to the middle of the galaxy.

You will find that there is no end to the universe or was there a beginning. Get your arms around that.

[jcyr]

The story so far...

The fourth U.S. Naval Observatory CCD Astrograph Catalog (UCAC4) contains 900 files of star data in binary. Totaling 8.3GB.

There is a python script to fix up a few errors in there. Done that.

There is a dump program included, in FORTRAN, that reads the binary files and spits out ASCII files, one star record per line. I managed to build that with GNU Fortran so now I have 30GB of CSV files!

These files are named z001.asc to z900.asc. That's 900 declination zones of 0.2 degree width, with zone number 1 starting at the South Pole and increasing toward north.

I'm tempted to stuff them all into my Cockroach SQL database to see how well it copes. From there I could select a nice busy patch of sky for our challenge data. Sort of equatorialish and looking to the middle of the galaxy.

At least 10^6 points, I hope. It's gotta be challenging after all!

[Heater]

jcyr,

At least 10^6 points, I hope. It's gotta be challenging after all!

I have been thinking about that...

The data set has to be big enough to be an interesting challenge.

On the other hand I thought it would be nice to be small enough to plot in an image to see, a grid of 2048 by 2048 for example. 4 million points about.

But heck the UCAC4 star positions are only 32 bit integers in each direction. There is only about 113 million points. Why not go for the whole hog? A grid of 4 billion by 4 billion!

That should sort the men from the boys.