Showing posts with label #DataScience. Show all posts
Showing posts with label #DataScience. Show all posts

10.27.2016

Hungarians and the dream of Austrian life standard

Hungary has a stormy historical relationship with Austria. The two countries were part of the same empire between the mid-16th century and 1920, though Hungarians revolted several times against the formation. Ever-since Hungarian people have a sight on the western brother-in-laws (as we call Austrian people) for reference on how well state affairs could go also in Hungary, if only we would be as smart, lucky, greedy, hard working, easy going, etc, as those living beyond the Leitha.
Especially when we talk about the life standard. In politics, the Austrian standard of living is the de facto standard, to which we always want to close the Hungarian one, and about which politicians state year by year that it is further ahead of us then ever.
Since I have a nice historical dataset about GDP-per-capita values from a nice range of countries including all European ones, I thought I will do some fact finding exercise to figure if those statements are true, and if so, why? (The original data is available form the site of the Maddison Project, and it is called: Historical Statistics of the World Economy:  1-2008 AD.)
Let's begin with my disclaimers: First, I am not an economist, just a guy who likes to play with data. It can easily happen that all my insights are wrong, or mis-interpreted. Second, I am aware that  GDP-per-capita is not a direct nor optimal measure of life standard, but 1) this is the best data I can access, 2) it is not that bad since so many economists are still using it for comparing living standards in different countries.
Now that these are off the table, lets have a look on a general picture how GDP-per-capita changed after the second world-war.

I use a few countries as references here in addition to Austria (green, as their lawn is always green). UK (red) as a general reference representing general trends in the world economy. Germany (blue) as a representative of WWII losers in Europe, Finland (orange) because during the past century, it had a very similar starting point and background as Hungary. I know, at this point people start to scream that how-so? Well, in the beginning of the 20th century Finland was in a very similar status just like Hungary. Not much of natural resources, some industry but nothing overwhelming, part of a huge empire which is hated by much of the population in the country. They enjoyed a wide autonomy, but gained independence during the endgame of WWI. Some red political adventures, but generally a capitalist backwater country with strong economical dependence on western allies. Parallel histories until the end of WWII. Besides, as I am a Hungarian living in Finland, the orange line is absolutely a must have in this mind-game.
So, let's have a look on the plot. The absolute GDP-per-capita values were more-or-less steadily growing in all countries. The exception is the period around the '90s representing the collapse of the Soviet economy. Hungary (black line) was inside the block, and Finland was a heavy exporter to the Soviet-union, so both of these countries experienced an economical shock when the rouble of Big Brother went to south. During this period, the growth in Hungary was much less than in those other, more lucky, countries, including Austria. Well, thank you socialist plan-economy.
So how faster Austria developed than Hungary?

Much. Just after the war, Hungary was able to produce at 90% level of Austria. Thanks to the hard-core communism between 1945 and 1956, this sharply dropped to 55% in the course of 11 years. After the '56 uprising, the new socialist regime realized that Hungarians will be content of the new world order if only they will be given perks. After exterminating all internal enemies, Kádár started their 'New Economical Mechanism', which was not successful stabilizing the economic backsliding compared to Austria. They started to finance workplaces, production and in a small part social expenditures from external loans in 1973, which finally stabilized the life standard at around 40-45% of Austrian people. (I have to add that it was somewhat expensive. There are calculations stating around 40-70% of today's public debt is the interests of external state loans obtained between 1973 and 1990.) Again, the collapse of the Soviet bloc and their economy shrunk further 10% and by 1992, Hungary was able to produce at 31% of Austria. I remember this time very well. People were driving to Vienna to but such goods as washing-machines or computer components. By this time where are the years after the WWII, when Austirans and Hungarians were really brother-in-laws in our misery of losing the war, living at almost the same levels?
Ever since the beginning of '90s, Hungarians are working day-and-night, and we can be proud (?) that we are closing to 40% of Austrian life standard.
Conclusion? Not much. Maybe I recommend these plots to those who still believe that socialism and socialistic economy was so good for those poor people. After 50 years such an adventure in Hungary, we can see these data and think. Without this experiment, where could we be? Where Austria or where Finland is?
(Code for generating the plots and some more is posted to Pastebin.)

4.28.2016

Mapping the Moon

I am fascinated by space exploration. The other day, inspired by the recent anniversary of the Apollo 16 mission, I was watching a short documentary about Moon exploration.

The documentary had an interesting note about the Lunar ReconnaissanceOrbiter, a space probe orbiting the Moon since 2009, and performing different measurement to prepare for future Moon expeditions. One of the six on-board instruments of the orbiter is called Lunar Orbiter Laser Altimeter, or LOLA. It measures the precise height of the lunar surface under the probe by centimeter accuracy, and it is used to build a topographic map of the Moon.

That has turned on my brain and I was carried away. What could be more cool than play around with the real 3D topographical map of the Moon! Then I figured, that NASA is a public institute, they publish practically everything they measure or record during their missions. So maybe, if I dig deep enough, I can find LOLA data out in the Internet wilderness.

And here we go! The orbiter itself has a great and informative website. They provide links not only to interesting images and multimedia, but also to actual real measurements and datasets via NASA's PlanetaryData System archives (sounds mind blowing!).
The actual measured raw data can be downloaded from the Lunar OrbitalData Explorer. For example, a single set of measurements can be imported into R like this:

raw.data<-read .table="" a="" href="http://pds-geosciences.wustl.edu/lro/lro-l-lola-3-rdr-v1/lrolol_1xxx/data/lola_radr/laser1/lro_no_01/lolaradr_092582345.tab">http://pds-geosciences.wustl.edu/lro/lro-l-lola-3-rdr-v1/lrolol_1xxx/data/lola_radr/laser1/lro_no_01/lolaradr_092582345.tab
")
head(raw.data)

This is a table file which can be nicely imported into an R data frame. Its structure is explained here.
The main point is that the first two columns contain longitude/latitude coordinates, while column 10 is the distance of the surface measured from the orbiter. It is a simplification, because the measuring laser is not always pointing “down”, but for our purposes that will do. Because after investigating a few of the files (and there are many of them), it is clear that they contain data segmented by collection time, meaning that in a file there are reads from a segment of a single orbit. Since I am more interested in all the measurements from a given lunar location, I was digging further on the website.

Fortunately, they offer also a tool to download all the elevation reads from a location, and it can be selected using simply the coordinates of the desired region: http://ode.rsl.wustl.edu/moon/indextools.aspx

So I have fired up Google Moon and picked up one of my favorite regions: the Posidonius Crater. There are multiple crater rings in this complex, and I have randomly picked up crater “J” and its surrounding and downloaded data between 30 to 32 E; and 33 to 35 N coordinates.

The data is kindly generated as a CSV file, which is very easy to handle with R:

dat.file<- font="" opofull_csv_table.csv="">
my.data<-read .csv="" dat.file="" font="">
head(my.data)
summary(my.data$topography)



The important information is stured in the first three columns of the created data frame: the longitudes, the latitudes, and the “topography”, which can be simply interpreted as elevation compared to a hypothetical “sea level”.
After a brief investigation of this topography data, it can be seen that it is actually “below” that level, so all the elevations are negative:

summary(my.data$topography)

It is easy to use the plot3d() function to draw a 3D scatterplot of the measured data. I wanted to use a color code which would resemble map colors, with high values as brown and low areas as green. The terrain.colors() function offers a handy solution for that:

zlim<-range font="" my.data="" topography="">
zlen<-zlim font="" zlim="">
colorlut<-terrain .colors="" font="" zlen="">
col<-colorlut font="" my.data="" topography-zlim="">

With the colors prepared, plotting is a matter of providing the proper parameters of plot3d():

plot3d(my.data$Pt_Longitude,
       my.data$Pt_Latitude,
       my.data$topography,
       col=col,
       zlim=c(-4000,11000),
       xlab="Longitude",
       ylab="Latitude",
       zlab="Elevation",
       main="Posidonius J crater on Moon",
       sub="Lunar Reconnaissance Orbiter LOLA data")



This is excellent, the passings of orbiter are clearly visible, and with a bit of turning the plot this and that ways, we can clearly see the shape of the terrain. Still, it is not what I wanted as a nice 3D topography. For that, the measurements taken under the orbital passings should e converted to a regular grid format, where the elevation is recorded in a grid like matrix. It is not a trivial task to convert our existing data to grid format as it involves heavy interpolation. It is a computationally expensive calculation, but nothing that a modern computer could not handle. R offers the akima package, which implements the interpolation of H. Akima (http://www.iue.tuwien.ac.at/phd/rottinger/node60.html ).

library(akima)

int.dat <- font="" interp="" my.data="" t_longitude="">
                  my.data$Pt_Latitude,
                  my.data$topography,
                  xo=seq(30,32,length=100),
                  yo=seq(33,35,length=100))

It takes some 10-15 minutes to run the interpolation, but the results are suitable for visualization with the rgl package. First, I wanted to convert the elevation values so that they scale close to 1:10 with the horizontal dimensions of the plot.

elevation<-int .dat="" font="" z="">
elevation<-elevation 1:10="" distortion="" font="" of="" sizes="" vertical="">

Again, the colors are recalculated so the usual topographical colors are scaled to this data:

zlim<-range elevation="" na.rm="TRUE)</font">
zlen<-zlim font="" zlim="">
colorlut<-terrain .colors="" font="" zlen="">
col<-colorlut elevation-zlim="" font="">

And finally, we are ready to see the surface of the Moon!

library(rgl)
rgl.open()
rgl.surface(100:1,1:100,elevation,color=col,back="lines")



Incredible! A piece of Moon on my computer screen!