Category Archives: Online Tools / Apps / Data Sources

Make/write/record music with R

Move over Garage Band, there is a new musical sheriff in town. For those of us who like to mess around in R – okay, realistically this gets added to my list of retirement projects a few decades down the road, assuming my body and brain manage to stick around for the next few decades.

gm: Generate Music Easily and Show Them Anywhere

Provides a simple and intuitive high-level language, with which you can create music easily. Takes care of all the dirty technical details in converting your music to musical scores and audio files. Works in ‘R Markdown’ documents <https://rmarkdown.rstudio.com/>, R ‘Jupyter Notebooks’ <https://jupyter.org/>, and ‘RStudio’ <https://www.rstudio.com/>, so you can embed generated music anywhere. Internally, uses ‘MusicXML’ <https://www.musicxml.com/> to represent musical scores, and ‘MuseScore’ <https://musescore.org/> to convert ‘MusicXML’.

CRAN

fun with coronavirus math

Let’s do some coronavirus math! This is a word problem, kids. I’m writing on January 14, 2021, and this post will be horribly outdated, but possibly of historical interest, when you read it.

The total number of cases confirmed to date as of today, in the U.S.: “23.1 million+” (New York Times)

The CDC’s ratio of actual cases to confirmed cases: 7.2 (CDC)

Number of cumulative cases in the U.S. so far: 23.1 million * 7.2 = 166 million (166,320,000)

Population of the United States: 328.2 million (Google)

% of our population that has had the coronavirus = 166 / 328.2 = 51%

% of our population that has been vaccinated: 3.1% (Financial Times)

But all other things being equal (which I am sure they are not), 51% of the people vaccinated will have already had the coronavirus, so the vaccine so far adds 1.6% to 51% of our population. Call it 53% to be generous.

We have heard a variety of estimates on what constitutes herd immunity, but the number 70% seems to be sticking at least in the media (I don’t have a source handy, and need to go do some other things now.) So we might not be that far off. The (painfully) slow but steady vaccine rollout tortoise will eventually get to the finish line, people are continuing to get infected at high rates every day in the meantime, and nobody wants to see another wave from the new variant, but if and when it hits us it might push us over the mark (at a horrific human cost, of course).

One last thought is that at the moment, I suspect we are immunizing people who are more likely to have already had an infection than the population as a whole. We are being told this is the most ethical approach, or the quickest way to lower risk for the population as a whole, or some combination of the two. The ethical statement may be true, although this seems subjective. I thought ethics was not up to ethicists, but rather ethicists were supposed to ascertain what our society as a whole considers ethical, and maybe compare that to other human societies past and present. I haven’t seen public polls of what people think is ethical, although they may exist. I can see a case that the way the vaccine is being rolled out is ethical, but I can also see a case for a random lottery being equally ethical.

Better planning and communication would not just be ethical, they are the common sense need and our government is continuing to fail, fail, fail and people are dying, which is the opposite of ethical governance. To my ears, it is arrogant to hear them lecturing us about ethics.

novel writing software, Scrivener, and R Bookdown

This is a pretty niche post, but a couple years ago I briefly entertained a fantasy of becoming a novelist. After I tried to write a novel, my the wisdom of not quitting my day job was affirmed. But in the course of that, I briefly investigated software that novelists are using to organize their books and research. The most popular is called Scrivener. I was thinking though that it would be neat if someone would customize the R packages Markdown or Bookdown for writing, because then especially if you are writing non-fiction you could have all your data, analysis code, tables, figures, references etc. in one place along with your prose. This post describes a system that allows the author to actually combine the use of Scrivener and R Bookdown.

advertising shits in your head

Some books just have good titles. And this one is called Advertising Shits in Your Head: Strategies for Resistance. From the publisher:

Advertising Shits in Your Head calls adverts what they are—a powerful means of control through manipulation—and highlights how people across the world are fighting back. It diagnoses the problem and offers practical tips for a DIY remedy. Faced with an ad-saturated world, activists are fighting back, equipped with stencils, printers, high-visibility vests, and utility tools. Their aim is to subvert the adverts that control us.

“PM Press is an independent, radical publisher of books and media to educate, entertain, and inspire.”

Really, they had me at the title. But it seems to me that the main way to counter the shit is to teach children from a young age to evaluate the source and quality of information they are taking in, expose themselves to multiple sources of information, think critically, draw their own conclusions, discuss their conclusions with other informed people, and be open minded.

Somewhat related is this new browser plug-in Newsguard, which provides “Trust ratings for all the news sites that account for 95% of engagement” and is “Written by journalists, not secret algorithms.” Sounds okay, although I think I have too many browser plug-ins already.

the Oura

The Oura is a fitness tracker, but unlike others it is a ring you wear on a single finger. See Wired article here and company website here. It seems to focus mostly on measuring your heart beat and temperature – all the time – and coming up with a variety of metrics and feedback for you based on that. The NBA and NASCAR have apparently given them to all their athletes (yes, I maintain that race car drivers are athletes, but we can continue the debate down at the corner bar when and if it reopens.)

I’m curious about fitness trackers, but if I ever invest in something I would like blood pressure and nutrition to somehow be incorporated.

the numbers on maternal mortality

The site Our World in Data likes to plot various statistics against GDP per capita. The U.S. is almost always below the middle, and sometimes towards the bottom, of the industrialized countries. On maternal mortality, the U.S. is orders of magnitude safer than many poor countries, but like other stats there is a noticeable gap with the leaders in our peer group (Finland, Iceland, France, Japan, Switzerland to name a few). Greece and Poland stand out as middle income countries that do much better than us. Interestingly, Belarus also stands out as a high-performance, lower-income country on this metric. The plot is animated, so you can see the U.S. drifting slightly worse over time even as our wealth grows, and even as other countries tend to make progress over time. I think I’ve said it before – we’re coasting on fumes, drifting behind the middle of the pack, and continuing to lose momentum.

modern monetary models

These two posts have a long explanation of monetary theory in general, and modern monetary theory in particular. It’s a little over my head, although I like challenging myself to try to understand it. It is a very abstract system to try to understand. I think that if you can understand monetary policy, you might have a chance to understand what money actually is. And if enough people understand it, they might stop believing in it and the world might end.

Basically, as I understand it, the government prints money (i.e., borrows money from itself) and spends it, usually more than it takes back in taxes, and this creates a surplus in the private sector. It can control the money supply by changing the amount it borrows and spends, or by changing the tax rate. I think what people find scary about “modern monetary theory” is that it suggests money doesn’t have to be taken seriously and any needed amount can just be printed any time. This is why politicians generally have not been given the keys to the printing press.

I have a metaphor in my mind of the real economy as a machine with pistons and gears turning. The fuel for the machine is maybe human effort and ideas (and some actual fuel). But the gears will grind without grease, so you have to lube it up. Not enough and the system will shut down violently. Adding extra will not make the machine turn faster, but it will not do any serious harm other than maybe a gunky mess someone has to clean up. Better to use a little too much lube than not enough. The lube for the economic machine is money.

There were a few other interesting things in the articles that I didn’t know or hadn’t thought about recently. It refers to the late Wynne Godley at Cambridge University as the “father of stock-flow consistent modeling”. I think a few people in a few different disciplines might claim that mantle, but that is the neat thing about system theory, it’s interdisciplinary. There is a certain irony if anyone is into it and doesn’t realize it is interdisciplinary.

There is a free(?) system dynamics system called Minsky, something like Stella but tailored specifically to finance and economics. Matlab also has a sort of stock-flow simulation module call Simulink that I hadn’t heard of. I am still waiting for that system dynamics R package.

The Minsky model also made me think of the late Jay Forrester, who advocated for a long time for stock-flow modeling in economics.

Traditional mainstream academic economics, by trying to be a science, has failed to answer major questions about real- life economic behavior. Economics should become a systems profession, such as management, engineering, and medicine. By closely observing the structures and policies in business and government, simulation models can be constructed to answer questions about business cycles, causes of major depressions, inflation, monetary policy, and the validity of descriptive economic theories. A system dynamics model, as a general theory of economic behavior, now endogenously generates business cycles, Kuznets cycles, the economic long wave, and growth. A model is a theory of the behavior that it generates. The economic model provides the theory, thus far missing from economics, for the Great Depression of the 1930s and how such episodes can recur 50–70 years apart. Simpler system dynamics models can become the vehicle for a relevant and exciting pre-college economics education.

Jay Forrester, quoted in a blog called Viewpoints that Matter (including the blogger’s viewpoint, presumably)

Imagine if the average high school graduate really had an intuitive understanding of how important systems like the economy are structured and why they function the way they do as a result. The world might be a different place.

I’ve been working on one more metaphor for awhile. Maybe the real economy is like a tightrope, and the financial economy is like a safety net stretched above a concrete floor. If we use too much food, water, energy, saturate the atmosphere and ocean with our waste, etc. we will fall off the rope. Hitting the concrete floor would be a failure of the real economy like starvation or freezing to death. The safety net would be a spike in prices for food or energy that slows down the economy short of (most people, right away) actually dying of exposure. The fall would still be very painful and you might break bones or even your neck if you fall just the wrong way. What about something like nuclear proliferation or all the ice in Antarctica suddenly melting? I don’t know, maybe dry rot in the old net that we are failing to do anything about. No price signal is going to save us from those.

Covid Act Now

This is a new site that gives a Covid risk rating based on five indicators: daily new cases, infection rate, test positivity, ICU headroom, and contacts traced. They try to give the same information by county, but they only have the data to provide a couple of the indicators at the county level. I know this data exists for my county, but it must be collected and stored (or not) differently in different counties and different states, so that there is no single organized database of it. This is the kind of thing the federal government could provide leadership on, and once again, they are just failing us in epic fashion.

I’ve added this to my running list of Covid data and simulation sites.

coronavirus trackers and simulations revisited

Update: December 13, 2020 (and from time to time since then, I update links if I notice they are broken)

This post is getting a surprising amount of attention. I don’t normally update posts, but I am updating this one since it is getting attention and the commentary in the original post is significantly outdated. Rest assured, if you are a historian in the far future studying what I was thinking back in June 2020, I have kept the original post at the bottom. I am keeping all the links, just grouping them somewhat and removing (from this section) the outdated commentary. (Thank you, Word Press, for making a simple copy-and-paste operation like this beyond excruciating.)

Data Trackers

  • Johns Hopkins – map, stats, access to data sets
  • New York Times – a national (U.S.) map by county and plots by state (now, with a paywall! as of 7/30/21. Which I will never pay because WEAPONS OF MASS DESTRUCTION!)
  • Financial Times – similar to others, but they look at excess deaths a little differently and have some interesting graphics
  • BBC – similar to NYT, but international
  • CDC – changed this link to their “COVID-19 by County” page on 2/26/22; the updated recommendation is to mask indoors if new cases in your county are 200,000 per 100,000 population per week, AND if the number of people entering the hospital and/or in the hospital is above certain thresholds. It’s a little hard to find the data and figure out yourself, so if you trust the CDC (and who wouldn’t?) you can just type in your county and they will tell you if it is high/medium/low.
  • https://coronavirus.thebaselab.com/ – a variety of maps and plots
  • City Observatory – intermittent data-based articles and maps
  • Our World in Data – excellent interactive country-level data, maps, and plots. A tip – you can also type in “world” or the name of a continent in the country box.
  • https://aatishb.com/covidtrends/ – a very clever animated time series of growth in cases over time, by country
  • Reuters – just more numbers and maps, similar to NYT
  • Covid Act Now – state-level data and communication in a simple, easy to understand index format
  • Harvard Global Health Institute COVID Risk Levels Dashboard – similar to Covid Act Now, but less simple and less easy to understand. Seems to have more ability to drill down into county-level data, although when you do that much of it is blank.
  • Wastewater surveillance from “Biobot Analytics” – added 4/30/22.

Simulations

  • University of Washington IHME – the best place I have found for understandable future projections. At the state level.
  • FiveThirtyEight – compares different models (no longer updating as of 7/30/21)
  • https://covid19risk.biosci.gatech.edu/ – This site calculates the probability that someone in a group of a given size is infected, based on the estimated rate of active cases in a U.S. state.
  • MicroCOVID – a risk calculator based on local data and allowing you to adjust your risk tolerance and try out various scenarios (added 8/8/21), such as “one night stand with a random person” (on the latter, please remember there are other diseases besides just Covid-19, for example antibiotic-resistant syphilis…)
  • Covid-19 Forecast hub – another visualization of various models and ensembles of models

Vaccine Trackers

Local Pennsylvania/Philadelphia Interest

  • The state of Pennsylvania has a useful dashboard which they have now made public (or it was public before and I didn’t notice.) It compares cases, positive tests, and hospital data for the current and last 7-day period, at the county level.
  • Speaking of Philadelphia, a shout out to the Philadelphia Health Department which provides some open downloadable data.

Miscellaneous Stuff

Original Post (June 27, 2020)

I decided to list out and summarize the variety of trackers and simulations I’ve mentioned in previous posts. Like many people (in the U.S. Northeast at least), I was glued to coronavirus info on various screens from roughly mid-March to mid-May, then my attention started to gradually drift to other things as the situation got better. Now, it seems that it has either stabilized at a not-quite-out-of-the-woods level, or is slowly reversing itself as we see other parts of the country start to be affected more seriously (sorry if you are reading this and are being affected, we in the Northeast take no pleasure in your suffering, I promise, although we suggest you turn out any bigoted anti-science politicians in your area who are letting this happen.) Anyway, I find that I am interested in starting to look at trackers and simulations again on a daily basis. These are in the order I discovered them.

  • Johns Hopkins – a neat map early on, although now the entire world has become a blob. Still a good place to stare at data.
  • New York Times – a national (U.S.) map by county and plots by state. seems to load even though I have used all my free articles for the month.
  • BBC – they update continuously but I’m not sure if this link will be to the latest
  • CDC – this is what I would have predicted would be the go-to source of information and expertise if you asked me before all this started…but it’s mediocre at best. Yes, that just about sums it up.
  • https://coronavirus.thebaselab.com/ – a variety of maps and plots to stare at, not my first stop but a little different if I am tired of others
  • University of Washington IHME – still the most informative state-level simulations I have found, accounting for hospital capacity among other things
  • City Observatory – they did an awesome analysis by U.S. metro area, which I have not seen anyone else do (human beings interact with each other socially and economically in cities and their suburbs, which often cut across states, and states often contain metro areas that are not connected much socially or economically. Economists, social scientists and urban planners know this of course, but nobody else studying the epidemic seems to have figured this out. Seriously, other data visualization and simulation sites, you can do this, it’s just a matter of grouping data by counties.) Unfortunately, they quit updating it and have not automated it. I still check every now and then to see if they have picked it up.
  • Our World in Data – pretty much every conceivable way of looking at data by country. I like to look at confirmed deaths per million across countries. By this measure, the starkest contrast is east vs. west. The eastern countries were hit first, hard, and without warning, and their death rates are very, very low. They have a variety of government types, responses, ethnicities and cultures. I just don’t think anybody has come close to explaining it. The U.S. is in the middle of the pack of western countries, which somewhat contradicts conventional wisdom and suggests news organizations are making the obvious error of not normalizing by population.
  • https://aatishb.com/covidtrends/ – an animated time series of new confirmed cases in the past week vs. total confirmed cases, both on a log scale, by country. As I write this, shows the beginning of a concerning uptick for the United States, and Brazil out of control.
  • Reuters – I actually never wrote about this one, but it has a map and some numbers.
  • FiveThirtyEight – they have an aggregation of various simulation models out there. New York and New Jersey look like a stream sprayed horizontally out of a garden hose, while Texas and Florida (today) look more like a fire hose.
  • https://covid19risk.biosci.gatech.edu/ – This site calculates the probability that someone in a group of a given size is infected, based on the estimated rate of active cases in a U.S. state. I assume it’s estimated active cases, anyway, or it wouldn’t make sense. It would be better by metro area (seriously guys, someone just get this done), but still a nice idea. I’m in Philadelphia, but I figure the New Jersey numbers are probably the most applicable.
  • Covid Act Now – provides a composite risk index at the state level, and county when county level data is available in the right format (which is not that often)
  • Harvard Global Health Institute COVID Risk Levels Dashboard – keeps it simple with just data on new cases, but gives you a variety of nice mapping, charting, and tabular formats to slice and dice the data at country, (U.S.) state or county level.
  • The state of Pennsylvania has a useful dashboard which they have now made public (or it was public before and I didn’t notice.) It compares cases, positive tests, and hospital data for the current and last 7-day period, at the county level.
  • Speaking of Philadelphia, a shout out to the Philadelphia Health Department which provides some open downloadable data.
  • I look at the FAO food price index on occasion. It’s falling lately. Sometimes I look at oil and gold prices, and how many Special Drawing Rights can be bought with one U.S. dollar. Oh and, the Rapture Index is at an all time high!

biodiversity, food and agriculture

Morally, biodiversity should matter to us just because it is. Life on Earth is special, and beautiful, and possibly unique in this universe. But it also matters because losing it could be bad for us humans. The more genetically uniform our sources of food are, the more vulnerable and less resilient they are.

The UN Food and Agriculture Organization put a massive report out on this last year. The verdict? Diversity is lower than it should be, it is declining, and some things are being done but not enough things are being done to reverse the decline. Doesn’t that describe most of the thorny problems facing our planet and species at the moment? We better pay attention to food though – cleaning up after storms, fires and floods is one thing; a few million babies and old people out of billions dying prematurely is another thing; but a serious food crisis could be the one that brings our civilization to its knees.