Tag Archives: computer programming

those f-ed up spreadsheets

I’ve seen a lot of fucked up spreadsheets in my time. In fact, you could probably say most spreadsheets contain errors, and this article confirms it. Nonetheless, quality control procedures that apply to other types of software (and spreadsheets are a very useful kind of software because they are very powerful and have an easy learning curve, which is also their downfall) can and should be adapted to spreadsheets. One of the most important ones (which really applies to any program or project) is to begin the quality control process in the planning and concept development phase.

Spreadsheet quality assurance: a literature review

Spreadsheets are very common for information processing to support decision making by both professional developers and non-technical end users. Moreover, business intelligence and artificial intelligence are increasingly popular in the industry nowadays, where spreadsheets have been used as, or integrated into, intelligent or expert systems in various application domains. However, it has been repeatedly reported that faults often exist in operational spreadsheets, which could severely compromise the quality of conclusions and decisions based on the spreadsheets. With a view to systematically examining this problem via survey of existing work, we have conducted a comprehensive literature review on the quality issues and related techniques of spreadsheets over a 35.5-year period (from January 1987 to June 2022) for target journals and a 10.5-year period (from January 2012 to June 2022) for target conferences. Among other findings, two major ones are: (a) Spreadsheet quality is best addressed throughout the whole spreadsheet life cycle, rather than just focusing on a few specific stages of the life cycle. (b) Relatively more studies focus on spreadsheet testing and debugging (related to fault detection and removal) when compared with spreadsheet specification, modeling, and design (related to development). As prevention is better than cure, more research should be performed on the early stages of the spreadsheet life cycle. Enlightened by our comprehensive review, we have identified the major research gaps as well as highlighted key research directions for future work in the area.

Make/write/record music with R

Move over Garage Band, there is a new musical sheriff in town. For those of us who like to mess around in R – okay, realistically this gets added to my list of retirement projects a few decades down the road, assuming my body and brain manage to stick around for the next few decades.

gm: Generate Music Easily and Show Them Anywhere

Provides a simple and intuitive high-level language, with which you can create music easily. Takes care of all the dirty technical details in converting your music to musical scores and audio files. Works in ‘R Markdown’ documents <https://rmarkdown.rstudio.com/>, R ‘Jupyter Notebooks’ <https://jupyter.org/>, and ‘RStudio’ <https://www.rstudio.com/>, so you can embed generated music anywhere. Internally, uses ‘MusicXML’ <https://www.musicxml.com/> to represent musical scores, and ‘MuseScore’ <https://musescore.org/> to convert ‘MusicXML’.

CRAN

2018 in Review

Most frightening and/or depressing stories:

  • JANUARY: Cape Town, South Africa looked to be in imminent danger of running out of water. They got lucky, but the question is whether this was a case of serious mismanagement or an early warning sign of water supply risk due to climate change. Probably a case of serious mismanagement of the water supply while ignoring the added risk due to climate change. Longer term, there are serious concerns about snowpack-dependent water supplies serving large urban populations in Asia and western North America.
  • FEBRUARY: Cape Town will probably not be the last major city to run out of water. The other cities at risk mentioned in this article include Sao Paulo, Bangalore, Beijing, Cairo, Jakarta, Moscow, Istanbul, Mexico City, London, Tokyo, and Miami.
  • MARCH: One reason propaganda works is that even knowledgeable people are more likely to believe a statement the more often it is repeated.
  • APRIL: That big California earthquake is still coming.
  • MAY: The idea of a soft landing where absolute dematerialization of the economy reduces our ecological footprint and sidesteps the consequences of climate change through innovation without serious pain may be wishful thinking.
  • JUNE: The Trump administration is proposing to subsidize coal-burning power plants. Meanwhile the long-term economic damage expected from climate change appears to be substantial. For one thing, Hurricanes are slowing down, which  means they can do more damage in any one place. The rate of melting in Antarctic ice sheets is accelerating.
  • JULY: The UN is warning as many as 10 million people in Yemen could face starvation by the end of 2018 due to the military action by Saudi Arabia and the U.S. The U.S. military is involved in combat in at least 8 African countries. And Trump apparently wants to invade Venezuela.
  • AUGUST: Noam Chomsky doesn’t love Trump, but points out that climate change and/or nuclear weapons are still existential threats and that more mainstream leaders and media outlets have failed just as miserably to address them as Trump has. In related news, the climate may be headed for a catastrophic tipping point and while attention is mostly elsewhere, a fundamentalist takeover of Pakistan’s nuclear arsenal is still one of the more serious risks out there.
  • SEPTEMBER: A huge earthquake in the Pacific Northwest could be by far the worst natural disaster ever seen.
  • OCTOBER: The Trump administration has slashed funding to help the U.S. prepare for the next pandemic.
  • NOVEMBER: About half a million people have been killed in Iraq, Afghanistan, and Pakistan since the U.S. invasions starting in 2001. This includes only people killed directly by violence, not disease, hunger, thirst, etc.
  • DECEMBER: Climate change is just bad, and the experts seem to keep revising their estimates from bad to worse. The Fourth National Climate Assessment produced by the U.S. government is not an uplifting publication. In addition to the impacts of droughts, storms, and fires, it casts some doubt on the long-term security of the food supply. An article in Nature was also not uplifting, arguing that climate change is happening faster than expected due to a combination of manmade and natural trends.

Climate change, nuclear weapons, and pandemics. If I go back and look at last year’s post, this list of existential threats is going to be pretty much the same. Add to this the depressing grind of permanent war which magnifies these risks and diverts resources that could be used to deal with them. True, we could say that we got through 2018 without a nuclear detonation, pandemic, or ecological collapse, and under the circumstances we should sit back, count our blessings, and wait for better leadership. And while our leadership is particularly inept at the moment, I think Noam Chomsky has a point that political administration after political administration has failed to solve these problems and this seems unlikely to improve. The earthquake risk is particularly troublesome. Think about the shock we felt over the inept response to Katrina, and now think about how essentially the same thing happened in Puerto Rico, we are not really dealing with it in an acceptable way, and the public and news media have essentially just shrugged it off and moved on. If the hurricanes, floods, fires and droughts just keep hitting harder and more often, and we don’t fully respond to one before the next hits, it could mean a slow downward spiral. And if that means we gradually lose our ability to bounce back fully from small and medium size disasters, a truly huge disaster like an epic earthquake on the west coast might be the one that pushes our society to a breaking point.

Most hopeful stories:

I believe our children are our future…ya ya blahda blahda. It’s a huge cliche, and yet to be hopeful about our world I have to have some hope that future generations can be better system thinkers and problem solvers and ethical actors than recent generations have been. Because despite identifying problems and even potential solutions we are consistently failing to make choices as a society that could divert us from the current failure path. And so I highlighted a few stories above about ideas for better preparing future generations, ranging from traditional school subjects like reading and music, to more innovative ones like meditation and general system theory, and just maybe we should be open to the idea that the right amount of the right drugs can help.

Fossil fuels just might be on their way out, as alternatives start to become economical and public outrage slowly, almost imperceptibly continues to build.

There is real progress in the fight against disease, which alleviates enormous quantities of human suffering. I mention AIDS, diabetes, and Alzheimer’s disease above. We can be happy about that, of course. There are ideas about how to grow more food, which is going to be necessary to avoid enormous quantities of human suffering. Lest anyone think otherwise, my position is that we desperately need to reduce our ecological footprint, but human life is precious and nobody deserves to suffer illness or hunger.

Good street design that lets people get around using mostly their own muscle power. It might not be sexy, but it is one of the keys to physical and mental health, clean air and water, biodiversity, social and economic vibrancy in our cities. Come to think of it, I take that back, it can be sexy if done well.

Good street design and general systems theory – proof that solutions exist and we just don’t recognize or make use of them. Here’s where I want to insert a positive sentence about how 2019 is the year this all changes for the better. Well, sorry, you’ll have to find someone less cynical than me, and/or with much better powers of communication and persuasion than me to get the ball rolling. On the off chance I have persuaded you, and you have communication and/or persuasion super powers, let me know.

Most interesting stories, that were not particularly frightening or hopeful, or perhaps were a mixture of both:

Whatever else happens, technology and accumulation of human knowledge in general march on, of course. Computer, robotics, and surveillence technology march on. The human move into space is much slower and painful than many would have predicted half a century ago, and yet it is proceeding.

I’ll never drop the waterless sanitation thing, no matter how much others make fun of me. It’s going to happen, eventually. I don’t know whether we will colonize Mars or stop defecating in our water supply first, but both will happen.

The gene drive thing is really wild the more I think about it. This means we now have the ability to identify a species or group of species we don’t want to exist, then cause it not to exist in relatively short order. This seems like it could be terrifying in the wrong hands, doesn’t it? I’m not even sure I buy into the idea that rats and mosquitoes have no positive ecological functions at all. Aren’t there bats and birds that rely on mosquitoes as a food source? Okay, I’m really not sure what redeeming features rats have, although I did read a few years ago that in a serious food crunch farming rats would be a much more efficient way of turning very marginal materials into edible protein than chicken.

The universe in a bottle thing is mind blowing if you spend too much time thinking about it. It could just be bottles all the way down. It’s best not to spend too much time thinking about it.

That’s it, Happy 2019!

Scratch and Raspberry Pi

Here’s an article about how Scratch (a programming language supposedly any child can learn) and Raspberry Pi (a computer that costs less than $50) fit together.

We’re huge fans of Scratch here at the [Raspberry Pi] Foundation. It was designed to teach young people how to program but it’s a great learning tool at any age: you can build your first program in minutes and pick up fundamental concepts very quickly. Whilst having fun. Sneaky!

If you’ve never tried Scratch before then today is the day to boot up your Pi and have a play. If you are Pi-less then you can use it online, but you’ll be missing out on the best bit of all: physical computing with Scratch. It’s probably the easiest way to hook up sensors, LEDs, buttons and motors to the Pi

February 2015 in Review

This blog got 173 hits in February! Pretty cool, considering I really just meant it as a place to collect my own scattered thoughts and refer back to them later. If 173 out of the 6 billion people out there like it, I am flattered. Okay, I understand there may have been a few repeat visitors. Also, judging from the most popular posts, there is one thing I mention occasionally that people really like: robots!

Negative trends and predictions:

  • Fresh Air had an interview with Elizabeth Kolbert, author of The Sixth Extinction. The idea here is that what humans are doing to other species is equivalent in scope to events that have killed off most life on Earth in the past.
  • The drought in the western U.S. continues to grind on.
  • There are some depressing new books out there about all the bad things that could happen to the world, from nuclear terrorism to pandemics. Also a “financial black hole”, a “major breakdown of the Internet”, “the underpopulation bomb”, the “death of death”, and more!
  • Government fragmentation explains at least part of suburban sprawl and urban decline in U.S. states, with Pennsylvania among the worst.

Positive trends and predictions:

  • Libraries are starting to go high-tech using warehouse robot technology.
  • I had a rambling post on technologies to watch: carbon fiber, the internet of things, self-driving cars and trucks, biotechnology for everything from carbon sequestration to cancer treatment to agriculture, and of course more automation, robots, and artificial intelligence. And yes, Clark W. Griswold’s cereal varnish is a real thing!
  • U.S. utility solar capacity is slowly ramping up.
  • A new study suggests a sudden, catastrophic climate tipping point may not be too likely.
  • Robots can independently develop new drugs.
  • According to Google, self-driving taxis are only 2-5 years away.
  • Complex ecosystems can be designed.
  • Compost toilets may save the world…if we can get over the ick factor and the sawdust problem.
  • There are lots of cheap new options for the aspiring high-tech handymen (and women and children) among us. Even better news, we may have reached the point where if you build a robot with your kid in the basement, and he then tells other kids about it, he might not get beat up on the playground.
  • New York City has some good examples of green stormwater infrastructure integrated in sidewalk and street design.

One thing that strikes me is that we keep hearing about biotechnology, but we haven’t seen big, obvious impacts in most of our daily lives yet. I suspect biotechnology is like computers and robots in the 70s, 80s, and 90s – slow but steady progress was being made in the background, the pressure was building, and then the wave suddenly broke onto the commercial and public consciousness. I suspect biotechnology is the next big wave that is going to break.

DIY programming and robotics stuff

Here’s some programming and robotics stuff that could be useful for education and do-it-yourself projects around the house. Oh, the things I could do if I didn’t have to work for a living…

  • The Raspberry Pi is a low cost, credit-card sized computer that plugs into a computer monitor or TV, and uses a standard keyboard and mouse. It is a capable little device that enables people of all ages to explore computing, and to learn how to program in languages like Scratch and Python. It’s capable of doing everything you’d expect a desktop computer to do, from browsing the internet and playing high-definition video, to making spreadsheets, word-processing, and playing games.

  • Arduino is an open-source electronics platform based on easy-to-use hardware and software. It’s intended for anyone making interactive projects.

  • VEX IQ is a robotics platform designed to transform STEM learning for young students and their teachers. Students as young as 8 can jump right in and snap robots together using this intuitive, toolless platform while educators can utilize the free VEX IQ Curriculum to help teach them valuable lessons and skills that are needed in today’s changing world. The VEX IQ Challenge, presented by the Robotics Education & Competition Foundation, gives students affordable access to the inspiration, excitement and learning that comes from participating in a STEM challenge.

R graph catalog

Here’s a nice catalog of graphs made with R, along with source code for each. Some of the images were broken or missing when I tried it, but hopefully they’ll get that fixed. (By they way, this is my personal experience with interactive “Shiny” apps so far – I love the idea and the look, but there always seems to be something wrong that needs to be fixed, and fixing it takes more time and requires more specialized training than just dealing with plain old code. At first, I thought it might be a productivity enhancer, but instead it’s a drag when your job is not to build cool-looking apps, but to produce useful data analysis results in a reasonable amount of time.)