Cloud-init and the curse of the YAML (and a potential solution! or at least idea)

 I recently took up cloud-init professionally as part of exploring modern(?) alternatives to kickstart for the on-premises private cloud. All worked fine, is all viable, after I actually figured out where to put the config entries...

Cloud-init has a couple of config files:

metadata:

#cloud-config
instance-id: test
hostname: test.localhost
network:
  version: 2
  ethernets:
    eth0:
      match:
        name: eth*
etc. etc. etc.

user-data:

users:
  - name: cloud-user
    ssh_authorized_keys:
      - ssh-rsa [our public key] root@our-bastion.localhost
    sudo: ALL=(ALL)     NOPASSWD:ALL
etc. etc. etc.

All looks good, looks reasonable. The challenge in getting this far was that in reading the docs it was not apparent at the time what config entries go in which file. For example, if I put things for the network in the user-data file it was parsed correctly and I could see the YAML was loaded correctly into the various python dictionaries in the code, but then later on it was just ignored. The docs note in vague ways that the metadata is for the environment and the user-data is specifics for this particular system. Which was what got me into trouble initially putting the network information in the user-data, IDK.

Looking at the code, using YAML for configuration and the easy way it's parsed and loaded into a dictionary is very powerful for adding configuration information to your software. The downside is if you name something slightly wrong or if you put something not quite in the right location, things will just be silently ignored. 

There are a couple of schools of thought here, and I've seen both professionally. One is that you should code things explicitly to look for the commands / statements and if there is something not explicitly recognized then you flag as an error. This style of coding is a bit of a challenge to modify, and in one particular case I've needed to extend multiple code locations so that a new argument/statement is a) parsed correctly (in one module), b) consumed correctly (in a different module), and then c) acted upon in a third module. A lot of work for just adding a similar statement, but it does mean if there's something not quite right, we will notice it right away. At the expense of making the code harder to work with and extend.

The second school of thought is that as configuration information is consumed, you only look for and act upon things you know you are looking for and acting on. This code is trivial to extend, you just add something to the "act upon" section and consume whatever it is you're looking for. The downside here is that if you add some configuration that's not quite right, likely it will just be ignored since nothing is looking for it.

This is the approach that cloud-init takes, and for good reason as there are multiple / modularized consumers all looking at the same dictionary and so any extension should be able to just consume whatever configuration it thinks it needs.

So this left me thinking, is there a way that we can have a central YAML file that is loaded into a python dictionary and that some overseer can look at the end and make sure that everything you specified was actually used? I think YES, based on a quick subclass of dict:

class NewDict(dict):

    def __init__(self, *args, **kwargs):

        super(NewDict, self).__init__(*args, **kwargs)

        self._used_flags = {key: False for key in self.keys()}

    def set_used(self, key, used_arg = True):

        # print(f"(Setting used flag for {key} to {used_arg})")

        if key in self:

            self._used_flags[key] = used_arg

   

    def is_used(self, key):

        _used_flags = self._used_flags.get(key,False)

        # print(f"(Getting used flag for {key} - it is {_used_flags})")

        return(_used_flags)

This NewDict provides all the same stuff for dict, but adds a set_used(), and an is_used() method where you can indicate if a dictionary config item has been used/consumed or not.

To use then you would do something like:

import yaml

with open('stuff.yaml', 'r') as file:

    data = yaml.load(file, Loader=yaml.FullLoader)

nd = NewDict

# Copy the yaml into our NewDict

for key in data:

    nd[key] = copy.deepcopy(data[key])

# Indicate some keys are being used.

nd.set_used('Stuff1')

nd.set_used('Stuff3')

# Report on keys not being used

for key in nd:

    if (not nd.is_used(key)):

        print(f"WARNING, {key} is actually unused you know.")

This way various modules can go through and consume (or re-consume!) whatever configuration is needed, and then at the end something could go through the dictionary and note when some key had actually not been used. Bonus points and cloud-init can also name the modules that were called so you get a sense of what modules you're actually using, and what config entries are actually needed.

Time to start blogging again.

 After a brief vacation, time to start brain dumping on this blog again. Who knows maybe I will tweet again too.

Lost In Translation

Uniqlo Jeans Innovation Center notes:

Great, sustainable denim, I'm all in!

Special waterless washing.  Awesome.  (Hopefully they are not washing in gasoline or something like that.)

Hmm, hold on, if this is waterless, why does the water need to be recycled?  There is no water, what's to recycle?

Natural drying, air drying, I get it, nice.  But, why are these being dried if they are not even wet due to being waterless.

I think what might be going on here is it's "water" "less", as in using less water.  Maybe that's OK with water, but can you imagine meatless hamburgers?  There's still meat in them, but less meat so that means more vegan.  Meat-"less".  Or how about homeless?  You're not without a house, you just downsized to a smaller house, less house, less home, homeless.

We're onto you marketing people!

Bash batch. Converting .avi to .mp4 for import to Davinci Resolve

Bash makes things so easy for doing batch operations on a bunch of files.  I have a bunch of .AVI files from my old camera that Davinci Resolve won't handle.  ffmpeg converts well enough to .mp4 for import.  Since I have a bunch of clips, conversion is just a bunch of command lines:

$ for filename in *.AVI; do filepart=${filename%%.AVI}; if [ ! -e ${filepart}.mp4 ] ; then echo ffmpeg -i ${filepart}.avi ${filepart}.mp4; fi;  done | sh -x

 Formatting and readability is a bit fugly on the command line but this does it nicely.  As a bonus it sees if things are converted already and don't do a conversion.

Telephones

Quick article about how the US senate almost banned rotary phones - https://www.senate.gov/artandhistory/history/minute/Senate_Considers_Banning_Dial_Phones.htm

An old boss of mine had an amusing anecdote about somebody years ago saying that with the enormous growth of telephones in the US everybody would be employed as a telephone operator.  The amusing part of my boss' anecdote was that this person was actually right.  Since we type in phone numbers when we want to talk to somebody, we are in effect working as a phone operator.

Of course in good old 2019, we pretty much have contact apps universally available so we aren't likely to have to type in phone numbers any more.  Unless it's from the menu of the new restaurant that opened and we want to get takeout.

Statistics, lies, whatever.

Every now and then I hear the oft repeated $verylarge% of drivers consider themselves to be above average.  Ha ha, stupid drivers, you suck, you are really bad drivers, you just don't know it or worse don't want to admit it.  After all, being bad is one thing, but being unaware that you are bad, that means you are a general moron as well.

But, if we have a sample of drivers, one rates at a 2, three rate at a 10, the average rating is 3.2.  This would mean that the three scoring at 10, or 75% of the drivers are above average.  Waaaaay above average.

What am I missing here?

On Poverty

Interesting post - https://www.irishtimes.com/culture/why-do-so-many-americans-hate-the-welfare-state-1.4057860

US culture based on Protestant ethics.  European roots were later changed by various world wars to be more supportive of welfare whereas US was largely unaffected by those wars so the ethics lived on.

Professor Elizabeth Anderson notes:

“You know,” she says, “America was so dominantly Protestant for such a long time. We have a substantial number of Catholics but the culture was really shaped by Protestants – in term of their total cultural domination of the United States at its founding, and really continuing.”

One aspect of this, she highlights, is the adoption of a Protestant work ethic as a core value in society. This has a positive side – in honouring human labour – but it also has a negative side.

“There is a profound suspicion of anyone who is poor, and a consequent raising to the highest priority imposing incredibly humiliating, harsh conditions on access to welfare benefits on the assumption you’re some kind of grifter, or you’re trying to cheat the system.

“There is no appreciation for the existence of structural poverty, poverty that is not the fault of your own but because the economy maybe is in recession or, in a notorious Irish case, the potato crop fails.”

Energy

Energy, here on earth, is apparently finite.  We power things with oil and natural gas which is a finite resource held within the bounds of the sphere that we live on.  In the world of the future though, we anticipate getting this energy from the sun, a large glowing sphere nearby.  You could imagine though that solar energy is also finite in the sense that the earth has a finite surface area with which to capture the solar energy.  Could it be in the future that our resource demands are so great that we would admonish people who were wasting solar energy?  Why isn't that square meter being used to capture solar energy or being used to power the photosynthesis for plant xyz?  What a waste, such a shame!

Say what?

Great article from Paul Graham - http://www.paulgraham.com/say.html.  It's from 2004.  It's timeless.  The man is a genius.  Or maybe he is totally wrong and I just agree with him.  Read the article.

Edit:  Interesting somewhat related article I was directed to.  https://slatestarcodex.com/2017/06/21/against-murderism/

Maintenance?

Now this is an interesting read.  http://themaintainers.org/blog/2019/7/30/why-do-people-neglect-maintenance.  I often see a sort of tension between maintaining something and ignoring it and replacing it after a period of time.  It's almost as if we think that one strategy is universally better than the other.  Really we should be thinking about characteristics that make it a better choice in some cases.

Smart coral polyps?

From https://phys.org/news/2019-10-dead-corals-regrow-fatal.html

But the researchers found that in 38 percent of the impacted colonies, the polyps had devised a survival strategy: shrinking their dimensions, partly abandoning their original skeleton, and gradually, over a period of several years, growing back and starting a new skeleton.

Should I actually read that as the polyps that did not shrink died off whereas those that were able to shrink survived and were  able to survive and grow back over a period of years?  Some polyps shrink due to the problems, some just die off?  Guess which ones we see around after a longer period of time?

This is just like the cats that died when exposed to sunlight.  Those all died off and were not able to reproduce and we are not aware of them at all since all we have left are cats that are OK with sunlight.

Never good at lisp

Back in the day when I was learning lisp, I would have a hard time understanding constructs that were just too nested.  I would see things like:

something(somethingelse(trythis(internalpart(updateothers(runstuff)))))

I can see a similar thing happening sometimes with Python where it's easy to keep wrapping object calls together and you end up with hard to decipher code:

something(parameter).object(otherthing(innerpart).state(optionalrewind))

There are some people who talk in similar ways at work and use parenthetical sub clauses before getting to the actual point and I have a hard time following what the actual point is:

We need to figure out, and not necessarily now, but at some point in the near future perhaps when we can get a consensus or when the time is right but not just because we want to be opportunistic agnostic, what, or perhaps even first why or maybe the rationale behind this and related activities that you and according to at least a couple of other people although this is just third party because I haven't heard directly from them but I have been able to glean by reading between the lines of interactions, our next steps should be.

Parsing that out, the real meat of the sentence is (in bold):

We need to figure out, and not necessarily now, but at some point in the near future perhaps when we can get a consensus or when the time is right but not just because we want to be opportunistic agnostic, what, or perhaps even first why or maybe the rationale behind this and related activities that you and according to at least a couple of other people although this is just third party because I haven't heard directly from them but I have been able to glean by reading between the lines of interactions, our next steps should be.

I think as a consumer or target of the conversation I would like to hear the bold part first and can then get the rest of the sentence after.  Or perhaps the rest of the sentence is background and I should get that first.  But I have a heck of a time trying to keep the idea in my head when it's at the beginning, middle, and end, of a bunch of parenthetical remarks.

My point of this whole post (irony, I'm only getting to it now), is that I have a heck of a time reading and understanding news headlines that seem to string together everything into one sentence

[something...] according to police reports via court order leaked by the assistant to the then deputy director who had been allegedly deposed by the now acting associate vice president under executive approval as reported to news personnel by an anonymous tip said the ex chief after hearing reports of this on Thursday from his staff.

What is there some sort of rationing on periods and capital letters that we could not break this up in to digestible sentences?

Well done!

I'm starting to look a bit more into neural networks.  More to come on that later, but in the deeper parts there are matrices and various calculations.  I noticed that the creators of the numpy package did a really great job in making sure that things are displayed in a nice way that makes it easier to see what is going off.  For example, if we create a matrix with:

(python) x=np.array([[1,2,3],[4,5,6],[7,8,9],[10,11,12],[13,14,15]])

when you go to print it out you see:

(python) x
array([[ 1,  2,  3],
       [ 4,  5,  6],
       [ 7,  8,  9],
       [10, 11, 12],
       [13, 14, 15]])

What is really great here is that as the numbers increase in length, the output is shifted so all the columns line up.  Yes, this isn't rocket science genius, but it goes a long way to making things readable and understandable.  In the "computer" world often we just print things out and leave it up to the person to mentally align and figure things out.  I find that people who often do that sort of messy output are people I have to remind that the computer actually works for us, not the other way around.  If we take 10 minutes to make the computer do a bit more work (ex. line up the columns) then it has made things (in this case comprehension) easier for us.  Let the tool do the work for you!

Jumping to conclusions, part 1

Just started reading American Overdose, the Opioid Tragedy in Three Acts.  I'm on the first page, Introduction: An Epidemic Foretold.  It starts out:

Karen Jennings had once been a manager at McDonalds.  Walking home a little the worse for drink, she fell into a creek and broke her back.  The doctor prescribed painkillers to ease her through the pain of recovery.

and then goes on to talk about the spiral of addiction.

So, leaping to conclusions here, it's almost like that second sentence, "the little worse for drink", which is perhaps just glossed over as accidental, could actual be contributing to the condition, namely that Ms. Jennings might have other issues contributing to addiction.

We will see how the book progresses, but I've always wondered why some individuals can apparently safely take opioids and not spiral into addiction.  What percentage of the patients is that, do 50% not become addicted?  75%, 90%?

Another article I have read in the past was related to a study of rats where in a controlled environment rats could self medicate by pressing a button to receive something (cocaine, opioids?) and they rapidly became addicted.  However, the article continued, the rats were in a cage with nothing but the button.  Other rats in a similar experiment were in a cage with rich rat entertainment, wheels, things to play with, etc.  They found those rats occasionally pressed the button to medicate, but not to the level of addiction that the isolated rats did.  The premise of the article was that addiction isn't fully the chemical itself but also the environment.  They drew parallels with the Vietnam war where there was a lot of drug use when in Vietnam due to needing to mentally escape the horrific situation, however when those soldiers returned back to the US, the drug use mostly dried up since they were back in a normal environment.

In Ms. Jenning's case when she was walking home "the little worse for drink" (kudos for not driving BTW) I have to wonder if she was also not in a bad place to start with that also "fed" into the addiction.

Anyway, good read so far, on page 1.  We will see where the book goes.

[Edit *2019-05-14*]
Became curious about why some people become addicted to alcohol and not others.  Google was challenging because a lot of the answers were about dangers of addiction and how to detect, why it is bad, etc.  But I was looking for scientific reasons...

Found this - https://www.healthline.com/health-news/lateral-habenula-responsible-for-alcholism-040814#3

which incidentally had this gem:

“Whether one is going to use drugs or not is a choice based on what the alternative reinforcers are,” he said. “If you have nothing else to do and life is not very pleasant, if you are a social creature—for example, a rat, and you are solo-housed in a boring environment as rats are generally housed for experiments, in that setting it is very easy to get animals to self-administer drugs. If one has them in an enriched environment in which they can socialize, have sex, groom, all those things that rats like to do, then it’s much more difficult to get them to self-administer drugs. It’s much more difficult to induce something that looks like an animal model of addiction.”It’s a fair bet that if humans were forced to live in the same conditions as the rats in Taha’s experiment—alone, in a tiny cage, with nothing to do—and given unlimited access to drugs, they might pass the time by using drugs, too.

They note that genetics may be involved namely that the inability of the lateral habenula region to respond to negative outcomes (painful stimulus for example) prevents us from avoiding those actions in the future.  In rat simulations they artificially damaged the lateral habenula in some rats and noticed that they did not avoid actions which later had negative outcomes.

Alcohol use is interesting as 'small' use has small euphoric effects and a small (non-existent?) negative effect, so continued/occasional use (which we note as not being an alcoholic) is not an issue.  If occasional 'large' use has a large negative effect but this negative effect does not cause us to avoid these large use activities in the future then this pattern could be repeated (also necessitating larger use due to tolerances increasing) then this is what we are calling alcoholism.

It's quite interesting that even in this "modern age"(!!!) we cannot have these conversations without dragging in stigma or perceptions of mental weakness.  "Why isn't he trying harder, he's just giving up too easy, he is too weak" type observations.

Go and read the healthline article, it's really good.

On the American Overdose book thread, unfortunately it's just a continuation of stories of addiction, how lives were ruined, how bad corporations took advantage of the situation to make lots of money.  There are occasional glimpses of questions as to why only certain regions were being hit by this (which may play into the conditions paragraph as noted by the Healthline article).  But otherwise it's just a "this is terrible and unacceptable" type book.  As with people with similar attitudes (serial complainers) I often find this tiresome.  Slightly less worse are people who try to do something about it but unfortunately have not taken the time to really understand the issue and so solutions are just random and amount almost to a cargo cult type response.

Disk Performance

Recently got a Synology DS1819+ and did some performance benchmarks.

Looked around at a bunch of perf benchmark tools and settled on CrystalDiskMark 6.0.1 (although I downloaded and installed from the Microsoft Store - quick/easy install (free)).

Ran on the old Synology DS215J and on the new Synology DS1819+

Results from a CIFS mount of a folder.

DS215J:




DS1819+:



OK, so way better caching on the random reads and writes.  The sequential is about 30% better.

Disks behind the test are:

DS215J    - 2 WD30EFRX-68EUZN0 - 3TB 5400RPM WD Red disks
DS1819+ - 2 WD40EFRX-68N32N0 - 4TB 5400RPM WD Red disks


Interesting to note the WD30EFRX has 147MB/s interface speeds, the WD40EFRX has 150MB.  The 2GB model WD20EFAX has 180MB/s interface speed and 256MB cache whereas the other models have 64GB cache.  Strange difference, 2GB model seems to be much better.

There are no real specs of I/O perf of these models but the assumption is newer ones have higher areal density so better (faster) I/O.

Group Consensus

Learned from Ted Talk How can groups make good decisions? that if asked to come up with estimates, individuals on average do worse than smaller groups of 5 (estimate on height of Eiffel tower).  In moral issues small groups tended to cancel out (suppress?) more extreme views.  Presenters didn't advocate but implied that this might help out in voting situations to mediate extremism.

Great Caesar!

WSJ artcle on Caesar salads - https://www.wsj.com/articles/caesar-salads-for-everyone-variations-on-a-beloved-classic-1513170000

Interesting that it came from a Tijuana restaurant.  The guy who invented it was an Italian/American guy from California named Caesar Cardini.  The restaurant was opened in Tijuana in the 1920s and was popular with people from California who would go there to eat and drink during Prohibition.  That's a very interesting statement in these times; it would be like somebody today who would take a road trip to Mexico for the afternoon so they could eat tacos and legally smoke weed.  People would assume they were a drug addict.  I wonder if in the 1920s it was assumed people were alcoholics if they crossed the border just so they could go to a restaurant and have drinks?  Different times.

You can actually buy Cardini dressing in the supermarket.  I regularly get it; it's quite good.

Active evolution? Hardly

From Discover Magazine.

1 Minute - The amount of time, on average per night, that a group of modern Tanzanian hunter-gatherers were all asleep at the same time.  Experts say the minuscule overlap supports the notion that humans evolved different sleep patterns as a way to ensure someone was up to alert others of nighttime threats, such as predators.

Pet peeve of mine, a lot of descriptions are like this where the implications are that the evolution was directed and intentional (as a way to ensure someone was up).  In reality it's less glamorous.  The humans who happened to be randomly de-synchronized in their sleep patterns didn't get eaten by tigers or murdered by bands of ruffians.  The ones who all slept at the same time ended up dying and didn't pass on these 'sleep at the same time' genes to their offspring.  That's the evolutionary pressure, not some sort of evolving different patterns as a way to ensure someone was up to alert others.  That sounds so carefully planned out.

Really what we think happens is random mutations give a survival advantage.

Strange conventions

Reading some non-fiction (Michio Kaku, Physics of the Impossible), I noticed that a paragraph describing metamaterials described properties of metamaterials and then the next paragraph started:

What are these metamaterials?  They are substances that have ...

What's interesting here is that this conversational method "what are..." seems perfectly normal in a book, makes the language interesting like it's really a conversation, yet when I'm reading a book and I'm reading facts it's very strange to think of this as a conversation.  What are metamaterials does absolutely nothing to tell me what they are, the sentence is pure throw away but yet it feels natural, even in written form.

I think this might be like when somebody calls you up and says "Hello, it's Joe", when we all know that I know it's Joe since the ID on the calls said so.  But, if Joe started up right away with the talking, the conversation would be thrown off.  Like I'm expecting an announcement of who it is.  I wonder if younger kids start off phone calls this way or if they just launch into it since they never had a time before caller ID...  (Plot twist, younger kids don't seem to call each other on the phone, they just text.  Plus they don't announce who they are at the beginning of the text.)

Planning Poker

https://www.planningpoker.com/ - online free planning poker (for up to 5 people) that can be used to estimate sizes of things, typically used in agile/scrum.  Doing it as poker avoids anchoring or influence across the team.

AI

Amusing yet very helpful AI info.  http://aiweirdness.com/faq

The Robots are Taking Over!

Good article, Wired, 25.09 Sept 2017, P064.Robopocalypse, talking about robots taking over, forecasts of people being out of work:

History suggests that the process is much more uneven than that.  The ATM, for example, is a textbook example of a machine that was designed to replace human labor.  First introduced around 1970, ATMs hit widespread adoption in the late 1990s.  Today, there are more than 400,000 ATMs in the US.  But, as economist James Bessen has shown, the number of bank tellers actually rose between 2000 and 2010.  That's because even though the average number of tellers per branch fell, ATMs made it cheaper to open branches, so banks opened more of them. ...  Taking a wider view, Bessen found that of the 271 occupations listed on the 1950 census only one - elevator operator - had been rendered obsolete by automation by 2010.

Follow-up: Perfect being the enemy of the good

Good WSJ article.  https://www.wsj.com/articles/fda-seeks-to-reduce-nicotine-levels-in-cigarettes-to-nonaddictive-levels-1501253894

The FDA wants to regulate nicotine levels to make them non addictive.  Also they will start encouraging people to switch to less dangerous products such as e-cigarettes and smokeless tobacco.

People have noted that low nicotine products have been tried before and didn't make the market.  I suspect that's because they tried going to 0 nicotine.  Instead they should work like the patch and have different levels of nicotine.  Note that e-cigarettes have varying levels of nicotine available.  I suspect regular cigarettes have the levels of nicotine carefully set in order to satisfy the customer.

In some ways this is a bit like coffee and caffeine and such.  We have regular coffee and we have decaf.  I think if coffee caused cancer we would be having similar discussions.  The only thing that's a bit strange is if we have cigarettes that have no nicotine, wouldn't smoking them still cause cancer, since it's not the nicotine that's cancerous, it's the smoking part.

Plus I don't think we've yet drawn a lot of light on the addictive qualities of caffeine.  People need to get their fix in the morning, it's a stimulant, there are withdrawal symptoms, etc.  Like drug use there's all sorts of paraphernalia for making coffee, etc.  The similarities are quite close!  Apparently the thing that gets us all up in arms is intoxication which is the only thing missing in the caffeine.  Although I know some people who become somewhat aggressive after having a few cups of coffee, perhaps like a lighter version of somebody going nuts on crack.

Will be interesting to see how this all goes with the nicotine levels and harm reduction with other product uses.  I think the tobacco companies will thrive and will adjust to this quite nicely.  I think they've seen this coming for years (not that they are prescient but I think they have noticed rightly so that these other products are less harmful).

Follow-up: Tech Giants

Recent Op-Ed in the WSJ - https://www.wsj.com/article_email/tech-giants-see-how-the-mighty-have-fallen-1500677602-lMyQjAxMTI3MjI1MzEyNjMyWj/

There was recent concern about Artificial Intelligence causing widespread loss of jobs.  A reader noted that there was a similar panic in the 1990's where people feared that computers would replace teachers in the classroom.  Instead, the widespread use of technology in education has revitalized curriculum, teaching, and learning.  Technology remains an important tool to advance and transform the nature of human work.

Perfect being the enemy of the good

Great WSJ article - https://www.wsj.com/article_email/big-tobacco-finds-surprise-allies-in-smokeless-push-1500629402-lMyQjAxMTI3MjI5MzEyMzM0Wj/?mg=prod/accounts-wsj

Many scientists agree that moist, smokeless tobacco, including chewing and dipping tobacco, is significantly less harmful than cigarettes. But rather than encouraging the country’s 37 million smokers to switch to less-risky products, U.S. health officials have so far stuck with an abstinence-only message to the public.

Online fact sheets published by the Centers for Disease Control, the Food and Drug Administration and the National Cancer Institute list multiple health risks associated with smokeless tobacco—including cancers of the mouth, esophagus, and pancreas—but give no indication it is less harmful than cigarettes. “There is no safe form of tobacco,” the cancer institute says on its website.

I think we have an all or nothing tendency which can be harmful.  This is another case of the perfect being the enemy of the good.

The Robots!

Laurie Penny noted in Wired Aug 2017, P014 that

A recent Oxford study predicted that 70 percent of US construction jobs will disappear in the coming decades.

Makes you wonder what construction jobs were like before large yellow metal machines were used to dig big holes and pour concrete.

Paleo Mainfesto

Diet ideas from the Paleo Manifesto.

Make Broths and Stocks.

Eat Fermented Foods.  Sauerkraut, Kimchi, pickles, kombucha, full-fat Greek yogurt, aged cheeses.  Avoid sterilized fermented foods.

Use traditional animal fats, beef fat, lard, butter (Kerrygold, Irish grass-fed butter).  Healthy plant oils include coconut and olive oil.  Avoid industrial vegetable oil such as canola or soybean oil.

Eat a variety of colors, different color plants have different chemical compounds.

Eat Eggs.

Eat Liver.  Duck liver pate is affordable and an easy way to get liver into the diet without having to cook it.

Eat Oily Cold Water Fish.  Mackerel, sardines, herring, anchovies.

Eat Seaweed.

White Rice is OK.

Drink Tea

Alcohol is OK

Avoid industrial foods, things made in a factory, contain lots of ingredients and that don't spoil. 

Avoid sugar, vegetable oils, cereal grains.

Avoid Cereal Grains.  Seeds have toxic proteins designed to pass through mammal digestive tract intact and dispersed and covered in manure for fertilized growth.  Toxic proteins are more heavily concentrated in the outer shell, the bran.

Avoid Legumes, which are grain-like seeds.  Soybeans, peanuts, lentils, peas, alfalfa, any variety of beans.  They have similar defenses like wheat.  White rice is OK.

Avoid nuts and seeds, almonds, cashews, sunflower seeds, like grains and legumes.

Avoid Potentially Problematic Foods, tomatoes, eggplant, peppers, potatoes.

Avoid fruit juices with added sugar.

Interesting note, Wheat contains opioid peptides and make eating wheat enjoyable, addictive, and difficult to stop.

Get Smart!

smartmontools are awesome!

Get status of a drive:

smartctl -a /dev/sda

USB connected drive:

smartctl -d sat,12 -a /dev/sdb

Run a quick test:

smartctl -d sat,12 -t short /dev/sdb

Run a more in-depth test:

smartctl -d sat,12 -t long /dev/sdb

Your License Please!

Great article in Barron's by Gene Epstein, July 10, 2017, P. 27.

Paraphrasing:

The pressure on legislature to license doesn't come from the public but the members of the occupation. (Milton Friedman).  55 years later two other commentators observe licensing having negative effect on employment (since it restricts access to the field).  It's also the case that occupational licensing "widens the gap between rich and poor by squelching employment opportunities at the lower end of the socioeconomic scale, and by inflating the compensation of highly skilled professionals at the top of that scale."  (The Captured Economy, How the Powerful Become Richer, Slow Down Growth, and Increase Inequality, by Brink Lindsey and Steven Teles).

No study has been done on degree to which occupational licensing has widened income inequality but we do know that since 1970, the share of workers subject to licensing has jumped from 10% to almost 30%.  There are lots of occupations paying reasonably well which people at the low end might normally be able to fill with minimal on-the-job training, but they may be out of reach due to money and time required for the license - beauticians, manicurists, barbers, preschool teachers, athletic trainers, gambling dealers, bartenders, massage therapists, interior designers, and florists.

Those who defend licenses confuse it with branding, meaning branding makes us better-informed consumers.  Private market already performs this in various ways, online evaluations for example.  The advantage of branding is the market doesn't place restrictions on people's right to enter a field.

Studies of licensing show little connection between quality and licenses.  Louisiana requires florists to be licensed which Texas doesn't.  An experiment involving florists from both states revealed no difference between floral-arranging skills of the licensed professions vs. unlicensed.  That's because licensing is mainly about barriers to entry, not enhancing skill.  

I agree with this but I would be concerned about skills at the higher end professions.  Licensing does force you to keep up with studies and updates that normally would languish.  Also, some things might lack adequate oversight or market pressure to weed out inferior people (sounds like I don't believe in market efficiency in some fringe cases) whereas routine testing or re-licensing should help that.  But I definitely do agree that there are some professions requiring licensing that are just way too much nanny state.


Also somewhat related from the Editorial Commentary by Thomas W. Hazlett

The Radio Act of 1927, the brainchild of then-secretary of commerce Herber Hoover, created a regulatory regime for carefully parceling out airwaves according to a "public interest" standard.  It was said to be necessary to prevent chaos - "etheric bedlam."
In fact, it was not.  Rather, it reflected Washington politics that favored incumbent interests - the first few visionaries who opened radio stations and enjoyed commercial success.  The scheme hamstring competition and flummoxed innovators for generations"

WeMo Insight

Handy data logger for electricity use.  Useful for diagnosing fridge issues since I wanted to know if it was consuming electricity or not.  Previously I'd been using Kill-A-Watt which is a nice tool but doesn't have data export stuff.  WeMo Insight has some nice export and remote access.  The app is decent and lets you set the cost for electricity but for some reason on the iPhone 6s I can't set it and get it to stick.  If I use the app on the iPad the setting sticks.  If I use the app on android it sticks.  But not on the iPhone 6s.  Strange.  I have a ticket open with their support but they can't reproduce the issue.  https://www.youtube.com/watch?v=RpcIWSBPDPg


Can chart daily or hourly electricity use via export to Excel:



Give a good sense of if the fridge is behaving and consuming electricity normally.

http://www.belkin.com/us/p/P-F7C029/

Data Logger

The Elitech RC5 data logger is really great.  The fridge has been on the fritz and so being able to see how the temperature has been going is really good.

Software is OK, works fine on Windows 10.  Stopping recording data and starting recording again requires a strange process which isn't documented (well?  at all?) but luckily Franklin (https://www.amazon.com/gp/pdp/profile/A2NA41EE5VUMY0/ref=cm_cr_dp_d_pdp?ie=UTF8) has the quick notes:

Quick start guide (since they didn't include one)

1) It will work plugged in to usb port or not, but your computer will heat it up, so don't measure temperature while it's plugged in
2) You have to hit the play button and HOLD IT DOWN for four seconds to start recording
3) You have to plug it into your computer, and use the software to stop recording. Click the stop recording button at the top right bar on the GUI
4) You can't start recording again, until you zero out the data. To do so go to the parameter set menu in the software, and click on the "Save Parameter" button. You don't have to change anything, but this will zero out the data set and allow you to log new data.

Step 4 for zeroing things out is important.  I couldn't for the life of me figure out how to start recording again until I googled and found that step 4.

Data export goes nicely into Excel and you can then slice and dice.  They produce a nice image for you:






Available on Amazon at https://www.amazon.com/Elitech-RC-5-Temperature-Recorder-Accuracy/dp/B00MQSCZF2

Think of the children!

So apparently Owl's Brew (http://www.theowlsbrew.com/) makes and sells beer/tea beverages.  Beer mixed with watermelon and white tea, ale with Darjeeling hibiscus and strawberry.  Won't children accustomed to Arizona iced tea be confused and mistake this new product for actual iced tea, think of the accidental ingestion and increased visits to the hospital emergency rooms!  We must ban this and put a stop to this practice now!

Advanced Technology

OK, Market Basket is now the most technologically advanced supermarket I know.  They are able to print their register receipts ON BOTH SIDES!  Seriously.  BOTH SIDES.  I've never seen this anywhere before.  My rough calculations indicate this saves about 1/2 the paper.

The Rise and Fall of Violent Crime in America

The Rise and Fall of Violent Crime in America.  Barry Latzer


I'm a sucker for fact based evidence and research.  Great science, economics.

Great book about rise and fall of violent crime over the past hundred years or so.  Reads somewhat like a college textbook.  Lots of great references and end notes.  Good fact based read.

Interesting takeaways:

Causes of drops in crime: Young men going off to war, correlation with alcohol abuse (higher abuse == higher violent crime).  Unemployment does not correlate; it's not the case that when there's lots of work available that crime drops.

Longer term trends, crime dips 1935-64, rises 64-95, dips 95.

Rise in 64 attributed to increase in young men (baby boom) exceeding a tipping point, a systemic overload of the criminal justice system and police/courts, and a subculture of black youth violence - swagger as defense against violence - respect and preemptive theft to increase respect.  Came from the southern redneck culture.  See "The Code of the Street".  Crack cocaine epidemic in the 80s also significant source of violent crime.

Social contagion - people commit crimes because their peers are doing it.  Things exceed a tipping point.

Dip in 95 - Cocaine epidemic ran its course.  Younger people saw negative effects on their older siblings, did not want the same fate.  More cops and policing about yes but the clearance rates of reported crimes being solved actually went down.  (But, total amount may have been better but % was worse; was not explored by the author).  (Thought experiment - do we have the expectation that percentage remains fixed as number scale (from a social perspective)?)

Definition of culture - the collective programming of the human mind that distinguishes the members of one group or category of people from the others.

Sources of fear (say of flying or being a crime victim) - likelihood is important but that is tempered by unpredictability of something happening, the inability for us to take precautions, and potentially devastating consequences of the thing happening make all the difference.

Windows 10 quirks, part 1

Windows 10, loving it.  However, a couple of strange quirks have popped up.  Sharing those here:

ASUS SBW-06D2X-U DVD/BD stopped working.  Had been using this fine.  Used it to install the computer in fact.  Had been working fine under Windows 10 for a few months.  Then mysteriously after a reboot, the device wouldn't show up.  I'd plug it in, get the sound that the device was recognized successfully, but it wouldn't show up on the system.  Looked in the Device Manager, it was seen there (after I elected to show hidden devices, ahem!), however, Windows said it wasn't attached.  Strange.  Googled some posts, lots of stuff about removing ATAPI drivers, IDE base, etc.  Not for me since this is a USB device.  I removed the device from device manager hoping that on next plug-in it would be recognized and new drivers would be installed.  No dice, plugging in got me the same success sound but the device didn't show up.  DANGIT!  Googled some more and found posts (http://answers.microsoft.com/en-us/windows/forum/windows_10-hardware/asus-sbw-06d2x-u-will-not-work-in-windows-10/9865fa58-b14b-4670-94fd-a542cc270051?auth=1) saying that there was another device it was showing up as - INITIO combo device.  On a whim, removed that device, unplugged and replugged in the device, and magic, it all works now.  Yay.

So, strange that this was working, looks like a new(er?) windows device driver showed up and decided it was in control of this device and blocked the ASUS driver from owning the device.

Here is some javascript for displaying random numbers:





<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <title>New Web Project</title>
    </head>
    <body>
        <h1>New Web Project Page</h1>

        Max Number:
        <input id="maxnum" type="number" />
        Seconds of delay:
        <input id="delay" type="number" />

        <br />
        <button type="button" onclick="go()">
            Go!
        </button>

        <button type="button" onclick="stop()">
            Stop!
        </button>

        <p id="message1">

        </p>
        <p id="message2" style="font-size: 200px">

        </p>

        <script>
var interval;

function go() {
var maxnum = document.getElementById("maxnum").value;
var delay = document.getElementById("delay").value;
document.getElementById("message1").innerHTML = "you typed in maxnum=" + maxnum + " and delay=" + delay + "";
clearInterval(interval);
update();
interval = setInterval(function() {
update();
}, delay * 1000);

}

function stop() {
   clearInterval(interval);
}

var prevnum = -1;
function update() {
var maxnum = document.getElementById("maxnum").value;
var number;
do {
number = Math.floor((Math.random() * maxnum) + 1);
} while (number == prevnum);
document.getElementById("message2").innerHTML = number;
// document.getElementById("message2").style.fontSize="xx-large";
prevnum = number;
}
        </script>

    </body>
</html>

systemd

After recently upgrading an ancient system, I realized that my custom init scripts don't quite start in the same (kludged) order now that my Fedora box is using systemd.  So, a quick update is in order.

Jumping into man systemd and systemctl isn't particularly enlightening.

Helpful sites are:
http://www.linux.com/learn/tutorials/788613-understanding-and-using-systemd
man systemd.service



Create /usr/lib/systemd/system/[thingy].service.  This is the master holding place for configs and init.

To get things to start automatically at boot, symlink:
ln -s /usr/lib/systemd/system/[thing].service /etc/systemd/system/[thing].service

Edit /usr/lib/systemd/system/[thingy].service:

[Unit]
Description=Network Stuff
After=syslog.target network.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/[thingy] start

[Install]
WantedBy=multi-user.target

For a more persistent service:

[Unit]
Description=Network Stuff
After=syslog.target network.target

[Service]
Type=forking   (could be simple if the process does not fork)
ExecStart=/usr/local/sbin/[thingy] start
PIDFile=/var/run/blah.pid

[Install]
WantedBy=multi-user.target

To activate:

systemctl daemon-reload
systemctl enable [thingy]

To see status:

systemctl list-unit-files --type=service
systemctl status [thingy]

SD Maid cleans up WSJ

I've been reading the WSJ on my Nexus 7 from time to time.  A bit of a buggy app, but tolerable.

A little while ago I noticed that my 13G of free storage on the Nexus 7 had dropped down to about 3G.

At the time I didn't realize these two things were related...  I just figured I'd used up the measly space with various things (books, music, videos, apps, etc.), and my little tablet was just overwhelmed.

So, first up, figure out what's using the storage so I can do a bit of spring cleaning.

The Android Settings app has a nice panel for Storage, and it shows you what's using storage - free storage, and storage used by apps, pictures,videos, music, downloads, cache, and misc.

Apparently I was using about 10G of Misc.  Nice.

I've been using ES File Explorer, so time to use it to trawl the filesystem looking for what's using what.  OK, apparently there is a folder on the root of the "sdcard" called ".wsj-External-data-cache" and in there are a bunch of directories, each one holding a bazillion files.  Probably about 10G worth.

OK, fine, uninstall the WSJ app.  Data cache remained.

Use ES File Explorer to delete the folder.  No go, it craps out with java memory exception when deleting folders with 10 bazillion files.  I was able to delete some folders, but there was a pernicious folder with too many files that ES File Explorer just could not delete.

I googled around and found other file explorer apps, but they all seemed to crap out when trying to delete the folder.  Argh.

I happened to stumble across "SD Maid", which also lets you trawl through the filesystem doing stuff.  This actually worked and was able to delete the remaining data cache.  It took about 10 minutes to delete everything, but was successful, no java memory exception.  Nice.


(This post reminds me of the time my MacBook filled up with disk because of the backups.  Maybe I'll post about that next.  - Apparently Time Machine on MacBooks makes backups on the local disk, for times you wanted to restore a file accidentally deleted.  Is all fun and games until you use about 400G of your disk space on backups.)

Disaster Recovery

So when I set up my shiny new Fedora 16 box, I was careful to set up a pair of disks as Raid 1.  Any disk failure and I would be covered, right?  Wrong.  What I neglected to do was put the GRUB2 MBR (Master Boot Record) boot loader on my second disk.  So, even though I had all my data mirrored onto the second disk, if the first disk failed, my system would not boot because the MBR was only on the first disk.  Did I say if the first disk failed, I meant to say when the first disk failed.  And sure enough, it failed.  Unbootable system.  I tried swapping disks around, etc, but no MBR on the second disk so no boot.  Here's what I did to recover.

The scenario - 2 disks, sda and sdb set up in raid 1 mirror.  Only sda has the MBR for grub2.  sda failed, leaving me with a good sdb but the system could not be booted.

First, I got a trusty Knoppix boot DVD.  At first I'd tried to use the Knoppix CD but it only boots at 32 bit and I couldn't chroot to my installed system (on second disk) because it was 64 bit.  Knoppix 7.0.4 only has grub and not grub2 so I need to use the grub2 code that's on my working sdb.  So, Knoppix DVD, available at http://www.knopper.net/knoppix/index-en.html.  I tried messing around creating a boot USB on my Mac using unetbootin, etc but ultimately wasn't able to create anything that booted.  (Had to use my Mac because obviously my Linux box was not working...)

Once I had a working Knoppix DVD, I booted on it (boot parameter "knoppix64") and accessed my working sdb (which Knoppix Linux had brought on as sda since the failed sda was at this point now removed and disk names are done as they are seen).  Since it's a metadisk, in order to mount the disk, I had to do:

mdadm --assemble --auto=yes /dev/md0 /dev/sda2

sda1 is swap.  sda2 is a large LVM partition.  The mdadm --assemble brought up metadevice md0 with member device sda2.  mdadm was kind enough to bring on the array even though it was missing 1/2 of the mirror.  So nice.

Next, to mount the filesystems:

mkdir /media/lvm

mount /dev/vg00/lvslash /media/lvm

mount /dev/vg00/lvboot /media/lvm/boot 

Mounting the LVM devices (use the "pvs", "vgs", and "lvs" commands (no args) to display the contents of the various LVM components.  The original / and /boot are mounted under /media/lvm.

Now at this point I want to run grub2 to get it to re-install the MBR.  But, no amount of messing around with various components or options could get it to run and see things and install the MBR.  I kept complaining that it couldn't find /boot/something/grub2 stage1.  But the file was there at the path it was looking at.  My only guess was that it was trying to find it by reading the filesystem directly and couldn't since the LVM or grub on Knoppix was different.

Regardless, I needed to do a chroot to fake out the system into thinking it was running for real on my disks and not Knoppix.  The chroot was easy enough, but then grub2 would refuse to run because it couldn't find running device info (which is on /dev but not the chroot /dev).  ARGH!  The fix is to remount /dev and /proc from the real system onto the chroot system.  Surprisingly this just works:

mount --bind /dev/ /media/lvm/devmount --bind /proc /media/lvm/proc

That done, I can chroot to the filesystem:

 chroot /media/lvm

and then run grub2 to install the new MBR

grub2-install --recheck /dev/sdagrub2-install /dev/sda 

After this was done, I exited out of the chroot, unmounted filesystems, rebooted on the new MBR, and like magic, I'm good to go, the working disk is fine.  Next up, to get a replacement disk and create the LVM for the mirror and resynchronize across two disks.  And also to put the MBR on the second disk :)

Need for Speed 2

Upgraded the wireless to a dual N band router.  Is a little faster than before.  See ancient history here http://gaplax.blogspot.com/2012/06/need-for-speed.html

Copying a 5.74G file around:

802.11g - 38 minutes
802.11n - 17 minutes


This router is a little frustrating as when I configure it as an Access Point, I can't get at the router to configure it even though I've told it to allow access to the management interface over any net.  Not a huge loss, but is nice to see the mgmt interface to see who's connected to it.

LIRC

I've been using XBMC for a little while.  I have a Rosewill remote RHRC-11002.  Had been working fine but then stopped working.

Turns out in the XBMC Lircmap.xml this remote is known as mceusb but somehow LIRC is now reporting that the device is mceusb_hauppauge.

The fix is to edit /usr/share/xbmc/system/Lircmap.xml and change the known device to mceusb_hauppauge:


root@xbmc:/usr/share/xbmc/system# diff -c Lircmap.xml.20120804 Lircmap.xml
*** Lircmap.xml.20120804 2012-03-23 20:25:30.000000000 -0400
--- Lircmap.xml 2012-08-04 05:44:31.617871872 -0400
***************
*** 9,15 ****
 
  
 
!
  Play
  Pause
  Stop
--- 9,15 ----
 
  
 
!
  Play
  Pause
  Stop

Done, and my remote is now working again.

Weekend temptation, the Internets themselves.

So I currently have the 25Mb Internets at home (beats the 10Mb at work, ahem...).  Verizon now has "Quantum Internet" so I can get 300Mb.  Schweet.  No pricing for 300Mb, but they quote 75Mb as +$10/month.  Very tempting.

According to Verizon:
  If you order 150 Mbps/65 Mbps or 300 Mbps/65 Mbps FiOS Internet, a Verizon technician may need to install a new router and wiring in your home. Please ensure your computer has a10/100/1000 Gigabit Network Interface Card capable of supporting these higher speeds.


Good call on checking to see you have a GigE interface on your computer!

Last time I did a speed test I got ~30Mb.  http://gaplax.blogspot.com/2011/11/fios-speed-test.html

Mass Energy

Annual check in on Mass Energy http://www.massenergy.org/heating-oil

Costs $20/year to join.  They state their prices are $0.20/gallon cheaper than the state average.  For us with annual use of ~700G this works out to about a $140 savings.

The (Honest) Truth About Dishonesty

Just finished Dan Areily's book "The (Honest Truth About Dishonesty)"

Take home points:

 Simple Model of Rational Crime (dishonesty based on rational analysis of cost/benefit of cheating, benefit of the crime, probability of getting caught, expected punishment) is completely bogus.  Study was set up (solving math problems for $$), subjects did not cheat more when the payout (for cheating) was higher.  It was actually less.   Probability of getting caught?  Study showed roughly the same amount of cheating as when there was NO chance of getting caught.

The more distanced from actual money (tokens vs. actual $), people were more likely to cheat.  People are more likely to steal $1 of pens (office supplies) than an actual $1 bill.  Stealing $1 and then using it to buy a pen is somehow worse than stealing a $1 pen.  Getting people to recite an honor code (thou shalt not steal) makes them less likely to steal.  So does signing *before* they fill out a form (like a tax form).  Filling out after does not have this effect.

The more distanced you are from the act, the more likely you are to cheat.  Golfers cheating by moving the ball are less likely to consider it cheating if they move the ball with their club or feet than by their hands.  This also ties into cheating by stealing $$$ vs. stealing supplies.

Resisting temptation somehow wears down our moral resistance, and after enough resistance we are apt to give up (run out of steam) and give in.

Wearing (known) fake products (sunglasses, bags) makes us more likely to cheat on various tests.

Cheating across cultures and nationalities, pretty much the same.

Being supervised decreases dishonesty.  Even a picture of somebody watching you decreased the likelihood that somebody would cheat on putting the correct change in a box to pay for a snack.

Essay mills - people paying for essay papers?  As of 2010 - a survey of the quality of these papers shows them to be so poor it's immediately apparent to professors that they are complete junk.

People are more likely to cheat in an altruistic situation where they are helping somebody else.

Forces that shape dishonesty:

• Ability to rationalize the situation
• Conflicts of interest
• Being creative
• One immoral act spawning another
• Being tired
• Altruistic benefit
• Watching others cheating

Forces that have no effect

• Amount of money to gain
• Probability of being caught

Forces that decrease dishonesty

• Pledging honesty
• Signing something
• Reminding of the moral code
• Being supervised

  

All in all, a good read, good studies back up the conclusions.  The book starts out strong, but by the later chapters seems to run out of steam and becomes more anecdotal.

Coach!

Oh man, I was saddened to learn that the McGurk effect (http://en.wikipedia.org/wiki/McGurk_effect) had nothing to do with Coach McGuirk.  Too bad.

QuackTime Player

Grrrrrr.  Why is it that the QuickTime player can only play MPEG-2 at OSX Lion levels.  It can play MPEG-4 just fine.  What is it about MPEG-2 that they couldn't handle?  VLC has been doing it for years.


This is the sort of crap that drives me bonkers with Apple stuff.  If you stay within the ecosystem, everything is fine, but once you stray out, you realize the ecosystem is actually quiet small and limited.


http://support.apple.com/kb/HT3775?viewlocale=en_US

Below are the video and audio file formats and codecs that QuickTime Player can playback in Mac OS X v10.6.x or OS X Lion:

Media type	File formats	Codecs or components
Video	QuickTime Movie (.mov) MPEG-4 (.mp4, .m4v) MPEG-2 (OS X Lion only) MPEG-1 3GPP 3GPP2 AVI DV	MPEG-2 (OS X Lion only) MPEG-4 (Part 2) H.264 H.263 H.261 Apple ProRes Apple Pixlet Animation Cinepak Component Video DV DVC Pro 50 Graphics Motion JPEG (Mac OS X v10.6.x only) Photo JPEG Sorenson Video 2 Sorenson Video 3