That pesky “soft lockup detected” message…

If you’ve read any of my more geekish entries, you’ll know that I bought a surplus Dell Poweredge 2900 server last year with the intent of setting up a MineCraft server or two for my son.  I decided to use VirtualBox VMs because I liked the notion of playing with the software, and it seemed to me to be a bit more secure. There’s nothing all that sensitive on the machine but still, security is security and in this case access to one server did not mean access to the rest.

Problems in the form of purported kernel panics reported as soft lockup messages began as soon as I started setting up VMs.

soft lockup detected on CPU#x

I tried several Ubuntu versions, MineOS, and straight Debian, but still the issues persisted.  A couple of months ago, I whined to my Facebook friends and Ian Mclaird suggested that it might be an issue with the symmetric multiprocessing (SMP) support.  Thus armed with new words, I began another search and finally discovered that the issue is very specific to SMP in VMs.  It’s also a bit rare, I guess, as there are only two posts that I’ve been able to find addressing the issue.

See

The short version is that you need to tweak a kernel setting and make sure it’s applied at boot.  Using nano or your favorite editor, edit /etc/sysctl.conf as root, simply add a line at the end as shown. Note: for the later kernels at least, you can use a / or a . to delinate the sub-directories.

  • with kernels from somewhere prior to 2013, try: kernel/watchdog_thresh=180
  • For Ubuntu 14 and 15, as well as Debian Wheezie and Jessie, try” /proc/sys/kernel/watchdog_thresh
  • For Ubuntu 16: try  /proc/sys/kernel/hung_task_timeout_secs

Two notes:

First: when I tried to set this Ubuntu 16 I wound up with an error at boot to the effect of “kernel parameters not loaded.” (Sorry, I’m not going to try and recreate the exact error and I can’t find it on line…) What happened is that the value at /proc/sys/kernel/hung_task_timeout_secs is protected. I think, but have not confirmed that there is a maximum value and the sysctl checks the value.  As I didn’t want to mess with it any more, I reviewed the document at the third link (https://www.kernel.org/doc/Documentation/sysctl/kernel.txt)  and set /kernel/watchdog=0.  This was not my first choice as it disables both the hard and soft lockup watchdogs. I may get back to the this but for now, the system(s) are stable, so I’m moving on!

Edit (6/17): one of my Debian Jessie servers hung so I turned the watchdog back on. The max for /proc/sys/kernel/watchdog_thresh appears to be 60. It tosses an error with anything larger.

Second: you can try settings in a non-persistant manner by running (as root, use sudo as necessary)

sysctl -w /proc/sys/kernel/hung_task_timeout_secs=xx

You can also reload the /etc/sysctl.conf file with:

sysctl -p /etc/sysctl.conf

Echo4Golf Clear

Me

I am not
who I want to be
is
better

I am
so many things bad and good but mostly just me
so many thoughts right and wrong
so many impulses (un)controlled
cowardly
weak
afraid
liberal conservative conservative liberal
privaliged
token

I am (not)
alone in a world filled with hate and discontent

An open letter to President Obama

Dear President Obama,

I’m a supporter. I voted for you twice and would do so again were it possible. I have watch through eight years of an obstructionist Congress making every one of your choices, good or bad, a huge battle. I have seen men and women on both sides of the aisle, men and women that I should be able to respect, act as spoiled little children, as traitors to our nation in the name of … well whatever it was in the name of. I understand at least some of the challenges that you have faced.

Having said that, I have a question for you:

With all due respect, sir, words are great, but what are you / we going to do about this?

In fact, I ask this question of our Government as a whole. Senators, Congress-people, appointed Cabinet Members, policy-makers all: what are we going to do about these fundamentalist fruitcakes who choose to commit suicide by cop in order to make one anti-other statement or another. If it’s not radical “Muslims,” it’s so-called Christians. Or radical-this or fundamentalist-that. Where is the plan to root out the causes of these evil acts performed by evil people?

Such actions are all grounded in fear and hatred and they are a much more significant issue on a national and international scale than any other. They have got to be stopped. Again, I ask: where is the coherent plan from our government? From the people? From *any* of the candidates for any office?

We all know, or at least we should, that America has some very hard decisions to make, even before the election. This especially given the nature of the two front-running candidates, neither of whom is acceptable to me or to many-many other Americans. The time has come for leadership and not finger pointing. The time has come to put away the political bullshit and start being Americans rather than “liberals,” “conservatives,” straight, gay, religious, atheist, or any of the many other things that divide us as a people.

Shame!

Our leadership should be ashamed of their example, as should those striving for higher positions within that leadership.

We, as Americans, should be ashamed for tolerating such obstreperousness in our leaders. They are supposed to be of us, not a separate ruling class. For shame America, for worrying more about how much each of us can have, rather than making sure of the supply for generations to come, and sharing our bounties with the world. For shame for our greed, our hatred, our ignorance, and for our baseless fears. For shame for our need to watch our brothers and sisters more closely that we watch ourselves.

For shame.

There will be a reckoning in November, but that may well be too late. The time to start leading is long past. I call on all those in positions of leadership to accept the responsibilities to which they aspired, or leave. It’s that simple: either lead, follow, or get the hell out of the way!

Further, I call on all Americans to come together and put an end to this “silly-season” before it’s too late and our great nation is no more. I call on all Americans to exercise discretion and tolerance before acting, and to engage their brains before running their mouths. We must help those who need help, and do away with the issues that allow these fundamentalist freaks to control us.

America was once a great nation. There is no question of that, unless we look at the past through rose-colored, revisionist lenses. But we are no longer. Although we may be the only cold-war “super-power” left standing, we have failed ourselves, our children and the rest of the world. It’s time to clean up our act and take our place among the greats again. Starting now!

@Barack Obama ‪#‎ForShameAmerica‬ ‪#‎AmericaMatters‬

Server Upgrade: OS, Virtualbox, and MineOS

Warning: if you’re not into computers, or get annoyed by amateurs who are, you might want to skip this post!

The new server is up and running with everything that used to run on the Raspberry Pi.  My original philosophy for the unit as a whole was to run a very basic Ubuntu server install with Virtualbox, and then to run the different servers and services as separate virtual machines.  My thinking was that even though that would duplicate the Linux installs, I have plenty of resources and this setup would allow me to work on one server issue while the others were unaffected.

But I’m getting ahead of myself!  First, let’s talk about basic hardware configuration. For those just joining us, I purchased on eBay a Dell Poweredge 2900 (see this post). in order to set up a MineCraft server for what was supposed to be a middle school club.  The club fell through but I set up the servers anyway. I also decided to move the servers from a Raspberry Pi to the “new” machine. The machine was set up a bit differently than advertised.  4GB of memory turned out to be 8, and the hard drives turned out to be hardware arrays (73GBx2 in a RAID 1, and 146×4 in a RAID5).  Once I re-seated the hard drives everything started right up. I discovered that parts for the machine, with one exception, are dirt cheap so I spent about $120, maxed the memory at 48GB, added 3 more 146GB drives, and the missing front bezel.  The machine came with dual-ethernet, a DRAC (remote access card) which provides dedicated access to the server outside of the bandwidth being used by ETH0 and ETH, redundant power supplies, hot swappable fans and drives, options to mirror or assign memory and drives as mirrors or hot-spares, and other things that I don’t think I’ll ever use.  Add to that two 1TB external drives, and I’m looking at about 3TB of storage.  Not to shabby.  In fact, if you’re keeping score (I am, and I suspect SWMBO is as well…) that’s about $180 so far, which isn’t bad for that system.  I also passed my old garage workstation on to Nick, so I’ve added a video card and a Sound-Blaster as well.  Unfortunately, those two were a bit more specialized and cost about the same as the entire rest of the system.  I also had to give up (for now, at least) the DRAC5 in the server, due to IRQ conflicts with the video cars.  I had thought those a things of the past, but we’ll see. It’s working so I’m not in a hurry to fix anything.

In any case, for less than $400, I won’t complain.

In terms of software, I installed a copy of the latest Ubuntu 14.04 as the OS.  I started with 15.04, but went back a version because the Debian folks have changed some things that I didn’t want to deal with (and don’t yet understand).  The install was reasonably straight-forward although, for some reason, a USB drive install didn’t work. The machine kept trying to find install files on the CD drive once booted, rather than looking at the USB stick. Not a big deal and I simply burned the ISO to a CD and ran a traditional install.

I next installed VirtualBox (Instrtuctions here) and set up virtual machines for MineCraft and a simple LAMP (Linux, Apache, MySql, and PHP) server. That’s where things started to go a bit downhill.  The LAMP server was no problem and is still running today.  The MineCraft server was an issue, however, in that I could not get a modded server to run stabally. As soon as I set them up on the physical machine, all three of the servers started to behave themselves. The way is stands now, minecraft.echo4golf.com shows server status and ports and resides on its own VM.  I plan to consolodate that to the LAMP server when I have time.  Again, it’s not broken, so I’m not in a hurry to fix it.

This blog, several other services including an OwnCloud installation and our eBook library (running Calibre ODPS (and HTML) PHP Server or COPs), and a couple of admin services such as Webmin and  Shell-In-A-Box are running on the VM LAMP. That machine has 2GB of RAM dedicated, and a 120GB virtual drive that I’m going to compress at some point.  It doesn’t need the space as the OwnCloud files are on a USB drive connected to the server and shared with the VM.

The physical service is running three MineCraft Servers controlled by the MineOS WebUI as well as Dell’s Open Manager Server Adminstrator.  As I noted above, I lost the DRAC, so the OMSA install is kind of important.  I also went ahead and installed LXDE (see LXDE.org | Lightweight X.11 Desktop Environment) as I like a GUI occasionally, and I’m familiar with this one from the Raspberry Pi.  It looks a lot like Windows, and is pretty responsive, both of which are a plus for me. This way, I can run the server either remotely or directly, and either as a console or GUI session.  I’ve added a Windows 7 VM as well and plan to move my workstation there from what is now Nicky’s desktop.

So, that’s where we are.  Coming soon:

  • Adding a Video Card and a Sound Card to the Dell PowerEdge 2900
  • Installing a simply LAMP Server, services, and utilities on the Dell PowerEdge 2900
  • Multiple sites in Apache
  • Simple (and FREE!) DNS, SSL, and MX services for your domain

In the meantime,
Echo4Golf clear!

 

 

 

 

 

For Shame for All of Us.

So glad to see such a humanitarian response to the burgeoning Syrian crisis from Saudi Arabia, Kuwait, Qatar, and Bahrain among many-many others. Good thing that there are no Muslims, Jews, or Christians involved or I’d suspect some sort of hypocrisy…

FYI: “You cannot attain to righteousness unless you spend (in charity) out of what you love.” (Holy Quran 3:92)

or: “And if thy brother be waxen poor, and his means fail with thee; then thou shalt uphold him: as a stranger and a settler shall he live with thee.” (Hebrew – English Bible, MT, Leviticus (Vayikra) 25:35)

or, if you prefer: “Hatred stirreth up strifes: and charity covereth all sins.” (Vulgate Bible: Douay-Rheims Translation, Proverbs 10:12)

or perhaps: “If I should have prophecy and should know all mysteries, and all knowledge, and if I should have all faith, so that I could remove mountains, and have not charity, I am nothing.” (Vulgate Bible: Douay-Rheims Translation, Proverbs 10:12, 1 Corinthians 13:2).

Our leadership, national and international, brings shame on all of us. Again. And once again, God, in many aspects, names, and forms is ashamed of us.

New Server Part 1: The new Server OS (Ubuntu) and Virtual Box

I recently upgraded to another new-to-me (i.e. used) computer.  My philosophy has been to, as a rule, buy decent quality used or reconditioned computer gear for some time.  This means I can get a decent box at a price much better than new.  Given that our computers rarely wear out before succumbing to general abuse or being passed on (and then succumbing to general abuse…), this policy has served us reasonably well thus far.

This latest acquisition was a bit larger than usual.  The kids (and I) wanted to run a dedicated Minecraft server.  I tried a virtual machine on my desktop and it worked reasonably well for the three of us, but would occasionally lag.  I moved it on to an old Dell Optiplex GX60.  I had to add a network card as the one on the motherboard was not working ($8), as well as maxing out the memory (2GB, $8).  Unfortunately the machine itself was not able to keep up with more than two players.  Given that I intend to have at least two servers up, and may wind up hosting the local middle school Minecraft Club, this was an issue.

I wound up purchasing a Dell Poweredge 2900 locally on eBay for $66.  No, that’s not a typo.  Although advertised with 4GB of memory and 73GB and 146GB drives, the listing was incorrect.  The server came with 8 gigs, 2 73gb drives in a RAID, and 3 146GB drives in another array. The latter needed reseated before they came up, but they’ve been working reliably since.  Did I mention $66? The 2900 also comes with dual processors… Those are dual-quadcore processors.  It’s also got all sorts of cool server goodies like redundant, hot- swappable power supplies, hardware RAID through BIOs, hot-swappable HDDs, and exessive memory capacity.  The best part is that there are lots of parts available inexpensively.  I get the feeling that there are a lot of these still running in commercial application.  I scored 32GB of memory for 29.99, for example, and used drives are less than $20.

The one issue is that the machine is huge.  Dell calls the form-factor a “tower” but in truth it seems to me more like an old mini-computer than anything. It also weighs in at a hefty 80 pounds or so.  (Amazon lists a shipping weight of 100, so I’m guessing 80. It’s heavy!)

All in all though, I’m very pleased with the purchase. I don’t see it replacing my desktop, and certainly not my laptop of tablet, but I am going to move all of the existing e4g servers off of my Raspberry Pi.

Next: Basic Configuration

Echo4Golf Clear!

Yet more technology

So. I bought a server. Wait: is too much. Let me sum up.

Son got daughter and me playing Minecraft.
Son wanted a server of his own to play on.
Dad: “No problem.”
Server 1 was a virtual machine on my desktop.
Server 2 was an old Dell tower
Server 3 is a Dell Poweredge 2900 server class machine.

In the immediate future, look forward to a series on transitioning my sites and services from the Raspberry Pi to the Dell.

Echo4Golf Clear

Desecration of our Flag

<rant>

http://www.usflag.org/uscode36.html

A note to all of my friends sharing the US Code (link above has a decent history of the code)  as it relates to desecration of the National Colors: the code was struck down by the Supreme Court in 1990. The Court argued that burning the flag was an act of free speech protected by the first amendment of the Constitution. You’re more than welcome to disagree. Heaven knows I hate seeing the Colors treated inappropriately! But consider this: the very system that you claim to support has created this decision.

It’s not *your* country. It’s *ours.* And I’m sick to death of small minds on both sides of the aisle trying to bring it down.

How about we stop with the indignation and the invitations to love it or leave it? How about we stop looking for things to hate and find things to love?

Echo4Golf Clear

</rant>

So many projects, so little time…

Just that.

I hope this post finds everyone well. I just realized that I have three posts started, another two in the planning stages, and several other projects happening at once. When did this happen? Stay tuned for more on the Raspberry Pi changeover, a potential Minecraft server for my son, shaving related posts and, as I get the shop back up to speed, posts about Echo4Golf Woodworking and my pens and other wood projects.

Echo4Golf Clear

Raspberry Pi Redux- Setup and Basic Configuration of the R-Pi V2.

As noted in a previous post, echo4golf.com and this blog are self-hosted. They run on a Raspberry Pi Linux based computer. If you’re not familiar with the Raspberry PI, check out the history of the Raspberry Pi foundation at the Raspberry Pi Foundation web site. Basically, it’s a complete linux-based computer system on a card about the size of a Altoids tin. A new Pi runs less than $50 shipped, and all the user needs to do is add an SD card, hook it up to an HDMI capable TV, plug it in to a cell phone charger and go! OK, it might be a bit more complicated than that. New users or those unfamiliar with Linux will want to read up on the process from the numerous sources available. But the Pi is a very useful little device. Furthermore, the latest version, the Raspberry Pi V2.0 has twice the memory and runs about six time faster than the original. The author has procured a v2 Pi, and this article is the first in a series describing the migration of the e4g servers from the original to the new PI, as well as some basic changes to stabilize the server configuration prior to the migration.

The author’s new Pi came from eBay seller PhotoDeals, who were not only offering the best price from a US based vendor, but had the unit in stock. The RPi2 is still new enough that it can be challenging to find. The vendor shipped quickly, and the unit arrived within three days.  Not bad considering it was ordered on a Friday!  The unit cost $49.99 with free shipping. As of this writing, the price has been reduced to $44.99.

http://www.ebay.com/itm/181670826756?_trksid=p2057872.m2749.l2649&ssPageName=STRK%3AMEBIDX%3AIT

As a bit of a side note, careful shopping on eBay will yield some great buys on the original RPi. Be careful though. It appears that a lot of vendors aren’t clear on the differences and are selling the original as RPi V2. The difference between the two is significant and the older slower ones should be dropping in price, one would think. PhotoDeals has the older model for about $35.

http://www.ebay.com/itm/Raspberry-Pi-Model-B-B-Plus-2-0-Version-512MB-RAM-Linux-System-Board-/181471424051

The new unit is pretty much the same form factor as the original Model B Pi.  It won’t use the same case, however as there are four USB ports instead of two.  Additionally, migrating the system is not as simple as moving a drive and installing new drivers.  The original Raspberry Pi uses a standard SD card as a boot drive and for storage.  The new Pi v2 uses a micro-SD.  Although the card-based system is acceptable for many users, those who intend to utilize databases (by installing WordPress or OwnCloud, for example) would do well to migrate the operating system to an external drive before discovering how easily they can be corrupted on an SD card. Yes, that’s the voice of experience! Before we get that far, however, some basic configuration is in order.

You’ll want to get your Pi out of the box and remove packing materials. For this stage of the operation a keyboard and monitor or TV with an HDMI port is necessary. Once the system is set up, you’ll be able to run it remotely via an SSH connection.

The first step in configuring a Raspberry Pi is the creation of the bootable SD card. There are all sorts of instructions online but, following the KISS principle, the author chose to install NOOBS (New Out of Box Software).

The alternative is not all that challenging, and merely involves getting the image for a chosen operating system and writing it the the card.  “Official” Images are available at http://www.raspberrypi.org/downloads/. There is the Debian based “Raspian,” a Fedora based “Pidora,” a couple of media center setups, and RISC OS (about which your author knows nothing except it’s not Linux, Windows, or Mac-based). Should you choose to go that route, instructions are freely available from many sites.  Basically though: get the image, unzip it, use SDFormatter from the SD Association (https://www.sdcard.org/downloads/formatter_4/ as of this writing) to format your card, write the image with Disk Imager (http://sourceforge.net/projects/win32diskimager/) from Source Forge, insert the card, and start the Pi by plugging it in…

Setting up NOOBS is simpler.  Download it from http://www.raspberrypi.org/downloads/.  Unzip it. Copy the files to the root of the card. Start the Pi. Done. If you have problems, run the SD Fomatter above, with the size correction setting enabled. Then restart the card again.

Once you’ve start the Pi, you’ll see a list of the operating systems available.  The author uses the Debian-based Raspian. In any case, check the box for the operating system of your choice and grab a cup of coffee or a beer while the OS installs.

As the install completes, the system will run the Raspi-Config utility.  There’s not a lot that you have to do here but you should set all three of the “Internationalisation Options.”  Among other things, this will avoid issues with your keyboard and, later on, with the SSSL Certificate system.

  • Select number 4. “Internationalisation Options.” and then I1, “Change Locale.” Select as appropriate. The author selected all three “en_us” options with utf-8 as the default.
  • Select 4 again and this time choose I2 “Change Timezone.” The list is alphabetical and US is at the bottom.  After that you’ll be able to select the specific timezone appropriate to your locale.
  • Once more, select 4 and, last but not least, choose I3 to change the keyboard layout.  There are specific keyboards listed but the author uses “Generic 101-key PC.”

A couple of other things that you might want to consider, while you’re in Raspi-Config, are enabling the SSH server for remote or “headless” access, if you don’t leave the Pi attached to a monitor and keyboard, and setting a network host name.

To enable the SSH server, select 8, “Advanced Options,” and then A4 “SSH.”  You’ll see a screen asking if you wish to enable the server.  Select yes and hit return. You’ll see another screen telling you that the server is enabled. Select OK and you will be taken back to the menu.

To set a host name instead of the default, select 8 again, and then A2 “Host Name.” Pay attention to the naming requirements as the utility will not tell you if you use an incorrect name.  Underscores (_), for example, are not valid, however dashes (-) are.

Once you’ve completed the configuration, select “Finish” and allow the system to reboot. Don’t disconnect it from the monitor and keyboard just yet, however, as the next step is to set up the Pi to run from an external hard drive, rather than the SD card.

Reboot the system and then we’ll look at moving the root to a hard drive.

Echo4Golf clear.