Mike Fedyk

"What version of trixbox should I use?"

2007-07-03T21:52:02Z

Trixbox 2.0 includes asterisk 1.2.

Trixbox 2.2 uses asterisk 1.4

I'd stick with asterisk 1.2 or callweaver for now.

No program "pvscan" found for your current version of LVM

2007-07-03T21:49:45Z

That's what I get when I run pvscan right after installing the lvm2 package. It doesn't make sure the lvm kernel modules are loaded before reporting this error.

This command fixes the problem:

/etc/init.d/lvm start

I need to report a bug...

Stats gathering library

2007-07-01T07:04:24Z

I mentioned previously about using sysstat to create an unified stats gathering lib for tools to use, and it looks like someone else has already started a project to fill that gap.

Check out libstatgrab:

The libstatgrab library provides an easy-to-use interface for accessing system statistics and information. Available statistics include CPU, Load, Memory, Swap, Disk I/O, and Network I/O. It was developed to work on Linux, FreeBSD, and Solaris. The package also includes two tools: saidar provides a curses-based interface for viewing live system statistics, and statgrab is a sysctl-like interface to the statistics.

Linux Firewall Distributions

2007-06-30T22:27:45Z

RedWall Firewall:

redWall is a bootable CD-ROM firewall which focuses on Web-based reporting of the firewall's status. It includes Snort, snortsam, dansguardian, and support for fwbuilder, squidguard, reporting (using BASE/sarg/ntop/webfwlog), VPN (Openswan/PoPToP/Openvpn), Spam Filtering (spamassassin, dcc, razor2, clamav, amavis-new, dspam and maia mailguard), and mail-based, alerting. Configuration data are stored on a floppy or USB disk.

SmoothWall Firewall:

SmoothWall is a popular Internet Security software package (based on Linux) offering automated modem/advanced ISDN autoprobing, ethernet ADSL/cable, USB ADSL (Alcatel Speed Touch Home only in 1.x; additional support for USR, Fuji, ECI, etc in 2.x), and multiple ethernet card support within 5 minutes of install. Web managed and with full facilities normally only seen in expensive commercial offerings, it also offers SSH, DHCP, and full firewall logging and auditing functionality.

IPCop:

IPCop Linux is a complete Linux Distribution whose sole purpose is to protect the networks it is installed on. By implementing existing technology, outstanding new technology and secure programming practices IPCop is the Linux Distribution for those wanting to keep their computers/networks safe.

Devil-Linux:

Devil-Linux is a special secure Linux distribution which is used for firewalls, routers, gateways, and servers. The goal of Devil-Linux is to have a small, customizable, and secure Linux system. Configuration is saved on a floppy disk or USB stick, and it has several optional packages. Devil-Linux boots from CD, but can be stored on CF cards or USB sticks.

My goal is to find one that I like and integrate bwmtools, bandwidthd and ShoreWall into their frontends.

The proprietary vendors need a bit more competition in this space IMO.

Also I think Shorewall would make a fine standard firewall framework for Linux. Either that or some standard API needs to be created for Linux Firewalls.

Database clustering with sequoia.

2007-06-26T05:34:47Z

I've been looking for something like sequoia for a while for multi-location database replication and failover.

Ubuntu Feisty and a Z22 Palm Pilot

2007-06-03T21:26:51Z

I'm trying to get a Z22 Palm Pilot working with Ubuntu Feisty today.

First I found this in the kernel logs:

usb 1-1.4: new low speed USB device using uhci_hcd and address 5
usb 1-1.4: device descriptor read/64, error -71
usb 1-1.4: device descriptor read/64, error -71
usb 1-1.4: new low speed USB device using uhci_hcd and address 6
usb 1-1.4: device descriptor read/64, error -71
usb 1-1.4: device descriptor read/64, error -71
usb 1-1.4: new low speed USB device using uhci_hcd and address 7
usb 1-1.4: device not accepting address 7, error -71
usb 1-1.4: new low speed USB device using uhci_hcd and address 8
usb 1-1.4: device not accepting address 8, error -71

Then I found ubuntu bug #54419 and it gave me three commands that were useful:

sudo modprobe -r ehci_hcd
sudo sh -c 'echo blacklist ehci_hcd > /etc/modprobe.d/blacklist-ehci'
sudo update-initramfs -u -k `uname -r`

That takes away the error messages in the kernel log, but I was still having trouble finding the USB serial device for communications from jpilot.

That brings me to ubuntu bug #108512. Apparently the visor module doesn't get loaded when a palm gets plugged into a usb port on feisty. The two commands below take care of that problem in a "works for me" kind of way. The permanent fix would probably be in the form of udev rules.

sudo modprobe visor
sudo sh -c 'echo visor >> /etc/modules'

With that, jpilot works on Ubuntu Feisty.

Servers either lost power or rebooted at PIHost

2007-05-29T14:20:25Z

$employer has servers located in PI Host.

All of them were either rebooted or lost power over the weekend.

Should I need a UPS at a co-lo facility?

Virtual Networking

2007-05-28T17:21:34Z

"Host only networks" in VMware are cool. If you want 50 networks on your system, you don't have to worry about having the physical network hardware (or VLANs) to separate the traffic.

Now I just need to find where vmware stores their configuration so I don't have to edit it through the vmware-config.pl program and find out how to turn off dhcp service to host-only-networks.

That's the one hole I see in this setup. The dhcp service for host-only-networks is from the host, and that can be an attack vector. So far I've just been killing them since I don't need them and don't want dhcp on those networks.

Routing, VMware & Wireless

2007-05-28T05:31:45Z

I spent today hacking with my friend Charles Wyble in addition to returning and borrowing some books. :)

We setup a Linux router in a VMware guest. His machine now has five ethernet ports (one built-in and four in a multi-port card), but the tulip driver in Ubuntu Dapper Server has a weird probing quirk. The ports that physically are ordered 1, 2, 3, 4 were detected as eth4, eth1, eth2, eth3. No problem. Just reorder the network interface names in /etc/iftab and we're set.

Next we configured the vmware networking so that each vmnet is associated with the correspondingly named ethX number. So eth1 -> vmnet1, etc. Then we added a "host only" interface, but it took over vmnet1 and that was fun debugging. "Why am I not seeing my public IP address on this interface?!"

After getting packets flowing, installed shorewall and setup a dmz. A cursory look at this setup shows that there is a price to pay in resource usage. What would take 2-7% cpu usage (a torrent of FC5 at 600Kbyte/s) to do on the the host system takes about 25% processor usage within the VM guest on an AMD Athlon XP 2200+. The security and flexibility of this setup is unquestionable, but it is good to know the cost.

Also Charles had a problem with Windows in a VMware guest on his Ubuntu Feisty 64bit laptop not being able to get out to the network. It turned out not being a windows problem, but a ndiswrapper problem with his bcm43xx chipset wireless card. Plugging in a wired ethernet connection worked around the problem and getting a Ralink mini-PCI card will be a more permanent solution.

I guess it was a windows problem, since a windows NDIS driver caused the problem. Heh.

Optimizing Linux Performance by Phillip G. Ezolt

2007-05-28T04:22:55Z

Today I finished reading Optimizing Linux Performance by Phillip G. Ezolt.

Even though I already knew a lot about the subject, it did fill in a few gaps in my knowledge of oprofile, kcachegrind, valgrind, vmstat, iostat, script, objdump, gdb and sar. It also clued me into a few tools I didn't know about like ltrace, gprof, memprof, ipcs and etherape.

One tool it didn't mention is dstat. The package description says it all:

<dstat>Dstat is a versatile replacement for vmstat, iostat and ifstat. Dstat overcomes some of the limitations of these programs and adds some extra features.
.
Dstat allows you to view all of your network resources instantly, you can for example, compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval).
.
Dstat also cleverly gives you the most detailed information in columns and clearly indicates in what magnitude and unit the output is displayed.
.
Dstat is also unique in letting you aggregate block device throughput for a certain diskset or network bandwidth for a group of interfaces, i.e. you can see the throughput for all the block devices that make up a single filesystem or storage system.
.
Dstat's output, in its current form, is not suited for post-processing by other tools, it's mostly meant for humans to interpret real-time data as easy as possible.

One thing about dstat is that it uses six times more memory than vmstat (but the output is prettier):
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mfedyk   23001 0.6 0.5   5452 3824 pts/5    S+   21:27   0:00 python /usr/bin/dstat 1
mfedyk   23027 0.0 0.0   1780   588 pts/6    S+   21:28   0:00 vmstat 1
</dstat>

I liked the examples used to for discovering performance problems but not most of the solutions proposed. I understand the point of the book isn't detailing the way optimize your system libraries (glibc, gtk, etc.), but building menu caches in nautilus (the latency performance example) might have gone over better with me if there was at least a mention of improving the library being called. Sometimes the best course of action is to improve the library you're calling in the Open Source Way(tm) instead of just working around problems you find in your application code.

The book also shows areas where Linux and its performance tools could be improved. It got me thinking about how to change the entire Linux performance monitoring ecosystem by first griping about how things are currently and pontificating on how things would be with a few changes.

Right now, every performance monitoring system (cacti, munin, etc.) has it's own performance gathering code. sysstat's sar (System Activity Reporter) is already written in C. It needs to store more statistics and have language bindings to Perl, Python, Ruby, PHP, etc. for polling its database, and a generic stats reporting API for stats gathering agents like snmpd, etc. This would allow one project to be the focal point for system monitoring statistics gathering and remove the duplication of work.

With this work done, performance monitoring *might* be close to what windows currently offers. I don't say that lightly. This is a serious area where Linux needs to catch up.

gnome-system-monitor (g-s-m): The lines are so thick it looks like they were drawn with a crayon. Vital stats like nice, system, hard and soft irq cpu usage are not shown. So much vertical space is wasted with the child sized buttons to change the colors of the lines. At least it doesn't flicker when updating its stats anymore. This tool has improved steadily over time but still needs a lot of work. Compare this with Windows' Task Manager. While not the best, it does present more information in less screen space than g-s-m.

Also since the book was written when Fedora Core 2 was current, there have been a few things that have improved since the book was written. For instance, major and minor faults are accounted for properly now. Let's get that per process block and socket/pipe statistics gathering into the kernel.

Quad core CPU support in Centos 4.5

2007-05-18T00:51:56Z

The machine I saw the error in with Vmware Server 1.0.2 (Yes, I need to upgrade...) was running on a 2xquad core xeon server (2 sockets). Now centos 4.5 has support for quad core.

I wonder if that has anything to do with the error I was seeing in the centos5 guest...

Contacted by Vircom (Makers of ModusMail)

2007-05-17T23:02:02Z

Here is an email I received from Vircom yesterday:

"While reading on the internet, I came across something you wrote in your blog on Vircom’s ModusMail (see below). I am interested in knowing more details so we can make sure that customers have a positive experience with Vircom’s products. For instance, did you contact Vircom support on this issue?

If you can be kind and provide me with more information, I would greatly appreciate it. As Product Manager, I am interested in improving Vircom’s products and not trying to sell you our solution. Detailed negative feedback is especially useful for me because it helps show weak areas in the product that we need to improve."

I sent him details on the service provider that is blaming them for the slow email server responsiveness. Hopefully they will be able to help them. Though I do not hold that service provider in high regard, so it will probably be difficult for Vircom to do anything about this.

We'll see. It's good to see Vircom making an effort in this area.

CPU errors with SMP Centos5 on VMware Server

2007-05-17T20:07:22Z

Looks like I'm running into the "Centos beta 5 shows cpu errors on vmware" with Vmware Server 1.0.2 on a Centos4 host. Switching the VM to one CPU seems to avoid the problems. We'll see.

Centos5 LVM disk stats bug filed upstream

2007-05-17T19:28:02Z

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=240476

No response on the list so far, so I'm filing a bug upstream.

RHEL5 Documentation fixes

2007-05-17T19:10:16Z

Yay, I found and fixed Red Hat's documentation typos.

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=240474

Vmware server and LVM

2007-05-17T17:16:40Z

I wanted to store my VMs on LVM with each partition on a separate Logical Volume (LV) and the MBR stored in the vmx file. It looks like I can do this with Xen, but vmware doesn't even allow you to select LVM devices. I tried putting in a full path, but it just says "Unable to determine the bus type of device /dev/mapper/VolGroup00-base_guest". This is on a Centos4 host. Boo vmware.

The nice thing about putting this on LVM is that each LVM LV is treated as a separate drive by the disk accounting subsystem in Linux so that you can see how much activity each VM is generating. I guess I'll just have to monitor each VM independently... :-/

Xen or VMware or OpenVZ?

2007-05-16T01:58:04Z

Of course openVZ is not mutually exclusive with either Xen or VMware. They can be used together, but that is not the purpose of this post...

I found an interesting link to a forum post on the centos mailing list talking about OpenVZ today.

It says that once the memory limits are reached in an OpenVZ Virtual Environment (VE) it kills the process that hit the limit.

<speculation>I suspect (since I haven't researched and found the details yet) that after reaching the memory limit in a VE, OpenVZ simply refuses allocating memory when processes request more memory. That can be bad because a lot of apps don't handle this very well and die. You would see this also if you simply limited the memory allowed to a process group or user.</speculation>

What happens differently when a virtualized kernel (like in Xen or VMware) hits its physical memory limit is that it starts using swap and if swap is full or unavailable[1] it will discard as many memory pages as possible, disk cache, exeutables mmmap()ed, etc. This can allow more memory to be allocated, though with a serious drop in performance.

But the kernel tries hard to satisfy the allocation before refusing.

If my speculation above is correct and OpenVZ hasn't made any modifications to the linux Virtual Memory Manager so it will simulate "memory pressure" in each VE independently, then OpenVZ (and presumably the generic Linux resource limits code) will simply reach the limit and refuse further allocations without trying hard to find space for allocations and have serious problems with applications that don't handle memory allocation failures properly (that would be most programs).

This needs to be looked into further...

1. Some people have gotten into the habbit of configuring systems without swap. This mostly started because of the terrible VMM in the early 2.4.x kernel series. If you're running a kernel from kernel.org after 2.4.16 or any distro kernel, you should configure swap space on your system(s).

Linux Disk accounting with LVM Snapshots

2007-05-16T01:18:51Z

I was reading a lot of data from one LVM logical volume and wanted to watch the disk statistics with iostat today. To my surprise, I was seeing the same activity on two logical volumes (LV) and the physical disk containing the two.

A little investigation showed that one LV was a snapshot of the other, and I was reading from one of the snapshots.

While I understand that the two LVs share some common Logical Extents (LE), I believe it is a bug that the disk statistics show activity on both logical volumes.

Should I report this in the centos bugzilla or on upstream's bts?

Thanks,

Mike Fedyk


http://lists.centos.org/pipermail/centos/2007-May/081227.html

Fun with xen on centos5

2007-05-09T18:14:57Z

(15:52:39) mm: ping
(16:18:03) mfedyk: pong
(16:18:54) mm: these boxes don't like me, I asked rails-02 to reboot to test the init stuff and xenstored appears to be hung
(16:19:13) mm: can't even get an xm list from it
(16:19:32) mfedyk: hmm
(16:19:52) mfedyk: oyga
(16:20:41) mm: it's been like this for about 20 minutes
(16:23:34) mfedyk: xenstored is stuck in a loop
(16:25:09) mm: figures
(16:25:30) mfedyk: you doing anything disk intensive?
(16:25:38) mm: not doing anything at all
(16:25:57) mfedyk: gah
(16:29:02) mfedyk: ok, restarted xend
(16:29:14) mfedyk: I can ssh into rails
(16:29:24) mfedyk: it's been up for 30 mins
(16:29:25) mm: what's the uptime?
(16:29:27) mm: haha
(16:29:42) mm: sounds like it rebooted, but something was unhappy
(16:29:50) mfedyk: yes
(16:29:58) mfedyk: let me investigate
(16:30:05) mfedyk: tell me if you get more hiccups
(16:40:27) mfedyk: oh, this is a nice read: http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00487.html
(16:41:47) mm: hahah
(16:41:47) mm: GET on /xend/domain/demo is resulting in 16 copies being made.
(16:42:04) mfedyk: and writes...
(16:43:26) mm: hopefully that's something that deadrat backports
(16:52:56) mm: how the heck can rails-02 have a load of 1, it's not doing anything
(16:53:28) mfedyk: probably has a process stuck in D state
(16:53:44) mfedyk: yup, xenwatch
(16:54:52) mm: I give up
(16:54:58) mfedyk: lol
(16:55:05) mfedyk: we should consider using vmware

Unattended upgrades in Ubuntu and Debian

2007-03-22T23:04:11Z

With Ubuntu Dapper (don't know about previous versions, Dapper is the first version of Ubuntu I've used) there is a nice feature to automatically update your system. I found it in the preferences of the orange update notification icon and then searched for a way to activate this functionality from the command line for use outside of a graphical environment.

It's a nice package called unattended-upgrades. Well isn't that an obvious enough name?

Before this I had been using the cron-apt package to notify me when there were updates in Debian. I didn't feel comfortable with running `apt-get upgrade -y` because that would change config files. Now I don't have to worry about that at all.

I typically use the dapper-updates and dapper-backports sets of repositories and those packages would need a manual apt-get upgrade (or do the same thing with the GUI). But I wanted them installed for me automatically also. Luckily the list of acceptable repositories for unattended-upgrades is not hard coded.

My /etc/apt/apt.conf.d/50unattended-upgrades file now looks like this:

Unattended-Upgrade::Allowed-Origins {
        "Ubuntu dapper-security";
        "Ubuntu dapper-updates";
        "Ubuntu dapper-backports";
};

And my /etc/apt/apt.conf.d/10periodic looks like:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "0";
APT::Periodic::Unattended-Upgrade "1";

Now I don't have to install any updates manually.

P.S. This works in debian also, just use "Debian testing" or Debian unstable" in the Allowed-Origins list.

sbackup (Simple Backup) in Ubuntu Dapper

2007-02-09T08:30:00Z

I have been helping John troubleshoot (and learn how to troubleshoot) sbackup for the last few weeks.

We would identify one issue and fix or work around it and then another issue would rear its ugly head a few days later. The fact it was a medium priority item on a long list of tasks also factors into why it took the time it did.

It turns out that some of the bugs were already fixed in the newer version already available in Edgy[1]. These fixes weren't brought to the "Long Time Support" release, though this package is in universe so it's up to the community to handle it.

After checking the dependencies I found that the Edgy sbackup package can be directly installed on Dapper thanks to having a short list of deps (also being written in python might have helped).

That fixed the problem of not restoring files beyond 2GB within the archive.

I also had John go through the process of fixing one bug, reporting another in launchpad.

The one we fixed:
sbackup denies group access to /var/backup

The one we reported:
sbackup lock not stored in /var/lock

So far there haven't been any responses to these bugs.

The 2GB restore issue wasn't in launchpad so I created one, marked it fixed with a comment "Fixed in Edgy" and nominated it to be released for Dapper:
Files beyond 2gb in tar not restored

Let's see how things turn out. I wonder if I should escalate this to upstream. I will if something doesn't happen in the next week or so.

1. I didn't want to get into apt_preferences at the time because while teaching I have learned that it is best to stay on topic. If you cover too much, the associations you want to be learned get lost in the glut of terms and information from the side-topics. This was especially true when I had him rebuild the jpilot package from edgy on dapper (but that is a topic for another post).

JBOD SATA enclosures: Multi-Lane SAS/SATA

2007-01-30T12:13:20Z

Port multiplier support in libata is just hitting lowercase 'experimental' status and is still a ways off from hitting mainline at kernel.org. The only other option is a hardware raid controller like the MegaRAID SATA 300-8X. But that is not an option in my case because the servers I am working with only have 5 volt PCI-X slots and all of the 8-port SATA cards I have found are 3.3v only.

Another option for economical external JBOD SATA is the multi-lane SAS variety. One of the nice things about multi-lane SAS/SATA (and eSATA with Port Multipliers) enclosures is that they are easily partitionable. The multi-lane will allow you to connect up to four (up to 5 with eSATA) drives directly to as many as three servers within cabling distance per 12-drive enclosure.

With this setup I will be able to plug a 4-port SATA controller into any quad cpu rackmount server bought off of ebay without the expense of SCSI drives and without the dependency on any single controller model by using software raid. Another option is connecting all drives to one Linux server and export it over iSCSI and/or NFS and boot off the network.

There are a couple more offerings now when compared to even two months ago. Let's review the contenders:

PC-Pitstop SC-SATA12RM $849
At first glance you can easily see that it is composed of three 4in3-bay internal enclosures like the ISTAR BPU-340SATA. At first I was surprised they didn't use the 5in3 ISTAR BPU-350SATA to fit 15 drives into the same space, but then I remembered. I have used an internal enclosure like the 4in3. If you want to keep your drives in a rattling, vibrating box, then use one of those internal aluminum 4in3 enclosures.

PC-Pitstop SC-SATA12RMD $1,199
As one of the new products available, this enclosure comes with dual-power supply, improved cooling and a front cover with lock. Yet it also noticably uses four of the 3in2 internal enclosures similar to the ISTAR BPU-230SATA. Maybe that says something about the 4in3. I hope I'm wrong about these enclosures from PC-Pitstop as far as vibration is concerned.

Norco DS-1240 (available from iPCDirect) $599
There are several reviews available. Reviews by Amug, ExtremeTech, newegg for the eSATA model and Firewire model are available. All of them are favorable to the quality of the enclosure with several mentioning the steel construction and approx. 30lbs weight without drives. One went so far as to call it a "tank" and there were no complaints about the trays or loading mechanism. Any criticisms were confined to the ease of accidental disconnection and speed of the firewire or usb interface, noise of the fans, quality of the included Silicon Image controller or incompatibility of the port multiplier with the eSATA model. None of those criticisms are relevant to this model because it will be on a rack next to equally loud rackmount servers and this model uses multi-lane SAS connectors with thumb-screws.

I was able to find one review for PC-Pitstop's 8 drive tower model. It uses two multi-lane connectors but is not rackmount. It also seems to avoid problems by not using internal X-in-Y drive enclosures. I haven't found any negative reviews of multi-lane SAS JBOD enclosures.

Good spam?

2007-01-30T01:26:53Z

I received a new type of spam today.

Missing children.

Some spam can be good.

Interesting usb devices

2007-01-30T01:24:46Z

Ever walk into an office or cube and see a heater plugged into the same outlet strip as the computer and shook your head?

Maybe usb can solve some of that:

USB Cup/can heater/cooler.

USB Mug heater with car adapter.

and yes:

USB foot heater.

The only thing I didn't find was a space heater. Even with the meager power available from a usb port, it should be able to warm the feet under a desk.

Yahoo acknowledges Firefox

2007-01-29T02:06:59Z

Just did a search on yahoo and noticed at the top right that it says:

"Search from your browser (red up arrow)"

When you mouse over that text, it pops up some instructions on how to search with yahoo from firefox.

I don't know how new this is (I don't search with yahoo very often). But I just noticed it.