Vmware server, iscsi, AOE, nas, fiber channel, WTF??

When navigating the remote storage forest you need to keep a level head and some focus about what you really want to accomplish. The jump from truly small business to enterprise level prices begins here, SAN, and ends with your clients wondering how somebody can ever afford to pay 63k for essentially a file server. Should Fiber Channel cards cost $750 when you can get a 1G ethernet for $19? Who knows. Whenever you tack “enterprise” to the front of something expect the price to rise accordingly.

What I know thus far is that I am having some real issues with vmware and their virtual filesystem. Let me contrast this to my experience with Xen. With Xen, I could create a LV (logical volume) and simply export that LV to the Dom (VM, Guest OS) and have it work. This was fantastic, I didn't realize just how fantastic it was until I decided to resize a Vmware disk.

I have Vmware running on my server that has Raid-1 with LVM so I can expand paritions with the best of them. For Vmware though, first you need to grow the file system, easy using their built in utility, I grew my disk from 10 to 20 gigs np. If you are not familiar with how Vmware stores its filesystem for Vms: it uses a bunch of files. When you expand it, it just creates a bunch more files. Here is the weird part, after the expansion I noticed that where I started the expansion, the filesize was normal size, but the file before it, the previous last file, was the same size that is was before, smaller. Now that I read that it sounds confusing. A better way to say it would be the last file in the list before I expanded it was smaller. This is just like any other archive format that I use, rar being the best example. When it expanded it just built a new file after the last old one, it did not start and change the old file, essentialy adding space to a single parition. Hmmmm. What does that mean? It seems it means that the disk was 'grown' by adding more files which makes it look like another partition on the drive.

Now I have multiple partitions that should would work as advertised. I cranked up the Vm and ran cfdisk, and there was the new space staring me down. Great, now just to take mounted drive off line and resize the partition, then the fs. What a pain. Yes, I know about mounting RAW partitions with Vmware, this however was not really supported with beta, so I steered away from it. Now I need to take a better look, because of what I am in need of right now: more space.

My client is really expanding fast. I sold them my current server for their shop of 8 people. They had burned through approximately 30 gigs in 20 years they had been around, so I figured 300 gigs should be about right. At least until I got a handle on what thier requirements were. Well now that they have 20 people in the course of six months, and a real addiction to Vmware-server, its time for plan B. I know they need a SAN, I feel it in my bones. I have not , as of yet , added some network storage to my hardware inventory. Its probably about time.

Kicking it up a notch is what its all about, so i decided to see what it would take to build a SAN. Madre de Dios , I have trouble believing people pay what they do for some hardware. I am really in the right business. My servers will perform with the best of them, I use all top rated hardware, so why not build a SAN that can do the same? The trouble is the communications channel that is needed to build the connection. If you go with 'enterprise' equipment, you are going to go with fiber channel, and of course pay out the wazoo. If you live someplace besides Boardwalk, you might opt for the next big performer, abet half as fast, ATA over Ethernet. This is Marvin Gardens right here, still pricey, like Applebees instead of Houstons (6k and below, sans drives). So down we go, gimme Baltic Ave, were looking at ISCSI. ISCSI will perform over standard Gigabit hardware, and can be utilized with Windows and Linux. Looks like we have a winner. Welcome to the Taco Bell of SANs, and right in my budget.

The dilemma here is the performance hit on the processors with ISCSI. You can offset this with a TOE device (TCP offload Engine). Is a TOE device really needed for my application, or could I deal with a hit in overhead? I think  the cost of trying to incorporate a TOE in my 1u servers would out-weigh the benefits of the added cycles. I might be horribly wrong here, so more research is in order.

The case is a different beast. You can buy a large case with a hot swap back plain, now you just need a raid card that supports SATA raid-5 with a ton of ports. Of course the hit there is for the card. There is a solution with a port multiplier. The SATA II spec allows for a port multiplier, a device that will marry multiple SATA drives to a single port, and my Asus MB in my server supports SATA II. Bingo. However, I would like to be able to do this without a little device mounted in my case. So I think I will opt for the card. That means raid-5 on the card and better performance. Enough to allow me a little slack if I skimp on the TOE for now, go figure.

Bottom line, a SAN is going to be a little more expensive with the added cost of the Gigabit switch and the added card but well under 2k. That suits me, and better yet, my client just fine. Expect to see me offer my own line of low cost SANs in the near future.

How long is too long in cyberspace. I am wondering if there is a change in the way that we conceptualize time and what it means when our thoughts and actions to a certain extent are cataloged and indexed for reference. Take this entry now. It will be stored as a collection of thoughts and pinged back to an aggregation site and some people will find me that way. Others will stumble upon this entry because I mention things like: Robert Anton Wilson , Naked Cheerleaders, teeth, plaque conspiracy, and Metallica. Whatever the case, you made it, and the information fresh to those who find it after the novelty of those search terms is forgotten. The linear way of associating thoughts and descriptions is indeed becoming old fashioned. The movement to the 'Andy Warhol' dissection of future fame becomes more substantial, and the power of this new media is being spawned and created by those who would consume it. In the midst of this change, the challenge for the future creators will be to consistently feed this need that the public is creating for content. I am torn between the need to express certain ideas and concepts, and the idea that the content should reflect quality of posts, not quantity.

Back to the idea of time. If I do not write for some time, but now I am indexed by google, I exist now in a finite state. I am preserved, tied not to time, but to a concept. When people think of an idea, or want to follow a trial of thought, they could stumble over one of my posts, my diatribes, even if it is weeks or months old. The relevancy will no longer be determined by the amount of time it took me to make the post but rather the idea that the post realized. The interface that the person makes with the search engine algorithm will determine the outcome of their search, and the amount of time that is spent on any particular 'conversation', which is what this blog is essentially: a one sided diatribe that gives you the reader a chance to respond en masse, Any Time you want.

What I am doing lately for mental stimulation is watching lectures on google video.

There are a few really good classes online that I enjoy, and many presentations that you could get no where else.

Google hires people to come and talk to their employess, pays them large amounts of cash, then puts them online for free. If you like hard to find media and other gems, make sure you turn to video.google

Here are a few of my favorites:

Seth Godin at google, fantastic! Must see if you want to know about web 2.0 and your business.

http://video.google.com/videoplay?docid=-6909078385965257294&q=presentation

Physics for future presidents, Berkley. This lecture is great for everybody. It's a real down to earth look at physics.

http://video.google.com/videoplay?docid=-7520664547647462372&q=physics+for+future

Alternative 3: Was it a secret plan to shuttle scientist and engineers off the earth to work on the dark side of the moon, or a hoax? England gets a taste of Orson Wells' style trickery in this weird documentary.

http://video.google.com/videoplay?docid=9098938544636366793&q=alternative+3

Install Vmware server on debian Etch (testing) in 7 steps: (should work same with stable)

1. Install Debian PXE boot as outlined here, make sure you select Desktop and Standard in package config during install. Installed already? Step 2.

2. Get your Key from Vmware here

3. Download code for Vmware Server and the Management UI

4. Download gcc-4.0 : apt-get install gcc-4.0 ; ln -fs /usr/bin/gcc-4.0 gcc

5. Download headers for kernel : apt-get install kernel-headers-2.6.16-2-686-smp (your kernel may vary, do a cat /proc/version and apt-cache search to find your headers)

6. Untar and Install Vmware server: /vmware-server-distrib/vmware-install.pl

7. Untar and Install Vmware-mui : /vmware-mui-distrib/vmware-install.pl

Connect to the MUI at https://servername:8333

Vmware loves to @$%& me.

I love Vmware. I simply love it. I have done fantastic things with this software, and made many a client's day with a resounding “yes, I can do that”, or “sure, I have an idea of something that will work”, and Vmware is the cure.

I can imagine in the offices of Vmware when they decided to release vmware-server for free. They were high five'n, maybe sipping lattes with their pinky fingers out, who knows. I do know they probably had a picture of me on a dartboard, or my head pasted on some sort of porno scene, laughing as they came up with this sorted scenario: Let's kill the beta software and force users to upgrade, effectively breaking our product during testing, that will really screw Doug good and hard. Thanks guys, you did, you screwed me until I spit qwerty.

Killing software that is “free” because it's a beta, is well, dumb. If you want me to test your software for you, and help you proselytize to the unwashed about your software, do not !@#$ me in the ass. Do not kill the software while people are using it, or prevent machines from rebooting with a vague reference in one of many log files that mentions “license expiration”. Do not leave me standing there like Daffy Duck in a rigid stance with one finger in the air declaring how great your product is, when your clever bomb explodes leaving me a pile of dust with just my beak saying how ”disthpicable” you are.

I run my business, and a lot of others, on free software. Never have I ever had to deal with the likes of this. If your reading this Vmware, let that sink in. You have actually created a NEW problem with free software. I didn't think that was possible. You sat around and came up with a completely new !@#!@#- up situation that I could deal with. I know you told me in your LA that I clicked through that it would probably not be reliable, and a bunch of other legal stuff. You probably hid it in there somewhere that you would have the right to break it at anytime. The sad part is I know why you did it. You wanted to make sure people would be using your final release, and not the beta, I understand that. Well, I would have upgraded, on MY time. You brought production in offices to a complete stop. You made a lot of enemies that day. The first time I caught it, I couldn't believe it. I upgraded before any production losses where felt. Then you did it to me AGAIN. I didn't catch it the second time, we call it V-day. You just killed my Virtual Machines in mid stream, and wouldn't let them reboot without installing new software. You forced me to install beta software over beta software, what the *&(*&(* are you doing?

I am sure that as time passes I will eventually forget transgressions, and let it all slip into the past. I mean your product is free, and I can deal with wonky stuff for free. It's not like there is any one else giving away a Virtual Server product for free. I mean Vmware has nothing to worry about, no competition, like Xen, or Microsoft. These little mistakes couldn't make me bitter enough to tell everyone I know about how evil this situation was. These little mistakes couldn't make me switch to another FREE competitor. No, I still love you Vmware, for now.

Windows Profiles and their idiosyncrasies in Samba.

Normal profile behavior is nice to have. But anyone administering a Windows environment with roaming profiles knows that certain behaviors are a cause of problems with the profile.

If you can replicate the behavior across a number of machines with a single user, then you have a profile problem.

If your problem is on one machine, check the Documents and Settings folder as administrator to make sure that no residual profiles are left over.

Strange program interaction, slowdowns, crashes, other such anomalous action:

This is an indication of some form of the profile being left on the machine. This occurs when the normal process of removing the profile is interrupted. Loss of connection to the Samba server can cause profiles to remain on the client , even if you have enabled delete cached profiles in the Group Policy.

Removing the profiles should be done with caution. I would suggest moving them to the desktop of the client you are working on. Then waiting to see if that user starts working again, or if the profile data on the server has to be rebuilt. You can use some of the data that is on the client to help rebuild the profile.

When you need to move profiles, reboot, and log in as administrator.

Navigate to:

C:\Documents and Settings\

Move the left over users. Now you might be subject to an error, not being able to move NTUSER.DAT. This is normal if you have not rebooted. Just reboot and log in as administrator and perform the task again.

Now log, and log in as the user. Everything should go back to semi-normal. Just check to see if the data is up to date. If not, you can navigate to their saved profile on administrator desktop, and copy the profile data over the current profile they have running. Log, and log in again. Do this until the profile is functioning.

There are a few things you need to do while setting up roaming profiles on windows clients, one of which is turn off the caching feature. Why? In my experience only bad things come from allowing users to cache their profiles to the server, evil and wicked things. Try to make a change to a profile on the server thats already cached on the client and you will see what I mean. Ok, on to the meat.

If you have not used the Microsoft Management Console before, just launch it with:

Start>Run>mmc

Once there, navigate to Group Policy Editor:

Add Remove Snapins>Add>Group Policy Editor

Next window says Local Computer, thats great, click finished.

Navigate to :

Local Computer>Administrative Templates>User Profiles

Enable Delete cached copies on roaming profiles

Enable Log user off when roaming profile fails

Thats it, done. Feels good, looks good. Everything should work great. I will assume that you are doing all this on the client as administrator, so navigate to:

C:\documents and settings\

And find the users that have cached data, move that someplace other then that folder. You might want to use it later.

Now, log, and log in as a user. Remember to check that profile size, large profiles make unhappy users.

What the BLEEP do we know?

I think as a species, we are still subdivided into groups that tend to think and act alike. Problems arise in these microcosms, these genetic or socially ordered groups, that reflect interactions of the larger social orders. Remember, what you are seeing and hearing most definitely is a reflection of these separate orders of thinking, and the truth is harder to grasp as it is being filtered through a sieve of internal reflection.

Finding sources of information that you can trust is a doggy quest. The best I believe we can do at this time is use an aggregate of these sources, and our judgment, to realize what is the truth. The algorithm for this problem should be derived from a very simple filter mechanism based on the oldest truth set in existence, our conscience.

The problem with 'truth' as a whole is the subjective nature of reality as we recognize some of the science to our being. As observers of the universe, we play a crucial role in the definition of the 'truth'. The natural order that states we manufacture some level of our reality logically means that for the most part all of us create the same thing. The truth is order, perfect order. The truth is all that is solid, and by relation, good and real. When I think of truth, I think of light. I think of things that lift me up, and carry me forward, bring me to a higher level of being. The truth will set you free. The truth is the Alpha.

So If we equate truth with light, then by relation, dark must mean something else, lies. With that mindset, evil would be absence of solid form, nothing. No solid form, lower, down to a zero state. Nothing, something, one and zero, notice the relation here. These concepts seem familiar because they are the manifestation of our physical reality from the truth. Evil, or social entropy, is the movement towards another state, that of nothing. Remember that entropy death and decay are essential parts of our reality. Without the canvas, there can be no painting. Evil is the movement of a state of being towards the unreality of everything. You are everything, when, you are no-thing. The truth is the Omega.

We all know these things, they are in us from the time we are born. There is no other description for good, or bad. The process of categorizing and recognizing patterns in this dichotomy is what we call morality.

Telling good from evil at times can be a difficult task. We are told often that the justifications for actions of others are based on proposed outcomes that will eventually lead to something good. Notice that it is counter intuitive to believe that any good action can ever come from an action that would by its nature be classified as evil. However, it is possible to add good to a situation to positively affect the outcome.

The easiest way to get at the truth is to first categorize particular actions as good, or bad.

Good

I know, its the weekend, I should be relaxing. Hey, this is relaxing, dropping some science to everyone out there makes me feel a little better. Ok, after this I am doing something that does not require a keyboard, promise.

For this exercise let's pretend that you have your network installed as outlined and locked down so user permissions are effective across all your clients. Everything is working great until an update for ActiveX comes down, and now your permissions to protect your clients are preventing them from updating. They are broke as the ActiveX content needed does not have the permission to update.

You could go to every machine, one at a time, and use runas to update IE, or, you could turn once again to our patron saints at sysinternals.

The name of the application you are going to need is called psexec , and it does something really cool.

This little guy will go out across the wire, and allow you to spawn commands remotely as Admin on remote machines. That means you can do all kinds of interesting and worth while things.

Here is a basic command for killing a process remotely, guess which process:

psexec \\10.10.10.11 -u victimdomain\domainadmin -p yourpassword "c:\windows\system32\taskkill" /F /IM iexplor*

Thats right, use your domain administrator account to send these commands, and kill IE. It's a hoot to sit in the same room and watch their expression as their myspace page continually dies.

In the same vein, you can also launch applications. I would like my users to be able to update their own ActiveX plugin, so I launch a process for them:

psexec \\10.10.6.16 -u victimdomain\domainadmin -p yourpassword -d -i "C:\Program Files\Internet Explorer\IEXPLORE.EXE" -k http://mfr.mlxchange.com/Default.asp

Notice the -k, that opens up IE in kiosk window mode. Why? All I want them to do is update one page, not fill their machine will malware! They have administrative rights on IE at this time, but no way to do anything but that one page.

Ok great, now they have one page open, but they can't do anything else until it is closed, so we need to add a timer to close in automatically:

sleep 300

This executable is not included with XP, so you are going to have to find something similar, or download it. I will place a link here.

Finally, close the IE page after they are done with the first command I showed you.

Now let's tie it all together into something ALIVE!

What if you are on vacation and the update comes down and you want your users to be able to execute these commands automatically? The solution I devised was to use Thunderbird and a plugin that allowed me to execute commands when my filter was triggered for a message. This extension is called mail box alert and can be found here: http://jelte.nlnetlabs.nl/News/index.html

I created a filter for each user, and inside I built a batch file that would execute upon receiving a particular subject, and body to a message. This would launch the bat file, in turn launching IE, then 300 seconds later, killing that window. They are then working and back to normal.

You can also launch the same commands to multiple machines at the same time. For example you can send a command to every machine on your domain. I use this little trick to logoff clients:

psexec \\10.10.10.11,10.10.10.11 -u victimdomain\domainadmin -d -i "shutdown" -t 300 -l -f -c" unexpected maintenance. Please save your work. Login at 1:30pm"

You can use the shutdown window from a domain admin login, but this is oh so much cooler.

I also have nuclear options that will reboot all machines, launch IE in admin on all machines, and other such fun.

Thats it for me, off to the beach.

Let me preface this by saying to do this quick, you are going to need a working nix box already, you could do it on any box as long as your tftp server is working.

• Download images/netboot

• install tftpd-hpa

• copy the files from the netboot folder to the var/lib/tftpboot
• Set your DHCP server with these extra options:

siaddr 10.10.6.14 (address of tftpd)

sname ntinstall(for Unattended, but why not?)

boot_file pxelinux.0

• restart the tftp daemon, pxe boot!

Debian should come up, and install like normal.

Today I am going to be doing some testing on a clients network. They have been updating me on a slowdown they are experiencing during the busy times in the afternoon.

They have about 16 users, running on 23 machines. 2 switches and one router make up the hardware for the lan.

My first thought is that this is just normal traffic, and the lan is struggling to keep up. They have a need to transfer images, not big on their own, but large enough to cause a slowdown when every one is sending at the same time.

My focus is, of course, small business environments. I have been associated with larger interests on and off, but small business is where I like to be. Why? I think it has something to do with white boards, my uncharacteristic fear of them. I know that when there is a white board, there will most likely be strategic planning. And where there is strategic planning, there will be dough nuts. I need no more dough nuts, I feel like a dough nut as it is.

My clients trust my judgment and decision process. When I diagnose a problem, I consider the situation, then I take action to gather intelligence. When I build a network, I consider the requirements, then I take action. Many of my buddies in the suit and tie world worry about my exploits, and cajole me for having such bravado with holy and intrinsically unknown devices. They worry about network downtime and data loss, and the justification to their superiors if they can't back it up with a nifty flow chart. A flow chart is not going to save you, not even close. I would give a room full of flow charts, and graphs, and technical manuals, for one guy that new what the hell he was doing, and was not afraid to get his hands dirty.

I remember when I started working with SGI, and they rolled in 6 Onyx machines. If you have not seen an SGI machine like the Onyx, well they were big. Refrigerator big. Multiple risc processors, scsi, and millons of dollars that were not mine. I was afraid, made feeble by the huge machine. The representative that was helping to install all the equipment turned to me and said “be not afraid, it is just a computer”, and so it was. Once I got to the console of the box, and all the normal commands I was familiar with worked, my fears eased. IRIX was kind to me, and made me realize that I could handle pretty much whatever was tossed my way. After all, its only a computer.

Today the plan is to go in, and wait until the bell. I will be preparing a rather large file that I will trigger the machines on the network to download from the server as I monitor the traffic. Using tcpdump I will capture the packets that I can and see what I can on the line. I hope to bring this network to its knees, twitching and stuttering so I can watch it all happen. After I get enough data I can go in and prepare some flow routes or some such fix to prioritize the traffic to better suit my client. Could I do this at my last fortune 500 employer? Nope, not a chance. I would probably have to submit a work order, and more paperwork, and a slash dot 7 in triplicate, and wait a week, well you get the idea. So off I go into the unknown, man, I love my job.