I started out this morning trying to gleam a little behind a network issue I was having with Asterisk. It did not take me long however to realize the problem was deeper and more sinister then I could have imagined.

I started by installing TCPdump on my Asterisk box, and Ethereal on my windows machine. I then ran the dump, and used my windows box to analyze the results. It was shocking to say the least.

I called my brother and tried to have a normal conversation, the phone kept dropping out. I was capturing the data however, and this is what I saw:

Not pretty, two glaring indications that I have a problem, right in front of me. The data indicated to me that for some reason, my server was dropping packets. Now the question is, was it just the Asterisk server, or was it the Xen box itself causing this issue. I decided to pursue the later.

I made a call from my cell to the Asterisk box while pinging the box from my client. Low and behold, this is what I saw:

Ouch, you can see the Asterisk box die, and then come back to talk a little more. Yikes. Now I took that train of thought a little further and made a call to my Asterisk box, while pinging my Apache server that runs on the Xen box, what I saw made me a little sick.....:

Ugh, so it seems the hickup is in the Xen box, more then likely stemming from the drive failure and the subsequent rebuild. So now I am going to have to rebuild that server, or build another and do I live transfer of the data. Neither of these options do I like. I do have a brand new server with a processor that sports the built-in hypervisor. So it looks like I will have to rebuild my server before tackling the network. Man, what a long day this is going to be.

UPDATE:

After a reboot and a careful look at the logs, it seems the other drive on that server, the older one, also went bad. Needless to say, I will not be buying that type of drive again.Mdadm did its job, and sent me an email indicating a failure, somehow I overlooked it. I rebooted the machine, it's running fine now. It will have to do untill maxtor sends me a replacement for the drive from Newegg.

Network problems arise from complexity. That's my story and I'm sticking to it.

I have been doing the best I can to expand my network lately. It has been tough, but I have managed to build a nice network in my home that can do close to everything I want. The problem now seems to be it does too much. I know this because of the daily routine I am having to perform to keep everything working the way I like it. That is bad. I find every so often I need to explode into a massive undertaking just to stay on top of a particular problem, and this seems to be one of those times.

The Story and structure:

Like I said my network is complex, and it is. I run cutting, no bleeding edge code, that I barely have time to implement, much less become familiar with all the details. This of course can lead to all sort of weird situations that normally do not exist. I am, in fact, the testing ground for all the technology I deploy. The weirdness lately seems to be a drop in packets that occurs on semi-random intervals across the network. This means when I am watching a movie from my server, or talking on the phone, you got it, everything pauses for a few seconds. This makes for some interesting phone conversations to say the least.

I used Xen to consolidate my servers a long time ago, well 6 months or so, with great results. I built a dual 2.0 Xeon with 4 gigs of ram and a RAID 1 with 250gig of SATA goodness. Its a really good board, an IBM server board, so It has all those nifty server features like a separate back plain that does all the monitoring. With Xen running on top of Debian, I made several machines inside that run my day to day. I run Apache, Asterisk, Samba, Postfix, and Mysql, all on separate virtual machines. When I first got everything working, it flew. I tried to put as much of a load on it as I could, I couldn't slow it down.

This was great and worked until a month ago, when I had a drive failure. No problem right? I had a RAID mirror, so theoretically I should be able to partition another drive, pop it in, and use mdam to add it back to the RAID. Well I pulled the bad drive and ran with just one, and the performance came back, but not to what it was. Then I ordered two new drives from Newegg. When the new drives arrived it turns out that one of them was bad. I did replace the missing RAID mirror with a new drive and rebuilt the partition. I had been running everything the 'old' way off of another server I had until I new that this machine was back up and running correctly. I transferred everything back to the Xen machine. It ran ok for a week, then the problems began. Of course the most noticable is the VOIP lag. So now I must run some diagnostics to determine what exactly is going on.

Not being the kind of person that was happy with a few problems, I decided to pile them on a little higher to make things more interesting. I got a letter from Verizon telling me about Fios being availiable, and I thought, hey why not. I liked the pitch about the speed, and I new I was lucky to be where I was so I could have it first. They installed it without issue, and I integrated the gateway into my network. I did this by ditching the Dlink they had provided and swaping it for a WRT54G by cloning the MAC address. Worked the first time. I then made the routers static, and setup a flat network with two gateways. Hmmmm interesting, but will it work? The answer is yes, and no. I did not want to ditch Brighthouse just yet, not until the 15mbs link had made me forget all about my 7mbs. This of course has not happened. The first problem I had is the Gateway, how does the network handle multiple gateways in a route? Simple, it picks the one that is least congested. Great for speed, bad for communication. This means that if I am trying to run, lets say Apache, off of a ethernet alias like eth0:0 and giving it a new IP address with the new gateway, when it talks back to the client it will pick the gateway that it likes the best. Only a bit of the traffic you are getting in, will go back out on the right gateway. Of course there are ways around this, and I have found a few, like iproute. It seems though I am to a point where it would be wise to begin segmenting my network as to allow for easier diagnosis, as well as reducing broadcast traffic. Segmenting will allow me to use QOS to my full advantage, as I will be able to add the subnet mask and pick the level of traffic I want.

Currently my biggest issue is my VOIP. Asterisk is my PBX and I have several lines coming in. My service provider is Telasip, and has been working well for quite some time. Telasip is tantamount to a majical school, a mystery school that seems to expect you to 'know' before doing. It took me a while to figure out some of the deeper secrets, but I have, like creating 'any' number on caller ID. It is troublesome trying to learn an application, without access to the log files of the server you are connecting to. With a humble heart I have wrote to Telasip asking for confirmation of a problem I was experiencing, with the only indication of a problem being “request denied”, or “wrong password”. They wrote back at first, letting me know 'nibbles' of information, then not writing back altogether. I took from this that my questions had become futile, and that in fact I needed to go back to 'school' and learn enough to ask the proper questions. So now I am ready to school myself in the ways of Asterisk, and this network rebuild is going to be one of the tools I use to do so.

The tools:

I am going to begin this adventure with a set of tools to monitor the traffic on my lan to determine exactly what is going on and how to diagnose problems like this in the future.

Ethereal – is a nice analyzer

TCPdump – a easy way to get packets

Iptables to push data to a log file.

Sipp – an Asterisk traffic/call generator and tester.

You have to remember that the traffic on a switch is well, 'switched'. This means that just putting your ethernet device is promiscuous mode is not enough to see all the traffic on your lan. To do this you need an angle, either a hardware device setup for this, or a mung 'hack' like Man in the middle ARP poisoning to dump all the traffic to one location, so you can see what is going on.

The Plan:

I believe it was Nietzsche that said 'A man without a plan is not a man' .

First thing I want to do before I do anything else is to figure out how Sipp works.

I have it installed on my Asterisk box, but when I try to generate calls, it gives me SIP 404 not found errors. Worse, it seems to be trying to connect to my outside Trunk, blasting my provider with useless calls that can never be connected. The more powerfull the tool, the greater the risk of damage.So now I am going to disable my registered connections to the outside and blast away until I can get a decent load test going and see what my Asterisk server is doing.

The next action will be to remove the FIOS all together and see if I can't get my performance back the way it was before the addition. If I can, I will begin segmenting this network and classifying data with QOS on the router. I hope to be able to watch this all in real time with my tools in place so I can determine where the problem is coming from, and how to correct it. Hopefully, I will be able to leave all of these tools in place so when another problem arises, I will be able to deal with it in a timely manner. I will be posting details of my adventure as I have time.

I just got back from staying in a condo on the beach in St. Augustine, it was great. We did all the normal stuff with the family, the tourist stuff. I think I am actually getting old enough to enjoy it, egads. The history aspect of St. Augustine is staggering, but I wonder how much has been “restored” just for the tourist industry, and how much is an extension of the populace through time. I was also looking at the walls of the inside of the fort and noticed carvings of graffiti that dated back to the 1800's. They have all kinds of rules about not carving, but I was thrilled to see the vandals of earlier generations handwriting on the walls. It only served to peek my interest and force me to wonder if generations after me will be discovering artifacts from my generation, or will our name be wiped away suffering under the condemnation of the 'now'. Are we not as important because we are still alive? Mabye, but not as interesting. In a hundred years or so, my signature will be very interesting, in a thousand, more so. Here are some pics of our journey. I would highly recommend it for a weekend. The Oasis restaurant, the Mill top bar and grill, and free parking downtown in the new parking garage this month are all good reasons to go. If you are coming from somewhere that has hostels, this area is one of the few in the US I have see with hostels, so cheap and local flavor will not be something unobtainable.

If you have verizon wireless and you are getting dinged with overages, well this could be the solution for you.

http://verizon-minutes-used.blogspot.com/

Set this up correctly and it will display how many minutes you have used at the bottom of your browser. Now you can talk with impunity!

I have to go, I have 100 minutes left, I want to make sure I use them all.

I've talked in previous posts about some of the structure of a Mimic IT network. I will define exactly what that structure is, what makes it different from the current network you are using, and most importantly, everything you will need to replicate it for deployment in your business.

The hardware:

Nothing fancy here, a WRT54G router flashed with DD-WRT firmware for extra functionality. The server is a 1u server that uses name-brand components like Asus, and Corsair. I usually run Cat 5e cable, but any cable will do. A little note about wireless links to your server from clients: try to avoid it. Although wireless is a great technology, the actual implementation is wrought with perils. Needless to say, my philosophy when deploying is to remove the doubt associated with inherent problems, and go with something that is stable out of the box. As for the clients, any cheap machine will do just fine. Make sure you keep to the specifications of your software, but for the most part a 1 GIG machine with 512 megs of ram should be able to handle everything you throw at it. This is especially true if you are just running normal office applications. When a client asks me what kind of new machine they should by, I point them to Dell. Why? A great warranty, and they are very easy to service. Plus, the cheaper machines are they way they hook most people in, so you can get a new client with monitor for under $400 USD.

The install itself should be in one place. This means the router, switch, cable modem and server, should all be physically located in one place, like a closet. Why a closet? Its out of the way, and easy to cool. This is most important. If you want enterprise class results, you need to try to emulate the environment that they have to give you those same results. This means a cool, dry , safe, out of the way area. When I mean cool, I mean as cool as physically possible. The cooler you make your server and hardware, the more efficient it's going to be. If the server is happy, then everyone is going to be happy with the performance. Another reason to isolate the server is the noise, 60k rpm fans in unison sound like a little jet taking off, bad for concentration. I would suggest scheduling a new line drop for your electrical with a good solid ground. I would also suggest a new AC drop in the closet. Once you have those things in place, and a good UPS, your ready to begin installing the hardware. As far as the clients go, the same mantra applies. A cool and dry environment is going to make your PCs happy. Conditioning the power with a UPS for each machine will also minimize problems on the client side. A client can bring down a network, so make sure you pay attention to that side as well.

The Software:

Here is where we really start to deviate from a normal small business network. If you are using a profile login with a windows domain, then you are already half way there. What we will attempt to do is lock down the clients to user-mode only, and adjust the registry to allow the software to work correctly. This is more of an art then a science at times, and I have outlined the process in previous posts. When we are done, your users will be able to use any machine in their office as if it is thier own.

Usually my server is ready to go before I walk in the door. This means a standard Debian install with a few extras. I install Unattended to facilitate a network reload. If you are rebuilding an existing network, and would like to test a client install, I would suggest Vmware. Vmware server is free, and it installs fairly easilly on Debian. Once there, you can use the PXE boot function of a Virtual Machine to test your client images before you begin to install them on a physical machine. I use Vmware to build up a template of what the basic client will look and feel like. I do have some standard software that I use, along with any proprietary software needed to run the business. I usually just install OpenOffice and Thunderbird. If your clients are insistent on using Microsoft products, and they use Outlook Express, then you need to persuade them ever so gently to use standard Outlook. Why? It seems that Express will not let us move the mail folder to a location on the network, it must be in the profile. This is bad for us as the profile size is indicative of the login-logoff speed of the client. Say no to Express if at all possible.

Propagating the old client data to the server is going to be your biggest challenge. You only get one shot before you slick the machine, so take your time.

You need to have the server in place and functioning on the network. Samba needs to be up and running to drop data to the server. I would create a share called 'shared' and place folders named after your users in which you will store their data. Tell your users to take all of their documents and place them under the folder that you have mounted for them. I would of course review what they have and remind them if you find any data they have not moved. The first thing I usually do is make sure the email is pulled over. Once you install Thunderbird, it will import the mail into its folder. From there you can navigate to :

C:\Documents and Settings\Username\Application Data\Mozilla\Thunderbird\Mail

I copy the entire folder to //server/shared/username. With that done, you have all the data needed for the install. You can go ahead and set up a domain and propagate the user structure to the server. This just means copying the data to the appropriate places so when a user logs in the domain they get all the information in the right places, including their email and documents. Please see my post on setting up clients for some of the details.

Now that you have all the data and you have used my posts to setup the client with locked down permissions and the appropriate software, you can log in. Use the Vmware image that you have created to see if it works with each user, and their email is there. Once you are satisfied that everybody has all their data, you can begin the client reload. Unattended should be ready to go, all you need to do is make sure the DHCP server on the router contains the necessary information to point it to your server for tftp boots. You will also need to load a tftpd that can be used with PXE, I would suggest tftpd-hpa, as it is availiable from the repository via APT. You will have to write some scripts for Unattended to load all of your software. This could take a while to test, as you have to wait for an install to see if the process works completely. There is probably a way to test just the scripts that install the software, I have not found it. If you know, please leave a comment, I would love to hear it. If you set up Unattended correctly, you should be able to specify the Product Key for Windows XP with the Mac address of the client. This can either be accomplished via a Database like Mysql, or a file, that you can add to. You can also specify the name of each machine along with the domain. You can set up Samba to allow machines to join the domain without a password. This would be good as the Domain Admin password is stored in clear text in the setup file for Unattended. When I get some time I will at least write a HASH storage of some sort for this file, but hey, it works, so I just manually type the password in, no problem.

I would create a document for the owner that would allow them to replicate the process when they buy a new machine. Each step should be documented from start to finish. Store this on the server under shared/docs so they have access to it later. If you have a big shop, recruit a couple of workers, familiarize them with the process, and begin the slicking and reloading.

If you are going to give your users access through the VPN to their machines, I would suggest using the Static feature of the DHCP server. Just write the mac address down along with the hostname and place it in the required fields of the router. Now when your client boots they will have the same IP. You can also change the IP of the client remotely, I know you like that.

Make sure if you mix static and dynamic IP's that you seperate the two effectively. If you pick a static that will be assigned dynamically, then UDHCPD will die, and your clients will go down as they try to renew their IP's. Not good.

I hope this post familiarizes you with some of the details, and gives a rough outline on what I do to install a network. Of course the devil is in the details, so I will be going into more specifics in later posts.

Run windows with no virus or malware scanners, wow, brave. Not if you use the methods outlined here. See without the access needed, bad code is prevented from doing ANYTHING to your system. No need to scan for something on the client if:

• You can catch it on the server, or over the line

• It can't do anything once it gets there

See how that works? But hey, don't take my word for it, listen to the guys at HP.

http://www.hpl.hp.com/techreports/2004/HPL-2004-221.pdf

How do you prevent a system wide malware infection while surfing with windows? Simple, use a restricted user account. But wait, my programs don't work in a restricted user account! No problem says Microsoft, Vista will take care of everything.

Looks like the Big boys are going to muscle in on my angle. Guess its time to kick it up a notch.
http://www.microsoft.com/technet/windowsvista/evaluate/feat/secfeat.mspx

I just found this site while surfing, the tools look awesome.

http://nonadmin.editme.com/UsefulTools

These tools look great for client administration on a Mimic IT network! Kudos to the devs.

Today's chat is all about Windows Roaming Profiles.Profiles are really the life blood of the kind of network we are building here. Your user profiles will contain all the data that you will need to encapsulate a user. Once your user has a profile, they are on the server, and can be manipulated rather easily. You can do a number of things to a user on your server via the profile. One of which is change their registry. The meat and potatoes of a Samba domain management scheme like this requires some tweaks to the user and computer registry. With the tools provided, its not a problem.

The registry file that is loaded in the windows registry under HKCU is located in the root of the profile, and its called : ntuser.dat

Now ntuser.dat is easy to edit. I like to mount the profile drive in samba and “Load Hive” the registry's from. To do this just run

• start>run>regedt32 and under File click “Load Hive”.

Make sure you have selected HKLM on the left for this to work. Once it asks for a key, type “default” and hit enter.

Now that users keys will be loaded as “default” under HKLM on your regedt32, you can affect them and then “Unload Hive” from the same menu.

Thats a great way to edit the registry remotely, but you can also edit the registry when the user is logged unto a machine. Then when the user logs off the registry will be saved back to the server under their profile.

This is great for little tweaks, but what about a wide scale registry change? That's where ntconfig.POL comes in.

Windows has a great way of doing things with clients called Active Directory. If we had a Windows server I would explain all the benefits of Active Directory, but we don't, so no crying.

Instead, embrace your Linux Samba box and dive in as we use a work around to get a lot of Active Directory style love.

First the tools.

In order to create an ntconfig.POL you are going to need :

• poledit.exe

You can download this from Microsoft's site. Just do a search for Orktools and it should come up.

I believe Orktools is a package for maintaining the ntconfig.POL with Ms Office. It works for us just fine.

Next you are going to need a sample policy to go by. This is called the ADM file, and is a template:

Download Sample Custom ADM

This was made by Nathan Lightfoot, I retrieved it from the Novell site, it works great for our purposes.

http://www.novell.com/coolsolutions/tools/14387.html

Here's the low down. Take the .ADM file and use that as a template to create the ntconfig.pol.

Once you have that created, modify the options to suit your needs.

Launch the poledit.exe and select options from the top then template, and select the .ADM file.

poledit.exe>options>templates>select your .ADM file

Now you should see two icons appear, one is for the Local Machine, the other is for the user. They both have some really cool features that we are going to use to customize the way these clients play with the network.

Go ahead and click through the options of Default Computer. Notice all the ways you can customize these clients. You can adjust Automatic Updates, Media Player Updates, shut down the Firewall, and on and on. Its great. Oh on a side note, if you are new to this business, for the love of god stop installing and running firewalls on client machines. Every time I have to disable multiple firewalls on a client a little piece of me dies inside. Don't you know that software firewalls step in between your TCP/IP stack and the network? Its like putting a traffic cop on the freeway. There is no need to have a firewall on your client. If you are that freaked out about being on the net and need TIA (total information awareness), then get a decent router that logs your connections.I could rant for a while on the futility of software firewalls on the client, but I must digress.

Lets focus now on the other icon Default User. Notice Folder Redirection. If you open that up there are a few folders you can redirect. This is very important. When you make the switch to a profile base system, your users will still want to treat the machine like the dirty scamp that they have been used to. This means crap upon crap on the Desktop, and loads of more crap in My Documents. What are we to do?

Luckly, when Samba makes a profile, it lets you create a home directory for the user. This is on the server, by default I believe its the users home directory. I move it to someplace a little easier like under my winstore/user/username dir. If you created a profile directory for your users, great. Now create a home directory for them right beside the profile dir . In that directory create two more dirs:

mydocuments

desktop

Change the folder option:

Default User>Folder Redirection>Custom Desktop Folder>//YOURSERVER/user/%username%/desktop

Thats it. Notice the environment variable %username%.

To see if your folder redirect in the registry its in HKCU\Software\Microsoft\Windows\Current Version\Explorer\Shell Folders

Now do the same with My Documents. Save the ntconfig.POL file to your Samba netlogon share as defined in the smb.conf. I place mine usually in /home/netlogon/ntconfig.POL , along with my logon.cmd.

logon.cmd is a script that defines what drives are mapped at logon along with some other stuff, like setting the clock, check it out.

Now you should be able to log into the user and have all the data on the desktop and in My Documents stay and be accessed on the server, rather then having to load each time with the users profile. It really keeps the load times down, and the users happy.

Come back again as I talk about setting up Thunderbird in this environment. I will also begin to touch on my philosophy and method of computer security.

Need some more info, are you that geeked out? Welcome to the club bretheren.

http://www.petri.co.il/understanding_administrative_templates_in_gpo.htm

Wow, what can I say, but Microsoft got a great deal. I guess I feel vindicated in some weird way. These tools are what lead me to think about custom security for Windows machines in the first place. Regmon showed me it was possible to run every app you could think of as a non-priv USER. Now only if developers could get their heads around writing apps for non-priv USER without modification, that would be even better. Don't get this confused with User-mode and Kernel mode, this has more to do with windows ACL and being an Administrator then anything else. Programs that make you run as Administrator are evil.