Starting Up the Cellular Modem at Boot Time

As usual, getting the cellular modem to startup at boot time was not as easy as expected.  The behavior I wanted was simple: When windows boots up, I want the modem to turn on and provide an internet connection.  The trick here is that the cellular modem acts like a modem — meaning that it doesn’t provide instant connectivity like a WiFi card does.  You have to actually “dial in” manually.  Usually, this is done using the little application that comes with the unit, but that requires a human to launch it.  I need connectivity to happen automatically and without requiring someone to log onto the machine.

I started by looking at using the various Startup folders that Windows XP has available, but couldn’t get any of them to work before someone had logged on.  In my research, I found out that windows has a command line utility called RasDial that allows you to automate logging on through a modem.  Once I found that, I figured the simplest approach would simply be to have my NT Services just launch this executable with the proper command line arguments as part of their startup.  Of course, I hit a few issues in doing this:

  • Since the robot is a really underpowered PC, I had to set the timeout on executing that command to 6 minutes or I would constantly get timeouts launching the EXE
  • Because that whole process took so long, the Service Control Manager (or SCM, the part of Windows that manages NT Services) would think the service was hung.  I ended up rewriting my service to do most of its work on another thread.
  • I have two services, but didn’t want both to have to do these machinations.  So, I tried just doing it in one, and then telling windows that the other service depends on the first service in the startup order.  I could never get that to work.  The dependent service would simply never start.  So, I ended up just having each reduntantly try to startup the modem in case they get loaded in a different order over time.

So now it all works:  I am using a cellular modem, using the .Net ServiceBus relay to communicate to the robot, and this all starts up automatically at boot time without requiring someone to logon.  However, it is very slow.  Takes about 6 minutes to boot the robot now between the cost of normal windows boot, starting up the modem, and logging into the .Net ServiceBus.  Pretty awful, but it is OK for my purposes for now.

Here’s the .Net code I’m using to startup the modem:

public static void EnsureNetwork()
{
	ProcessStartInfo info = new ProcessStartInfo("RasDial", "\"ATT Wireless\"");
	info.UseShellExecute = false;
	info.RedirectStandardOutput = true;
	Process process = Process.Start(info);

	if (!process.WaitForExit(360000))
	{
		throw new ApplicationException(string.Format("RasDial timed out:{0}", process.StandardOutput.ReadToEnd()));
	}
	else if (process.ExitCode != 0)
	{
		throw new ApplicationException(string.Format("RasDial returned exit code {0}: {1}", process.ExitCode, process.StandardOutput.ReadToEnd()));
	}
}
Posted in .Uncategorized., Networking, Software | Leave a comment

Microsoft AppFabric Service Bus running

I went forward with the plan to try out the Microsoft Azure AppFabric ServiceBus (quite a mouthful!).  Registering for an account was easy, and they have a new pricing plan that is free if you’re only using a small number of connections, which is all we’ll ever use.  I’ve posted the modified ServiceBus sample project that I ended up creating to figure out how to create and call a ServiceBus endpoint without using a config file, and which sets up streaming on the ServiceBus contract.  This last part was needed because I still plan to have part of the robot contract continuously stream images and wanted to ensure it worked.  I’m still working through some issues on that, however.  Right now, the ServiceBus is buffering up the stream I’m returning and doesn’t actually send until the streaming ends.  I’m sure there is some configuration I still need to modify to get this to work.  Given that all the rest of the operations on my service endpoint work fine, however, I’m going to keep plowing forward.  For the moment, the lack of streaming isn’t blocking me.
The only real issue I had was that the ServiceBus endpoint insisted on listening on Port 80, which was odd.  The whole point of the ServiceBus is to be a relay so that you don’t have to listen or open up any special ports.  It is supposed to just open an outgoing http connection to a service in the cloud and then have messages route through the cloud via this outgoing connection.  The fact that it also listened on Port 80 was very confusing.  After working with Clemens Vasters who is currently on the ServiceBus team, it turns out it was really a funky feature of WCF.  If you create a host in WCF by passing one more base addresses like this:

host =

new WebServiceHost(typeof(ImageService), httpsAddress);

Then a couple of default Behaviors (Behaviors are special extensions in WCF) will try to listen on Port 80 automatically: ServiceDebugBehavior and ServiceMetadataBehavior.  So you need to either remove those behaviors manually or remove the base addresses if you care.

Once I fixed that, I modified the robot code to use the ServiceBus, hooked the robot up to use the cellular modem and the new ServiceBus connectivity worked like a champ!  I was able to connect to and control the robot even though it was going through a cell modem that didn’t allow incoming connections.

Now, I can go back to figuring out how to get the robot to start the cellular modem connection when it boots (and also figure out why streaming doesn’t work for me on the ServiceBus).
Posted in .Uncategorized., Networking, Software | Leave a comment

Incoming tcp connection not supported by AT&T using cellular modem

OK, my first issue isn’t how to get the connection to start at startup, my first issue is really: how to get inbound TCP connections to work over a cellular modem.  My initial research indicates that the cell phone companies block incoming TCP connections unless you get a special account.  Here’s one example.  So, it looks like if I’m going to be able to connect to the robot over a cellular modem, I’ll need to use some kind of relay that has the robot connect to a server, the client connect to the same server, and have messages route through it like a bridge.  The .Net Service Bus is one such example.  Early on in my experimentation I used the relay for the robot, but I stopped when they started charging for it.  I’ll have to look into this again and see if it is a viable option

Posted in .Uncategorized., Networking, Software | Leave a comment

Cellular Modem

So far we’ve been using the Linksys WUSB100 Wireless N WiFi (g/b/a) USB Adapter as our wireless connectivity solution for the robot, but only for testing purposes.  When the robot is in the field, WiFi won’t be available, our plan is to use a cellular modem for connectivity.  I ordered the Zoom 4596 3G Mobile Broadband Modem and tried it out.  Was a very seamless install on both my Win7 desktop machine and the robot itself (which is running Windows XP).  The modem acts like a hard drive when you first plug it in and the hard drive contains the driver installation program.  You just run that, do a reboot and you’re all installed.  Great experience!  To try it out, I simply pulled the SIM card out of my phone (which has a data plan) and plugged it into the cellular modem.  I had to adjust the connection information for the modem to make it work with AT&T however.  Here’s the connection information I used:

APN: wap.cingular
Username: WAP@CINGULARGPRS.COM
Password: CINGULAR1

Once I set that up, it worked like a champ.  Very fast too!

The only issue is that it doesn’t appear that I can just leave the modem plugged in and have the machine boot up with the cellular modem acting as the network connection by default. There is a manual connection step required to “Dial In”.  I’ll have to figure out a way to automate that step.  Hmmm.

Posted in .Uncategorized., Hardware, Video | Leave a comment

Video causes CPU to be at 100% on the server

Now that I had a streaming interface that used WPF I was getting into the final stages of my design.  I started to notice a new issue though: When I turned on the video, the CPU would shoot to 100% and stay there.  This obviously affected the responsiveness of the robot immensely.

I ran a profiler on the code and didn’t get great data (I didn’t record what the issue was and don’t remember).  So I started creating some theories and trying things out.  One of the facts that led me down some of these paths was that I was pretty sure I hadn’t seen this CPU time issue in my previous code so I suspected it was something to do with my new WPF code that was the culprit.

Theory #1: Maybe I am creating and throwing away so many big objects that the .Net Garbage Collector is having a hard time keeping up and that is pegging the CPU?  I did a bunch of work to cut down object allocations and at this point all memory outside of a few strings is completely reused.  It is a better design and much more efficient.  I’m sure it helped something but it didn’t address the CPU problem.

Theory #2: Bad image processing architecture.  The basic architecture I used for processing the images used the DirectX DirectShow APIs to get images.  This is set up as a callback: you give it an interface and it calls you over and over with a new image from the camera, it can do this many, many times a second.  I would make a copy of the image, put it into a queue and then return from the callback.  Another thread would grab an image off of the queue and serialize it out to the network when a client was requesting a video feed.  My theory was that the callback thread was able to run in a continuous loop pushing images into the queue and this what was using up all the CPU.  So, I tried an approach where the callback thread would return immediately if an image wasn’t needed yet by the thread serializing images to the network.  Didn’t solve the problem either.

Theory #3: After some testing I figured out that the CPU was being used outside my code after the above fix and it appeared that, even though I was returning the callback thread immediately, the DirectShow API was using the CPU to grab the next image and feed it to me, only to have me throw it away.  It also turned out that the DirectShow API would block until the callback returned.  Thus, my final theory was that, if I blocked the callback thread until I wanted another image and I throttled the number of images I was requesting, I could keep the CPU usage down.  Turned out this worked.  So now, the Image Service allows you to set not only the image size, but also the framerate (1 frame per second seems to work well under current conditions and keeps the CPU usage around 50% or so).

Posted in .Uncategorized., Software, Video | Leave a comment

Using GDI+ or WPF on the Server for Serving Images

Next issue I began hitting intermittently as I was performance testing my image service was that everyone once in a while I would get an exception from GDI+ (a graphics library that is part of the .Net Framework): “Object is currently in use elsewhere”.  After doing some research on this it appears as though GDI+ expects many graphics operations against objects like bitmaps to occur on the same thread that created the bitmap.  I also found some scary warnings about not using GDI+ code on the server.  Finally, I realized that as part of my performance testing I’d like to be able to do various kinds of bitmap scaling so we can send less information over the wire.  All of this lead me to changing my code to use WPF (Windows Presentation Foundation, another .Net Framework library) on the server instead of GDI+.  I did find some warnings against using WPF on the server as well, but they seemed less strong and I organized my code to keep one thread operating against an object as well, so I thought I’d give it a try.

Overall, I found it to be a relatively easy change, outlined here.  There were no real perf wins, but I did get rid of that exception and the code is now set up to support any size of bitmap you want.  If the network gets really slow, you can scale down the graphics appropriately.

Posted in .Uncategorized., Software, Video | Leave a comment

Video Performance

Our biggest challenge so far on the “controlling the robot” side has been getting the video feed from the robot to work well.  The basic problem is that images in general are big and this means they are slow to transmit.  This causes latency and means that the picture you see on the screen when controlling the robot has a big delay from what is actually happening in the real world.  Now, this isn’t a fatal issue for us because our whole command system does not rely on real-time video.  We’ve designed the robot from the beginning around the fact that the GSM modem will be slow and the connection will drop often.  The way you command the robot is to tell it “Move forward for 5 seconds” and then wait to see what happens.  So, it isn’t like the robot will run off the stairs while you are waiting for the video to get updated.  However, it is quite annoying to wait for the video to catch up with your last command, so I wanted to see how small I could get the latency to be.

Our first assumption is that the connection is going to break often and be slow, so we have used an approach that simply sends a picture one at a time.  We are assuming that using video compression that requires a sequence of pictures in order won’t work reliably.  Thus, all the approaches we’re trying are simply finding ways to transmit a series of JPGs, one after another, with the smallest latency possible.

Initial Approach

We started simple: The client would ask the robot for a picture, the robot would send it back.  Request, response.  It worked fine, but the latency was, of course, bad.  We tried various encodings and found that JPG was the smallest so we stuck with that.  That sped things up a bit.  I set up the user interface to continuously poll for images so you didn’t have to request a new one manually each time you wanted to see what the robot saw and that sped things up further.  However, this polling really took a tool on how responsive the robot was to commands. 

You see, the service we have running on the robot is single threaded, meaning it will only process one request at a time.  This is because the .Net Serializer is a single physical board that can only take one command at a time.  When you asked for an image, that request went to the same service that processes .Net Serializer commands and thus stopped you from telling the robot what to do.  This really was unusable.  Fortunately, the video and the .Net Serializer commands didn’t need to be on the same service since video came from the camera and commands went to the .Net Serializer.  They could run commands in parallel.  To fix the issue, we created a new video service so now we could send commands in parallel.  This improved command performance immensely.  Latency was still not great, though.

Streaming Pictures

The next approach I took was to get rid of some of the latency inherent in the Request/Response pattern.  Why not just do one request and have the robot send response, response, response — i.e. why not have the robot return a stream of pictures one after the other instead of having the client have to ask each time.  This would get rid of some of the latency since the robot would just send images as fast as possible.  Windows Communication Foundation makes this easy with their streaming support.  Once this was in place, the latency was getting better, but it still wasn’t great.  I tried a few things to see if I could improve performance here:

  • Adjusting the TCP level buffer size to see if I could get transmits to happen more quickly (and not get batched on the server).  Didn’t help.
  • I tried getting WCF out of the picture completely by building my own socket based service (this post was very helpful on this front).  Turns out Microsoft did a good job tuning their service and I wasn’t able to beat the perf of WCF in the time I had available
    • 61.5 mSec using WCF with a worst latency of -122 and best -18
    • 95.2 mSec using my custom TcpServer with a worst latency of -139 and best -66

So, I stuck with the WCF service as my approach for now.  My next angle was to quick focusing on the networking side and see if there were optimizations I could do in the service that could cut down the time it takes to get the image from the camera to the network.  See my next post for more details

Posted in .Uncategorized., Software, Video | Leave a comment