Coding Notes

On the hunt

2010-12-07T08:27:00.001-08:00

It's the second full day of being unemployed. Quite an odd feeling. I feel as if I should be scared, but recent events make it seem like the future is full of possibilities - many of which may be better than the recent past - and that was pretty damn good. I'm very conflicted - we're conditioned to think that being unemployed shouldn't feel like this.

In my free time (which isn't as considerable as I had hoped), I'm working as hard as I ever had. And all of it is in Rails, which also fills me with joy.

Of course, it's not all roses! I'm learning how to write extensions for Spree, and I've just encountered my third "wrong turn". While it's frustrating, each correction results in significantly less code. I suppose that means I'm on the right path!

Here's a sample of my schedule for the next couple of days:
- Today (tuesday): Should hear back on whether I'll get a second interview with a primarily .net shop
- Tomorrow: Phone interview for 3 month contract position. Possible call with HR department of large open-sourcish company (their HR guy contacted me - never burn bridges).
- Thursday: New office space! It's only a short term deal, but it's free for now!
- Early next week: If the phone interview goes well tomorrow, I'll do a conference call with their developers - one of which I interviewed while at my last job - the recruiter said he was thrilled that they might be able to hire me (never, ever burn bridges).
- Later next week: Interview with extremely large company. I used to work for the manager of the group I'd be working in - he said yesterday how great it would be to have another developer he knew he could trust to write code correctly (never, ever burn bridges, pee off the side of them or do anything else that might soil your relationship with the other side of the bridge).

Why Javascript Sucks (a.k.a. Fuck you Microsoft)

2010-08-04T07:38:00.000-07:00

Javascript is a very cool language. With many of the features that make Ruby fun to work with, Javascript gets a lot of unwarranted criticism.

But, even with it's cool features, it's a poor choice for writing rich client interfaces. I'd rather use Java, C/C++, QT or even Flash. Thank goodness for GWT, which makes it possible to create Javascript through the use of Java (which, even in this case, provides a bit of "write once, run anywhere").

But again, it's not because Javascript is a bad language; it's because to use Javascript, your code has to work in a browser. And as any programmer who has ever written web code knows, Microsoft has fucked that up for everyone.

Fuck you Microsoft!

Sites vs Applications vs Software

2010-07-16T15:30:00.000-07:00

Just because it's made up of "webpages", not every coding effort is equivalent. I really like this article. Of course, I replace ".Net" with "Java" when I read it, but the concepts are right on!

If you're simply building a website for a client, you can figure out the single best way (in someone's opinion) to accomplish a task or complete a workflow. Our designer loves consistency and simplicity in products we release. And he's got good reasons for saying that.

But when you're trying to provide solutions to evolving and complex problems, you need to build in flexibility. Business needs today will not be the business needs of tomorrow. And the business may need multiple ways of accomplishing the same task.

The "single approach" systems work well with scripting languages. Many times, when a business need changes, you can simply change the way the code works. The tests protect you from breaking other parts of the system, and you end up with a little more (or a little less code) than you had before the refactor.

More complex systems benefit from statically typed languages (and the tools that make them such a pleasure to work with). When a business requirement changes, it often comes with many rules and variables. Often, a simple refactoring is not sufficient to meet the new needs (which is likely some crazy-difficult logic that some VP dreamed up in a scotch-induced haze that will do little to advance the company but will surely drive you to the edge of insanity).

We build "sites", "applications" and "software". Our event registration product is an "application" which helps our clients create "sites" which sell tickets to their events. But, because we have many clients all using the same application (in pretty different ways), and because we want the event registration system to be integrated into other applications, we've built "software" that allows selling tickets or trinkets or anything else. Had we simply built an event registration application, we'd lack the ability to provide the type of fluidity in reporting, analysis and new product creation that a well thought out software platform provides. Trying to write the software in a scripting language would quickly become unwieldy. However, trying to write the applications or sites in Java is slow and cumbersome (with the exception of using GWT for Javascript - gotta love that for a productivity boost).

The worst fail I've ever seen

2010-06-29T06:58:00.000-07:00

If you spend anytime working with computers in an office setting, you've undoubtedly experienced "an outage". This is the dreaded, panic-stricken period of time where something goes wrong. It could be a complete loss of power, despite on-site generators that "were supposed to kick on". It could be a car hitting a telephone poll that knocks out your T1 line. Or, it could be a hard drive failing, taking all your data with it. Some of these are avoidable, and others are not.

But IT is expected to be somewhat proactive. They are expected to have basic monitoring in place to ensure they notice a machine dying before paying customers do. And when the machine dies, they should probably have some plan for bringing it back online. I've seen this done well, and I've seen this go awry (and when I was a sysadmin, was even responsible for a few less than perfect recoveries).

But what happened last week was an epic fail of monumental proportions. Our company continues to support a legacy e-commerce system. It continues to generate revenue for us, but more importantly, it is the platform on which a number of small businesses run. In other words, without this service up and running, our customers are not able to do business. I'd like to think I work at a company that cares for it's customers and wants to see them succeed.

And I understand - shit happens. So, it when the system went down , it was not a surprise. After all, one reason the system is being replaced is because it . . . well, frankly it sucks!

But, this didn't just lead to an outage - the system was down for days. Here's what happened, according to our "IT Director":
[T]he server that holds the database files ran out of disk space. When this happened, it corrupted both the database file, and the transaction log file that's used to recreate activity in the event of the database file becoming corrupted. This left us with no way to recreate the activity on the platform and has left us with no any easy way to get that information back.

No big deal. That's what backups are for, right? Well, here's the explanation from our marketing folks a few days later:
During a reconfiguration of our backup utility (on April 22), another server took over mirroring the backups during the reconfiguration. After the reconfiguration was complete, the backup process was transitioned back to its normal process. As a precaution with the updated system, the backups were left on the temporary server and that is where the recovery came from.

In other words, the backup strategy in place was fucked up and no one noticed. The most recent backup was from 2 months ago. 2 fucking months! Are you kidding me! I've seen a lot of IT disasters that have cost people and companies a lot of money. While this may not have been the most expensive, it certainly was the most careless and avoidable - by an extremely wide margin!

In large enterprises, file systems running out of space can result in firings; there would at least be a lot of post mortem meetings. Downtime of hours (much less days) causes more unemployment. Lack of backups for 2 months would get most IT departments "swept clean". In smaller companies, those responses aren't possible since the "IT department" may be a single person. But it certainly is a sign that something is seriously fucked up and mismanaged.

Since we only have 2 people in our IT department (the IT Director and a sysadmin), no one has been fired. As far as I know, there have been no reprimands or repercussions. There are hints about not having enough staff and already working 9 - 12 hours a day coming from the IT guys. But this isn't about staff or hours, it's about planning and management. The current "IT Director" (said sarcastically) has proven an inability to manage an IT department and maintain systems.

Even if the company isn't going to attempt to fix this, the IT Director should man up and step down. Or at least, take responsibility. The sysadmin has admitted publicly to making mistakes and dropping the ball, but the "IT Director" is just waiting with his head down, hoping the storm will pass.

And, if you think I'm being harsh, let me assure you, I am not. After all, 2 1/2 years ago, the same system had a similar multiday outage, because . . . you guessed it . . . no fucking backups! You can bitch about not having enough people or working too many hours already, but there's no excuse for that!

I've seen a lot of IT fuck ups over the years, and most I can look back on and laugh about. I'm not laughing now!

MapReduce - just another design pattern!

2010-06-03T07:03:00.000-07:00

A co-worker mentioned that we was playing around with MapReduce recently. He told me how "fast" is was and how he could process a huge batch of log files "in real time". And the whole time, there's something in the back of my head telling me that something isn't right about all of this. And it's not just because he's a sysadmin, and not a developer.

Because he's testing on a laptop (with a single CPU no less), he's not getting any benefit from using a MapReduce approach. Sure, he's using a shiny new technology that helps him sound cool and cutting edge. But in reality, the processing his application is performing could be done, probably more efficiently, using a different approach.

MapReduce on a single machine is nothing more than a design pattern! On a single processor, it's not even a good design pattern. You're taking a large data set and breaking it up to be "mapped" by a bunch of workers that are going to end up waiting on each other, only to "reduce" the data into a single collection. There's no advantage over processing the data serially. In fact, the cost of breaking up the input and merging the results is additional overhead.

I suppose the "style" of typical map and reduce code may seem novel. Since these often take a more functional approach, programmers who have only worked with object-oriented code see the simplisity of passing functions-as-arguements a wonderous and magical thing. But this is not a MapReduce specific feature. Again, the same result can be achieved in many other ways.

I'm currently finding many business people (non-programmers) around the office with huge misconceptions about MapReduce. There seems to be a concensus that MapReduce is synonomous with NoSQL. And you can take billions of rows of data, run an ad hoc query, and get results in sub-second times. After all, that's the experience one gets when searching Google. Right?

At least I think I've figured out where these misconceptions are coming from! And I'm pretty sure, as he continues to tinker with his super-speedy log splicer, he's going to report his magnificent performance results. I trust he'll never benchmark his ingeniuos invention against a simple Perl script (since there's no buzz around such an old school approach to parsing log files).

Too bad he doesn't realize his "big data" is laughably small compared to the amounts of data being processed when MapReduce was conceived. The bright spot in all of this is, as long as he continues to use MapReduce for purposes it is not intended for, he won't have to ask management to fund more servers. Maybe it'll also keep him busy and out of the developers' hair.

Learning UI programming all over again

2010-04-02T05:44:00.000-07:00

We're adding a new feature to one of our client applications that is really an application in it's own right. The user interface is often described as "burley", at least from a coding perspective. From the users point of view, this thing is will nothing short of elegantly awesome, for there's a lot of work that goes into convert the vision to reality.

It's interesting working through front end code again. It's something I've haven't done in years, at least as far as hard core end user interaction goes. I'm not talking about putting html and css on the page and responding to asynchronous method calls. I'm talking about really cool stuff that designers come up with when they believe there are no limits to what a user interface can contain.

My last experience with UI work was with Swing. I did a lot of interactive mapping functionality for open source projects, as well as a visual inspection and packaging tool for an employer. I went through the whole experience of discovering (and understanding) the main event thread. I really learned about the MVC and Observer patterns in a way that reading alone can't provide.

While our "burley" feature is implemented in javascript, we've chosen to use GWT. While there is a bit of familiarity between Swing and GWT, they are quite different. We're also not using GWT the way it's intended - instead of building the UI from new Widgets, we're taking an HTML page (provided by the end user) and wrapping Elements from the DOM in custom Widgets. It made us question whether GWT was the right tool for this job, but so far, writing in Java instead of javascripts has proven to be a huge win for us. Object oriented or not, navigating a slew of javascript files or a few monolithic js files, javascript has proven to be fairly non "team-friendly". Rafactoring Java in Eclipse is trivial, making it easy to keep clean and allowing us to spike out new ideas quickly and easily. And don't even get me started on the testability of how this code is shaping up, compared to old javascript that remains largely uncovered.

Regardless of how it's implemented, a UI is a UI, and there are common problems that arise. We initially spiked out a

with a mouseover event that caused a "masking"

to cover the original. On mouseout, we remove the mask. In hind sight, it's obvious that once the mask is positioned over the underlying element, the underlying element will loose mouse focus. Fortunately, none of the developers have Photosensitive epilepsy.

While I love to think our team is good enough to start off on the right path everytime, that was obviously not the case. Besides learning the GWT api, refreshing our knowledge of browser events, and determining how to navigate any dom a user could throw at our code, we ran a lot of spikes that challenged our knowledge and assumptions. Once we determined what we wanted the code to do, we realized the code we'd developed during the spikes was less than optimal. Much of what we had was untestable, tightly bound and just plain messy. Fortunately, all of that can be fixed by following some common patterns and best practices.

Separating concerns (MVC) and using a publisher/subscriber (Observer) approach to events has allowed us to come up with code that's enjoyable to work with, and easy to understand. The learning curve for developers jumping in (and out) of this project is incredibly low. The ability to test logic (while not messing around with GWTTestCase and it's long run times) provides a sense of confidence that browser testing alone can not match.

In the end, it just shows that languages and frameworks are not as important as good coding practices. Clean, simple and tested code that follows proven patterns and practices provides a quality product that can be extended, modified and maintained long into the future. If you start with mimicing what's been shown to work well, you'll have a better chance of ending up with something that serves you (and your customers) well.

Agile Planning

2010-01-20T06:46:00.000-08:00

Agile suggests doing "just what is needed". Sometimes that just doesn't feel right. Sometimes it doesn't even seem possible. When it comes to planning, I'm finding an agile approach to be a bit nerve-racking.

We're getting ready to build some functionality that . . . well . . . scares me. It's basically a specialized WYSIWYG editor in javascript, but that "specialized" functionality is anything but trivial.

The first fear comes from a language requirement which is not coming from the development team. The developers provided a few options that might provide the functionality we need. Another group (that includes a designer, a project manager, a salesperson, etc.,) decided using javascript provides enough benefits to justify the risks. I'm confident we'll be able to make this option work (JQuery to the rescue), but I'm not sure how long it's going to take (JQuery is powerful, but browser inconsistencies are more powerful).

Another fear comes from not having a clear spec. The stakeholders have seen mockups that include features that may be extremely expensive to create. They've shared ideas that are fuzzy at best, and ill conceived at worst. And since there is no clear documentation, there's no way to know what each persons expectations are, and therefore no way to know how much effort is required. How much do we have to build in order to meet the minimum functionality required? How do we break up this huge monster of vision into iterations? Just because we're using Agile approaches doesn't mean we should completely skip planning.

Before we introduce any work to make this feature set a reality, our customer (management) wants to know how long it will take to build. So, without really knowing all the details about what we're supposed to build, we have to tell how many iterations we'll need to build it, so that we can schedule the first iteration to determine what we need first.

What would make everyone happy is to figure out exactly what this editor is supposed to do (full specs). Then, we'd break down the functionality to figure out the server-side interfaces so our javascript can make the proper ajax calls. We'd figure out exactly how the javascript is going to persist and reload data. We'd make sure we at least had a proposed architecture, and had identified the major parts of it, deciding whether any needed further analysis to expose potential roadblocks. At that point, we could come up with a guesstimate of how long the entire project would take. It's about as unagile as you can get.

So, how do you strike a balance and give a time estimate? Demand specs (even if what you end up is useless for the development team)? Demand more time to investigate (even if you have no time to devote to interviewing stakeholders)? Give up and get a job waiting tables?

Simple: you guess! And if at all possible you let the customer provide you with an answer! If you can illicit a desired date from the client (say, a month), you have a starting point for an iteration. Starting with a timebox of a week, we can run a bunch of spikes, perhaps coming up with some usable code. At that point, we'll be able to see what's possible to complete in the rest of the iteration. At that point, we set the expectation clearly and repeatedly. If there's a complaint or resistance, we're in a much better position to renegotiate (and well ahead of the deadline, so that no one panics).

Of course, as I write this, it's all conjecture. We have negotiated 5 weeks, but have yet to schedule it into our sprints. I'm curious (no . . . I'm nervous) to see how it all works out. Will that be enough time to get a quality solution? Or is that enough time to get an underwhelming, buggy turd propped up? Is renegotiating for more time, if it comes to that, going to be more confrontational than just dealing with misaligned expectations upon delivery? Guess I'll find out!

The problem with QA

2010-01-10T12:01:00.000-08:00

As far as Java goes, we've solved most of our testing problems. We built enough infrastructure to make unit testing easy and effective, even when complex mock objects are involved. Fitnesse provides acceptance testing, which doubles as functional tests. While we haven't had time to put resources into JMeter testing, we've got the infrastructure set up and ready to go (hopefully we'll eventually be able to afford additional headcount to build out a full performance test suite).

For our client apps, unit testing is done in a similar fashion to Java. Mocking is generally pretty easy and we've had no trouble putting the test suites into our automated build process. Coverage tests help identify risks we expose when we forget to follow TDD practices.

While we've done some unit testing for Javascript in the past, it's been pretty minimal. One developer on our team really likes working with Javascipt, and he's put effort into improving our testing capabilities. Unfortunately, we still have issues with testing events (mainly due to the inability to run multiple threads in our tests). We still haven't attempted to automate the tests within our build, but I'm confident we won't have any issues with finding a solution.

To really test our entire application stack, we need more infrastructure. We're using Selenium in conjunction with Fitnesse to write tests, but we're far from a sustainable solution. We're able to run multiple systems with the exactly the same software as the production systems (config changes, but no code changes - mocks are not allowed).

But keeping this running has been an incredible challenge. Besides having to solve the technical challenges, we'll never be able to automate our entire testing infrastructure. Because we have multiple systems involved, we can't (nor want) systems to be automatically deployed. We don't want our front-end tests to break just because we're making changes to how the web server works. Sure, once we think the change is "complete", then we want tests to show us where we've missed things. Having multiple systems deployed on every code change would just lead to tests always failing without indicating which ones are serious and which are trivial. This makes the tests useless as developers quickly learn to ignore the results.

We also have the issue of running tests across multiple machines. Even though Selenium provides the ability to control other machines, just keeping those machines running is a challenge. While VM's may reduce our need for hardware (and space), they certainly don't reduce our need for admin assistance.

The real issue with QA is that, like code, it takes people and time to provide a high quality solution. If you don't have people to build and maintain your test suite, your code will suffer. On the other hand, if all of you developers are forced to become QA implementers, your code will suffer (at least as far as scheduled delivery is concerned). I believe building software without balancing QA staff with development staff leads to technical debt that's risky and dangerous.

Inheritance gone wrong

2010-01-08T08:21:00.000-08:00

Wow. I just got done refactoring a class with one of the worst designs I've ever seen. An abstract class contained an abstract variable that was used in two protected methods.

Most of the concrete implementations (writing by the same developer who wrote the abstract class) set the protected in the "setArgs" method, and used it in the "getArgs" method. Imagine my surprise when I overwrote the "getArgs" method and didn't set the protected variable.

The silver lining is this ugly, smelly design led me to other ugly smellies in the code. While I wouldn't call the design elegant yet, the next developer that builds a concrete implementation of this abstract class will have a much easier time.

Google Wave is my friend

2010-01-05T06:49:00.000-08:00

After falling madly head-over-heels for Google Wave, I've discovered that my infatuation was based more on lust than love. Sure, at first sight, it seemed beautiful and wonderful, despite warning me that it was not ready for a long-term relationship. So now, we remain great friends, though don't talk as often as we once did (nor perhaps as much as we should). But I'll stay in touch because I now realize a good tool, like a good friend, is sometimes exactly what you need to make a bad day better, or a good day great.

I've attempted to categorize the types of waves I've seen. There could be others, but these seem to be prevalent in my inbox:

IM style messages
Voting systems
Long term documentation
Forum style/interest topics
Running jokes

Of these, the first two are where I think Wave excels. Or maybe, that's just where I've learned to use it best. Both of these share a common trait - they are short lived. These are replacements for IM messages and short lived emails. Waves persist, while IM messages don't. The ability to see messages being typed, and the ability to change them could make IM obsolete, and make email laughable as a collaboration tool. I've seen the real time, multiple user style IM communication wow many new users.

I find waves very helpful for conversations that last only a day or two. Distributed standup meetings, getting concensus on design decisions and helping other developers get up to speed on new technologies are all situtations where Wave has been a tremenous help. These are all conversations that stale quickly.

Some of the limitations of Wave could be improved in future iterations. A better alert system, the ability to hide waves (having them only reappear if content is updated) and the ability to remove users from a wave (or even removing ones self), would go far in making Wave better for longer term communication.

Wave is now one of my main communication tools. Right up there with email, IM and RSS, Google Wave is something I'll continue to use daily.

The Duct Tape Programmer and surviving the "death trap"

2009-09-28T07:46:00.000-07:00

I really like Joel Spolsky's blog, but I have a lot of problems with one of his recent posts about the Duct Tape Programmer.

He is the guy you want on your team building go-carts, because he has two favorite tools: duct tape and WD-40. And he will wield them elegantly even as your go-cart is careening down the hill at a mile a minute. This will happen while other programmers are still at the starting line arguing over whether to use titanium or some kind of space-age composite material that Boeing is using in the 787 Dreamliner. When you are done, you might have a messy go-cart, but it’ll sure as hell fly.

Seriously? When you're done, you're lucky to still be alive! If you make it down the hill in one piece!

Of course, overengineering can be just as bad as fly-by-the-seat-of-your-pants programming. But, just because you win the race doesn't mean you're going to win. Go ahead and win the first heat. I'll gladly take your spot in the second heat, while you're trying to figure out how to keep your front wheels from falling off.

Remember, before you freak out, that Zawinski was at Netscape when they were changing the world. They thought that they only had a few months before someone else came along and ate their lunch. A lot of important code is like that.

Netscape did win the first heat. Great for them! I'll bet it was high-times while all this was going on. Of course, since that time, Netscape hasn't had their go-cart in any races. I'll bet after the initial thrill of winning, life wasn't so happy for the employees and developers at Netscape.

No, I think I'll stick with taking a little extra time to make sure my shit works. If you don't have time to do unit tests, you don't have time to write the code. There's nothing about testing that requires you to use "templates, multiple inheritance, multithreading", etc., approaches.

In fact, I do agree that it's best to go for the simple approach. But, most duct tape programmers I've met aren't going for simple - they're going for "works when I try it" without any regard for readability. And since they don't use unit tests, if it doesn't work for you, the only way you can refactor their code is by crossing your fingers and feeling like you're flying down a hill at a mile a minute on a bucket of bolts held together by duct tape and wd-40! No thank you!

I'm such an asshole (and I really think it's going to pay off big)!

2009-09-24T07:28:00.001-07:00

The company I work for moved into new office space awhile back. We simply outgrew the old space (well, that, and the boss went in with some other investors to buy a building in China town).

One of the worst parts of the old space was that the developers sat in "the back room". To get to the restrooms, you either went out the front door, or through the back door (which was in the back room). This resulted in numerous "drive bys".

A "drive by" can range from a random thought to "the next big thing". It usually starts in the mind of a salesperson or marketing type. The process can start anywhere, but the urinal seems to be a treasure trove of drive by opportunities. As soon as the thought forms (but hopefully not before hand washing is complete), the thinker looks for the first developer they can find. Then, the thinker unloads his wonderous wisdom upon the developer who is supposed to stand in awe, absorbing all of the ingenious and innovate ideas spurting forth. Once the stunned feeling subsides, the developer is naturally going to figure out some way to fit the new feature into the product as soon as is humanly possible (or faster).

Needless to say, drive bys suck! These are ideas that, if fed, will be spread to clients and throughout your entire organizations sales engine. Before you know it, what you thought was an innocent head nod has come back to bite you. If you're lucky, you find that this great feature (which has no specification, definition or clear vision) is already overdue. If you're not so lucky, you get a bug report from a client asking why they just paid your company money for a product when they can't get that feature to work (because, of course, that's the one thing the client bought your software instead of your competitors).

At our new office, we're no longer in the "bathroom path". Drive bys have been reduced exponentially! I owe a big thank you to Cody, who was the dev team manager when we moved into the new space - he worked hard to make sure drive bys wouldn't be an issue, including getting us a door - that locks!!!!

But yesterday, while trying to troubleshoot a failing unit test, here comes Mr. Salesman with "the next big thing". To be honest, as he stood over my desk and asked whether our puny brains could fathom how to realize his incredible concept in software, I didn't even listen. All I heard was "will the new product have blah blah blah" and "no one else does that". And, instead of politely nodding, I shook my head and said "I don't know. You'll have to ask Jeff and Sheryl what's going into the new product". When this didn't seem to satisfy him, I added "I don't know what's going to be in there, but I don't think we'll have that."

First of all, I have since found out, we will, indeed, have this innovative concept in our new product. In fact, it's so innovative that it's already been thought of and defined. Apparently, even though I still don't know what the feature is, it's something we've already built.

Secondly, I suppose I could have saved Mr. Salesman a few minutes by actually listening to him. I could have listened and realized that "yes, we've already built that". I could have even taken some of the credit and made myself look smart by explaining how well we wrote that particular branch of code. I should have probably flushed the error I was trying to track down from my mind, realizing that however long it took to get my head back in the code, it was time well spent to make sure that this saleperson could turn his attention to even grander innovations.

But, my reaction was to basically tell him he was talking to the wrong person. I was polite enough to not be offensive, but steadfast enough to make sure there was no question that I would not play the "define a feature" game. If you want to talk about whether a feature is technically possible, then by all means, let's chat! If you want to understand how our architecture will impact future growth, integration possibilities or performance, then I'm all in! But, if you want to tell me about a rough draft of an idea that occurred to you in the shower, then fuck off! I have better things to do!

So, as bad as I feel about turning Mr. Salesman away with the understanding that I will not take part in inflating his ego, I think it will pay off! I'm pretty sure I've helped cure someone of "drive by disease".

Zend_JSON_Decode barfs on funky characters

2009-09-22T08:51:00.001-07:00

Zend works fairly well with json, but we've found many problems with funky characters. The latest character was an "end of text" - \u0003. No idea how it got input into the system, but now that it's there, we need Zend to be able to deal with it.

Thanks to Katie for helping us figure out what characters are causing problems!

Is Java dead? Are you kidding?

2009-09-21T06:59:00.000-07:00

Posts that contemplate whether or not java is dead are amusing at best and annoying at worst. Regardless of which company bought who, or which scripting/functional/newfangled language is so much better, or how many people are "jumping ship", Java will continue to thrive for years!

This is no different than people proclaiming that Linux would spell the demise of Windows (I must admit, I was among those fools). True, Windows has much less market share than it once did, but it is by no means dead.

Heck, if COBOL continues to run "almost three quarters of the world’s business applications", then Java will surely survive 2009! And if Java is currently the most popular language, saying that Java is doomed seems pretty ignorant.

Of course, if by dead you mean something you won't use anymore, then sure . . . move along! Those of us who view languages as tools will continue to use it (and claim the money that companies are still willing to hand out). You go ahead and make decisions based on what language "feels" better and which one you think will get you the most respect. In the meantime, I've got work to do, and that my friend, involves writing heaps and heaps of Java code!

Creating an Easy Mock Custom Matcher

2009-09-16T08:34:00.000-07:00

This blog isn't really supposed to contain "how-to" entries (it's really just a place for me to vent). However, in the past I've found certain tasks worth putting somewhere online in case I need them in the future. I've used planet-source-code for snippets, sourceforge for projects, and maybe a few other sites here and there. Custom matchers in Easy Mock are just hard enough to want a reliable tutorial. While there may be better posts about this same subject elsewhere, there's something about reading code you've written to remind you how to write more code!

So, here's my "CollectionMatcher" class:


package project.test.common;

import java.util.Collection;

import org.easymock.IArgumentMatcher;
import org.easymock.classextension.EasyMock;

public class CollectionMatcher implements IArgumentMatcher {

 private Collection collection;
 private String notFound;
 
 public CollectionMatcher(Collection collection) {
  this.collection = collection;
  notFound = "";
 }

 public static Collection collectionEq (Collection collection) {
  EasyMock.reportMatcher(new CollectionMatcher(collection));
  return null;
 }
 
 public void appendTo(StringBuffer buffer) {
  buffer.append(notFound).append(" not found in {");

  String comma = "";
  for (Object o : collection) {
   buffer.append(comma).append(o.toString());
   comma = ",";   
  }
  
  buffer.append("}");
 }

 public boolean matches(Object otherCollection) {
  if (!(otherCollection instanceof Collection)) {
   return false;
  }

  Collection otherText = (Collection)otherCollection;
  
  for (Object o : otherText) {
   if (!collection.contains(o)) {
    notFound = o.toString();
    return false;
   }
  }
  
  return true;
  
 }

}

So, now we can do things like:


package com.project.api.delivery;

import static project.test.common.CollectionMatcher.*;
import static org.easymock.classextension.EasyMock.*;

//other imports

public class PreviewCommandTest extends TestCase {

  init(new PreviewCommand(preview));
  
  {
   ContentReplacementManager manager = managerContext.getContentReplacementManager();
   expect(manager.replaceValues(content, null)).andReturn(content);
   expect(manager.replaceTags(eq(content), (Preview)eq(null), (Collection)collectionEq(entity.getPreviews()))).andReturn(content);
   managerContext.doReplay();
  }
  
  command.invoke(webContext);
  
  managerContext.doVerify();
  
 }
 
}

Testing the internet (or how to mock a web service)

2009-09-01T06:46:00.000-07:00

Part of the architecture of project Awesome involves an engine that processes a high number of tasks, each requiring a significant amount of work. While I can't explain what the engines main responsibility is, this is related to the persistent queue I mentioned in a previous post.

At the end of each unit of work, a message is sent to an external system for additional work to be done. However, because the external system is essentially a non-blocking remote queue, we get no feedback or information about the state of the work being done. To solve this, there is yet another service which returns feedback on the process of each unit of work as it executes on the remote system. We reconcile the feedback with each unit of work sent for processing.

Because there are so many steps involved, testing this entire process is a bitch. Testing this entire process in Fitnesse is even harder. The engine spans threads to periodically check the incoming queue and poll the feedback service. Since Fitnesse runs in a single thread, we will need to test some of the threading issues with JMeter (for example, if two callers attempt to place the same item in the queue, only one should be accepted). However, we don't have the resources to do JMeter, so for now, we'll have to hope that our unit tests are enough.

So, the external system gets mocked in Fitnesse! To accomplish this, we create an actual HTTP server on the local system which we can control within test tables. Fitnesse starts up this mockable server, and then starts the engine, telling it to call the address of our mocked service. With no changes to it's existing code, our engine is now processing records that we can include directly from our wiki.

While this doesn't test the entire engine, it does allow us to test one critical piece. We can now verify our engine performs correctly, regardless of the state of actions of the external service (in fact, we can even make it do things it shouldn't ever do - just in case).

Careful with those dates!

2009-08-27T07:12:00.001-07:00

Crap! I've been bit in the ass by dates AGAIN.

Back at Microsystems/Biotronik, I got screwed assuming the SQL Server could tell the difference between dates that were milliseconds apart. Turns out, they have to be several milliseconds apart, or they look like the same date. When you're tracking items as they moved through a manufacturing line and machines are pushing the parts around, milliseconds are critically important. If you want to use date as part of a compound primary key or unique index, you have to use a epoch-based value (or something other than SQL Server).

And once again, dates are back to haunt me! This time, we're using dates to organize files on the filesystem. Awesome has a concept of "Assets", which is simply binary content (pictures, spreadsheets, documents, etc.). For each binary file, we keep a record (in the database) of metadata. The actual file is stored on the file system. We decided that in order to potentially partition and sort the files, we would store them based on their created date. In other words, the "date added" value in the metadata record tells us where the binary file is stored. This works great within the overall Awesome architecture.

That is, until yesterday!

Turns out, the database was misconfigured. While we're on the West coast, the database thought it was somewhere in the midwest. One of our diligent IT staff noticed this incosistency, and like any good IT person would, he corrected the problem. Hours later, we were troubleshooting a problem with Assets not being found. Now, the metadata that used to point to 2009/06/12/10/10/24 as a path for the Asset was looking in 2009/06/12/12/10/24. Opps, no file!

BTW, if you're wondering why we use the date and don't store the actual path as a varchar, consider what happens when we want to change how the binary files are stored. While it would be painful to move each file to a new directory structure, changing the code to determine the path is much easier (and safer) to change than trying to update each record in the database. Just sayin' . . . :-)

Anyway, the database's timezone is back to central time and all is good. For the time being, we're going to write a script that moves all of the files to a different "hour" directory and then change the timezone. Fortunately, we don't have very many Assets that were uploaded between midnight and 2 am. Since the added time is only critical for finding the binary file represented by the record, updating a few records before we run the script should be pretty painless.

So, crisis averted. But next time, I'll pay more attention when choosing whether or not to use a date as an identifier!

There's always a beginner in the crowd

2009-08-18T20:27:00.001-07:00

Each morning, my RSS reader is full of posts that seem very rudimentary. Absolutely not trying to pick on any one particular post, but there are many of them each day. These are things that you run across when working in a new language or with a new API. These articles strike me as a "here's a puzzle I ran into and here's how the puzzle is solved" type entry.

These used to annoy me. I envisioned a blogger who was so proud of solving a problem that they felt they should share their brilliance with the world. I felt they were trying to announce their advancement from one level of knowledge to the next.

I now realize that that's not the case. People have lots of reasons to post a blog entry. Maybe it's to share an experience, share knowledge, or just (like in my case) to vent.

But the best thing these entries do is remind other developers of the basics. Sure, these are valuable to beginners, but we're all, in one way or another, beginners. Programming has so many back alleys and dark corners that no one is an expert. Sure, maybe you're very knowledgable about Java concurrency, or the latest scripting language, or . . . whatever. But there's a lot of stuff you don't know (and if you deny it, you're so oblivious to reality that you are probably dangerous to your clients well being).

There are professional developers working every day that completely overlook (or ignore, or forget) the basics. Because we work with multiple languages, it's pretty easy to forget the language specific shortcuts or optimitazations that each language offers. There are highly paid developers with many years of experience who do not use version control, unit testing and similar strategies that most of us take for granted (I swear this is true - I've seen it with my own two eyes). If you're a developer with a lot of experience, don't discount the ideas and discoveries that the newbie at the desk next to you, or the blogger that got stuck in your RSS feed, is so excited about -- there's a chance she's found something of value that you've overlooked.

My language is better than your language

2009-08-18T20:18:00.000-07:00

I'm flabbergast at the number of blog posts that try to convince readers why one language (or framework) is better than others. What a waste of bits!

You can find fault in any language. PHP lacks a finally clause, Java is too cumbersome, Ruby uses a poor threading model, and on and on. The funny thing is, while I might argue that each of the above statements show these languages downsides, I'm sure there are arguments that can be used to contradict or refute those statements.

Who cares? If you don't like a language, don't use it! If it solves a problem for you, then use it. If you only know one language, and you know how to solve a particular problem with it, then by all means, use it! The end user doesn't give a rat's ass what language you use (unless it won't work on their computer).

Languages are just tools. Use what works to solve a particular problem. Some languages solve certain problems better than others, but that doesn't make them better overall languages. Saying that a language is the best is the same as saying you know that language -- if you say you know a language (unless you're the creator of that language), then you don't realize how much you don't know!

Terracotta is a JMS killer

2009-08-11T07:37:00.000-07:00

I often feel bad (or maybe it's guilt) at the decision to not use Spring in our software platform. I don't think it would have helped our project, and I certainly don't miss the XML "sit-ups" we would be stuck with now. But, when one of our team leaves (even if that's me), it's going to be one major item missing from their resume.

So, now we come to our current dilemma. Awesome (the code name for our platform) currently has a queue for asynchronously processing tasks. It's not a bad design and is throughly tested. But it's also not the cleanest design. We've encountered another place where asynchronous processing is required. So the time has come to refactor the current queue logic so these, and all future queues, are coded consistently.

My experience tells me that this new queue especially, should be consumed on a remote machine. Warnings of millions of events, each requiring significant processing power, occurring repeatedly for sustained periods of time, are common. Of course, there's no way to currently verify the loads we'll have to support, or the amount of power required to process each event, but my general sense is, we could cripple most machines.

So, my first reaction was to use JMS to manage our queues. This way, we could move them off of our main server easily, and let some other machine churn away on the events without effecting anything else.

But then I realized, at least for the time being, we are developing for a single JVM. Of course, we're planning on using Terracotta to make it a really big JVM, but it's still acting as a single machine. We're also working from a single database and file system. Being able to code for a single machine definitely reduces the complexity of distributed processing, but at some point, this model will not be optimal. I'm pretty convinced that we'll eventually want to move some of the processing off to a separate environment.

So, to use JMS or not? It would certainly look good on everyone's resume. And while we have a persistent queue already coded up and working in production, JMS would handle the persistence for us, but that's probably a "push". JMS would allow us to use a subscription scheme where we currently only support one consumer per message, should we ever need it. But introducing JMS would add more complexity to the overall architecture. The only way to really know which route is best would be to research, test and benchmark. Since our timeline is so compressed and management views QA as a secondary concern, guesswork is all I've got.

All our team can do is build the features we're being told to build (if this sounds bitter, it's not - we have a huge feature list and a very tight deadline). I'm certainly not turning a blind eye to the potential for performance problems down the road, but I don't have the time or resources to make an informed decision. So, for now, we hope that one big JVM can handle the load we're going to encounter.

I hope when shit hits the fan, the switch is set to low.

Cart before the Horse

2009-07-30T20:00:00.001-07:00

Sometimes you can't avoid it. You just can't write featureA until you at least define featureB. It would sure be nice if everything was separate, but on occasion, it's unavoidable.

For example, consider a situation where you're interfacing with an external service. Say this external service accepts requests to do some work and also provides feedback on work that is either completed, or in process. Classic consumer/producer type stuff. Your typical remote work queue.

But, what if you're responsible for both of these systems? It's pretty hard to write the interface that calls the external service if you don't know what the external service will need to complete it's work. At the same time, it's difficult to write the external service if you don't know what work is to be performed. Until you know what feedback is required by your system, or what data your system will have to reference the external system, it's pretty difficult to know how you'll gather and return the feedback.

Cart before the Horse. You need to know at least a little bit about each system in order to even abstract out the parts that are generic. Until you know what data each system will have, and what each system needs in order to do it's work, you can't create the protocol that they'll use to talk to each other. And until you do that, trying to define one side of the equation is pretty much guesswork.

A good post on clean coding

2009-07-29T09:21:00.000-07:00

Here's a great post on keeping code clean. Some PHP guys say Smarty isn't useful; maybe that's why there's so much PHP code that sucks to work with.

Building just what you need, even when you know it's wrong

2009-07-28T09:44:00.001-07:00

I just completed some functionality that I know is incomplete. Sure, all the tests pass, but I know there are other tests that could be written. But since we don't have time to make the tests pass, I'm just not writing them.

The functionality basically merges values into a document. Think "mail merge" type functionality or smarty-style tags. The current implementation will work for all of the "official" user stories we have documented. However, I can think of certain situations where, especially if escaping is involved, the merged output will not be what is intended.

So, should the time be spent now to make sure that any possible input can be handled? Or do we stick with something that may handle all of our needs, and add tests only when we run into situations where a customer need additional functionality?

Regardless of the "right" answer, we're moving on . . .

My love and hatred of Fitnesse

2009-07-28T09:24:00.000-07:00

I honestly can't remember how I survived without Fitnesse. What a marvelous tool!

Fitnesse provides regression and intergration testing of our backend api, acceptance testing that is clearly understandable to non-programmers (even managers can understand it) and specs for developers. It allows us to put our datastores into a known state before tests run without having to expose the functionality through the api. Truly, exactly what the doctor ordered.

But there's a dark side to Fitnesse too. And it goes beyond the fact that, as a wiki, it's easy to get a big mess of tests with very little organization. After using Fitnesse for over 2 years, we've been able to rangle in the mess and figure out a good organizational strategy.

The biggest problem we're facing now is a limited number of people that can write quality tests. In fact, the only one who puts any serious time into writing tests is . . . me! Sure our project manager spends time putting in user stories, but these are little more than "items should be able to be deleted" type entries. The act of converting that into actual executable tables falls on my shoulders.

Perhaps it's because I'm too controlling, or perhaps it's because I know that having quality tests ready to go means that the team will always have well defined work to complete. I wonder if I should try offloading some of the testing onto other developers (the only ones who have any chance of creating quality tests), or if I should keep them working on making the tests pass.