Category: Unit Testing

Jan 20 2011

Looking for a vertical software vendor? Here is a quiz to separate the first-class from the fly-by-night vendors.

If your company is looking for a big-ticket software vendor and you want to take some early steps to find the right vendor, here is a brief quiz you could give to your candidates.

These are mostly adapted from THE JOEL TEST but I have added some things that are relevant since The Joel Test is pretty out of date.

When interviewing the vendors, set it up so that they would only have an hour or so to answer and return the test (so they can’t fake or fudge the answers.)

I would not expect perfection but on the whole most software shops (by that I mean a company whose primary line of business is selling packaged software) should be doing a lot of what is suggested by the questions.   This “test” would do a good job of separating the first class operations from the fly by night ones.  Even the first class ones that are not doing all of what is here will have strong answers and justifications.  The only way a fly by night one could pass this test is by lying through there teeth.

Speaking of lying through their teeth.  Before making a final decision maybe you should ask to SEE the operation.  Have them show you their environment and step you through the use of their ticket system, or release process, or bug-tracking software.

An irony of the economics of vertical market software is that it is traditionally a LOT more expensive but is actually LESS likely to have quality built-in.   Inexpensive single-purpose software vendors can’t afford to deliver crappy software because the support costs can easily overwhelm the revenue stream.  But vendors that charge $100,000 to &1,000,000 per installation can throw money and people at their problems (and very often bill the costs to their customers.)  I am not saying that all really expensive software is crappy.  But the potential is there so you must tread carefully.

Enjoy.

Here is the quiz.

ABOUT VERSION CONTROL

  • Do you use Source/Version Control?
  • Which Version Control system do you use? 
  • Can your developers that are working  remotely (for example, doing customer customizations) do commits and updates to your version control system (as long as there is a internet connection)?

SOFTWARE BUILDS

  • Can you make a releasable build in one step?
  • Do you make daily builds when you are product is in active development?
  • Do you use a continuous integration system?
  • What do you use for continuous integration?

BUG MANAGEMENT

  • Do you use bug tracking software?  Please describe briefly.
  • What bug tracking software do you use?
  • Do you fix bugs before writing new code? (When working against a schedule do you fix bugs before starting new features?)

METHODOLOGY

  • Do you follow a software development methodology? 
  • What is the name of the methodology you follow?
  • Is it an agile or waterfall methodology?
  • In what ways do you NOT follow the recommendations of your proscribed methodology? 
  • What role do developers play in driving your release schedule (as opposed to managers)?
  • Describe your methodology in two paragraphs or less.

PROGRAMMERS

  • Describe the method you use for hiring developers.  What steps do you try to complete?
  • Describe the working condition of your developers. 
  • How often are developer computers replaced in your company?
  • What do your developers do for professional development?

QUALITY ASSURANCE

  • Do you have in-house testers that are not developers?
  • Can developers close their own tickets in your bug tracking software?
  • Do you do usability testing as part of your UI design process?  Please describe briefly.
  • Do you have in house designers for the UI of your software?

UNIT TESTING

  • Do you use Unit Testing in the development process at your company?
  • Would you say that you use Unit Testing as part of the design process or just to be sure that changes don’t break the build.
  • What is the code coverage of your unit test suites?
Aug 12 2010

Differences between Unit Testing Mocks, Fakes, and Stubs

Martin Fowler’s page on the matter is a well-known guide.  Here is how he defines the differences in summary:

  • Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
  • Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
  • Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what’s programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it ‘sent’, or maybe only how many messages it ‘sent’.
  • Mocks are what we are talking about here (the article): objects pre-programmed with expectations which form a specification of the calls they are expected to receive.

Happy Unit Testing

Seth