Planet OSL

I’ve recently been asked to join as an adviser to the Open Source Digital Voting Foundation.  In my view, this is one of the most important open source projects around for the US system of democracy.  I was deeply impressed by their open standards specification,  public trust approach and the work they’ve done thus far - with little public fan fare - to establish the non-partisan initiative which has become known as “TrustTheVote!”.

Recognizing a large, active OSS community exists in Oregon, the OSDV is coming to Portland on February 18th to introduce their project.  Although the meeting content is designed for a technical audience, the project overview and progress-to-date would be of interest to many.

Here’s a description of the event.  You can also view details including a map to the event at Portland’s CubeSpace on Yahoo or Calagator.  If you’re in the area, hope to see you there!

Discover this imperative “public digital works project” of the Open Source Digital Voting Foundation. The OSDV Foundation is a Silicon Valley based public benefits corporation whose mission is to work to restore trust in how America votes through the design, development, and demonstration of open source digital voting technology.

Join us to learn details about the “TrustTheVote Project,” a well funded non-profit effort which has been under the radar for 2 years. The OSDV Foundation is now raising public awareness, and expanding efforts including a planned development center in Portland, Oregon.

Our guests are two executives of the Foundation including its Chief Technology Officer. Their presentation will:

• Introduce the project, its motivation, founding, and development efforts to date;
• Walk through the TrustTheVote technology road map and review major projects underway;
• Discuss development philosophies and approaches including experience-driven design and test-driven agile development;
• Review opportunities for systems architects, software developers, SDQA/test specialists, and user experience designers;
• Cover plans to expand the volunteer developer teams, future opportunities for senior members of technical staff, and opportunities for you to get involved.

Presenters:
Gregory Miller, Chief Development Officer
E. John Sebes, Chief Technology Officer

event on: Yahoo or Calagator

In my first post on my new blog I mentioned that I decided to use Drupal for my main blog site. As you can probably notice by the footers on this page I’m no longer using Drupal and decided to go back to WordPress. I noticed several people at the last Corvallis Beer and Blog asked me about it. That made me think I should write a blog post about my journey, or at least the short version of it.

There were several reasons for this but it basically boiled down to using the right tool for the right job. Drupal is an amazing platform for managing and generating dynamic html content. Its plugin and theming system is one of the best out there. Unfortunately, I’m no Drupal guru (even though we host them at the OSL) and had several issues getting some things working like I wanted.

So why WordPress? Well, it seems like everyone in my region raves about it plus I tried the 2.7 version and saw how simple it was to administrate and setup. The ability to install plugins without ssh’ing into the machine was  a key feature that I just loved. Here’s a few of the things that I really like about WP.

• Installing/Upgrading plugins easily
• Categories/Tags are easy to setup
• TinyMCE is installed and works flawlessly
• Comment spam is easier to handle
• The editing flow is less busy
• Found a theme that is simple, elegant, and easy to customize the header image

While I agree Drupal can essentially do all the things above, it requires more researching, googling, tweaking, and trail and error to get it right. I spent a week or two combing the interwebs trying to find a defining site or blog post that explained everything. I found several sites that had decent information, but it wasn’t explained in a manner that made sense to me. The whole concept of taxonomy is just strange to me and in WP its stupidly simple. To me Drupal is like the Gentoo of blog software. It’s great and awesome if you know how to tweak it and use it properly, but for a blogging n00b like me, its just too much hassle for me to do deal with. I think Drupal has some great documentation, its in no comparison to the simple ones like Gentoo has (which is a large part of why people can install it).

So what does Drupal need to do? A few simple things would have made my life easier:

• Create a page or site describing in detail how to setup similar features that modern blogs like WordPress have (categories, tags, etc)
• Implement module installation via Drupal
• Implement module upgrades via Drupal
• Create a Blog centric theme similar to K2
• Make it easier for people to use Drupal but keeping it simple for normal bloggers

I may go back to Drupal someday but I need to understand it more. I really think Drupal is an awesome project and its ability is far superior than any other CMS out there. But for a blog its just a bit too hardcore for the average blogger. Hopefully that will change in a few years.

Today I looked at this blog’s usage statistics and as it turns out, a whopping 54 percent of my visitors use Firefox, followed by Internet Explorer, then Safari, Opera, Chrome.

It’s interesting to see how much difference the “clientele” of a page makes for its statistics. The overall market share of Firefox has topped 20% a few months ago, but since this blog has a lot of tech content, a higher number of Firefox users is probably not surprising.

By the way, almost 80% of my Firefox visitors surf with Firefox 3.0.5, followed by only 5% of 2.0.0.20 users and a long tail of various other, outdated, browser versions.

Here's the wordle generated cloud of my blog posts. Seems pretty cool! I think I say "community" too much :)

I’m taking a trip to Matsue between March 8th and March 14th 2009.

Lecture, participate in Shimane University’Project: “Stabilization and Business Models for Open Source Software through the Cooperation of Industry, Government, Academia, and the Software Developers’ Community”

There’s been a nice little meme going on in the Mozilla blogosphere during this last week: Running your blog through the wordle “word cloud” generator, then posting it. Here is mine:

Comes to no surprise that “German” and “Germany” are two of the most-used words here — I sure hope I am not boring my readership to death by giving them intercultural lessons here

Anyway, if you post yours, feel free to send a ping to this article, or leave a comment with the link to your blog. I am looking forward to seeing how others’ word clouds look!

Wow … the past six months have been interesting in Moodle-land. As of this morning we’re up to 10,850 users (both teachers and students), 1488 courses, and 173 Moodle instances. Clearly classes in Oregon’s K-12 public schools have started.

lars@bozeman:~$ history|awk ‘{a[$2]++} END{for(i in a){printf “%5d\t%s\n”,a[i],i}}’|sort -rn|head
133 cd
114 ls
44 svn
31 vi
28 python
24 ssh
21 ./ConfigurationManager.py
17 make
13 rsync

It looks to me like I spend too much time moving around the file system. I should try to type more pathnames and stick around in one place…

Try this in Linux: open Firefox on your local machine. Then open a terminal window and ssh to another machine using the -X option for X forwarding. On the remote machine, start Firefox. The behavior I get is so bizarre that it cannot be a bug — somehow this looks intentional. The Firefox process on the remote machine sits for a few moments and then dies. Then a new local Firefox window opens. WTF?

I thought I was going insane. The people at the OSL that I told about this thought I was insane. The Mozilla developers that I work with and tried to explain this to thought I was insane.

Some research shows this: the remote Firefox actually starts and communicates with the X server running on the local machine. The X server tells the remote Firefox that there is already a process called Firefox running. The remote Firefox then sends a message to the local one to open a new window and then the remote Firefox dies. This protects a user from creating too many instances of a Firefox process on their machine. Clever, huh? But totally WRONG and counterintuitive!

Apparently you can stop this behavior if you start the remote Firefox with the intuitively named “no-remote” switch. That prevents the remote Firefox from “connecting” to the local Firefox.

Sigh, there goes an hour of my life that I’d like to have back…

Here is an example of a utility that uses a recursive generator. It is a command line utility that assists Java programmers in finding missing classes. I wrote this script several pears ago when I was dragged kicking and screaming into a Java project.  The script recursively searches a directory tree for jar files. When it finds a jar file, it scans the file’s directory for the target Java class.

#!/usr/bin/python

import sys, os, os.path
import fnmatch

def findFileGenerator(rootDirectory, acceptanceFunction):
  for aCurrentDirectoryItem in [ os.path.join(rootDirectory, x) for x in os.listdir(rootDirectory) ]:
    if acceptanceFunction(aCurrentDirectoryItem):
      yield aCurrentDirectoryItem
    if os.path.isdir(aCurrentDirectoryItem):
      for aSubdirectoryItem in findFileGenerator(aCurrentDirectoryItem, acceptanceFunction):
        yield aSubdirectoryItem

if __name__ == "__main__":
  rootOfSearch = '.'
  if sys.argv[1:]:
    rootOfSearch = sys.argv[1]
  if sys.argv[2:]:
    classnameFragment = sys.argv[2].replace('.', '/')
    def anAcceptanceFunction (itemToTest):
      return not os.path.isdir(itemToTest) and fnmatch.fnmatch(itemToTest, '*.jar') and
             classnameFragment in os.popen('jar -tf %s' % itemToTest).read()
  else:
    def anAcceptanceFunction (itemToTest):
      return not os.path.isdir(itemToTest) and fnmatch.fnmatch(itemToTest, '*.jar')

  try:
    for x in findFileGenerator(rootOfSearch, anAcceptanceFunction):
      print x
  except Exception, anException:
    print anException

The focus is on the generator function findFileGenerator. It creates an iterator for the results of a recursive search through a directory tree. It accepts as parameters a path to begin the search and a function to determine if a given file satisfies the search parameters.

Generators can be kind of confusing because even though they look like a function, they do not execute immediately when called. They return a reference to an object that works like an iterator. The code defined in the generator function is executed by that iterator object. The first time that the iterator’s ‘next’ function is called, execution begins at the beginning of the code and goes until it encounters a ‘yield’ statement. The ‘yield’ statement returns the next value of the iterator. The next time the ‘next’ function is called, execution resumes at the next statement after the ‘yield’.

Let’s examine this example closely. Imagine that the first call to the iterator has happened and we’ve got the resultant iterator-like object. The first call on that object to ‘next’ starts execution at this line:

for aCurrentDirectoryItem in [ os.path.join(rootDirectory, x) for x in os.listdir(rootDirectory) ]:

Here we’re getting our first directory listing of all the files in the current directory. Because the call to ‘os.listdir(rootDirectory)’ returns a list of file names with their paths stripped off, we’re going to have to re-attach them. The list comprehension (the code between the [ … ]) welds the current directory path to each of the files in the list and returns a new list. The for loop then sets us up to iterate through that list.

   if acceptanceFunction(aCurrentDirectoryItem):
      yield aCurrentDirectoryItem

Here’s where we decide if the current entry in this directory is interesting or not. We call the acceptance function on the item. Since the acceptance function is passed in when we originally called this generator, it could be anything the programmer desired. In the case of this particular utility, we’re looking for Java Jar files that meet a certain criteria. But it really could have been anything at all: find all files that have vowels in their name, or all files that have a specific type or content.
If the acceptance function returns ‘True’, then we yield. The current file is returned by the iterator and execution stops until the ‘next’ function is called.

  if os.path.isdir(aCurrentDirectoryItem):

If the acceptance function rejected the item, this is immediately the next line to execute. If the acceptance function accepted the item, this line won’t be called until after the next call to ‘next’. In either case, our goal is to find the next item for the iterator to return.

Since we’re iterating through a list of entries in a directory, some of those will be directories themselves. The item that we sent to the acceptance function could have been a subdirectory. Regardless of the outcome of the acceptance function, we need to recurse into subdirectories.

   for aSubdirectoryItem in findFileGenerator(aCurrentDirectoryItem, acceptanceFunction):
        yield aSubdirectoryItem

Hang onto your hat, here’s where your brain may explode. We’ve got a sub-directory and we need to recurse into it and iterate through its entries. Well, we’ve got this handy generator that does exact that: it returns an iterator that will cycle through the contents of directory. ‘for’ statements in Python have a special relationship with iterators. You can provide one instead of a list and the ‘for’ loop will dutifully iterate through them for you. We recursively call the generator, passing in the subdirectory and the acceptance function. The generator returns an iterator to us and the for statement starts the iteration by silently calling the next function. Remember that the iterator returns only items that have passed the acceptance function, so each item that we get here we’re just going to pass on as the next item in our iterator. Hence, we yield every item that we get in this loop.

The rest of the file is in the problem domain: a command line utility that will find Java Jar files with certain classes in them.

if __name__ == "__main__":

Perhaps someday in the future, we’ll want to use the generator in another application. By putting the code of the command line utility under this ‘if’, we’ll prevent it from executing when we use the ‘import’ statement on this file.

  rootOfSearch = '.'
  if sys.argv[1:]:
    rootOfSearch = sys.argv[1]

The root of the path that we’re to search is option on the command line. If no path is specified, we’ll assume that we’re to start in the current working directory.

  if sys.argv[2:]:
    classnameFragment = sys.argv[2].replace('.', '/')
    def anAcceptanceFunction (itemToTest):
      return not os.path.isdir(itemToTest) and fnmatch.fnmatch(itemToTest, '*.jar') and
             classnameFragment in os.popen('jar -tf %s' % itemToTest).read()

The name of the class that we’re to search for is also optional. If the user does not provide one, then we’ll assume that we’re to just find all jar files regardless of their content.

This code fragment is the other case: a fragment of a class name has been given. It is our task here to create an acceptance function that meets the criterion.

First thing to do is cook the class name a bit. In Java, class names are qualifies with paths. Inside Java code, ‘.’ is used as a separator. However, inside jar files, ‘/’ is the separator. To be friendly, we want Java programmers to be able to use either notation. We make sure the command line argument is converted to the ‘/’ notation and stored in ‘classnameFragment’. Next we define an acceptance function that receives a pathname as a parameter. All we have to do is subject that pathname to some tests and give it either a thumbs up or down. In this case, we test to see if the pathname represents a directory, then test to see if it is a jar file and finally we run the command line function ‘jar \-tf’ to give us a listing of the jar to see if our class name fragment is in there. Since Python can do “short-circuit” expression evaluation, if any of the earlier tests fail in the boolean expression, the other tests do not get executed.

  else:
    def anAcceptanceFunction (itemToTest):
      return not os.path.isdir(itemToTest) and fnmatch.fnmatch(itemToTest, '*.jar')

In the case where the user did not provide a class name fragment, we assume that we’re looking for all jar files. The acceptance function here just drops the additional criterion where we looking into the content of the jar file.

  try:
    for x in findFileGenerator(rootOfSearch, anAcceptanceFunction):
      print x
  except Exception, anException:
    print anException

Finally, we ‘re ready to actually use the tools. We call the generator function with the path from which to start the search and our acceptance function. That returns an iterator that we loop through and print the matching jar files.

The encoding job is the object that the producer SDK revolves around. It contains configuration for what, when and how something is encoded. It is defined by IHTXEncodingJob and implemented by CHXTEncodingJob. Most of the producer client logic will deal with creating and manipulating this object and its children

This example shows an input being created and added to the job.

Encoding jobs can also be stored or read from xml files. CHXTEncodingJob implements IHXTUserConfigFile which contains the methods WriteToFile(…) and ReadFromFile(…). This feature won’t be needed by all clients but its easy and unobtrusive to add.

IHXTEncodingJob isn’t the only interface that can be serialized. Audiences can be stored separately so they may be reused in different jobs.

This code snippet taken from the command line producer app shows an audience being read from a file

//deserialize audience
if ( SUCCEEDED(res) )
res = spSerial->ReadFromFile( audpath.c_str(), !m_Params.bDisableUpdateCodecs);