IPython Sprint

What: A one day concentrated effort to advance IPython development.

When: Saturday, 28 April 2007, 9am to 6pm

Who: FernandoPerez and BrianGranger will lead the IPython team, the Front Range Pythoneers, and anyone else interested in joining the party

Where: bivio Software, Inc., 28th and Iris, Boulder, CO (Above Hair Elite in Suite S) Google Maps link

Net presence: if you can't be physically present, join us on channel #ipython over IRC at FreeNode, we'll start the channel around 9am when we get going and will keep it open through the day. Some of us will also have Skype available in case anyone wants to talk about the work from other locations.

Quick News

• The sprint branch is here (make the local checkout with whatever name you want):

svn co http://ipython.scipy.org/svn/ipython/ipython/branches/sprint1

Signup

If you plan on coming, please add your name to the list below by picking the next consecutive available number in the 10.20.30.X address block:

# IP aliases for sharing code during sprint
192.168.0.200    fernando
192.168.0.201    steve
192.168.0.202    jbaker
192.168.0.203    bgranger
192.168.0.204    bwaters
192.168.0.205    dobbs
192.168.0.206    robert
192.168.0.207    justin
192.168.0.209    matt
192.168.0.210    kkinder
192.168.0.211    cary
192.168.0.101    ikelly ian
192.168.0.104    kip

The code sharing page explains the details, but basically after configuring your IP alias with the number you chose, you can add this list to your /etc/hosts file and will then be able to find others on the network regardless of the actual IP address the DHCP server gave them.

Homework

Here are a few things you may consider doing before the sprint, to ensure we can get going quickly on Saturday, without wasting much time on setup issues.

• Install the saw branch of IPython, as described in the Parallel Computing page. Play with it and let us know how it all goes. Note that right before the sprint, we'll clone this branch to have a safe sandbox to play with on Saturday, but if you get the 'real' one working, you'll already have all the right tools and dependencies in place.

• Install Mercurial and read our code sharing instructions. We'll use this workflow on Saturday.

Ideas

• Tutorials that cover various usage cases. Tutorials for IPython1 are located here. It would be wonderful if people could develop further tutorials that illustrate how to use IPython1 for various things. Some critical things: task farming, using IPython1 with MPI, and domain specific examples (from Physics, Biocomputing, Chemistry, Image/Signal processing, etc.)

• Investigations of new styles of high level parallelism. There is a lot of fun stuff to do here, ranging from a Map/Reduce controller and client to applications of generators and iterators in parallelism. The sky is the limit here.

• Update the basic tutorial that is here to reflect the current state of saw. It would also be nice to have examples of the task farming stuff here. The python script that the tutorial is based on is here. One of the main things that will need to be added is a good discussion of the PendingResult object.

• IPython for distributed unit testing. Basically, use an ipython-based cluster to run unit tests in parallel, which could significantly speed up large test runs.
• Update ipython to use optparse instead of DPyGetOpt.py; generate the usage, man page and latex manual from the getop info. The code to do this would be generically very useful, and is sorely missing in python.
• Refactor the IPython.ultraTB module to separate the coloring from ANSI terminals, so that in the future the highlighting can be made via HTML or any other mechanism. Note that ultraTB came originally from the cgitb module in the stdlib, so look there for inspiration.
• Complete deployment and monitoring library, to start engines and controllers, to monitor engines and kill them locally, etc.
• Begin to move the IPython trunk into saw.
• Finish the multienginehttp.py client and add a taskhttp.py client. We would probably want to use httplib2 for the client part of this. This would be low-level Twisted hacking.
• Work on the security of the network protocols. Also low-level Twisted hacking.
• Add more tests.
• Performance benchmarks.

• Task system: Min has implemented a task farming system already. Discuss, look into getting it in shape for release, talk about scheduling design. Interesting link on this front: the new upstart system from Ubuntu, which has a clean state machine design for this type of problem.

• Go over setuptools issues: use of ez_setup, best practices, etc. Robert is an expert on this topic, poke his brain on it.

• Design/implement a system for interactively searching the documentation for all available modules. - SteveRogres and CaryMiller tackled this, mostly doing discovery on open source search tools with Python bindings. This sub-project will be documented on the Developer_Zone/SearchDocs page.

Resources

One of the main goals of the merge/refactor is to make it easy to embed IPython inside of other applications, mostly GUI-based. The Accerciser team seems to have successfully embedded the current IPython inside a GTK text widget. This code will provide useful pointers both on how to do things, and on the needs of such a project (so that in the future they don't need to hack quite so much).