This is my page for IPython related notes.
Release of Saw
- Finish client work
- pending Deferreds
- barrier
- Replace vanilla at the high level
- timeouts on blockOn
- length error on pulling too bug object
- refactoring ipcluster
- tests+documentation+tutorials
Refactoring Notes
- Expand the tests for multiengine.
- Clean up multienginepb.py.
Make a PB version of the RemoteController class
- Implement a custom paging mechanism for PB sends.
- Add security to the client.
- New controllers.
Error Propagation
Currently we are tyring to design a good way for errors to propagate from Engines to the Controller and eventually back to the user. My overall idea is that each interface will document clearly how errors are handled. The main interfaces we need to worry about are:
- IEngineBase
- IControllerBase
- IMultiEngine
The challenge is that most of these interfaces can be present on either side of a network connection. Thus we must be able to propagate things over the network and have them adhere to the same interface. The other thing that we need to do is begin to get a hand on the diferent types of Exceptions to see if they need to be handled differently. Some general notes:
- Exceptions that are known or expected should have custom exception classes that record extra info like the Engine's id.
- Exceptions should be logged if nothing else.
- I want to be able to raise the original exception and have access to the original traceback.
Here is a list of the interface methods and notes of the types of errors that can occur:
IEngineCore
Error\method |
How to handle? |
execute |
push |
pull |
pullNamespace |
getResult |
reset |
kill |
keys |
Object cannot be serialized |
raise generic serialization error and propagate down errback chain. |
0 |
0 |
1 |
1 |
0 |
0 |
0 |
0 |
Object cannot be unserialized |
raise generic unserialization error and propagate down errback chain. |
0 |
1 |
1 |
1 |
0 |
0 |
0 |
0 |
Object doesn't exist |
raise KeyError and propagate down errback chain |
0 |
0 |
1 |
1 |
1 |
0 |
0 |
0 |
Unexpected errors |
log them |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
Network protocol error |
log them |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
Object too big to send |
for push, raise exception/Failure. For pull, propagate equivalent down chain. |
1 |
1 |
1 |
1 |
1 |
0 |
0 |
1 |
Exception in user code |
create failure and return in results dict |
1 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
IEngineSerialized
Error\method |
How to handle? |
pushSerialized |
pullSerialized |
Object cannot be serialized |
|
0 |
1 |
Object cannot be unserialized |
|
1 |
1 |
Object doesn't exist |
|
0 |
1 |
Unexpected errors |
|
1 |
1 |
Network protocol error |
|
1 |
1 |
Object too big to send |
|
1 |
1 |
- IEngineQueued
- clearQueue
- queueStatus
- IControllerCore
- registerEngine
- unregisterEngine
- IEngineMultiplexer
- execute
- push
- pull
- pullNamespace
- getResult
- reset
- kill
- keys
- pushSerialized
- pullSerialized
- clearQueue
- queueStatus
- IEngineCoordinator
- scatter
- gather
- IMultiEngine
- getIDs
- verifyTargets