eagain.net Open in urlscan Pro
2606:4700:3033::6815:1cd9  Public Scan

Form analysis
0 forms found in the DOM

Text Content

 * About
   
   About me and this site

 * Blog
   
   A journal of my thoughts and things in my life

 * Articles
   
   Collection of articles I've published

 * Talks
   
   Presentations I have delivered recently

 * Software
   
   Software I have written


TV'S COBWEB


BETTER COREOS IN A VM EXPERIENCE 2016-03-31



(This was all tested with CoreOS beta 991.2.0, the stable 835.13.0 fails to
mount 9p volumes.)

CoreOS comes with instructions how to run it in QEMU . After the setup, it comes
down to something like

#!/bin/sh
exec ./coreos_production_qemu.sh \
    -user-data cloud-config.yaml \
    -nographic \
    "$@"


with cloud-config.yaml looking like

#cloud-config

hostname: mytest
users:
  - name: jdoe
    groups:
      - sudo
      - rkt
    ssh_authorized_keys:
      - "ssh-ed25519 blahblah jdoe@foo.example.com"


I used that for many experiments, but felt it was less than ideal.

I wanted to move beyond -net user. That just means calling QEMU directly,
instead of using coreos_production_qemu.sh, no big deal. But it meant I would be
writing my own QEMU runner, no matter what.

I also wanted more efficient usage of my resources -- after all, the whole point
of me running several virtual machines on one physical machine is to make these
test setups more economical.

The provided QEMU images waste disk and bandwidth. Every VM stores two copies of
the CoreOS /usr image, just like a physical machine would. Copy-on-write
trickery on the initial image will not help beyond the first auto-update, as
each VM independently downloads and applies the updates. This means if you run a
small test cluster of say 5 VMs, you'll end up with 10 copies of CoreOS, and 5x
the bandwidth usage needed.

Imitating physical computers with virtual machines is great if you're trying to
learn how the CoreOS update mechanism works, but once you're to the point of
wanting to just run services, it's simply not needed.

CoreOS does have a supported mode where it does not use the USR-A and USR-B
partitions: PXE booting , starting a computer by requesting the software over
the network. I could even skip the virtual networking and use this with QEMU by
launching the kernel and initrd directly , no need for PXE itself. However, this
is wasteful in another way: it holds the complete /usr partition contents in
RAM, using about 180MB. Once per each VM. There is also an annoying delay of 15+
seconds in VM startup, presumably related to the large initrd image, and later
the kernel spends 1.2 seconds uncompressing it into a tmpfs (measured on a
i5-5300U laptop).

Digging into the PXE image, I find that it actually stores the /usr contents as
a squashfs -- which is a real filesystem that can be stored on block devices, as
opposed to just unpacking a cpio to a tmpfs. The PXE image does what's called a
"loopback mount", where a file is treated like a block device. In the PXE
scenario, the file is held in RAM in a tmpfs; I can just put those bytes on a
block device, and boot that!

(The Live CD seems to also hold /usr contents in tmpfs just like the PXE
variant, even though it could fetch them on demand from the ISO. The squashfs
image is random-access, unlike the usual cpio.gz that's used for initramfs
contents. In later versions, CoreOS could switch their ISO images to use the
trick I'll explain below -- at the cost of physical machines needing to spin up
a CD more often than once per boot. The live CD has another downside that made
me avoid it: to pass kernel parameters, I'd have to resort to kludges like
creating a boot floppy image with syslinux and the right parameters on it.)

So, I set about fixing the wasted disk and bandwidth problem. Here's a story of
an afternoon project.


USING THE /USR IMAGE DIRECTLY

Instead of holding an extra copy of the /usr image data in RAM, we can make it
available as a block device, and load blocks on demand.

For that, we need the /usr squashfs image as a standalone file, not inside the
cpio. It's not available as a separate download, but we can extract it from the
PXE image:

wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_pxe.vmlinuz
wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_pxe.vmlinuz.sig
gpg --verify coreos_production_pxe.vmlinuz.sig
wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_pxe_image.cpio.gz
wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_pxe_image.cpio.gz.sig
gpg --verify coreos_production_pxe_image.cpio.gz.sig

zcat coreos_production_pxe_image.cpio.gz \
    | cpio -i --quiet --sparse --to-stdout usr.squashfs \
    >usr.squashfs



PREPARE A ROOT FILESYSTEM

We also need to make prepare a disk image that will be used for storing the root
filesystem. CoreOS won't boot right with a fully blank disk. If it had, I would
have used qcow2 as the format, but now I need to provide some sort of structure
for the root filesystem, so let's go with a raw disk image.

I might have been able to set up the right GPT partition UUIDs for the initrd to
mkfs things for me, but that seemed too complicated, and I doubted it'd support
my "just the root" scenario as well as their nine-partition layout.

To keep it simple, we won't bother to use partitions; the whole block device is
just one filesystem.

>rootfs.img
chattr +C rootfs.img
truncate -s 4G rootfs.img
mkfs.ext4 rootfs.img



PREPARE USER_DATA

This was previously done inside coreos_production_qemu.sh with a temp dir, but
we'll just pass a directory as virtfs following the "config drive" convention.
Let's move our previous file into the right place:

mkdir -p config/openstack/latest
mv cloud-config.yaml config/openstack/latest/user_data



FINALLY, RUN QEMU

qemu-system-x86_64 \
    -name mycoreosvm \
    -nographic \
    -machine accel=kvm -cpu host -smp 4 \
    -m 1024 \
    \
    -net nic,vlan=0,model=virtio \
    -net user,vlan=0,hostfwd=tcp::2222-:22,hostname=mycoreosvm \
    \
    -fsdev local,id=config,security_model=none,readonly,path=config \
    -device virtio-9p-pci,fsdev=config,mount_tag=config-2 \
    \
    -drive if=virtio,file=usr.squashfs,format=raw,serial=usr.readonly \
    -drive if=virtio,file=rootfs.img,format=raw,discard=on,serial=rootfs \
    \
    -kernel coreos_production_pxe.vmlinuz \
    -append 'mount.usr=/dev/disk/by-id/virtio-usr.readonly mount.usrflags=ro root=/dev/disk/by-id/virtio-rootfs rootflags=rw console=tty0 console=ttyS0 coreos.autologin'


You'll be greeted with the Linux bootstrap messages and finally

This is mycoreosvm (Linux x86_64 4.4.6-coreos) 06:14:10
SSH host key: SHA256:t+WkofIWxkARu1hezwPnS/vgTJXUcPidA3UxKr+1uGA (DSA)
SSH host key: SHA256:cT32H33EVCHSnrCRsB+I9GG7AgXQWfyjk7JFuEzAqFU (ECDSA)
SSH host key: SHA256:NFgc7BLbeyS3SslpscSSNHNzc7lXzx6vKqBmUp+5T7Q (ED25519)
SSH host key: SHA256:pK8Dknoib61FnIwMQ6u4F4FxeSMIRq9zYsrJd0N3MPY (RSA)
eth0: 10.0.2.15 fe80::5054:ff:fe12:3456

mycoreosvm login: core (automatic login)

CoreOS stable (991.2.0)
Last login: Fri Apr  1 06:02:25 +0000 2016 on /dev/tty1.
Update Strategy: No Reboots
core@mycoreosvm ~ $


Success!

As usual with QEMU, press C-a x to exit.

Stay tuned for part 2, where we will make the VM even leaner.



posted on 2016-03-31, tagged as coreos virtualization qemu kvm linux howto

--------------------------------------------------------------------------------


UNMARSHALING A JSON ARRAY INTO A GO STRUCT 2016-01-13



Sometimes, you see heterogeneous JSON array like

["Hello world", 10, false]


Dealing with such an array in Go can be very frustrating. A []interface{} hell
is just about as painful as the map[string]interface{} hell (See my earlier
article about that ).

The natural way to deal with data like that in Go would be a struct like

type Notification struct {
	Message  string
	Priority uint8
	Critical bool
}


See how much more meaning we've added?

Now, you can't just json.Unmarshal an array into a struct. I'll show you how to
make that work.

Read more...



posted on 2016-01-13, tagged as go json programming article

--------------------------------------------------------------------------------


GO JSON UNMARSHALING BASED ON AN ENUMERATED FIELD VALUE 2016-01-12



In a previous article , we talked about marshaling/unmarshaling JSON with a
structure like

{
	"type": "this part tells you how to interpret the message",
	"msg": ...the actual message is here, in some kind of json...
}


Last time, we left a repetitive switch statement in the code, where each message
type was unmarshaled very explicitly. This time, we'll talk about ways to clean
that up.

Read more...



posted on 2016-01-12, tagged as go json programming article

--------------------------------------------------------------------------------


DYNAMIC JSON IN GO 2015-08-27



Go is a statically typed language. While it can represent dynamic types, making
a nested map[string]interface{} duck quack leads to very ugly code. We can do
better, by embracing the static nature of the language.

The need for dynamic, or more appropriately parametric, content in JSON often
arises in situations where there's multiple kinds of messages being exchanged
over the same communication channel. First, let's talk about message envelopes,
where the JSON looks like this:

{
	"type": "this part tells you how to interpret the message",
	"msg": ...the actual message is here, as some kind of json...
}


Read more...



posted on 2015-08-27, tagged as go json programming article

--------------------------------------------------------------------------------


NOTES FROM SCALE12X 2014-02-24



(Sidenote: this little blog engine had bitrotted pretty bad.. I reimplemented it
with markdown, go & bootstrap, and it's much more pleasant to work with now.
Time for new content!)

I spent Saturday & Sunday at the Southern California Linux Expo (SCALE) , and
here's my very personal report of how I experienced it.

SCALE is not your typical tech conference, it brings in very diverse groups of
people. The organizers are actively trying to reach out to e.g. kids that are in
that "might grow up interested in things" age. Just about every age group,
techie background, and personal interest is present -- the common theme really
is only Linux (and a few BSD-based vendors trying to sell their gear). Of course
this means that SCALE won't ever serve my desires perfectly -- but it serves the
community well, and the feel of the conference is very friendly and engaging.


FRIDAY

First of all, I was too busy to go on Friday, and the streaming video had some
sort of audio codec trouble, so I won't comment about content of the devops day
. What I will say is that I'm impressed by the strength of the devops presence
at SCALE. It's becoming a significant backbone of SCALE, year by year. Kudos to
the organizers. And they're at it all year long -- the local ops-oriented
meetups have a great community going. Heartily recommended, whether you carry a
pager or not. Also see hangops .

SCALE also hosted another sub-event on Friday called Infrastructure.Next ,
@infranext . It looked interesting, though I fear overpresence of Red Hat and
vendor agenda. I'm still waiting for slides and/or video of How to Assemble A
Cutting Edge Cloud Stack With Minimal Bleeding . (The archived live streams for
all three days are useless because of audio problems.)

I also missed Greg Farnum's talk on Ceph . I worked at Inktank for almost two
years, and this technology is one of a kind, and a good indicator of what
direction the future lies. If you deal with >20 machines, you should definitely
take time to look into Ceph.


SATURDAY

Saturday started off with a talk about SmartOS vs Linux performance tooling
(slides ). There wasn't much new there this time around, but Brendan is a good
speaker, and SmartOS is probably the most serious server-side alternative to
Linux I'd personally consider these days, so it's good to keep tabs on what
they've been working on.

My interestests drew me next to the talk about Presto (slides ). Takeaways:

 * batch and interactive systems have fundamentally different needs, e.g. for
   monitoring grace periods, how and when maintenance can be performed; they
   require a different ops culture.
 * Dain shared background on Facebook's internal networking challenges, and how
   data center power limits forced them to essentially trade off other servers
   for Presto servers, to avoid network bottlenecks.
 * Presto is integrating the BlinkDB research on approximate queries, e.g. <10%
   error for 10-100x faster queries sounds like a very good trade-off.
 * many "big data" stores don't store enough statistics about index hit rates to
   guide query planning

I'm sad I missed Beyond the Hypervisor (slides ) due to a schedule conflict.

The OpenLDAP talk (slides ) was really largely about LMDB , and that's what I
came for. LMDB is a library that implements a key-value store, with an on-disk
B-tree where read operations happen purely through a read-only mmap. This is a
really nice architecture, pretty much as good as a btree gets -- that is, it's
probably happiest with read-mostly workloads, and probably at its worst with
small writes to random keys. Pretty much the opposite of LevelDB, there. I wish
the benchmarks were less biased, but that seems to be the unavoidable nature of
benchmarking. LMDB has a lot of the kind of mechanical sympathy that may remind
you of Varnish : all aspects of caching are offloaded to the kernel, and data
can be accessed in zero-copy fashion because the read-only mmap prevents
accidents. For Go programmers, Bolt is a reimplementation of the design in pure
Go, avoiding the Cgo function call overhead, and offering a much nicer api than
the direct wrapper szferi/gomdb . My quick microbenchmarks say that, when used
from Go, Bolt can be faster.

Next up was High volume metric collection, visualization and analysis . If I
could take back those 20 minutes, I would.

I spent the rest of the day catching up with old friends and making new ones.


SUNDAY

Clint Byrum is now at HP and working on TripleO, a project that aims to make
OpenStack do bare-metal deploys, and then run a public-facing OpenStack on top
of that. His talk was a good status report (slides ), but in situations like
this I always end up wanting more details.

For the next slot, I bounced between three different talks, not 100% happy with
any one of them. First, Hadoop 2 (slides ) was an intro to YARN et al that
started off like an apologist "I swear Hadoop and Java don't really suck as much
as they seem to". Mark me down as unconvinced.

Second, Configuration Management 101 was a good effort from a Chef developer to
be party neutral, and talked about the common things you find in all the common
CM frameworks. His references to promise theory are pretty much dead on, and in
the 3 years since I fiddled momentarily with https://github.com/tv42/troops , my
thoughts have gone more and more into thinking about distributed CM as an
eventual consistency problem. With Juju-inspired notifications about config
changes, using more gossip & vector clock style communication to update peers on
e.g. services provided, this might result in something very nice. That one is
definitely on the ever-growing itches to scratch list.

Third, Seven problems of Linux Containers was an OpenVZ-biased look into
remaining problems. Some of it was a bit ridiculous -- who says containers must
share a filesystem, just mount one for each container if you want to -- and some
of it was just too OpenVZ-specific to be interesting. Still, a good topic, and
OpenVZ was groundbreaking work.

For the next slot, I returned from lunch too late to fit in the packed rooms,
and enjoyed breathing too much to try harder. I watched three talks, mostly from
open doorways. The hotel's AC was not really keeping up anymore at this point,
and only the main room was pleasant to be in.

Big Data Visualization left me wishing that 1) it wasn't fashionable to say "big
data" 2) he'd have shown more visualizations 3) he'd talked about the hard
parts.

ZFS 101 (slides ) is interesting to me mostly to see what people think about &
want from storage. Btrfs is really promising in this space, feature-wise; it
still has implementation trouble like IO stalls, but the integrated snapshots
and RAID are just so much more useful and usable than any combination of
hardware RAID, software RAID, and LVM. Snapshots really need to be a first-class
concern. So far, my troubles with Btrfs are of a magnitude completely comparable
to my troubles with the combination of LVM, LVM snapshots, HW-RAID cards dying,
and SW-RAID1 sometimes booting the drive that was meant to be disabled. All in
all, I find the "not yet stable" argument a bit boring; there's a whole lot of
code and complexity in Btrfs, but it also removes the need for a whole lot of
other kinds of code and complexity. If nothing else, the ZFS/Btrfs feature set
should be a design template for future efforts; I understand e.g. F2FS has a
very specific design goal (think devices rather than full computers), but not
supporting snapshots in a new filesystem design is a bummer.

And finally, I spent time in Jordan Sissel's fpm talk . fpm is a tool that
converts various package formats into other package formats, a lot like Alien .
Jordan's viewpoint on this is a frustrated admin who just wants the damn square
peg to fit in the round hole, and fpm is the jigsaw & hammer that'll make that
happen. I fundamentally disagree with him about the role of packaging; the whole
point of packaging is destroyed if the ecosystem has too many bad packages, and
the reason e.g. Debian packaging can be a lot of work is not because cramming
files in an archive should be hard, but because making all that software work
together and upgrade smoothly actually is a difficult problem. But Jordan is an
entertaining speaker, and his point is valid; there are plenty of cases where
you don't care about the quality of the resulting package. Just.. please don't
distribute them, ok?



posted on 2014-02-24

--------------------------------------------------------------------------------


SLIDES FOR MY RECENT TALKS 2012-01-08



I just put up a bunch of slides from talks I've presented lately:

 * concurrency-oh-my/ Concurrency, Parallelism, Events, Asynchronicity, Oh My:
   concepts and Python applications of concurrency, 2011-11-10 at SoCal Piggies
   .

 * ceph-overview/ Ceph Overview for the OpenStack Conference, 2011-10-07 at
   OpenStack Conference .

 * teuthology/ Teuthology: a multi-machine test runner, 2011-07-01 at DreamHost
   .

 * intro-to-boto/ Introduction to Boto, and an even briefer intro to Gevent,
   2011-07-01 at DreamHost .



posted on 2012-01-08

--------------------------------------------------------------------------------


KEYBOARDS INFLUENCED BY TOUCHSCREENS 2011-04-30



By now, you have probably used a touchscreen keyboard. We've come a long way
from the clumsy "kiosk" computers that brought touchscreen keyboards to
mainstream, a decade or two ago. But a classic keyboard with physical keys is
still preferable for the tactile feedback we get from pressing the keys. Until
touchscreens can provide that, we'll be using traditional keyboards for a while.

But how do touchscreen keyboards differ from physical keyboards, and what ideas
could we copy from them to improve the user experience of traditional keyboards?
Well, for one, most touchscreen keyboards these days don't do key repeat --
instead, they'll pop up a menu of alternatives, often the same base letter with
with various accents, diacritics and umlaut. And you slide your finger to pick
one of these options.



Now, that sounds good. I for one don't know how to type the various variations
of the letters on a US keyboard, and as a Finn I actually need "ä" and "ö"
sometimes. I can type them in Emacs, but not in my IM application or web
browser.

Here's the idea: we don't really use autorepeat on the A-Z characters. Instead,
make a long press of a letter key bring up an on-screen menu with variations of
the basic letter.

On a touchscreen, choosing from the menu is immediate and fairly intuitive.
While using the mouse for that would be straight forward -- and probably what a
first time user will try -- nobody wants to type like that. We're stuck with one
finger holding down the original key. We need to make a selection from up to
about 9 alternatives. Here's two easy ideas (assuming the initial highlighted
alternative is the base letter):

 * make space move to the next highlighted alternative, or wrap around to the
   left
 * make the four primary home row keys of the other hand highlight alternatives
   1-4; make pressing the same home row key again act like space, above

For example, say I want to type "ä". My options are (assuming US keyboard
layout):

 1. hold "a", wait till menu pops up, while still holding "a" press space 4
    times until "ä" is highlighted, release "a"

 2. hold "a", wait till menu pops up, while still holding "a" press ";", release
    "a"

I think I'd use that more than auto-repeat. What about you?



posted on 2011-04-30, tagged as keyboard touchscreen human-computer-interface
idea wishlist future

--------------------------------------------------------------------------------


DEPLOY TOOLS 2011-02-27



I've been looking at the world of deployment tools lately. Outside of Puppet and
Chef (and ignoring the old beards Bcfg2 & Cfengine), what other things are
there?

Fabric lets you write Python functions to describe "tasks" to be run. The Python
functions are run on a client machine -- for example, the sysadmin's laptop --
and each task can be directed to operate on hosts or roles (groups of hosts),
over SSH. The functions can run remote commands with run("echo hello, world")
and sudo("chmod u=rw,go=r /etc/passwd"). Fabric is a very useful piece of the
puzzle, but doing more complex operations one shell command at a time gets
frustrating. I keep wanting something that can run whole chunks of Python
easily, on the target machine. Fabric also does nothing to solve the problems of
e.g. multiple admins running a deploy command at the same time.

Kokki is closer to Chef in it's style (and literally, "Chef" in Finnish). It's a
framework for writing cookbooks, with actions like File("/etc/greeting",
content="hello, world") in them. Then a configuration for a machine can invoke
certain recipes. Kokki seems to be aimed fully at running things locally; that
is, if you're deploying things, you'd run Kokki on the target machine. Kokki is
still in fairly early development, it's website and source code don't match each
other at all, and many of the cookbooks no longer work with the current version.
It also still inherits a bit too many non-Pythonic elements from Chef, for my
tastes. Still, to this Pythonista, it looks very promising, and I will be
exploring it further.

Poni ("Pony" in Finnish -- I want one too!) is another Python project that takes
a very different tack. It is built on a command-line tool that lets you define
your infrastructure through hierarchical collections of key-value settings; that
is, you describe the whole multi-machine service with database servers, load
balancers, app servers and all. You can use inheritance much like other deploy
tools use cookbooks. The command-line tool seems to be meant to be used for
everything; the stored data is not really meant to be edited directly. While I
appreciate the polish of the command-line tool, the editor-hostility comes off a
bit odd. Especially so when the getting started guide has me "uploading"
template files and Python source code to Poni's internal configuration storage.
Am I really supposed to have two copies of these files?

Once you have your infrastructure defined, Poni provides you two main methods to
actually make changes: you can create files based on templates (that have a
strong mechanism for referring to any values from the configuration, including
things like sharing the database connection information between the DB server
and the client config), or you can run custom functions ("control commands").
The Python functions run locally, but Poni provides a remote execution framework
very similar to the one in Fabric, though at least for now it is significantly
more verbose. And, to my disappointment, doesn't really allow running full
Python functions remotely either.

Somewhat confusing is the difference between the "create a file" and the "run
control command" functionality. It is not quite clear how the whole is intended
to orchestrate the full deployment, and the examples are both lacking and
misleading. For example, right now the Puppet deployment example requires you to
run a command to create some files, watch it fail, run another command to
install software, then run the first command again to create the rest of the
files. (Kind of weird to deploy Puppet with Poni in the first place..)

There is one thing about Poni that I am already starting to dislike. Currently,
every identifier you need to refer to on the command line is given as a regexp,
and commands act on all matches. This leads to high risk for operator errors:
for example, the documentation itself uses $find("webshop/frontend") as an
example; yet that would also match webshop/frontendforsomethingelse. I do hope
the author changes his mind about regexps everywhere.

Much like Kokki, Poni is very early on in its development; it's command-line
tools and things like variable referencing are top notch, but the picture is
very much not complete yet. But this one is definitely a project to watch.

--------------------------------------------------------------------------------

For posterity: I filed a bunch of issues about the things I bumped into:

 * https://github.com/melor/poni/issues/2
 * https://github.com/melor/poni/issues/3
 * https://github.com/melor/poni/issues/4
 * https://github.com/melor/poni/issues/5
 * https://github.com/melor/poni/issues/6



posted on 2011-02-27, tagged as deploy software fabric kokki poni

--------------------------------------------------------------------------------


EUROPYTHON 2008 VIDEOS ARE UP 2008-08-11



Update: Well, they used to be. blip.tv pulled a nasty stunt and removed tons of
user content. And EuroPython itself has a nasty habit of not providing archived
websites for previous years. This all has linkrotted. Don't you hate the
internets?

My talks were:

 * My God, it's Full of Files -- Pythonic filesystem abstractions
 * Version Control for Du^H^HDevelopers



posted on 2008-08-11, tagged as europython2008 europython python talk video

--------------------------------------------------------------------------------


EUROPYTHON 2008 WRAP UP 2008-07-19







EuroPython 2008 was fun. I presented two talks (My God, it's Full of Files --
Pythonic filesystem abstractions and Version Control for Du^H^HDevelopers ) and
one lightning talk (RST+S5 for your slides), participated in a bunch of open
space sessions, listened to about 13 talks, took a bunch of pictures, but most
importantly had interesting hallway conversations with interesting people.





As usual, PyPy was heavily represented, and seems to be making nice progress
toward being the nice and featureful default Python implementation of the
future. I especially liked the restricted execution features and the LLVM
backend. The zc.buildout talk made me decide I will try to replace (part of?)
one custom deploy mechanism with zc.buildout -- most likely I'll end up
rewriting most of the current things as zc.buildout recipes, but hopefully some
of the pre-existing recipes will be useful, and hopefully I can then later reuse
the recipies I create for this setup.

Personally, I think my talks went ok. I understand videos will be available
later, as soon as transcoding etc are finished. I'm anxious to see them myself,
as I'm still finetuning my public speaking skills. I'm learning, though -- this
year I had no trouble staying within my time slot, even when I was adjusting
verbosity on the fly.

For some reason, I felt underprepared for the filesystem API talk , but
ultimately people liked the idea of a consistent Pythonic filesystem API enough
that we had an open space session on it, and people were enthusiastic about a
sprint to prototype the API. Which is what we ended up doing, too -- I'll blog
separately about the results of that.

My decentralized version control talk seemed to me to go over more smoothly; I
guess that's just because I've been thinking about version control and project
management a lot lately, so it was easy to talk about the topic in a relaxed
way. On the other hand, it wasn't as much a call to action, and it really was
overly generic, so I didn't get as strong audience participation there. We did
have an interesting conversation about branch management strategies and such,
though. I consciously tried to keep the talk on a generic level, as I felt a
pure git talk would have alienated some listeners, but I did end up feeling
restricted by that. There was some interest on a Teach me git -style session,
but what we ended up doing was just talking one on one about getting started
with git, during the sprints. Sorry if I missed any one of you -- grab me on
#git to continue, or find me in future conferences ;)





I was requested to organize an open space session for Twisted Q&A, and that is
exactly what we did. We went through a bunch of things related to asynchronous
programming concepts, Deferreds, working with blocking code and libraries,
database interfaces, debugging and unit testing.

I was also pulled in to another Twisted open space session, that was mostly
about what greenlets are and how to use them. I tried to explain the differences
between classical Deferreds, deferredGenerator/inlineCallbacks, and greenlets,
to the best of my understanding. As a summary, with greenlets any function you
call can co-operatively yield execution (I mean yield in the scheduling meaning,
giving away your turn to run, not in the Python generator meaning --
interestingly inlineCallbacks etc actually make those be the same thing... my
kernel instincts make me want to say "sleep"). Yielding in any subroutine means
anything you do may end up mutating your objects -- which is the root evil
behind threading we wanted to get away from. All the other mechanisms keep the
top-level function in explicit control of yielding. Around that time, most
people left for lunch, but about three of us stayed and talked about debugging
Deferreds and network packet processing with twisted.pair and friends.





One of the interesting hallway conversations was about what happens when
upstream web hosting listed on PyPI is failing. It seems PyPI already does some
sort of mirroring, but even that might not be enough. Many companies seem to be
bundling eggs of their dependencies in their installation package, which sounds
like a good setup for commercial click-to-install deployment. But it would still
be good to see a CPAN -style mirror network for PyPI, and at least some people
seemed even motivated to donating servers and bandwidth. Personally, I'm mostly
spoiled by the combination of Debian/Ubuntu and decentralized version control,
and my level of paranoia is too high to automatically install unverified
software from the internet anyway. My primary motivation in the conversation was
to point out that PyPI already has some sort of mirroring/upload setup, and that
you'd really want to specify exact versions and SHA-1 hashes of your
dependencies. Optionally, you could delegate the known good hash storage to PyPI
(assuming you trusted PyPI not to attack you), but that would require a full
Debian-style signature chain from a trusted key, or you'd be owned by anyone
capable of MITM attacks, DNS forgery, or cracking a PyPI mirror.



posted on 2008-07-19, tagged as europython2008 europython python

--------------------------------------------------------------------------------


EUROPYTHON 2008 TALK #1: MY GOD IT'S FULL OF FILES 2008-07-07



I published slides for my first Europython talk. Note the slides might not easy
to understand without the actual talk -- the video streaming at Ustream will
work if the network infrastructure can take it, and a downloadable video should
be available later.

My God, it's Full of Files

Pythonic filesystem abstractions: An overview of different filesystem(-like)
APIs in Python and attempts for unifying them.

There's a lot of different filesystem(-like) APIs in Python. I intend to provide
an overview of existing projects, their status and capabilities, and hopefully
inspire you to work on improving things.

Read more...



posted on 2008-07-07, tagged as europython2008 europython talk python filesystem
programming

--------------------------------------------------------------------------------


INCREMENTAL MAPREDUCE 2007-11-24



So Google has their MapReduce , and the people behind CouchDB are throwing
around their ideas . I spent some time thinking about incremental mapreduce
around July, and it's time I type out that page full of scribbles.

First of all: I think the ideas thrown out by Damien above aren't really
mapreduce, As Google Intended. The real power of mapreduce is in its inherent
combination of parallelism and chainability, output of one mapreduce is input to
another, each processing step can run massively in parallel with each other,
etc. The proposed design is like a one-iteration retarded cousin of mapreduce.

With that bashing now done (sorry), here's what I was thinking:

The way I imagined building an incremental mapreduce mechanism, without storing
the intermediate data and just recomputing chunks that are out-of-date (which
would be lame), is to add one extra concept into the system: call it "demap". It
will basically create "negative entries" for the old data. This is basically
what Damien did by providing both the old and new data map calls, all the time,
just said differently, and I think my way might make the average call a lot
simpler. And I don't see any reason why my version wouldn't be parallelizable,
chainable, and generally yummy.

Read the article



posted on 2007-11-24, tagged as mapreduce couchdb parallel cluster programming
database search

--------------------------------------------------------------------------------


KVM, THE VIRTUALIZATION MECHANISM, ROCKS 2007-10-12



For years now, my primary machine has been a laptop. I've been avoiding running
Xen 1 because the hypervisor it injects between hardware and Linux hasn't been
very friendly for power saving. Modern laptops are too often hot enough without
any extra help.

Well, I'm happy to say that as of now, KVM is definitely stable enough to
replace my use of Xen for test setups. I expect that shortly it will become what
I recommend for server use, too. And because it lets Linux be Linux, instead of
doing any oddities with the hardware, all the powersaving etc goodness still
works perfectly.

So far, I've stumbled on two things:

 1. The version of KVM in Ubuntu, even gutsy, is just too old. With a bit of
    fiddling with the patches, KVM v46 built a perfectly working deb.

 2. The isolinux graphical bootup, used in Ubuntu, crashes KVM (and with v28, it
    crashes the host machine -- beware!). See the bug report . I got around that
    by using mini.iso , but you could always fall back to debootstrap; that was
    all we ever really had with Xen.

And while I have the soapbox: I have a dream. It's KVM running with SDL/VNC
graphics in a window that's resizeable all the way to the virtual machine, with
XRandR. Please make it happen!

Patches

So, the edited version of the patches:

 * 01_use_bios_files_in_usr_share_kvm.patch
 * 06_no_system_linux_kvm_h.patch
 * from-debian-qemu/04_do_not_print_rtc_freq_if_ok.patch

And just remove from-debian-qemu/62_linux_boot_nasm.patch, it seems to have made
it upstream.

--------------------------------------------------------------------------------

 1. What's up with http://www.xensource.com/ , by the way? That website is just
    a pile of horrible non-informative enterprise speak. Utterly useless, and
    that tends to alienate the techie community pretty fast. At least it's
    alienating me. ↩︎



posted on 2007-10-12, tagged as kvm xen virtualization xrandr idea

--------------------------------------------------------------------------------


SNAKEPIT AND GITOSIS, THINGS I'VE BEEN WORKING ON 2007-10-12



A brief update of things I've been working on:

Snakepit

Snakepit is a port of (part of) HiveDB to Python and SQLAlchemy . It will help
you write database-backed applications that need to scale further than one
database server, or even a master-slave setup, can take you. It's MIT licensed,
that is, pretty much as free-for-all as it can be. And it's still work in
progress, so don't be too harsh yet ;)

See https://github.com/tv42/snakepit for more.

gitosis

gitosis aims to make hosting git repos easier and safer. It manages multiple
repositories under one user account, using SSH keys to identify users. End users
do not need shell accounts on the server, they will talk to one shared account
that will not let them run arbitrary commands. gitosis is licensed under the
GPL.

First real release will come as soon as I have to time to go through a couple of
really minor nits. It's been self-hosting for a long time now.

See https://github.com/tv42/gitosis for more.



posted on 2007-10-12, tagged as software programming git database scalability

--------------------------------------------------------------------------------


ROTACLOCK -- A UNIQUE CLOCK WHERE THE WHOLE WRIST DISPLAY THE TIME 2007-06-06



(This post didn't originally get published due to technical problems with the
graphics. It was written in 2007 and was rescued & inserted into the archive in
2015.)

Imagine a wristwatch with nothing but the wristband. Time flows around your
wrist; parts of the wristband light up to indicate what time it is, and you can
watch seconds race loops around your wrist.

The concept may need some tuning, as some of the readings are awkwardly behind
your wrist. Perhaps there's a gap there that the indicators jump over, around
the closing mechanism of the wrist band.





posted on 2007-06-06, tagged as idea gadget

--------------------------------------------------------------------------------


GIT FOR COMPUTER SCIENTISTS 2007-03-29



I wrote a brief introduction to git for people who are not scared by words like
Directed Acyclic Graph . Read it here: Git for Computer Scientists



posted on 2007-03-29, tagged as git howto

--------------------------------------------------------------------------------


HOWTO HOST GIT ON YOUR LINUX BOX 2007-03-23




WARNING

This solution is obsolete. Use gitosis instead!

Gitosis is basically git-shell-enforce-directory's big brother, and an actual
software project. Use it.

Updated to drop --use-separate-remote from git clone, it's the default.

Updated to add --read-only to git-shell-enforce-directory.

I've run repeatedly into cases where I want to provide services to people
without really trusting them. I do not want to give them shell access. I don't
want to even create separate unix user accounts for them at all. But I do want
to make sure the service they use is safe against e.g. password sniffing.

Instead of trying to run the version control system over HTTPS (like Subversion
's mod_dav_svn that will only work with Apache, which I don't run), I want to
run things through SSH . SSH is the de facto unix tool for securing
communications between machines.

Now, I said I don't want to create a unix user account for every developer using
the version control system. With SSH, this means using a shared account, usually
named by the service it provides: svn, git, etc. To identify different users of
that account, do not give the account a password, but use SSH keys instead. To
avoid giving people full shell access, use a command="..." when adding their
public key to ~/.ssh/authorized_keys.

For Subversion, I submitted an enhancement to add --tunnel-user, to make sure
the commit gets identified as the right user, and then used command="..." with
the with arguments, like this (all on one line):

command="/srv/example.com/repo/svn/svnserve -t
  --root /srv/example.com/repo/svn/view/examplegroup
  --tunnel-user jdoe" ssh-rsa ... jdoe@example.com


Where the view directory is a bunch of symlinks to the actual repositories,
allowing me to do group-based access control.

With git , the author of the changeset is recorded way before the SSH connection
is opened. Without building some sort of access control in git hooks on the
server, every developer can pretty much ruin the repository by overwriting
branches with bogus commits. What they will not have is access outside of the
repository, or a way to actually remove the old commits from the disk (unless
you run git prune on the server). The distributed nature of git makes this
reasonably easy to detect, and pretty much trivial to recover from. For any real
trust in the code, you should look at signed tags anyway. The included wrapper
allows you to have read-only users, but provides no detailed access control
against developers with write access; they just won't be able to escape to the
rest of the filesystem.

So, with that introduction out of the way, let's get to configuring:


INSTALL GIT ON THE SERVER

sudo apt-get install git-core git-doc



CREATE THE DIRECTORY STRUCTURE STORE THE REPOSITORIES AND RELATED FILES

sudo install -d -m0755 \
	/srv/example.com/repo/git \
	/srv/example.com/repo/git/.ssh \
	/srv/example.com/repo/git/repos \
	/srv/example.com/repo/git/view



CREATE THE SHARED USER ACCOUNT FOR THIS SERVICE

sudo adduser \
	--system \
	--home /srv/example.com/repo/git \
	--no-create-home \
	--shell /bin/sh \
	--gecos 'git version control' \
	--group \
	--disabled-password \
	git



LIMIT ACCESS

Set up a script that makes sure only relevant git commands can be run via SSH,
and to limit the visible section of the filesystem to things you actually want
to give access to; put this file in /usr/local/bin/git-shell-enforce-directory (
download ) and chmod a+x it

#!/usr/bin/python

# Copyright (c) 2007 Tommi Virtanen <tv@eagain.net>
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
# BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

# Enforce git-shell to only serve repositories
# in the given directory. The client should refer
# to them without any directory prefix.
# Repository names are forced to match ALLOW.

import sys, os, optparse, re

def die(msg):
    print >>sys.stderr, '%s: %s' % (sys.argv[0], msg)
    sys.exit(1)

def getParser():
    parser = optparse.OptionParser(
        usage='%prog [OPTIONS] DIR',
        description='Allow restricted git operations under DIR',
        )
    parser.add_option('--read-only',
                      help='disable write operations',
                      action='store_true',
                      default=False,
                      )
    return parser

ALLOW_RE = re.compile("^(?P<command>git-(?:receive|upload)-pack) '[a-zA-Z][a-zA-Z0-9@._-]*(/[a-zA-Z][a-zA-Z0-9@._-]*)*'$")

COMMANDS_READONLY = [
    'git-upload-pack',
    ]

COMMANDS_WRITE = [
    'git-receive-pack',
    ]

def main(args):
    os.umask(0022)

    parser = getParser()
    (options, args) = parser.parse_args()
    try:
        (path,) = args
    except ValueError:
        parser.error('Missing argument DIR.')
    os.chdir(path)

    cmd = os.environ.get('SSH_ORIGINAL_COMMAND', None)
    if cmd is None:
        die("Need SSH_ORIGINAL_COMMAND in environment.")

    if '\n' in cmd:
        die("Command may not contain newlines.")

    match = ALLOW_RE.match(cmd)
    if match is None:
        die("Command to run looks dangerous")

    allowed = list(COMMANDS_READONLY)
    if not options.read_only:
        allowed.extend(COMMANDS_WRITE)

    if match.group('command') not in allowed:
        die("Command not allowed")

    os.execve('/usr/bin/git-shell', ['git-shell', '-c', cmd], {})
    die("Cannot execute git-shell.")

if __name__ == '__main__':
    main(args=sys.argv[1:])



CREATE YOUR FIRST REPOSITORY

cd /srv/example.com/repo/git/repos
sudo install -d -o git -g git -m0700 myproject.git
sudo -H -u git env GIT_DIR=myproject.git git init


(with git older than v1.5, use init-db instead of init)


SET UP AN ACCESS CONTROL GROUP AND GIVE IT ACCESS TO THAT REPOSITORY

cd /srv/example.com/repo/git/view
sudo install -d -m0755 mygroup
cd mygroup
sudo ln -s ../../repos/myproject.git myproject.git


You can also use subdirectories of view/mygroup to organize the repositories
hierarchically.

Note, one SSH public key will belong to exactly one group, but if necessary you
can create a separate group for each account for absolute control.

Note, access to repository implies write access to repository, at least for now.
You could make


AUTHORIZE USERS

Get an SSH public key from a developer and authorize them to access the group

cd /srv/example.com/repo/git
sudo vi .ssh/authorized_keys


How the developer generates their key is out of scope here.

Add a line like this, with the public key in it (all on one line, broken up in
the middle of word to make sure there is no misunderstanding about when to use a
space and when not to):

command="/usr/local/bin/git-shell-enforce-directory /srv/exampl
    e.com/repo/git/view/mygroup",no-port-forwarding,no-X11-forwar
    ding,no-agent-forwarding,no-pty ssh-rsa ... jdoe@example.com


Or to allow only read-only access, add --read-only as an option.


YOU CAN NOW PUSH THINGS TO THE REPOSITORY WITH

git push git@myserver.example.com:myproject.git mybranch:refs/heads/master


Note that before the first push, your server-side repository will not contain
even an initial commit, and can't really be cloned.


NOW THE DEVELOPER CAN CLONE THE REPOSITORY

git clone git@myserver:myproject.git


or to avoid some behavior of older git that I consider confusing (needs git v1.5
or newer):

git clone -o myserver git@myserver:myproject.git


They will probably want to set up ssh-agent to avoid typing the passphrase all
the time.

And you're done! Good luck with your adventures with git, and welcome to the
21st century and to distributed version control systems.



posted on 2007-03-23, tagged as git admin howto ssh

--------------------------------------------------------------------------------


HOWTO BUY A USED CAR IN CALIFORNIA 2007-02-20



Tomorrow, if everything goes well, I will buy a used car from an individual in
California. Here's a checklist of things for that, to help others in similar
situations. Some things may have already been omitted because they weren't
relevant for me, so you may want to independently browse the websites I'm using
as sources. I'm skipping everything related to finances and haggling. I'm not
covering cases where the car isn't in good condition. Also, buying from
dealerships is different. Good luck.

(Updated to mention REG 262.)

 1.  Find an ad on craigslist or whatever. I liked
     http://losangeles.listpic.com/car/

 2.  Call the seller. Don't just email, start evaluating the seller. Things to
     ask: http://www.carbuyingtips.com/questions.html (though that list is
     horribly extensive; I'd rather just pick the three best-looking candidates
     and do that thing on the spot; you'll need to doublecheck anyway to make
     sure the seller wasn't lying).
     
     Ask for:
     
     * the VIN of the car (usually 17 characters, on windshield)
     * full name of the seller (you'll need it anyway for the check)

 3.  Check the CARFAX report for the car. Go for the $24.95 30-day option, you
     should always look at more than one car.
     
     * compare odometer, accident history etc with what seller said (better yet,
       just avoid accident cars, the risk isn't worth it)
     * make sure it's not an old junk car poorly repaired ("salvage" title)

 4.  Check the smog check history (fails are a sign of trouble). In California,
     in most of the cases, the seller is required to provide a certificate of
     check newer than 90 days old -- it seems the information flow to website is
     slow enough that this latest check does not show up, or something.

 5.  Check that the car hasn't suffered storm damage . (Annoying website demands
     cookies for a simple form. Suck.)

 6.  Make an appointment and go see the car. In good sunlight, you want to see
     the car.
     
     * for some things to check, read more , especially On-the-Lot Checklist,
       Road and Test Checklist
     
     * you have to drive the car; leave a photocopy of your drivers license if
       needed for assurance
     
     * bring a car nut friend who knows what to check; play good cop bad cop
       (this checklist is not as good, as I am not a car nut)
     
     * on the lot:
       
       * paint chips
       * cracked windows
       * accident damage
       * signs of water damage
       * tires
       * check the VIN you were given against windshield, doors, engine,
         dashboard, major body parts; if they don't match you are dealing with a
         criminal
     
     * while driving:
       
       * brake hard, is it even
       * rev the engine, does it sound healthy
       * make plenty of starts and stops
       * go through a manual gearbox, there should be no grinding noises
       * see if the car will drive straight with your hands off the wheel
       * how does the clutch feel?
       * listen for noises throughout the test drive
     
     * after driving:
       
       * check for leaks
     
     * to be really careful, you should check a lot of things like
       
       * AC
       * windows up/down
       * radio/cd/speakers
       * etc

 7.  Take the car to a mechanic of your choosing for evaluation. Alternatively,
     choose to trust on service from brand name vendor.

 8.  Check the price against Kelley Blue Book etc.

 9.  Now you're in business. Only bureaucracy and doublechecking left.

 10. If you know what you're willing to pay, go to your bank and get a cashiers
     check. Or two alternatives, if that works out. Otherwise, you will need to
     go back to the bank after haggling, and without a deposit the seller may
     have sold the car to someone else. Or something. Tuff.
     
     (Some people say haggling at dealerships is easier if you show up with a
     cashiers check just a bit short from what they're asking.)
     
     Don't pay with cash, there's even less chance of getting it back than
     cancelling a check.
     
     If you happen to be a seller reading this, do the actual sale in a bank to
     be sure you aren't being cheated.

 11. Download and print PDF forms from DMV .
     
     * Bill of Sale (PDF )
     
     * Statement of Facts may interest you (PDF )
     
     * things the seller needs: Notice of Release of Liability (PDF )

 12. In California, sellers are required to provide a smog certificate. Make
     sure you get one. Stuff on smog checks: 1 , 2 , 3 . Frankly, I'm still a
     bit confused myself when I will need to do a smog check.

 13. Check that the registration is current and that the car wasn't repurchased
     under the California "Lemon law". (Err, how? As I understand it should say
     so in the Certificate of Title paper)

 14. The seller should find the "pink slip" aka Certificate of Title
     
     * if said paper does not have form fields for ownership transfer and
       odometer reading, you need form "REG 262" from the DMV, which is printed
       on special paper and not available as PDF. SUCK!
     
     * check seller name against his drivers license
     
     * fill it out with both of your info and both sign it; for instructions
       search for "Where do I sign?" on this page
     
     * if a bank or something still owns a chunk of the car, their signature is
       also needed on the pink slip; I'd be inclined to avoid the complexity
     
     * fill in the odometer value, both sign (read more )
     
     * seller keeps the Notice of Transfer and Release of Liability part and
       submits to DMV.
     
     * buyer fills in the back of the title to transfer ownership

 15. Things to ask for before leaving
     
     * is a special wheel lug key needed, get it
     * are there extra keys
     * how does the car alarm work
     * on a convertible, how does the roof work
     * any "tricks" you should know

 16. Now you're ready to leave with your new car! Check that you have
     
     * Certificate of Title, signed by both, also odometer section
     
     * Bill of Sale, signed by both
     
     * maintenance records
     
     * smog certification
     
     * owners manuals, repair manuals
     
     * spare tire, jack
     
     (sources 1 , 2 )

 17. Seller has 5 days to submit Notice of Transfer and Release of Liability to
     DMV. Do it online .

 18. Buyer has 10 days days to report ownership change to DMV and max 30 days to
     pay the fees. Read more: checklist , things to send to DMV , more info .

 19. Taxes?

General resources:

 * The other side of the story: seller howto . The person you are buying from
   should have done all of this, and this is what you can demand from him. Also,
   see the links and actual transaction guidance inside.

 * Used car buying tips at http://www.carbuyingtips.com/used.htm



posted on 2007-02-20, tagged as car california los-angeles howto

--------------------------------------------------------------------------------


SCALE5X: TALK SUMMARY OF THE OPENWENGO TALK 2007-02-10



More SCALE5x : Dave Neary is talking about OpenWengo . Note to self: Wengo =
TelCo, WengoPhone = software, OpenWengo = project developing WengoPhone -- or
something. At least it's not just .org for community and .com for services, even
if the names are way too close to eachother.

Good quote (not his, didn't catch the name):

"People don't want to buy a quarter-inch drill.
They want a quarter-inch hole!"


He recommended this blog for anyone interested in user interface design:
http://headrush.typepad.com/

Choice quotes:

"Cross Platform (but sound on Linux is a disaster)"

"Surprisingly, for Microsoft, it's not SIP... pure SIP."
(talking about MSN Messenger)


They intend to implement XMPP-based transport mechanisms. Mentioned inkboard, an
Inkscape extension(?) for whiteboard-style sharing of drawing over the internet.

They have games over OpenWengo (I guess XMPP?), like chess.

"Oh did I mention sound on Linux is horrible?"

Heh, we're calling audience members during the talk. From France. And it didn't
work ;)

OpenWengo has cross-platform video conferencing. Wow.



posted on 2007-02-10, tagged as voip sip im software scale5x conference talk

--------------------------------------------------------------------------------


SCALE5X: TALK SUMMARY OF THE HORRIBLY NAMED RED HAT XEN TALK 2007-02-10



More SCALE5x : Sam Folk-Williams is doing a talk called Xen Virtualization in
Red Hat Enterprise Linux 5 and Fedora Core 6: An overview for System
Administrators (UNGH!). And demonstrates why I hate "big" companies like Red
Hat: they sent a non-technical, but well-practised, person to talk about Xen. He
sounds convincing, but ended up explaining Xen domU migration without
understanding the concept of shared storage. Gah.

Also note how talk summary promises live demos of Xen integration features only
Red Hat has, and how the actual talk contained no such thing. If I didn't have
wireless right now I'd be annoyed. Thank you SCALE5x organizers, the wifi is
just great.



posted on 2007-02-10, tagged as admin redhat xen scale5x conference talk bad

--------------------------------------------------------------------------------


SCALE5X: TALK SUMMARY OF ADMIN++, WHAT ROOT NEVER TOLD YOU 2007-02-10



So I'm at SCALE5x , listening to Ron Gorodetzky talk about what he learned about
sysadmining for Digg and Revision3 (who try to be an "Internet television
network"; in effect, they distribute loads of big files). Most of the tools he
mentioned I already knew, but it was nice to get independent reviews of "hey I
think this is good". Here's what I took home from his talk:

 * He really thinks highly of the OSCon 2005 talk Livejournal's Backend (A
   history of scaling) (PDF ).

 * He liked memcached and MogileFS .

 * Between the lines I understood Revision3 has outsourced their big bandwidth
   use -- the CDNs he mentioned by name were Cachefly (the color scheme hurts
   even my eyes and real designers think I'm colorblind), BitGravity (caution
   hideous flash site) and of course Akamai .

 * He spoke about outsourcing data center operations, using things like Amazon
   EC2 and S3 . I need to come up with a budget and time to play with EC2.

 * He stressed the importance of setting up KVMs etc properly for the data
   center.

 * Set up your infrastructure and plan for scaling before you get popular,
   because you will be too busy to do them afterwards. That's nice, I like
   building things scalable from scratch.

 * Specific infrastructure management tools:
   
   * Puppet -- seems pretty much a reimplementation of cgengine
   
   * Bcfg2 -- smells like academentia to me
   
   * ISconf -- from the Bootstrapping an Infrastructure people, seems to be
     based on the idea of a p2p distributed cache that stores pretty much a
     version control history of commands ran.
   
   As usual, I haven't yet seen anything that would actually seem to work in the
   real world, unless you give up everything you already have (like package
   management etc), and do things 100% their way.
   
   His suggestion: as the tools are based on very different worldviews, look at
   everything and try to pick the one that matches your opinions.

 * One thing he wouldn't skimp on: "Don't skimp on RAM."

 * At Revision3, they use long-life server hardware and don't upgrade the
   servers, instead they go for a full new deployment.



posted on 2007-02-10, tagged as admin cluster configuration-management scale5x
conference talk livejournal digg

--------------------------------------------------------------------------------


IMAP OVER SSH HOWTO 2007-02-09



Tired of managing n+1 passwords? Hate having an extra network port open on that
server box? Want to have automated replication of email to your laptop in a Unix
command line geek-friendly fashion?

Here's how to make OfflineIMAP synchronize mail between local and remote
Maildirs .

 * on the client:
   
   * create an SSH key pair with no passphrase:
     
     $ ssh-keygen -t rsa -N '' -f ~/.ssh/imap-preauth-key
     

 * on the server:
   
   * install Binc IMAP on the server; no need to have it actually listen for
     network connections
   
   * I store my mail as ~/.Mail on the server; create a ~/.bincimap on the
     server and adjust to fit:
     
     Mailbox {
       depot = "IMAPdir",
       umask = "0077",
       path = ".Mail",
     }
     
   
   * create a shell script ~/bin/imapd-preauth that'll start the IMAP daemon in
     a preauthenticated mode; note that OfflineIMAP wants a certain style of
     handshake bincimapd doesn't know how to do, so we fix that with sed:
     
     #!/bin/sh
     set -e
     
     export BINCIMAP_LOGIN=PREAUTH+FAKE
     bincimapd|sed --unbuffered '1s/^FAKE OK PREAUTH/* PREAUTH/'
     
     
     Make the script executable (duh).
   
   * authorize the previously generated SSH key to run only the above script --
     add the following to ~/.ssh/authorized_keys (split here for readability,
     make it all one line; replace THINGS to fit):
     
     command="/home/USERNAME/bin/imapd-preauth",no-port-forwarding,
     no-X11-forwarding,no-agent-forwarding,no-pty SSHPUBLICKEYHERE
     

 * on the client:
   
   * tell OfflineIMAP about the preauthenticated IMAP connection:
     
     [Account SOMETHING]
     localrepository = local-SOMETHING
     remoterepository = remote-SOMETHING
     
     [Repository local-SOMETHING]
     type = Maildir
     localfolders = ~/data/mail/SOMETHING
     
     [Repository remote-SOMETHING]
     type = IMAP
     remotehost = HOSTNAME
     preauthtunnel = env -u SSH_AUTH_SOCK ssh -q -i ~/.ssh/imap-preauth-key %(remotehost)s fake-command
     

That should be it! Have fun.

(And if you just broke it, feel free to give one of the halves to me.)



posted on 2007-02-09, tagged as imap email offline software offlineimap ssh
howto

--------------------------------------------------------------------------------


WEB 2.0 EXPLAINED 2007-02-05



Here's a nice video on what people mean when they say Web 2.0:
http://youtube.com/watch?v=6gmP4nk0EOE

I like the style of it, though it is too fast paced if you don't already know
what it is talking about.

(Found via Mickipedia )



posted on 2007-02-05

--------------------------------------------------------------------------------


SIX WORD SCIFI 2006-12-02



Sixwordscifi.com is so much fun it has to be wrong, somehow. Here are my
favorites so far (yes, one of them is mine):

> Back me up before I die.
> 
> — boogah

> Even at light speed, I wait.
> 
> — jefft

> All alone in his light cone.
> 
> — tv



posted on 2006-12-02

--------------------------------------------------------------------------------


THE PHONE KILLER PHONE 2006-12-02



I now know what I want from my next phone. And it'll totally blow the whole
phone concept out of the water.

Start with a mostly-open hardware platform like Neo1973 , add Linux (OpenMoko )
on top. And no need to cram in a clumsy qwerty keypad, just carry a one-hand
keyboard when you care about it. Less clumsy when you don't want more than a
phone, full SSH sweetness when you want it. The phone itself is purely
touch-screen, and the keyboard can actually get respectable WPM with real
keypress feedback. And the big part is, because the phone is actually Open,
plugging all this in is not a big problem! That's just great!



posted on 2006-12-02

--------------------------------------------------------------------------------


I NEED A BAG 2006-11-17



I have a shoulder strap-style bag, manufactured and used by the German army,
bought as surplus and dyed black. My 12" thinkpad fits perfectly inside of it.
But the strap is sewn in place, and seems to fail every few years -- and now's
the time.

The bag is really good, and I am going to get it fixed, but that doesn't mean I
can't look at alternatives. So, I need something that fits a 12" thinkpad, isn't
too big, preferably comes in black, and is otherwise non-attention grabbing and
doesn't look like it'd contain a laptop. According to Lenovo, my laptop is about
268x211x20mm.

My options:

 * small messenger bag: 1 , 2
 * hard-shell attache case (not really my style, but a 12" one in black might
   rock it -- though I'm not going to buy a 15" case with internal padding to
   make a 12" laptop not bounce around, I want a smaller case too; it'd still be
   plenty big for the obligatory dead tree notes and docs; for pics, see 3 , 4
 * the manliest purse ever from Maxpedition : I'd be willing to carry that (in
   black of course;) if it fit a 12".. And I don't think it will.
 * more classical military style: M-51 Engineers Field Bag , Urban Explorer
   Black Canvas Shoulder , etc ..
 * something from http://booqbags.com/ but the style doesn't really smack me in
   the face with want
 * something from Timbuk2 , but they really don't seem my style

None of those really work for me. I guess considering the bags I already own, I
might go for something I don't have. Mmm, a smallish hard-shell attache case in
black, with a good enough shoulder strap that I can sling it over my head. That
might just do it. Now where do I get one?



posted on 2006-11-17

--------------------------------------------------------------------------------


A REVVER COMMAND LINE VIDEO UPLOAD TOOL 2006-11-13



Update: It seems the script had gone missing at some point. It's back .

As you may or may not have noticed, I do a bunch of stuff for Revver. I ended up
writing a sort of a tutorial to the Revver API, and as I like to collect all
kinds of code samples here, I thought I should crossblog it here. The original
is on the Revver developer blog .

One day, I was on a slow internet connection and wanted to upload a few files. I
wanted something more batch-oriented than the web-based upload, and I have a
personal bias against most current Java runtimes. So I decided to use the cool
new API and write a video upload client , and will walk you through what it does
in this blog entry. Feel free to "just" use the tool, but hopefully this will
also help you in writing your own API clients.

First of all, I wanted to write something that's usable just about everywhere. I
tend to use Python , so that's what the tool is written in. The Python standard
library didn't seem to be able to do HTTP POST file upload (think web forms) of
large files, so I ended up using curl for that. This should work on any Linux/OS
X/etc box with Python and curl installed. All you Ubuntu /Debian people just get
to say sudo apt-get install curl and that's it.

So, let's dive right in. The tool is imaginatively named revver-upload-video.
The first bit is the command line parser. Don't be intimidated by the length,
this is pretty much boilerplate code, the actual API-using bits are really
small. The full file is 155 lines, total.

There are basically three kinds of options: mandatory, optional and for
developer use. Mandatory options are enforced later on, and developer options
are mostly meant for playing with the staging environment and reusing upload
tokens from previous, failed, uploads.

#!/usr/bin/python
"""
Guerrilla command line video upload tool.
"""
import optparse, getpass, urlparse, urllib, xmlrpclib, subprocess

def getParser():
	parser = optparse.OptionParser(
		usage='%prog --title=TEXT --age-rating=NUM [OPTIONS] FILE..',
		description='Upload videos to Revver')

	parser.set_defaults(
		api_url='https://api.revver.com/xml/1.0',
		upload_url='http://httpupload.revver.com/',
		login=getpass.getuser(),
		)
	parser.add_option('--login',
					  help='login name to use (default: %s)' %
					  parser.defaults['login'])
	parser.add_option('--passphrase-file',
					  help='read passphrase from (prompt if not given)',
					  metavar='FILE')
	parser.add_option('--age-rating',
					  help='MPAA age rating (mandatory)',
					  type='int')
	parser.add_option('--title',
					  help='title for the video (mandatory)',
					  metavar='TEXT')
	parser.add_option('--tag',
					  help='tags (mandatory, repeat for more tags)',
					  metavar='KEYWORD',
					  action='append')

	parser.add_option('--author',
					  help='author of the video',
					  metavar='FULLNAME')
	parser.add_option('--url',
					  help='website for extra info')
	parser.add_option('--credits',
					  help='extra credits',
					  metavar='TEXT')
	parser.add_option('--description',
					  help='a brief description',
					  metavar='TEXT')

	parser.add_option('--api-url',
					  help='API URL to contact (developers only)')
	parser.add_option('--upload-url',
					  help='Upload URL to send the file to (developers only)')
	parser.add_option('--upload-token',
					  help='use preallocated token (developers only)',
					  metavar='HEX',
					  action='append')

	return parser


If you've used optparse before, there's not much interesting there. It just
instantiates a parser object and returns it, for the main function to use.
Nothing there touches the Revver API yet.

Next up, we have some utility functions. getPassphrase will read a passphrase
form the file given to --passphrase-file=, or prompt the user for one. getAPI
instantiates an XML-RPC client object with the login and passphrase, and caches
it in options so if you call getAPI more than once, you're still only prompted
for the passphrase at most once. Nothing in revver-upload-video uses that, but
these are meant to be reusable functions.

def getPassphrase(filename=None):
	if filename is not None:
		f = file(filename)
		passphrase = f.readline().rstrip('\n')
		f.close()
	else:
		passphrase = getpass.getpass('Passphrase for video upload: ')

	return passphrase

def getAPI(options):
	api = getattr(options, 'api', None)
	if api is None:
		passphrase = getPassphrase(filename=options.passphrase_file)

		(scheme, netloc, path, query, fragment) = \
				urlparse.urlsplit(options.api_url,
								  allow_fragments=False)
		query = urllib.urlencode([('login', options.login),
								  ('passwd', passphrase)])
		url = urlparse.urlunsplit((scheme, netloc, path, query, fragment))
		api = xmlrpclib.Server(url)
		options.api = api
	return api


All right, now we're getting to the actual meat. getToken calls the API method
video.getUploadTokens to allocate an upload token, that lets you upload a file
to the Revver archive.

def getToken(api):
	url, tokens = api.video.getUploadTokens(1)
	assert len(tokens)==1
	token = tokens[0]
	return url, token


createMedia creates a new video in the archive from your uploaded file by
calling video.create in the API. It also adds metadata like your website URL to
the video. Finally, it returns the media id of the newly-created video.

def createMedia(options, token):
	data = {}
	if options.credits is not None:
		data['credits'] = options.credits
	if options.url is not None:
		data['url'] = options.url
	if options.description is not None:
		data['description'] = options.description
	if options.author is not None:
		data['author'] = options.author
	api = getAPI(options)
	media_id = api.video.create(token,
								options.title,
								options.tag,
								options.age_rating,
								data)
	return media_id


Finally, we have the main function, and the bits that call it when you run the
tool. Here, we actually parse the command line arguments, enforce the presence
of the mandatory options, and bail out unless you gave it actual files to
upload.

For each file given on the command line, we either use one of the tokens given
to us with --upload-token=, or get one from the API with getToken. Then we join
the upload URL and the token to get the place to upload the file to, and run
curl as a subprocess to do the actual upload. Checking that curl worked takes 8
lines, and then we use createMedia to actually create the video. And that's it!

def main(progname, args):
	parser = getParser()
	(options, args) = parser.parse_args()

	if options.login is None:
		parser.error('You must pass --login=LOGIN')
	if options.title is None:
		parser.error('You must pass --title=TEXT')
	if not options.tag:
		parser.error('You must pass --tag=KEYWORD')
	if options.age_rating is None:
		parser.error('You must pass --age-rating=NUM')
	if not args:
		parser.error('Pass files to upload on command line')

	for filename in args:
		if options.upload_token:
			token = options.upload_token.pop(0)
			url = options.upload_url
		else:
			api = getAPI(options)
			url, token = getToken(api)
			print '%s: allocated token %s' % (progname, token)

		upload_url = urlparse.urljoin(url, token)
		retcode = subprocess.call(['curl',
								   '-F', 'file=@%s' % filename,
								   '--',
								   upload_url,
								   ])
		if retcode < 0:
			print >>sys.stderr, '%s: upload aborted by signal %d' % (
				progname, -retcode)
			sys.exit(1)
		elif retcode > 0:
			print >>sys.stderr, '%s: upload failed with code %d' % (
				progname, retcode)
			sys.exit(1)
		print '%s: used token %s for %s' % (progname, token, filename)

		media_id = createMedia(options, token)
		print '%s: created media %r from %r' % (progname,
												media_id,
												filename)

if __name__ == '__main__':
	import os, sys
	main(progname=os.path.basename(sys.argv[0]),
		 args=sys.argv[1:])


At this point, we have all we need to do the same thing as the web-based upload,
or the Java upload client. And you can do something similar, by yourself. This
file is copyright Revver, Inc, but licensed under the MIT license -- that means
you can use it as a base for writing your software, without any real
restrictions. Download the whole thing here: revver-upload-video .



posted on 2006-11-13

--------------------------------------------------------------------------------


NEW DOMAIN NAME 2006-09-30



I had a bit of fun thinking up of puntastic DNS domain names. I ended up
registering eagain.net , the old tv.debian.net name will soon start redirecting
there. Need to set up email too.. Vanity domains are soo much fun.

For the rare non-nerd reading this, EAGAIN is the error code you get when you
are doing asynchronous programming with non-blocking sockets and would block
next. Err, let's just say "I write async code and it's a neat insider joke", ok?

For the nerds out there, here's a bunch of wild ideas I had while figuring out
what domain name to register. Many of them are invalid (too short), and most of
the good ones are already taken, but in case you need some inspiration:

 * asyn.ch
 * dot.at
 * ex.plo.de / im.plo.de
 * plea.se
 * fal.se
 * belong.us
 * chi.hu (as in Chihuahua, our dogs..)
 * celci.us
 * blo.gr
 * co.de
 * co.ff.ee
 * oh.no
 * wh.ee
 * wh.at
 * sh.it
 * stre.am (taken)
 * up.stre.am
 * down.stre.am
 * upstre.am
 * em.ploy.ee
 * s.tre.am
 * 3x2.net (as in triple-double-w)
 * tribledouble.net
 * 2by4.net (as in clue-by-four)
 * be.am (la.ser.be.am, scotty.up.be.am ;-)
 * dre.am
 * progr.am
 * pu.sh
 * pu.bli.sh
 * form.at
 * anon.ymo.us
 * pla.net
 * mag.net
 * carwa.sh
 * blo.at
 * cave.at
 * repe.at
 * pho.to
 * oct.et
 * pron.to
 * plu.to
 * lot.to
 * dit.to
 * adju.st
 * ang.st
 * arti.st
 * ava.st (type like a pirate)
 * broadca.st
 * inner.net
 * comm.it
 * mis.info
 * foc.us
 * fung.us
 * geni.us
 * gur.us
 * octop.us
 * styl.us
 * stat.us
 * tor.us
 * line.br
 * deco.de
 * while.do
 * un.do
 * weir.do
 * if.then.fi
 * cook.ie
 * newb.ie
 * zomb.ie
 * disk.io
 * comm.it
 * subm.it
 * bat.ch
 * bran.ch
 * epo.ch
 * fet.ch
 * zil.ch
 * gra.ph
 * boo.st
 * fibona.cc
 * from.to
 * justa.com

And for the Finns:

 * mi.au (cat lovers)
 * vai.nu (dog lovers)
 * kir.nu
 * masen.nu
 * tie.dos.to (taken, by non-Finn)
 * hit.to

I'm not going to say anything about the Cook Island's subdomain for commercial
entities.



posted on 2006-09-30, tagged as computer nerd dns

--------------------------------------------------------------------------------


IN CASE YOUR XEN DOMU'S HAVE NETWORKING TROUBLE 2006-05-21



If your domUs have networking trouble with TCP, or some other protocol that ends
up needing fragmentation such as large ICMP pings, you need to read this.

If it seems TCP handshakes complete, but no data is transferred -- especially,
no actual data gets sent out from the domU -- you're likely hitting a bug in how
Xen interacts with TCP segmentation offload.

The bug seems to depend on the actual network interface card the traffic is
going out from. I hear tg3 is one of the cards that triggers it, and I'm seeing
it on my home box with 8139too's.

The fix is pretty simple, but hard to figure unless you know what to look for:
inside the domU, run

ethtool -K eth0 tx off


for each interface affected.

See
http://wiki.xensource.com/xenwiki/XenFaq#head-4ce9767df34fe1c9cf4f85f7e07cb10110eae9b7
for the very small amount of extra information that is out there.



posted on 2006-05-21, tagged as xen networking computer

--------------------------------------------------------------------------------


IBOOK--, THINKPAD++ 2006-05-21



Life sucks and then your computer breaks. As soon as a new X release is out, and
it seems dual head on iBook is a possibility , the darn thing decides to fry its
logic board. Again. Thankfully, Apple may make the repair for free, if the
symptoms match the manufacturing problem . Again.

Well, the good news is that after 3 years of using the iBook, I got a new
laptop. A Lenovo Thinkpad x60s, weighing just 1.3kg. It's so light I always
think I forgot to put it in the backpack.

Ubuntu Dapper (flight 7) seems to work pretty well on the x60s. Trouble spots so
far:

 * install CD corrupted display during X autoconfig: screen was in text mode,
   mostly black, with two or three character-size grey rectangles -- hitting
   enter blindly let it continue and reboot the machine

 * suspend and hibernation fail on resume

 * wlan hanged once, and didn't recover until I rebooted into Windows -- unngh.
   Look at this:
   
   ipw3945: Error sending SCAN_ABORT_CMD: time out after 500ms.
   ipw3945: Radio Frequency Kill Switch is On:
   Kill switch must be turned off for wireless networking to work.
   ipw3945: Error sending ADD_STA: time out after 500ms.
   ipw3945: Error sending RATE_SCALE: time out after 500ms.
   
   
   After that, any attempt to use the interface ended with:
   
   ADDRCONF(NETDEV_UP): eth1: link is not ready
   

 * hotplugging the UltraBay docking station does not seem to work in Linux

I especially love the dual headedness, after fighting with the ATI driver in the
iBook.

Now I need to see about hooking the fingerprint reader up to PAM.



posted on 2006-05-21, tagged as computer hardware ubuntu

--------------------------------------------------------------------------------


MY IBOOK HAS TWO HEADS 2006-04-23



Finally, after two years of hacks , my iBook 2.2 knows how to multihead! And no
silly clone mode only, totally different image and external output at 1600x1200
at 85Hz. This is nice! Thank you X.org people for version 7, thank you X Strike
Force !

Update: well, now suspending fails and booting the machine results in a black
screen in over half of the tries. Bah.



posted on 2006-04-23, tagged as ibook computer hardware

--------------------------------------------------------------------------------


RENDER_PATTERN: REPEAT PATTERNS EASILY IN NEVOW TEMPLATES 2005-12-21



After render_fragment, dialtone mentioned render_pattern, that would get one or
many patterns from the page and put them in the current tag. Well, that's easy
to write:

def render_pattern(self, name):
   """
   Find and render a pattern.

   Example:

   <span nevow:pattern="foo">
	 I'm very repetititive.
   </span>
   <ul>
	 <li nevow:render="pattern foo">
	   this text will get removed when rendering
	 </li>
	 <li nevow:render="pattern foo"/>
   </ul>
   """
   def f(ctx, data):
	   doc = self.docFactory.load(ctx)
   patterns = inevow.IQ(doc).allPatterns(name)
   return ctx.tag.clear()[patterns]
   return f


Updated to adapt doc to inevow.IQ before calling allPatterns.



posted on 2005-12-21, tagged as nevow twisted python programming

--------------------------------------------------------------------------------


RENDER_FRAGMENT: REUSABLE FRAGMENT EMBEDDING IN NEVOW TEMPLATES 2005-12-21



This Nevow renderer came up on #twisted.web. Thanks to rwall and dialtone for
input.

def render_fragment(self, name):
   """
   Find and render a fragment, with optional docFactory.

   Find a fragment factory from self via attributes named
   fragment_* and replace content of current tag with said
   fragment.

   If pattern docFactory is found under this tag, pass it as
   docFactory to the fragment factory.

   Example:

   class MyFrag(rend.Fragment):
	   ...

   class MyPage(rend.Page):
	   fragment_foo = MyFrag
   ...

   and give MyPage a template with

   <!-- no docFactory -->
   <div nevow:render="fragment foo">
	 this text will get removed when rendering
   </div>

   <!-- with docFactory -->
   <div nevow:render="fragment foo">
	 this text will get removed when rendering
 <span nevow:pattern="docFactory">
   but this whole tag will be passed as docFactory to MyFrag.
 </span>
   </div>
   """
   def f(ctx, data):
	   callable = getattr(self, 'fragment_%s' % name, None)
   if callable is None:
		   callable = lambda ctx, *args: ctx.tag[
			"The fragment named '%s' was not found in %r." % (name, self)]
	   kwargs = {}
   try:
	docFactory = ctx.tag.onePattern('docFactory')
   except stan.NodeNotFound:
		pass
	   else:
		kwargs['docFactory'] = loaders.stan(docFactory)
   return ctx.tag.clear()[callable(**kwargs)]
   return f




posted on 2005-12-21, tagged as nevow twisted python programming

--------------------------------------------------------------------------------


RENDER_IF: CONDITIONAL PARTS IN NEVOW TEMPLATES 2005-12-17



This Nevow renderer has saved me a lot of time:

def render_if(self, ctx, data):
   r=ctx.tag.allPatterns(str(bool(data)))
   return ctx.tag.clear()[r]


Use it like this:

<nevow:invisible nevow:render="if" nevow:data="items">
  <ul nevow:pattern="True"
  nevow:render="sequence">
	<li nevow:pattern="header">The items are a-coming!</li>
	<li nevow:pattern="item">(the items will be here)</li>
  </ul>
</nevow:invisible>


And now, if the list returned by data_items is empty, there will be no <ul> tag
at all in the output.

I just realized non-boolean tests may be wanted -- for example, test if a string
matches a regexp. You could do that by mangling the data before render_if, but
that's not nice, because then you don't have access to the original data inside
nevow:pattern="True". So, instead let's parametrize the test:

def render_ifparam(self, name):
   tester = getattr(self, 'tester_%s' % name, None)

   if tester is None:
	   callable = lambda context, data: context.tag[
			"The tester named '%s' was not found in %r." % (name, self)]
	   return callable

   def f(ctx, data):
	   r=ctx.tag.allPatterns(str(bool(tester(data))))
   return ctx.tag.clear()[r]

   return f


Note how we still cast the return value of the tester to boolean. You could
avoid that and call the renderer render_switch. Adding support for Deferred
tests would be quite easy, too. The only ugly part is I don't know of any way to
make the same renderer work nicely for nevow:render="if" and
nevow:render="ifparam foo".

[Updated to add return f, also renamed second render_if to render_ifparam to
clarify things a bit. Thanks k3mper.]



posted on 2005-12-17, tagged as nevow twisted python programming

--------------------------------------------------------------------------------


MY REVIEW OF TWISTED NETWORK PROGRAMMING ESSENTIALS 2005-11-29



I've just finished my review of the book Twisted Network Programming Essentials
by Abe Fettig. Now go read it .

You may also be interested in an earlier review I wrote about Foundations of
Python Network Programming .



posted on 2005-11-29, tagged as book-review twisted python programming

--------------------------------------------------------------------------------


TURKU-DEV: KEHITTÄJÄTAPAAMINEN TURUSSA 2005-11-22



This entry is about a local software developer gathering, and written in
Finnish. My apologies if it is complete gibberish to you, but atleast you can
stare at the pretty ä dots.


MIKÄ?

Vapaamuotoinen tapaaminen ohjelmistoja työkseen ja/tai harrastuksekseen
tekeville, tai muuten aiheesta kiinnostuneille.

Tutustutaan ihmisiin, puhutaan mukavia, syödään ruokaa. Jos haluat kertoa
hienosta uudesta softasta, jota olet juuri tekemässä, löydät varmaan jonkun
samanmielisen. Jos tarvitset apua hankalaan ongelmaan, joku varmaan on joskus
tehnyt jotain samankaltaista. Eikä ihmisten tunteminen ainakaan haittaa
urakehitystäkään.

Sillä ei ole väliä onko työkalusi C, Perl, Java, Python, Ruby vai PHP; tai
Linux, BSD, OS X vai jopa Windows. Vapaat/avoimet ohjelmistot ovat monelle
meistä tärkeitä, joten niiltä et kokonaan pysty välttymään, mutta tarkoitus on
vain saada samanhenkisiä ihmisiä kokoon.


MISSÄ?

Turun keskustassa oleva ravintola Harald , katso kartta .

Meitä kiinnostaa eniten Turun seudun toiminta, mutta ajatuksia "road showsta" on
heitetty ilmaan, eli jatkossa kehittäjätapaaminen saattaa olla sinunkin
lähikuppilassasi.


KOSKA?

Nyt lauantaina, 26.11. n. klo 12:30 alkaen. Niin pitkään kun intoa riittää.

Seuraava kerta varmaan joskus tammikuussa, ja siitä sitten eteenpäin vaikka
kuukauden tai parin välein.


KUKA?

Tällä tapahtumalla ei ole virallista järjestäjää, eikä se liity minkään
yhdistyksen tjms. toimintaan. Minä aloin asiasta tutuille puhumaan, Tero Kuusela
on tehnyt lähes kaiken valmistelutyön.

Tällä hetkellä aiheesta kiinnostuneiden ihmisten taustoja ja kiinnostuksia:
VSTKY, Linux-Aktivaattori, Debian, Python, Linux kernel, Google Summer of Code,
jne..


TAHTOO!

Liity postituslistalle. Listan osoite on turku-dev@lists.inoi.fi ja liittyminen
tapahtuu lähettämällä viesti osoitteeseen turku-dev-subscribe@lists.inoi.fi ja
vastaamalla vahvistus-pyyntöön.

Ihmismäärän arvioimiseksi pyydän, että ilmoitat tulostasi etukäteen osoitteeseen
Tero Kuusela tero@teroajk.net .



posted on 2005-11-22, tagged as turku-dev finland software-development meeting
lang:fi

--------------------------------------------------------------------------------


THE MOCHIKIT SCREENCAST IS VERY NICE 2005-11-21



"It's simply a more convenient syntax."

"MochiKit is full of more convenient syntax."

The MochiKit screencast is great. I think screencasts are a great way to
introduce people to new software.



posted on 2005-11-21, tagged as javascript ajax twisted python

--------------------------------------------------------------------------------


I REGISTERED AT TECHNORATI.COM 2005-11-21



I just registered at Technorati , and it requires me to post an entry claiming
my Technorati Profile .

..and when I registered an alternate URL for it, they required me to do it all
over again: Technorati Profile .



posted on 2005-11-21, tagged as technorati

--------------------------------------------------------------------------------


NEW WEBSITE TEMPLATE 2005-11-20



Just finished a new website layout. I'm reasonably pleased with it.



posted on 2005-11-20, tagged as web

--------------------------------------------------------------------------------


PYTHON IS CONFUSING 2005-11-02



>>> def simple(): yield 'a'
...
>>> ', '.join(simple())
'a'
>>> def horrible():
...     if ' ' not in False: yield 'a'
...
>>> ', '.join(horrible())
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: sequence expected, generator found
>>>


But it does accept generators!

(Yes, I know what triggers it to say that. It's still horribly misleading.)



posted on 2005-11-02, tagged as python programming

--------------------------------------------------------------------------------


USING NEVOW.GUARD THE SMART WAY 2005-09-16



<ronwalf> ok, I give up... How do I get the AVATAR_LOGIN stuck
between the SessionWrapped resource ul and the current
resource url


Well, we aim to please.

def getActionURL(ctx):
    request = inevow.IRequest(ctx)
    current = url.URL.fromRequest(request).clear()
    root = request.getRootURL()
    root = url.URL.fromString(root)
    assert root is not None
    root = root.pathList()
    me = current.pathList(copy=True)
    diff = len(me) - len(root)
    assert diff >= 0
    action = current
    if diff == 1:
        action = action.curdir()
    else:
        while diff > 1:
            diff -= 1
            action = action.parent()
    action = action.child(guard.LOGIN_AVATAR)
    for element in me[len(root):]:
        action = action.child(element)
    return action


Comment from ronwalf (on IRC) on 2005-09-17T00:06:57:

<ronwalf> Better.  after root = root.pathList()
<ronwalf> if root == ['']: root = []




posted on 2005-09-16, tagged as python twisted programming

--------------------------------------------------------------------------------


TURUXI GATHERING LOOKS A BIT TOO BORING :( 2005-09-16



The Turuxi gathering today looks like a bit too boring for me, personally. I
think I'll go xycle instead.



posted on 2005-09-16

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------