kylelemons.net Open in urlscan Pro
2606:50c0:8000::153  Public Scan

Form analysis
0 forms found in the DOM

Text Content

KyleLemons.net
 * Home
 * Blog
 * Go Projects
 * GitHub


WELCOME

I am a software engineer, and long-time proponent of Go. This site hosts some of
the blog entries I've written over the years, along with details about some of
the projects I've created, and write-ups from interesting problems I've solved.


TEST DRIVING THE NEW GENERICS LINK

edit Kyle Lemons
label Generics,, Go
folder Blog
today 17 June 2020

This is a collection of my thoughts as I examine the latest Go generics
proposal, announced in this blog post.

I don’t come across many times where I think that generics in Go would solve a
problem, but there are a few times where I’ve wanted something like it on
multiple occasions:

 * Channels that represent mutexes.
 * Metrics with arbitrary value and dimension types.


CHANTEX

Chantex, which is my head-canon name for channels that are taking the place of
mutexes, are a thing that I have started doing a lot after spending some time
mulling over Bryan C. Mills’ awesome Rethinking Classical Concurrency Patterns
talk.

I won’t dive too much in detail, but here is what it can look like:

type Server struct {
    state chan *state // chantex
}

func (s *Server) Login(ctx context.Context, u *User) error {
    var state *state
    select {
    case state = <-s.state:
        defer func() { s.state <- state }()
    case <-ctx.Done():
        return ctx.Err()
    }
    return state.addUser(u)
}

type state struct{
    activeUsers map[userID]*User
}

func (s *state) addUser(u *User) error {
    if _, ok := s.activeUsers[u.id]; ok {
        return fmt.Errorf("user %q is already logged in", u.name)
    }
    s.activeUsers[u.id] = u
    return nil
}


There are a number of benefits here over the standard mutex pattern, but here
are my favorite three:

 * You can attach methods to the state type that are very difficult to call
   without “holding” the mutex.
 * You can wait for the mutex with a context, cancellation channel, timeout,
   deadline, etc.
 * It is obvious what data is covered by the mutex.

So, I won’t try to convince you to start using this pattern if you’re not
already inclined to do so, but I wanted to try to see if it could be improved
with generics. Here’s what I came up with:

type Chantex(type T) chan T

func New(type T)(initialValue *T) Chantex(T) {
	ch := make(Chantex(T), 1)
	ch <- initialValue
	return ch
}

func (c Chantex(T)) Lock() *T {
	return <-c
}

func (c Chantex(T)) Unlock(t *T) {
	c <- t
}

func (c Chantex(T)) TryLock(ctx context.Context) (t *T, err error) {
	select {
	case t = <-c:
		return t, nil
	case <-ctx.Done():
		return t, ctx.Err()
	}
}


Using this with the example above, it turns into this:

type Server struct {
	state Chantex(state)
}

func (s *Server) Login(ctx context.Context, u *User) error {
	state, err := s.state.TryLock(ctx)
	if err != nil {
		return fmt.Errorf("state lock: %s", err)
	}
	defer s.state.Unlock(state)
	return state.addUser(u)
}

type state struct {
	activeUsers map[userID]*User
}

func (s *state) addUser(u *User) error {
	if _, ok := s.activeUsers[u.id]; ok {
		return fmt.Errorf("user %q is already logged in", u.name)
	}
	s.activeUsers[u.id] = u
	return nil
}


It’s only two lines shorter, so you’d have to use it over a dozen times before
it makes up for its line count delta. Aside from that, though, it is going to
look a lot more familiar to readers who are familiar with the mutex, and it
doesn’t have the somewhat-unusual-looking anonymous-defer-in-a-select bit.
Luckily, however, with this approach you don’t actually lose the flexibility to
use it in a select if you need something more custom:

func (s *Server) ActiveUsers(ctx context.Context) ([]*User, error) {
	var state *state
	select {
	case state = <-s.state:
		defer func() { s.state <- state }()
	case <-time.After(1*time.Second):
		panic("state: possible deadlock")
	case <-ctx.Done():
		return nil, fmt.Errorf("state lock: %w", ctx.Err())
	}

	var users []*User
	for _, user := range state.activeUsers {
		users = append(users, user)
	}
	return users, nil
}


So, you can actually use this with effectively same code as from before the
Chantex(T) refactor if it makes sense in context.

Overall I am pretty happy with how this one came out. Check out the full code
onthe playground if you’re interested. I think it would be even more useful for
some of the other types discussed in the Rethinking Classical Concurrency
Patterns talk, in particular the Future and Pool types.

The last thing that took me a bit to figure out: how to make a Locker method.
This required me to swap over to requring a pointer type explicitly for Chantex,
when I had originally just made it type Chantex(type T) chan T, but this lines
up with how I normally use it:

type Chantex(type T) chan *T

func (c Chantex(T)) Locker(ptr *T) sync.Locker {
	return locker(T){ptr, c}
}

func (l locker(P)) Lock() {
	*l.ptr = *l.mu.Lock()
}

func (l locker(P)) Unlock() {
	l.mu.Unlock(l.ptr)
}


This seems like a worthwhile change, since it potentially avoids any confusion
about copying values if the underlying implementation is not well-understood,
and it had effectively no change on the caller side of the API.


METRICS

To track data at scale, we have a metrics pipeline that collects data from
running servers. It is similar to prometheus. Each metric can be one of a fixed
number of types (basically numbers, floats, and strings) and can have a variable
number (fixed on a per-metric basis) of dimensions, which can also be of a
(slightly smaller) set of fixed types (strings and ints basically).

Yes, I realize that this example is actually in the generics proposal, but I
wanted to play around with other approaches too.

Code could look something like this:

package mine

import (
    "log"

    "example.com/monitoring/metrics"
    "example.com/monitoring/fields"
) 

var (
    serversOnline = metric.NewStoredIntCounter("servers_online", field.Int("zone"))
)

func init() {
    zone, err := currentZone()
    if err != nil {
        log.Panicf("Failed to determine local zone: %s", err)
    }

    scrape := time.Tick(30*time.Second)
    go func() {
        for {
            <-scrape

            servers, err := countServers()
            if err != nil {
                log.Errorf("Failed to count servers: %s", err)
                continue
            }

            serversOnline.Set(servers, zone)
        }
    }()
}


There are a number of downsides of this approach:

 * The Set method is defined as (m *intMetric) Set(value int, dimensions
   ...interface{})
   * The variadic interface does not provide compile-time type safety
   * The interface boxing is costly when used in tight loops
 * This requires creating specific types for each allowable value
   * Each metric type (int, float64, string, etc) needs its own type
   * Each metric “style” (“counter”, “gauge”, “histogram”, etc) needs its own
     constructor
   * Each field type (int, string, etc) also requires its own constructor and
     type
 * The implementations require using reflection and/or type switches even though
   the API surface is required to enumerate possible value types

In other words, this approach is getting the worst of all worlds: it is not
performant, not type-safe, and it requires lots of copy/pasted code.

I would like to think that generics could solve this problem, and while I think
the current proposal does help, it still doesn’t leave us in what might be the
optimal place. I don’t think it precludes it in the future, however, but let’s
get to it.


ENUMERATED DIMENSION COUNTS

So far, this is the only approach that will actually work under the generics
proposal. As you will see below though, it is not entirely satisfying.

The “generic” version of the (relevant pieces of) the code could look like this
under the current proposal:

serversOnline := metric.NewStoredCounter(int)("servers_online", field.New(string)("zone"))
serversOnline.Set(servers, zone)


Unfortuantely, this doesn’t get us type-safety: the Set method must still be
defined as (m *Metric(T)) Set(value T, dimensions ...interface{}). So, instead,
we could change the API to look more like this:

https://go2goplay.golang.org/p/UPsvxoDw9m6

package metric

type Sample1(type V, D1) struct {
	Timestamp  time.Time
	Value      V
	Dimension1 D1
}

type Metric1(type V, D1) struct {
	Name   string
	Values []Sample1(V, D1)
}

func NewMetric1(type V, D1)(name string) *Metric1(V, D1) {
	return &Metric1(V, D1){Name: name}
}
func (m *Metric1(V, D1)) Set(value V, dim1 D1) {
	m.Values = append(m.Values, Sample1(V, D1){time.Now(), value, dim1})
}


With this approach, we do manage to get type-safety for our Set function. Note
the major downside here though: instead of having to define one top-level type
for each value and dimension type (generics gives us that), we have to define a
new top-level type (two, actually) for each number of arguments… you would also
need this:

https://go2goplay.golang.org/p/zpMDuqd9s-F

package metric

type Sample2(type V, D1, D2) struct {
    Timestamp  time.Time
    Value      V
    Dimension1 D1
    Dimension2 D2
}

type Metric2(type V, D1, D2) struct {
    Name   string
    Values []Sample2(V, D1, D2)
}

func NewMetric2(type V, D1, D2)(name string) *Metric2(V, D1, D2) {
    return &Metric2(V, D1, D2){Name: name}
}
func (m *Metric2(V, D1, D2)) Set(value V, dim1 D1, dim2 D2) {
    m.Values = append(m.Values, Sample2(V, D1, D2){time.Now(), value, dim1, dim2})
}


… and so on and so forth.

Hand-crafting this would likely be onerous, so it would likely require code
generation for the generic implementations at each number of dimensions up to
some maximum number, which seems like it almost defeats the purpose, because you
could then just generate the code for the metric directly.

There are a few kinds of API that I could envision that might work with some
other kind of generic:


“LISP” PATTERN

m := field.Add(int)("age", field.Add(string)("name", metric.New(float64)("height")))
m.Set(age, name, height)


This seems like it should be somewhat possible at first glance, particularly if
you expand the Set method to be something like

m.With(age).With(name).Set(height)


However, this requires the With method to include the proper type for the
sub-function in its type parameters, which is not practical recursively.


“VARIADIC” PATTERN

m := metric.New(float64, string, int)("height", "name", "age")
m.Set(height, name, age)


This pattern is simimlar to the Set problem above: there is no way to write the
Set method, which would require some form of variadic type parameters:

// Set with [] type parameters defines the recursive component of the type.
//
// This is only what a variadic type parameter *could* look like.
func (m *Metric(T,[D0,DN...])) Set(value T, dim0 D0, dims ...DN) *Sample(T, DN...) {
    return m.Set(value, dims...).addDimension(m.d0.name, dim0)
}
// Set without the [] type parameters defines the "base case" for the recursion.
func (m *Metric(T)) Set(value T) *Sample(T) {
    return &Sample(T){Value: value}
}



“BUILDER” PATTERN

m := metric.New(float64)("height").WithField(string)("name").WithField(int)("age")
m.Create().With(name).With(age).Set(height)


The Set is not possible to write for the same reasons as above, and the
WithField method is not possible to write because methods may only include the
type parameters of the type itself.


CONCLUSIONS

Over the course of experimenting with this, I have a few overall impressions:

 * It’s pretty inconvenient to predeclare variables of type T just to get access
   to the zero value. A zero builtin or allowing nil to be used for the zero
   value of type parameters would be useful.
 * It gets very confusing to deal with both pointer- and non-pointer versions of
   a type parameter. In particular, if you want to let the user choose whether
   they’re dealing with a pointer or a value type but intend it to be thought of
   as a reference by using the (type *T)
 * Thinking and reasoning about self-referential, recursive, and/or mutually
   recursive types will melt your brain. This is going to be very important for
   deciding when to use them and when not to use them. I suspect that for the
   time being, we will want to stick to reflection when the readers will not
   already have a good mental model of the type relationships.
 * Type parameters as proposed should be used when the type parameters can be
   inferred unless the types being specified are simple and it is easy to
   understand what the types mean in context. In particular, if the user has to
   specify multiple different versions of a type in a single instantiation
   (think x := F(T1, []T1, T2(T1))(y, z) or something)), it is probably too
   complex.
 * Having interface types that can’t be used anywhere that interfaces are
   allowed feels strange. I am used to Go features being generally orthogonal:
   any feature can be used with any other, and they interact in ways that
   intuitively make sense.

Also, I have to say, I really appreciate that the team got this up and running
on the playground, it enables a ton of fun experimentation. And double kudos for
fixing the bug I found so fast!


MOVED TO GITHUB PAGES LINK

edit Kyle Lemons
label GitHub
folder Blog
today 4 July 2019

Every few years I decide that my current hosting solution isn’t ideal and go
looking for a new one.

This year, I’ve decided to reduce costs (because my website receives so little
traffic) by moving my static content over to GitHub Pages.

Feel free to check out the source if you want to see the content of my old
blog’s posts, but realistically I probably won’t spend the required amount of
time to update them to Jekyll beyond what I’ve done so far (which was to dump
them from the Ghost sqlite database and hackishly template that out to files).

If you’re looking for my CS1372 resources, you can find them here.

The most popular references are these:

 * An old proposal for how generics could look in Go.
 * A Guide to pointers in C


WHY LEAP SECONDS SHOULD WORRY YOU LINK

edit Kyle Lemons
label Time, Postmortems
folder Writeup
today 1 May 2015

Here are just a few examples of leap second woes:

 * Beidou day numbering bugs
 * Leap Second 2012 collection


BIRTH OF A CLUSTER LINK

edit Kyle Lemons
label Docker, Google Cloud, Kubernetes
folder Writeup
today 4 April 2015


> BUILDING BLOCKS
> 
> Using Kubernetes or Google Container Engine there are a few building blocks
> that you need to understand. There are great docs on their sites, but I’ll
> give an overview of them here.

Read more...


STATIC IPS WITH GOOGLE CONTAINER ENGINE LINK

edit Kyle Lemons
label Google Cloud
folder Writeup
today 3 April 2015

The big trick with using a static IP is to specify in your service config

portalIPs:
  - put.your.ip.here


and, in particular, you must not have

createExternalLoadBalancer: true


because that will attempt to create one for you.


GOOGLE CONTAINER REGISTRY WITH GOOGLE APPS LINK

edit Kyle Lemons
label Google Cloud, Docker
folder Writeup
today 31 March 2015

> The Container Registry allows you to easily push your docker images to Cloud
> Storage.
> 
> Nominally, the registry entry for an image will be
> gcr.io/projectname/imagename, where projectname is the name of the project on
> the Developer Console and containername is whatever id you want. At this
> point, however, both of these only reliably seem to support A-Za-z_-.
> 
> TL;DR: If you're using my container script:
> 
> echo REMOTE=gcr.io/projectname/imagename &gt;&gt; container.cfg
> 
> 
> Or, for Google Apps:
> 
> echo REMOTE=b.gcr.io/bucketname/imagename &gt;&gt; container.cfg
> 
> 
> Then, you can simply
> 
> ./container.sh push

Read more...


DOCKER RECIPES LINK

edit Kyle Lemons
label Docker
folder Writeup
today 28 March 2015

> This page is primarily for my benefit, but hopefully somebody else will find
> it useful as well!
> 
> 
> GENERAL NOTES
> 
> 
> OPEN SOURCEABLE DOCKERFILES
> 
> I always like sharing what I do with other people. That’s part of what I love
> about the internet. Thus, I try to make my Dockerfiles reusable by anyone. I
> also don’t want to include anything in my Dockerfiles that I would consider
> sensitive or overly site-specific, like paths to my data volumes. Here are
> some tips on making release-ready Dockerfiles.

Read more...


RX: MY PRESCRIPTION FOR YOUR GO DEPENDENCY HEADACHES LINK

edit Kyle Lemons
label Go, Programming
folder Writeup
today 22 April 2012

> There has been a lot of discussion on the Go Nuts mailing list about how to
> manage versioning in the nascent Go package ecosystem. We muttered about it at
> first, and then muttered about it some more when goinstall came about, and
> there has been a pretty significant uptick in discussion since the go tool
> began to take shape and the Go 1 release date approached. In talking with
> other Gophers at the GoSF meet-up recently, there doesn’t seem to be anyone
> who really has a good solution.
> 
> TL;DR: kylelemons.net/go/rx
> 
> 
> 
> Oddly appropriate, don’t you think?

Read more...


GO: A NEW LANGUAGE FOR A NEW YEAR LINK

edit Kyle Lemons
label Go, Programming, Open Source
folder Blog
today 6 January 2012

> As 2011 makes its way out and 2012 takes its place, it’s time for a bit of
> reflection and a bit of looking forward. I haven’t been writing software
> professionally for particularly long, but I have written software in a number
> of languages (everything from Pascal to Python), and I think we can all agree
> that none of the usual suspects are particularly ideal. If you’re like me, you
> hunker down with your favorite set of language features and write your code in
> as much isolation as possible, so that you can work around or ignore whatever
> problems your language and/or environment cause you. You probably have some
> tools lying around to help you do various things for which the language or
> your editor/environment aren’t well-equipped. You probably don’t even realize
> all of the things that bother you about the language after awhile, because
> it’s the norm: every programmer has to deal with them, it just comes with the
> territory. Please note: I will be comparing Go to other languages extensively
> in this blog post; do not take it to be an indictment of them or even, really,
> as reasons to not use them. I’m simply giving my opinion on their differences
> and why I personally find that Go is a more suitable language for my
> development. I have used all of the languages that I discuss and will continue
> to use them when their particular strengths are required.

Read more...



Generated by Jekyll — Hosted on GitHub Pages — View Source