Re: Shared Metrics Between Workers

[Jeff Utter <jeff.utter@firespring.com>]

 
Thanks for the quick reply. I very much appreciate the insight.

On November 12, 2015 at 7:23:13 PM, Eric Wong (e@80x24.org(mailto:e@80x24.org)) wrote:
> Jeff Utter wrote:  
> > The earliest, promising solution I considered was raindrops, but it looks  
> > as though you need to know all of the possible metrics up front - which  
> > won't necessarily work as prometheus could use metrics based on parameters  
> > which could vary.  
>  
> You don't have to return all the data you'd aggregate with raindrops,  
> though. Just what was requested.  

Just to make sure I understand this correctly though, in order for the metrics to be available between workers, the raindrops structs would need to be setup for each metric before unicorn forks? 

> GDBM (in the stdlib), SQLite, RRD, or any "on-filesystem"[1] data store  
> should work, even. If you do have a lot of stats; batch the updates  
> locally (per-worker) and write them to a shared area periodically.

Are you suggesting this data store would be shared between workers or one per worker (and whatever displays the metrics would read all the stores)? I tried sharing between workers with DBM and GDBM and both of them end up losing metrics due to being overwritten by other threads. I imagine I would have to lock the file whenever one is writing, which would block other workers (not ideal). Out of the box PStore works fine for this (surprisingly). I'm guessing it does file locks behind the scenes.

Right now I'm thinking that the best way to handle this would be one data store per worker and then whatever reads the metrics scrapes them all read-only. My biggest concern with this approach is knowing which data-stores are valid. I suppose I could put them all in a folder based off the parent's pid. However, would it be possible that some could be orphaned if a worker is killed by the master? I would need some way for the master to communicate to the collector (probably in a worker) what other workers are actively running. Is that possible?